Font size:
On Sept. 27, 2001, MEA efforts culminated in the public release of a document critical of Mackinac Center for Public Policy education research. Mackinac Center supporters and interested persons have asked the Mackinac Center to explain the union's actions. These remarks were published October 23, 2001.
What Is the Mackinac Center for Public Policy?
The Mackinac Center for Public Policy is a 14-year-old nonprofit, nonpartisan research and educational institute devoted to improving the quality of life for all Michigan citizens by promoting sound solutions to state and local policy questions. The goal of all Center reports, commentaries, and educational programs is to equip Michigan citizens and other decision-makers to better evaluate policy options. Working with scores of scholars and policy experts from public and private institutions in Michigan and elsewhere, the Mackinac Center advances reforms in education, labor law, tax policy, and other disciplines. The advances in education and labor law, in particular, help tens of thousands of children access better educational opportunities and greater quality of life.
Some Organizations Have a Strong Self-Interest to Oppose New Public Policies
But there are organizations that benefit financially and politically from the education and labor status quo of ever-increasing public spending, rules that force teachers to support unions, and laws that prevent parents from freely choosing the safest and best schools for their children. Not surprisingly, those are the organizations that tend to vehemently oppose Mackinac Center research when it suggests that there may be benefits from academic and fiscal accountability, freedom of association for teachers, and more choices for students and their parents. The Michigan Education Association (MEA), the state's largest and wealthiest school employee union, risks losing financial and political clout as policy makers adopt education reforms proposed by the Mackinac Center.
The MEA's self-interest in opposing reform is clear. The Mackinac Center, entirely independent of government funding and forced union dues, neither profits nor is harmed by changes in school choice, privatization, or union membership. But MEA wealth can be directly and immediately affected by any such reforms.
For example, merely making teacher union membership voluntary-as it already is in 10 states-could potentially cost the union millions of dollars it now collects through forced union dues, or cause the union to have to work harder to earn the voluntary support of its members. There is no question that the MEA benefits from the current arrangement that forces hard-working teachers, bus drivers, custodians, and cooks to support the union.
Effectiveness of Mackinac Center Research
The Mackinac Center has a history of effective influence on public policy in education, and that has at times significantly affected MEA finances. In just one instance, a 1993 Mackinac Center study of the MEA's $600 million health insurance subsidiary (Michigan Education Special Services Association, or MESSA) precipitated state regulatory scrutiny that resulted in MESSA having to return $70 million in overcharges to public school districts.
To counter the Mackinac Center's effectiveness, the MEA has over the years followed Mackinac Center publications with contradictory documents of its own, published under the MEA's name. Few independent observers took them seriously.
MEA Forms New Group to Oppose Independent Education and Labor Research
Now the MEA has a different approach. The union created a new organization that appears to be functioning as the MEA's alter ego. The group, the Great Lakes Center for Education Research and Practice, is continuing the MEA's standard practice of publishing opposing positions in response to previous Mackinac Center research. The group's inaugural document is a criticism focusing on fourteen previous Mackinac Center education policy studies.
The new group, which the MEA calls a "think tank," is not explicitly owned by the union like its MESSA insurance subsidiary, but it appears to be effectively under the union's control. The MEA conceived, finances, and dominates the governing board of the new group.
MEA Executive Director Chuck Anderson wrote that "The Great Lakes Center was developed as a concept from an MEA strategic planning process.." and "The MEA Board agreed to provide `seed money' to launch the Center, but to broaden the control and influence beyond MEA."1 The group's ten-member governing board includes seven NEA-affiliated union officials from states in the Great Lakes region, three of whom (including the board's chairman) are also the president, executive director, and a vice president of the Michigan Education Association.2
MEA President Luigi Battaglieri, acting in his alternate role as chairman of the new MEA-created group, stated that the MEA provided "$200,000 to get the Great Lakes Center off the ground."3 Apparently referencing the MEA's desire to counter the influence of Mackinac Center research, Mr. Battaglieri stated, "Frankly, I admire what [the Mackinac Center has] done."4
Weaknesses of the MEA Group's Document
The MEA group's document, "Let the Buyer Beware," focuses almost exclusively on Mackinac Center research that has included critical analyses of public policies supported by MEA officials. The mere facts that the group was conceived by MEA, and has received $200,000 so far from the union, and is chaired by the MEA president, do not necessarily mean that any document it produces is unreliable, even if such a document is an attack on critics of MEA's positions. To assess a document's value, one must also examine the document itself.
"Let the Buyer Beware," even viewed independently of its publisher, is an unpersuasive analysis of Mackinac Center for Public Policy research. The document describes itself as "an analysis of the social science value and methodological quality" of Mackinac Center education policy studies published from 1990 - 2001. It concludes that Mackinac Center education studies are "often inadequate as a basis for formulating education policy."
The document fails to persuade for at least five reasons.
The 41-page document fails to cite a single error in any Mackinac Center study.
The document applies evaluation criteria that are typically used by professional social science and other academic societies to determine what they might print in the journals they read. It is not criteria used by, or generally familiar to, real-life policy makers as they formulate policy.
Subjective application of evaluation criteria amplifies the authors' biases.
The authors failed to subject their document to any peer-review process, while simultaneously stressing the importance of peer reviews and criticizing the Mackinac Center's peer review process.
Evidence suggests that at least one of the document's authors compromised the report's objectivity by forming his conclusions about the value of Mackinac Center research four years before he actually co-wrote the document.
The five major weaknesses are each discussed below.
First, the document fails to identify any errors in any Mackinac Center study. And if an error had been found, the authors could have brought public attention to it by contacting the Mackinac Center. The Mackinac Center actively encourages critique of its work through its "Guarantee of Quality Scholarship," which is printed inside every study.5 The Mackinac Center's guarantee promises to publicly correct, in writing, any error of fact.
To take 41 pages to conclude that certain research is "inadequate," and then fail to identify a single factual error in that research, invites the reader to presume that evidence of other supposed flaws in the research must be substantial, clear, and compelling. But the MEA authors do not show such evidence.
Second, the document's authors used evaluation criteria that are useful for determining if Mackinac Center research would likely be reprinted in academic society journals, rather than actually be useful and accessible to the policy makers who comprise the intended audience of Mackinac Center publications. Academic journals run the gamut from "International Journal of Scholarly Academic Intellectual Diversity" to "The Journal of Mundane Behavior," but it's safe to say that few policy makers study them closely.
Although criteria for papers submitted for peer-reviewed academic journals may be very important and useful to academic professors on a tenure track, it does not follow that Mackinac Center research is incorrect in either its facts or its conclusions. Many influential and intellectually rigorous books, journal articles, policy analyses, reports, and other publications would not be accepted by academic journals; likewise many interesting and useful academic journals sadly go unnoticed by policy makers.
This does not mean that the methodological principles of formal social science research are unimportant, but merely that they are not the sole determinant of value of public policy research and analysis.
Third, the authors of the MEA document subjectively applied the academic journal criteria in a way that amplifies their own biases against Mackinac Center findings. One purpose of the criteria is to screen out evaluators' personal biases. So even if the Mackinac Center's goal were to have its research published in academic journals instead of having it actually be used by policy makers, the academic journal criteria would have to be applied as objectively as possible. The authors failed to do this.
For example, one Mackinac Center study, Unused Capacity in Privately Funded Michigan Schools, was given the lowest possible score on one part of the criteria: "Is the research question significant, and is the work original and important?" Although MEA authors may not desire to know how many open seats exist in privately funded schools, this was an important question for Michigan citizens and policy makers in 1999 and 2000, the period the study was published.
As citizens debated expanded school choice (Proposal 1 of 2000), there was legitimate concern that the demand for openings in private schools would exceed supply. Dozens of news stories included school officials and policy makers debating this question. Only after Mackinac Center researchers conducted a survey of private schools was this question empirically answered.
Subjective application by the authors of the academic journal criteria exists throughout the document.
Fourth, the authors document and criticize the peer-review process for Mackinac Center research, explain at length the importance of peer-reviews, but then fail to peer-review their own "research." Their document reads:
One of the hallmarks of a high quality academic journal is, for example, the strict adherence to the principle of "blind" review. A blind review is one where the reviewer does not know the name(s) of the researchers or the researchers' institutional affiliations. This ensures that quality is not compromised by favoritism or institutional loyalty. This review process is rigorous; a rejection rate of ninety percent is common. Blind reviews go through several revisions; it is unheard of that an article would be accepted for publication without edits and revisions.
This form of peer-review relies on the absolute integrity of the process. Editors of scientific journals are chosen for their reputations as scholars and violations of the process are a matter of great concern. By utilizing this process, social scientists can have confidence in the research published, whether or not they find the conclusions personally to their liking.
As Molnar has pointed out concerning the publication of non peer-reviewed reports, "amid the welter of such reports and the media coverage of them, there seem to be few guideposts for readers to measure the quality of the information that they are receiving." The danger posed by widely publicized non peer-reviewed reports of questionable social science value has been the subject of considerable discussion. Several recent articles and essays have addressed the issue.6
Despite the MEA authors' emphasis on peer reviewing, one co-author stated that their own document was never submitted to a peer review process.7
Neutral peer reviewers, or any kind of peer review, could have added credibility to the MEA document. The MEA-funded group selected as its authors three academics who are active opponents of reforms advanced by the Mackinac Center. Their history includes projects for the National Education Association, of which the MEA is an affiliate, to publish documents that disparage market-based approaches to improve educational quality such as outsourcing, charter schools, and other forms of parental choice.
Perhaps a defense of the failure to conduct a peer review could rest upon the fact that the MEA document was not intended for publication in a peer-reviewed academic journal. But Mackinac Center researchers subject their work to a peer-review process, even though Mackinac Center reports are not written primarily for peer-reviewed academic journals.
The peer-review process is very important. Mackinac Center research is reviewed by experts on its Board of Scholars, and it is submitted for review by outside experts who arrive at conclusions that differ from the Mackinac Center's. Luigi Battaglieri, MEA president, publicly pointed this out-perhaps unintentionally-acting in his role as Great Lakes Center chairman to release the MEA document.
Mr. Battaglieri cited the September 2000 Mackinac Center study, The Cost of Remedial Education: How Much Michigan Pays When Students Fail to Learn Basic Skills, which found that Michigan businesses and post-secondary institutions spend more than $600 million annually to provide basic reading, writing, and arithmetic skills to employees and college students. Mr. Battaglieri suggested that the $600 million figure was erroneous by offering as evidence a quote from Dr. David Breneman in a Lansing State Journal news story.
Dr. Breneman, a University of Virginia researcher who has conducted one of the only national studies on the costs of remedial education, has found that the national cost of addressing the lack of basic skills in public higher education is approximately $1 billion annually. According to Mackinac Center calculations reported in the Lansing State Journal news story, if the results in Michigan were extrapolated to the rest of the nation the total expenditure for remedial education would be over $16 billion per year. Mr. Battaglieri presented this apparent discrepancy as evidence that the Mackinac Center was in error. But a reading of the study itself tells a different story.
Dr. Breneman-despite his disagreement with Mackinac Center policy recommendations -was both a peer-reviewer and contributor to the Mackinac Center study, a fact evidently unknown to Mr. Battaglieri. Contrary to Mr. Battaglieri's interpretation at his news conference, Dr. Breneman's work actually substantiates the Mackinac Center findings.
Dr. Breneman, when peer-reviewing the Mackinac Center study, anticipated that someone might draw flawed conclusions based on comparisons of his work and the Mackinac Center's work, so he wrote in the addendum to the Mackinac Center study:
Prior work, cited by the author [Dr. Jay P. Greene] in the first footnote of the study, notes that William Haarlow and I have estimated the national cost of remedial education to be about $1 billion annually, while Ronald Phipps estimates the number to be closer to $2 billion. In the Executive Summary, which is all many people will read, Greene puts his estimate at "around $16.6 billion." At first glance, one would assume that these different studies have come to wildly different conclusions, but that is not so. Indeed, his figure of $16.6 billion and ours of $1 or $2 billion are comparisons of apples to oranges."8
Dr. Breneman explains the apparently dissimilar findings and then validates the results of the Mackinac Center report:
It should be emphasized that when Greene makes the same precise extrapolation to the United States based on his data for Michigan, he reaches a figure less than ours, $773 million. So on an apples-to-apples basis, we now have three estimates that are relatively close, and in particular, Greene's number and ours are very close.9
Mr. Battaglieri's observation and statements at his news conference provided perhaps the best illustration possible of the importance of a sound peer review process. In this case, the peer review process itself nullified public criticism aimed at a Mackinac Center study.
A fifth weakness of the MEA document involves a cardinal rule of research: an investigator must not begin with his conclusions in mind and then seek evidence to support them. Sadly, one of the document's authors may have crossed that line.
Dr. Alex Molnar, a document co-author, trustee of the MEA's Great Lakes Center, and Arizona State University professor may have formed his conclusions about the Mackinac Center, a tax-exempt institute, four years before he co-wrote the Great Lakes Center document. In the April 1997 issue of New York-based Stay Free! magazine, Dr. Molnar said, "I don't think there ought to be any tax-exempt foundations. F- - - it."10
The context of this interview involved other tax-exempt foundations, some of which support public policy research that Dr. Molnar may consider of little value. But his statement that these foundations should not exist is comparable, for instance, to another tenured professor stating that those who support labor unions or their goals should not exist.
Anyone who would make such a strong statement about the very existence of those who support labor unions, without explanation or retraction, would be considered by most reasonable people unqualified to objectively evaluate the contributions of labor unions, and rightfully so. Dr. Molnar's objectivity regarding the Mackinac Center is similarly impugned.
Perhaps Dr. Molnar was able to set aside his apparently strong bias against the existence of organizations like the Mackinac Center while he assessed the quality of its work. Perhaps the other authors were able to keep his bias from infecting their views and skewing their collective conclusions. Or perhaps these biases are deemed unimportant to the authors. But in any case, an investigator with strong views against an institute must offer a persuasive explanation that his views will not interfere with the objectivity of his review of that institute's work. The document lacks any such explanation or acknowledgement of bias.
Conclusion
These five weaknesses of the document-failure to identify any errors, use of academic journal criteria to draw conclusions about the value of research to policy makers, subjective application of those criteria, lack of a peer review, and predetermined conclusions of a co-author-allow the reader to conclude that it is an unpersuasive critique of Mackinac Center for Public Policy research.
The group that produced the document is conceived, funded, and dominated by the Michigan Education Association, a labor union that stands to lose political clout and millions of dollars if policy makers continue to be influenced by research that shows the benefits of school choice and freedom of association for teachers. The document itself is weak, and it is further weakened by the obvious financial self-interest of its publisher. Although the document offers some specific opportunities for improvement, it is of little value as a guide for evaluating Mackinac Center research.
Matthew J. Brouillette is director of education policy, and Joseph G. Lehman is executive vice president of the Mackinac Center for Public Policy.
These statements are contained in an email sent by Mr. Anderson on Sept. 18, 2001.
Great Lakes Center Board of Trustees, accessible at www.greatlakescenter.org/whoswho.htm, as of Oct. 22, 2001.
Stated by MEA President Luigi Battaglieri at a news conference in Lansing, Michigan on Sept. 27, 2001.
Mackinac Center for Public Policy's "Guarantee of Quality Scholarship" accompanies every Center study. It reads:
"The Mackinac Center for Public Policy is committed to delivering the highest quality and most reliable research on Michigan issues. The Center guarantees that all original factual data are true and correct and that information attributed to other sources is accurately represented.
"The Center encourages rigorous critique of its research. If the accuracy of any material fact or reference to an independent source is questioned and brought to the Center's attention with supporting evidence, the Center will respond in writing. If an error exists, it will be noted in an errata sheet and will accompany all subsequent distribution of the publication, which constitutes the complete and final remedy under this guarantee."
Peter W. Cookson, Alex Molnar, and Katie Embree, "Let the Buyer Beware: An analysis of the social science value and methodological quality of educational studies published by the Mackinac Center for Public Policy (1990-2001), Sept. 2001, p. 9.
Phone conversation with co-author Dr. Peter W. Cookson, Jr., on Oct. 1, 2001.
David W. Breneman, "The Problem in Clear, But Solutions May Vary" in The Cost of Remedial Education: How Much Michigan Pays When Students Fail to Learn Basic Skills, Mackinac Center for Public Policy, September 2000, p. 29.
Interview of Alex Molnar by Jay Huber of Stay Free! magazine; accessible at www.stayfreemagazine.org/archives/13/molnar.html .
The Mackinac Center for Public Policy is a nonprofit research and educational institute that advances the principles of free markets and limited government. Through our research and education programs, we challenge government overreach and advocate for a free-market approach to public policy that frees people to realize their potential and dreams.
Please consider contributing to our work to advance a freer and more prosperous state.
