The International and Historical Evidence Is not Applicable – J.P.G.
JPG: Andrew is also correct in noting that the experience from U.S. programs is limited in number, scope, and duration, motivating him to look across countries and through history for further evidence. Unfortunately, those models are often as limited if not more so in their usefulness. First, the political and cultural contexts are so different in Holland, New Zealand, Chile, and ancient Rome that it is hard to know whether successes or failures of those programs are attributable to universal features of the programs or particular features of those times and places.
This view does not address the use of natural experiments in general, or my own three-pronged comparative method in particular. As I argue on pages 7 through 9, natural experimentation allows researchers in a variety of fields to turn the diversity of historical and international experiences to their advantage, by looking for patterns in system performance that persist despite cultural, economic, and technological variations. The more disparate the range of settings in which a particular school system performs well, the more likely it is that that system is intrinsically well designed.
I’m certainly open to arguments about the validity or applicability of natural experimentation, but clearly natural experimentation is respected and used widely enough that it cannot be reasonably dismissed without so much as an argument.
This Paper Doesn’t Present all the Evidence – J.P.G.
JPG: It took Andrew several hundred pages of careful historical work in Market Education to sort these matters out; something that cannot be replicated in this paper.
Forging Consensus and Market Education were never intended to serve the same purpose. Market Ed was a survey of historical and international case studies that attempted to discern any patterns that might exist across time and place in the effectiveness of different school governance systems. Forging Consensus is a higher level document. It doesn’t try to re-derive the patterns extracted by Market Education, it takes the book’s findings as given and uses them to compare the merits of alternative contemporary policy proposals. That’s why Forging Consensus spends only 4 pages on the territory that Market Education covers in 400 pages, and why the bulk of Forging Consensus is a comparison of vouchers and universal education tax credits.
The way I envisage people reading Forging Consensus, assuming that they have the time and inclination to take it seriously, is to dig down into the referenced citations any time they find a point with which they disagree. For example, if someone does not believe that direct parental financial responsibility is important, I would hope that they would have a look at the supporting references I provide on that point. If they still disagree after reviewing the supporting evidence, c’est la vie.
Historical Evidence Alone Can Be Unpersuasive – J.P.G.
JPG: Even that lengthy historical analysis cannot fully satisfy doubts, making historical and international comparisons usually as limited as U.S. examples that can be studied with more modern social science techniques.
To clarify, my recent survey of the evidence from developing countries makes extensive reference to highly sophisticated modern econometric studies, and that quantitative work is cited in the paper. Market Education also cites modern research on schooling in the United States, England, Canada, Japan, and the U.K., in addition to its historical coverage.
So the judgment calls that each reader has to make are: How reliable is natural experimentation? And, which experimental conditions are most likely to affect the predictive power of a "school choice" study? If you think the consistent performance (either good or bad) of a particular school system across cultures is a strong indication of its merits, then you’ll value surveys of the international and historical evidence more highly. If you think that you can learn more about the long term effects of large-scale school voucher programs by looking at the 90-year-old national voucher program in Holland than at the 14-year-old city-wide program in Milwaukee, then you’ll give serious consideration to the international evidence.