The Michigan Senate has already passed and the House is considering a major expansion of Michigan’s corporate welfare regime. The House Tax Policy Committee held a hearing on March 8, and two officials from the agencies in charge of current business subsidy programs came to explain and justify the state’s corporate welfare apparatus.

Rep. Jim Tedder had some questions about current programs. In particular, about so-called “return on investment” claims made on behalf of one of them (taxpayer-funded “Pure Michigan” ads), which were generated by a consultant called Longwoods International who has been repeatedly hired by the agency and — until recently — under a no-bid contract.

Tedder asked, “[T]he statistic that’s provided — the $7.67 return for every $1.00 — how is that quantified? What is the process in deriving that number?”

The officials were Steve Arwood, chief executive officer of the Michigan Economic Development Corporation and Jeremy Hendges, chief deputy director of the Michigan Department of Talent and Economic Development. Jeremy Hendges fielded the response, a truncated and edited version of which is below (see the embedded video for the entire answer).

“That is the number from Longwoods that does the study on the return from the investment essentially and it is based on state and local tax dollars that come back and Longwoods has been studying the program probably from day one. We have contracted that out a few times. Their methodology was verified a couple years ago. We had Michigan State [University] double check their work essentially and verify their methodology for the study on that.”

The response didn’t answer the question as directly as it was asked, if at all. It was almost a “nonresponse response,” and may leave the reader wondering why a top official at the MEDC can’t provide an answer to this seemingly basic question.

Hendges failed to provide a specific, detailed answer because he probably just does not know the answer. Longwoods refuses to provide meaningful details of the methods they used to create the ROI figure for Pure Michigan. So no one knows exactly how the figure is calculated, which the MEDC knows and seems perfectly comfortable with (or, if MEDC officials do know, they respect Longwoods’ wishes and keep it hidden from the public). The Mackinac Center has written extensively about a lack of transparency in the Pure Michigan and other programs.

Moreover, the MSU review Hendges references was actually performed by another no-bid contractor called Certec, Inc. This company was owned by a former vice president of Longwoods who coauthored the report with an MSU scholar. The report covers an ad campaign in 2003, which predates the Pure Michigan program by several years.

The companies that produce responses that government agencies want to hear are part of a cottage industry that exists to serve the “budget justification” needs of ineffective state economic development agencies. Their products are nothing more than PR puffery garbed in scholarly robes. Here is a relevant excerpt from a previous blog where I explain why the MSU validation study doesn’t validate the Pure Michigan program’s ROI claims:

Who Watches the Watchers?

So it is interesting that in 2011, when pressed on similar questions about earlier Longwoods’ reports, the then-head of Michigan’s travel promotion bureau pointed to a “validation study” for which the MEDC paid a different contractor $5,000. While this was supposed to confirm that the Longwoods methodology was valid, it also suggests the agency knew these reports had a credibility problem.

Moreover, even this second vendor said Longwoods had a “lack of total transparency,” and acknowledged that reviewing its process would “require costly replication to totally confirm” that its claims were valid. But even if cost were no object, Longwoods’ secrecy would make confirming its claims impossible.

Dodgy Methods Acknowledged

We do know some things about the method, although the details provide little comfort. Bill Siegel, founder and CEO of Longwoods, has said his company uses a hybrid system based on what are called “conversion” and advertising tracking studies. The former are said to measure how many people responded to a travel ad by later visiting the destination.

Nevertheless, traditional conversion study methods appear to have been what the validation study authors used for their work. Conversion studies have been widely criticized by academic economists, who consider them highly dubious. In his book, “Tourism Analysis: A Handbook,” author Stephen Smith writes that — especially with tax-funded travel promotion agencies — “Conversion studies are increasingly popular … as a tool to justify budgets. Because of survival motivation, many conversion studies fall short of full adherence to rigorous scientific procedures.” Smith wrote this was especially true of tax-funded agencies that promote tourism interests.

Even Siegel (writing with colleague William Ziff-Levine) said in 1990 that, “The main value of a conversion study would appear to be to get diagnostic information on a fulfillment piece, not to provide hard numbers on the bottom-line return on-investment of a campaign.” More recently, he said that this hybrid approach uses the parts of the conversion study system that he believes are its most useful and effective.

And then there is the secrecy. While you can read an online copy of the survey Longwoods used to produce its 2014 report, it is meaningless without a detailed explanation of how the company used the survey to derive its claim of a 587 percent return from an ad campaign. And such an explanation is exactly what you won’t find on any Longwoods website or published document.

The MEDC’s validation contractor was supposed to evaluate the legitimacy of Longwoods’ claim. Notwithstanding Longwood’s lack of transparency, this second outfit chose to attempt its own version of a conversion study.

Process Fails Smell Test

As if doubts about the underlying contract weren’t enough, there are many reasons to question the practice of paying a second contractor to validate the claims of a “budget justification” contractor.

To begin with, the MEDC did not issue a request for proposals, which could have given it a number of validation contractors to choose from. Instead, agency officials hand-picked a contractor, which at the very least limited the scope of possible perspectives. Also, it’s hard to ignore that the co-author of the validation study is a former vice president of Longwoods International, the company being evaluated.

That revolving-door relationship is part of a pattern. The MEDC official who mentioned the validation study in 2011 was George Zimmerman, whose title was vice president of the MEDC's Travel Michigan office, which oversees state promotion efforts. Mr. Zimmerman has since been hired by Longwoods International as its chairman.

The questions multiply when details of this second contractor’s work are examined. For instance, the survey research it used involves asking respondents to report the degree to which the state of Michigan’s advertisements influenced their decision to visit. This introduces the possibility of response bias — a pitfall of survey research — which would influence the output. Moreover, the authors simply lump in two categories of responses at different rates before making their calculations. They claim this is a common technique but offer no supporting evidence. Curiously, one result derived by the validation authors apparently matched the Longwoods International conclusions to the penny ($3.42).

Confirmation Study Confirms Nothing

Perhaps not surprisingly, the validation study authors so thoroughly hedged and qualified their report that they appear to doubt whether “return on investment” claims like Longwoods’ can actually be measured. In one place they wrote, “The ‘real’ conversion rate in [sic] unknown and next to impossible to derive given the many variables involved, many beyond the control of the researcher.”

Elsewhere, contractor No. 2 admitted, that “the possible confidence interval for our estimate is wide,” and that “our bottom line estimate will likely be inflated to the degree that other Michigan advertisers were also engaged in these markets at this time at a level beyond the precision of this estimate.” Translation: They came up with a ballpark estimate in a very large ballpark. Still, the contractor suggests that even if they’re off by 25 percent the result is still positive.

The outfit that was paid $5,000 to prepare this report can at least be commended for admitting they don’t know what they don’t know.

Perhaps capping all the preceding, 2011 press reports describe former MEDC and current Longwoods International official George Zimmerman repeatedly pointing to the validation study to support his company’s claims about MEDC ad spending. However, the validation study contract was entered into in 2007 to cover a Longwoods’ report on a 2003 tourism ad campaign. Published reports indicate that the state had validated Longwoods’ technique but failed to mention that the validation work involved ads unrelated to the Pure Michigan campaign.

Representative Jim Tedder asked a pointed and good question and got an empty response. He and his colleagues should remember that the next time corporate welfare legislation or its proponents are asked to approve spending taxpayer dollars on it.

Permission to reprint this blog post in whole or in part is hereby granted, provided that the author (or authors) and the Mackinac Center for Public Policy are properly cited.