Evidence-based policy is hard. The art is long, but everything else—personnel, time, independent analysis, the funds to find, and the will to act on, evidence—is short. Politicians, itching to do something often seek ‘policy-based evidence’ for some unique ‘solution’. But policies based on hunches, sentiment or dogma can lead to costly errors that are difficult to recover.
All policy is experiment
“All policy effectively is experimentation. But that does not mean flying blind… policy experiments need to be monitored and evaluated and, over time, corrected or terminated if they turn out to be failures. These are things that Governments typically find hard to do—particularly the termination part.”
Applied to climate policy, for example, this ‘experimental’ approach argues for small experiments first:
“The only sensible way forward, therefore, is to start gradually, to monitor, to learn by doing as we develop institutions and see the effects of carbon pricing on our economy and community, and as we wait for others to come to the party—in other words, an adaptive response.”
Aside I: This is an argument also made by another of my favorite economists, John Kay about successful public policy or business strategy (see, for example, this recent article on The Great Leap Forward in British health records).
Aside II: The telltale of a rich idea like this is that you can’t help thinking of more applications. How would this approach apply in the case of the macro-economic stimulus packages now being debated?
The heart of Gary Banks’ speech is the section on the Essential Ingredients of evidence-based policy making. Here’s a Readers’ Digest summary of the argument:
“In situations where government action seems warranted, a single option, no matter how carefully analysed, rarely provides sufficient evidence for a well-informed policy decision.”
“Most evidence-based methodologies fit broadly within a cost-benefit (or at least cost effectiveness) framework, designed to determine an estimated (net) payoff to society… but [they haven’t] been all that commonly or well used, even in relatively straightforward tasks such as infrastructure project evaluation.”
Good data is a prerequisite
“…Australia has been very well served by the Australian Bureau of Statistics and the integrity of the national databases that it has generated. But in some areas we are struggling. Apart from the challenges of valuing impacts, and disentangling the effects of simultaneous influences, we often face more basic data deficiencies. These are typically in social and environmental rather than economic domains, where we must rely on administrative collections—or indeed there may be no collections at all”
“A major failing of governments in Australia, and probably world-wide, has been in not generating the data needed to evaluate their own programs. In particular, there has been a lack of effort to develop the baseline data essential for before-and-after comparisons. “
Real evidence is open to scrutiny
“… no evidence is immutable. If it hasn’t been tested, or contested, we can’t really call it ‘evidence’. And it misses the opportunity to educate the community about what is at stake in a policy issue…”
Good evidence requires good people
“You can’t have good evidence, you can’t have good research, without good people. People skilled in quantitative methods and other analysis are especially valuable. It is therefore ironic that we appear to have experienced a decline in the numbers with such skills within the Public Service at the very time when it has been called upon to provide an evidence-based approach that relies on them.”
Independence can be crucial
“Independence is even more important when dealing with technical research than with opinions. People are better able to judge opinions for themselves, but the average person is naturally mystified by technical research. “
A receptive policy-making environment
“…The final and most important ingredient on my list. Even the best evidence is of little value if it’s ignored or not available when it is needed. An evidence-based approach requires a policy-making process that is receptive to evidence; a process that begins with a question rather than an answer, and that has institutions to support such inquiry.”
You can download the paper as a PDF file (about 100kb) from the Productivity Commission website.
p>Update: David Uren in the Australian has gone through the speech with a highlighter marking every sentence that could be considered controversial. All the quibbly bits in one place. Worth a look, of course, but not really the flavor of the speech.