Gary Banks on evidence-based policy

Evidence-based pol­icy is hard. The art is long, but every­thing else—personnel, time, inde­pen­dent analy­sis, the funds to find, and the will to act on, evidence—is short. Politi­cians, itch­ing to do some­thing often seek ‘policy-based evi­dence’ for some unique ‘solu­tion’. But poli­cies based on hunches, sen­ti­ment or dogma can lead to costly errors that are dif­fi­cult to recover.

All pol­icy is experiment

All pol­icy effec­tively is exper­i­men­ta­tion. But that does not mean fly­ing blind… pol­icy exper­i­ments need to be mon­i­tored and eval­u­ated and, over time, cor­rected or ter­mi­nated if they turn out to be fail­ures. These are things that Gov­ern­ments typ­i­cally find hard to do—particularly the ter­mi­na­tion part.”

Applied to cli­mate pol­icy, for exam­ple, this ‘exper­i­men­tal’ approach argues for small exper­i­ments first:

The only sen­si­ble way for­ward, there­fore, is to start grad­u­ally, to mon­i­tor, to learn by doing as we develop insti­tu­tions and see the effects of car­bon pric­ing on our econ­omy and com­mu­nity, and as we wait for oth­ers to come to the party—in other words, an adap­tive response.”

Aside I: This is an argu­ment also made by another of my favorite econ­o­mists, John Kay about suc­cess­ful pub­lic pol­icy or busi­ness strat­egy (see, for exam­ple, this recent arti­cle on The Great Leap For­ward in British health records).

Aside II: The tell­tale of a rich idea like this is that you can’t help think­ing of more appli­ca­tions. How would this approach apply in the case of the macro-economic stim­u­lus pack­ages now being debated?

The heart of Gary Banks’ speech is the sec­tion on the Essen­tial Ingre­di­ents of evidence-based pol­icy mak­ing. Here’s a Read­ers’ Digest sum­mary of the argument:

Method­ol­ogy matters

In sit­u­a­tions where gov­ern­ment action seems war­ranted, a sin­gle option, no mat­ter how care­fully analysed, rarely pro­vides suf­fi­cient evi­dence for a well-informed pol­icy decision.”

Most evidence-based method­olo­gies fit broadly within a cost-benefit (or at least cost effec­tive­ness) frame­work, designed to deter­mine an esti­mated (net) pay­off to soci­ety… but [they haven’t] been all that com­monly or well used, even in rel­a­tively straight­for­ward tasks such as infra­struc­ture project evaluation.”

Good data is a prerequisite

…Aus­tralia has been very well served by the Aus­tralian Bureau of Sta­tis­tics and the integrity of the national data­bases that it has gen­er­ated. But in some areas we are strug­gling. Apart from the chal­lenges of valu­ing impacts, and dis­en­tan­gling the effects of simul­ta­ne­ous influ­ences, we often face more basic data defi­cien­cies. These are typ­i­cally in social and envi­ron­men­tal rather than eco­nomic domains, where we must rely on admin­is­tra­tive collections—or indeed there may be no col­lec­tions at all”

A major fail­ing of gov­ern­ments in Aus­tralia, and prob­a­bly world-wide, has been in not gen­er­at­ing the data needed to eval­u­ate their own pro­grams. In par­tic­u­lar, there has been a lack of effort to develop the base­line data essen­tial for before-and-after comparisons. “

Real evi­dence is open to scrutiny

… no evi­dence is immutable. If it hasn’t been tested, or con­tested, we can’t really call it ‘evi­dence’. And it misses the oppor­tu­nity to edu­cate the com­mu­nity about what is at stake in a pol­icy issue…”

Good evi­dence requires good people

You can’t have good evi­dence, you can’t have good research, with­out good peo­ple. Peo­ple skilled in quan­ti­ta­tive meth­ods and other analy­sis are espe­cially valu­able. It is there­fore ironic that we appear to have expe­ri­enced a decline in the num­bers with such skills within the Pub­lic Ser­vice at the very time when it has been called upon to pro­vide an evidence-based approach that relies on them.”

Inde­pen­dence can be crucial

Inde­pen­dence is even more impor­tant when deal­ing with tech­ni­cal research than with opin­ions. Peo­ple are bet­ter able to judge opin­ions for them­selves, but the aver­age per­son is nat­u­rally mys­ti­fied by tech­ni­cal research. “

A recep­tive policy-making environment

…The final and most impor­tant ingre­di­ent on my list. Even the best evi­dence is of lit­tle value if it’s ignored or not avail­able when it is needed. An evidence-based approach requires a policy-making process that is recep­tive to evi­dence; a process that begins with a ques­tion rather than an answer, and that has insti­tu­tions to sup­port such inquiry.”

You can down­load the paper as a PDF file (about 100kb) from the Pro­duc­tiv­ity Com­mis­sion website.


p>Update: David Uren in the Aus­tralian has gone through the speech with a high­lighter mark­ing every sen­tence that could be con­sid­ered con­tro­ver­sial. All the quib­bly bits in one place. Worth a look, of course, but not really the fla­vor of the speech.

No Comments

Leave a Reply

Your email is never shared.Required fields are marked *