Gary Banks on evidence-based policy

Evi­dence-based pol­i­cy is hard. The art is long, but every­thing else—personnel, time, inde­pen­dent analy­sis, the funds to find, and the will to act on, evidence—is short. Politi­cians, itch­ing to do some­thing often seek ‘pol­i­cy-based evi­dence’ for some unique ‘solu­tion’. But poli­cies based on hunch­es, sen­ti­ment or dog­ma can lead to cost­ly errors that are dif­fi­cult to recover.

All policy is experiment

All pol­i­cy effec­tive­ly is exper­i­men­ta­tion. But that does not mean fly­ing blind… pol­i­cy exper­i­ments need to be mon­i­tored and eval­u­at­ed and, over time, cor­rect­ed or ter­mi­nat­ed if they turn out to be fail­ures. These are things that Gov­ern­ments typ­i­cal­ly find hard to do—particularly the ter­mi­na­tion part.”

Applied to cli­mate pol­i­cy, for exam­ple, this ‘exper­i­men­tal’ approach argues for small exper­i­ments first:

The only sen­si­ble way for­ward, there­fore, is to start grad­u­al­ly, to mon­i­tor, to learn by doing as we devel­op insti­tu­tions and see the effects of car­bon pric­ing on our econ­o­my and com­mu­ni­ty, and as we wait for oth­ers to come to the party—in oth­er words, an adap­tive response.”

Aside I: This is an argu­ment also made by anoth­er of my favorite econ­o­mists, John Kay about suc­cess­ful pub­lic pol­i­cy or busi­ness strat­e­gy (see, for exam­ple, this recent arti­cle on The Great Leap For­ward in British health records).

Aside II: The tell­tale of a rich idea like this is that you can’t help think­ing of more appli­ca­tions. How would this approach apply in the case of the macro-eco­nom­ic stim­u­lus pack­ages now being debat­ed?

The heart of Gary Banks’ speech is the sec­tion on the Essen­tial Ingre­di­ents of evi­dence-based pol­i­cy mak­ing. Here’s a Read­ers’ Digest sum­ma­ry of the argument:

Methodology matters

In sit­u­a­tions where gov­ern­ment action seems war­rant­ed, a sin­gle option, no mat­ter how care­ful­ly analysed, rarely pro­vides suf­fi­cient evi­dence for a well-informed pol­i­cy decision.”

Most evi­dence-based method­olo­gies fit broad­ly with­in a cost-ben­e­fit (or at least cost effec­tive­ness) frame­work, designed to deter­mine an esti­mat­ed (net) pay­off to soci­ety… but [they haven’t] been all that com­mon­ly or well used, even in rel­a­tive­ly straight­for­ward tasks such as infra­struc­ture project evaluation.”

Good data is a prerequisite

…Aus­tralia has been very well served by the Aus­tralian Bureau of Sta­tis­tics and the integri­ty of the nation­al data­bas­es that it has gen­er­at­ed. But in some areas we are strug­gling. Apart from the chal­lenges of valu­ing impacts, and dis­en­tan­gling the effects of simul­ta­ne­ous influ­ences, we often face more basic data defi­cien­cies. These are typ­i­cal­ly in social and envi­ron­men­tal rather than eco­nom­ic domains, where we must rely on admin­is­tra­tive collections—or indeed there may be no col­lec­tions at all”

A major fail­ing of gov­ern­ments in Aus­tralia, and prob­a­bly world-wide, has been in not gen­er­at­ing the data need­ed to eval­u­ate their own pro­grams. In par­tic­u­lar, there has been a lack of effort to devel­op the base­line data essen­tial for before-and-after comparisons. “

Real evidence is open to scrutiny

… no evi­dence is immutable. If it hasn’t been test­ed, or con­test­ed, we can’t real­ly call it ‘evi­dence’. And it miss­es the oppor­tu­ni­ty to edu­cate the com­mu­ni­ty about what is at stake in a pol­i­cy issue…”

Good evidence requires good people

You can’t have good evi­dence, you can’t have good research, with­out good peo­ple. Peo­ple skilled in quan­ti­ta­tive meth­ods and oth­er analy­sis are espe­cial­ly valu­able. It is there­fore iron­ic that we appear to have expe­ri­enced a decline in the num­bers with such skills with­in the Pub­lic Ser­vice at the very time when it has been called upon to pro­vide an evi­dence-based approach that relies on them.”

Independence can be crucial

Inde­pen­dence is even more impor­tant when deal­ing with tech­ni­cal research than with opin­ions. Peo­ple are bet­ter able to judge opin­ions for them­selves, but the aver­age per­son is nat­u­ral­ly mys­ti­fied by tech­ni­cal research. “

A receptive policy-making environment

…The final and most impor­tant ingre­di­ent on my list. Even the best evi­dence is of lit­tle val­ue if it’s ignored or not avail­able when it is need­ed. An evi­dence-based approach requires a pol­i­cy-mak­ing process that is recep­tive to evi­dence; a process that begins with a ques­tion rather than an answer, and that has insti­tu­tions to sup­port such inquiry.”

You can down­load the paper as a PDF file (about 100kb) from the Pro­duc­tiv­i­ty Com­mis­sion website.


p>Update: David Uren in the Aus­tralian has gone through the speech with a high­lighter mark­ing every sen­tence that could be con­sid­ered con­tro­ver­sial. All the quib­bly bits in one place. Worth a look, of course, but not real­ly the fla­vor of the speech.

Leave a Comment

Your email address will not be published. Required fields are marked *