The Randomistas just won the Nobel Economics prize. Here’s why RCTs aren’t a magic bullet.

Duncan Green on why RCTs should not be seen as the new magic bullet for evaluating aid and development interventions. 

Lant Pritchett once likened Randomized Controlled Trials (RCTs) to flared jeans. On the way out and soon we’d be wondering what on earth we’d seen in them.

Not so fast. Yesterday, three of the leading ‘Randomistas’ won the Nobel economics prize (before the pedants jump in, strictly speaking it’s the Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel). Congrats to Esther Duflo, who becomes only the second woman to win the prize (yes really, the other was Elinor Ostrom), with Abhijit Banerjee and Michael Kremer.

There will be lots of plaudits in the press, and a fair amount of magic-bulletism about RCTs as the alleged ‘gold standard’ for evidence of what works in development. So, channelling Pulp Fiction, ‘allow me to retort’. For the ‘bah humbug’ corner, here (in reverse chronological order) are some FP2P links to various sceptical views of RCTs going back several years.

Naila Kabeer on Why Randomized Controlled Trials need to include Human Agency (2019)

Quote: ‘RCTs need to acknowledge the central role of human agency in enabling or thwarting project objectives at every stage of the processes they study. It is unlikely they will be able to do this by confining themselves to quantitative methods alone.’

How did the Randomistas get so good at influencing Policy? (2019)

Stefan Dercon (2018) on Duflo and Banerjee: ‘Everything has to be inductive, experimental. Lots of little solutions will move us forward. They have no big theory of what causes low growth, no big questions, just ‘a technocratic agenda of fixing small market failures’. Getting institutions right is not crucial – we can do lots of bad policies in good institutional settings, and lots of good policies in bad institutional settings.

Lant Pritchett v the Randomistas on the nature of evidence – is a wonkwar brewing? (2012)

Quote: ‘‘RCTs are a tool to cut funding, not to increase learning.’  ‘Randomization is a weapon of the weak’ – a sign of how politically vulnerable the argument for aid has become since the end of the Cold War. ‘Henry Kissinger wouldn’t have demanded an RCT before approving aid to some country.’

And a quote from me in that blog: ‘On one side are the ‘best fit’ institutionalists and complexity people, with their focus on path dependence, evolution and trial and error. On the other are the ‘universal law’ experimentalists, offering the illusory certainty of numbers, and (crucially) comfort to the political paymasters seeking to prove to sceptical publics that aid works. It’s hard to see how they can both be right, or happily coexist for long.’ 

Poor Economics – a rich book from Abhijit Banerjee and Esther Duflo (2011)

Quote: ‘But is it a Big Book? Yes in terms of the approach – I think it will leave a lasting impact on its readers in showing the merits of a bottom-up, evidence-driven approach. But not, I think, in terms of content – lots of interesting, surprising facts and analyses, but no one big message. Given their suspicion of grand narratives, I’m sure the authors would be quite happy with that.’

Well, I got that wrong…..

And here’s another economics Nobel, Angus Deaton, on why RCTs aren’t all they’re cracked up to be (2m). 2017 paper with Nancy Cartwright here.

In my experience, those at the centre of a hype cycle are often highly aware of the limitations and weaknesses of their product. It’s the acolytes and spin doctors who remove all the caveats and nuances, and a magic bullet is born. Let’s hope we can generate a better discussion about the strengths and weaknesses of RCTs in response to the Nobel prize.

Please add your own links!

This first appeared on Duncan's blog FP2P.