On the Overuse of Randomized Controlled Trials in the Aid Sector
In development aid administration and research, randomized controlled trials (RCTs) have become extraordinarily popular as an approach for evaluating development projects and programs. The design of an RCT places a village/community within an experimental design. One group is assigned to receive an intervention (i.e. free mosquito nets) while another serves as a control selected for its similarity and comparability to the test group. Baseline surveys or other forms of systematic measurement are carried out before and after to show whether (in fact) a given aid intervention has worked. Such work has uncovered some striking and sometimes counter-intuitive findings.
When it comes to projects that link digital technologies to socio-economic goals (my area of research) there are a number of examples of the application of RCTs. For example, an study of the One Laptop Per Child program in Peru found little improvement in conventional math and language skills (as assessed by standard testing) but measurable improvements in verbal fluency and general cognitive skills. A study of school children in urban India found greater improvements among students who used a computer-assisted math learning program than those who worked with a personal tutor.
The popularity of RCTs relates to an important concern, namely figuring out which aid programs and projects work and how well they work. Aid money is not infinite and the history of such efforts are littered with innumerable failed projects. In the name of accountability and transparency, such highly structured evaluations should ideally keep aid efforts honest and help to elevate successful programs while dispensing with proposed solutions that may have seemed promising but simply don’t deliver. In the NGO world it’s all bundled under M&E (monitoring and evaluation) but seemingly as of late M&E is often synonymous with RCTs.
The rise of RCTs is also a reflection of some influential champions. In particular, the Poverty Action Lab (at MIT) and the work of economists Esther Duflo, Abhijit Banerjee there as well as Dean Karlan at Yale. There are some known limitations to RCTs. Though they may powerfully illuminate effects out in the real world, those effects do not necessarily generalize beyond the studied context. Prominent economist Angus Deacon notes that “It’s like designing a better lawnmower—and who wouldn’t want that? —unless you’re in a country with no grass, or where the government dumps waste on your lawn….RCTs can help to design a perfect program for a specific context, but there’s no guarantee it will work in any other context.” RCTs also bend research toward the measurable, rather than necessarily what is the main matter of interest. For example, all sorts of concepts and issues are not readily or uncontroversially operationalized, such as political freedom, empowerment, corruption, and learning. The already mentioned RCT study of the OLPC in Peru, by applying standardized testing of math and language skills is completely at odds with the philosophy of learning behind the project. OLPC has aimed from the beginning at helping children “learn how to learn.” So why is OLPC being held to standards that it explicitly rejects? The likely reason is that this is what is readily measurable whereas the capacity to “learn how to learn” is not so readily measurable. Even more concerning are some of the efforts to standardize RCT designs into out-of-the-box evaluation procedures for NGOs and aid agencies. This idea arises from the expense and limited supply of experienced, PhD-level researchers who can develop customized and appropriately matched research designs.
Beyond these specific problems of RCTs, the primary purpose to which RCTs have been put is for evaluation work. That is, to measure the consequences and effects of a project or program after it has been conceived and implemented. Such attention to RCTs as the go-to for any and all development related research (academic or applied) means that program innovation work escapes the scrutiny and rigor it also requires. Just as RCTs offer the promise of transparency and accountability, similarly the process of devising such programs in the first place is in need of such attention. Just as evaluation work benefits from being grounded in systematically collected observations out in the world, the same would likewise benefit innovation work. However RCTs are very ill suited for innovation and design research which requires a much more rapid, flexible, and inductive analytical approach; a built-in openness to the discovery of unanticipated possibilities, not a narrow process of deductive reasoning.
In development policy and program development work, there are some well-known approaches that fill this need for a grounded, rapid, and flexible process. Much of it has come under the label of participatory development. Development scholar Robert Chambers, in particular, has consistently argued against the viability of quantitative metrics-driven, top-down, or non-empirical ‘Eureka’ moment approaches to devising novel aid programs. He points to the way experts and scholars function with biases and assumptions that shape the solutions they come up with and that, by misunderstanding the conditions or priorities of receiving populations, easily and frequently lead to project failure. His set of participatory methodological approaches (and what he specifically has labeled Participatory Rural Appraisal) include collaborative community-based exercises such as drawing maps or developing charts. These are all aimed at giving communities a chance to express what they know about their own needs and problems. RCTs by depending upon experimental designs that largely anticipate what there is to find and measure in the field do not formally incorporate such opportunities for discovery.
The emerging field of ICTD (information and communication technology and development) which considers how digital and network technologies may affect development outcomes is beneficially influenced by various design research communities. Approaches from the human-computer interaction (HCI) field and from human-centered design are often put to use by ICTD researchers. Such approaches have been respecified to address the special challenges of remote, rural, low income, and low-literacy populations. My colleague Tapan Parikh has made use of paper prototyping as a low cost and unintimidating way of testing design ideas. Kentaro Toyama (formerly deputy director of Microsoft Research in India) in reviewing methodological developments in ICTD refers to the “Bollywood method” devised by Apala Chavan of Human Factors International as a way to make use of narrative styles familiar to the groups meant to benefit from design projects. These narratives are used to illustrate scenarios about technology use, to present hypothetical situations, and to gain feedback from potential project beneficiaries. Three scholars Tacchi, Slater and Hearn, with support from UNESCO, developed ‘Ethnographic Action Research’ a method for understanding communication tools (digital and non-digital) and communication processes by training and equipping community members to do research themselves, not just to provide data. In all three examples, the authors argue that researcher immersion, living in and among the people in the setting of design intervention, is an essential part of design innovation processes. While such immersion has long been essential to anthropological research, it is interesting to see it taken up by others and specifically by engineers to answer questions of system and interface design.
The field of socio-economic development research and practice seems to be especially prone to focusing on a single solution or approach which may be treated as a panacea and ultimately misapplied. Angus Deacon argues that we must “take the halo off of RCTs.” Development work has also often treated technology deterministically, as singularly responsible for positive developmental impact. RCTs offer value in unmasking projects (including technological interventions) that fail to realized impact. However, RCTs alone do not equally address the needs of program evaluation and program innovation and design work.
