The REF’s Focus on Linear and Direct Impact is Problematic and Silences certain Types of Research
In the last Research Excellence Framework (REF), the new element of research impact was understood in very linear and direct terms. Aoileann Ní Mhurchú, Laura McLeod, Stephanie Collins and Gabriel Siles-Brügge consider how accepted definitions of impact may have had the effect of silencing certain types of research. Research and impact should be seen as a two-way street, where academics engage with their research subjects as part of a process of co-production. Moreover, impact must be thought of as a collective endeavour that captures the broader social and cultural benefits of academic work. The Stern Review recommendations are to be welcomed but whether and how they are adopted remains to be seen.
Research “impact” was the new component of the UK Research Excellence Framework (REF) – a government-sponsored evaluation of research at UK universities last carried out in 2014, with the next iteration foreseen for 2021. According to the Higher Education Funding Council for England, research has “impact” when it has “an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life beyond academia” (emphasis added). But the perceived need for academics to speak beyond the proverbial “ivory tower” is not confined to discussions surrounding the REF, as the existence of this very forum and many other pieces extolling the virtues of academics not “talking only to their peers“ suggest.
Our point as researchers located in the REF Unit of Assessment (UoA) of Politics and International Studies – and as we have argued in our recent Political Studies Review article – is not to criticise the focus on “impact” itself or imply that it necessarily silences certain types of research. Indeed, in the wake of the problematic EU referendum campaign in the UK, it strikes us as crucial that researchers remain engaged beyond the academy. However, we do want to suggest that the definition of “good impact” is linked to the REF process in our field. This may have made it difficult for (and therefore silenced) certain types of research (unable) to fit in as a “safe option”.
We would like to point to the Politics and International Studies UoA sub-panel report which “noted that ‘additive’ case studies – in which a large number of separate, and typically loosely connected, claims to impact were made – were invariably judged less favourably than more clearly focused case studies”. Similarly, in the introduction to its book, The Impact of the Social Sciences: How Academics and Their Research Make a Difference, the LSE Public Policy Group emphasises the importance of “primary” and “measurable” impacts flowing directly from specific academic outputs.
This focus on linear and direct impact is problematic. Firstly, the relationship between research and impact is a two-way street. Academics do not just swoop in from on high with their research and insights to resolve some intractable or “wicked“ policy problem, but rather engage with their research subjects in (co-)producing this work. For example, in his research on EU trade and investment policy (including on the EU-US Transatlantic Trade and Investment Partnership), Gabriel Siles-Brügge has not just acted as an adviser to NGOs and trade unions, but also incorporated their insights into his research.
We also question the emphasis on intentionality built into this linear definition of impact, for example through the “pathways to impact” statements now ubiquitously found on Research Council grant forms. After all, much advancement has been the result of curiosity-driven, user-disinterested research: think of the importance of formal logic and pure mathematics in underpinning the development of computers. Closer to our own field we find how Amartya Sen’s “Capability Approach” to human wellbeing has been highly influential with the UNDP without originally being conceived as a policy-oriented piece of research. Does the impact agenda potentially lead us down the road of only formulating certain types of research questions in response?
Secondly, we want to stress that impact is a collective endeavour and thus often indirect (also argued by John Dunn). One researcher can still have impact by influencing another. In this vein we should recognise that there may be a “division of labour” between academics, with some closer to “users” than others but all contributing to the same enterprise. Stephanie Collins, for example, has worked on the moral philosophical concept of “care ethics” – which may have applications to, for example, the nursing profession, but which might require specialist expertise in that field to translate the findings.
A related criticism of a focus on direct impact is that it misses the broader social and cultural benefits that academic work might have, especially upon our students and the connection they represent between academia and “the real world”. Teaching is explicitly excluded from the HEFCE impact definition. And yet broader critical skills and open-ended thinking we might instil in our students can have important non-academic impacts on society. One example here comes from Aoileann Ní Mhurchú, who developed community-oriented activities outside the classroom as part of her research-based teaching on “citizenship”.
Since our piece was accepted for publication, the “Stern Review” of the REF (an independent piece of research commissioned by the UK Government) has been published. This is likely to feed into a revision of the process for the next REF, and so what it says on impact potentially matters – especially as it describes impact as “one of the success stories of REF2014”. Is it good news from our perspective?
In part. We welcome that Stern is critical of (a) the need to link impact to specific research outputs; (b) the fact that interdisciplinary and collaborative impacts were not sufficiently recognised; and (c) “incentivis[ing] universities to separate inappropriately or dichotomise their research and teaching missions”. We therefore also welcome the ensuing recommendations to broaden the focus of impact case studies to also include “a research activity and a body of work” (no. 6); “institution-level” impact case studies (no. 5); and that “impact case studies should not be narrowly interpreted”, but should also incorporate “impacts on teaching” (no. 7).
The proof of the pudding, as the proverb has it, will be in the eating – and specifically whether and how these recommendations will be adopted for the next REF. How do we define “a research activity and a body of work” without falling back on the existing output-based model – or operationalise Stern’s wider definition of impact in recommendation no.7 without reproducing the biases of clear-cut measurability? For our part, in our PSR article we argue for the need to reflect upon alternative ways of measuring messier forms of (indirect, teaching-oriented and collaborative) impact rather than remaining beholden only to dominant measurement principles that require a clear-cut causal relationship between the academic research and the non-academic community. It is worth noting that while Stern speaks to the problem of equating impact with direct impact, he accepts the basic proposition that it is linear by underscoring that “impact should [still] be based on research of demonstrable quality” (recommendation no.6, emphasis added).
This blog post is based on the authors’ article, “The Present and the Future of the Research Excellence Framework Impact Agenda in the UK Academy: A Reflection from Politics and International Studies”, published in Political Studies Review (DOI: 10.1177/1478929916658918). This post first appeared on the LSE's Impact blog.
Photo credit: <a href="http://foter.com/re/932291">Foter.com</a>
