This is a version of a blog post previously published on Tracey's personal blog in 2018.
It has been interesting to see how impact has become embedded in the academic research community since it was introduced into the UK research excellence framework (REF) in 2014 (for the first time since such research excellence exercises emerged in the 80s).
Having been UoA lead for Business and Management (Derby) in REF2014, being joint UoA lead for REF2021 (Derby) and also having an interest in evaluation practice (my Ph.D. focused on evaluation), I led a University-level session on REF impact.
The slides are available below and other materials I used are linked at the end of this post. I'll attempt to summarise the key messages below:
1. Understand that dissemination and impact are distinct
Impact is not the same as activity or dissemination. If your case study outlines only your inputs, activity and outputs (see impact process attached) and doesn't progress into outcomes and impact (righthand side of the red line) then you are probably not alone but may want to review it. Many REF2014 panels commented that case studies often confused dissemination with impact (UoA25 Education and UoA17 Geography were examples of this). I recently attended the CABS Research Conference where the same issue was noted by former sub-panel members of UoA 19 (now 17 Business and Management). See slides 7 and 8 for more.
2. Don't forget your underpinning research
The REF2014 'Assessment framework and guidance on submissions', quite clearly stated the role of underpinning research to the impact case studies, and there was a section in itself for underpinning research in the impact case template (pages 52-54 of that guidance). Yet, in recent discussions I have had with researchers the role of this underpinning, 2*+, 'excellent' research appears to have been overlooked (I've even attended an external session on REF impact where underpinning research was not mentioned at all!).
According to the REF2014 criteria, underpinning research should make 'a distinct and material contribution' to the impact being claimed. Make sure that the contents of the papers/research outputs are understood and a clear link can be made between them and the impact claimed. Remember, research dating back to 1st January 2000 can be used and the individual researchers don't even need to still be with your institution...perhaps this opens up some opportunities?
3. Revisit every claim made and ensure there is evidence included to support
In the last REF, I did this with a good old highlighter pen. On an almost-there-draft, I highlighted the claims being made and then cross-checked these against the evidence provided. Many REF2014 panels noted that evidence didn't always support the claims being made, making it hard to assess them. Similarly, ensure any testimonies are clear, direct and support the claim, 'we worked with the University and they really helped us' isn't really good enough.
4. Impact pathways are good but idealistic - think past it if needed.
A perfect impact process would see a clear and targeted effort to secure opportunities to conduct research (grant applications for instance) and later make impact. Yet, sometimes things don't go to plan, or great-but-unexpected-opportunities arise which are worth making the most of. I've tried to show this in the Research Impact process below... making room for both proactive and reactive approaches to impact.
5. The importance of record keeping
I learned this the hard way in the last REF. There were few shared computer folders in the department and so lots of knowledge and reports were lost when someone moved on. Pulling together an impact case study wasn't just about joining the dots, I had to find the dots first! Hopefully, the open access movement helps to ensure underpinning research papers are easily retrievable but evaluation reports, details of contacts for testimonies etc are hard to find if you don't know where to look or that they exist. Good records and team collaboration software (shared folders, virtual workspaces etc, specific research impact software) are your friend!
6. Engage in analysis from REF2014
Many institutions and organisations have analysed and summarised impact case studies from 2014. The University of East Anglia have prepared impact case study packs to examine strong and weak case studies in each of their UoAs - these are really useful to understand strengths/weaknesses identified in each UoA, and are available here. (link will open a broken link - use the search bar that appears and search 'Impact case study pack').
Clarity is coming
More details including panel criteria are due from the REF team in summer 2018. The panel criteria were incredibly useful in the last REF particularly in the examples of impact and details of types of evidence that might be included. For updates, it may be worth following REF2021 on Twitter: @REF_2021 or checking for updates on their website: http://www.ref.ac.uk/
Dr Tracey Wond is Head of Research for the College of Business Law and Social Sciences. Tracey led the Business and Management UoA for the University of Derby for REF2014 and is currently co-lead for the same UoA for REF2021. With a keen interest in advancing evaluation practice, Tracey is a steering group member of the UK Evaluation Society Midlands Regional Network which she helped to found.