Dr Tracey Wond, UK Evaluation Society Treasurer, Director and Evaluation Consultant at HEIER, and published evaluation expert muses over 'evaluation' within Michael Gove's Ditchley Annual Lecture.
I was enraged when he famously professed that people had had 'enough of experts' during Leave EU campaigning (although pretty impressed that researchers at the University of Sheffield even went as far as to find out if this were true - they concluded it wasn't)
And so, I was somewhat surprised to hear of Gove's words at the Ditchley lecture last weekend (29th June 2020). The speech can be read here. In short, it really does advocate for evaluation.
What did Gove say?
Here are some extracts (in blue):
At the heart of our programme must be a focus on what works – what actually helps our fellow citizens to flourish.
- As evaluators we've been on this quest for some time.
And that means, as I have emphasised, rigorous evaluation of Government programmes. What value do they add? What incentives do they provide for better performance and better service to others?
- As Treasurer of the UK Evaluation Society, evaluator and advocate for all things 'good evaluation', I welcome this statement. I'm less sure about the 'incentives' part - whilst performance measurement can be a motivator they can also lead to game-playing, and (the feature of several of my publications to date) when we make the stakes of an evaluation higher we tend to see more politicising and evaluation challenges. We also need to be mindful that doing evaluation is one thing - using it is another.
Government needs to be rigorous and fearless in its evaluation of policy and projects. And in doing so, we need to ask not only questions about spending per se, but about effectiveness against ambition.
- I like the fearless here. How can we endeavour to be fearless evaluators? It's got a certain ring to it. I also like pitching 'effectiveness' against 'ambition'.
I didn't like that this statement suggests this isn't happening in Government to date especially allied with his later point that 'all across government at the moment that widespread rigour is missing' - there is a great deal of good work happening and there are many members of UKES who remind us of that, through their conference, article, and webinar contributions regularly. These may not be all about Bayesian statistics or RCTs. Although to be fair, Gove does go on to give credit to some in the civil service:
There are many brilliant people in our civil service, and I have never come across any civil servant who did not want to do his or her best for the country. But, nevertheless, there are a limited number, even in the Senior Civil Service, who have qualifications or expertise in mathematical, statistical and probability questions – and these are essential to public policy decisions. As governments in developed nations go, we in the UK are lagging behind many others in terms of numerical proficiency. But so many policy and implementation decisions depend on understanding mathematical reasoning...That means we need to reform not just recruitment, but training. We need to ensure more policy makers and decision makers feel comfortable discussing the Monte Carlo method or Bayesian statistics
- BAMMM!!?!?- a smack in the face for any qualitative evaluator (don't worry we'll be there to figure out why it worked when you need us!). A clear preference for quants.
Although, it's good to see capability building and training being advocated here and I really hope that the UK Evaluation Society can step up to support this to happen across evaluation.
What are the metrics against which improvement will be judged? How are appropriate tools such as randomised controlled trials being deployed to assess the difference being made? How do we guard against gaming and confirmation bias?
- We're no strangers to the growing preference for randomised controlled trials (whether we like it or not - as a qualitative researcher you can probably guess my reaction to that).
What does this mean for evaluation?
With my UK Evaluation Society hat on, it's good that evaluation has been increasingly mentioned by politicians and across mainstream media (COVID related updates have also brought evaluation to the fore a little more). It reinforces our purpose as the UK community for those involved in evaluation. However, we must be mindful that evaluation isn't politicised more than it ought to be. Whilst, it's well recognised that evaluation is inherently political, there are risks that it becomes gamed or a mere symbolic action.
Gove's speech clearly makes a call to plug the capabilities gap and so there are opportunities here for the evaluation community to step up to this (with a clear steer on the quants!).
However, I think there's also a role for qualitative and mixed methods researchers to step-up, to call out the limitations of the methodologies being used, and to continue to advocate the strengths of various qualitative and mixed approaches.
So, as it turns out, we do need experts - albeit ones who are good with number-crunching!
p.s. to my friends undertaking evaluation in government - I'm in no doubt that your work is already rigorous - keep going!
To learn more about the UK Evaluation Society visit www.evaluation.org.uk
Views are my own.