1. Introduction
2. Why you might use a co-creation approach to evaluation
3. What do we mean by co-creation?
4. How do you co-create an evaluation purpose and process?
5. Conclusion
6. Doing an 'after action' review
7. Key questions
8. Glossary of terms
9. Further reading and resources

Introduction

Do you want to move beyond evaluation that ‘extracts’ data from participants to turn it into neat conclusions that sit neatly on a desk? Do you want to recognise the different perspectives of different stakeholders you work with? Do you want to change your evaluation from reporting your impacts to learning together?

If so this guide provides guidance for developing an approach that is rooted in co-creation.

You can introduce this at all stages of an evaluation process, from planning, to action, to review, so that your evaluation becomes a shared learning journey. This helps build in reflection and ownership, leading to action.

This guide may be helpful if you want to:

• involve the people taking part in your project or work in the evaluation and learning, whatever role they play
• encourage those involved to find their own meaning and learning as part of contributing to the evaluation process
• include participant voices in the process, the analysis and the deliverables – and in the actions or changes that might follow

Why you might use a co-creation approach to your evaluation

Many of the frustrations expressed about evaluation come from it feeling unequal and extractive. This can feel very separate from the kind of connections that experiencing or taking part in art or heritage can create.

Co-created evaluation is more of a collaborative, peer-learning process, that can flex to your particular circumstances. If you see your work as working with a specific community or communities of people rather than on them or even for them, you might want to think about how you can also learn with them, rather than report their experience to others on their behalf.

This might seem easier when you already have an identified set of participants or potential beneficiaries, such as a group of users in a particular setting, or a group of collaborators, with whom you have an ongoing relationship, or would like to establish one.

But if you don’t know exactly who will take part in your project when you start planning, you can still bring a spirit of co-creation to the work at key points, by consciously reaching out to people you are not already talking to and trying to capture different perspectives on your work.

A co-created learning process can be a powerful way to build trust in each other, one of the key factors of building or maintaining long-term collaboration.

What do we mean by co-creation?

Co-creation is such a powerful idea, the word has become something of a buzzword in recent years. It is now used in lots of different contexts. A recent report for Arts Council England, by Heart of Glass and Battersea Arts Centre, explored different views of co-creation. They concluded that it was a process and a methodology where responsibility, authority and agency are shared, with people working with others as equal but different contributors. They identified a set of principles for co-creation that included being transparent, honest and fair; based on trust, respect and care. This requires that we ensure appropriate time and resources are in place; good communications; equality and reciprocity; and clarity on aims and expectations.

These are all good principles to reflect on when co-creating your evaluation, or indeed for any evaluation. They are very much in tune with the Centre for Cultural Value’s argument that evaluation should be Beneficial, Robust, People-centred and Connected.

So how do you co-create an evaluation purpose and process?

I argue for approaches to building creative community which are centred on three key verbs: Connect, Collaborate and Multiply. I first wrote about this in a report on leadership for Creative People and Places, and also cover it in more detail in my new book, Tactics for the Tightrope. For me this approach builds trust, relationships and increases the chance of more equal flows of opportunity and learning. By working together, especially with people with different perspectives and skills, we can mutually amplify learning and bring others in as a result.

I try and bring this spirit to all my evaluation work, wherever possible, to encourage degrees of co-creation, with funders, workers and where possible beneficiaries or participants. This may include co-creating the purpose of the evaluation, although in many circumstances, it may be more about giving space for adapting and evolving as the evaluation develops and participants become clear.

For evaluation to be robust and beneficial it needs to accommodate differences of opinion about experience and evidence, rather than smooth them over to produce simple messages. Learning, like life, is often messy and contradictory. Including different voices in the reports and the data considered, and in the reflection on it is important. Ideally, everyone should hear those different views – not just the evaluator(s) in one-to-one interviews – to maximise the opportunities for shared learning. Reports produced from an evaluation process can usefully include points of dissent as well as agreement. They may reflect multiple perspectives, and different kinds of expertise. You should trust the process and the ultimate readers/listeners/users to examine outliers in light of the overall conclusions reached.

Below I have outlined the steps necessary in designing and delivering any evaluation and the considerations that need to be taken when adding in co-creation. I have worked on the basis that an evaluation is first a partnership between the people known as ‘evaluators’ and the commissioning organisation: that’s the ‘you’ in the information below. Obviously in many circumstances these might be one and the same, and equally I hope it might evolve to mean ‘all the people working on this’.

Common step 1: Understand who the evaluation is for, who the beneficiaries and potential collaborators are, and what they might be looking for from the work/project.

Co-creation considerations

This might be people taking part as audience, makers or learners, say, but might also be artists, staff, volunteers, or even funders and partners depending on the project. Be as creative as you can about who you could collaborate with, but allow time for them to explore what they bring/want from that, rather than seeing them as serving your ends.

Map out the different beneficiaries/groups and how you might best involve them – e.g. workshops, phone conversations, 1-1 meetings, asking for comments on draft plans. (You can use a stakeholder mapper tool such as this one based on Lusthaus’s work. Categorise people and note their interests and roles in the project or evaluation.)

Think about whose goals, objectives and experiences your evaluation needs to include – and then how you could actively involve them in the design. Be clear and open about any ‘givens’ or non-negotiables in terms of the purpose, and how data will be used. What, fundamentally, is your evaluation for?

Some experiences

I worked with ARC on an evaluation of the Great Place Tees Valley programme devised by partners across the Tees Valley and brought together by Tees Valley Combined Authority with from Arts Council England and National Lottery Heritage Fund. We needed a process that would involve and influence all of these stakeholders, and the dozens of delivery partners. This meant involving them all in the process somehow.

When working with Imogen Blood and Lorna Easterbrook on the long-term programme evaluation of Arts Council England and Baring Foundation’s Celebrating Age programme, we need to understand that the stakeholders included the funders, the 36 funded projects, but also the artists working on the project, multiple partners and older people. Although we were not looking at individual projects, we still wanted the voice of older people to be in the mix, in the data, in the reflection and in any outputs.

Common step 2: Clarify the purpose and objectives of the evaluation.

Co-creation considerations

One characteristic of co-creation is not setting outcomes too strictly. It is important to be flexible and iterative about the evaluation process, ensuring you spend time at the start (or better still before, in the project planning stage) being clear what you want to find out from the evaluation.

It may be things you might call ‘results’ or impact. It might be process learning – what worked where, with whom, and how might we work better? Or a very particular aspect such the creative process, the organisational impact, or social capital.

The key is to decide collaboratively what matters most to you. I find asking ‘What’s the one thing you need this evaluation to do?’ is a helpful starting point. It’s also beneficial to hear each others’ responses. If there are many different ones, you can use a range of methods to narrow it down – discussion, voting, or sorting into themes.

Some experiences

For the Great Place Tees Valley evaluation, we worked with the Steering Group and project team to identify a set of aims for the evaluation, centred on providing learning and recommendations for future approaches to increasing cultural vibrancy in the Tees Valley. This way of working started to share responsibility and ownership right from the start, rather than the evaluation team being expected to sift the evidence for future recommendations, as we might have with a less co-created approach.

Our aims included capturing and sharing learning amongst delivery partners in ways that built skills and capacity and provided time for partners and other stakeholders to reflect on both the achievements and remaining gaps/needs of the Tees Valley cultural sector and co-create a set of recommendations, supported by stakeholders.

Common step 3: Develop an evaluation framework which includes central evaluation questions

Co-creation considerations

Funders or commissioners may have particular needs they need to make sure are addressed even in a co-created process. Respect this whilst connecting those needs, which may focus on monitoring impact more than learning, to the broader context of people- centred learning rather than evaluation-as-advocacy.

In a co-created process with many stakeholders, it can be more difficult to narrow down the factors you are going to track. There may be some things you need to monitor for accountability purposes – e.g. how many people came.

Involve potential participants at the earliest stage to define the scope, and to make the methods as suitable and practical as possible.

Think about whose questions, whose answers, whose reflection who is involved, when define measurables and other indicators, if these are necessary for your evaluation.

Some experiences

I worked on one evaluation of a programme with a funder whose main driver was quite specific health and well-being indicators. This had to be included but had a knock-on effect to the methods, with some projects finding the detailed individual surveys intrusive. A way of showing the impact on well-being was, however, necessary to show the extent of impact on certain health indicators. As the evaluation team we had to balance and share the different perspectives and help people find practical ways forward. A key point here is to allow enough time to test the framework with people.

Managing multiple stakeholder viewpoints can be tricky in a co-created process. This can start with the evaluation framework, which needs to be inclusive and robust, but not try to track and learn from every single possible piece of evidence. (In my experience this a recipe for not seeing the wood for the trees, leading to confusion, overwhelm and less credible conclusions.)

Common step 4: Design and plan for appropriate methods of data collection and analysis

Co-creation considerations

Agree the most appropriate and proportionate methodology with the people involved in delivering it (which may include participants even if only in responding to questions), getting as many perspectives as practical.

Workshop the ideas people have for what would and would not be appropriate ways to collect data that can tell you something about the key parts of your evaluation framework. (Or pick up unexpected results, impacts or learning.)

Asking people to share what works for them is a good starting point in co-creation as it can find solutions to some of the common barriers for different groups.

Think about how you can use a mixed methods approach, combining different approaches – qualitative and quantitive, one-to-one, one-to-many, online and in-person, peer and independent views. It can be good to share a range of potential methods with co-creators at the design stage.

Test your methods with the people who will be delivering them.

Some experiences

Arts partners in the well-being related project mentioned above also wanted to learn about the creative process, and in people’s other responses to the activity. They tended to feel participants would find the survey intrusive and that the funder’s expectations of evidence were unrealistic. The design stage attempted to meet the needs of all, but failed to do so universally. This meant an adaptive approach as the project rolled out, and flexibility in how the survey was used. However, the survey received many more responses than anticipated, providing rich quantitive data for all stakeholders to consider. The evaluation combined surveys with stakeholder peer learning sessions to enable as much co-creation of themes and feedback as possible alongside the well-being-focused quantitive data.

In the Celebrating Age evaluation, we had some negative reactions to the Most Significant Change method at first. Some felt it sounded cumbersome in relation to their time and capacity. Some felt that written stories might not work well for the older people in their projects. We were able to adapt the methods to also include videos, small group discussions. We also worked with the Family Arts Campaign, the funders and other strategic networks to run a series of reflection panels in an even broader way than anticipated, and to involve policy makers at the final stages.

Common step 5: Collect data in line with design

Co-creation considerations

Involving participants in this process can be powerful as a way to maintain the spirit of co-creation. This might be artists collecting stories from participants – or vice versa.

Make sure everyone has chance to input and comment on summaries and write ups. The sharing of reports and records and the creation of feedback loops is really important to co-creation.

Some experiences

Both the Celebrating Age and Great Place Tees Valley evaluations adapted the Most Significant Change methodology, because it involves people in sharing, collecting and reflecting upon stories of change. This also gave some sense of ownership of the conclusions that came of the reflection on what mattered most to different groups of people. In this ways ‘data collection’ becomes the first stage of analysis.

Common step 6: Analysis and reporting

Co-creation considerations

Take every opportunity to get others to think about things with you in peer learning sessions or emerging findings workshops. Share data with stakeholders and participants and convene conversations about what it might mean to them. (This is central to the Most Significant Change process, which gets ‘panels’ of informed stakeholders to reflect on sets of stories and think about their significance, and any patterns or outlier learning in them. It can, though, be incorporated into other methods.) They will see patterns and outliers any one individual might miss. Ask them what the implications are for themselves, their organisations or sector.

Participants and stakeholders can also be asked to comment and contribute to drafts. It can be helpful to share notes and records as you go along, giving everyone involved the opportunity to comment, suggest alternative interpretations or conclusions and add to the record.

Some experiences

The key from all the co-created evaluations I have been involved with is repeatedly stressing that all perspectives are welcome, useful and inform the results of the evaluation. Even where the process requires some independent reflection based on evidence, I would build in some workshopping of emerging conclusions. My experience is that people relish this. An hour spent thinking something through with others can be more productive than an hour ploughing through a detailed report. (Don’t be frightened to give up being The Evaluator with capital letters.)

The health-related project built in elements of peer learning. This gave the opportunity to shape the themes of the evaluation report, how success was described and what was considered in group reflection. This was then able to sit alongside an analysis of the quantitive ‘hard’ data and impact on well-being. This gave a richer picture than either would have alone, meeting the needs to more stakeholders.

Common step 7: Sharing of findings

Co-creation considerations

W.H. Auden famously said “poetry makes nothing happen.” Lots of people think this is true of evaluation. Ending a co-created process by involving participants, partners and stakeholders in sharing the findings can help change that.

Ask people to share their own conclusions, and their actions in response to recommendations arising from the evaluation. (You can build this into any final report if you plan accordingly.) That way any reader or listener can see the difference being made as a result of learning.

Ask people to share the findings through their own networks. Producing snappy summaries, images to share on social media and short videos can be good – things that people can share.

A final report is of course typical, but do think – together – whether that’s really what you need. Reports are useful for transparency outside your network, or if the learning has wider relevance you want to highlight. (And funders often require them, of course.) But if the actions and learning for your collaborators are the focus, a report in any form may not be necessary.

Some experiences

The work led to a report and a set of recommendations that informed the Tees Valley Combined Authority creative industries recovery plans in the wake of Covid. The work also informed a new wave of projects and follow up with the steering group to identify a set of ‘principles for practice’ to help them shape projects and how they are delivered. These are explicitly drawing on the learning from this collaborative evaluation approach. Artist AJ Garret was commissioned to turn the 10 Principles into a poster, and a film was commissioned to illustrate them.

In the Celebrating Age evaluation, the final round of reflection panels involved national networks, funders and stakeholders – the people who have a major influence on cultural opportunities for older people. Out of those discussions have come plans to share the key themes emerging from the evaluation, and any final reporting.

Interim sharing was done in collaboration with Family Arts Campaign and ACE, sharing learning at events and in online reports. This would have been less integrated in a more top-down process.

Conclusion

Co-creation is often about the art of the imperfect and the possible-at-that-time, and this applies to evaluation too. For me, success is an evaluation that fulfils some of these characteristics:

• Those involved use the process to learn from and improve how they work together in future. This is much more likely to come from a process participants and beneficiaries have been involved in shaping for two reasons. Firstly, and fundamentally, it will matter to people because it’s about what is significant to them. Secondly, they will co-create enthusiastically not be examined reluctantly
• Participants, beneficiaries, commissioners, funders and those working on the project all buy into the point and purpose of the evaluation and work together to make it robust, beneficial and reflective of a wide range of perspectives
• Those involved feel their time spent on evaluation has been productive and beneficial
• The evaluation process has brought people together, or deepened their understanding of each other

The spirit with which you co-create can take you a long way, even if you cannot co-create every aspect. If you give yourselves time, space and a clear process, you can involve people at key points to shape the aims, method and analysis arising from any evaluation.This takes time, though, and practice, so make sure you allow time, and learn from each attempt.

Doing an ‘after action’ review

I would suggest conducting some kind of ‘after action’ review about the process and what you got out of it, so you can always be working to improve and to consolidate what works. This review basically works through these five questions:

1. What was supposed to happen?
2. What actually happened?
3. Why were there differences?
4. What have we learnt?
5. What does this mean for us in the future?

When thinking about an evaluation process, it is important to focus on whether learning was generated – co-created or arising from individual insight – and whether it was practically applied. This is often what motivates people to engage. When evaluation is seen to make a real difference – to the quality of a service, to the richness of creative practice, or to the intended outcomes– it is amazing how much more people are willing to contribute, which in turn makes learning even richer.

Key questions

If you’re thinking about a co-created approach to evaluation, consider at least these four questions:

1. How much flexibility can you create in terms of what your evaluation needs to deliver? (E.g. what funder or participant non-negotiables are there to bear in mind?)
2. How early in the design process can you involve the people your evaluation matters to the most, or a sample of them?
3. How might you involve participants and stakeholders in data collection and analysis – including shaping any recommendations?
4. How could everyone involved in the co-creation process help share the findings of your evaluation once they are produced, in whatever form that takes?

Glossary of terms

Beneficiary: people who get some kind of benefit from a process or activity

Data: a piece of evidence or information, in whatever form

Most Significant Change: a qualitative evaluation technique based on collection and reflection on stories

Participant: people who take part in the project, usually not those delivering or managing the activity

Stakeholders: people to who the outcome of a process matters or who will be influenced by it

Further reading and resources

• Practice Example - Most Significant Change: Learning from collaboration in local arts & culture: Great Place Tees Valley Mark Robinson
• SERA, the Story-based Evaluation and Research Alliance, also has other examples of use of Most Significant Change and guidance on using and adapting the technique
• Multiplying Leadership in Creative Communities, Mark Robinson (Creative People and Places, 2020)
• Evaluating Creative Civic Change: an outline of the approach taken by Local trust
• Co-Creating Change: definitions and principles from the network
• The Failspace Project’s Wheel of Failure can be a really useful tool for reviewing your evaluation, especially if your desired impact relates to policy
• Considering Co-creation - Arts Council England commissioned report and podcast series on co-creation by Heart of Glass and Battersea Arts Centre

You may also want to take a look at FailSpaces's How to... fail well guide and our co-created Evaluation principles.

Have you co-created the purpose of your evaluation with others, including participants? What did you learn? Share your reflections and learning with us at ccv@leeds.ac.uk and we’ll feed this into future versions of this guide. Or let us know if you’d like to submit a case study or contribute to a podcast or webinar on this theme.

Image: Dance to Health Norwich. Photo: Becky Demmen