Making Data Work: A scoping study to develop a mixed-methods evaluation framework for culture
The cultural sector makes a significant contribution to people’s lives, to society and the economy – so why do we struggle to convey its impact? This study, led by the Centre for Cultural Value, shares findings from a national, interdisciplinary research project that aims to tackle the crisis the cultural sector is currently facing in harnessing and evaluating its data.
Undertaken over a course of 15 months (from January 2021 to March 2022), this study was led by the Centre for Cultural Value, funded by the Economic and Social Research Council (ESRC) through UK Research and Innovation (UKRI). The research team comprised researchers from arts management, cultural policy, psychology and quantitative sociology, working closely with industry experts from The Audience Agency and MyCake.
What we wanted to achieve
We set out to explore the tensions, challenges and incompatibilities of data collection, assessing the abundance of evaluation methods currently being used in the cultural sector.
At the heart of our research, we observed different and divergent protocols for data collection, collation, storage, analysis and impact evaluation across the cultural sector. These varied and excessive approaches to data management have led to a chaotic and sporadic landscape for cultural evaluation, making it harder to demonstrate impact at the level of national policy.
Our research sought to untangle these incompatibilities and to make a case for a more standardised data approach in the arts.
Within our research, we also focused on the variety of voices and perspectives evident within the cultural sector. We considered the role of creative practitioners and the part artists, audiences and communities play in data collection. We also identified opportunities for funders, sector bodies and policymakers, exploring how they might be able to improve cultural data practices.
Through this project, the research team aimed to shape and inform the policymaking process, offering alternative, multidimensional approaches to understanding the cultural sector and its impact.
- There are significant strengths in terms of current quantitative data resources, with substantial benefit anticipated if these can be linked and made more accessible.
- The primary source for quantitative data is developed for, with and by cultural sector organisations. Data relates to programming, management and marketing/fundraising offers opportunities for learning and evaluation.
- Cultural organisations lack the resources to employ data scientists to benefit from data already collected or generated;
- The driver for evaluation data is often to meet funding requirements, leaving creative practitioners less able to develop skills or processes for good qualitative or mixed-methods data gathering and evaluation.
- Current methods and evaluation approaches fail to convey the social impact of cultural activity in any compelling way.
Our work also focused on a specific locality to explore how data and evaluation can help cultural organisations understand the value and meaning of the activities they undertake. Situated in the Bradford cultural community, it offered an in-depth insight into practitioners’ experiences and aspirations relating to data and evaluation and the challenges of capturing the impact of their work in appropriate ways. Some key themes emerged here:
- Planning for impact and improvement, using data and evaluation to set baselines or assess past performance and using this to help plan for future impact.
- Robust data, methods and frameworks, requiring good quality data to be collected and analysed using common standards impact.
- Methodological diversity and innovation, adopting mixed methods and data types to convey the distinctiveness and value of the cultural sector)
- Building communities of practice around cultural data and evaluation.
- Common, underlying data infrastructure, combining qualitative and quantitative approaches to connect different methods and sources, develop common standards, and act as a repository of data and research finding.
There is substantial potential to build upon the cultural sector’s strengths, data resources, and capacity to improve impact and communicate value.
Future development work could include:
- Developing common standards in quantitative data.
- Piloting a Cultural Data Trust.
- Developing policy/practice networks on cultural sector data and evaluation at local and national levels.
- A Cultural Sector Data Observatory to bring together, research and analyse mixed methods data sources.
- Testing and refining the evaluation planning framework with practitioners.
We would like to thank our project funders, the Economic and Social Research Council, for their generous support for this research and also the three core funders of the Centre for Cultural Value for providing the research infrastructure that made this research possible in the first place: the Arts and Humanities Research Council, Arts Council England and Paul Hamlyn Foundation.
Other formats of this document are available on request. Please contact: email@example.com
This publication is licensed under the terms of a Creative Commons Attribution-Non-commercial-ShareAlike 4.0 International license. View a copy of this license. It is available to be used, shared, and adapted for non-commercial activity. Please credit the copyright holder Centre for Cultural Value. Any enquiries regarding this publication should be sent to us at firstname.lastname@example.org
If you want to cite this work, please use: Walmsley, B. et al. (2022). Making Data Work: A scoping survey to develop a mixed-methods evaluation framework for culture Leeds: Centre for Cultural Value. © Centre for Cultural Value 2022