Background

My essential reads have been chosen to give you food for thought regarding the diversity of approaches to evaluation and impact. In the UK, discussion around best evaluation practices to capture cultural value has evolved since the 1980s. By 2021, some approaches are more dominant than others and this, in turn, influences the way success and relevance is argued and the way funding for culture is prioritised. It is important for the sector to understand the choices available and the reasons why some approaches have become dominant over others.

I started my career as a classic academic, specialising in cultural policy research and the symbolic narratives of major international events, but have dedicated the last fifteen years to advancing the case for holistic evaluation frameworks, working closely with major event host cities and cultural sector representatives in the areas of both policy and practice.

My work over this period has blurred the line between theory and practice as I have had the chance to apply and test the validity of my research recommendations, particularly when it comes to evaluation. The readings I suggest here have been determinant in my understanding of core evaluation principles and possibilities. I include classic references, from the 1990s, as I consider them particularly helpful in setting the scene and remind us of essential choices. I also include more recent works that should help you understand later developments.

My Essential Reads

1. Defining Values. Evaluating Art Programmes by François Matarasso
2. Measuring the economic and social impact of the arts by Michelle Reeves
3. Capturing cultural value. How culture has become a tool for government policy by John Holden 
4. International measurement of the economic and social importance of culture by John C Gordon & Helen Beilby-Orrin 
5. Measuring the value of culture: a report to the UK Department for Culture, Media and Sport by Dave O'Brien 

1. Defining Values. Evaluating Arts Programmes

by François Matarasso, 1996

This piece was influential in my original understanding of evaluation and the notion of social impact of the arts. It is an early example of reflections around approaches to evaluation. It is part of ‘The Social Impact of the Arts’ working papers throughout the 1990s, a seminal exercise that is worth revisiting.

The report reflects on the nature of evaluation, outlines ethical issues, reflects on the context to evaluation, methodological issues and practical consequences. It reminds us of the importance of creativity in evaluation and proposes a working model.

It is an essential read because it takes us back to basics and reminds us that the issues at stake today, in 2020, were already being identified back in the 1990s and retain full currency. You can find related papers around ‘monitoring, evaluating and researching culture” in Matarasso’s dedicated site, the Parliament of Dreams.

2. Measuring the economic and social impact of the arts

by Michelle Reeves, 2002

This report was useful to inform my understanding of what we mean by impact at large. It is a comprehensive review of all existing research on the economic and social impact of the arts up that point, including an assessment of the quality of existing evidence throughout the UK.

I consider it an essential read because it takes us through all key concepts relevant in this area (value, impact, evaluation models, economic and social frameworks) and includes detailed listings of methodological typologies. It is a practical document, with many examples of what is meant by each method identified and the types of evidence it may generate.

Much of what has been published in the last ten years has taken more specific routes and prioritised certain approaches over others. This replaces the more exploratory and open-ended take of earlier exercises. See some examples below.

3. Capturing cultural value. How culture has become a tool for government policy

by John Holden, 2004

This read offers a contrast and a critique on many of the practices outlined by the Reeves paper, above. It is one of a collection of essays published throughout the 2000s on the notion of ‘cultural value’ by the thinktank Demos. Holden contests the dominant approaches to arts and culture evaluation on the basis that they focus on the instrumental value of culture alone.

I consider this essential reading as it offers a counterbalance and presents arguments to advance more creative approaches to evaluation and produce ‘better evidence – not just data, but knowledge that people can act on’. The reflections in section 4 (pp. 17-21) are particularly useful and still pertinent today.

4. International measurement of the economic and social importance of culture

by John C Gordon & Helen Beilby-Orrin for the OECD, 2007

This international overview is valuable for anyone interested in the technicalities. It is thorough and it takes us beyond the UK environment, offering comparator evaluation models from Australia, Canada, France, the United Kingdom, the USA and a selection of other OECD countries. The emphasis is on cultural statistics and on economic as well as participation indicators. It offers an account of what is possible – and what is not possible – when assessing culture quantitatively and via standardised measures.

This is a relevant read for those wanting to understand the types of measures that other sectors, outside of culture, tend to prioritise and excel at. The measures and evaluation frameworks proposed here wouldn’t suffice in order to capture cultural value, but it is important to understand the kind of international data that most countries can provide as a basis for comparisons. It is also helpful to be familiar with the language presented here and the conventions it builds on, in order to identify what is useful as well as what is missing and requires alternative models in order to make the case for culture.

5. Measuring the value of culture: a report to the UK Department for Culture, Media and Sport

by Dave O’Brien, 2010

This is a useful update on the cultural evaluation and measurement debate. It brings to the fore the need to define value – and cultural value more specifically – by moving beyond the previously dominant focus on ‘impact’ measures. In this way, the work of the cultural sector can be appraised, from a policy point of view, through tools as advanced and diverse as those used in the health and environment sectors.

The report is a response to the Green Book produced by Treasury in 2003, which stressed the need for Cost Benefit Analysis in government decisions. In the view of O’Brien, the value of the culture sector tends to be disregarded or misunderstood by Treasury, largely due to the lack of measures and arguments that are in line with its priorities. This, in turn, affects the quality of public decision-making when it comes to culture.

I consider it an essential read because it offers a review of notions of value and cultural value and offers an array of useful options that are compatible with the approaches preferred by government when it comes to public funding decisions.

Summary

Discussion on approaches to evaluating culture in the UK have evolved since the late 1980s. After the initial articulation of the ‘economic value’ of the arts (Myerscough 1988) an important development in this field at the time, much evaluation has been dominated by economic frameworks to the detriment of other dimensions and approaches.

Since 2004, there has been a growing emphasis on holistic evaluation frameworks, accounting for multiple dimensions and measures simultaneously (across economic, social, cultural and environmental realms, to name a few of the most dominant). Important advancements have also been made on specific case studies or specific areas of evaluation (e.g. evaluating the creative city, evaluating culture-led regeneration, evaluating arts practices in specific regions, evaluating specific art sectors) with UK-based organisations and projects taking a leading role.

Since 2010, there has also been a growth in comparative and internationally transferable evaluation frameworks, with distinct advancements being made across the EU as well as via global platforms, with the support of networks such as IFACCA.

By 2021, however, we are still debating best practices in evaluation, trying to find the balance between robust evaluation, accessible evaluation, comparative evaluation and holistic evaluation frameworks. It is important to remember that these questions are not new and to ensure we build on previous practices as well as working to push this field forward – and advancing the case for complementary evaluation techniques, including more qualitative and creative evaluation frameworks.

About the author

Dr Beatriz Garcia is Senior Research Fellow in International Cultural Policy and Media Events at the University of Liverpool. She is a member of the European Capital of Culture Selection Committee – nominated by the European Commission; expert member at the Culture & Olympic Heritage Commission –nominated by the International Olympic Committee; director of the Cities of Culture Research Observatory and Associate Director at the Centre for Cultural Value.

Beatriz has been at the forefront of research on the rhetoric, impact and long-term legacy of culture-led city regeneration interventions since 1999 and was a founding member, head of research and director of the Institute of Cultural Capital (2010-2019).

She has conducted fieldwork on the cultural impact of every edition of the Olympic Games since Sydney 2000. Other projects directed by Beatriz include a review of all available evidence on the first three decades of the European Capital of Culture programme; the pioneering Impacts 08 ~ 18 programme, a 20-year study on the socio-economic and cultural impacts of Liverpool as European Capital of Culture; and the first nation-wide assessment of a Cultural Olympiad, focusing on the London 2012 Olympic Games. To conduct this research, Beatriz has been funded by national and international bodies such as the European Parliament, European Commission, International Olympic Committee, United Cities and Local Governments, British Council, British Academy and Research Councils UK.

See: www.beatrizgarcia.net; www.citiesofculture.org; www.impacts08. net; www.iccliverpool.ac.uk ; @beatriz_garcia

Linktr.ee/drBeatrizGarcia

Call to action...

What’s the most useful reading you’ve done in this topic area? What did you learn? If you’d like to share your reflections and learning with us get in touch at ccv@leeds.ac.uk and we’ll feed this into a future resource. Or let us know if you’d like to submit a case study or contribute to a podcast or webinar on this theme.