In Meeting 3 we aimed to cover a number of areas.
Introduce Theory of Change. In the first meeting, a number of the group reported negative experiences of Theory of Change, in particular feeling that it was difficult to apply the model to their work. Having spent time considering the drivers for impact assessment, the benefits of adopting an outcomes approach and how this might become embedded in their organisations, the group was interested in revisiting Theory of Change and in looking at how the model could support them
Consider how Impact Assessment can support business planning and organisational development
Discuss how Impact Assessment translates into advocacy
Gain the perspective of a major grant-making Trust
For a list of participating organisations, please click here.
To find out more about the closing Seminar event in November 2013, please click here.
Marina Svistak, Consultant Measurement & Evaluation, Research & Consulting, New Philanthropy Capital
Tim Joss, Director, The Rayne Foundation
According to recent research by New Philanthropy Capital (NPC), 75% of charities are now measuring the impact of their work to some degree. Over half measure all aspects of their work while a quarter measure some. For NPC, whose aim is to support and strengthen the charitable sector, this is very encouraging, although clearly some challenges remain to engage the remaining 25%.
The group considered this information. If 75% of charities were able to demonstrate impact to some degree, then what implications did this have for other charities’ effectiveness in fundraising and advocacy?
NPC’s approach to measuring social impact comprises four components, which it believes organisations must have in order to assess impact effectively:
Strategic vision and goals
A well-developed Theory of Change
Appropriate measurement tools
NPC’s work supports the charitable sector to develop the skills and knowledge underpinning these components.
A Theory of Change is a conceptual map of how activities lead to outcomes. It shows the causal links that exist between activities and the outcome(s) that they achieve
Overall the map charts the journey from an identified need through to the final outcome, by showing the incremental steps of cause and effect taken in that journey. In so doing, it explains in detail how and why an intervention works in practice. These steps are known as ‘intermediate outcomes’
A Theory of Change is also sometimes called a ‘Logic Model’, because it demonstrates a logical chain of cause and effect
In her presentation (downloadable through the link button, left), Marina Svistak used the example of a project supporting children who are falling behind at school because of problems at home. Slide No 8 simply links need, activities and final outcome. In Slide No 9 the intermediate outcomes are filled in, demonstrating the causal links underpinning this project’s methodology.
Causal links should not be assumptions. They should be already evidenced, or be the subject of testing.
Marina Svistak discussed sources of evidence and advised the group that it is not always necessary for an organisation to collect its own data to support every outcome. Research done by others may already support the same causal link as the one you want to make.
Slide 10 of Marina’s presentation (downloadable through the link button, above left) provides a checklist for organisations to use when thinking about evidence for causal links.
The group returned to the theme of levels of evidence and the challenge of attributing outcomes to a particular intervention. The same topic was discussed at Meetings 1 and 2.
Marina Svistak presented a diagram showing how different types of evidence rated in terms of credibility.
It was acknowledged that advanced forms of evidence-gathering – control groups and randomised control trials - were beyond the scope and resources of many organisations
However, organisations can look to other sources – such as national statistics or national surveys – to which they can match their data. See Meeting 4 for further information about the recently available access to Department of Justice data.
It was also noted that schools provide an easier environment in which to work with control groups, because directly comparable cohorts are available and there is clear knowledge of other interventions that a control cohort may have encountered
The ethics of working with one cohort and denying the same experience to another were again questioned, and many members of the group expressed their discomfort with this idea. Other group members felt that the value of the evidence they stood to gain should outweigh any ethical discomfort, and commented that control trials are routinely used in much more ethically sensitive situations, such as drug trials
Tim Joss , Director of the Rayne Foundation, gave a thought-provoking presentation that looked at the benefits of assessing impact. He offered a funder's perspective on how effective arts organisations were in this area compared to other parts of the third sector. Tim gave examples of organisations for which Impact Assessment had led to replication and growth.
The group undertook an exercise looking at the potential for replication within their programme. This had mixed results and most organisations identified difficulties, believing that their key resource and unique selling point - the artistic programme - could not be stretched in this way. The artistic capital of an arts organisation is its greatest asset, and is also the element that enables high quality and effective work to happen - but it is finite, which can create problems. For example, the Royal Ballet has a particular number of dancers with a particular number of available hours. To replicate its Chance to Dance programme in many places would mean using different dancers - this would no longer be the Royal Ballet.
Tim Joss gave a brief background to the Rayne Foundation, which works in the areas of arts, education, health and medicine, and social welfare and development. Recently the Foundation has moved to an outcomes-based approach. There are four cross-cutting outcomes, and applicants must demonstrate how their work relates to one or more.
Tim described how monitoring of grantees' activities had changed as a result of the outcomes-based approach. This had shifted from ‘long reports of anecdote & flattery’ to more specific reporting against outcomes.
In Meeting 1 the group had considered to what extent their drivers for impact assessment coincided with funders' needs. Tim offered a funder’s perspective on the same question. In his opinion, when comparing the needs of an arts organisation true to its mission with those of a funder, these needs were broadly the same, namely:
To check effectiveness
To identify areas for improvement
To interrogate value
To support development and growth
Additionally, a funder will want to know how the grant contributed to the funder's objectives, and whether there are lessons for sharing. However, this assumes a grantee whose funded work is true to the organisation’s core mission. In reality some organisations suffer from ‘funder-led mission drift’. In other words, they undertake activities in response to funding opportunities, not because they are linked to the core mission. If this is the case then the common ground described above is less likely to exist.
Impact Assessment is currently more important than ever due to:
Growth of evidence-based policy and practice
Treasury Green Book and competition between government departments
New financial instruments which depend on measurable outcomes
The Rayne Foundation funds work across a number of sectors. The group asked Tim Joss to comment on how the arts sector compared to other parts of the third sector in terms of impact assessment.
Good intentions and received wisdom about the power of the arts are not enough. Cost and benefits must be measured
The arts sector argues for its uniqueness but this does not necessarily equate to strength. Rather, the sector might be stronger if it identified itself as part of a movement, with a clear sense of where it fits into society
As a sector it suffers from atomisation. Subsectors do not communicate with each other, and therefore the sector as a whole is unable to make common cause
The arts sector suffers from ‘projectitis’ – developing new projects and initiatives all the time, but it is less adept at scaling, replication and adoption of existing models that work well
Slide 11 of Tim’s presentation (see link above) gives further detail about characteristics of the arts sector and what the positive image of this might look like.
A strong arts sector would:
Be part of a movement
Know where it fits into society
Make common cause
Learn / benchmark
Demonstrate scaling, replication and adoption of existing models
Undertake reality checks
Contribute to policy making
The arts sector needs set out itself what a reasonable range of cultural outcomes might be. If the sector itself does not do this, then others will fill the vacuum. Furthermore, it is vital that artistic capital is placed at the forefront of an outcomes model for the arts. Artistic capital is needed before it can be exploited for social outcomes.
To illustrate this point Tim’s presentation compares the Big Society Capital outcomes matrix (culture, sport and heritage) with BOP Consulting’s logic model of the Edinburgh Festival Evaluation Framework demonstrating how artistic inputs (capital) lead to wider economic, social, health and wellbeing outcomes.
Tim Joss discussed the need for the arts world to be more joined up in its advocacy and to make the case for its position in the broader health, wellbeing and 'happiness' agenda. If the arts sector does not make its own case then others will decide where it does (or does not) fit in.
There was a brief discussion about the possibility of the arts community working together to develop a set of shared measures.
Other sectors have developed shared measures, but at the moment the arts suffers from atomisation of subsectors and a lack of a co-ordinated approach.
Tim Joss advocated strongly that arts organisations use impact assessment to inform organisational strategy and growth. In particular he was interested to see organisations move away from ‘projectitis’ and towards a culture of scaling up successful models.
The group undertook an exercise looking at how their organisations might develop their activities, asking:
What does your organisation offer?
Could you develop any of your projects into a service?
Could this be replicable?
Is it desirable for the organisation? Why / why not?
Is it sustainable?
What are the barriers?
The exercise had mixed results and the following issues were raised:
Several members of the group felt that the identity and quality of their programmes were inextricably linked to specific artistic assets. For one this was their building, for another it was company artists. These assets had finite capacity, so that upscaling would mean that the nature of an activity would have to change. For example, taking place in an outside venue, or using freelance or non-company artists. Organisations felt that this compromised identity and quality
Another organisation felt that developing a greater focus on one area and rolling it out more widely could lead to mission drift in the organisation
A further organisation identified that if one activity were to increase another would have to decrease or cease – there wasn’t capacity to do everything
On the one hand these responses could be seen as a reluctance to make choices or move away from a ‘projectitis’ model. On the other hand the point about finite artistic resources has validity.
The group undertook an exercise looking at intermediate measures and the barriers they perceived to adopting a Theory of Change approach.
London Transport Museum. Photo: Diane Auckland Fotohaus