Arts Impact Assessment

Meeting 2

In Meeting 2 our key theme was Embedding Impact Assessment within an organisation. The agenda focused on how organisations can plan their work with outcomes and measurement in mind. Our guest speaker Dr Douglas Lonie of the National Foundation for Youth Music gave an account of how his organisation had embedded impact assessment within its culture, as well as offering a funder’s perspective of an outcomes-based approach.

18th June 2013

at the Roundhouse

For a list of participating organisations, please click here.

 

To find out more about the Seminar event in November 2013, please click here.

Content Overview

Agenda

Guest Speaker

Download Meeting 2 Agenda

Dr Douglas Lonie, Research & Evaluation Manager, National Foundation for Youth Music.

Dougie Lonie Biography Meeting 2
  • Dougie Lonie described the process of shifting to an outcomes-based approach in Youth Music and placing impact assessment at the heart of the organisation’s culture. Click here to visit the Youth Music's Planning and Evaluation: An Outcomes Approach

  • This had taken place over many years, and he emphasised that such a change would inevitably take time

  • Part of the homework from this session was reading a publication by New Philanthropy Capital about the conditions and factors required to embed impact assessment into an organisation Click here to download 'A Journey to Greater Impact' from NPC's website

Topic

Discussion and Learning

Embedding Impact Assessment within an organisation

Need was defined in two ways:

  • The needs of participants – for example ‘to develop better interpersonal skills’

  • The need (or case) for a particular intervention in order to achieve that outcome

 

The group asked:

  • Are we able to clearly articulate why our intervention is needed?

  • Why now?

  • Why for these particular participants?

  • Why is our approach the best one?

  • Can we show evidence as to how we know this?

 

Participants agreed that they were not as effective in this area as they would like, and identified a number of barriers:

  • Establishing need is time-consuming. When so much of an organisation’s capacity is focused on delivery it is difficult to direct resources towards identifying need thoroughly

  • Understanding need was easier for organisations that always worked with the same participants, type of participant or locality and therefore developed a body of knowledge and experience

  • However, many of the group worked with multiple participants, often covering a number of different areas such as formal education, youth justice and health. Understanding need required a more in-depth knowledge of these areas than they felt they could offer

  • This in turn raised a point about the nature of arts organisations and differences from other parts of the third sector, where organisations more commonly specialise in one area

  • The group noted that establishing need required access to data – for example deprivation indicators for a locality, health statistics relating to a subsection of the population. They did not always feel confident as to how to access such data or interpret it

  • Working in partnership with other agencies or organisations who had more specialist knowledge of participant needs was one way of overcoming some of these barriers.

 

If the need for a project or intervention is not established then it is extremely difficult to adopt an outcomes-based approach to planning and consequently it is difficult to demonstrate impact.

 

If an organisation is able to demonstrate impact and show how its approach achieves outcomes then this information will in turn provide evidence of need for their activity.

 

Impact Assessment therefore becomes a cyclical process, in which need is evidenced; activity is delivered; outcomes are demonstrated; and the need for the activity is evidenced  - see diagram, left.

  • In order to be able to assess our impact we have to plan with an outcomes-based approach in mind

  • There are various tools which support this. Dougie introduced the Charities Evaluation Service Planning Triangle, which plots the relationships between Overarching Aim, Intended Outcomes, Activities and Outputs. The group undertook a project planning exercise using this model.

  • For more information about the CES Planning Triangle, including case studies demonstrating how it works in practice, click here to visit the CES website.

  • The group found that it was sometimes difficult to distinguish between outputs and outcomes. However, it is extremely important to get this distinction right from the start, in order that appropriate outcomes are set. The Charities Evaluation Service ‘Jargonbuster’ offers information about the language and definitions used in evaluation and impact assessment – for more details click here.

How effective are we in identifying the need for our work?

Planning for effective Impact Assessment

In order to demonstrate impact we need to collect evidence. But how do we ensure that our evidence is of good quality? And how do we use the evidence that we gather?

 

The group discussed this area with Dougie Lonie and some key points arose:

  • Evidence forms an important part of evaluation planning. After setting specific outcomes for a piece of work it is then vital to identify outcome indicators – in other words, what might the outcome look like in practice? How are we likely to be able to demonstrate it?

  • Once outcome indicators are in place, the next step is to identify the sources of evidence that will be needed and at what point(s) it should be gathered. Evidence can come in many forms and from many sources, but it is strongest when it is clearly linked to an outcome indicator and an outcome

  • If this planning takes place then evidence sources e.g. questionnaires, documentary evidence can be timetabled well in advance, and responsibility for these areas can be allocated

  • A multi-layered approach should be considered in which each outcome can be demonstrated through more than one type of evidence. This makes the case for attribution and/or correlation more compelling

  • An organisation may offer many different types of activities. However if outcomes across more than one activity are the same, then it is worth standardising some outcome indicators and types of evidence. This will enable an organisation to aggregate the data collected and to create a more compelling evidence base

Developing an evidence base

How can we prove that our intervention has led to a particular impact, especially as we often work with participants who have multiple interventions?

 

In Meeting 1, the group identified attribution as one of the greatest barriers to doing Impact Assessment and this theme was revisited with our guest speaker in Meeting 2.

  • The NESTA Standards of Evidence for Impact Investing were examined

  • Randomised Control Trials – producing the highest level of credibility – are financially out of reach for most arts organisations and are not relevant to short-term interventions

  • It was agreed that while attribution was extremely difficult to prove, correlation was possible. Identifying need correctly and adopting an outcomes-based approach with robust evaluation planning were essential in giving the best chance of demonstrating correlation

What can we do ourselves, and when do we require additional expertise?

 

A brief discussion raised the following points:

  • Invest in training for the organisation’s staff, and develop systems and expertise

  • It is important to have clarity about evaluation aims in order to decide whether it can be undertaken in-house or whether it requires additional expertise. Going through a rigorous evaluation planning process will highlight where additional expertise might be required

  • Alternatively, it may be appropriate to seek support with the evaluation planning process, but then undertake the process in-house, or with light support

  • A good understanding is required in order to write an evaluation brief and to identify and manage an evaluation partner

  • A brief needs to be detailed and specific, outlining the scope of the intervention and what the organisation wants to know. This clarity will make it easier to identify the most appropriate methodology and to manage the partnership

Attribution and correlation

Skills and knowledge required to do effective Impact Assessment

Need is evidenced diagram

Homework

The group undertook two pieces of homework:

 

1. To read the New Philanthropy Capital publication ‘Journey to Greater Impact’, which gives an account of the conditions and factors required to embed impact assessment into an organisation

 

2. To complete an evaluation planning grid, plotting Need, Activity, Outputs, Outcomes, Outcome Indicators and Evidence

A Journet to Greater Impact Dougie Lonie's Presentation