In Meeting 2 our key theme was Embedding Impact Assessment within an organisation. The agenda focused on how organisations can plan their work with outcomes and measurement in mind. Our guest speaker Dr Douglas Lonie of the National Foundation for Youth Music gave an account of how his organisation had embedded impact assessment within its culture, as well as offering a funder’s perspective of an outcomes-based approach.
For a list of participating organisations, please click here.
To find out more about the Seminar event in November 2013, please click here.
Dr Douglas Lonie, Research & Evaluation Manager, National Foundation for Youth Music.
Dougie Lonie described the process of shifting to an outcomes-based approach in Youth Music and placing impact assessment at the heart of the organisation’s culture. Click here to visit the Youth Music's Planning and Evaluation: An Outcomes Approach
This had taken place over many years, and he emphasised that such a change would inevitably take time
Part of the homework from this session was reading a publication by New Philanthropy Capital about the conditions and factors required to embed impact assessment into an organisation Click here to download 'A Journey to Greater Impact' from NPC's website
Need was defined in two ways:
The needs of participants – for example ‘to develop better interpersonal skills’
The need (or case) for a particular intervention in order to achieve that outcome
The group asked:
Are we able to clearly articulate why our intervention is needed?
Why for these particular participants?
Why is our approach the best one?
Can we show evidence as to how we know this?
Participants agreed that they were not as effective in this area as they would like, and identified a number of barriers:
Establishing need is time-consuming. When so much of an organisation’s capacity is focused on delivery it is difficult to direct resources towards identifying need thoroughly
Understanding need was easier for organisations that always worked with the same participants, type of participant or locality and therefore developed a body of knowledge and experience
However, many of the group worked with multiple participants, often covering a number of different areas such as formal education, youth justice and health. Understanding need required a more in-depth knowledge of these areas than they felt they could offer
This in turn raised a point about the nature of arts organisations and differences from other parts of the third sector, where organisations more commonly specialise in one area
The group noted that establishing need required access to data – for example deprivation indicators for a locality, health statistics relating to a subsection of the population. They did not always feel confident as to how to access such data or interpret it
Working in partnership with other agencies or organisations who had more specialist knowledge of participant needs was one way of overcoming some of these barriers.
If the need for a project or intervention is not established then it is extremely difficult to adopt an outcomes-based approach to planning and consequently it is difficult to demonstrate impact.
If an organisation is able to demonstrate impact and show how its approach achieves outcomes then this information will in turn provide evidence of need for their activity.
Impact Assessment therefore becomes a cyclical process, in which need is evidenced; activity is delivered; outcomes are demonstrated; and the need for the activity is evidenced - see diagram, left.
In order to be able to assess our impact we have to plan with an outcomes-based approach in mind
There are various tools which support this. Dougie introduced the Charities Evaluation Service Planning Triangle, which plots the relationships between Overarching Aim, Intended Outcomes, Activities and Outputs. The group undertook a project planning exercise using this model.
For more information about the CES Planning Triangle, including case studies demonstrating how it works in practice, click here to visit the CES website.
The group found that it was sometimes difficult to distinguish between outputs and outcomes. However, it is extremely important to get this distinction right from the start, in order that appropriate outcomes are set. The Charities Evaluation Service ‘Jargonbuster’ offers information about the language and definitions used in evaluation and impact assessment – for more details click here.
In order to demonstrate impact we need to collect evidence. But how do we ensure that our evidence is of good quality? And how do we use the evidence that we gather?
The group discussed this area with Dougie Lonie and some key points arose:
Evidence forms an important part of evaluation planning. After setting specific outcomes for a piece of work it is then vital to identify outcome indicators – in other words, what might the outcome look like in practice? How are we likely to be able to demonstrate it?
Once outcome indicators are in place, the next step is to identify the sources of evidence that will be needed and at what point(s) it should be gathered. Evidence can come in many forms and from many sources, but it is strongest when it is clearly linked to an outcome indicator and an outcome
If this planning takes place then evidence sources e.g. questionnaires, documentary evidence can be timetabled well in advance, and responsibility for these areas can be allocated
A multi-layered approach should be considered in which each outcome can be demonstrated through more than one type of evidence. This makes the case for attribution and/or correlation more compelling
An organisation may offer many different types of activities. However if outcomes across more than one activity are the same, then it is worth standardising some outcome indicators and types of evidence. This will enable an organisation to aggregate the data collected and to create a more compelling evidence base
How can we prove that our intervention has led to a particular impact, especially as we often work with participants who have multiple interventions?
In Meeting 1, the group identified attribution as one of the greatest barriers to doing Impact Assessment and this theme was revisited with our guest speaker in Meeting 2.
The NESTA Standards of Evidence for Impact Investing were examined
Randomised Control Trials – producing the highest level of credibility – are financially out of reach for most arts organisations and are not relevant to short-term interventions
It was agreed that while attribution was extremely difficult to prove, correlation was possible. Identifying need correctly and adopting an outcomes-based approach with robust evaluation planning were essential in giving the best chance of demonstrating correlation
What can we do ourselves, and when do we require additional expertise?
A brief discussion raised the following points:
Invest in training for the organisation’s staff, and develop systems and expertise
It is important to have clarity about evaluation aims in order to decide whether it can be undertaken in-house or whether it requires additional expertise. Going through a rigorous evaluation planning process will highlight where additional expertise might be required
Alternatively, it may be appropriate to seek support with the evaluation planning process, but then undertake the process in-house, or with light support
A good understanding is required in order to write an evaluation brief and to identify and manage an evaluation partner
A brief needs to be detailed and specific, outlining the scope of the intervention and what the organisation wants to know. This clarity will make it easier to identify the most appropriate methodology and to manage the partnership
The group undertook two pieces of homework:
1. To read the New Philanthropy Capital publication ‘Journey to Greater Impact’, which gives an account of the conditions and factors required to embed impact assessment into an organisation
2. To complete an evaluation planning grid, plotting Need, Activity, Outputs, Outcomes, Outcome Indicators and Evidence