Meeting 2

For a list of participating organisations, please click here.

To find out more about the Seminar event in November 2013, please click here.

18th June 2013 at the Roundhouse

Content Overview

In Meeting 2 our key theme was Embedding Impact Assessment within an organisation. The agenda focused on how organisations can plan their work with outcomes and measurement in mind. Our guest speaker Dr Douglas Lonie of the National Foundation for Youth Music gave an account of how his organisation had embedded impact assessment within its culture, as well as offering a funder’s perspective of an outcomes-based approach.

Agenda

Guest Speaker

Dr Douglas Lonie, Research & Evaluation Manager, National Foundation for Youth Music.

Embedding Impact Assessment within an organisation

  • Dougie Lonie described the process of shifting to an outcomes-based approach in Youth Music and placing impact assessment at the heart of the organisation’s culture. Click here to visit the Youth Music's Planning and Evaluation: An Outcomes Approach
  • This had taken place over many years, and he emphasised that such a change would inevitably take time
  • Part of the homework from this session was reading a publication by New Philanthropy Capital about the conditions and factors required to embed impact assessment into an organisation Click here to download 'A Journey to Greater Impact' from NPC's website

Planning for effective Impact Assessment

Need was defined in two ways:

  • The needs of participants – for example ‘to develop better interpersonal skills’
  • The need (or case) for a particular intervention in order to achieve that outcome
    • The group asked:

      • Are we able to clearly articulate why our intervention is needed?
      • Why now?
      • Why for these particular participants?
      • Why is our approach the best one?
      • Can we show evidence as to how we know this?
        • Participants agreed that they were not as effective in this area as they would like, and identified a number of barriers:

          • Establishing need is time-consuming. When so much of an organisation’s capacity is focused on delivery it is difficult to direct resources towards identifying need thoroughly
          • Understanding need was easier for organisations that always worked with the same participants, type of participant or locality and therefore developed a body of knowledge and experience
          • However, many of the group worked with multiple participants, often covering a number of different areas such as formal education, youth justice and health. Understanding need required a more in-depth knowledge of these areas than they felt they could offer
          • This in turn raised a point about the nature of arts organisations and differences from other parts of the third sector, where organisations more commonly specialise in one area
          • The group noted that establishing need required access to data – for example deprivation indicators for a locality, health statistics relating to a subsection of the population. They did not always feel confident as to how to access such data or interpret it
          • Working in partnership with other agencies or organisations who had more specialist knowledge of participant needs was one way of overcoming some of these barriers.
            • If the need for a project or intervention is not established then it is extremely difficult to adopt an outcomes-based approach to planning and consequently it is difficult to demonstrate impact.

              If an organisation is able to demonstrate impact and show how its approach achieves outcomes then this information will in turn provide evidence of need for their activity.

              Impact Assessment therefore becomes a cyclical process, in which need is evidenced; activity is delivered; outcomes are demonstrated; and the need for the activity is evidenced - see diagram, left.

Need is evidenced diagram

In order to demonstrate impact we need to collect evidence. But how do we ensure that our evidence is of good quality? And how do we use the evidence that we gather?

The group discussed this area with Dougie Lonie and some key points arose:

  • Evidence forms an important part of evaluation planning. After setting specific outcomes for a piece of work it is then vital to identify outcome indicators – in other words, what might the outcome look like in practice? How are we likely to be able to demonstrate it?
  • Once outcome indicators are in place, the next step is to identify the sources of evidence that will be needed and at what point(s) it should be gathered. Evidence can come in many forms and from many sources, but it is strongest when it is clearly linked to an outcome indicator and an outcome
  • If this planning takes place then evidence sources e.g. questionnaires, documentary evidence can be timetabled well in advance, and responsibility for these areas can be allocated
  • A multi-layered approach should be considered in which each outcome can be demonstrated through more than one type of evidence. This makes the case for attribution and/or correlation more compelling
  • An organisation may offer many different types of activities. However if outcomes across more than one activity are the same, then it is worth standardising some outcome indicators and types of evidence. This will enable an organisation to aggregate the data collected and to create a more compelling evidence base

Attribution and correlation