Arts Impact Assessment

Barriers

During Meeting 1, the group of participating organisations identified a significant number of barriers to undertaking Impact Assessment.

Timeframe and short-term planning

The majority of initiatives that organisations undertook were short – including many one-off events. It was potentially difficult to keep in contact with participants after an initiative had come to an end. Measuring change over time therefore presented a challenge to many participants.

Volume of projects

Tools and methods

Lack of clarity and mission

Staff turnover

A lack of consistent metrics

Number of funders

Most organisations had many different initiatives taking place concurrently. With the focus being on delivery there was insufficient remaining capacity for impact assessment.

Organisations had multiple funders to report to and this generated heavy workload. Organisations described situations where funders supporting the same project had different reporting requirements. One organisation had 26 funders involved in current projects.

 

The burden of reporting was cited as a further reason why organisations would like to negotiate reporting requirements with their funders.

The organisation must be clear about what it is doing before it can be clear about what it is measuring.

Changes in staff mean that it can be difficult to embed and then maintain a coherent approach to information gathering within an organisation.

There are a lot of different approaches to assessing impact. Organisations felt confused about different methodologies and found the technical language around methodologies challenging. They were unconfident about navigating the information available and choosing an approach that was best suited to them.

Organisations were frustrated that so many different metrics were used across the sector and between sectors. This meant that they were unable to compare their results to those of peers and that it was more difficult to cite the findings of similar projects in their case for support.

 

They also believed that a lack of consistent metrics meant that the sector as a whole was unable to demonstrate its impact effectively.

Attribution

Requirements of funders and how they are perceived

Quality of Evidence

Ethics: the collection and storage of sensitive information

Two barriers relating to the requirements of funders were identified

  • Some funders had extremely high expectations of reporting which were difficult to meet given limited capacity. Sometimes the work generated by reporting was felt to out of proportion to the amount of funding given

 

Conversely, the perceived attitude of other funders sometimes created a disincentive to assess impact thoroughly

  • Some funders did not ask for detail other than confirmation that the activity had taken place and the donation used as per the funding agreement

  • Organisations were not certain how funders used the information provided to them or indeed whether it was used at all

Organisations were concerned that it was extremely difficult to attribute outcomes to specific activities. Many participants were recipients of multiple interventions, in some cases involving multiple agencies, and therefore pinpointing the role played by a single intervention in an overall outcome was challenging. Organisations were cautious about over-claiming.

Linked to the topic of attribution, generating a high quality evidence base was also felt to be a barrier. Evaluation involving a control group requires significant resource and expertise. A Randomised Control Trial would be beyond the resources of most individual organisations. There was therefore concern that the quality of evidence attainable within the resources and expertise of an individual organisation may lack the necessary credibility.

Our group raised concerns around the collection and storage of sensitive information and commented that lack of confidence in this area was a barrier to undertaking some aspects of information gathering.

  • Some members of the group felt strongly that having a control group and therefore denying a cohort access to an activity was counter to their mission and too uncomfortable.

World of cranes clockers clauser cropped