During Meeting 1, the group of participating organisations identified a significant number of barriers to undertaking Impact Assessment.
The majority of initiatives that organisations undertook were short – including many one-off events. It was potentially difficult to keep in contact with participants after an initiative had come to an end. Measuring change over time therefore presented a challenge to many participants.
Most organisations had many different initiatives taking place concurrently. With the focus being on delivery there was insufficient remaining capacity for impact assessment.
Organisations had multiple funders to report to and this generated heavy workload. Organisations described situations where funders supporting the same project had different reporting requirements. One organisation had 26 funders involved in current projects.
The burden of reporting was cited as a further reason why organisations would like to negotiate reporting requirements with their funders.
The organisation must be clear about what it is doing before it can be clear about what it is measuring.
Changes in staff mean that it can be difficult to embed and then maintain a coherent approach to information gathering within an organisation.
There are a lot of different approaches to assessing impact. Organisations felt confused about different methodologies and found the technical language around methodologies challenging. They were unconfident about navigating the information available and choosing an approach that was best suited to them.
Organisations were frustrated that so many different metrics were used across the sector and between sectors. This meant that they were unable to compare their results to those of peers and that it was more difficult to cite the findings of similar projects in their case for support.
They also believed that a lack of consistent metrics meant that the sector as a whole was unable to demonstrate its impact effectively.
Two barriers relating to the requirements of funders were identified
Some funders had extremely high expectations of reporting which were difficult to meet given limited capacity. Sometimes the work generated by reporting was felt to out of proportion to the amount of funding given
Conversely, the perceived attitude of other funders sometimes created a disincentive to assess impact thoroughly
Some funders did not ask for detail other than confirmation that the activity had taken place and the donation used as per the funding agreement
Organisations were not certain how funders used the information provided to them or indeed whether it was used at all
Organisations were concerned that it was extremely difficult to attribute outcomes to specific activities. Many participants were recipients of multiple interventions, in some cases involving multiple agencies, and therefore pinpointing the role played by a single intervention in an overall outcome was challenging. Organisations were cautious about over-claiming.
Linked to the topic of attribution, generating a high quality evidence base was also felt to be a barrier. Evaluation involving a control group requires significant resource and expertise. A Randomised Control Trial would be beyond the resources of most individual organisations. There was therefore concern that the quality of evidence attainable within the resources and expertise of an individual organisation may lack the necessary credibility.
Our group raised concerns around the collection and storage of sensitive information and commented that lack of confidence in this area was a barrier to undertaking some aspects of information gathering.
Some members of the group felt strongly that having a control group and therefore denying a cohort access to an activity was counter to their mission and too uncomfortable.