This is the first in a series of blogposts related to the often challenging topic of impact measurement. This post describes how we identify the key outcomes for each of the ideas on ChangeX.

How do you know that the work you’re doing is having a positive impact on the communities you engage with?

Everyone, from big corporations to social enterprises and small community organisations, need to answer this question and it’s often one of the most difficult questions to clearly and confidently answer.

It certainly has been one of the more difficult questions for us at ChangeX. Our ultimate ambition is to activate 20 million teams around impactful ideas by 2030. We plan to do that by building a portfolio of ideas aligned with the Global Goals and equipping local community leaders with the information and resources they need to mobilise around those ideas.

Fundamental to achieving this goal, is accurately tracking our progress along the way. Our impact happens at a number of different levels, the first level being the direct impact of the ideas that are replicated through ChangeX.

For this reason, one of our key metrics is the number of “Active Starters”. An active starter being an individual or a team who successfully replicates one of the ideas from ChangeX.org in their local community, for example a new GIY Group of people coming together to grow their own food, a new Men’s Shed where men are regularly meeting and carrying out projects, a school successfully introducing Playworks in their yard. This is, in effect, our key output, the growth of which we believe will lead to social impact.

But how do we know that the growth of Active Starters is having a social impact and more importantly how do we know what exactly that social impact is?

There are approximately 50 ideas on ChangeX. They are all, to some extent, proven to work and to achieve positive social or environmental impact. In order to gain a deeper understanding of how they affect the people engaging with them, we look at the research that sits behind those ideas. Some have a very high standard of academic research directly related to their idea or programme, others rely on existing research in their domain to demonstrate likely outcomes.

Identifying the outcomes of each idea on ChangeX

The Logic Model is a well-established framework within which to evaluate the social impact of a policy or intervention. Applying this methodology consistently across the ChangeX portfolio allows us to build a clearer picture of impact and more importantly, it helps us to understand what impact each new Active Starter has. At its simplest, you look at Inputs > Outputs > Outcomes and over time you get a clearer picture of longer term impact.

One big challenge for us is that there isn’t a consistent standard of research to validate the outcomes for each idea on ChangeX. The highest standards of academic research require a very significant investment of both finance and time, neither of which are abundant in any fast-growing non-profit.

Inspired by the approach of the University of Wisconsin, in building their What Works for Health database, we believe that using existing research relating to a strategy or intervention, while not perfect, presents a good solution to help gain an understanding of an idea’s outcomes.

Let’s take GIY groups as an example. A wide array of academic research exists to show the positive impact of community food growing, both for the environment and for the individual involved. While there isn’t extensive academic research on GIY particularly, the research already available identifies 2-3 key outcomes of community food growing, therefore allowing us to build a logic model for GIY:

As ideas replicate and grow through the ChangeX platform, we extrapolate the resulting outcomes per new replication and the final step is to validate these outcomes with direct feedback from beneficiaries.

Here’s a similar logic applied to Playworks:

Closing the feedback loop

Our next challenge is to close the feedback loop back to the ultimate beneficiaries – the community members engaging in the GIY Group or the kids experiencing Playworks in their school – to confirm that the outcomes, as expected, are happening. Over time, this will allow us to identify differences as they arise  – context, geography and also monitor impact over time.

We want to make it really easy, through the ChangeX platform, to gather this feedback directly from those participating. We’re looking at a number of ways to do this including SMS, automated surveys, phone interviews and we’ve already been inspired by some of the work of Feedback Labs and Acumen in how to approach it.

The Lean Data Field Guide by Acumen and this article on the Power of Lean Data from the Stanford Social Innovation Review are two great resources that we’ve found helpful.

There are a few other big areas we’re excited about in this field including how we map our progress to the Global Goals and looking at the personal impact successfully starting an idea has on individual starters over time. 

As we progress over the coming months we plan to share more of our learnings and we’d love to learn from others in the field about the challenges you face and some of the ways you’re overcoming those challenges.

I’ve pulled together a reading list of some of the resources and organisations that have inspired our thinking so far, you can find that here.