Warming impact measurement in a changing world

Have you ever been in a situation where people you are working with to develop and implement outcome or impact measures for tell you that they don’t make sense, and they don’t capture the real value of what they are doing? But you say we need metrics for the funders if they are to have any hope of being funded again. Sometimes people agree to fit themselves into boxes that don’t feel right because of the threat of not being funded. Others continue to resist and ask, but isn’t there another way to represent what we do to funders, so that we don’t lose what’s real and authentic about what’s happening here? And honestly, in many complex situations I’ve been in, I can see that the metrics being suggested or generated are only a very small part of the picture, and that they do the people doing the work, and the value they are creating, a huge disservice.

Meanwhile, funders keep asking, what impact is that programme or initiative we funded having? Surely, they say, there must be metrics or data that can show us their results by now (in my experience, this question is sometimes asked in the first year of funding, when people haven’t even moved into offices or opened bank accounts yet).

It is reasonable for funders to ask how well resources are being used, but often the power imbalance between funders and grantees means funders’ view of the world and their need for mostly simple metrics wins out, and time and again, I hear from grantees that they don’t see themselves in the reporting.

I’m sure there is a middle ground somewhere. It shouldn’t be a zero-sum game.

Having recently worked on a few projects developing outcome and impact frameworks where design has been inclusive, engaging, highly participatory and grounded in what’s valued by grantees and funders, resulting in grantee-led outcome specification, multi-method data collection, led and owned by grantees, with subsequent analysis and sense-making a collaborative and engaging process, with reporting also co-designed, I think we need more of this if we really want to understand the value and impact of what funders and investors are applying their resources to.

A couple of questions I’ve been asking myself are:

· How do we develop participatory and inclusive approaches to impact measurement that authentically bridge the gap between the board table of investors and funders and realities on the ground?

· How do we design outcome and impact measurement systems that are relatively low-cost, participatory, inclusive and account for complexity?

Features of warm data systems

Given the complexity of the challenges faced by people and the planet, and the many difficulties of impact measurement (see below), there is a key adaptive opportunity to ‘warm’ our impact data systems, so they can more authentically represent what is valued and support the decisions we need to make about our futures.

In contrast to much current impact measurement practice, I would argue that the features of warm[1] impact data systems have:

· a relational and participatory underpinning

· are designed with the things grantees, communities or those with a stake value as their basis

· are built with trusted experience and expertise

· support collaborative sense-making and evaluative reasoning, and

· their development and use create a range of different kinds of value (capability, understanding, insight, etc).

On the one hand, there is an acknowledgement of complexity, but on the other hand, there is a growing focus on impact

With growing complexity, some funders are shifting away from focusing on what is being spent and what can be easily counted, such as the number of grants and the size of grants, to understanding the difference funding makes, to whom, and the extent to which it contributes to value creation. At the same time, there is a growing acknowledgement that creating valuable and meaningful change is hard. Often, change takes a long time and happens due to many things; there is rarely a single contributor, and the trajectory of change is often unexpected or surprising.

Along with this focus on results and value, there is also an increasing focus in New Zealand and globally on creating and measuring impact. The landscape of options for impact measurement and management is expanding, with a plethora of approaches and ideas about how impact measurement and management should or could be done. In the impact investment world, a thriving movement is developing principles and standards for impact measurement and management. There are high hopes that just as we have built systems for understanding financial data, we can create and build infrastructure for impact data.

However, there are also concerns. Evaluating and measuring impact is challenging, especially for complex settings including, but not limited to:

• Multi-system initiatives such as sustainable and environmental development, cultural revitalisation and many other initiatives, where there are many moving parts

• Portfolios of investment with multiple grantees across different sectors and locations

• Many smaller project and program interventions that do not have the human and financial capacity for impact evaluation

• Situations where impacts are unlikely to be realised within decision-making timeframes

• Unrealistic expectations about the capabilities of communities and organisations to undertake impact measurement.

Despite the challenges of impact measurement, organisations face increasing pressure to measure and demonstrate their impact as demand for limited funding and resources continues to grow. The experience of impact measurement is not always positive, with measures sometimes imposed by funders and narrowly focused on what can be measured rather than what matters to people in communities. The cost of impact measurement can be high, with communities frequently shouldering a considerable burden of the effort, and the return is quite often not considered worth the investment for the limited insight gained. With growing complexity, these issues are magnified, rather than resolved, especially as the vast majority of impact measurement is typically not designed to capture the nuances of complex situations.

With organisations and communities still investing precious time and considerable resources in impact measurement, there is a need to find an approach that is not imposed, but designed and built by and with communities and organisations, that expresses what matters, what is valued, can take account of complexity, is practical and delivers something useful to communities, organisations and funders.

What underlies the complexity/impact conundrum?

One reason for the challenges of impact measurement is that many approaches to impact measurement and evaluation are underpinned by assumptions about the world that are no longer helpful as we navigate our way through system change and polycrisis. Industrial managerialism assumes the world is a controllable and predictable place, like a machine, and that we can manage our way out of any problem with better data. Our bureaucracies and managers continue to operationalise impact, with ever more detailed measurement, privileging what we can measure over what might actually really matter. However, as much as we want to believe that this measurement will give us insight into people’s lives, these measures are most often far removed from people’s lived realities.

One impact measurement specialist, Jess Daggers, has written that she is ‘concerned that the majority of impact measurement (and management (IMM) generates spurious insight that only has passing resemblance to the reality on the ground’. She argues that there is a general lack of correspondence between the data generated by a lot of impact measurement and what’s actually happening on the ground.

The measurement of impact isn’t neutral either. The measurement of our lives and the ‘impacts’ of social programming construct new realities. The mostly unseen decisions about what and how to measure are social and political choices determining which outcomes are more important, leading to communities and providers prioritising these things over the things that really matter to them. As Sam Pressler argues, “In the logic of managerialism, everything is reduced to the instrumental: participants are expected to complete a program in order to achieve an outcome, while nonprofits and governments are expected to report on these outcomes in order to receive more funding.”

The connective tissue of our lives, including relationships, cultures, the happiness and sadness of daily life, and the feelings of connection and disconnection, is de-emphasised, even disregarded, because it is harder to measure. Graciela Selaimen has written that “The language of…impact investing has long been structured around metrics, scale, and measurable results. But this logic risks blinding us to what truly matters: that which is alive, relational, and emergent.” The engine of ‘impact’ measurement, instead of amplifying better impacts, can inadvertently constrain the possibilities of what living is all about. And unfortunately, it’s this kind of thinking that has led us into the mess we’re in.

Nora Bateson argues that “[t]he irony of this is that the most pernicious problem facing humanity at this time is the habit of attempting to respond to living, non-linear, multi-causal issues with responses geared toward specific outcomes. To do so is to produce actions that are fundamentally out of sync with the livingness of the problems”.

There are now many who recognise the lack of authenticity in much impact measurement and are uncomfortable with this ‘cold’ data, and who are searching for other approaches that are ‘warmer’ and closer to the livingness of things[2].

So, what can we do about this situation?

As the calls for systems change and transformation grow louder, so too does the recognition that we must build new kinds of capabilities and infrastructures to do this work. Given the massive entanglement of wicked issues facing humanity and the planet, if we care about and value creating and building an abundant, regenerative future, then we need to lean into developing different ways of understanding our impact.

If we understand the world as complex, then ‘knowing’ about the world is better approached using methods that reflect this complexity and are able to capture context, path dependencies, patterns, feedback loops, interconnections, and so on. It is unlikely to be well understood or captured using methods designed for control and prediction. In other words, how we come to know about the world (the epistemology) has to mirror what we understand the nature of reality to be (the ontology). It is not logical to analyse a situation as if it operates like a machine when it does not.

Relying only on methods and measures (that are mainly quantitative) grounded in a different understanding of reality is a folly. It is not rigorous to analyse a situation as if it operates like a machine when it does not. How we choose to know about the world has to match what we think about the nature of that reality.

Conventional impact measurement is not usually grounded in this way of thinking and may even act as a barrier to understanding in these uncertain times. Among some in the impact measurement field, there is a growing understanding of the need to stretch current practice so that different approaches, mental models, and ways of seeing the world can be included in impact measurement and management practice. For example, shifting our measurement practice to be more collaborative and to ‘ move towards shared learning and sensemaking’.

Making sense of complex situations so that we can take meaningful action requires approaches and methods that allow for multiple perspectives, drawing on human experience and expertise, as well as contextual knowledge such as cultural and spiritual processes and practices, intuition and instinct, alongside more rationally based approaches. Our methods must be able to accommodate paradox and tensions, and also allow for situations to evolve over time and for new features to emerge. Approaches that involve engagement, dialogue, narrative, and relationality become imperative in complex situations.

One of our challenges is that we are deeply entrenched in using legacy approaches and systems, making it feel almost impossible to completely overhaul these systems. So, one way forward is to find ways to work within and adapt them. And as we reform and ‘warm’ the systems, we will begin to reveal which aspects are worth holding onto, and which parts are no longer helpful and can be left behind. Learning to spot the landscapes, openings, potential, tensions and challenges of new opportunities and learning spaces will be part of the development and adaptive work of impact measurement.

Bright spots are emerging internationally, with increasing recognition of the need for iterative, adaptive impact designs and practices, as well as for more participatory models that recognise the value, not just the impact, to grantees and the people and ecosystems supposedly served by investment.

I am engaged in some grounded work on this, and will be watching, tracking, and engaging with others who are wrestling with the challenges, whilst also recognising the need for more adaptive, inclusive, and authentic ways of doing this stuff.

I haven’t yet answered my original questions; I’m still experimenting and grappling with the tensions in small ways, and I’m open to others’ thoughts.

All thoughts are my own, and I’m grateful to Julian King for reviewing my thoughts before putting them out into the world

---

[1] Credit for this terminology is to Nora Bateson, see https://batesoninstitute.org/warm-data/

[2] See https://batesoninstitute.org/warm-data/