The sector doesn’t talk enough about scale.
My opinion has always been one borne out of frustration. One where successful businesses can produce their products at a cheaper cost when they sell more, yet one where nonprofits are ultimately punished for their successful programs because they have to hire more staff and seek more funds to support their work.
But evidently these frustrations lacked the nuance to hold up in the face of basic econometrics (of which I am beginning to learn, understand, and appreciate).
You see, I was looking at economies of scale and not scaling true social sector innovation. As you know nonprofits are not that simple, not that binary, and definitely not structured in a way where the average costs per unit of output decrease with the increase in the scale or magnitude of the output being produced.
The Voltage Effect
I write this on the back of spending time this week with John List, the current Chief Economist of Walmart, The Kenneth C. Griffin Distinguished Service Professor in Economics at the University of Chicago and formerly at the White House and as Chief Economist at Uber and Lyft. John was in town as the closing plenary speaker for Society for the Advancement of Economic Theory (SEAT) 2022 conference and on the back of his new book The Voltage Effect: How To Make Good Ideas Great And Great Ideas Scale.
John also looks into the ways people make decisions regarding charity. Charitable giving is an important component of each country’s national economy, yet few organisations understand the best way to promote their cause. Over the years John has conducted multiple field experiments to examine the factors that influence people’s charitable decision-making which make for fascinating reading.
John’s speech had my mind buzzing for the full 90 minutes and contained threads of economic theory in which many lessons and opportunities could be observed for nonprofit organisations. Our work too often rests on hunches and intention and not on a careful examination of data and the science of using science to identify and implement promising ideas – the science of the economics of philanthropy as it were.
A good review of the book I found online by Yael Levin Hungerford succinctly summed up the key lessons found within its pages – the “Five Vital Signs” – and why desirable results don’t replicate when a policy scales: false positives; failure to know one’s audience; scarce ingredients; spillover effects; and cost of sustaining at scale. Once an idea passes the Vital Signs test, The Voltage Effect offers insights from the field of economics to ensure its success at scale: from experimenting on the margins to setting up the right kind of incentives (hint: we are loss averse) to creating a healthy culture.
Scaling Promising Social Programs
At the end of the day not all great ideas scale and that’s ok, but we have to identify whether that is because of the chef or the ingredients as John would say. For example, a restaurant can be wildly successful in one location but not work in others which is largely down to the chef. If it does take off in multiple locations and is franchised then that’s attributable to the ingredients and systems. From a nonprofit standpoint that could also be linked to local leadership versus a common social issue such as cancer or homelessness.
The key here is in testing a program’s ability to scale and this is where we apply our philanthropic lens.
John shared his opinions on A/B testing, sharing that in the academic world, the petri dish that forms the outcomes of that research are largely in a controlled state which are bolstered by the best inputs available (e.g. we do all we can to ensure our hypotheses are correct) .
“People tend to have unrealistic expectations about A/B testing. Vendors and agencies alike would have you believe that experimentation platforms are magical money machines that turn button colours and copy tweaks into double-digit conversion rates increases.
One highly resourced program I know of built its business case on a strike rate of 50%. Every second experiment they ran, they proposed, would generate a revenue uplift! That’s complete and utter madness.
“The vast majority of experiments will not impact your tracked metrics one way or another”
A meta-analysis of 20,000 anonymised experiments by over a thousand Optimizely customers conducted by Stefan Thomke and Sourobh Ghosh gives a more realistic view. Just 10% of experiments had a statistically significant uplift for their primary metric. Airbnb, Booking.com and Google all report a similar rate of success:
At Google and Bing, only about 10% to 20% of experiments generate positive results. At Microsoft as a whole, one-third prove effective, one-third have neutral results, and one-third have negative results. All this goes to show that companies need to kiss a lot of frogs (that is, perform a massive number of experiments) to find a prince.
The vast majority of experiments will not impact your tracked metrics one way or another. Many optimisation programs struggle to acknowledge this fact. They plod along in obscurity, not wondering why they’re not getting the same results as ‘everyone else’. They sheepishly sweep their ‘failures’ under the rug which ensures that they will never learn from them, stagnating at level 1.”
Funding Replication To Replicate Successful Innovations.
What is missing from this process and what in fact shows the potential for scale is in the replication. Replication in its simplest terms is being able to copy the original experiment to see if you can produce those outputs with some degree of fidelity.
For John, replication is the option c for scaling (following the A/B component). It is where a number of critical scale features are added to the process and where important learnings innovation can occur. List out all your concerns and opportunities to scale and add them to the study, take into account quality, capacity, access, finances, geography, you name it.
It was mentioned at this point that not many replication studies were posted in science journals ( of which I found interesting), a fact that John ended up addressing by creating his own journal. He also found in his own research that, “The sciences are in an era of an alleged ‘credibility crisis’ (…) By combining theory and empirical evidence, we discuss the import of replication studies and whether they improve our confidence in novel findings. The theory sheds light on the importance of replications, even when replications are subject to bias. We then present a pilot meta-study of replication in experimental economics, a subfield serving as a positive benchmark for investigating the credibility of economics. Our meta-study highlights certain difficulties when applying meta-research to systematise the economics literature.
After doing a bit more reading, something really rang true for me in regards to scaling. If innovation doesn’t have to be new, just new to you (and potentially in a localised context) then why do we not fund replication efforts?
We fundraise for loan loss reserves in impact investing, we fundraise for organisational capacity building and we fundraise for flexible capital options, so why are we not investing in scaling solutions to the most critical social issues of our time? Is it time to publish a new cook book for our local nonprofit chefs?
Social Replication Toolkit
The great news is that cook book indeed exists and was developed Spring Impact (formerly The International Centre for Social Franchising), a non-profit that helps social organisations systematically replicate to scale.
As I mentioned previously, Hunger, poverty, disease: societies globally face a range of perverse problems. Spring Impact capture this by stating ‘Though promising solutions exist, the frustration is that we can’t seem to scale their social impact to match the true size of those problems.’
Their Social Replication Toolkit tests for ‘replication readiness’ and helps you assess your readiness for replication and help you to better prepare to replicate your solution.
Successes In The Field
Since moving back to Australia I have been impressed by the work, direction and innovation of Social Ventures Australia (SVA). A systems focused organisation who learn in their communities and support scale predominantly through impact investing, SVA recently undertook a review of 7 initiatives that have replicated their model to increase social impact.
The review is definitely worth reading regardless of the localised content as it helps readers understand the process, methodology and definitions (which you will know from my book are important in demystifying much of the new tech and trends out there in the sector).
Using the Spring Impact replication toolkit to guide their findings, SVA concluded with seven lessons and observations around replicating successful programs with the following summary takeaways:
“The experience of ventures that have succeeded in replicating for scale in Australia in recent years offers emerging lessons for aspiring founders, organisations, supporters and the social sector more broadly. The ‘how’ is important – identify and maintain a convincing value proposition, maintaining some measure of control over design and delivery, and measure and track just the essential.
However, these lessons have largely been learned by private sector licence and franchise businesses and applied by the social sector. The ‘why’ is more interesting. Not only is there a good reminder never to forget to deliver on your purpose but that, even as a small organisation, you can aspire to contribute to broader social change. A number of the success cases reviewed demonstrate the potential for relatively small-scale ventures to influence the underlying conditions of the social sector in which they operate.”
Testing Out Tired Funder Tropes and Baked In Biases
Funding replication efforts also lends itself to start breaking down a few structural issues associated with grantmaking too, all of which we tackled in Future Philanthropy with our feature on Laura Tomasko from the Urban Institute.
Many funders use data to inform grantmaking decisions and evaluate the effectiveness of a particular program or organisation. But do they think critically about where those data come from and what purpose they serve?
Laura recognises how current funding practices compound historical and current inequities, privilege, and power dynamics in the sector.
“There’s a lot of focus in the sector on achieving scale and supporting evidence-backed interventions,” she said. “I think it’s important to pair that goal with critical thinking about the extent to which systemic racism and bias might have created conditions that enable some organisations to grow and build evidence of success and others to stay afloat with few resources. Did the larger organisations have leaders who gained expertise through academic training and professional experience, with strong networks in philanthropy that enabled them to secure grants for evaluations? If given the same access to funding and evaluation, would the programs at less well-resourced organisations demonstrate similar or greater effectiveness?”
She went on to say this: “When making grantmaking decisions, funders who want to support organisations likely to achieve programmatic objectives would be wise to consider organisations that might have fewer resources and little to no formal evaluations, but whose leaders come from the community they serve. These leaders know what might work from lived experience and have the knowledge and trust of the community to bring about change. Funding within established networks can perpetuate inequities by overlooking community leaders and solutions that derive from that lived experience.”
Funding replication efforts here should start by (in the words of Professor List) ‘imagining what success would look like at scale, applied to the entire population with their varying situations, over a long period of time. With that being said, funders should think less about data as a tool for compliance and punitive action against organisations that have not achieved certain projections, and more as a tool for evaluation and to support an organisation’s learning, growth, and storytelling capacity.
“We want grantmakers to recognise the ways that data can empower their grantees to thrive while also acknowledging the flaws and limits of data,” Laura said. “When gathering data to evaluate effectiveness, think about who pays for the assessment, who determines the research questions, and what information is available and collected and by whom.
All of these factors influence the data that ultimately helps determine the effectiveness of a program or grant. And who benefits from the data? Do the data collection efforts burden grantees and ultimately not benefit the organisation itself? Or do they empower organisations to evaluate and improve programs, as well as equipping them with information to tell their story to their stakeholders?”
Scaling New Ideas
Not every social program needs to scale. There are people who are not scalable, some products or service are purposely not designed to scale. In other words, scalability isn’t for everyone. But we must approach the results of each program in a way that captures the issues of power, race and privilege (amongst other factors) and give the programs every chance of success especially if they have shown early promise.
This may result in multi-year grants and the opportunity to discuss the end of year results rather than be judged on them. So often it is seen that a funder will walk away from supporting a new partner due to limited success in year one, where it would be much more constructive to perform some self-reflection and perhaps providing capacity building support or increasing funding to help them achieve their vision.
This is something that would be a hallmark of funding replication efforts and something that should merit more attention in our sector. A first step of course would be to get yourself a copy of The Voltage Effect which is essentially a guide on how to get rid of bad ideas and make good ones better. So, whether you choose to scale or not to scale, understanding the techniques behind scalability will help you in all aspects of your work.
From my perspective, that work begins now by building awareness around the need and impact of funding replication in our sector. Scaling a narrative of impact after all is another part of our journey to be more effective in our outcomes.
Photo Credit – The University of Nottingham
Future Philanthropy Blog 