Over the past 10 years, the Central New York Community Foundation has invested more than $1.6 million in helping local nonprofit organizations acquire and cultivate the skills needed to measure their effectiveness. (View list of participants.) Now, we look to the next stage: investing in nonprofits that are utilizing performance management skills in action.
Written by Colby Cyrus, CAPM, Program Manager and Frank Ridzi, Ph.D., Vice President, Community Investment
Download as PDFIntroduction
How would you measure making a difference? After all, this is why nonprofits exist – to provide a service to the public, to meet a greater good. And although anecdotal stories about an organization’s services are compelling, they can come up short when it comes time for the nonprofit to prove that what it does actually works. Our experience working with nonprofits has taught us that quantifying how much of a difference has been made is frequently an area of difficulty.
The Central New York Community Foundation’s (CNYCF) capacity building initiatives are designed to meet the pressing needs of nonprofits. In 2009, we found that one of the most immediate of these was the need to develop a framework to demonstrate success and impact. Nonprofits, however, had little capacity or know-how when it came to measuring and managing their own performance. This is the first link in a chain that looks something like this:
Since 2009, CNYCF has been working on the first link of the chain: helping local nonprofits to increase their capacity and learn best practices for measuring their performance. Now, it’s time to move ahead and support those same organizations in implementing their newly acquired best practices, because we have proved that organizations’ capacity and practice have improved.
The following sections will detail why measuring success is so critical (and difficult) for nonprofit organizations, how CNYCF addressed this challenge, and what we are doing going forward to meet the changing needs of our partners.
Why & How We Measure ImpacT
At the most basic level, nonprofits address social needs that government and for-profit initiatives fail to meet. There will always be a need for the services that they offer. In fact, the sector is only expanding, with nonprofit jobs jumping 18.6% from 2007 to 2017 (Johns Hopkins University 2020). So, if this part of the job market is growing, and nonprofits are playing an increasingly important role in meeting society’s needs, how can we be sure that the intended effects are actually coming to fruition? In the for-profit world, indicators for measuring success are clear: high profit margins and more sales, and, ultimately, higher monetary returns for shareholders, mean that the company is effective at doing its job. Yet, in the nonprofit sector, there is no way to measure sales, and there is no profitability. What, then, does success look like? Just as doing the work of nonprofits is not easy, the work of proving impact is equally difficult.
Efficiency is a metric that is commonly used in discussions of business, but is it effective for the nonprofit sector? In the for-profit world, goods and services must be delivered in a manner that allows the income derived from the sale of those goods or services to be greater than the cost of producing those goods or services. Efficiency and optimization are key components of that. The most efficient companies often make the most profit, and it is these that come to mind when we think of “successful companies”. But for a nonprofit, the goal is to make a long-lasting difference. For that reason, measuring efficiency will not do the trick. How do we measure something so seemingly intangible as societal benefit? We can see, of course, that the benefit is real. There are countless nonprofits doing countless amazing things all over the world, and there is no disputing the benefit that they bring.
Imagine that you are the executive director of a highly reputable nonprofit that provides school supplies to local K-12 students. Parents tell you all the time that without your nonprofit, they would not be able to acquire the items their children need. Your staff is always busy, and there is never a shortage of families waiting to receive the supplies you give out. You know, intrinsically, that you are making a world of difference. After all, you can see it and hear it, every day. But how can you quantify that? When it comes time for you to write a grant, for example, you will need some concrete numbers to demonstrate your effectiveness.
It all comes down to measuring impact, and being able to illustrate the change that a nonprofit has made in peoples’ lives, with data. It’s also about using those same numbers to inform an organization’s operations. In other words, the key to proving impact is performance management: using data to prove your impact, and continually identifying areas where you can improve. A solid framework, though, must precede performance management efforts if they are to succeed.
Investing in Infrastructure for Performance Management
In 2009, we noticed a need in the community to develop this very kind of framework. Specifically, local nonprofits often fell short of what they needed in terms of data collection and management, which severely hindered their goal of demonstrating and increasing impact. Around this same time, government funding decision making began to turn towards the focus of measurable accomplishments. The most popular question then became “how do we know that our investment is actually doing anything?”
Federal funding, upon which many nonprofits rely to sustain and carry out their programming, increasingly required proven impact, often at quite detailed levels. But across New York State (and the nation), few nonprofits had the capacity in place to be able to meet this requirement. This was certainly true of the nonprofit community in Central New York – there simply wasn’t enough of an infrastructure among nonprofits to be able to handle the level of performance measurement and management that was being requested.
In response, we began working with our nonprofit partners to determine what their needs were, and how those needs could be addressed. While at first an informal process, these conversations eventually resulted in the design and implementation of the Performance Management Learning Community (PMLC), which offered grants and learning opportunities to nonprofits who sought to improve data measurement through increasing skills and competencies including database management and statistical analysis. The theory was that with better data and performance management, nonprofits could more clearly illustrate what was working and what was not, leading to more funding opportunities and more impactful storytelling. In other words, PMLC sought to instill in participants the very framework that they needed.
In light of the fact that participants would be partaking in PMLC with differing levels of experience and comfort in performance management, the program was eventually structured into prep, intermediate, and pro levels. In this way, PMLC was able to accommodate any nonprofit that wished to participate. Those taking part in PMLC Prep would learn the basic concepts of gathering and utilizing data during a four-month course, while their colleagues in PMLC Intermediate would share their project implementation experiences with each other in a much more intensive one-year course. PMLC Pro was also a four-month course, which allowed advanced participants to collaborate, share, and develop a project to implement within the Central New York region. In sum, over 100 organizations participated in the program and benefited from the grant awards made available:
In 2022, CNYCF’s 5-year strategic plan came to a close and PMLC phased out after running for 10 cycles. Ample data from participant exit surveys made it possible to assess whether the program had accomplished what it set out to do at its inception and to what degree participants and their organizations took their new performance management framework and put it into practice. By determining the success of certain metrics from the data, it is possible to gain insight into how much more capacity nonprofits have gained in terms of measuring their own impact. The data provide insight into three important measures: individual capacity, organizational capacity, and organizational practice. In other words, using the data, we can identify changes in knowledge and know-how of both participants and their organizations as well as changes in the way organizations do what they do.
Data analysis showed significant increases in participants’ comfort levels with technical skills that form the basis of being able to measure impact. Using Excel, creating logic models, using Access, and using pivot tables were all topics addressed in PMLC, and participants reported feeling considerably more comfortable in these areas after the program.
“As someone who uses Excel but was never formally trained in it, I learned new skills for analyzing data,” said Todd Goehle, director of community engagement at PEACE, Inc. and former PMLC participant. “Additionally, by learning about and discussing performance in group settings among different agencies, I developed new skills and confidence in discussing data and performance with staff.”
We hoped, then, that higher individual capacity would increase the capacity of the organization as a whole. With that in mind, we also used exit data to determine how the knowledge and capabilities of participating nonprofits changed after PMLC.
As discussed, organizations must have at least a moderate performance management framework in place in order to effectively measure their impact. Using metrics taken from the McKinsey Capacity Assessment Grid, organizations assessed the state of their framework (in other words, their level of organizational “know-how” and processes). For example, a “high capacity” category describes organizations that have a small number of concise and easy-to-measure Key Performance Indicators (KPI’s), use internal and external benchmarks to measure progress, and then use progress along those benchmarks to improve practice. We saw a notable jump in the number of organizations who fall into the category of high capacity and therefore embody the best practices of performance management.
Accordingly, we hoped that organizations that have increased capacity would undertake more of those best practices. In regards to carrying out performance management, some of those best practices include getting buy-in from an organization’s executive director and other staff leadership, constructing a logic model for the program, identifying indicators for specific outcomes, and using findings from data analysis to provide direction to staff. Data analysis showed that a greater number of organizations performed the tasks after PMLC as opposed to prior, demonstrating that the program succeeded in fundamentally changing the way that organizations go about measuring impact (Cyrus, 2022). At a high level, the process takes the following form:
Many of those who engaged with the course continue to see the value of PMLC in their daily work. Here is what past participants shared:
Patricia Sullivan, deputy director of the Syracuse Northeast Community Center (SNCC):
“SNCC is able to look at and identify trends that need to be addressed. For example, when we learned a majority of the people that we were serving didn’t have access to food, we expanded our pantry hours. When we learned that people in the community struggled with transportation, we provided bus passes. Being able to analyze the data that we collect has helped us to improve our programs tremendously,”
This example demonstrates one of data’s fundamental purposes: it tells us where the gaps are, allowing us to address needs where they arise. Data-driven decisions can also be monumental in an organization achieving its strategic goals.
Todd Goehle, director of community development, PEACE, Inc:
“Participating in PMLC proved to be a critical ingredient for reaching one of the agency’s strategic plan goals, which was ‘to align programs with community needs and goals’ – as part of the agency’s move towards this goal and towards developing a data-driven decisions culture in practice, PMLC provided the critical spaces and resources the agency needed for both training and collaboration.”
Investing in Organizations and Projects that Use Best Practices
With PMLC phasing out, and participants having developed the frameworks and infrastructures they need, CNYCF is entering a new phase focusing on investing in nonprofits that utilize performance management skills in action. In other words, we are shifting our investment focus from learning skills to using skills. This is a key adjustment: we spent a decade empowering nonprofits to learn what they needed to learn in the realm of performance management, but the real change in the sector will come when those organizations are further enabled to put what they have learned into action. The ultimate goal is to foster a more data-savvy nonprofit sector. We are now at the point where we can support more advanced initiatives, such as helping organizations to document the needs of their clients and then record the impact that the organization is having.
To help organizations in this phase, we set up a free performance management data system, the Life Needs Assessment (LNA), which assists organizations in monitoring their clients’ needs, identifying which needs are not being met, and then honing in geographically and otherwise to address severe service gaps (Ridzi, 2020). This usage of the LNA is exactly what the PMLC was designed to do. Not only do participating agencies have the performance management framework in place, they are actively putting it into practice by managing the data that comes from LNA, and continually using those data to determine the impact their services are having over time. LNA can adapt to measure any needs and outcomes both within and across organizations. It also supports analysis of needs and outcomes through a diversity, equity and inclusion lens. But perhaps most importantly, the LNA is designed to foster data-driven collaboration across organizations (through such features as “data dating”) in ways that include the voice of community residents and nonprofit clients.
We have begun to turn the corner toward a focus on higher impact grant making, focusing on scaling up programs that demonstrate promising outcomes. PMLC was a natural predecessor to this phase, and allowed participants to develop the framework they needed to be able to demonstrate the promising outcomes that grant makers such as CNYCF seek through their funding and strategic work (Ridzi, 2012).
Conclusion
Performance management will likely always be an uphill climb for nonprofits. The data needed to determine impact simply require more effort to collect and interpret than, for example, profit and loss margins. In addition, some constraints on nonprofits are endemic and performance management must be implemented and improved upon despite these challenges. Namely, this includes issues of funding, staffing levels, and time management for the often-overworked staff that nonprofits have. In addition, given the high turnover of staff in the nonprofit sector, which we experienced directly among participants and grantee organizations during the duration of the PMLC, developing and retaining high levels of evaluation expertise on staff will likely persist as a challenge for most organizations. This underscores the fact that the work of performance management is never done – an organization can always strive to improve its practice in this area and infuse this work into its operational culture.
To that end, we will continue to meet nonprofits where they are and will offer programming that reflects the needs of our community’s service providers. When PMLC was first conceived, those needs consisted of developing a solid performance management framework and incorporating best practices into everyday business. In that aspect, as we look back on the 10-year run for PMLC, much of the mission that we defined has been accomplished. Now, in 2022 and onward, the need of our partners will be to receive support to actually carry out what they have learned. This comes with its own unique set of challenges, but will ultimately be carried forward with the benefit of data-informed insight and decision making.
PMLC participants
References
- Cyrus, Colby. (2022). An Analysis of Improvements in Capacity and Practice in PMLC Participants. Central New York
Community Foundation. - Johns Hopkins University. (2020). The 2020 Nonprofit Employment Report: The 3rd largest employer faces the COVID-19 crisis – Johns Hopkins Center for Civil Society Studies (jhu.edu) June 16, 2020 Chelsea Newhouse. http://ccss.jhu.edu/2020-nonprofit-employment-report
- Ridzi, F. (2012). Managing Expectations When Measuring Philanthropic Impact: A Framework Based on Experience. The Foundation Review, 4(4). https://doi.org/10.4087/FOUNDATIONREVIEW-D-12-00007.1
- Ridzi, Frank. (2020). Goldilocks Data-Connecting Community Indicators to Program Evaluation and Everything in Between.” (p. 15-35) In Frank Ridzi, Chantal Stevens & Melanie Davern (eds.) Community Quality-of-Life Indicators Best Cases VIII. Switzerland: Springer.10.1007/978-3-030-48182-7.
