The Strategic Value of Program Evaluation

Program evaluation is a valuable tool to help economic development programs and organizations improve their impact and effectiveness. Faced with increased competition for limited funds and higher expectations from stakeholders, organizations can use program evaluation for to advance multiple goals:

• Documenting program impacts to build support from funding sources and stakeholders 
• Improving understanding of what works and what doesn't 
• Determining critical success factors to better target resources 
• Identifying ways to improve operations 
• Creating tools to regularly document impact and assess effectiveness

Our Approach to Program Evaluation

Karl F. Seidman Consulting Services uses a three-dimensional approach to program evaluation to maximize the evaluation's strategic value to your organization. The three dimensions of the evaluation include:

1. Program Impacts: documenting the contribution of your program to positive change, quantifying program benefits, and comparing impacts to goals, 
2. Program Operations: identifying best practices and areas for improvement, finding ways to reduce process time and costs, and benchmarking outcomes, processes and costs to comparable organizations 
3. Assess key relationships: gauging client perceptions, identifying who is critical to your program's success, and determining ways to extend and improve these critical relationships.

Defining the "Program Model" is a key first step to inform a sound evaluation. This "Program Model" is a concise, logical flowchart of your program's strategy that relates your program "outputs" to the desired economic development "outcomes". It identifies the chain of actions needed for your program to be effective and clarifies the appropriate impact measures for both "outputs" and "outcomes". By elaborating the critical path to achieve your goals, the program model also helps identify key operating issues and relationships to evaluate.

Evaluating Impacts: Addressing the Attribution Problem

A critical issue for any program evaluation is determining whether the observed outcomes were caused by your program or would still have occurred in the absence of your assistance. This issue is akin the problem that Clarence the angel faced in move, It's a Wonderful Life: how to show what Bedford Falls would be like if George Bailey had not lived. There are three approaches to addressing the attribution problem:

1. Experimental Design which involves prior random assignment to experimental and control groups to compare the outcomes for those who participate in the program (the experimental group) with those who do not participate in program (control group). 
2. Quasi-experimental Design where a comparable control group is constructed after the fact, i.e., after the program assistance occurs to compare outcomes between those receiving assistance and those who did not receive assistance. 
3. Client surveys where program participants project what would have happened without the program intervention

The most effective approach to addressing attribution is chosen based on the nature of the program, the timing of the evaluation, and the resources available for the evaluation.

Evaluating Program Operations

Beyond documenting impacts, the evaluation process is extremely useful in identifying ways to improve operations in ways that use resources more effectively, enhance client satisfaction and increase impacts. Our evaluation approach employs four tools to assess the effectiveness of program operations:

1. Mapping the program's transactions and processes to generate a detailed understanding of program operations and procedures. This mapping often uncovers duplication, inefficiencies, and potential areas for improvements. 
2. Benchmarking against comparable programs establishes performance standards for judging program impacts, benefits and costs. It can also identify where a programs' greatest strengths and weaknesses lie. 
3. Surveying clients is an invaluable way to get input from your most critical stakeholders on what is working and where problems lie with your program. 
4. Interviewing key stakeholders and partners to gain their experience working with a program and their perceptions and recommendations about best practices and areas for improvement.

For all these steps, a knowledgeable independent evaluator is an essential and critical asset. The independent evaluator provides confidentiality to clients and stakeholders yielding more honest and comprehensive feedback on your program while also bringing an objective and new vantage to the analysis. .

Assessing Relationships

Successful economic and community development programs are built on strong relationships with multiple partners. With the devolution of federal programs, a greatly expanded non-profit sector, and the growth in corporate leadership, today's environment demands attention to building a strong network of relationships to support and enhance your economic development program. The third dimension of our evaluation approach, assesses the strength and strategic content of your current relationships while identifying new relationship-building opportunities. Three tools guide our evaluation of program relationships:

1. Identify critical relationships based on your program model and operations. This analysis encompasses relationships needed to: (a) market the program and secure referrals; (b) secure complimentary inputs and services, (c) generate desired goals and outcomes from extending your program's direct activities. 
2. Assess the scope and effectiveness of relationships from program data, staff and stakeholder interviews, and client surveys. 
3. Scan the environment to uncover new relationship opportunities.

Gain the Advantage of Direct Program Management Experience

Unlike many program evaluation firms, Karl F. Seidman Consulting Services brings direct program management experience to the task of program evaluation. Karl F. Seidman uses his knowledge and insight, gained from over ten years experience managing state and local government economic development programs, to create sound strategies and recommendations to improve your effectiveness and increase your accomplishments.

Learn more about our Program Design and Evaluation Experience