News

Did law school teach me something about evaluation?

May 3rd, 2013 by Joe Kriesberg

My teenage sons enjoy making fun of me for spending three years in law school and then getting a job that does not involve being a lawyer. I try to explain that you learn a lot in law school that is useful for other professions (how to write, think, and argue, for example), but they remain skeptical.

I was thinking about this recently because I was engaged in several conversations about program evaluation and how we can better understand whether our efforts are having the desired impact.  As I wrote in my last article, our field is often wrongly and harshly judged because poverty rates in America have not declined in the last fifty years. For some, this is evidence that what we are doing is not working. What if poverty rates are the wrong benchmark to measure our field’s success?  If so, how do we measure our efficacy and overall performance?

I certainly don’t have a simple or complete answer for these questions, nor does anyone else. At some level, it is impossible to answer given that the community development field seeks to serve many different constituencies with different needs and goals. And many actions and programs may benefit some at the expense of others.

Given the complexity of the questions and the murkiness of the data, I would like to suggest that we consider applying some legal thinking to the question.  Perhaps, the way our legal system resolves conflicts – in particular civil law suits – offers some guidance.

First, in a civil legal case, the jury must make a decision based on a preponderance of the evidence, unlike a criminal case in which the burden of proof is “beyond a reasonable doubt.”  This strikes me as the right standard for community developers because the complexity of human life means that there will always be some doubts about what caused certain outcomes. But that does not mean we can’t make judgments and decisions about what is or is not working; about which programs should be funded or not funded.  Most of the time, a preponderance of the evidence will point in one direction or the other; at a minimum it can significantly reduce the guess work.

Second, in a civil case, the attorney assembles as much evidence and data as s/he can to support his or her version of the case.  This evidence can take many forms – physical evidence, testimonials, statistical data, photos, and other sources of information. The fact that there may be some evidence that is contradictory, confusing or incomplete does not automatically negate the case depending on the overall weight of the evidence. This can be applied in our context, where community developers and others can use stories, anecdotes, output data, surveys, population level data, pictures, and other forms of information that begins to paint a picture of what is happening.

Third, and most important, a good trial attorney puts the evidence into a story that the jury can understand. Like all good stories, a lawyer’s story must have a beginning, middle and end, with a logical flow throughout that allows a jury to conclude, “yeah, that makes sense.”  Without a story, the evidence is just noise and is likely to be unconvincing. Without a story, jurors will have a hard time reconciling contradictory evidence and they will be unable to fill in the blanks if the evidence is incomplete. With a clear and logical story, jurors can make reasonable assumptions and inferences based on the inevitably incomplete evidentiary record. Jurors can also rely on scholarly research to help them understand the evidence and how it fits in the story. In fact, jurors can even apply common sense!  All of this is also true in our context. We need to have a story (in evaluation jargon we might say “theory of change” or “logic model”) about how we think the world operates and how we can change or alter its course. Data and evidence can then be used to see if things are playing out in a manner consistent with our story/theory/logic model that we have developed – or not.

As we work with DHCD and our members to devise an evaluation system for the newly enacted Community Investment Tax Credit program, I hope we can apply some of these ideas. CDCs should be able to articulate a theory or story about why they think their efforts are creating the desired change. They should have both quantitative and qualitative evidence that can shed light on whether they are having the desired impact.

We should not expect proof beyond a reasonable doubt, but rather a preponderance of the evidence as to whether our efforts are working. Gaps in data or knowledge should not be seen as evidence that programs are failing, nor should the complexity of human and community behavior deter us from seeking better understanding about the impact of our work. 

In future articles, I will offer some thoughts on an overall theory of change for the CITC program. I will also reflect on what I have learned from collecting data for the GOALs initiative for the past 10 years and how those lessons might be applied in the CITC context.   With a clear story or theory about our work and with more accurate and complete data, I am confident we will be able to effectively evaluate the work of our member organizations and the CITC program itself.  Of course, this model does leave a major challenge which is aggregating and comparing the work of different CDCs given that each group may have a slightly different approach and set of goals.

I’m not sure I needed three years of law school to figure this out – most people know the basic elements of a civil trial.  And I suspect my kids will always think that I wasted my time and money going to law school.  But for me, I continue to believe, based on a preponderance of the evidence, that I made the right decision to go to law school.  If nothing else, I met my wife during law school and my kids ought to see the value in that – even if meeting someone was not part of the logic model that motivated me to enroll in the first place.

Commenting Closed

Tell Your Story with a Map

May 1st, 2013 by John Fitterer

As a fundraiser, I want to tell my story as clearly and concisely as possible. This can be quite challenging, particularly because CDCs are diverse organizations with a number of programs addressing numerous causes in their community. One tool that I've used for years is maps, and with new data sources, the power of maps keeps growing.

Maps help remove many of the storytelling barriers by showing complex data almost instantly.  Where are the homes you’ve built located?  What are the busy transportation corridors in your service area?  Where are there gaps in critical services, such as poor access to supermarkets, pharmacies and job hubs?  Plot this information out quickly to tell your story concisely and compellingly by sharing your successes and opportunities for future projects and programs.  For a place-based organization, like a CDC, it’s a powerful tool to know how to use.

I recently attended a presentation at the Boston Foundation by Professor Sarah Williams of MIT.  Her project, “Million Dollar Blocks”, is on display in New York’s MOMA.  In this project, she mapped out the addresses of New York residents who are incarcerated.  Those areas with phenomenally high rates of incarceration are called Million Dollar Blocks (CLICK HERE to see one of her maps). (Check out more work by Professor Williams and her colleagues) This is a map that tells a story and conveys an enormous amount of information in seconds.  For civil rights advocates and professionals in Community Economic Development this isn’t necessarily revolutionary information.  However, by presenting it in such a convincingly clear and concise manner, it tells a powerful story in seconds. If you’re a nonprofit looking to reduce incarceration rates, strengthen community engagement and organizing efforts, create opportunities for advancement and raise funds toward these causes, this could be a very compelling part of your presentation. 

Many of us can’t afford GIS mapping tools, but take the time to learn about some of the resources that are available online for free.  Google Fusion Tables are worth learning more about.  You can upload a spreadsheet with data points, property listings etc. that are geo-located in minutes.  The Commonwealth offers numerous data sets through their Oliver system for free.  Also, MAPC has the DataCommon project.  The Boston Foundation with MAPC runs the Indicators Project, which looks to be expanding some of its services for the entire state in the near future. Of course, Census.gov has resources, such as the FactFinder.  Classes are offered frequently for Census resources, some of which are through the Mel King Institute, and MAPC’s DataCommon.

It only makes sense for CDCs as place-based organizations to use maps to advance how they share their projects, programs and, of course, performance to mission.  With free online resources to get you started, learning how to use these tools is well worth the time commitment.

Commenting Closed
Subscribe to News