Who said it would be easy?
Oh, measurement. I would have never thought that the thing I once had so much faith in would come back to haunt me. I shouldn’t say haunt – it doesn’t really haunt me. It has just become much more difficult and complex than I ever imagined.
I am an economics major. My favorite classes have been the two econometrics classes I have taken. I am fascinated by the way regression analysis works, and the methods economics have come up with to tease out causal relationships (Instrumental variables, difference-in-differences, regression discontinuity, fixed effects, etc. – if you don’t know what these mean just ignore this. Or Wikipedia it). When I stumbled upon econometrics (and things like randomized control trials) I was at a point where I was frustrated with the non-profit world. I was so frustrated with the “everyone wants to do good, but no body knows what they are doing” situation. Measurement techniques seemed like the answer. Since then, I have realized that measurement has its limitations. A few thoughts:
- Some things simply cannot be measured (often the important things). The most important things like happiness, personal growth, hope, paradigm and behavioral change, etc. are extremely hard, if not impossible, to measure.
- Even if certain things can be measured, conclusions are often difficult to draw. Education is a great example. We know some things about education. For example, it seems that in the U.S. an extra year of schooling leads to approximately a 10% return in income (averaged across years and with several other caveats). We don’t, however, know much else. Especially when it comes to education reform (teacher effectiveness/compensation, class room size, charter schools, voucher programs, home schooling, etc.) We can measure a lot of things (test scores, graduation rates, etc.) but the main take away from my Economics of Education class is that “we don’t really know”. There are mixed results for almost everything.
- Often we measure the wrong thing. After a few years of being in the “invisible hand” camp of economic theory, I am beginning to question a few things. I still think there are good points, but I feel it is extremely damaging to always view increased efficiency as a good thing. Especially because economists (and the world) often measures efficiency, success, development, happiness, poverty, etc. etc. in dollars. We measure GDP. We measure income. We measure consumption. What if none of these things really make people happy?
- Quantitative research must be combined with qualitative research. Why can’t economists and sociologists just get along? We really need a combination of the multiple perspectives to truly understand what is going on. Economists can’t do it themselves, and neither can sociologists (or psychologists, anthropologists, and all the other -ists in the world). We need a much more holistic approach of what and how we measure.
Don’t get me wrong. I still have lots of faith in measurement and believe it to be a key to success no matter how we define it. I have just become acutely aware of its limitations over the past year. We need to understand these limitations in order to unlock the true potential of measurement.
That was a really long post. Soap box topic. Congratulations if you read the whole thing.