Can you measure good education?
Below I've posted the first lecture for the "Research Methods in the Social Sciences" course I am teaching at VCU this semester. The first 20 minutes serve as a pretty good introduction to my general approach to social science, so I thought I would post it here as well.
In summary, the social sciences can never be as scientific as disciplines like chemistry or physics, simply because even the most basic situations in social life are more complicated than basic-level hard science situations. I do not believe that social scientists can ever develop parsimonious models that apply as broadly as "force equals mass times acceleration." Social life is more like quantum physics, except if every reasonable person accepted that a Unified Field Theory were impossible. Social science is never going to be able offer rules or formulas that describe every aspect of every situation. It just isn't.
However, this truth doesn't allow us to throw up our hands at the messiness of social life, forget measurement entirely, and just do whatever we want. What social science can do is offer analysis that can make our decisions a whole lot more informed and a whole lot less prone to disaster. If you use the right method of data collection and analysis, and if you are careful about the conclusions you draw, you can make decisions that will be better than wild guessing, gut feel, and the like.
As I allude to in the clip, I believe my attitude to be a healthy way to look at measurement in education. Let's take Value Added Methodology (VAM), which is about as controversial a measurement tool as you will find in education. I would argue that those who worry that VAM incorrectly identifies good and bad teachers are right, but only up to a point. If the only definition of success or failure is the ability to correctly identify the level of skill for 100 percent of teachers, 100 percent of the time, every evaluation instrument, no matter sophisticated, will fail. Social life is messy, and evaluating teachers is not F=ma.
As you can surely guess, I think that "100 percent/100 percent" standard is silly. A useful method of evaluation is, among other things, one that correctly sorts MOST teachers correctly. I'm a big supporter of VAM precisely because its ratings are largely consistent with those derived from rigorous classroom-based evaluations. Teachers and their advocates should insist that VAM ratings be only one of multiple measures used in teacher evaluation and question their use for teachers outside of the subject area of the test that generates the VAM score. But as long as policymakers heed these caveats, VAM is an example of social science at its best.