Questioning randomized evaluations

Randomized evaluations seem to be all the rage for figuring out whether to charge for bednets or the value of consumer credit. But some economists have started to ask questions. Chris Blattman raises the question of opportunity cost:

Randomized trials are expensive and time-consuming, and it's worth specifying under what circumstances (if any) they ought to be absorb the time and energy of the world's best and brightest development economists.

Meanwhile, Evelyn Stark over at CGAP has similar concerns. While randomized experiments may provide useful information, they can be expensive and time-consuming: "Expense - cited as equal to the cost of the microfinance intervention being tested." In some circumstances, simple monitoring and evaluation without randomization might be enough, even if it leaves us with less certainty about the outcomes. Perhaps it's time to develop more formal cost-benefit guidelines for randomized evaluations?   

Update: Owen makes a valuable point in response to this posting:

Our problem is that we are investing too little in rigorous evaluations, not too much.  We’ve been giving aid for 50 years now with pitifully little evidence about what really works. 

Of course it is true in principle that we could invest too much in evaluation - but the point of diminishing returns is a long way above where we are now. The sums of money involved are trivial by comparison with the huge amounts of aid we are spending on the basis of far too little information.

I couldn't agree more! See, for instance, some skepticism on OLPC in the absence of proper evaluation. In aggregate, we almost certainly invest too little in evaluation, and randomized evaluation is the most promising type of evaluation that we have available. However, let's suppose that many more resources are devoted to evaluation - we still need to make a judgement call about how best to spend that money, and a cost-benefit analysis would help.

Let's take a specific example. The Inter-American Development Bank is currently carrying out an evaluation of OLPC in Haiti. Price tag? More than $5 million. Let's assume for a second that the evaluation comes up roses. What then? Can we assume that the OLPC would have a similar impact in Peru? Or Jamaica? How about Azerbaijan? Or should we carry out a randomized evaluation in each of these countries? We may not be at the point where we need to answer that question, but it's worth thinking about it now.