John List (on whose recent comment in Science I commented on here) wrote a brilliant piece for the Journal of Political Economy late last year, along with another article co-authored by Steve Levitt (of Freakonomics acclaim) and finally another (again co-authored with Levitt) in the Perspectives section of Science (Feb, 2008). I am going to do a sequence of reports on each of the articles, starting with the List & Levitt (2007) JEL piece, then on to List (2007) and ending with Levit and List (2008). I will assume that readers are aware of the differences between the dictator game, the ultimatum game, trust games and public goods games (PGGs).

Levitt and List (2007)
The basic message of the article is that we need to ensure that our interpretations of experimental results are nuanced and that we must not over-generalize, or in some cases under-generalize, our results. Specifically, we must not interpret incorrectly the results of experiments such as the ultimatum game, or the dictator game, as being truly reflective of 'social preferences' (for a definition of social preferences see my recent post on Bowles (2008)).

They provide us with 5 specific points to consider:

1. the presence of moral and ethical considerations;
2. the nature and extent of scrutiny of one’s actions by others;
3. the context in which the decision is embedded;
4. self-selection of the individuals making the decisions; and
5. the stakes of the game

In order to do this, they present a utility function that is affected by factors other than those which normally appear as arguments in utility functions. I present it briefly as a point of reference.

U_i(a, v, n, s) = M_i(a, v, n, s) + W_i(a, v)

The utility that individual i achieves is relative to the moral value (M) that they experience, which is a function of the actions they take (a), the stakes involved (v), the social norms which are pertinent when they act (n) and the degree of scrutiny (s), their wealth (W) is affected only by the actions and the stakes. For situations without moral components, "the model reverts back to a standard wealth maximisation problem" (157). In general you could expect a trade-off between morality and wealth and that as the stakes (v) of the game rise, "wealth concerns will increase in importance relative to fairness concerns, although this need not always be the case" (157). To me this last caveat is incredibly important, it could be blind naiveté, but I think that certain people would not place a price on certain (generally extreme) actions, or at least they would rather not have to do so. I would say that, in general, their construction holds, but at extremes of 'moral action' it probably won't. They note though that as stakes go up, the salience of n increases, i.e. people can forgive someone for shoplifting, but aren't so happy to forgive the Enron chiefs.

Moving, the imperative of their research:

Results from... experiments [dictator game, ultimatum game, public goods games] have been used to argue that pro-social preferences are important in a wide range of real-world settings an inference based on the assumption that the experimental findings are equally descriptive of the world at large... Rather, we are interested in the extent to which the lab provides reasonable guidance as to the importance of such behavior in a wide range of naturally-occurring settings (158).

Let me summarize the main ideas:
Unparalleled Scrutiny: you are never watched as much in real life as you are in an laboratory experimental setting. People have acted differently in the lab than they have when being observed without knowing their actions are observed. The incidence of giving decreases when scrutiny levels increase.
Anonymity in lab and field: 2 types: anonymity between experimenter and subject and anonymity between subjects. Incidence of 'giving' behaviour in a dictator game decreases when anonymity between experimenter and subject is applied ("double blind" game). Also, as scrutiny levels increase giving increases. In PGGs, as anonymity increases (using randomized application of indeterminacy of contribution) contributions decreased, conversely, as anonymity decreased, contributions increased.
Context matters (and is not always under the control of the experimenter): Experimenter lacks complete control in the experiment, subtle manipulations affect individual behaviour (choice of words such as 'partner' or 'opponent' affect contributions, similarly 'punish' or 'assign points', similarly again for the name of the game 'wall street' vs. 'community'. Contributions and behaviour are also strongly culturally dependent (Henrich, et al 2005). [Personally, I think that we can add Herrmann, et al's 2008 piece I referred to here and here as an indicator of cultural salience in experimental interactions.]
Stakes: Giving behaviour is sensitive to dramatic increases in stakes ($10 to $100), but not to smaller increases in stakes($10 to $40). Responses are also sensitive to stakes increases $1 of $5 is rejected more regularly than $100 of $500. There is conflicting evidence on the 'breakdown of trust' for high vs. low levels of stakes.
Selection into the experiment: Lab experiments possibly the science of "punctual college sophomore volunteers", or individuals interested in the research, or individuals who are naturally more cooperative and/or seek approval. Contrasted with market experiments which may select people who place a relatively higher value on wealth than morality. They contrast results from Fehr and List (2004) where actions by CEOs are compared with those of students. The CEOs were more cooperative. But you still have the problem of 'volunteers' being more cooperative and self-selecting into experiments.
Artificial restrictions: One of the problems is that "Real-world contexts typically offer the option of both giving and receiving, which may help explain in part why, contrary to the lab environment, people rarely receive anonymous envelopes with cash inside." We must also compare the dimension of generosity and the cost of it. In terms of dimension individuals may contribute time and effort (rather than money) to some volunteer porgram. With respect to cost of generosity, some individuals may avoid acting generously entirely in order to avoid the costs thereof. The aspect of time is particularly important, we cannot directly infer from short period experimental interactions that people will act similarly in a long-run field experiment, or the real world. Immediate emotional response in experiments must also come under consideration, with immediate 'hot' responses being contrasted with longer term 'cold' responses.

Generalizability & Conclusions
Moving on, they consider the reliability of generalizing experimental results to the 'real world'.
As an example: "In financial markets, for instance, the stakes are large, actors are highly anonymous, and little concern seems to exist about future analysis of one’s behavior" (168). The representativeness of the situation and the sample are both incredibly important in attempting to generalize. In small communities there are close ties and a high degree of scrutiny - cooperation will be high. In other interactions, specifically repeated interactions, it is difficult to differentiate between strategic action and prosocial preferences.

They comment further, "Any empirical estimate requires an appropriate theory for proper inference—and this lesson holds whether the data are obtained in the lab, from coin collector shows, or from government surveys" (170). In addition, combining emprics and theory is a better way to go than simply propounding the evidence against homo economicus. Also, 'nesting' experiments can help to isolate the effects of specific outcomes. Lastly, using multiple supporting inputs for greater theor is important: briding the gap between randomized field experiments and controlled laboratory experiments is crucial.

My comments
What I want to know is, can we overcome the sampling problem of experiments? I would love to be able to convert lectures of economics suddenly into on the spot experiments in order to get a more representative sample, or to use participation in an experiment as, say, a prerequisite for 'class attendance', or something equivalent, in order to get larger and more representative samples. Moreover, I would prefer it if experiments were cross-departmental. Often experiments end up sampling students from one department (econ or psych) without getting students from the humanities or sciences. Thus, samples are, once more, problematic.

In terms of scrutiny, one of the examples that Levitt & List propose was of an experiment where eyes came up on the computer screen, which they said indicated scrutiny. I don't think we can legimitately claim this unless we see the actual graphic, because of another possible interpretation which is the 'humans respond sympathetically to seeing eyes' interpretation (and I can't for the life of me recall the references, but remember this from the MIT OCW psych lectures).

Once more, I think that the adoption of 'best practice' in experiments (similar to work by Duflo and Banerjee for Randomized Evaluation) is crucial. I know that attempts have been made in the past (Roth, Camerer and others) and experiments are, predominantly, standardized with similar methods in most of the Western world. However, problems arise when you aren't operating in a normal Western environment and you are, instead, trying to do experiments in the developing world where you can't use the generally accepted computer program, because you don't have computer access, and pen and paper are the normal mode. I'll leave it at that for the moment.

References

Bowles,
S. (2008). Policies Designed for Self-Interested Citizens May Undermine
"The Moral Sentiments": Evidence from Economic Experiments. Science, 320(5883), 1605-1609. DOI: 10.1126/science.1152110


Herrmann, B., Thoni, C., Gachter, S. (2008). Antisocial Punishment Across Societies. Science, 319(5868), 1362-1367. DOI: 10.1126/science.1153808Bowles, S. (2008). Policies Designed for Self-Interested Citizens May Undermine "The Moral Sentiments": Evidence from Economic Experiments. Science, 320(5883), 1605-1609. DOI: 10.1126/science.1152110Levitt, S.D., List, J.A. (2007). What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?. Journal of Economic Perspectives, 21(2), 153-174. DOI: 10.1257/jep.21.2.153

List, J.A. (2007). On the Interpretation of Giving in Dictator Games. Journal of Political Economy, 115(3), 482-493. DOI: 10.1086/519249

Levitt, S.D., List, J.A. (2008). ECONOMICS: Homo economicus Evolves. Science, 319(5865), 909-910. DOI: 10.1126/science.1153640