Journal of Economic Literature Vol. XLII (December 2004) pp. 1009-1055 Field Experiments GLENN W. HARRISoN and JOHN A. LIST 1. Introduction experimental environment. We do not see the notion of a "sterile environment" as a Iin e e of an pet negative, provided one recognizes its role in Every researcher who behaves as if an exoge- the research discovery process. In one nous variable varies independently of an error sense, that sterility allows us to see in crisp term effectively views their data as coming relief the effects of exogenous treatments on from an experiment. In some cases this belief behavior. However, lab experiments in isola- tion are necessarily limited in relevance for is a matter of a priori judgement; in some cases it is based on auxiliary evidence and predicting field behavior, unless one wants inference; and in some cases it is built into the to insist a priori that those aspects of eco- design of the data collection process. But the nomic behavior under study are perfectly distinction is not always as bright and clear general in a sense that we will explain. Testing that assumption is a recurring difficul- Rather, we see the beauty of lab experi- ty for applied econometricians, and the search ments within a broader context-when they always continues for variables that might bet- are combined with field data, they permit ter qualify as truly exogenous to the process sharper and more convincing inference. under study. Similarly, the growing popularity In search of greater relevance, experi- of explicit experimental methods arises in mental economists are recruiting subjects in large part from the potential for constructing the field rather than in the classroom, using the proper counterfactual. field goods rather than induced valuations, Field experiments provide a meeting ground and using field context rather than abstract between these two broad approaches to empirical economic science. By examining the 2 When we talk about combining lab and field data, we nature of field experiments, we seek to make it do not just mean a summation of conclusions. Instead, we have in mind the two complementing each other in some a common ground between researchers. functional way, much as one might conduct several lab We approach field experiments from the experiments in order to tease apart potential confounds perspective of the sterility of the laboratory For example, James Cox(2004) demonstrates nicely how 'trust" and"reciprocity" are often confounded with "other egarding preferences," and can be better identified sep- Harrison: Department of Economics, College of arately if one undertakes several types of experiments Business Administration, University of Central Floridag with the same population. Similarly. Alvin Roth and List: Department of Agricultural and Resource Economics Michael Malouf(1979) demonstrate how the use of dollar and Department of Economics, University of Maryland payoffs can confound tests of cooperative game theory and NBER. We are grateful to Stephen Burks, Colin with less information of one kind (knowledge of the utili- Camerer, Jeffrey Carpenter, Shelby Gerking, R. Mark ty function of the other player), and more information of Isaac, Alan Krueger, John McMillan, Andreas Ortmann another kind (the ability to make interpersonal compar Charles Plott, David Reiley, E. Elisabet Rutstrom, isons of monetary gain), than is usually assumed in the Nathaniel Wilcox, and the referees for generous comments leading theoretical prediction. 1009
1010 Journal of Economic Literature,Vol.XLII (December 2004 terminology in instructions.3 We argue that Our second point is that many of the char there is something methodologically funda- acteristics of field experiments can be found mental behind this trend.Field experiments in varying,correlated degrees in lab experi differ from laboratory experiments in ma ments.Thus,many of the characteristics that imply mpting to view field people identify with field experiments are not only found in field experiments,and nts.w argu should not be used to differentiate them e them from lab experiments. "in lab tory exper Our third point,following from the first two is that th ere is much to learn from field t be precisely the oppo if it is arti icial to the subject or context of the returning to the lab.The task.In the en we se e field experiments as methodologically comple entary to rol in the field are traditional laboratory experiments. f the Our primary point is that dissecting the en e lab.T characteristics of field experiments help field experimen s can nelp one c gn bett define what might be better called an ideal and have a methodological exneriment in the sense that one is able to role quite apart from their complementarity observe a subject in a controlled setting but at a substantive level where the subiect does not n ceive In section 2 we offer a typology of field the ontrols as being un dth is n experiments in the literature,identifying the de otion bei key characteristics defining the species.We idea tha At first blush sub ects in setting et have eriments or more se contr ctory,t ague thgh accurately to identify different characteris. We explain this jargon n from experimental economic a bright line to define ne field and other Murphy (1973 set of hat expe egrees in a field sub ents in a e propos d to de ermine the ield context of an h the cl brea experiment:t nature of thes ect poo the nature of the information that the sub group a fr jects bring to the task,the nature of the com modity,the nature of the task or trading rules applied,the nature of the stakes,and the environment in which the subjects operate. ud not seem unnatural to th Having identified what defines a field exper oing what come iment.in section 3 we put experiments in Along s omists can identify see Ed es to in ch a the ing-Re (1) d auctions in tute field ments.I through an of rev and we four mor single-unit auction format the des broad types of field eaknesse
Harrison and List:Field Experiments 1011 literature review is necessarily selective material,language,animal,etc.,and not in although List(2004d)offers a more complete the laboratory,study,or office."This orients bibliography. us to think of the natural environment of the In sections 7 and 8 we review two types of different components of an experiment. It is impe ortant to identify what factors xperiments.One is called a social p riment so that we that it is a delib a what facto cial re di peri ma mot ivate ample when pe of some governmen progr n is implemented periment that diffe They have become popular in ce areas counterpart lab experiments o f Ronald such as employment schemes and the detec ummings,Glenn Harrison,anc Laura tion of discrimination.Their disadvantages Osborne (1995)and Cummings and Laura have been well documented.given thei Tavlor (1999).what explains the differencer political popularity,and there are several Is it the use of data from a particular marke important methodological lessons from those whose participants have selected into the debates for the design of field experiments. market instead of student subjects;the use The other is called a"natural experiment. of subjects with experience in related tasks: The idea is to recognize that som event that the use of private sports-cards as the under naturally occurs in the field ha ens to have odity instead of an environme som e of the characteristics of a field e lying co tal wd. the of streamlined me nt The an be attra ctive sc sof dat ntal thods the less-intrusive e at e cost due to mun some combin and simila cing the certai Finally,in section S we briefly exa related types of experiments of the mind.In ed.)defines the one case these are the thought exper tion ments"of theorists and statisticians.and in rgnltcgetonofohowyome the other they are the ord means "neuro-economics experiments" provided by technology. The dane things objective is simply to identify how they differ minded that d to be son 2.Defining Field Experiment ethingo y on to bes Th e are several ways to define One is asc ormal d b up ir onary.An Eve n tha identify w t it is that you want the I to differen o阳ere is the lab The Oxf ord English Dictionary (Secon Edition)defines the word"field"in the fol ory of the in publi policy.If th is son lowing manner: "Used attributively to hat rem denote an investigation,study,etc.,carried optimum,itdoe natic out in the natural environment of a given to the optimum
1012 Journal of Economic Literature,Vol.XLII (December 2004 differences?We believe field experiments Blackburn,Harrison,and have mature d to the point that some frame. Rutstrom 1994).Alternatively,the subject work for addressing such differences in a pool can be designed to represent a target systematic manner is necessary. population of the econom (e.g.,traders at 2.1 Criteria that Define Field Experiments the Chicago Board of Trade in Michael Haigh and John List 2004)or the general Running the risk of oversimplifying what oulation (e.g.,the Danish population in is inherently a multidimensional iss we rison.Morton Igel Lat. and Melonie Williams 2002). riment addition. the nature of the subject po monstandard subject p rience with the commodit ·the nat of the info mation that the the tasl exper apar the e nature o the task In the mg ce of the or trading rules to their trading activities in applied addition to their knowledge of e trading the nature of the stakes,and institution In abstract settings the impo the nature of the environment that the tance of this information is diminished.by subject operates in. design.and that can lead to behavioral We recognize at the outset that these changes.for example.absent such informa characteristics will often be correlated to tion.risk aversion can lead to subjects varying degrees.Nonetheless.they can be requiring a risk premium when bidding for used to propose a taxonomy of field experi objects with uncertain characteristics. ments that will,we believe,be valuable as The commodity itself can be an ir een lab and field experi tant part of the field.Re mental results become common ent years seen ents er can be ed as the used by valua g000 ha menters ions ove rtual g00 ie ce sa when n physical s or use s field subje ts,th abstra ly de goods latte as nonstandar in this have been the staple of xperimental eco e.But we a argu that the use of onstar nomics since Edward Chamberlin (1948) and Vernon Smith(1962),but impo ses ar ify the experiment as a field experiment.The artificiality that could be a factor influenc experiments of Cummings,Harrison,and E ing behavior.'Such influences are actually Elizabet Rutstrom(1995),for example,used of great interest,or should be.If the nature individuals recruited from churches in order of the commodity itself affects behavior in to obtain a wider range of demographic that is not accounted for by thethe characteristics than one would obtain in the has at standard college setting.The im of be st a lin domain of a a nonstandard subiect aries fron should be aw t worse ment to nt.in this ply false In on y pro et of soc cha Smith(962sgi3t mportan when dev loping statistica gh Smith (1962)does models to adjust for hypothetical bias p.121)in which monetary payoffs were employed
Harrison and List:Field Experiments 1013 understand the limitations of the generality this is an important component of the inter- of theory only via empirical testing. play between the lab and field.Early illus- Again,however,just having one field char trations of the value of this approach include acteristic,in this case a physical good,does David Grether,R.Mark Isaac,and Charles not constitute a field expe riment in any fun Plott [1981,1989],Grether and Plott [1984] damental se nse.Rutstrom (1998)sold lots and James Hong and Plott [19821. and lots of h truffles in a lab h nature of the stakes also affec ren ake in the laboratory designed to diffe th was svery much a lab expe er n the e field. we an eff tastiness commodi imilarly,Iar vior.If val Bateman et al.(1997)elicite I valuat ons ove when they are in the tens of ars,or in the pizza and dessert vouchers for a local restau ut are made in er ently when rant.While these commodities were no the price is less than one dollar,laboratory or actual pizza or dessert themselves,but field experiments with stakes below one do vouchers entitling the subiect to obtain lar could easily engender imprecise bids.Of them,they are not abstract.There are many course,people buy inexpensive goods in the r o omme field as well,but the valuation process they se might be keved to different stake levels The nature of the task that the subject is Alternatively,field experiments in relatively being asked to undertake is an in portant ortunity to eval epect that sino omtafan s within a n budget role in helping Th The vior vironme s of gel and Dan Levin illustrate th can provide super-expe heuristic a lab sett g might 0 experimenters always wondered inexperiencec terms of their whether the use of classrooms might gen propensity to fall prey to the winners' curse der role-playing behavior,and indeed this is An important question is whether the suc one of the reasons experimental economists cessful heuristics that evolve in certain field are generally suspicious of experiments settings“travel”to the other field and lab without salient monetary rewards. Even settings (Harrison and List 2003).Another with salient rewards however environmen- aspect of the task is the specific parameteri tal effects could remain.Rather than view zation that is adopted in the e then as uncontrolled effects,we see them as One can nt worthy of controlled study. m the so as study lab be or in a field 2.2 A Proposed Taxonomy is of of field n-specifi vior can always be ors tha diffe e exper convention again,Lis lab experiment (on There Is some owever,in having ive power of broad terms to differentiate wha we see a ura义wo the key differences.We propose the following terminology: a conventional lab experiment is one