1014 Journal of Economic Literature,Vol.XLII(December 2004 impr nnd that employs a standard subject pool of extent of discrimination in the sports-card marketplace. set of rules; an artefactual field experiment is the 3.Methodological Imp ortance of Field same as a conventional lab experime Experiment but with a nonstandard subject pool:1 a framed field experiment is the same as an artefactual field experiment but with Fieldpecause they mechanicologically ecause they mechanically f field context in either the co pay attention to issues tha great researcher seem task,or information set that the subiects to intuitively address.These issues cannot be comfortably can use. forgotten in the field,but they are of more ·a general importance. The goal of any evaluation method for "treatment effects"is to construct the prop- these task nere the subjects dokow er counterfactual,and economists have that they are in an experiment. rnative methods We recognize that any such taxonom of nstructing the counterfactual:co leaves gaps,and that certain studies may not fall neatly into our classification scheme trolled experir ments,natural (PSM) Moreover,it is often appropriate to con- nstr duct several types of experiments in order to nd t 3 identify the issue of interest.For example approache with treatme e y as the th ment.and le me w tho Harrison and List(2003)conducted artefac tual field experiments and framed field treated and eriments with the same subject pool.pr The treatment cisely to identify how well the he effect for unit i can then be measured as istics might naturally t=y-yo.The major problem,however,is the latte one of a missing counterfactual:t.is unknown It we could observe the outcome for an untreated observation had it been conducted ctual,fr and natur treated,then there is no evaluation problem experiments to investigate the nature and Controlled" experiments.which include laboratory experiments and field experi ments esent the most onvincing method of creating the counterfa al,sinc side trol group ents the popu field with We simplify by c the lo of peter Boh ontin of st albeit wit ics.one the Vicl might have some measure of risk ave ne pre run in which the ntrol cking-Reiley (2002
Harrison and List:Field Experiments 1015 average treatment effect is given by individuals with the same value for these fac- y*o where y*and y*are the treat- tors will display homogenous responses to ed and nontreated Iaverage outcomes afte the treatment,then the treatment effect can the treatment.We have much more to be measured without bias.In effect,one about controlled exp eriments,in particulan use statistical methods to identify which field experime ts.bel individuals are s lab rat Na experiments"the treat- for th eatme ent and find a na fo rmally.th tha th compar Z)∈(0，l,where terenc outcome lenotes independence after rfor the treate group with the before Another alterative to the DID model is and after outc nes for the nontreated group. the use of instrumental variables(IV),which Estimation of the treatment effect takes the approaches the structural econometric form Y=XB+tT.+n.where i indexes the method in the sense that it relies on exclusion unit of observation.t indexes vears.y.is the restrictions (Joshua D.Angrist,Guido W outcome in cross-section i at time t,X is a Imbens and Donald B Bubin 1996.and vector of controls,T.is a binary variable Joshua D.Angrist and Alan B.Krueger 2001) n=a++E.and t is the difference-in-dif The IV method,which essentially assume ferences(DID)ave age treatment effect.If that some components of the no we assume that data exists for ntal data are he then t=(y* utilized for ome 6 g 2000).Th to find variab of theg nd Kenn ed from proa the e outcome equation ut whi -specific sh ch is relate to unit to ent status and has no direct association are correlated with treatment status with the outcome.The weakness of the IV and that selection into treatment is inde approach is that such variables do not often pendent of temporary individual-specific exist.or that unpalatable assumptions must effect:E(maXwD)=E(aIX,D)+入，If be maintained in order for them to be used to E,and t are related,DID is inconsistently identify the treatment effect of interest. estimated as E(t)=t+E(Er-Eo D=1) A final alternative to the DID model is -E(e,-￡oD=0) structural modeling.Such models often entail One alte native method of assessing the a heavy mix of identifying restrictions (e.g. impact of the treatment is the method of ensity and dor 1(1983) ed in eaker ethod has be nsively the debat nd ence nt effec uld have been had the that sele 100g and Te that the di Smith and Petra Todd(2000). en matchi goa dbymay limit the use PSM is to make no perimental data baum and（a9s3, 9s4)sh d that matchn like" experimental data.The intuition to g behind PSM is that if the researcher can treated p(Z). the select observable factors so that any two
1016 Journal of Economic Literature,Vol.XLII (December 2004 separability),impose structure on technology could be applied to real people,but to actu- and preferences (e.g.,constant returns to ally do so entails some serious and often scale or unitary income elasticities).and sim unattractive logistical problems. plifying assumptions about equilibrium out- A more substantial response to this criti- comes (e.g.,zero-profit conditions defining cism is to consider what it is about students equilibrium industrial structure).Perhaps the that is viewed.a priori.as being nonrepre best-known class of such structural models is sentative of the target population.There are putable general equilibrium models. at least two issues here.The first is whether which have been extensively a endogenous sample selection or attrition has ate trade policies,for example ncourred due to incom lete control ove relies on c estimation strategi so that the eters that mple is sta simulatio sen e.g. onsistent sitivity analys trea al mode n this s se ether can be inform een the c ative on the behav or the population, experimenta tax and assuming away sample selection issues. policies (R.Blun ll and Thomas MaCurdy 4.2 Sample Selection in the Field 1999:and Blundell and M.Costas Dias 2002) Conventional lab experiments typically 4.Artefactual Field Experiments use students who are recruited after being told only general statements about the 4.1 The Nature of the Subject Pool experiment.By and large.recruitment pro A common criticism of the relevance of cedures avoid mentioning the nature of the inferences drawn from laboratory experi- task,or the expected earnings.Most lab ments is that one needs to undertake an xperiments are also one-shot.in the experiment with"real"people,not students that they do ot involve ated obse This criticism is often deflected by experi of a mpl ect to menters with the following imperative:if you eith ed to of the think that the experiment will generate differ If one w with spec ent results with"real"p ople,then go ahead rest i a task ould be o do and run the expe le.A (e.g.,Peter m and Han ind 1993).An ant of this is to challer the if one wanted to recruit subjects for several ics'asse tudents ses super-expenenced or to conduct pre-tests of such trust.or other an things as risk aversion st response,to suggest regarding preferences that could be built the experiment with real people into the design as well often adequate to get rid of unwanted refer One concern with lab experiments con ees at academic journals.In practice.howev ducted with convenience san oles of students er,few experimenters ever examine fielo behavior in a serious and large-sample way It is relatively easy to say that the experiment onard reen (19)for dramati theory to organ For example.the Pl king and David Tarr (197 or exampe John Kagel and Dan Levin( Harrison and H.D.Vinod (1992)
Harrison and list:Field Experiments 1017 is that students might be self-selected in allows one to remove this recruitment bias some way,so that they are a sample that from the resulting inference excludes certain individuals with cha racteris Some field ex eriments face a more seri tics that are nts of sa selection tha unde dep problem on the of the task.Once th ev ent has be it is not a importanc he el inf n flo overemphas lways possible to sim th of the .Th matte degree can ead to endoge pop lation are not represented,at nous subject attrition from the experiment least under the tentative assumptio that it is uch attrition is actually informat ive about only observables that matter.In this case it subject preferences,since the subject's exit would behoove the researcher to augment from the experiment indicates that the sub the initial convenience sample with a quota ject had made a negative evaluation of it sample,in which the missing strata were sur (Tomas Philipson and Larry Hedges 1998) veved Thus one tends not to see many con The classic problem of sample selection victed mass murderers or brain surge ons in refers to possible recruitment biases.such that the c where to natur of the ther n in our sa rm2naeraton ent.Th fo Another proble, of any exper pe the pos is th recruitme typ treatmen dures his issue is popu whi s ar es the role o ruited has c vers and plau recruitment fees in biasing the sample sibly expects s the experiment to ave some subjects that are obtained Ihe context for element rand then the her experiment is particularly relevant here observed sample will tend to look less risk- since it entails the elicitation of values for a averse than the population. It is easy to private commoditv She finds that there are imagine how this could then affect behavior some significant biases in the strata of the differentially in some treatments lames population recruited as one varies the Heckman and leffrey Smith (1995)discuss recruitment fee from zero dollars to two dol- this issue in the context of social experi- lars,and then up to ten dollars.An im ortant me nts,but the concern applies equally to finding how r is that field and lab experiments. a s del 4.3 Are Students Different? them. ects and th This wortditdOGo addre ssed in In a group subjects e tre ent ichte stein and Pa ment has 60 cent fem les and the ot (1973) and Penny Burns (1985 sample of subjects in another trea ment has lenn Harrison and James Lesley (1996 only 40 percent females,provided one con (HL)approach this question with a simple trols for the difference in gender when pool statistical framework.Indeed,they do not ing the data and examining the key treatment consider the issue in terms of the relevance effect.This is a situation in which gende might influence the response or the effect of oercholegoaemtche 2 the treatment,but controlling for gender
1018 Journal of Economic Literature,Vol.XLII (December 2004 of experimental methods,but rather in the bject was asked whether he or she terms of t the relevance of convenience sam would be willing to pay $X towards a public ples for the contingent valuation method good,where $X was randomly selected to However,it is easy to see that their methods be $10,$30,$60,or $120.A subject would apply much more generally. respond to this question with a "yes,"a The HL approach may be explained in “no,”ora"not sure.”A simple statistical terms of their attempt to mimic the results model is developed to explain behavior as a of a large-scale national survey conducted for the Exxon Valdez oil-spill litigation.A hanerc observable socioeconomic major national survey was undertaken in this case by Richard Car on et al.(1992)for the oped.HL the ceeded to the eral of the state of Alaska Thi ed ther stat -of-th e sta mor y for preve on this is the HL asked apply to t can obtain is simply mai the sam resu ts us then the statistical mode may be ience sample of stud used to predict the behavior of the target of South Carolina.Using students as a con population if one can obtain information venience sample is largely a matter about the socioeconomic characteristics of methodological bravado.One could readily the target population. obtain convenience samples in other ways The essential idea of the hl method is but using students provides a tough test of hTipnoe oroceeded by developing a sim sentative in the sense of allowing the survey instrument than the one used in the rcher to develon a d"statistical of this is pu model of the behavi or under the of the a not essential to the method.Th at se pi available of the survey wa to a relative the beha of student with stude ta tha rvey th a nts"is the to contro t attributes,is the col variability in their socio-demographic char lection of a range of tandard socioeconom acteristics,not necessarily the unrepresen- ic characteristics of the individual (e.g.,sex tativeness of their behavioral responses age,income,parental income,household conditional on their socio-demographic size,and marital status).Once these data characteristics. are collated,a statistical model is developed To the extent that student samples exhibit in order to explain the key responses in the limited variability in some key characteris- survey.In this case the key re nonse is a tics.such as age.then one might be wary of the veracity of the maintained assumption ce valuation question. In other words volved here.Howey r the sample do e to look like the order for the statistical n odel to be an adequate e one 4 Th xact form of that statistical odel is not impe (9)for et for of this method