What is the biggest weakness of the use of laboratory experiments in social research?

Abstract

Laboratory experiments in sociology have become increasingly oriented to testing, refining, and extending theories and testing their applications. The increasing number of theoretically oriented experiments has led to an increasing number of experimentally oriented theoretical research programs and, with them, to an increasing number of standardized experimental settings capable of comparing and contrasting conditions between experiments with the same confidence as within experiments. The increase in the number of theoretical research programs has led to more theory growth; to growth, of both theory and research, that is more cumulative; and to a greater impact of experiments on the application of theory. The chapter ends with a brief discussion of the challenge of generalizing from experiments.

Abstract

Laboratory experiments in social science developed most rapidly in the years since the end of World War II, fostered by the growth of technology for observation and recording. Experiments offer powerful advantages for testing predictions, although their advantages and proper uses, and relations to other kinds of research designs, still are not well understood by many social scientists. Experiments are most useful when investigating predictions derived from explicit theories, and it is theories, rather than experimental results, that are properly used to explain features of natural settings. Theoretical foundations, concrete and theoretical concepts, abstract design, operations, and interpretation of outcomes all are parts of experimental research programs. Although not every ­research question lends itself well to experimental research, when questions are formulated abstractly, the range of experimental usefulness is much broader than many people appreciate.

Conclusions

Laboratory experiments in social science require creating artificial settings and asking participants to engage in specific kinds of behaviors that are recorded. In a well-designed experiment, all operations are concrete manifestations of abstract factors in a theory. Experimental research should be readily interpretable by moving from concrete measures produced in a laboratory to abstract theoretical dependent variables and that information should be used to assess the derivations and assist in developing theories.

In addition to thoughtful design, it is important to include provision for monitoring the quality of an experiment. Useful techniques include pretesting and measures during the conduct of an experiment and in a postsession interview. The more aspects of a design that can be assessed in these ways, the stronger the confidence an investigator may place in conclusions drawn from this type of research.

In addition to scientifically adequate design, social science experiments must incorporate ethical concerns for the welfare of participants, during the experiment itself and afterward. Responsibility for ethical concerns properly rests with the investigators; relying on participants to know their limits and to understand the dangers through varieties of informed consent generally abdicates that responsibility and places it on people lacking the experience they would need to protect themselves.

Single measures and single experiments can mislead because of measurement error. Multiple measures should always be used for important variables and replication is essential for important theoretical questions. Programmatic experiments generally result in more reliable information than one-shot studies. They always partially replicate past work (in operations and measures, most typically), and through differences in setting and populations studied, they permit assessing different aspects of the design.

Laboratory experiments are complicated, and effective design presents many challenges. In return for the effort involved, they often return reliable data permitting strong inferences that are useful for theory building.

Laboratory Assessments of Aggression

Laboratory experiments provide the strongest evidence that a particular factor may play a causal role in aggression. This is because that factor can be manipulated whilst all other factors are (in theory) held constant (e.g., all participants may have an identical experience in the laboratory except for watching a violent or a nonviolent movie clip). Aggression experiments typically measure short-term increases in mild forms of aggression or in known precursors such as aggressive thoughts and feelings. For example, researchers might measure whether aggression-related thoughts are more activated in one group of participants compared with another by testing reaction times to identify aggression-related (hit, blood) versus neutral (sew, rose) words. Aggressive feelings are typically measured by having participants rate the degree to which they feel emotions such as anger, antagonism, and unfriendliness.

Measuring aggressive behavior itself has a long history involving ethical, reliability, and validity concerns. For ethical reasons, serious harm cannot be used as an aggression measure in laboratory experiments. However, numerous valid and reliable aggression measures have been developed, usually involving a contrived laboratory situation that allows participants to behave in a way that they believe will harm another, but in which no person is actually hurt. Early measures included counting the number of aggressive acts a child would make toward a target, and the willingness of an adult to deliver a (fake) electric shock to another person purportedly being tested for their ability to memorize stimuli under conditions where they would be ‘punished’ for mistakes. More recent methods include measuring the duration and/or loudness of aversive ‘noise blasts’ delivered to an opponent in a competitive reaction time (CRT) game, the amount of hot chili sauce assigned for eating by a stranger known to dislike hot foods, and the number of difficult puzzles that require solving by another person in order to win a reward. Although such measures have been criticized for being unlike ‘real-world’ situations and subject to biases such as the desire to please (or displease) the experimenter, well-designed modern experiments overcome such problems using careful cover stories and scripts, and have been shown to predict real-world aggression.

I Introduction

Laboratory experiments in the social sciences are concentrated in four disciplines: psychology, sociology, political science, and economics. We focus on the recruitment of participants in the first three. (Davis and Holt (1993) and Kagel and Roth (1995) provide extensive details of experimental methods in economics.) Experiments using human participants represent a relatively small proportion of the research in each of these disciplines. For example, much of psychology is not social, focusing instead on individual differences unrelated to social interaction or on the social behavior of nonhuman animals. The subdiscipline of social psychology is primarily experimental in psychology but spills across the disciplinary boundary with sociology. In sociological social psychology, experiments have had a place for several decades, but in uneasy coexistence with less intrusive observational methods. In sociology, experiments are concentrated in a subdiscipline known as “group processes.” Compared to psychology and sociology, laboratory experiments in political science have only recently become influential in the direction of research in the field. In all of these disciplines, however, the influence of experimental research on the field has been disproportionately large relative to its volume.

Szmatka and Lovaglia (1996) explain the peculiar “convincingness” of the results of laboratory experiments compared to results from other research methods as a combination of reproducibility, incremental adjustment of research design to counter criticism, and transparency. A critic can be invited into the laboratory to see for him- or herself with relatively little investment of time and resources. (See also Lovaglia (2003) on the power of experiments.)

Methodological issues are often couched in theoretical and even moral terms as practitioners of a particular technique seek to carve out a resource-rich niche for their work (Szmatka & Lovaglia, 1996). The treatment of participants in experiments is no exception. Whether participants ought to be volunteers, paid for their efforts, or required to participate to complete a class or degree has been a hotly contested issue that sociological experimenters have used to distinguish their research from that of psychologists. Sociological researchers have argued for greater care in avoiding coercion to participate, while psychological researchers have maintained that participation in experiments is a vital part of undergraduate education and thus should be required.

Within sociology, the debate over the ethics of deceiving participants has polarized experimental and nonexperimental social psychologists. Should “informed consent” require that all the relevant issues involved in the research be explained to participants before they agree to continue with it? On the one hand, experimenters using deception note that few important topics could be effectively researched in the lab without deception. For example, explaining in advance that the study investigates racist tendencies of participants would certainly alter their behavior during the study, thereby masking any relevant effect. On the other hand, those decrying deception maintain that the damage to the reputation of the discipline caused by deceiving participants far outweighs any contribution to knowledge that such techniques produce. Experimenters counter that participants should expect the reality they encounter in an experiment to match the reality outside the lab no more than they expect the same congruence from a theatrical production. Ironically, opponents of constructing alternate realities in the lab often come from a social constructivist ideological position that also denigrates experimenters as “positivist,” often using the term in straw-man fashion to misrepresent scientific activity (for related arguments, see Turner, 2006). In practice, the issue has been resolved by institutional review boards (IRBs) that approve experimental designs. IRBs have consistently upheld the value and ethical soundness of experiments using deception when approved procedures are followed and participants thoroughly debriefed after the experiment.

However, as Hertwig and Ortmann (2008) have suggested, upholding the rights of research participants and ensuring their welfare is arguably not the only ethical matter at stake when it comes to deception. The spirit of their argument is “Fool me once, won’t be fooled again.” A participant once deceived in a study (or who knows someone once deceived) is likely to expect to be deceived in other studies, which undermines the foundation of experimental research. Although the potential for this problem has led to a general prohibition against the use of deception in economics (Hertwig & Ortmann, 2008), a less extreme alternative is to point out that researchers have an ethical obligation to take seriously how the use of deception may impact not only participant welfare but also the integrity of a study and its findings.

Although theoretical and moral debates about participants are interesting and important, the main purpose of this chapter is to describe the ways that participants are recruited for laboratory experiments in enough detail to allow a researcher setting up a laboratory to efficiently recruit participants.

13.2.1 Laboratory Experiments

Laboratory experiments, as the name implies, are those which occur within the confines of a designated experimental or laboratory space—quite often a bespoke computer room. These laboratory experiments were initially adopted to test economic theory for the same reasons’ experiments are used in the physical sciences: the ability to control confounding factors and manipulate single variables to determine their effect. Because these environments are highly controllable and allowed the researcher the ability to isolate every variable and manipulate the one of interest, the laboratory became the workhorse for testing behavioral economic theories (List & Cherry, 2008), as its design construct limits that allows the scientist to engage with a small piece of the world (Friedman & Cassar, 2004, p. 12). It allows to closely monitor the environment and the institutional conditions through initial endowments, monetary rewards, language used, and defined rules (Smith, 1994). Negative shocks in a lab experiment in the area of disaster research can be modeled via increasing the show-up fees (initial endowment)2 so that individuals do not end with a financial loss which would otherwise be problem from an ethics perspective. However, lab experiments have problems in exploring emotional and psychological aspects related to disasters. In addition, using experimental methods in this way has allowed researchers to test theoretical models or generate (substitute) data that is normally unavailable from the field, which is often the case for disasters (Abbink, 2012). The experimental method particularly allows exploration of cognitive aspects and discrimination between potential theories or understands the causes of a theory failure. This allows to compare between environments, as well as between formal and information institutions, and therefore use the laboratory as a testing ground for institutional, environmental, and policy design strategies (Smith, 1994). Laboratory experiments can help to understand the implications of extreme parameters which are important for disaster research (e.g., extreme environmental conditions). Furthermore, the dynamics in an experiment can identify whether some properties (e.g., social norms) may begin to break down.

However, unlike the physical sciences, our particles (aka humans) are heterogeneous in nature and can alter their choices or behaviors dependent upon many (sometimes unobservable) factors. This is very different from the physical sciences where the aggregate laws of nature are more absolute, such that an identical force applied to the same object will always react in the same manner, that is, they will always behave in the same way when exposed to the same stimuli. Therefore it is important to replicate experiments to check the robustness of the results, although it should be noted that few replications in social science are able to exactly replicate former conditions due to many contextual aspects that cannot be controlled for (e.g., different subject pool). However, external validity can be enhanced if the experiment simulates cognitive processes that are relevant in the real world. Laboratory experiments focused on understanding individual behavior in disaster research would also benefit from enlisting a broader population set; that is, recruiting beyond students.3 Researchers who apply experiments in the area of disaster research will also need to be more inventive in the way they design their experiments. They cannot rely on common and widely used experiments. This induces the Duhem–Quine problem that each test of a theory is a joint test of the theory tested and the way the experiment is conducted. As the design needs to be subjected to a test, it may attract scholars who are less risk averse. Smith (1994) points out that a “theory always swims in the rough water of anomaly. You don’t abandon a theory because of a (or many) falsifying observation(s)” (p. 129). He refers to Einstein who emphasized that “Only after a diverse body of observations becomes available will it be possible to decide with confidence whether systematic deviations are due to a not yet recognized source of error or to the circumstances that the foundations of the theory [of relativity] do not correspond to the facts” (Einstein in Smith, 1994, p. 129). Thus Smith concludes that the procedures under which a theory is tested are also required to be part of theory itself, a procedural aspect that researchers have strongly neglected so far.

Another issue with conducting experiments on humans is that they are self-aware and will be cognizant that in laboratory settings their actions are being observed, which may result in them deviating from their normal behavior or preferences. This is known as the Hawthorne4 effect and it difficult to generalize results because the actions or choices made in the lab may not be an accurate or representative reflection of that individual’s normal behavior or true preferences (Levitt & List, 2009). However, this instability of action/choice makes the testing of causal and treatment effects more complicated; therefore, economists have addressed this through the implementation of a range of instrumental variables and other econometric techniques. While one could argue that laboratory experiments are of limited use due to the conditions under which the experiments occur, that is, nonnatural and sterile environments which may evoke nonnatural responses from participants, it must be noted that the absolute inability of experimenters to access real disasters has made the laboratory environment very attractive. Furthermore, while it may not be possible to replicate or even approximate the levels of stress and danger to provide sufficient threat of death in test subjects it does allow for a highly controllable environment in which to explore low cost decisions. As such, it remains somewhat unclear how much of the insights gained in the laboratory can be extrapolated to the world beyond it (i.e., a lack of generalizability). When looking at average effects one also needs to be aware of further selection problems (e.g., those who select themselves into experiments may have a higher willingness to seek social approval or a higher risk tolerance to accept environmental or institutional changes). The relative differences in a between-treatment design between the control and treatment group may still hold, but care must be exercised in interpreting the overall average effects.

Criterion 4: Experimental Evidence – Laboratory Research

Laboratory experiments cannot fully settle the question of correlation and causality. Nevertheless, if controlled exposure to the thin beauty ideal and associated features (e.g., sexual objectification) makes girls and women feel immediately worse about their bodies, this would suggest that media exposure could cause body image to become more negative over the long run.

Meta-analyses and other reviews permit three general conclusions. First, compared to participants randomly assigned to control conditions, girls and women exposed to a set of images of the thin beauty ideal featured in the media experience a moderately large increase in body dissatisfaction. This is called a ‘contrast effect’ because increased dissatisfaction reflects a disparity between perception and evaluation of the self in contrast to perception and evaluation of the standard (the ideal). This contrast effect has been produced by content from magazines, television, and video games, as well as pro-ana Web sites. Experiments conducted in Great Britain demonstrated that in girls the contrast effect is produced by the thinness of fashion models, not their attractiveness.

Second, there is some compelling evidence that girls and women who demonstrate higher ‘state’ body dissatisfaction immediately after seeing the ads tend to have the highest levels of ‘trait’ body dissatisfaction and of desire for thinness 2 years later. Third, there is substantial variability in the effects generated by experimental presentations of thin-ideal media. In fact, in one Australian study of girls in the 7th or 10th grade, approximately 25% of those viewing magazine images of slender attractive models exhibited an increase in state body satisfaction. This is called an assimilation effect because evaluation of the self is pulled toward the standard represented by the ideal. Assimilation effects were also found in two studies of Canadian college students. Restrained eaters showed moderate to large increases in body satisfaction following exposure to the thin ideal in magazine images, whereas unrestrained eaters demonstrated very large contrast effects. Assimilation, rather than contrast, is clearly a real possibility, but the contrast effects produced in a very large number of studies fulfill Criterion 4.

A Energy Intake

Laboratory experiments in animals and human clinical studies have repeatedly shown that the level of fat and energy intake in the diet is strongly and positively related to excess body weight. Examination of secular trends in self-reported overall energy intake, eating frequency, and energy density of diets suggest that trends of food consumption roughly parallel the pattern of obesity observed in the United States over the past 40 years [72–74]. Examination of trends in energy intake from nine consecutive NHANES show that energy intake increased beginning in 1976–80, and peaked in 2003–04 for most groups (1946 kcal/day to 2269 kcal/day). Among men, average daily energy intake increased from 2401 kcal/day in 1976–80 to a peak of 2701 kcal/day in 2005–06 (p<0.01). Among women, average daily energy intake peaked in 2003–04, increasing from 1521 kcal/day in 1976–80 to 1885 kcal/day in 2003–04 (p<0.01) [72]. Declines in energy intake were observed between the 2003–04 and 2009–10 surveys, with average daily energy intake decreasing from 2269 kcal/day to 2195 kcal/day [72]. The decrease in energy intake seen in U.S. adults roughly aligns with stabilizing obesity prevalence among adults between 2003 and 2012 [8]; it is possible that these apparent declines in energy intake contribute to the recent leveling off of obesity rates.

Secular trends in energy intake of children and youth have also been examined [75–80]. Data from NHANES found that mean energy intake remained relatively stable from the 1971–74 to 1999–2000 surveys, except for an increase among adolescent girls of 80 kcal/day [76]. Similarly, examination of data from NHANES 1988–94 through 2003–08 showed no increase in total energy intake over time among children aged 2–19 years [78]. In contrast to analyses that use NHANES baseline data, comparisons of the 1977–78 Nationwide Food Consumption Survey and 1989–91 Continuing Survey of Food Intakes by Individuals with the 2003–06 and the 2005–10 NHANES surveys show increases in daily energy intake of 180 kcal/day [80] and 108 kcal/day [79] for children aged 2–18 years, respectively. The biggest increase is seen among preschool aged children (aged 2–6 years), whose daily energy intake increased from 1433 kcal/day in 1989–91 to 1664 kcal/day in 2003–06 [75]. Recently, Mendez and colleagues [77] compared energy intake across the 2003–10 NHANES. Results suggest differences in secular trends in energy intake by child age. Daily energy intake decreased between 2003–04 and 2009–10 among children aged 2–5 years (decrease of 220 kcal/day for boys and 132 kcal/day for girls) and 6–11 years (decrease of 230 kcal/day for boys and 142 kcal/day for girls). In contrast, for adolescents aged 12–18 years daily energy intake decreased between the 2003–04 and 2007–08 surveys, but increased between the 2007–08 and 2009–10 surveys (increase of 60 kcal/day for boys and 97 kcal/day for girls).

Changes and patterns in secular-trend surveys should be interpreted cautiously due to a number of limitations, including weaknesses in study design, methodological flaws, different survey methodologies, and random or systematic measurement error in the dietary data [81,82]. For example, the procedural changes between NHANES II and III in dietary survey methodologies, survey food coding, and nutrient composition databases have made comparisons between the two surveys difficult. These limitations could explain inconsistent results in secular-trend surveys.

The Nature of Allozyme and DNA Diversities

Critical laboratory experiments on marine model organisms have shown that allozymic diversity is adaptive and responds rapidly to environmental inorganic and organic pollution stresses (Nevo, 1998); a large amount of allozymic diversity can be gained or lost rapidly. Our experiments demonstrated unequivocally that ecological stresses of thermal and chemical pollution affect the level and pattern of allozyme allele frequencies and heterozygosity of marine organisms, thereby corroborating the environmental theory of genetic diversity. The survivorship values of heterozygotes and homozygote allozyme genotypes vary in accordance with the pollutant concentration. The broader the ecological niche, the higher the heterozygosity. Furthermore, heterozygote sites mutate more frequently than equivalent homozygous sites, possibly because mismatch repair between homologous chromosomes during meiosis provides extra opportunities to mutate. At medium levels of pollution, heterozygotes seem to be superior, whereas at high pollution levels specific homozygotes become superior. Our laboratory results on mercury pollution were confirmed in the sea. Remarkably, Schulman and colleagues described dynamic and rapid changes in genome size mediated by an increase or decrease in copy number of the retrotransposon BARE-1 in wild barley, H. spontaneum, both regionally and locally in Israel were described in 1999 by Schulman and colleagues.

3.4 Experiment setup

The laboratory experiment is divided into four sessions as shown below:

Information session

Questionnaire

Learning session

Introduction to VR

Familiarization with the driving network

Actual experiment

3 experiment per user

2 scenarios per experiment

De-briefing

Feedback from the participants

3.4.3 Actual experiment session

The actual experiment is conducted in the VIRE driving simulator. Every participant is asked to try 3 experiments each consisting of 2 scenarios which will be discussed in more details in the next paragraph. At the end of each experiment participants are asked to select which scenario they preferred the most, driving themselves or being driven.

3.4.3.1 Experiment scenarios

As mentioned earlier, each participant has to go through 3 experiments with 2 scenarios. The scenarios are as follows: HDV and E2ECAV. In the case of both HDV and E2ECAV there are two possible road networks, the familiar network where participant does the learning session and the unfamiliar network different than the learning session network. Further there are two different traffic conditions, low and high. In the case of E2ECAV the users also have the option of multi-tasking and non-multi-tasking. In total there are 8 experiments (two scenarios each) and out of these 8, we randomly assign 3 experiments to each participant in such a way that all experiments are repeated equal number of times. Fig. 27.3 presents the breakdown of possible scenarios for HDV and E2ECAV.

Fig. 27.3. HDV and E2ECAV scenarios.

Before the start of each experiment, participants are assigned an origin-destination pair (in total there are two O-D pairs one for familiar network and one for unfamiliar network). Depending on a scenario they either have to drive themselves or be driven by a CAV. In the case of the HDV scenario they are also given a static map as shown in Fig. 27.4A and B. At the end of each scenario, their travel time is shown on the screen in order for them to compare the two alternatives. The two networks used in this study are both part of downtown Toronto network, which was also used by Farooq and Djavadian (2019) for their case study. The reason for choosing the same network is to utilize the E2ECAV data collected from their agent-based simulation to model the movement of E2ECAV in VIRE simulator.

Fig. 27.4. Test networks. (A) Familiar network (from learning session) and (B) Unfamiliar network.

The two OD pairs used are: intersection of King St. & Simcoe to intersection of Queen St. W. & Bathurst St., and intersection of Wellington St. & University Ave. to Queen St. E. & Jarvis St.