Quasi-Experimental Designs, Research Paper Example

1. Jackson (2012), even-numbered chapter exercises, p 360.
2. Based on the question stimulus, I would recommend that the psychology professor use: a time series non-equivalent control group design. One could also recommend a pretest/posttest comparison; however, due to the accretive nature of the learning process, it might be more useful to see how students test over time in order to see if the change in testing is a profitable idea.
3. There are numerous confounds that might impugn the results such as attrition and a nonequivalent baseline group.
4. There are three reasons why a single case design might be selected:
5. a) The researcher wants to identify a potential hypothesis, but wants to try it out before acquire funding for a larger group of participants;
6. b) The researcher only wants information on the individual(s) selected for research.
7. c) The researcher may want to avoid unpleasant ethical problems with selecting a potential helpful treatment for one group, while giving nothing to the other group
8. A multiple baseline differs the intervention over different time frames and behaviors- thus it is able to deal with confounders through delivering it via multiple baselines. This is often times an improvement on the limitations of a single case design.

Describe the advantages and disadvantages of quasi-experiments? What is the fundamental weakness of a quasi-experimental design? Why is it a weakness? Does its weakness always matter?

The main advantages of a quasi-experiment are that it is more generalizable and easier to conduct than a strict experiment. Another way to say this is: A quasi-experimental design exchanges problem related to external validity for problems related to internal validity. Although results from an experiment may be internally valid between the independent and dependent variable, they often times fail to translate to other situations due to restrictions. Quasi-experimental results, on the other hand, may lack a robust internal validity relationship- however, they are often times transferrable to other situations.

The fundamental weakness of a quasi-experiment is the inability to infer causality from results. As the textbook explains, the quasi-experimental design sits between correlation studies and experimental studies on the causal explanatory line. That is, while a quasi-experimental study provides more causal evidence than a correlation coefficient between two variables, because it relies on observational data that lacks an experimental intervention and randomization, there may numerous confounders explaining the results between variables.

If you randomly assign participants to groups, can you assume the groups are equivalent at the beginning of the study? At the end? Why or why not? If you cannot assume equivalence at either end, what can you do? Please explain.

If participants are randomly assigned to a group at the beginning of an experiment, this does not necessarily mean that the groups are completely “equivalent.” Randomization at the beginning of an experiment protects against differences due to systematic differences between the groups; however, differences in the group may still exist due to chance. At the end, randomization is still technically protective against systematic differences in the population; however, this protection may be impugned if the randomization is broken by attrition: if individuals drop out of the experiment, one cannot be assured the differences between groups were due to chance. If the individuals drop out randomly (that is, not due to observed variables), attrition does not affect results; however, if the attrition is due to the treatment (or unobserved variables), the groups are no longer equivalent.

Although complete (random) equivalence cannot be assumed at either end, statistical correction methods can be used to help narrow the gap.

Explain and give examples of how the particular outcomes of a study can suggest if a particular threat is likely to have been present.

The threats related to outcomes of a study are essentially a function of study design. For example, a one-group post-test design will have numerous threats to the outcomes of the study design, regardless of the question being asked: history, maturation, etc. While some of those threats will be obviated by introducing a pre-test post-test design, or through introduction of a control group- they will still exist.

1. Describe each of the following types of designs, explain its logic, and why the design does or does not address the selection threats discussed in Chapter 7 of Trochim and Donnelly (2006):
2. Non-equivalent control group pretest only- a non-equivalent control group pretest only would be marginally better than say a research design that featured a single group with a post-test design. This is because: The control group pretest would allow the researcher to assess the treatment group’s baseline- at the least, this design would take out threats such :

• History- does not address history- only a pre-test will not pick up an event that influenced both groups over the experiement’s duration.
• Maturation- Without a post-test for the control group, it will not be possible to understand if both groups matured for reasons not related to the intervention.
• Testing- would help with testing bias as only one group takes the test twice.
• Instrumentation and mortality- would not address these threats.
• Regression to the mean- would not address this

Non-equivalent control group pretest/posttest

• History- would address history because the pre- and post-test in the control group would allow the researcher to strip out the differences due to history in groups.
• Maturation- would address maturation because pre- and post-test in control group would allow research to strip out differences.
• Testing- would not address testing since both groups are testing twice.
• Instrumentation and mortality- would address these threats.
• Regression to the mean- would address this because the control group scores could be used to calibrate variance.
  Cross-sectional
• A cross-sectional design is less optimal, at least compared to a longitudinal design that compares respondents over time.
  Regression-Discontinuity
• Regression discontinuity designs are used when cutoffs are needed for a quasi-experimental design. The regression discontinuity design offers the benefits of having properties of randomization, without the need to actually undertake an experimental analysis.

Why are quasi-experimental designs used more often than experimental designs?

Quasi-experimental designs are used more often than experimental designs for a number of reasons. First, they are typically much cheaper than experimental designs: while conducting experiments is quite expensive, observational data is relatively cheap. Second, there are fewer ethical restrictions with quasi-experimental studies. Third, quasi-experimental studies are more generalizable than experimental studies.

One conclusion you might reach (hint) after completing the readings for this assignment is that there are no bad designs, only bad design choices (and implementations). State a research question for which a single-group post-test only design can yield relatively unambiguous findings.

One potential research question would be: how does the use of a computer program assist the learning of spelling words in an elementary class?

Part II

What research question(s) does the study address?

The study’s primary research question is: what is the impact of television advertising on the consumer preferences of children (arguably via the proxy of parent purchases)?
2. What is Goldberg’s rationale for the study? Was the study designed to contribute to theory? Do the results of the study contribute to theory? For both questions: If so, how? If not, why not?

One of Goldberg’s main rationales is to understand the “true” impact of television advertising with differing results from experimental and quasi-experimental results. Experimental results on the topic of television advertising’s impact find a robust relationship between advertising and its impact on children’s putative preferences; quasi-experimental studies, however, find a less robust relationship. Thus, Goldberg hopes that studying a “natural experiment”, whereby English speakers in Quebec are exposed to advertisement vis US television and French speakers have no exposure, will help reconcile the dissonant results.

What constructs does the study address? How are they operationalized?

There are three main constructs addressed in the study:

• Television watching, particularly focused on exposure to commercial time. This was addressed via a survey to children and their watching habits.
• The second main construct was impact on children’s consumer preferences via adult purchasing- this construct was operationalized via two variables: 1) toy awareness- to what extent children were able to recognize toys advertised on TV; 2) cereals purchased- assessing to what extent cereals purchased were similar to those advertised.
• The third main construct was culture: culture was operationalized through language.

What are the independent and dependent variables in the study?

Because the researchers used two designs (a between group and within group design); there were different independent and dependent variables selected for the different analyses.

Between-group independent- The child’s cultural affiliation was the independent variable for the between group analysis. This is because if the cultural affiliation is the same for the two groups, the difference between groups is (potentially) due to television advertising.

Within-group independent- The child’s exposure to American children’s commercial television (ACTV) served as the independent variable for this analysis.

Dependent variables: The study had two dependent variables that were used for both between-group and within group analysis:

1) toy awareness (to what extent children were aware of different toys); 2) mean number of children’s cereal purchased.

Name the type of design the researchers used.

The researchers used a quasi-experimental design, focused on survey design.

What internal and external validity threats did the researchers address in their design? How did they address them? Are there threats they did not address? If so how does the failure to address the threats affect the researchers’ interpretations of their findings? Are Goldberg’s conclusions convincing? Why or why not?

Goldberg identifies a number of important threats that he tries to address.

The first threat to internal validity of his analysis: the confounding factor of culture. That is, Goldberg is trying to measure the differing impact of television commercial exposure on toy awareness and brand name cereal purchases. In order for his “natural experiment” to work, however, he needs to confirm that the differences in results are truly a function of exposure to independent variable rather than individuals placed in the wrong group.

The author also employed the between group and within group analysis to correct for potential differences between groups.

Sources:

Goldberg, N. (1990). A quasi-experiment assessing the effectiveness of TV advertising directed to children. Journal of Marketing Research, 27, 445-54.

Jackson, S. (2012): Chapter 13
Trochim W. M. K., & Donnelly, J. P. (2008): Chapters 7, 10, and Pages 308-330

Social Research Methods. Available at: socialresearchmethods.net.