Samples, Power Analysis, and Design Sensitivity Samples, Power Analysis, and Design Sensitivity, Essay Example

Compare and contrast internal and external validity. Describe and give examples of research questions for which external validity is a primary concern. Describe and give examples of research questions in which internal validity is a primary concern. Discuss strategies researchers use in order to make strong claims about the applicability of their findings to a target population.

Internal and external validity are important concepts in conducting research. Internal validity refers to the relationship between the independent and dependent variable; that is, internal validity pertains to whether the effect observed (if any) is due to the independent variable on the dependent variable, or potentially due to other spurious (confounding factors). The issue of external validity is one of generalization: that is, do the findings acquired apply to the population in question, or do they simply apply to the sample examined.

The question of whether internal versus external validity is a concern is usually connected to the study design.  For example, external validity is typically a concern in randomized control trials.  This is because randomized control trials have the more robust controls (randomization) to ensure that internal validity is maintained.  However, the tradeoff for having stronger internal validity is weaker external validity.  That is, the controls on the sample population and inclusion factors mean that the results may not be generalizable to other populations. Internal validity is typically a problem in observational or studies that do not include randomization.  This is because without the effect of randomization, the results received may be due to the relationship between the independent and the dependent variable, they may also be due to confounding (or other factors).  However, the study design does not allow one (without sophisticated statistical correction methods) to assess which is the problem. Overall, because there is an explicit trade-off between the two, one needs to make the decision of what study design (and thus what threats to validity) one is willing to deal with to get the needed result.

Compare and contrast random selection and random assignment. Be sure to include a discussion of when you would want to do one or the other and the possible consequences of failing to do random selection or random assignment in particular situations.

Random selection and assignment are two important concepts in methodology that can overlap, as well as be mutually exclusive.  Random selection is related to how a sample is drawn from the larger population.  That is, if I am looking to select a population of 100 children from a class, I have a number of options.  One is to choose 100 names randomly from the list based on  a random number generator; another way would be to simply take the first 100 names on the list.  The first method is random selection; the second is non-random selection. The issue of random selection is connected to the issue of external validity.  In theory, if a random sample is drawn, the sample should mirror (to some extent) the observable characteristics found in the population; the results from the exercise should be (but certainly not necessarily) externally valid (generalizable) to other populations.

The second issue is one of random assignment. That is, once a sample is selected, how are individuals selected to receive different treatments?  A random assignment would mean that of the individuals selected, the treatment is allocated based on a random assignment such as a random number generator or a random selection of names. Random assignment is related to internal validity.  That is,  if random assignment is performed, then the results have a greater claim to be internally valid.  If random assignment is not performed, the results have a less robust claim to be internally valid.  A researcher can implement both concepts, one of the concepts, or neither in designing a potential experiment.

Explain the relationship between sample size and the likelihood of a statistically significant difference between measured values of two groups. In other words, explain why, all else being equal, as sample size increases the likelihood of finding a statistically significant relationship increases.

The basic relationship between sample size and the likelihood of a statistically significant difference is: an increase in sample size (n) leads to the increased likelihood of statistical significance in an observation. This is because an increase in the total sample size will increase the variance in the examined sample that will be helpful in assessing whether the results are due to chance or actual differences in the population.

Compare and contrast probability and non-probability sampling. What are the advantages and disadvantages of each?

Non-probability sampling is a method such as snowballing in which a researcher is not necessarily worried about getting a representative sample of the population; but rather acquiring a sample of certain characteristics they are interested in. For example, a researcher may want to use non-probability sampling techniques to understand more the factors behind drug abuse or to understand minority underperformance in schools.

Probability sampling, on the other hand, deals more with trying to get an accurate sample population that mirrors population. Thus, many intricate sampling methods are used  such as SRS and other techniques to get the right representation from the population.

Part 2

• The sample size needed for these factors are:

one-tailed t-test with two independent groups of equal size
small effect size .05  alpha =.05
beta = .2

The initial N= 2469

The initial N divided into half: 1234; the alpha and beta for that calculation is: Alpha= .086 and Beta is .34. This result was achieved with the compromise function.
Rationale for the smaller study:

Overall, the initial levels of alpha (.05) and beta (.80) could be met with a sample size around 2469.  However, that sample size due to a confluence of factors including money and logistics could not ultimately be met.  Thus, the sample size was cut in half in order to go forward with the experiment.  There were sacrifices made in order to achieve this goal: Foremost, the initial levels of alpha were raised from .05 to .086 that means a higher level of uncertainty will exist whether our result was due to a chance or a difference between the two populations.  Also, the beta level increased from .2 to .34 meaning the study may be underpowered and fail to pick up a difference between the populations.

• Calculate the sample size needed given these factors:

ANOVA (fixed effects, omnibus, one-way)

small effect size

alpha =.05

beta = .2

3 groups

The total sample size needed for the original specifications is 3858. When the compromise button was selected to halve the study’s sample size, there was a significant increase in both the alpha and beta elements of the study. The alpha increased from .05 to .27, meaning that it will now be quite difficult to tell whether the study’s result is due to chance or is due to an actual difference between the populations.

The same is true with beta.  Beta is used as a key input to calculate the power of the study.  For this study, the beta was originally set at the typical level of .20 meaning that power would be at .8.  However, after halving the sample size, the study’s beta shot up to .34.  This means that the study may ultimately be unable to detect differences between the population and the sample.

There are ultimately two different designs that can be used to address my research questions: ANOVA and multivariate regression.  Both of these statistical methods could be used to explore the four factors of the research question that include the demographic variables income, education, location, and parents’ education level.  The difference in the two would be the sample size needed for an alpha of .05 and a beta of .2 that would render a power of .8.  The ANOVA takes a lower sample size of 1500, while the multivariate regression takes a larger sample size of around 2000.  The difference between the two is a function of having more control variables for the linear multiavariate regression that essentially consume more degrees of freedom, and thus need a larger sample size.  Overall, due to sample size characteristics needed for this study,  I would likely suggest using multivariate regression rather than the ANOVA.

References

Houser, J. (2007). How many are enough? Statistical power analysis and sample size estimation in clinical research. Journal of Clinical Research Best Practices, 3(3), 1-5.

Acheson, A. (2010). “Sample Size.” Encyclopedia of Research Design. SAGE Publications.

Piasta, S.B., & Justice, L.M. (2010). “Cohen’s d Statistic.” Encyclopedia of Research Design. SAGE Publications.

McCready, W. (2006). “Applying Sampling Procedures.” The Psychology Research Handbook. SAGE Publications.

G*Power 3 software and documentation

Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175-191

Dupont, W. D., & Plummer Jr., W. D. (1990). Power and sample size calculations: A review and computer program. Controlled Clinical Trials, 11 (2), 116-128.