Designing a Research Project

Abstract

All science advances through the rigorous application of the scientific method. Part of this process involves the development of an empirical research design that can help researchers determine whether or not the hypothesis being tested is likely to be true. Good research design is based on a researcher's empirical observations and a review of the scientific literature. The information garnered from these sources is then formulated into a testable hypothesis that can be analyzed using inferential statistics. The research design used to test this hypothesis needs to not only consider the effect of various levels of the independent variables on the dependent variables, but also control as much as possible any extraneous variables that are not related to the research question but that might affect the results. The experimental data are analyzed to determine their statistical significance and the likelihood the null hypothesis is true using statistical tests.

Overview

Progress in the physical, behavioral, and social sciences is made through the systematic and rigorous application of the scientific method to observed real-world phenomena. The scientific method comprises the general procedures, guidelines, assumptions, and attitudes required for the organized and systematic collection, analysis, interpretation, and verification of data that can then be reproduced. The goal of the scientific method is to articulate or modify the laws and principles of a science. As shown in Figure 1, steps in the scientific method include

• problem definition based on observation and review of the literature;
• formulation of a testable hypothesis;
• selection of a research design;
• data collection and analysis;
• extrapolation of conclusions; and
• development of ideas for further research.

Observing & Researching Phenomena. Typically, scientific research begins with the scientist's empirical observations. For example, I might observe that when I wear a business suit to a meeting even when other people are wearing more casual clothes, I tend to be afforded more respect than when I wear less formal attire. If curious, I might next look at social science literature to see if anyone else has observed such incidents and hypothesized an underlying cause. I might find that there is a large body of research on how to "dress for success." My literature review might reveal that other scientists have not only observed these behaviors but also theorized about their causes and conducted research to test their theories. Problem definition relies on both of these sources of information: the researcher's observations of real-world phenomena and the research results and theories that are described in the scientific literature.

If the literature review has not answered all my questions and I am still curious about the nature of this phenomenon, my next step would be to formulate a testable hypothesis. This is not necessarily as easy as it sounds; although it might be relatively easy to articulate a naïve theory concerning the relationship between attire and success in the workplace, such as that people who wear business attire are more likely to be successful at work, such statements are vague and not testable. To be able to test this tentative hypothesis using the scientific method, one must determine what factors are important in this theory and then operationally define the associated terms.

Identifying & Defining Variables. In the simplest research design, a stimulus, such as a person wearing either business attire or casual attire, is presented to the research subjects—in this case, potential customers, supervisors, or other people who might be encountered in a business setting. The responses of the subjects are then observed and recorded. From a research design point of view, both the stimulus and the response are called variables. The variables of most concern in the design of a research study are the independent variable, which is the stimulus or experimental condition that is hypothesized to affect the outcome (e.g., how one dresses in the workplace), and the dependent variable, which is the observed effect on outcome caused by the independent variable (e.g., the reactions of research subjects to people wearing business attire). As shown in Figure 2, researchers must also consider extraneous variables, or variables that can affect the outcome of the experiment but have nothing to do with the independent variable itself. For example, if the person wearing either business or casual attire is rude when interacting with a research subject, this rudeness will probably have a much stronger effect on the subject's response than how the person is dressed. Similarly, if the person has visible tattoos, is poorly groomed, or looks like the subject's ex-spouse, the subject's response may be a reaction to these extraneous variables rather than to the independent variable. Any number of extraneous variables may affect an experiment. The more of these variables that are accounted for and controlled in the experimental design, the more meaningful the results of the research study will be.

In addition to determining which variables are important in the research study, it is also essential to operationally define them. An operational definition is a definition that is stated in terms that can be observed and measured. In this example, the researcher might operationally define "business attire" as a dark suit with a white shirt or blouse. "Casual attire" might be operationally defined as shorts and a t-shirt. However, by operationally defining these terms in this manner, the researcher is by necessity limiting the generalizability of the research results. With these definitions, the researcher will only be able to draw a conclusion about subjects' reactions to people wearing dark suits and white shirts or blouses versus their reactions to people wearing shorts and t-shirts. A whole range of other work-appropriate attire exists: sports coats and blazers, colored shirts or blouses, Bermuda shorts, polo shirts, and any other type of clothing that could be worn in the workplace. The researcher must decide how many of these options are important to the theory and should be tested in the experiment. The researcher, believing that it is important to look at a range of clothing options, might decide that several conditions of the independent variable are needed—formal business attire, informal business attire, and business casual clothing, for example—and design an experiment that examines subjects' reactions to all three levels of formality. Or, based on the research literature, the experimenter might conclude that formal business attire has already been demonstrated to result in better treatment in the workplace and decide to examine the limits of this conclusion. Accordingly, he or she might design an experiment in which subjects are exposed to people variously wearing black suits, charcoal gray suits, and navy suits with white shirts or blouses to see if there is any difference in the way that the subjects react. At some point, however, the researcher will have to limit the definitions of the variables to a manageable number, which is done in part by determining which inferential statistical techniques are available to analyze the data.

Constructing a Hypothesis. After the variables are identified and defined, the researcher will develop a formal hypothesis for the experiment that can be analyzed with inferential statistics. For this purpose, hypotheses are stated in two ways. The first of these is called the null hypothesis (H0), which is a statement that asserts that there is no statistical difference between the status quo and the experimental condition. In other words, the null hypothesis states that the manipulation of the independent variable being studied made no difference on the dependent variable, or the subjects' responses. For example, a null hypothesis might state that there is no difference between the way people in the workplace react to people who wear dark suits and the way they react to people who wear business casual clothing. In addition, the researcher will develop an alternative hypothesis (H1) that states that there is a relationship between the two variables—for example, that people tend to be more respectful in the workplace of people who are wearing dark suits.

Designing an Experiment. Once the null hypothesis has been formulated, an experimental design is developed that allows the researcher to empirically test the hypothesis. Typically, the experimental design includes a control group that that does not receive the experimental condition and an experimental group that does receive the experimental condition. The presence of a control group helps minimize the influence of the extraneous variables and determine how accurately the data collected from the experimental group describes the relationship between the independent and dependent variables.

For example, if one wanted to determine whether or not hearing a political candidate's speech changed people's minds about that candidate, the researcher could divide a sample of people into two groups and collect data about their initial opinions regarding the candidate. One of the groups would then hear the speech, and afterwards, all of the subjects would be asked for their opinions again. If the opinions of the control group, or the group that did not hear the speech, did not change and the opinions of the experimental group, or the group that did hear the speech, did change, then the researcher may be able to conclude that the speech was influential. However, if both groups show a similar change in opinion, then the change is more than likely due to something other than the speech, since the control group was not exposed to the speech.

After running the experiment, the researcher then collects data from the people in the study to determine whether or not the experimental condition had any effect on the outcome. Once the data have been collected, they are statistically analyzed to determine whether the null hypothesis—that there is no difference between the control and experimental groups—should be accepted or rejected. By accepting the null hypothesis, the researcher is concluding that the independent variable had no effect on the dependent variable (e.g., that a political speech had no effect on the people who listened to it, or that the way people dress in the workplace does not affect the way they are treated). If, on the other hand, it is found that the results of the data analysis are statistically significant, the researcher will conclude that it is probable that the difference observed between the experimental and control groups is due not to chance but to a real underlying relationship between the independent variable and the dependent variable. Additionally, particularly in the case of certain types of studies, some researchers argue that there are specific statistical calculations that should be factored into the conclusion regarding the degree of effect or lack of effect to ensure greater validity (Beard, Dienes, Muirhead, & West, 2016).

Applications

Part of research design and hypothesis development is determining how the research data will be statistically analyzed. It is important to note that the design of the experiment limits one's choices of how to analyze the data, and the researcher must determine which statistical tools will be used to analyze the data before data collection begins so that he or she can be assured that all the necessary information will be collected for analysis. In most cases, it is impossible to go back and collect additional data, meaning that the research study would need to be performed again from the beginning.

Inferential Statistics. In research studies, inferential statistics are used to test hypotheses to determine if the results of a study occur at a rate that is statistically significant, meaning that they are unlikely to be due to chance. There are a number of statistical methods for testing hypotheses, each of which is appropriate to a different type of experimental design. One commonly used class of statistical tests is the various t-tests. These tests are used to analyze the mean of a population or compare the means of two different populations. Another frequently used technique for analyzing data in applied settings is analysis of variance (ANOVA). This family of techniques is used to analyze the joint and separate effects of multiple independent variables on a single dependent variable and determine the statistical significances of the effects. For example, analysis of variance might be used if one wished to examine the differences between research subjects' reactions to people wearing black suits, navy suits, and grey suits. For more complicated situations, multivariate analysis of variance (MANOVA), an extension of this set of analysis of variance, allows researchers to test hypotheses involving the simultaneous effects of multiple independent variables on multiple dependent variables. Other inferential statistical tests include correlation, which determines the degree to which two variables are related, and regression analysis, which is used to build models of complex real-world data. Again, which statistical tool is used depends on the kinds of data available and the hypothesis the researcher is testing.

Because not every experimental situation in the behavioral and social sciences yields neat or ideal data, inferential statistical tools such as t-tests and analysis of variance are called parametric statistical tools, meaning that they make certain assumptions about the underlying distribution of the data they analyze. For instance, these tools assume that the measurement scale used to articulate the data has a meaningful zero point and intervals of equal size.

Fortunately, researchers do not need to misuse parametric statistics or forgo statistical analysis completely in situations where data do not meet the assumptions underlying parametric statistics. A number of nonparametric procedures that correspond to common parametric tests and do not make assumptions about the underlying distribution can be used when the shape and parameters of a distribution are known. These statistical tools are not as powerful as standard parametric statistics, but for situations in which the data set is less than perfect, they do allow the analyst to derive meaningful information. For instance, in cases where subjects have dropped out of an experiment or become unavailable for follow-up, statistical adjustments such as weighting can be used to achieve validity by compensating for such imbalances (Mercer, Kreuter, Keeter, & Stuart, 2017).

Conclusion

In order for any branch of science to advance, it is necessary to conduct rigorous empirical research that adheres to the principles of the scientific method: observation and review of the literature, formulation of a testable hypothesis, selection of a research design, data collection and analysis, extrapolation of conclusions, and development of ideas for further research in the area.

In the simplest research design, a stimulus is presented to the research subjects. The responses of the subjects are then observed and recorded. The researcher needs to determine which independent and dependent variables are important to the research question and then operationally define the variables so that they can be statistically analyzed and used to draw meaningful conclusions. This information is then turned into a formal hypothesis that is stated two ways: a null hypothesis, which states that the manipulation of the independent variable being studied has no effect on the dependent variable, and an alternate hypothesis, which states that the value of the independent variable does have an effect on the dependent variable. Based on this information, an experimental design is developed that allows the researcher to control any extraneous variables as much as possible, thus allowing him or her to observe changes in the dependent variable that are concurrent with changes in the independent variable. Part of the process of designing an experiment involves determining how the research data will be analyzed so that the researcher can establish the statistical significance of the results and either accept or reject the null hypothesis.

Terms & Concepts

Analysis of Variance (ANOVA): A family of statistical techniques that analyze the joint and separate effects of multiple independent variables on a single dependent variable and determine the statistical significances of the effects.

Correlation: The degree to which two events or variables are consistently related. Correlation may be positive (as the value of one variable increases, the value of the other variable increases), negative (as the value of one variable increases, the value of the other variable decreases), or zero (the values of the two variables are unrelated). Correlation does not imply causation.

Data: In statistics, quantifiable observations or measurements that are used as the basis of scientific research.

Dependent Variable: The outcome variable or resulting behavior that changes depending on whether the subject receives the control or experimental condition.

Empirical Evidence: Evidence that is derived from or based on observation or experiment.

Independent Variable: The variable in an experiment or research study that is intentionally manipulated in order to determine its effect on the dependent variable.

Inferential Statistics: A subset of mathematical statistics used in the analysis and interpretation of data. Inferential statistics are used to make inferences, such as drawing conclusions about a population from a sample, as well as in decision making.

Mean: An arithmetically derived measure of central tendency in which the sum of the values of all the data points is divided by the number of data points.

Null Hypothesis (H0): The statement that the findings of an experiment will show no statistical difference between the control condition and the experimental condition.

Operational Definition: A definition that is stated in terms that can be observed and measured.

Sample: A subset of a population. A random sample is a sample that is chosen at random from the larger population with the assumption that it will reflect the characteristics of the larger population.

Scientific Method: The general procedures, guidelines, assumptions, and attitudes required for the organized and systematic collection, analysis, and interpretation of data that can then be verified and reproduced. The goal of the scientific method is to either articulate or modify the laws and principles of a science. Steps in the scientific method include problem definition based on observation and review of the literature, formulation of a testable hypothesis, selection of a research design, data collection and analysis, extrapolation of conclusions, and development of ideas for further research in the area.

Statistical Significance: The degree to which an observed outcome is unlikely to have occurred due to chance.

Subject: A participant in a research study or experiment whose responses are observed, recorded, and analyzed.

Bibliography

Armore, S. J. (1966.) Introduction to statistical analysis and inferences for psychology and education. New York: John Wiley & sons.

Beard, E., Dienes, Z., Muirhead, C., & West, R. (2016). Using Bayes factors for testing hypotheses about intervention effectiveness in addictions research. Addiction, 111(12), 2230–2247. Retrieved October 22, 2018, from EBSCO Online Database Sociology Source Ultimate. http://search.ebscohost.com/login.aspx?direct=true&db=sxi&AN=119309842&site=ehost-live&scope=site

Cooley, W. W., & Lohnes, P. R. (1971). Multivariate data analysis. New York: John Wiley and Sons.

Dallal, G. E. (2007). Nonparametric statistics. Retrieved August 20, 2007, from http://www.jerrydallal.com/LHSP/npar.htm

Hanson-Hart, Z. (n.d.). Statistical reasoning. Retrieved July 24, 2007, from http://www.math.temple.edu/~zachhh/ch5.pdf.

Hollander, M. & Wolfe, D. A. (1973). Nonparametric statistical methods. New York: John Wiley and Sons.

Huff, D. (1954). How to lie with statistics. New York: W. W. Norton & Company.

MacKinnon, D. P. (2011). Integrating mediators and moderators in research design. Research on Social Work Practice, 21, 675–681. Retrieved November 5, 2013, from EBSCO Online Database SocINDEX with Full Text. http://search.ebscohost.com/login.aspx?direct=true&db =sih&AN=66817231&site=ehost-live

Mercer, A. W., Kreuter, F., Keeter, S., & Stuart, E. A. (2017). Theory and practice in nonprobability surveys: Parallels between causal inference and survey inference. Public Opinion Quarterly, 81, 250–271. http://search.ebscohost.com/login.aspx?direct=true&db=sxi&AN=123082527&site=ehost-live&scope=site

Mitchell, Lada. (2003, Sep). Book review: Applied multivariate statistics for the social sciences. Journal of the Royal Statistical Society: Series D (The Statistician), 52 , 418–20. Retrieved August 20, 2007 from EBSCO Online Database Business Source Complete. http://search.ebscohost.com/login.aspx?direct=true&db=bth&AN=10637883&site=ehostlive

Schaefer, R. T. (2002). Sociology: A brief introduction (4th ed.). Boston: McGraw-Hill.

Welsh, B., Peel, M., Farrington, D., Elffers, H., & Braga, A. (2011). Research design influence on study outcomes in crime and justice: A partial replication with public area surveillance. Journal of Experimental Criminology, 7, 183–198. Retrieved November 5, 2013, from EBSCO Online Database SocINDEX with Full Text. http://search.ebscohost.com/login.aspx?direct=true&db=sih&AN=60874161&site=ehost-live

Wendt, O., & Miller, B. (2012). Quality appraisal of single-subject experimental designs: An overview and comparison of different appraisal tools. Education & Treatment of Children, 35, 235–268. Retrieved November 5, 2013, from EBSCO Online Database SocINDEX with Full Text. http://search.ebscohost.com/login.aspx?direct=true&db=sih&AN=74231752&site=ehost-live

Witte, R. S. (1980). Statistics. New York: Holt, Rinehart and Winston.

Suggested Reading

Brown Urban, J., & van Eeden-Moorefield, B. M. (2018). Designing and proposing your research project. Washington, DC: American Psychological Association.

Calfee, R. C. (1975). Human experimental psychology. New York: Holt, Rinehart and Winston.

Gravetter, F. J. & Wallnau, L B. (2006). Statistics for the behavioral sciences. Belmont, CA: Wadsworth/Thomson Learning.

Seidman, E. (2012). An emerging action science of social settings. American Journal of Community Psychology, 50(1/2), 1–16. Retrieved November 5, 2013, from EBSCO Online Database SocINDEX with Full Text. http://search.ebscohost.com/login.aspx?direct=true&db=sih&AN=78190954&site=ehost-live

Webster, M. & Sell, J. (2007). Laboratory experiments in the social sciences. New York: Academic Press.

Young, R. K. & Veldman, D. J. (1977). Introductory statistics for the behavioral sciences (3rd ed.). New York: Holt, Rinehart and Winston.