Basic Research Methods and Statistical Concepts, Case Study Example

Briefly describe your area of research interest (1-3 sentences is sufficient).

Bidding on large-scale construction projects requires a substantial investment of time and resources on the part of the bidder.  Improved profits can be generated by determining in advance whether a bid is both likely to be successful, and likely to generate appropriate rewards for the company.  Models of bid/no-bid decision processes thus can generate significant cost-savings by significantly reducing unrewarding bids. The research planned is in modeling the bid/no bid decision process using two techniques: fuzzy logic and neural networks to determine which does the best in decision-making about whether to bid on complex construction projects.

List 4 variables that you might assess in a research project related to your research area. List one for each type of measurement scale: Nominal, ordinal, interval, and ratio. If you cannot think of a variable for each measurement scale, explain why the task is difficult. 

Variables that may be important in this research include the following, with those variables that may be fuzzified in the fuzzy model indicated:

• Nature of project (nominal category)
• Competition for project (nominal category—possible fuzzy)
• Approximate cost (interval)
• Reputation of client (nominal category—possible fuzzy)
• Ranking of potential bidders (ordinal—possible fuzzy)
• Risk/reward estimate (ratio—possible fuzzy)
• Estimated cost to do bid (interval)
• Bid result (nominal category)
• Project profits (if successful) (interval)
• Estimate of risk (interval—possible fuzzy)

In the neural network model, these variables will have to be normalized to a standardized input interval.

Create one alternate hypothesis and its associated null hypothesis related to your research area.

Hypothesis-1:  A fuzzy logic bid/no-bid decision model on major construction projects can increase profits more than a neural network-based model.

Null Hypothesis:  A fuzzy logic bid/no-bid decision model for major construction projects shows no difference in profits over a neural network-based model.

Briefly describe whether you think your area of interest is more conducive to experimental or correlational research. What are the costs/benefits of each as they relate to your research area? 

My research effort is related to experimental research.  It consists of building multiple models of bid/no-bid decision process (the experiment), and  then doing an analysis of the results of each to determine the best model for the decision-making process. The goal of the project is to find the type of bid/no-bid decision model that best increases profits.  Because each complex construction project is unique with unique problems and issues, simple correlation models are inappropriate.  Instead, it is important  to construct a model that considers multiple aspects of the bid opportunity.

Reliability vs. Validity. Considering your area of research interest, discuss the importance of reliability and validity. Can you have one without the other? Why or why not?

Issues of reliability and validity are going to be critical in determining the success of the models constructed.  In essence, reliability measures how consistent the models are in generating answers; validity measures how accurately the models reflect what they’re intended to model—that is, whether they produce the correct decisions.  A model can be utterly reliable—and deterministic models (such as neural network models) are always reliable in the sense that the same inputs always yield the same outputs.  (Fuzzy models have a probabilistic aspect to them that can generate some variation in outputs, depending on how the model is constructed.) Despite that reliability, if the model does not accurately model reality—if it doesn’t give the “right” answer—it would have little validity.  Thus reliability and validity are different measures.

Since neural network models are deterministic once created, test-retest reliability is not relevant since the same inputs will always generate the same output. In the fuzzy model, test-retest reliability measures can be important if different people fuzzify the same input in different ways.  Under such circumstances, the same raw data can result in different fuzzy model output. Also, fuzzy models have an intrinsic probabilistic aspect, which also can vary outputs.  Thus test-retest reliability for the fuzzy model is important.

Validity of both models is a measure of how accurately they model the desired outcome, which is whether a bid should be offered on a particular construction project.  It essentially measures whether either (or both) models accurately models the decision-making process.

Sample vs. Population. Considering your area of research interest, describe the difference between a sample and population. Why is it important to understand the difference between a sample and population in a statistics course?

Sample size and population are critical in developing both neural network and fuzzy models.  In both these models, the total collection of bidding opportunities that have data available  (i.e., the population) must be broken into at least two samples, one of which is a training set, used during the model’s construction. The other sample is the ‘validation’ sample used to confirm that the models accurately generate appropriate decisions.

The issues involved in identifying appropriate samples for each of these two data sets are critically important in building a reliable, valid model. The training set must be representative of the overall population and cannot be a convenience sample.  This typically is done by taking the entire population and constructing a random training set from that data.  Sometimes k-1 sampling methodologies are used to determine an appropriate sample set.

In some model development techniques, the total population has three data samples: a training sample, a test sample, and a validation sample.  The training sample is used to construct the model  as noted above. Since determining the stopping point in the model construction can be challenging, the test sample is intermittently used to determine how well the model can solve cases not used in training. However, such cases, since they are part of the training process, cannot be used in the final validation process which requires using a sample set the model has never before seen.

Measures of Central Tendency. 

Below is a set of data that represents weight in pounds for a particular sample. Calculate the mean, median and mode. Which measure of central tendency best describes this data and why? You may use Excel, SPSS, some other software program, or a hand calculator for this problem.
110.00 117.00 120.00             118.00             104.00             100.00             107.00
115.00             115.00             115.00             114.00             100.00             117.00             115.00
103.00             105.00             110.00             115.00             250.00             275.00

(Answers  developed using Excel)

Mean:  126.84 (arithmetic average)

Median:  115  (half of scores above; half below)

Mode:  115 (most frequent score)

The median is the best measure of central tendency because this distribution has two extreme outlier data points (250.00 and 275.00).  The mean is strongly skewed when one or two extreme outliers are included in the data set. (As an example, if the two largest data points are ignored, the mean changes to 110.88, a dramatic difference; in contrast, the median changes to 114, only a moderate change.) The mode does not change because it simply represents the single data value that occurs most frequently.

Measures of dispersion. For the data set above, calculate the range, the interquartile range, the variance, and the standard deviation. What do these measures tell you about the “spread” of the data? Why is it important to spend time performing basic descriptive statistics prior to conducting inferential statistical tests?

(Answers developed using Excel)

Range:                                    100.00 to 275.00  (minimum to maximum value in set)

Interquartile Range:                105.00  to 117.00  (the range from 25^th %ile to 75^th %ile)

Variance:                                2340.70 (average squared deviation from mean)

Standard Deviation:                48.38 (square root of variance; a measure of dispersion of data)

The range of the data shows how tightly clustered the data is. In particular, in this example, the range is quite large (100 to 275), yet the IQR is quite small (105 to 117). This means that half the data (the 25^th to 75^th percentiles) is very tightly clustered in a small range. The variance and standard deviations are relatively huge compared to that IQR. This implies some significant outliers in the data. An inspection of the data reveals two values (250 and 275) that are far outside the range of the rest of the data. The computation of the standard deviation for all but those two values is only 6.54, a dramatic drop.

Because of the outliers, they may require special consideration when performing statistical tests. Depending on the purpose of the data sample and the goals of the analysis, it may be appropriate to drop those data points (documenting that decision). Or it may mean that the sample selection process is inappropriate, particularly in a convenience sample selection process. In any event, it requires careful consideration of the data and the sample characteristics to determine how best to process the data.

Descriptive Statistics. Why is it important to perform basic descriptive statistics prior to conducting inferential statistical tests?

As explained above, descriptive statistics can help determine whether an appropriate sampling process was used in the research program.  This is especially true when a convenience sampling technique was used, because that may yield a sample that is not truly representative of a population.

Statistical Significance. Revisit the hypotheses you created above in #5. If you conducted a statistical test based on these hypotheses and found a statistically significant result, what would that mean from both a statistical and practical standpoint? (be sure to use the phrases “null hypothesis” and “effect size” in your answer).

The null hypothesis is always one that implies that the effect size will be zero (or its equivalent) in the total population. If the main hypothesis proves statistically significant, it implies that a non-zero effect size was observed  in the sample. From a pragmatic perspective, that would imply that the fuzzy logic decision model proved more effective at increasing profits in complex construction projects than a neural network model.

Type I and Type II Error. The concept of Type I and Type II Error is critical and will come into play not only with each and every statistical test you perform, but when you are asked to conduct an a priori power analysis for your Dissertation Proposal. Considering your answer to #10, discuss the implications of making both a Type I and Type II error.

The Type I error is one where the null hypothesis is rejected when it is in fact true. Since the null hypothesis always pertains to the overall population rather than the specific sample chosen, this can be a result of invalid sample selection process.  In effect, this is a “false positive” result—claiming a positive result when it should be negative.

A Type II error is the inverse of the Type I error. This constitutes not rejecting the null hypothesis when it should be rejected.  This is a “false negative” result, claiming a negative result (i.e., the null hypothesis) when it should have been positive.

The impact of these errors strongly depends on the specific application being chosen. In the case of the proposed hypotheses, the results may result in incorrect bid decisions being made. In this case, whether bids are not made that should be made, or whether bids are made that should not be submitted, the result is a loss of profits.

References

Chan, A. P., Chan, D. W. M., Yeung, J. F. Y. (2009). Overview of the application of “fuzzy techniques” in construction management research. Journal of Construction Engineering & Management, 135 (11), 1241-1252.

Hua, G. B. (2008). The state of applications of quantitative analysis techniques to construction economics and management (1983 to 2006). Construction Management & Economics, 26 (5), 485-497.

Swanson, G.C. (2011). Statistics: A User Friendly Guide (Especially for the Mathematically Challenged, 2^nd Ed. Gray Heron Press. Kindle Edition.

Urdan, T. C. (2010). Statistics in Plain English 3^rd Edition. New York, NY: Routledge.