Home 
Miscellaneous

Research Design II
Embed Code  If you would like this activity on your web page, copy the script below and paste it into your web page.
Normal Size Small Size show me how
Normal Size Small Size show me how
Research Design II
RDII MidTerm
Question  Answer 

CauseandEffect Sequence  the framework for explanation in scientific research policy sciences are a special case because you are evaluating a policy or program intervention to see if it work and if it makes a difference 
Proof  not the outcome of social science research the most we can do is gradually strengthen our confidence in the validity of a causal sequence by eliminating possible alternative explanations 
Repeated testing  through repeated testing we build confidence in a causal sequence 
Controls  In testing controls must be introduced for alternative explanations that are suggested by the theory, intuition, and observation 
Science is empirical  Measurement in the defining feature of science and precision is achieved though quantitative measurement “When you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind…” 
“When you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind…”  Measurement is the defining feature of science but quantification isn’t enough (“When you can measure it, when you can express it in numbers, your knowledge is still of a meager and unsatisfactory kind…”) 
The principal task of all science is…  to explain 
The framework for explanation is the _____ sequence  cause and effect 
The process by which we build confidence is a causal sequence is ____  Repeated Testing 
Repeated testing  involves introducing controls for alternative explanations suggested by theory, intuition, or observation 
Science is empirical with ___ as its defining feature  Measurement is the defining feature of science. 
Data Sets  Collections of measurement for a specified u/a 
clean data  Not collected but developed by filtering out unwanted influence by statistical analysis and research design 
Better data=  more reliable study = better validity to relationships 
two ways to develop data and to filter out unwanted influences  Statistical analysis (most common) and Research design 
What is the time orientation, relative emphasis, and variance Control strategy of data development through statistical analysis?  ex post facto ("mopping up"), computational, data adjustment 
What is the time orientation, relative emphasis, and variance Control strategy of data development through research design  ex ante, conceptual, eliminate at source 
Research design  the study of techniques for organizing a research process to achieve variance control prior to and during data development (compare to afterthefact cleaning and controlling achieved through statistics and econometrics) 
Bias  Errors that seep into a structured inquiry at one or more junctures 
5 sources of bias  sampling, selection, measurement, model specification, statistical 
Steps of a Structured Inquiry; Step 1: universe/population  Identify "target" population (no sources of bias) 
Steps of a Structured Inquiry; Step 2: Sampling  Select a workable data subset (Sampling bias: “Is the sample representative of the larger population?”) 
Sampling bias  “Is the sample representative of the larger population?” 
Steps of a Structured Inquiry Step 3: Model Specification  Define the basic concept in the Model (Model Conceptualization and Omitted Variable Bias: “Is the model complete?""Are all of the major influences present?") 
Omitted Variable Bias  “Is the model complete?""Are all of the major influences present?" 
Steps of a Structured Inquiry Step 4: Measurement: Variation Creation  Assign values to U/A attributes (Measurement Bias: “Do we have sloppy measurement?""Is indicator measurement reliable (accurate) and valid?""Do variables vary sufficiently?") 
Measurement Bias  “Do we have sloppy measurement?""Is indicator measurement reliable (accurate) and valid?""Do variables vary sufficiently 
Steps of a Structured Inquiry Step 5: Data Analysis Approach  Select appropriate data analysis model (Statistical Bias: “Are the assumptions underlying the analysis model met?") 
Statistical Bias  “Are the assumptions underlying the analysis model met?" 
Steps of a Structured Inquiry Step 6: Data Analysis Execution  Estimation and Inference (Selection Bias: “How adequate are the controls for eliminating alternative explanations?") 
Selection Bias  “How adequate are the controls for eliminating alternative explanations?" 
“All models are wrong, but some are useful."  A fantastic design helps limit the bias in a model so it is less wrong; even though a model may not be perfect it can still help say things about the world 
Unit of Analysis  The major entity that you are analyzing in your study U/A determined by the actual analysis done in the study; all of the study factors are things that describe the u/a 
3 Types of Data Structures  1) Panel 2) Cohort 3) Cross Sectional 
Panel Data Structure  Has an internal focus Use U/As as own controls controls for unwanted variance associated with individual differences 
Cohort Data Structure  External focus Controls for sources of unwanted influence associated with period influences controls for temporally defined influences 
Cross Sectional Data Structure  Controls specified variables least useful data structure 
Types of Design Structures  True Experiment ** Best QuasiExperiment Cross Sectional 
True Experimental Design Structure  Controls for all extraneous influences (known and unknown)but effectiveness can be diminished over time 
QuasiExperiment Design Structure  Depends on design structure Uses either time or comparison groups or both to diminish/eliminate specific threat to internal validity Introduced in 1950s by Campbell Do not use randomization to create comparison groups 
CrossSectional Design Structure  Depends on theory guided model specification Weakest and least effective control strategy 
Experimental Design Notions from 19th Century  IV and DV, effect, comparison group, randomization, ceteris paribus 
Randomization  assigning units of analysis randomly to differing levels of the IVrefers to external validity 
Control  rule out threats to causal inference 
5 Schools of Thought  Positivism (Hume), Essentialism (probabilistic versus deterministic), Concomitant Variation (Mill), Falsification (Popper), Activity (cause" as manipulation) 
Philosopher for the Positivist Tradition  Hume 
Hume's 3 conditions for inferring cause  1)contiguity (no middle factor x→Y) 2) temporal precedence (order) 3) constant conjunction (apparent all over  when X varies Y always varies) 
Positivism  guided by Hume's 3 conditions for inferring cause Hume only recognized observable phenomena causal relationship based solely on high correlations between variables 
Why is the concept of "cause" even needed? Challenger to Positivism?  B. Russell challenged positivism and the need for causal relationships Causal chains or intervening variables established the unacceptability of positivism 
Essentialist Tradition  associated with reductionism and determinism cause suggests variables that are necessary and sufficient for the effect factor must work all of the time to be a cause observed causal relationships are more likely to be probabilistic than deterministic 
Philosopher of Concomitant Variation  Mill 
Concomitant Variation  key to design is establishing counterfactual implied comparisons sets up elementary comparison with and without treatment 
Mill's Criteria for CauseandEffect Relationships  1) Variables must corelate 2) Time order (cause before effect) 3) Eliminate alternative explanations (the reason control is critical) 
Falsification  from Popper base of knowledge is on ruling out alternative explanations can never prove a theory to be true  but can prove it to be false the more causal variables are ruled out, the more confident we become with the relationship between IV and DV 
Activity theory of Causation  purposefully move X to observe effect on Y the concept of cause implies manipulation started idea of bringing idea into the lab 
Great Contribution of Concomitant Variation  Great Contribution: to realize the comparison of situations where a particular threat to valid inference was/wasn't operating is key to assessing whether the threat may occur for any observed relationship 
Variance of a random variable (or distribution)  the expected, or mean, value of the square of the deviation of that variable from its expected value or mean. 
Statistical Variation  is a measure of the amount of variation within the values of that variable, taking account of all possible values and their probabilities the “currency for scientific exchange” 
The purpose of research is to  maximize variance explained and minimize error variance 
Three kinds of variance  1) experimental 2) extraneous 3) error 
Experimental Variance is associated with  the Main IV 
The goal of experimental variance is to  MAXimize the variation 
Extraneous variance is associated with  the covariates (Zs) 
The goal of extraneous variance is to  CONtrol by transforming all other possible variables into constants 
The goal of error variance if to  MINimize 
MaxConMin  MAXimize the required systemic variance (MAX variance in X) CONtrol “unwanted” variance (CONtrol variance in Zs) MINimize the error variance 
We want to MAXimize  required systematic variance as a way of ensuring internal validity (“the whole point of science”) 
Extraneous variability is  variability that comes from outside the model and is unexplained by the model; it is a change that involved an alteration of a variable that is autonomous (unaffected by the workings of the model) 
the “operational key to a model’s internal validity” (Mill’s 3rd criterion)  CONtrolling extraneous variability 
To ensure internal validity  must control the influence of covariance and the influence of error 
We want to CONtrol  extraneous variability to protect internal validity (the X > Y relationship, does it exist, how strong is it, is it a plus or minus?) we want to pull out those Zs “cleanly”and turn them into constants so we have controlled for the other influences 
Five techniques for CONtrolling the influence of an extraneous variable  1) Eliminate the variable (ex: only study females is gender is an issue) 2) Randomization (best way!) 3) Build the variable into the model 4) Match U/As (“poor man’s substitute”) 5) Statistical control (ex post facto) 
Statistical Control as a Technique for CONtrolling the influence of an extraneous variable  this is a method of data adjustment where you subtract the variance attributable to the control variable from the group means 
We wish to MINimize  error variance which is variability as a result of randomness and are selfcorrecting has little to do “error” but variability due to errors can be a part of the error variance really refers to sources of variability we are not focused on 
Two sources of error variance  1) Specification error (ex: omitted variable bias) 2) Measurement error 
Because systematic variance is due to those variables under investigation and error variance encompasses every other source of variability, the two together equal  the total observed variance 
What lowers error variance?  1) Controlled conditions of the design 2) Measurement reliability 3) High experimental variance 
The greater the reliability of the measurement process, the ____ the error variance  lower 
The larger the experimental variance, the ___the error variance  lower 
Three types of Descriptive Studies  correlational, case study, crosssectional 
Correlational Descriptive Study  uses measures that represent attributes of entire populations (aggregates) 
Advantages and Disadvantages of Correlational Descriptive Study  advantages: easy to get data; measures of association are easy to compute disadvantages: ecological fallacy; inability to control for confounding factors; no comparison group; correlation does not imply causation 
Advantages and Disadvantages of Case Study Descriptive Study  advantages: useful emphasis on description; valuable insights for conceptualization and measurement disadvantages: no variation (n 
CrossSectional Survey  measures are taken on individual nits of analysis within some population at a given point in time 
Advantages and Disadvantages of CrossSectional Survey  advantages: useful for generating variance for statistical analysis disadvantages: must use expost facto statistical adjustments of data; unable to infer causality – time is constant and constants can’t explain variance 
Crosssectional study and time  time held constant (therefore no valid causal inference is justifiable); variation in all variables takes place at one point in time. Offers no real basis for inferring a causal relationship. The basis for the bulk of social science research findings 
Trend study and time  Consecutive crosssectional snapshots of DV across time offers no real basis for inferring a causal relationship can compare groups sharing the same age, but “period” effects are left to vary time varies opening up possibilities for other influences 
Longitudinal Design Types  time permitted to vary. While more costly to execute, only longitudinal designs can provide data capable of demonstrating a causal relationship. 
Panel Study and Time  powerful way to elimiate effects of individual differences (and their correlates) follow individuals in groups across time use Ss as their own controls e.g. pre/posttest designs and timeseries designs 
Cohort Study and Time  Measurements are taken on different units of analysis across time Groups of units of analysis whose timing on a background factor (birth, date of entry into the labor force) are used in sequence The groups themselves are different people 
Temporal Order: Addressing Mill’s Second Criterion  Time allows separation of events, things in space, relative to one another. A huge factor of structured pattern influence. 
Designing with Time (Mill’s second criterion) means  keeping the order of cause preceding effect 
TimeMapping  moving from “pointintime” to “process” and dealing with a discrete point versus a dynamic process. 
Micromediation  the detailed linkage and understanding of the mechanism by which the change generated 
Conventional View of Time Mapping  Regression coefficients are the ‘laws’ of social science but these coefficients usually come from cross sectional designs so the alternative is that we must focus of dynamic processes using longitudinal designs 
Process focus of time mapping  covariation, observed variation of variable over time 
Theory guidance of time mapping  when data should be collected, length of interval between measurements, overall duration of study 
Six Dimensions of the Dynamic Process of Time Mapping  Continuity Magnitude Rate of Change Trend Periodicity Duration 
Continuity  whether a variable has a consistent nonzero value through time 
Temporally recursive regression  shows the lifecycle effect and allows us to see the coefficient in the process of becoming; We are trying to determine the variety of forces influence the DV over time. 
Forwardmoving strategy  anchored starting year  we open the window of time wider year by year keeping the base time the same 
Backward moving strategy  anchored ending year 
Diagonal moving strategy  Looking at a period of time, sequentially, increment on a yearly basis, continued influence of a time period e.g. the 40’s on 
Magnitude  the absolute level of a variable at any time 
Rate of Change  how fast the magnitude changes per unit of time 
Trend  longterm change in magnitude 
Periodicity  the length in time between a common point in a cycle (if one exists) 
Duration  the length of time a variable holds a nonzero value 
The elaboration paradigm address Mill's ___ Criterion and ______ bias.  Mill's Third Criterion; Omitted Variable Bias 
Elaboration of a Model  adding influences into the equation one by one (allows control of more and more of error variance) 
Replication  shows same relationship the new test factor doesn't change the original relationship the test factor may have an independent effect on the DV but we don't know the original relationship found in both partials so can get rid of test factor 
Explananation  The original result goes away in the partials  effect reduced to zero or close to it the third variable is antecedent (Z>X>Y) so the original relationship is spurious 
Six Outcomes of the Elaboration Paradigm  Replication Explanation Interpretation Specification Distorter Suppressor 
Interpretation  The original result goes away in the partials  effect reduced to zero or close to it the third variable is intervening (between) (X>Z>Y) so the original relationship is spurious 
Specification  when the tf introduced, the partials respond differently tf may be either antecedent or intervening 
Suppressor  No original relationship but one emerges in the partials 
Distorter  When tf introduced, the direction of the original relationship is reversed in the partials the tf distorted the original relationship 
Created by:
18801171