Internal validity
Encyclopedia
Internal validity is the validity
of (causal) inference
s in scientific studies, usually based on experiments as experimental validity
.
A causal inference may be based on a relation when three criteria are satisfied:
In scientific experimental settings, researchers often manipulate a variable (the independent variable
) to see what effect it has on a second variable (the dependent variable) For example, a researcher might, for different experimental groups, manipulate the dosage of a particular drug between groups to see what effect it has on health. In this example, the researcher wants to make a causal inference, namely, that different doses of the drug may be held responsible for observed changes or differences. When the researcher may confidently attribute the observed changes or differences in the dependent variable to the independent variable, and when he can rule out other explanations (or rival hypotheses), then his causal inference is said to be internally valid.
In many cases, however, the magnitude of effects
found in the dependent variable may not just depend on
Rather, a number of variables or circumstances uncontrolled for (or uncontrollable) may lead to additional or alternative explanations (a) for the effects found and/or (b) for the magnitude of the effects found. Internal validity, therefore, is more a matter of degree than of either-or, and that is exactly why research designs other than true experiments may also yield results with a high degree of internal validity.
In order to allow for inferences with a high degree of internal validity, precautions may be taken during the design of the scientific study. As a rule of thumb, conclusions based on correlations or associations may only allow for lesser degrees of internal validity than conclusions drawn on the basis of direct manipulation of the independent variable. And, when viewed only from the perspective of Internal Validity, highly controlled true experimental designs (i.e. with random selection, random assignment to either the control or experimental groups, reliable instruments, reliable manipulation processes, and safeguards against confounding factors) may be the "gold standard" of scientific research. By contrast, however, the very strategies employed to control these factors may also limit the generalizability or External Validity
of the findings.
s cannot be ruled out, rival hypothesis to the original causal inference hypothesis of the researcher may be developed.
During the selection step of the research study, if an unequal number of test subjects have similar subject-related variables there is a threat to the internal validity. For example, a researcher created two test groups, the experimental and the control groups. The subjects in both groups are not alike with regard to the independent variable but similar in one or more of the subject-related variables.
Likewise, extreme outliers on individual scores are more likely to be captured in one instance of testing but will likely evolve into a more normal distribution with repeated testing.
If this attrition is systematically related to any feature of the study, the administration of the independent variable, the instrumentation, or if dropping out leads to relevant bias between groups, a whole class of alternative explanations is possible that account for the observed differences.
For eight of these threats there exists the first letter mnemonic THIS MESS, which refers to the first letters of Testing (repeated testing), History, Instrument change, Statistical Regression toward the mean, Maturation, Experimental mortality, Selection and Selection Interaction.
Validity
In logic, argument is valid if and only if its conclusion is entailed by its premises, a formula is valid if and only if it is true under every interpretation, and an argument form is valid if and only if every argument of that logical form is valid....
of (causal) inference
Inference
Inference is the act or process of deriving logical conclusions from premises known or assumed to be true. The conclusion drawn is also called an idiomatic. The laws of valid inference are studied in the field of logic.Human inference Inference is the act or process of deriving logical conclusions...
s in scientific studies, usually based on experiments as experimental validity
Validity (statistics)
In science and statistics, validity has no single agreed definition but generally refers to the extent to which a concept, conclusion or measurement is well-founded and corresponds accurately to the real world. The word "valid" is derived from the Latin validus, meaning strong...
.
Details
Inferences are said to possess internal validity if a causal relation between two variables is properly demonstrated.A causal inference may be based on a relation when three criteria are satisfied:
- the "cause" precedes the "effect" in time (temporal precedence),
- the "cause" and the "effect" are related (covariation), and
- there are no plausible alternative explanations for the observed covariation (nonspuriousness).
In scientific experimental settings, researchers often manipulate a variable (the independent variable
Independent variable
The terms "dependent variable" and "independent variable" are used in similar but subtly different ways in mathematics and statistics as part of the standard terminology in those subjects...
) to see what effect it has on a second variable (the dependent variable) For example, a researcher might, for different experimental groups, manipulate the dosage of a particular drug between groups to see what effect it has on health. In this example, the researcher wants to make a causal inference, namely, that different doses of the drug may be held responsible for observed changes or differences. When the researcher may confidently attribute the observed changes or differences in the dependent variable to the independent variable, and when he can rule out other explanations (or rival hypotheses), then his causal inference is said to be internally valid.
In many cases, however, the magnitude of effects
Effect size
In statistics, an effect size is a measure of the strength of the relationship between two variables in a statistical population, or a sample-based estimate of that quantity...
found in the dependent variable may not just depend on
- variations in the independent variable,
- the powerStatistical powerThe power of a statistical test is the probability that the test will reject the null hypothesis when the null hypothesis is actually false . The power is in general a function of the possible distributions, often determined by a parameter, under the alternative hypothesis...
of the instruments and statistical procedures used to measure and detect the effects, and - the choice of statistical methods (see: Statistical conclusion validityStatistical conclusion validityStatistical conclusion validity refers to the appropriate use of statistics to infer whether the presumed independent and dependent variables covary...
).
Rather, a number of variables or circumstances uncontrolled for (or uncontrollable) may lead to additional or alternative explanations (a) for the effects found and/or (b) for the magnitude of the effects found. Internal validity, therefore, is more a matter of degree than of either-or, and that is exactly why research designs other than true experiments may also yield results with a high degree of internal validity.
In order to allow for inferences with a high degree of internal validity, precautions may be taken during the design of the scientific study. As a rule of thumb, conclusions based on correlations or associations may only allow for lesser degrees of internal validity than conclusions drawn on the basis of direct manipulation of the independent variable. And, when viewed only from the perspective of Internal Validity, highly controlled true experimental designs (i.e. with random selection, random assignment to either the control or experimental groups, reliable instruments, reliable manipulation processes, and safeguards against confounding factors) may be the "gold standard" of scientific research. By contrast, however, the very strategies employed to control these factors may also limit the generalizability or External Validity
External validity
External validity is the validity of generalized inferences in scientific studies, usually based on experiments as experimental validity....
of the findings.
Confounding
A major threat to the validity of causal inferences is Confounding: Changes in the dependent variable may rather be attributed to the existence or variations in the degree of a third variable which is related to the manipulated variable. Where Spurious relationshipSpurious relationship
In statistics, a spurious relationship is a mathematical relationship in which two events or variables have no direct causal connection, yet it may be wrongly inferred that they do, due to either coincidence or the presence of a certain third, unseen factor In statistics, a spurious relationship...
s cannot be ruled out, rival hypothesis to the original causal inference hypothesis of the researcher may be developed.
Selection (bias)
Selection bias refers to the problem that, at pre-test, differences between groups exist that may interact with the independent variable and thus be 'responsible' for the observed outcome. Researchers and participants bring to the experiment a myriad of characteristics, some learned and others inherent. For example, sex, weight, hair, eye, and skin color, personality, mental capabilities, and physical abilities, but also attitudes like motivation or willingness to participate.During the selection step of the research study, if an unequal number of test subjects have similar subject-related variables there is a threat to the internal validity. For example, a researcher created two test groups, the experimental and the control groups. The subjects in both groups are not alike with regard to the independent variable but similar in one or more of the subject-related variables.
History
Events outside of the study/experiment or between repeated measures of the dependent variable may affect participants' responses to experimental procedures. Often, these are large scale events (natural disaster, political change, etc.) that affect participants' attitudes and behaviors such that it becomes impossible to determine whether any change on the dependent measures is due to the independent variable, or the historical event.Maturation
Subjects change during the course of the experiment or even between measurements. For example, young children might mature and their ability to concentrate may change as they grow up. Both permanent changes, such as physical growth and temporary ones like fatigue, provide "natural" alternative explanations; thus, they may change the way a subject would react to the independent variable. So upon completion of the study, the researcher may not be able to determine if the cause of the discrepancy is due to time or the independent variable.Repeated testing
Repeatedly measuring the participants may lead to bias. Participants may remember the correct answers or may be conditioned to know that they are being tested. Repeatedly taking (the same or similar) intelligence tests usually leads to score gains, but instead of concluding that the underlying skills have changed for good, this threat to Internal Validity provides good rival hypotheses.Instrument change
The instrument used during the testing process can change the experiment. This also refers to observers being more concentrated or primed. If any instrumentation changes occur, the internal validity of the main conclusion is affected, as alternative explanations are readily available.Regression toward the mean
This type of error occurs when subjects are selected on the basis of extreme scores (one far away from the mean) during a test. For example, when children with the worst reading scores are selected to participate in a reading course, improvements at the end of the course might be due to regression toward the mean and not the course's effectiveness. If the children had been tested again before the course started, they would likely have obtained better scores anyway.Likewise, extreme outliers on individual scores are more likely to be captured in one instance of testing but will likely evolve into a more normal distribution with repeated testing.
Mortality/differential attrition
This error occurs if inferences are made on the basis of only those participants that have participated from the start to the end. However, participants may have dropped out of the study before completion, and maybe even due to the study or programme or experiment itself. For example, the percentage of group members having quit smoking at post-test was found much higher in a group having received a quit-smoking training program than in the control group. However, in the experimental group only 60% have completed the program.If this attrition is systematically related to any feature of the study, the administration of the independent variable, the instrumentation, or if dropping out leads to relevant bias between groups, a whole class of alternative explanations is possible that account for the observed differences.
Selection-maturation interaction
This occurs when the subject-related variables, color of hair, skin color, etc., and the time-related variables, age, physical size, etc., interact. If a discrepancy between the two groups occurs between the testing, the discrepancy may be due to the age differences in the age categories.Diffusion
If treatment effects spread from treatment groups to control groups, a lack of differences between experimental and control groups may be observed. This does not mean, however, that the independent variable has no effect or that there is no relationship between dependent and independent variable.Compensatory rivalry/resentful demoralization
Behaviour in the control groups may alter as a result of the study. For example, control group members may work extra hard to see that expected superiority of the experimental group is not demonstrated. Again, this does not mean that the independent variable produced no effect or that there is no relationship between dependent and independent variable. Vice-versa, changes in the dependent variable may only be effected due to a demoralized control group, working less hard or motivated, not due to the independent variable.Experimenter bias
Experimenter bias occurs when the individuals who are conducting an experiment inadvertently affect the outcome by non-consciously behaving differently to members of control and experimental groups. It is possible to eliminate the possibility of experimenter bias through the use of double blind study designs, in which the experimenter is not aware of the condition to which a participant belongs.For eight of these threats there exists the first letter mnemonic THIS MESS, which refers to the first letters of Testing (repeated testing), History, Instrument change, Statistical Regression toward the mean, Maturation, Experimental mortality, Selection and Selection Interaction.
See also
- External validityExternal validityExternal validity is the validity of generalized inferences in scientific studies, usually based on experiments as experimental validity....
- Construct validityConstruct validityIn science , construct validity refers to whether a scale measures or correlates with the theorized psychological scientific construct that it purports to measure. In other words, it is the extent to which what was to be measured was actually measured...
- Content validityContent validityIn psychometrics, content validity refers to the extent to which a measure represents all facets of a given social construct. For example, a depression scale may lack content validity if it only assesses the affective dimension of depression but fails to take into account the behavioral dimension...
- Statistical conclusion validityStatistical conclusion validityStatistical conclusion validity refers to the appropriate use of statistics to infer whether the presumed independent and dependent variables covary...
- Validity in statisticsValidity (statistics)In science and statistics, validity has no single agreed definition but generally refers to the extent to which a concept, conclusion or measurement is well-founded and corresponds accurately to the real world. The word "valid" is derived from the Latin validus, meaning strong...
- Ecological validityEcological validityEcological validity is a form of validity in a research study. For a research study to possess ecological validity, the methods, materials and setting of the study must approximate the real-life situation that is under investigation. Unlike internal and external validity, ecological validity is not...