“Furr & Bacharach (2014). Estimating practical effects: Binomial effect size display, Taylor-Russell tables, utility analysis and sensitivity / specificity.” – Article summary

Validity refers to the degree to which evidence and theory support the interpretations of test scores entailed by the proposed uses (e.g. to what degree does it measure what it is supposed to measure). Items of a test itself cannot be valid or invalid, only the interpretations can be valid or invalid.

Validity is a property of the interpretation (1), it is a matter of degree (2) and the validity of a test’s interpretation is based on evidence and theory (3). Validity influences the accuracy of our understanding of the world, as research conclusions are based on the validity of a measure.

Construct validity refers to the degree to which test scores can be interpreted as reflecting a particular psychological construct. Face validity refers to the degree to which a measure appears to be related to a specific construct, in the judgement of nonexperts, test takers and representatives of the legal system. Convergent validity refers to the degree to which test scores are correlated with tests of related constructs.

Validity is important for the accuracy of our understanding of the world (1), decisions on societal level (e.g. laws based on ‘invalid’ research) and decisions on individual level (3) (e.g. college admissions).

The validity of test score interpretation depends on five types of evidence: test content (1), consequences of use (2), association with other variables (3), response processes (4) and internal structure (5).

Test content can be seen as content validity. There are two threats to content validity:

1. A test including construct-irrelevant content
   The inclusion of content that is not relevant to the construct of interest reduces validity.
2. Construct underrepresentation
   A test should include the full range of content that is relevant to the construct.

Construct underrepresentation can be constrained by practical issues (e.g. time of a test). The internal structure of a test refers to the way the parts of a test are related to each other. There should be a proper match between the actual internal structure of a test and the internal structure a test should have. The internal structure can be examined through the correlations among items in the test and among the subscales in the test. This can be done using factor analysis.

Factor analysis helps to clarify the number of factors within a set of items (1), reveals the associations among the factors within a multidimensional test (2) and identifies which items are linked to which factors (3). Factors are dimensions of the test.

Response processes refers to the match between the psychological processes that respondents actually use when completing a measure and the processes that they should use.

In order to assess validity, the association with other variables (e.g. happiness and self-esteem) should be assessed. If a positive relationship is to be expected between two variables, then, in order for the interpretation of a measure to be valid, this relationship needs to exist. The association with other variables involves the match between a measure’s actual associations with other measures and the associations that a test should have with other measures.

Discriminant evidence refers to the degree to which test scores are uncorrelated with tests of uncorrelated constructs. Concurrent validity evidence refers to the degree to which test scores are correlated with other relevant variables that are measured at the same time as the primary test of interest. Predictive validity evidence refers to the degree to which test scores are correlated with other, relevant variables that are measured at a future point in time.

Value judgements have influences on the scientific process. Consequential validity states that these influences should be recognized and evaluated as clearly as possible in the testing context.

There are three alternative perspectives on validity:

1. Criterion validity
   This perspective emphasizes the conceptual meaning or interpretation of test scores. It refers to the degree to which test scores can predict specific criterion variables.
2. Inductive approach
   This perspective emphasizes learning what the test scores mean rather than testing hypotheses about test scores.
3. Connection between tests and psychological constructs
   This perspective states that a test is a valid measure of a construct if the intended construct truly influences respondents’ performance on the test.

Reliability refers to the degree to which differences in test scores reflect differences among people in their level of the trait that affects test scores. Reliability is a property of test responses. In other words, it means that different people score the same score in different times on the same test.