Simpson’s Paradox in Psychological Science: A Practical Guide - Kievit - 2013 - Article

Simpson (1951) showed that a statistical relationship observed in a population could be reversed within all the subgroups that make up that population. This has significant implications for medical and social sciences; because, a treatment that may seem effective at the population-level may, in fact, have adverse consequences within each of the population's subgroups. The Simpson's paradox (SP) has been formally analyzed by mathematicians and statisticians. But there hasn't been much work focused on the practical aspects of the SP for empirical science; how might the researchers prevent the paradox, recognize it, and deal with it upon detection?

In this paper they state that (a) SP occurs more frequently than commonly thought, and (b) inadequate attention to SP results in incorrect inferences that may compromise not only the quest for truth, but may also jeopardize public health and policy. 

What is the Simpson's paradox?

Strictly speaking is SP not a paradox but a counterintuitive feature of aggregated data, which may arise when inferences (causal) are drawn across different explanatory levels: from populations to subgroups, or subgroups to individuals etc. 

Pearl (1999) states that SP is unsurprising, 'seeing magnitudes change upon conditionalization is commonplace, and seeing such changes turn into sign reversal, is also not uncommon'. The Simpson's paradox is linked to a lot of statistical challenges and the underlying shared theme of these techniques is that they are concerned with the nature of causal inference. According to Pearl it is the tendency of men to automatically interpret observed associations causally that renders SP paradoxical. To be able to draw conclusions, you must know what the underlying causal mechanism is of the observed patterns, and what data we observe is informative about these mechanisms. 

What is the role of the Simpson's paradox in individual differences?

Literature has documented inter-individual differences in, for example, personality. Cross-sectional patterns of inter-individual differences are often thought to be informative about psychological constructs. The idea that differences between people can be described using these constructs, mean to some that these dimensions play a causal role within individuals. But this kind of inference is not warranted: you can only be sure that a group-level finding generalizes to individuals when the data are ergodic. The dimensions that appear in a covariance structure analysis describe patterns of variation between people. 

A recent study showed that markers are known to differentiate between cultures and social classes did not generalize to capture individual differences within any of the groups. So correlations at one level pose no constraint on correlations at another level. Similarly, two variables may correlate positively across population of individuals, but negatively within each individual, over time.

In the cognitive psychology the direction is reversed within individuals is the speed-accuracy trade-off. The interindividual correlation between speed and accuracy is generally positive, within subjects there is an inverse relationship between speed and accuracy, reflecting differential emphasis in response style strategies. 

What is the survival guide to Simpson's paradox?

The Simpson's paradox occurs in a wide variety of research designs, methods and questions. So, it would be useful to develop means to 'control' or minimize the risk of SP occurring, much like we wish to control instances of other statistical problems such as confounding variables. 

What we can do is considering instances of SP that we are most likely to encounter, and investigate them for characteristic warning signals. The most general 'danger' for psychology is therefore well-defined: we might incorrectly infer that a finding at the level of the group generalizes to subgroups, or to individuals over time. There are certain strategies for three phases of the research process: prevention, diagnosis and treatment of SP. 

How do you prevent the Simpson's paradox?

The first step in addressing SP is to carefully consider when it may arise. The mechanistic inference we propose to explain the data may be incorrect, and this danger arises when we use data at one explanatory level to infer a cause at a different explanatory level. But, when you have absence of top-down knowledge, we are far less well-protected against making incorrect inferences. You have, in essence, a cognitive blind spot within which we are vulnerable to making incorrect inferences. 

When you want to be sure that the relationship between two variables at the group level reflects a causal pattern within individuals over time, the most informative strategy is to experimentally intervene within individuals. When you can model the effect of some manipulation, and therefore rule out SP at the level of the individual, the strongest approach is a study that can assess the effects of an intervention, preferably within individual subjects.