Why Summaries of Research on Psychological Theories Are Often Uninterpretable – Meehl - 1990 - Article

What is the scientific method?

The scientific method is an empirical and sceptical approach to asking questions and obtaining answers.

Good habits for a researcher include enthusiasm, open-mindedness, common sense, role-taking ability, inventiveness, confidence in own judgement, consistency and care for details, ability to communicate, and being honest.

What is the issue in psychological research?

Evidence shows a deficiency in most social sciences to match the growth and theoretical integration that characterizes the history of more successful scientific disciplines. Meehl presupposes that, with certain exceptions, theories in ‘soft areas’ of psychology tend to go through periods of initial enthusiasm leading to large amounts of empirical investigation with ambiguous overall results. This is followed by various kinds of amendment and the generation of ad hoc hypotheses. In the long run, experiments lose interest instead of theories actually being falsified. “They never die; they just slowly fade away.”

This discussion constitutes research that shares three properties: (a) theories in ‘soft areas’, (b) data correlational, and (c) positive findings consisting of refuting the null hypothesis. Soft areas could include counselling, personality theory, and social psychology.

What is Meehl’s thesis?

“Null hypothesis testing of correlational predictions from weak substantive theories in soft psychology is subject to the influence of ten obfuscating factors whose effects are usually (1) sizeable, (2) opposed, (3) variable, and (4) unknown. The net effect of these ten obfuscating influences is that the usual research literature review is almost uninterpretable.”

The derivation of observational conditional

Theories, taken with a statement of conditions, entail relations between single observations (particulars). The derivation of a prediction about observational facts involves a conjunction of several premises.

• Derivation of observational conditional: T – A1 – A2 – Cp – Cn à (O1 É O2)
  • T: substantive theory of interest
  • A1, A2…: one or more auxiliary theories which aren’t the main focus of the investigator’s interest.
  • Cp: ceteris paribus clause, a negative statement not formulated with concrete content like an auxiliary but states ‘other things being equal’.
  • Cn: statements about the experimental ‘conditions’.
  • O1 & O2: observational statements.

Meehl’s ten obfuscating factors

The combined operation of the ten factors makes it impossible to tell what a score of statistical significance in research proves about a theory’s plausibility.

1. Loose derivation chain

There are very few tight derivation chains in most areas of psychology and almost none in soft psychology. Some parts of the chain are well explained, while others are vague or not explicitly stated. Meehl states that if your observation 2 doesn’t follow observation 1 as expected, you won’t know why it happened and which part of the derivation chain was problematic.

2. Problematic auxiliary theories

Auxiliary theories are explicitly stated. Sometimes it happens in soft psychology that each auxiliary theory is almost as problematic as the main theory we’re testing. When there are multiple problematic auxiliary theories, the joint probability that they have could be lower than the prior probability of the theory of interest. We intended to investigate T, but the logical structure leaves us not knowing whether the prediction failed because our T was false or because one or more of the conjoined statements (A1, A2, Cp, or Cn) were false. – this reasoning applies to sections 3 and 4 as well.

3. Problematic ceteris paribus clause

Ceteris paribus means ‘everything else being equal’, but everything else isn’t always equal. This clause means that when we test a theory by showing a statistical relationship, while individuals may vary in certain factors that we have not controlled but allowed to vary, the alleged causal influence doesn’t have a significant additional effect operating in a direction opposed to our theory. There is usually a problem of control in the study. There are some situations where we can’t or don’t know how to control for certain factors, which could to some extent counteract the induced state we’re trying to test.

4. Experimenter error

Experimenter mistakes in manipulation. Includes errors due to biases, expectancies etc. on the part of the experimenter (or others involved on the input side of a testing process like research assistants). The experimenter may be enthusiastic about their theory and, without consciously intending, slant results into their favour, e.g. by the way they interact with their subjects/keenness.

5. Inadequate statistical power

Most tests in psychology can be expected to fail because of insufficient statistical power (usually due to small sample sizes) to detect a ‘real effect’.

6. Crud factor

In social sciences, everything correlates to some extent with everything else. Any measured trait is some function of a list of causal factors in the genes and history of the individual. Genetic and environmental factors are also known from empirical research to be correlated. 

7. Pilot study

A pilot study essentially duplicates a main study. A true pilot study is a mini version of a main study with a few minor improvements perhaps. Pilot studies have two aims, firstly they try to see whether an effect exists or not (to then decide whether to pursue the research to a larger extent). Secondly, they try to get a rough idea of the relationship between a mean difference and the approximate variability as a basis for inferring the number of cases that would be necessary to achieve statistical significance. If nothing significant is found, the pilot will be discarded and not published (because no effect = evidence against a theory). This means there’s a large collection of data missing from the collective understanding of the human psyche.

8. Selective bias in submitting reports

A researcher is more likely to submit a study if significant results were found. This comes partly from thinking that a null result ‘doesn’t prove much’. This creates bias in the literature available to the population.

9. Selective editorial bias

People who have been editors or referees say the same thing as investigators: they’re somewhat more inclined to publish a clear finding of a refuted null hypothesis than one that simply fails to show a trend. There’s a failure to report everything. Non-significant results are deemed to say that ‘there’s nothing here’.

10. Detached validation claim for psychometric instruments

Testing should be done with valid instruments. Sometimes, theories and instruments are validated at the same time. This isn’t good practice because you should establish validity of an instrument before you use it.