“LeBel & Peters (2011). Fearing the future of empirical psychology: Bem’s (2011) evidence of psi as a case study of deficiencies in modal research practice.” - Article summary

Psi refers to the anomalous retroactive influence of future events on an individual’s current behaviour. There are three important deficiencies in modal research practice: an overemphasis on conceptual replication (1), insufficient attention to verifying the integrity of measurement instruments and experimental procedures (2) problems with the implementation of null hypothesis testing (3).

The interpretation bias refers to a bias towards interpretations of data that favour a researcher’s theory. A potential consequence of this is an increased risk of reported false positives and a disregard of true negatives. The knowledge system of psychology consists of theory relevant beliefs (1), this is about the mechanisms that produce behaviour and method-relevant beliefs (2), this is about the procedures through which data is obtained.

Deficiencies in modal research practice bias systematically bias the interpretation of confirmatory data as theory relevant (1) and the interpretation of disconfirmatory data as method relevant (2).

Central beliefs are beliefs on which many other beliefs depend. Conservatism refers to choosing the theoretical explanation consistent with the data that requires the least amount of restructuring of the existing knowledge system.

If method-relevant beliefs are central in a knowledge system, it becomes more difficult to blame methodology related errors for disconfirmatory results. If theory-relevant beliefs become central, it poses the threat of becoming a logical assumption. A hypothesis under test should be described in a way that is falsifiable and not logically necessary.

An overemphasis on conceptual replication at the expense of direct replication weakens method-relevant beliefs in the knowledge system. A statistical significant result is often followed by a conceptual replication. A failure of the conceptual replication leads to the question whether the negative result was due to the falsity of the underlying theory or to methodological flaws introduced by changes in conceptual replication.

The failure to verify the integrity of measurement instruments and experimental procedures weakens method-relevant beliefs and leads to ambiguity in the interpretation of results. The null hypothesis can be viewed as a straw man, as two identical populations are almost not possible. Basing theory choices on null hypothesis significance tests detaches theories from the broader knowledge system.

In order to overcome the flaws of the modal research practice, method-relevant beliefs must be strengthened. There are three ways in order to do this:

1. Stronger emphasis on direct replication
   A direct replication leads to greater confidence in the results. They are necessary to ensure that an effect is real.
2. Verify integrity of methodological procedures
   Method-relevant beliefs are more difficult to reject if the integrity of methodological procedures are verified and this leads to a less ambiguous interpretation of results. This includes routinely checking the internal consistency of the scores of any measurement instrument that is used. This includes the use of objective markers of instruction comprehension.
3. Use stronger forms of NHST
   The null hypothesis should be a theoretically derived point value of the focal variable, instead of a hypothesis of no difference.