Critical thinking
Article: Nosek, Spies, & Motyl, (2012)
Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability

Abstract

An academic scientist’s professional success depends on publishing.

• Publishing norms emphasize novel, positive results.
• Disciplinary incentives encourage design, analysis, and reporting decisions that elicit positive results and ignore negative results .
• When incentives favor novelty over replication, false results persists in the literature unchallenged, reducing efficiency in knowledge accumulation.

This article develops strategies for improving scientific practices and knowledge accumulation that account for ordinary human motivations and biases.

A true story of what could have been

Incentives for surprising, innovative results are strong in science.

• Science thrives by challenging prevailing assumptions and generating novel ideas and evidence that push the field in new directions.

Problem: the incentives for publishable results can be at odds with the incentives for accurate results. This produces a conflict of interest.

• The conflict may increase the likelihood of design, analysis, and reporting decisions that inflate the proportion of false results in the published literature.

The solution requires making incentives for getting it right competitive with the incentives for getting it published.

How evaluation criteria can increase the false result rate in published science

Publishing is the ‘very heart of modern academic science, at levels ranging from the epistemic certification of scientific thought to the more personal labyrinths of job security, quality of life and self esteem’.

With an intensely competitive job marked, the demands on publication might seem to suggest a specific objective for the early-career scientists: publish as many articles as possible in the most prestigious journals that will accept them.

Some things are more publishable than others

Even if a researcher conducts studies competently, analyses the data effectively, and writes the results beautifully, there is not guarantee that the report will be published.
Part of the process is outside the researcher’s control.

High demand for limited space means that authors must strive to meet all publishing criteria so that an editor will do the unusual act of accepting the manuscript.

• the success in publishing is partly a function of social savvy of knowing what is publishable and empirical savvy in obtaining publishable results.

A disconnect between what is good for scientists and what is god for science

On its own, the fact that publishing is essential to success is just a fact of the trade.

• Conducting more high-impact research defines better scientists.
  • The research must be published to have impact.

But, publishing is also the basis of a conflict of interest between personal interests and the objective of knowledge accumulation.

• Published and true are not synonyms.
  • To the extent that publishing itself is rewarded, then it is in scientists’ personal interests to publish, regardless of whether the published findings are true.

The present authors have

• Accuracy motives: to learn and publish true things about human nature
• Professional motives: to succeed and thrive professionally.

Because we have directional goals for success, we are likely to bring to bear motivated reasoning to justify research decisions in the name of accuracy, when they are actually in service of career advancement.

Motivated reasoning is particularly influential when the situation is complex, the available information is ambiguous, and legitimate reasons can be generated for multiple courses of action.

• can occur without intention
• we are more likely to be convinced that our hypothesis is true, accepting uncritically when it is confirmed and scrutinizing heavily when it is not.
• with flexible analysis options, we are more likely to find the one that produces a more publishable pattern of results to be more reasonable and defensible than others.
• once we obtain an unexpected result, we are likely to reconstruct our histories and perceive the outcome as something that we could have anticipate all along.

Even if we resits the reasoning biases in the moment, after a few months, we forget the details.

• instead, we might remember the gist of what the study was and what we found.

Forgetting the details proves an opportunity for re-imagining the study purpose and results to recall and understand them in their best light.

Novelty and positive results are vital for publish-ability but nor for truth

Direct replication of another’s study procedures to confirm the results is uncommon in the social sciences.

Publishing a result does not make it true.
Many published results have uncertain truth value.
Replication is a means of increasing the confidence in the truth value of a claim.

• its dismissal as a waste of space incentivizes novelty over truth.
• as a consequence, when a false result gets into the published literature, it is difficult to expel.
• there is little reinforcement for conducting replications to affirm or reject the validity of prior evidence and few consequences for getting it wrong.

The dominant model of null hypotheses significance testing has become a de facto criterion for publishing.

• achieving a positive result does not mean that the effect is true, nor does it indicate the probability of its truth.

Psychologists perceive a bias against negative (null) results and are less likely to continue perusing or report negative results. Journals are less likely to publish negative as compared to positive results.

• Negative results are less likely to appear in literature.

The demands for novelty and positive results create incentives for:

• generating new ideas rather than pursuing additional evidence for or against ideas suggested previously
• reporting positive results and ignoring negative results
• pursuing design, reporting, and analysis strategies that increase the likelihood of obtaining a positive result in order to achieve publish-ability.

Practices that can increase the proportion of false results in the published literature

Other contributors have detailed a variety of practices that can increase publish-ability but might simultaneously decrease validity

Practices that are justifiable sometimes but can also increase the proportion of published false results:

• Leveraging chance by running many low-powered studies, rather than a few high-powered ones
• Uncritically dismissing ‘failed’ studies as pilot test or because of methodological flaws but uncritically accepting ‘successful’ studies as methodological sound
• Selectively reporting studies with positive results and not studies with negative results, or selectively reporting ‘clean’ results
• Stopping data collection as soon as reliable effect is obtained
• Continuing data collection until a reliable effect is obtained
• Including multiple independent or dependent variables and reporting the subset that ‘worked’
• Maintaining flexibility in design and analytic models, including the attempt of a variety of data exclusion or transformation methods, and reporting a subset.
• Reporting a discovery as if it has been the result of a confirmatory test
• Once a reliable effect is obtained, nor doing a direct replication.

The lack of interest in replication is striking given its centrality to science.

• the scientific method differentiates itself from other approaches by publicly disclosing the basis of evidence for a claim.

Other scientists must be able to independently replicate and verify, or dis-confirm the original scientist’s results.

• this allows scientists to work independently toward a shared objective without relying on accuracy or trust in any single source.

In principle, open haring of methodology means that the entire body of scientific knowledge can be reproduced by anyone.

• this makes replication a central principle of the scientific method.

But they are seldom published because they are not novel enough.

The consequences of the publish-ability-improving practices can be severe.

Strategies that are not sufficient to stop the proliferation of false results

False effects interfere with knowledge accumulation.

• If common scientific practices are increasing the rate of false effects, then changing some practices could improve efficiency in scientific progress.

These innovation are not sufficient to address the proliferation of false effects

Conceptual replication

Whereas a direct replication is able to produce facts, a conceptual replication may produce understanding.

• Because features of the original design are changed deliberately, conceptual replication is used only to confirm (and abstract) the original result, not to disconfirm it.
  A failed conceptual replication is dismissed as not testing the original phenomenon.

As such, using conceptual replication as a replacement for direct replication is the scientific embodiment of confirmation bias.

The mythology of science as self-correcting

False effects can remain for decades, slowly fading or continuing to inspire and influence new research.
Even when it becomes known that an effect is false, retraction of the original result is very rare.
Researchers who do not discover the corrective knowledge may continue to be influenced by the original, false result.

The truth will win eventually, but we are not content to wait.

Journals devoted to publishing replications or negative results

Defining a journal based on negative results or replications is self-defining it as a low importance outlet.

Education campaigns emphasizing the importance of replication and reporting negative results

It has not worked yet.

Publishing practices are hard to change because innovative research is more important than replication and negative results.
Innovation is the key driver of scientific progress.

Increasing expectations of reviewers to catch motivated reasoning and other signs of false results

Rejected because:

• peer reviewers are volunteers, they already work hard for no reward
• reviewing is hard work, even diligent reviewers miss lots of errors
• peer reviewers review only the summary report of the research, not the research itself.

Raising the barrier for publication

The standards for publication are already extremely high, especially in the social and behavioural sciences.

Requiring replication of everything could stifle risk taking and innovation.

Strategies that will accelerate the accumulation of knowledge

The key for improving the efficiency of knowledge accumulation is to capitalize on existing motivation the be accurate and to reduce the emphasis on publication itself as the mechanism of achievement and advancement.

Promoting and rewarding paradigm-driven research

Paradigm-driven research can be used for both confirming and disconfirming prior results.

• accumulates knowledge by systematically altering a procedure to investigate a question or theory, rather than varying many features of the methodology, by design or by accident.

This offers an opportunity to incorporate replication and extension into a single experimental design.

• balances novelty and replication by building new knowledge using existing procedures.

Effective use of this approach requires development of standards, sharing and reuse of materials, and deliberate alteration of design rather than wholesale reinvention.
It is easy to do more paradigm-driven research if authors make their paradigms available to others.

Primary risk: research questions can evolve to being about the method itself rather than the theory that the method is intended to address.

Author, reviewer, and editor checklists

Checklists are an effective means of improving the likelihood that particular behaviours are performed and performed accurately.

• it is easy to conceive of statistical and disclosure checklists for authors and editorial teams.

Checklists can ensure disclosures of obvious items that are sometimes forgotten.
They can also define best practices and methodological standards for domain-specific applications.

Why checklists?

• key information is left out frequently
• advisable methodological practices are not identified ‘naturally’ or systematically in the review process.

Challenging mindsets that sustain the dysfunctional incentives

Metrics to identify what is worth replicating

Even if valuation of replication increased it is not feasible, or advisable, to replicate everything.
Solution: metrics for identifying replication value, what effects are more worthwhile to replicate than others.

Crowd sourcing replication efforts

Individual scientists and laboratories may be interested in conducting replications but do not have sufficient resources available for them.
It may be easier to conduct replications by crowd sourcing them with multiple contributors.

Some important findings are difficult to replicate because of resource constrains.
Feasibility could be enhanced by spreading the data collection effort across multiple laboratories.

Journals with peer review standards focused on the soundness, not importance, of research

The basis of rejection for much research is that it does not meet the criterion of being sufficient ‘important’ for the journal considering it. Even if the research is sound and reported effectively.

With a publishing model focused on soundness, negative results and replications are more publishable, and the journal identity is not defined as publishing research that is otherwise unpublishable.

Lowering or removing the barrier for publication

To discard publishing as a meaningful incentive.
Make it trivial to publish.

Postpublication peer review can separate gatekeeper and evaluator.

• then peer review operates solely as an evaluation mechanism.

The priorities in the peer review process would shift from assessing whether the manuscript should be published to whether the ideas should be taken seriously and how they can be improved.
This would remove a major barrier to publishing replications and negative results if and when they occur.

This change would alter the mindset that publication is the end of the research process.

The ultimate solution: opening data, materials, and workflow

Openness provides scientists with confidence in the claims and evidence provided by other scientists.

Open data

With the massive growth in data and increased ease of making it available, calls for open data as a standard practice are occurring across all of the sciences.

Errors can occur in data coding, data cleaning, data analysis and result reporting. None of those can be detected with only the summary report.

Making data openly available increases the likelihood of finding and correcting errors and ultimately improving reported results.

• also, it improves the potential for aggregation of raw data for research synthesis, it presents opportunities for applications with the same data that may not have been pursued by the original authors and it creates a new opportunity for citation credit and reputation building.

Open methods and tools

Open methods facilitate confirmation, extension, critique, improvement, adaptation.
Facilitates replication and paradigm-driven research.

Open workflow

Public documentation of a laboratory’s research process makes bad practices easier to detect and could reduce the likelihood that they occur at all.