- What is the problem?
- What are measures that can be implemented when performing research? (methods)
- What are measures that can be implemented when communicating research? (reporting and dissemination)
- What are measures that can be implemented to support verification of research? (reproducibility)
- What are measures that can be implemented when evaluating research? (evaluation)
- What role do incentives play?
- Conclusion
This paper argues for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation, and incentives.
What is the problem?
Scientific creativity is characterized by identifying novel and unexpected patterns in data. A major challenge is to be open to new and important insights while avoiding being misled by our tendency to see structures in randomness. Apophenia (seeing patterns in random data), confirmation bias (focusing on evidence that’s in line with our expectations/favoured explanation), and hindsight bias (seeing an event as having been predictable after it has occurred) combined can lead us to false conclusions.
If several potential analytic pipelines can be applied to high-dimensional data, false-positives are highly likely.
What are measures that can be implemented when performing research? (methods)
Protecting against cognitive biases
Effective solution to diminish self-deception and unwanted biases is blinding. Preregistration is an effective form of blinding because the data doesn’t exist and outcomes aren’t yet known.
Improving methodological training
Threats to the robustness of science could be improved by statistical training. Training in research practice that can protect against cognitive biases and the influence of distorted incentives is most important. Without formal requirements for continuing education, the best solution may be developing educational resources that are accessible, easy-to-digest and immediately and effectively applicable to research (e.g. brief, web-based modules on specific topics).
Implementing independent methodological support
Many clinical trials have multidisciplinary trial steering committees to give advice and oversee the design and conduct of a trial. The need for these committees came from the fact that financial conflicts of interest could exist in clinical trials (e.g. sponsors may be companies manufacturing the product being tests and (un)intentionally influence design, analysis, and interpretation). Including independent researchers could alleviate these influences.
Encouraging collaboration and team science
Solution to ‘lack of resources to improve statistical power’. Distributed collaboration across study sites facilitates high-powered designs and more potential for testing generalizability (as opposed to relying on the limited resources of single investigators).
What are measures that can be implemented when communicating research? (reporting and dissemination)
Promoting study preregistration
Can include the registration of a basic study design as well as a detailed pre-specification of study procedures, outcomes, and analysis plan. This was introduced to solve two problems: publication bias and analytical flexibility (outcome switching).
- Publication bias: file drawer problem, the fact that more studies are conducted than published.
- Outcome switching refers to changing the outcomes of interest in the study depending on the observed result.
Improving the quality of reporting
Preregistration improves discoverability of research, but this does not guarantee usability. Improving the quality and transparency in reporting research is necessary to address this. The Transparency and Openness Promotion (TOP) guidelines offer standards as a basis for journals and funders to incentivize or require more transparency in planning and reporting research.
Registered reports (RR) – initiative to eliminate various forms of bias in hypothesis-driven research, specifically evaluation of a study based on the results. RR’s divide peer review processing in two stages, before and after results are known. First reviewers assess a detailed protocol that includes study rationale, procedure, and an analysis plan. Publication of study outcomes are guaranteed if the authors adhere to the approved protocols, meet pre-specified checks, and conclusions are appropriately evidence-bound. They prevent publication bias by accepting articles before results are known. They also neutralize P hacking by knowing the hypotheses and analysis plans in advance.
- Main objective against RR’s – format limits exploration or creativity by making authors follow a pre-specified methodology. But they place no restrictions on creative analysis practices or serendipity.
Authors can report the outcomes of any unregistered exploratory analyses, as long as those tests are clearly labelled as post-hoc.
What are measures that can be implemented to support verification of research? (reproducibility)
Promoting transparency and open science
How credible scientific claims are based on the evidence supporting them, which includes methodology applied, data acquired, and process of methodology implementation, data analysis, and outcome interpretation.
Open science is the process of making the content and process of producing evidence and claims transparent and accessible to others.
- There are barriers to meet these ideals, including vested financial interested (scholarly publishing), and few incentives for researchers to pursue open practices.
- Commercial and non-profit organizations are building new infrastructures like the Open Science Framework to make transparency easy and desirable for researchers.
What are measures that can be implemented when evaluating research? (evaluation)
Diversifying peer review
Pre-and-post-publication peer review mechanisms accelerate and expand the evaluation process. Sharing preprints enables researchers to get quick feedback on their work from a diverse community instead of waiting months for a few reviews in the conventional, closed peer review process.
Data sharing includes sharing data in public repositories, offering advantages in terms of accountability, data longevity, efficiency, and quality (reanalysis could catch crucial mistakes or fabrication).
- Badges acknowledging open science practices – Center for Open Sciences suggested journals to assigned badges to articles with open data (and other open practices like preregistration and open materials).
- The Peer Reviewers’ Openness Initiative – researchers signing this initiative pledge commit to not offer comprehensive review for any manuscript that doesn’t make its data publicly available without clear justification.
- Requirements from funding agencies – NIH intends to make public access to digital scientific data the standard for all NIH-funded research. NSF requires submission of a data-management plan outlining how data will be stored and shared.
What role do incentives play?
Publication is the currency of academic sciences and increases likelihood of employment, funding, promotion, and tenure. Positive, novel, clean results are more likely to be published than negative results, replications and results with loose ends. Consequently, researchers are incentivized to produce the former, even at the cost of accuracy. Ultimately, incentives increase the likelihood of false positives in published literature. Shifting the incentives offers a chance to increase credibility and reproducibility in published results. There will always be incentives for innovative outcomes, but there should be incentives and rewards for transparent and reproducible research.
Conclusion
Challenges to reproducible science are systemic and culture, but that does not mean they can not be met. The measures described offer practical and achievable steps to improve rigor and reproducibility.
Join with a free account for more service, or become a member for full access to exclusives and extra support of WorldSupporter >>
Contributions: posts
Spotlight: topics
Online access to all summaries, study notes en practice exams
- Check out: Register with JoHo WorldSupporter: starting page (EN)
- Check out: Aanmelden bij JoHo WorldSupporter - startpagina (NL)
How and why use WorldSupporter.org for your summaries and study assistance?
- For free use of many of the summaries and study aids provided or collected by your fellow students.
- For free use of many of the lecture and study group notes, exam questions and practice questions.
- For use of all exclusive summaries and study assistance for those who are member with JoHo WorldSupporter with online access
- For compiling your own materials and contributions with relevant study help
- For sharing and finding relevant and interesting summaries, documents, notes, blogs, tips, videos, discussions, activities, recipes, side jobs and more.
Using and finding summaries, notes and practice exams on JoHo WorldSupporter
There are several ways to navigate the large amount of summaries, study notes en practice exams on JoHo WorldSupporter.
- Use the summaries home pages for your study or field of study
- Use the check and search pages for summaries and study aids by field of study, subject or faculty
- Use and follow your (study) organization
- by using your own student organization as a starting point, and continuing to follow it, easily discover which study materials are relevant to you
- this option is only available through partner organizations
- Check or follow authors or other WorldSupporters
- Use the menu above each page to go to the main theme pages for summaries
- Theme pages can be found for international studies as well as Dutch studies
Do you want to share your summaries with JoHo WorldSupporter and its visitors?
- Check out: Why and how to add a WorldSupporter contributions
- JoHo members: JoHo WorldSupporter members can share content directly and have access to all content: Join JoHo and become a JoHo member
- Non-members: When you are not a member you do not have full access, but if you want to share your own content with others you can fill out the contact form
Quicklinks to fields of study for summaries and study assistance
Main summaries home pages:
- Business organization and economics - Communication and marketing -International relations and international organizations - IT, logistics and technology - Law and administration - Leisure, sports and tourism - Medicine and healthcare - Pedagogy and educational science - Psychology and behavioral sciences - Society, culture and arts - Statistics and research
- Summaries: the best textbooks summarized per field of study
- Summaries: the best scientific articles summarized per field of study
- Summaries: the best definitions, descriptions and lists of terms per field of study
- Exams: home page for exams, exam tips and study tips
Main study fields:
Business organization and economics, Communication & Marketing, Education & Pedagogic Sciences, International Relations and Politics, IT and Technology, Law & Administration, Medicine & Health Care, Nature & Environmental Sciences, Psychology and behavioral sciences, Science and academic Research, Society & Culture, Tourisme & Sports
Main study fields NL:
- Studies: Bedrijfskunde en economie, communicatie en marketing, geneeskunde en gezondheidszorg, internationale studies en betrekkingen, IT, Logistiek en technologie, maatschappij, cultuur en sociale studies, pedagogiek en onderwijskunde, rechten en bestuurskunde, statistiek, onderzoeksmethoden en SPSS
- Studie instellingen: Maatschappij: ISW in Utrecht - Pedagogiek: Groningen, Leiden , Utrecht - Psychologie: Amsterdam, Leiden, Nijmegen, Twente, Utrecht - Recht: Arresten en jurisprudentie, Groningen, Leiden
JoHo can really use your help! Check out the various student jobs here that match your studies, improve your competencies, strengthen your CV and contribute to a more tolerant world
1231 |
Add new contribution