Science and Ethics in Conducting, Analyzing and Reporting Psychological Research – Rosenthal - 1994 - Article

The article discusses scientific and ethical issues relevant to conducting psychological research. Looking at considerations of research design, procedures, and recruitment of human participants.

What are issues of design?

A safe research proposal can be ethically questionable because of design issues. For example: research hypothesizing private schools to improve children’s intellectual functioning more than public schools. This design does not allow reasonable causal inference because there is no randomization or consideration of other hypotheses. Research is questionable if:

• Participants’ time is taken from more profitable experiences/treatments.
• Poor quality design leads to unwarranted/inaccurate conclusions possibly harming the society funding the research.
• Giving money and time to poor quality research keeps resources from better science.

What are issues of recruitment?

Hyperclaiming: telling prospective participants (+ granting agencies, colleagues etc.) that research is likely to achieve goals that it is unlikely to achieve. Colleagues and administrators can evaluate our claims fairly but our participants cannot. We should be honest with them about the realistic goals of the study.

Causism: tendency to imply a causal relationship where it has not been established.

Characteristics of causism:

• No appropriate evidential base.
• Presence of language implying cause (‘the effect of’, ‘as a result of’ etc.) where appropriate language would be ‘was related to’ or ‘could be inferred from’.
• Self-serving benefits to the causist because it makes the result appear more important than it really is.

If the causist is unaware of the causism, it reflects poor scientific training. If they are aware of it, reflects unethical misrepresentation and deception.

A description of a proposed research study using causal language represents an unfair recruitment device used to increase potential participation rates.

How does bad science make for bad ethics?

The author proposes that institutional review boards should consider the technical scientific competence of investigators whose proposals they evaluate. Poor quality research can make for poor quality education. Asking a participant to participate in bad research increases the likelihood of them acquiring misconceptions about the nature of science and psychology rather than benefitting educationally.

What are costs and utilities?

When presented questionable research proposals, investigators/review boards employ a cost-utility analysis where the costs of doing a study (time, money, negative effects on participants) are evaluated against utilities (benefits to participants, science, the world etc.). High quality studies/studies with important topics have a utility outweighing the costs. But it is hard to decide if a study should be done when the costs and benefits are equal. However, the costs for failing to do research also should be evaluated – focuses on the benefit for future generations or participants. Example: if people can receive free care during a study that otherwise they couldn’t afford, is it ethical to not conduct this research?

What does data dropping entail?

The goal to have more support for your hypothesis:

1. Outlier rejection: it is likely that researchers drop outliers that are inconsistent with their hypothesis than those falling in line with their hypothesis. Outlier rejection should be reported. Results with outlies should be reported if you decide to drop them.
2. Subject selection: different type of data dropping. Subset of the data is not included in the analysis. Even if there may be good technical reasons, there are still ethical issues like when just subsets not supporting the researcher’s hypothesis are dropped. If you drop subsets, readers should be informed about it and what the results were. Similar considerations apply when results for one or more variables are not reported.

Is data exploitation always bad?

This issue has subtler ethical implications. We are taught that it is improper to snoop around in our data (analyze and reanalyze). It makes for bad science because while snooping affects p values, it is likely to turn up something new. It makes for bad ethics because data are expensive in time, effort, and money and looking further into data may turn up something you may not have found otherwise.

The author says if the research is wroth conducting, it is also worth taking a closer look at it. Replications are needed anyway whether you snoop or not. Bonferroni adjustments can help with the significant p-value that you may find after exploiting your data.

Can meta-analysis be used as an ethical imperative?

Meta-analyses are a set of concepts and procedures used to summarize any domain of research. We see them as more accurate, comprehensive, and statistically powerful compared to traditional literatures review because they have more information. This leads to:

• More accurate estimates of effect sizes and relationships.
• More accurate estimates of overall significance levels of the research domain.
• More useful information about the variables moderating the magnitude of the effect.
• Increase of utilities: time, effort, costs are all more justified when datasets are in a meta-analysis because we can learn more from our data.
• Ethical implications of not doing meta-analyses: failing to employ met-analytic procedures means you lose the opportunity to make use of past research.

Meta-analyses try to explain the variation in effect sizes from different studies. It seems to no longer be acceptable to fund research resolving a controversy unless an investigator as already done a meta-analysis to decide if there really is a controversy.

Pseudocontroversies: meta-analysis resolves controversies because it eliminates two common problems in evaluating replications:

1. When failing to get a significant effect in the replication study, we fail to replicate – failure to replicate is measured by the size of the difference between the effect sizes of two studies.
2. Believing that if there is an effect in a situation, each study of that situation will show a significant effect – the chance to find an effect when there really is one is often quite low.

Significance testing: meta-analysis tries to record the actual level of significance obtained (instead of whether a study reached a certain level) on the standard normal deviate that corresponds to a p value. The use of signed normal deviates:

• Increase a study’s informational value.
• Increases a study’s utility.
• Changes a study’s cost-utility ratio and ethical value.

Meta-analyses increase research utility and ethical justification, providing accurate effect size estimates.

What are issues in reporting psychological research?

Misrepresenting findings

Some misrepresentations of findings are more obviously unethical than others.

• Intentional misrepresentation: includes fabricating data, intentionally/knowingly allocating subjects to experimental and control conditions to support the hypothesis. Not being blind to treatment condition and recording responses, and research assistants recording responses knowing the hypothesis and treatment conditions.
• Unintentional misrepresentation: recording, computational, and data analytic errors can all lead to inaccurate results. This includes causist language and questionable generalizability. Errors in data diminish research utility and shift cost-utility ratio in an unfavourable direction.

Misrepresenting credit

• Problems of authorship: many papers are multi-authored making it difficult to allocate authorship credit. Who becomes a coauthor vs a footnote? Who is assigned first or last coauthor in the listing?
• Problems of priority: an issue between research groups. Who got the idea first?

Failing to report/publish

What was not reported and why? The two biggest forms of failure to report are self-censoring and external censoring.

• Self-censoring: can be admirable when a study is done badly, could be a service to science to start over. Less admirable reasons could be failing to report data that contradict earlier research or personal values – poor science and ethics.
  • Good practice is to report all results that give information on the hypothesis and give data that other researchers could use.
• External censoring: progress and the slowing of progress in science depend on external censoring. Sciences would maybe be more chaotic if it weren’t for censorship by peers who keep bad research from being released. Two major bases for external censorship:
  • Evaluation of methodology used in a study.
  • Evaluation of results obtained in a study.

What can we conclude?

The ethical quality of our research is not independent of the scientific quality of our research.