Critical thinkingArticle: Simmons, Nelson, & Simonsohn (2011)False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant This article is about two things:despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings, flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to false find evidence that an effect exists than to correctly find evidence that it does not.a solution to that problem. One of the most costly errors is a false positive.The incorrect rejection of the null hypothesis.Once they appear in the literature, they are persistent.Because null results have many possible causes, failures to replicate previous findings are never conclusive.Because it is uncommon for prestigious journals to publish null findings or exact replication, researchers have little incentive to even attempt them.False positives waste resourcesThey inspire investment in fruitless research programs and can lead to ineffective policy changes.Ambiguity is rampant in empirical research. As a solution to the flexibility-ambiguity problem, there are offered six requirements for authors and four guidelines for reviewers.This solution substantially mitigates the problem but imposes only a minimal burden on authors, reviewers, and readers.Leaves the right and...

Critical thinkingArticle: Dienes (2003)Neyman, Pearson and hypothesis testing In this article, we will consider the standard logic of statistical inference.Statistical inference: the logic underlying all the statistics you see in the professional journals of psychology and most other disciplines that regularly use statistics.The underlying logic of statistic (Neyman-Pearson) is both highly controversial, frequently attacked (and defended) by statisticians and philosophers, and more frequently misunderstood. The meaning of probability we choose determines what we can do with statistics.The proper way of interpreting probability remains controversial, so there is still debate over what can be achieved with statistics.The Neyman-Pearson approach follows form one particular interpretation of probability. The Bayesian approach considered follows form another.Interpretations often start with a set of axioms that probabilities must follow.Two interpretations of probability:the subjective interpretation: a probability is a degree of conviction of a beliefthe objective interpretation: locate probability in the world.The most influential objective interpretation of probability is the long-run relative frequency...

Critical thinkingArticle: Dienes (2008)Degrees of falsifiability A potential falsifier of a theory: any potential observation that would contradict the theory.One theory is more falsifiable than another if the class of potential falsifiers is larger.Scientists prefer simple theories.Simple theories are better testable.A theory can gain in falsifiability not only by being precise, but also be being broad in range of situations to which the theory applies.The greater the universality of a theory, the more falsifiable it is. Even if the predictions are not very precise.Revisions to a theory may make it more falsifiable by specifying fine-grained causal mechanisms.As long as the steps in a proposed causal pathway are testable, specifying the pathway gives you more falsifiers.Psychologists sometimes theorize and make predictions by constructing computational models.A computational model is a computer simulation of a subject, where the model is exposed to the same stimuli subjects receive and gives actual trial-by-trial responses.A theory that allows everything explains nothing.The more a theory forbids, the more it says about the world. The empirical content of a theory increases with its degree of falsifiability.The more falsifiable a theory is, the more open it is to criticism.So the more falsifiable our theories are, the faster we can make progress, given progress comes from criticism.Science aims at the maximum falsifiability it can achieve: successive theories should be successively more falsifiable. Either in terms of universality or precision.Make sure that any revision or amendment to theory can be falsified. That way theory development is guaranteed to keep its...

Critical thinkingArticle: Pearl (2018)Confounding and deconfounding: or, slaying the lurking variable Confounding bias occurs when a variable influences both who is selected for the treatment and he outcome of the experiment.Sometimes the confounders are known. Other times they are merely suspected and act as a ‘lurking third variable’.If we have measurements of the third variable, then it is very easy to deconfound the true and spurious effects.Statisticians both over- and underrate the importance of adjusting for possible confoundersOverrate in the sense that they often control for many more variables than they need to and even for variables that they should not control forUnderrate in the sense that they are loath to talk about causality at all, even if the controlling has been done correctly. Knowing the set of assumptions that stand behind a given conclusion is not less valuable than attempting to circumvent those assumptions with and RCT, which has complications on its own. The one circumstance under which scientists will abandon some of their reticence to talk about causality is when they have conducted a randomized controlled trial (RCT).Randomization brings two benefits:It eliminates confounder biasIt enables the researcher to quantify...

Critical thinkingArticle: Marewski, & Olsson, (2009)Beyond the null ritual, formal modeling of psychological processes Rituals can be characterized by a range of attributes including:Repetitions of the same actionFixations on special features such as numbersAnxieties about punishments for rule violationsWishful thinkingEach of these characteristics is reflected in null hypothesis significance testing. One good way to make theories more precise is to cast them as formal models.In doing so, researchers can move beyond the problems of null hypothesis significance testing, and simple difference searching. In the broadest sense, a model is a simplified representation of the world that aims to explain observed data.A model is a formal instantiation of a theory that specifies the theory’s predictions. This category also includes statistical tools, such as structural equation or regression models. Statistical tools are not typically meant to mirror the workings of psychological mechanisms. What is the scope of Modeling? Modeling is not meant to be applied equally to all research questions. Each method has its specific advantages and disadvantages. Modeling helps researchers answer involved questions and understand complex phenomena.In psychology, modeling is especially suited for basic and applied research about the cognitive system. ...

Critical thinkingArticle: Kievit, Frankenhuis, Waldorp, & Borsboom (2013)Simpson's paradox in psychological science: a practical guideIntroduction Simpson’s paradox: the direction of an association at the population-level may be reversed within the subgroups comprising that population.Simpson showed that a statistical relation observed in a population could be reversed within all of the subgroups that make up that population. Simpson’s paradox is a counter-intuitive feature of aggregated data, which may arise when (causal) inferences are drawn across different explanatory levels. (like population to subgroup or subgroup to individual).Simpson’s paradox is conceptually and analytically related to many statistical challenges and techniques.The underlying shared theme of these techniques is that they are concerned with the nature of (causal) inference. The challenge is what inferences are warranted based on the data we observe. One can only be sure that a group-level finding generalizes to individuals when the data are ergodic, which is a very strict requirement.Since this requirement is unlikely to hold in many data sets, extreme caution is warranted in generalizing across levels.The dimensions that appear in a covariance structure analysis describe patterns of variation between people, not variation within individuals over time.A person X may have a position on five dimensions compared to other people in a given population, but this...

Critical thinkingArticle: Scott O. Lilienfeld (2005)The 10 commandments of helping students distinguish science from pseudoscience in psychology The first commandment It is important to communicate to students that the differences between between science and pseudoscience, although not absolute or clear-cut, are neither arbitrary or subjective.Warning signs that characterize most pseudoscientific disciplines:A tendency to invoke ad hoc hypotheses, which can be thought of as ‘escape hatches’ or loopholes, as a means of immunizing claims from falsification.An absence of self-correction and an accompanying intellectual stagnationAn emphasis on confirmation rather than refutationA tendency to place the burden of proof on sceptics, not proponents, of claimsExcessive reliance on anecdotal and testimonial evidence to substantiate claimsEvasion of the scrutiny afforded by peer reviewAbsence of ‘connectivity’, a failure to build on existing scientific knowledgeUse of impressive-sounding jargon whose primary purpose is to lend claims of facade of scientific respectabilityAn absence of boundary conditions. A failure to specify the settings under which claims do not hold.Non of these warnings signs is by itself sufficient to indicate that a discipline is pseudoscientific.But, the more of these warning signs a discipline exhibits, the more suspect it should become.The second commandmentLearning to distinguish scepticism from cynicism.One danger of teaching students to distinguish science from pseudoscience is that we can...