Weaknesses of the Bayesian approach are:
For more information, look at the (free) summary of 'Bayesian versus orthodox statistics: which side are you on?'
Bayesian Versus orthodox statistics: which side are you on? - summary of an article by Dienes, 2011
Critical thinking
Article: Dienes, Z, 2011
Bayesian Versus orthodox statistics: which side are you on?
doi: 10.1177/1745691611406920
The orthodox logic of statistics, starts from the assumption that probabilities are long-run relative frequencies.
A long-run relative frequency requires an indefinitely large series of events that constitutes the collective probability of some property (q) occurring is then the proportion of events in the collective with property q.
The logic of Neyman Pearson (orthodox) statistics is to adopt decision procedures with known long-term error rates and then control those errors at acceptable levels.
Thus, setting significance and power controls long-run error rates.
The probability of a theory being true given data can be symbolized as P(theory|data).
This is what orthodox statistics tell us.
One cannot infer one conditional probability just by knowing its inverse. (So P(data|theory) is unknown).
Bayesian statistics starts from the premise that we can assign degrees of plausibility to theories, and what we want our data to do is to tell us how to adjust these plausibilities.
The likelihood
In the Bayesian approach, applies to the truth of theories.
We can answer the questions about:
Neither of these can be do using the orthodox approach.
Likelihood: the probability of obtaining the exact data given the hypothesis.
Posterior is given by likelihood times prior.
The likelihood principle: all information relevant to inference contained in data is provided by the likelihood.
When we are determining how given data changes the relative probability of our different theories, it is only the likelihood that connects the prior to the posterior.
The likelihood is the probability of obtaining the exact data obtained given a hypothesis (P(D|H).
This is different from a p value, which is the probability of obtaining the same or more extreme data given both a hypothesis and a decision procedure.
In orthodox statistics, p values are changed according to the decision procedure; under what conditions one would stop collecting data, whether or not the test is post hoc, how many other test one conducted.
None of these factor influence the likelihood.
The Bayes factor
The Bayes factor pits one theory against another.
Prior probabilities and prior odds can be entirely personal and subjective.
There is no reason why people should agree about these before data are collected if they are not part of the publically presented inferential procedure.
If the priors form part of the inferential procedure, they must be fairly produced and subjected to the tribunal of peer judgement.
One data are collected we can calculate the likelihood for each theory.
These likelihoods are things we want researchers to agree on. Any probabilities that contribute to them should be plausibly or simply determined by determined by the specification of the theories.
The Bayes factor (B): the ratio of likelihoods.
Posterior odds = B x prior odds.
The evidence is continuous and there are not thresholds in Bayesian theory.
B automatically gives a notion of sensitivity, it directly distinguishes data supporting the null from data uninformative about whether the null or you theory was supported.
For both p values associated with a t test and for B, if the null is false, as a number of subjects increases, then test scores are driven in one direction.
When the null hypothesis is true, p values are not driven in any direction, only B us. B is then driven to zero.
Stopping rule
In the Neyman Pearson approach, one must specify the stopping rule in advance.
Once those conditions are met, there is to be no more data collection.
Typically, this means one should use a power calculation to plan in advance how many subjects to run.
The Bayes factor behaves differently from p values as more data are run (regardless of stopping rule).
Planned versus post hoc comparisons
When using Neyman Pearson, it matters whether you formulated your hypothesis before or after looking at the data (post hoc vs. planned comparisons).
Predictions made before rather than after looking at the data are treated differently.
In Bayesian inference, the evidence for a theory is just as strong regardless of its timing relative to the data.
This is because the likelihood is unaffected by the time the data were collected.
The likelihood principle follows from the axioms of probability.
It is not the ability to predict in advance per se that is important, that ability is just an (imperfect) indicator of the prior probability of relevant hypotheses.
When performing Bayesian inference, there is no need to adjust for the timing of predictions per se.
Multiple testing
When using Neyman Pearson, one must correct for how many tests are conducted in a family of tests.
When using Bayes, it does not matter how many other statistical hypotheses are investigated. All that matters is the data relevant to each hypothesis under investigation.
Once one takes into account the full context, the axioms of probability lead to sensible answers.
In the Bayes approach, rather than the Neyman Pearson approach, that is most likely to demand that researchers draw appropriate conclusions from a body of relevant data involving multiple testing.
If we want to determine by how much we should revise continuous degrees of belief, we need to make sure our system of inference obeys the axioms of probability.
If researchers want to think in terms of degree of support data provide for a hypothesis, they should make sure their inferences obey the axioms of probability.
One version of degrees of belief are subjective probabilities.
Subjective probabilities: personal convictions in an opinion.
When probabilities of different propositions form part of the inferential procedure we use in deriving conclusions from data, then we need to make sure that the procedure is fair.
Thus, there has been an attempt to specify objective probabilities that follow from the informational specification of a problem.
In this way, the probabilities become an objective part of the problem, with values that can be argued about, given the explicit assumptions, and that do not depend any further on personal idiosyncrasies.
One notion of rationality is having sufficient justification for one’s beliefs.
If one can assign numerical continuous degrees of justification to beliefs, then some simple minimal desiderata lead to the likelihood principle of inference.
Hypothesis testing violates the likelihood principle.
Bayes factors demand consideration of relevant effect sizes.
Neyman developed two specific measures of sensitivity:
For any continuous measure based on a finite number of subjects, an interval cannot be an infinitesimally small point.
A null result is always consistent with population values other than zero.
That is why a non-significant result cannot on its own lead to the conclusion that the null hypothesis is true.
Theories and practical questions generally specify, even if vaguely, relevant effect sizes.
The research context, usually provides a range of effects that are too small to be relevant and a range of effects that are consistent with theory or practical use.
Researchers have relevant intuitions, and that is why it has made sense to them to assert null hypotheses.
Bayes makes them explicit.
If we want to use null results in any way to count against theories that predict an effect, we must consider the range of effect sizes consistent with the theory.
Effect size is very important in the Neyman Pearson approach.
On the other hand, Fisherian significance testing leads people to ignore effect sizes.
One must specify what sort of effect sizes a theory predicts to calculate a Bayes factor.
Because it takes into account effect size, the Bayes factor distinguishes evidence that there is not relevant effect from no evidence of a relevant effect.
One can only confirm a null hypothesis when one has specified the effect size expected on the theory being tested.
In specifying theoretically expected effect sizes, we should ask ourselves “What size effect does the literature suggest is interesting for this particular domain?” Rather than following the common practice of plucking a standardized effect size of 0.5 out of thin air, researchers should get to know the data of the field.
Confidence intervals themselves have all the problems for Neyman Pearson inference in general (unlike credibility or likelihood intervals).
Because confidence intervals consists of all values non-significantly different from the sample mean, they inherit the arbitrariness of significance testing.
To calculate a Bayes factor in support of a theory, one has to specify what the probability of different effect sizes are, given the theory.
Bayes gives us the apparatus to flexibly deal with different degrees of uncertainty regarding the predicted effect size.
Logically, one needs to know what a theory predicts in order to know how much it is supported by evidence.
Three distributions
In terms of predictions of the theory (or requirements of a practical effect), one has to decide what range of effects are relevant to the theory.
Three ranges:
Different ways of using Bayes factors
Bayes factor is suggested to be used on any data where the null hypothesis is compared with a default theory.
Or when inference is based on the posterior and thus takes into account the priors of hypothesis.
Also for specific hypotheses that interest the researcher and allows priors to remain personal and not part of public inference.
By following Bayes rule, each of these approaches means rational answers are provided for the given assumptions, and researchers may choose each according to their goals and which assumptions seem relevant to them.
Bayes factors are just one form of Bayesian inference, namely a method for evaluating theories against another.
With Bayes factors, one does not have to worry about corrections for multiple testing, stopping rules, or planned versus post hoc comparisons.
Bayes factor just tells you how much support given data provides for one theory over another.
There is no right Bayes factor.
Strictly, each Bayes factor is a completely accurate indication of the support for the data of one theory over another.
The theories are defined by the precise predictions they make.
The crucial question is which of these representations best matches the theory as the researcher has described it and related it to the existing literature.
One constraint on the researcher will be the demand for consistency: arguing for one application of a theory ties one’s hands when it comes to another application.
The solution is to use a default Bayes factor for all occasions, though this amounts to evaluating a default theory for all occasions, regardless of one’s actual theory.
A default Bayes factor will only test your theory if it happens to correspond to the default.
Another solution is to define the predictions according to simple procedures to ensure the theory proposed is tested according to fair criteria.
When using Bayes in multiple testing, one can use the fact that one is testing multiple hypotheses to inform the results if one believes that testing these multiple hypotheses is relevant to the probability of any of them being true.
Calculating a Bayes factor depends on answering the following question about which there may be disagreement: What way of assigning probability distributions of effect sizes as predicted by theories would be accepted by protagonists on all sides of a debate?
Ultimately, the issue is about what is more important to us: using a procedure with known long-term error rates or knowing the degree of support for our theory.
WSRt, critical thinking - a summary of all articles needed in the fourth block of second year psychology at the uva
This is a summary of the articles and reading materials that are needed for the fourth block in the course WSR-t. This course is given to second year psychology students at the Uva. The course is about thinking critically about how scientific research is done and how this could be done differently.
Je vertrek voorbereiden of je verzekering afsluiten bij studie, stage of onderzoek in het buitenland
Study or work abroad? check your insurance options with The JoHo Foundation
Add new contribution