#statstab #309 The statistical significance filter leads to overoptimistic expectations of replicability
Thoughts: Not sure how many researchers interpret p-values are indexes of replicability, but they shouldn't.
Recent searches
Search options
#pvalues
0 posts0 participants0 posts today
#statstab #300 (!!!) 
Beyond the forest plot: The drapery plot
Thoughts: I can't believe they didn't call these 'tepee plots'.
I think they are cluttered, but can be useful.
#metaanalysis #pvalues #consonancecurve #pvaluefunction #dataviz #prediction #plots #figures #R
#statstab #298 Replication: Do not trust your p-value, be it small or large
Thoughts: Even under exact replications, a p-values is not very good at predicting the p-value in a future study. p=.05 ~ 50% rep.
#statstab #295 The Fallacy of the Null-Hypothesis Significance Test
Thoughts: "the [..] aim of a scientific experiment is not to precipitate decisions, but to make an appropriate adjustment in the degree to which one accepts, or believes, the hypothesis"
#NHST #Bayes #ConfidenceIntervals #pvalues #significance #testing #hypotheses #likelihood #critique #fallacy
#statstab #289 The meaning of significance in data testing
Thoughts: Fisherian significance testing =/= Neyman-Pearson statistical hypothesis testing. Many debates on p-values and frequentist stats are due to this confusion.
#statstab #287 Dance of the p Values
Thoughts: One of my go-to demonstrations for the variability of p-values, and why they say so little about a study.
#pvalues #NHST #education #estimation #frequentist #replication #error #visualization #teaching
youtu.be- YouTubeEnjoy the videos and music you love, upload original content, and share it all with friends, family, and the world on YouTube.
#statstab #272 Different meanings of p-values
Thoughts: A riveting (& confusing) discussion on the definitions & properties of p-values. W/ guest appearance from some big names in stats, from all camps.
statmodeling.stat.columbia.edu
4 different meanings of p-value (and how my thinking has changed) | Statistical Modeling, Causal Inference, and Social Science
General PsychiatryThe p-value and model specification in statisticsThe p value has been widely used as a way to summarise the significance in data analysis. However, misuse and misinterpretation of the p value is common in practice. Our result shows that if the model specification is wrong, the distribution of the p value may be inappropriate, which makes the decision based on the p value invalid.
Replied in thread
"In real life, we weigh the anticipated consequences of the decisions that we are about to make. That approach is much more rational than limiting the percentage of making the error of one kind in an artificial (null hypothesis) setting or using a measure of evidence for each model as the weight."
Longford (2005) http://www.stat.columbia.edu/~gelman/stuff_for_blog/longford.pdf
Replied in thread
Surveys, coincidences, statistical significance 
"What Educated Citizens Should Know About Statistics and Probability"
By Jessica Utts, in 2003: https://ics.uci.edu/~jutts/AmerStat2003.pdf via @hrefna
#statstab #217 The distribution of p-values obtained in replications depends only on the original p-value. How can it be true?
Thoughts: A great discussion where the author @thenewstats chimes in to explain the issue.
Jacob Cohen: we, as teachers, consultants, authors, and otherwise perpetrators of quantitative methods, are responsible for the ritualization of null hypothesis significance testing,,, to the point of meaninglessness and beyond.
Wendy's teenager: Um, sir...
![An awareness of the history of p-values might elp deflate their swollen stature and encourage more judicious use. We were surprised to learn, in the course of writing this article, that the p < 0.05 cutoff was established as a competitive response to a disagreement over book royalties between two foundational statisticians. In the early 1920s, Kendall Pearson, whose income depended on the sale of extensive statistical tables, was unwilling to allow Ronald A. Fisher to use them in his new book. To work around this barrier, Fisher created a method of inference based on only two values: p-values of 0.05 and 0.01 (Hurlbert and Lombardi,
2009). Fisher himself later admitted that Pearson's more continuous method of inference was better than his binary approach: "no scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects [null] hypotheses; he rather gives his mind to each particular case in the light of his evidence and ideas (Hurlbert and Lombardi, 2009: 316). A fair interpretation of this history is that we use p-values at east in part because a statistician from the 1920s was afraid that sharing his work would undermine his income](https://s3.eu-central-2.wasabisys.com/mastodonworld/media_attachments/files/113/404/999/832/750/171/original/197090ae839ab845.jpeg "An awareness of the history of p-values might elp deflate their swollen stature and encourage more judicious use. We were surprised to learn, in the course of writing this article, that the p < 0.05 cutoff was established as a competitive response to a disagreement over book royalties between two foundational statisticians. In the early 1920s, Kendall Pearson, whose income depended on the sale of extensive statistical tables, was unwilling to allow Ronald A. Fisher to use them in his new book. To work around this barrier, Fisher created a method of inference based on only two values: p-values of 0.05 and 0.01 (Hurlbert and Lombardi,
2009). Fisher himself later admitted that Pearson's more continuous method of inference was better than his binary approach: \"no scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects [null] hypotheses; he rather gives his mind to each particular case in the light of his evidence and ideas (Hurlbert and Lombardi, 2009: 316). A fair interpretation of this history is that we use p-values at east in part because a statistician from the 1920s was afraid that sharing his work would undermine his income")
#statstab #209 Limitations of empirical calibration of p-values using observational data
Thoughts: Obs. research doesn't need p-values (imo) but ppl keep tryin to make'em happen
#pvalues #observational #empirical #causal
https://pmc.ncbi.nlm.nih.gov/articles/PMC5012943/
rebuttal
https://pubmed.ncbi.nlm.nih.gov/27592566/
PubMed Central (PMC)Limitations of empirical calibration of p-values using observational dataControversy over non-reproducible published research reporting a statistically significant result has produced substantial discussion in the literature. P-value calibration is a recently proposed procedure for adjusting p-values to account for both ...
#statstab #186 Are Multiple Contrast Tests Superior to the ANOVA?
Thoughts: Focuses more on the statistical aspects of the comparison than the theoretical ones. But insightful for newbies.
#nhst #anova #ttest #errorcontrol
#typeI #pvalues
https://www.degruyter.com/document/doi/10.1515/ijb-2012-0020/html?lang=en
De Gruyter · Are Multiple Contrast Tests Superior to the ANOVA?Multiple contrast tests can be used to test arbitrary linear hypotheses by providing local and global test decisions as well as simultaneous confidence intervals. The ANOVA- F -test on the contrary can be used to test the global null hypothesis of no treatment effect. Thus, multiple contrast tests provide more information than the analysis of variance (ANOVA) by offering which levels cause the significance. We compare the exact powers of the ANOVA- F -test and multiple contrast tests to reject the global null hypothesis. Hereby, we compute their least favorable configurations (LFCs). It turns out that both procedures have the same LFCs under certain conditions. Exact power investigations show that their powers are equal to detect their LFCs.
#statstab #165 Approximate Objective Bayes Factors From P-Values and Sample Size: The 3p√n Rule
Thoughts: p-values can't quantify evidence, but maybe if we apply a transformation they can? Is JAB_01 the future? Debate!
Replied in thread
There even wikipedia on the "Misuse of p-values": https://en.wikipedia.org/wiki/Misuse_of_p-values
I therefore am adding to my guidelines: "Instead of telling researchers what they want to know, statisticians should teach researchers which questions they can ask. […]
Before we can improve our statistical inferences, we need to improve our statistical questions."
Excerpt from Daniël Lakens (2021) https://journals.sagepub.com/doi/10.1177/1745691620958012
en.wikipedia.orgMisuse of p-values - Wikipedia