Has “many analyst” #research been done on quaLitative #data?

How much do *independent* teams of qualitative coders/analyzers converge/diverge when trying to answer one question with one dataset?

Suggestions (and self-promotion) welcome.

#statstab #315 What fraction of repeat experiments will have an effect size within the 95% confidence interval of the first experiment?

Thoughts: CV is a great place to learn. We see the same answer as Cumming: 83%

#replicability #confidenceintervals #replication #metascience #estimation #research #science

https://stats.stackexchange.com/questions/218602/what-fraction-of-repeat-experiments-will-have-an-effect-size-within-the-95-conf

#statstab #313 Analyzing Replication Studies

Thoughts: Replications and RR are on the rise, but not all are conducted appropriately, and become meaningless.

#stats #replication #metascience #openscience #metaresearch #metaanalysis

https://lakens.github.io/statistical_inferences/17-replication.html#analyzing-replication-studies.

"From Hypothesis to Publication: A Comprehensive Survey of AI-Driven
Research Support Systems"

The table shown below from this paper
that shows AI tools for research paper assistance is the most informative one of this type that I have seen in looking at many, many AI and ML research papers.

https://arxiv.org/abs/2503.01424

The authors of this paper have a wonderful bibliography on this topic at the Github repo:
https://github.com/zkzhou126/AI-for-Research

#statstab #298 Replication: Do not trust your p-value, be it small or large

Thoughts: Even under exact replications, a p-values is not very good at predicting the p-value in a future study. p=.05 ~ 50% rep.

#pvalues #replication #estimation #metascience

https://physoc.onlinelibrary.wiley.com/doi/10.1113/JP281614

Interesting paper by J. M. Schauer: "On the Accuracy of Replication Failure Rates. It provides IMHO very convincing evidence that common estimates of replication rates are likely subject to substantial bias and noise and are thus very hard to interpret.
https://doi.org/10.1080/00273171.2022.2066500
#metascience #replication

New in our #digital #collection:

Science of science : understanding the foundations and limits of science from an interdisciplinary perspective

https://doi.org/10.1093/9780198937401.001.0001

'Science of Science' provides an integrated framework for the #science of science as an emerging field, and a comprehensive understanding of the foundations and limits of science.

Via @deevybee review of nowadays much needed didactical #scicomm book

» provides practical tools for verifying the statistical consistency of published research.

It's available as a free ebook, making it accessible to the scientific community.
...
made the subject highly approachable. «

An Introduction to Forensic #Metascience

by @jamesheathers

https://goodscience.substack.com/p/book-review-an-introduction-to-forensic

• Knuth #LiterateProgramming approach

• 'It says in mice' in pg 45 2 same mice

• Didn't expect @sTeamTraen AI critique!

Excited to announce that we are recruiting a Research Associate for this #UKRI-funded #metascience research grant investigating the roll-out of Electronic Research Notebooks (ERNs)

For full details of the role, and information on how to apply, please visit https://www.jobs.manchester.ac.uk/Job/JobDetail?JobId=31714

This is a great opportunity to enhance research practice, support #OpenResearch and #FAIR data, and have a positive impact on researchers in #Manchester and across the UK

List of grant winners from first UKRI Metascience call

https://www.gov.uk/government/news/fellowships-launched-to-explore-how-ai-could-change-the-way-scientists-drive-new-discoveries

(published 13 feb, but I missed it)

Congratulations all

#statstab #290 An Evaluation of Four Solutions to the Forking Paths Problem

Thoughts: Multiverse analysis & the forking paths problem must be considered methodologically but also logically: based on one' stat philosophy.

#metascience #forkingpaths #NHST

https://doi.org/10.1037/gpr0000135

#statstab #288 {Multiverse} An R package for creating multiverse analysis

Thoughts: Doubting the testing framework can be a good way to check how your inferences change if you'd made different (reasonable) assumptions.

#multiverse #r #metascience

https://cran.r-project.org/web/packages/multiverse/readme/README.html

"Funding of Clinical Trials and Reported Drug Efficacy" indicate that industry-sponsored trials are more likely to report an effective drug
https://www.journals.uchicago.edu/doi/full/10.1086/730383 Publication bias is a possible explanation, hintingat valueof preregistration and registered reports #MetaScience @openscience

Was müssten wir in der Wissenschaft verändern, um ein Klima der Prosozialität, der Zusammenarbeit und des Vertrauens unter Forschenden zu fördern? Was sind eure Ideen? #metascience #openscience

New Blogpost! Read about the last meeting of the Platform for Young Metascientists and their plans for the future: https://reproducibilitynetwork.nl/2025/02/20/platform-for-young-meta-scientists-pyms-discussing-the-future-of-meta-science/
Save the date for the next meeting on June 5th in Tilburg!

You can sign up for their newsletter here: https://github.us10.list-manage.com/subscribe?u=ee35f157fb8122562afaa4156&id=c49d6dce15
and follow them on LinkedIn to get the latest news: https://www.linkedin.com/company/platform-for-young-meta-scientists

MEETING

2025 Science of Science Funding https://www.nber.org/conferences/si-2025-science-science-funding

Cambridge, MA. 17-18 July 2025; paper deadline March 20th

Vous avez raté la conférence Métascience, Recherche sur la Recherche et Science Ouverte en Dialogue, à Lyon, le 15 novembre dernier ? C'est fort dommage, car elle fut passionnante.

Mais pas de panique, une deuxième chance s'offre à vous : les enregistrements effectués ce jour-là sont dès à présent disponibles sur Canal U.

https://www.canal-u.tv/chaines/universite-lyon-1-icap/journee-metascience-2024

#MetaScience
#scienceOuverte
#Lyon

#TIL about The Unjournal and it checks sooo many boxes! https://unjournal.pubpub.org/

If you're interested in #metascience and understanding the implications of #AI for the #science and #research ecosystem, this looks like a great postdoctoral opportunity: https://sloan.org/programs/digital-technology/aipostdoc-rfp

UKRI Metascience AI early career fellowships

https://www.ukri.org/opportunity/ukri-metascience-ai-early-career-fellowships/

Call Open today, closes 10 April 2025 4:00pm UK time

If you have the AI/ML skills and want to bolster the scale and impact of your research by plugging into the most exciting metascience team in the world, please get in touch with @RoRInstitute about us supporting your fellowship application