mischbeck/MiningTransparencyManuscript
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 1 Wiki Activity

revision 1, almost done

Browse Source
This commit is contained in:
Michael Beck
2026-05-18 22:43:11 +02:00
parent ada154a107
commit c4b94c7f8f
9 changed files with 489 additions and 227 deletions
Download Patch File Download Diff File Expand all files Collapse all files
index.qmd
+225 -224
View File
@@ -6,7 +6,7 @@ execute:
freeze: auto
abstract: |
This pilot study addresses the current lack of systematic, large-scale evidence on Open Science Practices (OSPs) adoption in criminology and legal psychology. A scalable, machine-learning-based text classification pipeline is introduced to map the prevalence of Open Access (OA), Open Data (OD), Open Materials (OM), and Preregistration (PR). The analysis is based on publication metadata and a year-stratified sample of full texts from the top 100 journals in Criminology & Penology, Law, and Psychology (2013-2023). After identifying articles containing statistical inference (SI) via a high-performing classifier, the author utilized GPT-assisted coding and supervised learning to train specific classifiers for OD, OM, and PR. OA was classified using publicly available metadata. Among 1,763 SI articles with usable full text, design-based estimates reveal a significant disparity in OSP adoption. OA is relatively common (40.9%, 95% CI: 38.8-43.1) and has steadily increased from approximately 20% in 2013 to 50% in 2023. By sharp contrast, trends for OD, OM, and PR cannot be reliably quantified. Extreme class imbalance and the minimal number of positive cases indicate a very low underlying true prevalence for these practices in the assessed field. Methodologically, the study confirms that GPT-assisted coding supports accurate SI detection, but robust prevalence estimation for extremely low-frequency OSPs remains challenging for downstream classifiers. Overall, this project establishes a transparent and reproducible pipeline and provides critical baseline estimates for future, larger-scale assessments of research transparency in crime-related fields.
This pilot study addresses the current lack of systematic, large-scale evidence on open science practices (OSPs) adoption in criminology and legal psychology. A scalable, machine-learning-based text classification pipeline is introduced to map the prevalence of Open Access (OA), Open Data (OD), Open Materials (OM), and Preregistration (PR). The analysis is based on publication metadata and a year-stratified sample of full texts from the top 100 journals in Criminology & Penology, Law, and Psychology (2013-2023). After identifying articles containing statistical inference (SI) via a high-performing classifier, the author utilized GPT-assisted coding and supervised learning to train specific classifiers for OD, OM, and PR. OA was classified using publicly available metadata. Among 1,763 SI articles with usable full text, design-based estimates reveal a significant disparity in OSP adoption. OA is relatively common (40.9%, 95% CI: 38.8-43.1) and has steadily increased from approximately 20% in 2013 to 50% in 2023. By sharp contrast, trends for OD, OM, and PR cannot be reliably quantified. Extreme class imbalance and the minimal number of positive cases indicate a very low underlying true prevalence for these practices in the assessed field. Methodologically, the study confirms that GPT-assisted coding supports accurate SI detection, but robust prevalence estimation for extremely low-frequency OSPs remains challenging for downstream classifiers. Overall, this project establishes a transparent and reproducible pipeline and provides critical baseline estimates for future, larger-scale assessments of research transparency in crime-related fields.
---
```{=latex}
@@ -33,15 +33,15 @@ debug_mode <- FALSE
# Introduction
When evidence makes headlines, influences public opinions, shapes policing, sentencing or rehabilitation, it touches lives. But over the last decades, social scientists have learned how easily impressive results evaporate. Criminology is not insulated from these pressures. In this paper, open science practices in criminology and legal psychology are monitored to assess if the field is wired to catch errors before they might become policy.
When evidence makes headlines, influences public opinions, shapes policing, sentencing or rehabilitation, it touches lives. But over the last decades, social scientists have learned how easily impressive results evaporate. Criminology is not insulated from these pressures. In this paper, open science practices (OSPs) in criminology and legal psychology are monitored to assess if the field is wired to catch errors before they might become policy.
> "Only by [...] repetitions can we convince ourselves that we are not dealing with a mere isolated 'coincidence', but with events which, on account of their regularity and reproducibility, are in principle inter-subjectively testable." [@popperLogicScientificDiscovery2005, p. 23]
To challenge bias and to support replication of research, a movement has formed within the scientific community, fueled by the "replication crisis" that was especially prevalent within the field of psychology [@dienlinAgendaOpenScience2021]. The open science movement tries to establish open science practices (OSPs) to challenge many of the known biases that endanger the reliability of the scientific process and enable access to the scientific discourse for a broader public [@banksAnswers18Questions2019]. The ongoing debate of the last decades was especially focused on two OSPs:
To challenge bias and to support replication of research, a movement has formed within the scientific community, fueled by the "replication crisis" that was especially prevalent within the field of psychology [@dienlinAgendaOpenScience2021]. The open science movement tries to establish OSPs to challenge many of the known biases that endanger the reliability of the scientific process and enable access to the scientific discourse for a broader public [@banksAnswers18Questions2019]. The ongoing debate of the last decades was especially focused on two OSPs:
*First*, openly sharing materials, data and code enables replication that reduces p-hacking, surfaces errors, spreads methodological knowledge and might reduce burdens on the researcher, driving broader adoption across science [@freeseAdvancesTransparencyReproducibility2022; @freeseReplicationStandardsQuantitative2007; @finkReplicationCodeAvailability2024]. *Second*, preregistration involves thoroughly outlining and documenting research plans and their rationale in a repository before conducting the research, reducing deliberate or unconscious decisions taken to improve findings, challenging publication bias and other biases [@managoPreregistrationRegisteredReports2023; @hardwickeReducingBiasIncreasing2023; @mertensPreregistrationAnalysesPreexisting2019].
*First*, openly sharing materials, data and code enables replication that reduces p-hacking, surfaces errors, spreads methodological knowledge and might reduce burdens on the researcher, driving broader adoption across science [@freeseAdvancesTransparencyReproducibility2022; @freeseReplicationStandardsQuantitative2007; @finkReplicationCodeAvailability2024]. *Second*, preregistration (PR) involves thoroughly outlining and documenting research plans and their rationale in a repository before conducting the research, reducing deliberate or unconscious decisions taken to improve findings, challenging publication bias and other biases [@managoPreregistrationRegisteredReports2023; @hardwickeReducingBiasIncreasing2023; @mertensPreregistrationAnalysesPreexisting2019].
The initial plan for this work was to study the proposed effects of OSPs on reported effect sizes in published papers. During a first literature review, it appeared to me that there were only few publications that used preregistration in data-driven Criminology and Legal Psychology. Instead of assessing effect sizes, this raised the question of how OSPs have been adopted within criminology at all. Motivated by the expected positive impact of OSPs, this work studies the use of OSPs in the field.
The initial plan for this work was to study the proposed effects of OSPs on reported effect sizes in published papers. During a first literature review, it appeared to me that there were only few publications that used PR in data-driven Criminology and Legal Psychology. Instead of assessing effect sizes, this raised the question of how OSPs have been adopted within criminology at all. Motivated by the expected positive impact of OSPs, this work studies the use of OSPs in the field.
@scogginsMeasuringTransparencySocial2024 did an extensive analysis of nearly 100,000 publications in political science as well as international relations and observed an increasing use of OSPs, with levels still being relatively low. Their extensive research not only revealed the current state of open science in political science, but also generated rich data to perform further meta research. Inspired by their work, I adopt their research questions to assess OSPs in the fields of criminology and legal psychology:
@@ -49,7 +49,7 @@ The initial plan for this work was to study the proposed effects of OSPs on repo
> $RQ_2$: What proportion of statistical inference publications were preregistered?
This work gathers data about papers in a subset of Criminology and Legal Psychology journals, categorizes those papers by application of open science practices using machine learning methods and explore the patterns over time. The methods will closely resemble and try to improve the approaches taken by @scogginsMeasuringTransparencySocial2024. The research will contribute to the ongoing discussion about the use of OSPs by painting a clearer picture of their adoption in the field. The improved approach will serve as a starting point for a more extensive exploration of OSPs in criminology and legal psychology and will contribute to the growing literature of machine learning and LLMs in classification tasks of scientific literature.
This work gathers data about papers in a subset of Criminology and Legal Psychology journals, categorizes those papers by application of OSPs using machine learning methods and explore the patterns over time. The methods will replicate and try to improve the approaches taken by @scogginsMeasuringTransparencySocial2024. Alongside the two research questions, the share of openly accessible publications is reported as a secondary, descriptive analysis. While open access adresses a broader aspect of transparency, it is tracked alongside these practices and provides additional context for the overall trend in open-science adoption. The research will contribute to the ongoing discussion about the use of OSPs by painting a clearer picture of their adoption in the field. The improved approach will serve as a starting point for a more extensive exploration of OSPs in criminology and legal psychology and will contribute to the growing literature of machine learning and LLMs in classification tasks of scientific literature.
But first, a closer look at the underlying issues leading to the recent development of the open science movement will be taken to gain a deeper understanding of its context, the intended goals, implemented methods and their expected impact on the ever progressing scientific discourse.
@@ -57,45 +57,45 @@ But first, a closer look at the underlying issues leading to the recent developm
In his widely reviewed standard reading "Seven rules for social research", @4ff8afa9-5c92-3c50-b832-a1756ccbeedc emphasizes the importance of the replication of research findings. But already in the title of the chapter or the rule itself, Firebaugh cuts back on his appeal: "replicate *where possible*". He notes increasing data availability, yet acknowledges challenges for true replication. Given the books influence since 2008, one might expect replication and replication-enabling practices to be widely adopted today. But is this the case?
Besides the theoretically driven discourse, there are quite tangible reasons to talk about the scientific method, replication and the publication process. Analyzing 77 research teams assessing the same dataset for a single hypothesis, @breznauObservingManyResearchers2022 found extremely diverse results, ranging from strong positive to strong negative outcomes. They termed this phenomenon "researcher degrees of freedom", explaining that most of the variance in results was not explained by assigned conditions, research decisions, or researcher characteristics. Instead, idiosyncratic researcher variability accounted for more than 90% of the variance.
Besides the theoretically driven discourse, there are quite tangible reasons to talk about the scientific method, replication and the publication process. Analyzing 77 research teams assessing the same dataset for a single hypothesis, @breznauObservingManyResearchers2022 found extremely diverse results, ranging from strong positive to strong negative outcomes. @simmonsFalsePositivePsychologyUndisclosed2011 first termed this phenomenon "researcher degrees of freedom" (RDOF), explaining that most of the variance in results was not explained by assigned conditions, research decisions, or researcher characteristics. Instead, RDOF accounted for more than 90% of the variance [@breznauObservingManyResearchers2022].
This raises the question: if modern research practices are so prone to bias and error, what steps can be taken to mitigate these issues? A closer look at an ongoing debate resulting from cases around replication failures helps shed light on the whole complex, its implications and the today's research culture.
## From Replication Crisis to Credibility Revolution? {#sec-replication-crisis}
The publication of Firebaugh's text coincided with the onset of the replication crisis, a period where widespread replication failures especially but not exclusively in psychology revealed systemic issues in research culture. This crisis wasn't limited to a few fraudulent cases but exposed a broader problem where seemingly robust, highly cited studies could not be reproduced. Examples ranged from unintended to outright data fabrication [@barghAutomaticitySocialBehavior1996; @callawayReportFindsMassive2011; @crockerRoadFraudStarts2011a]. While the crisis began in psychology, it soon spread to other fields like in political science and economics [@breznauDoesSociologyNeed2021]. For instance, a classic social priming study by @barghAutomaticitySocialBehavior1996, finding that participants primed with an "elderly" stereotype walked more slowly, failed to replicate. A follow-up-study suggested, that the original results were likely influenced by experimenter expectations rather than the hypothesized mechanism of unconscious priming [@doyenBehavioralPrimingIts2012]. While some extreme cases are well-documented, the crisis is largely seen as a result of systemic pressure - such as institutional incentives to "publish or perish" - and normal human behavior or misconduct rather than serious intent [@diekmannII2Probleme2022; @crockerRoadFraudStarts2011a; @4ff8afa9-5c92-3c50-b832-a1756ccbeedc].
The publication of Firebaugh's text coincided with the onset of the replication crisis, a period where widespread failures to *replicate* - that is, to obtain consistent results when re-running a study with new data - revealed systemic issues in research culture. This is distinct from computational *reproducibility*, the narrower question of whether a given analysis can be re-executed on the same data; both matter for open science, but the crisis is primarily about the former. This crisis wasn't limited to a few fraudulent cases but exposed a broader problem where seemingly robust, highly cited studies could not be replicated. Examples ranged from unintended to outright data fabrication [@barghAutomaticitySocialBehavior1996; @callawayReportFindsMassive2011; @crockerRoadFraudStarts2011a]. While the crisis began in psychology, it soon spread to other fields like in political science and economics [@breznauDoesSociologyNeed2021]. For instance, a classic social priming study by @barghAutomaticitySocialBehavior1996, finding that participants primed with an "elderly" stereotype walked more slowly, failed to replicate. A follow-up study suggested, that the original results were likely influenced by experimenter expectations rather than the hypothesized mechanism of unconscious priming [@doyenBehavioralPrimingIts2012]. While some extreme cases are well-documented, the crisis is largely seen as a result of systemic pressure - such as institutional incentives to "publish or perish", described later - and normal human behavior or misconduct rather than serious intent [@diekmannII2Probleme2022; @crockerRoadFraudStarts2011a; @4ff8afa9-5c92-3c50-b832-a1756ccbeedc].
The term crisis not only implies alarmingly high proportions, but also creates pressure to act. This is supported by findings spanning many fields: Finance [@jensenThereReplicationCrisis2023], economics [@briggsPartialSolutionReplication2023], sociology [@auspurgAusmassUndRisikofaktoren2014] or medicine [@begleyRaiseStandardsPreclinical2012], with some authors even claiming that most published research findings in the social sciences are false [@ioannidisWhyMostPublished2005]. But what drives this crisis?
The term crisis not only implies alarmingly high proportions, but also creates pressure to act. This is supported by findings spanning many fields: social-psychology [@callawayReportFindsMassive2011], economics [@jensenThereReplicationCrisis2023], economics [@briggsPartialSolutionReplication2023], sociology [@auspurgAusmassUndRisikofaktoren2014] or even medicine [@begleyRaiseStandardsPreclinical2012], with some authors even claiming that most published research findings in the social sciences are false [@ioannidisWhyMostPublished2005]. Criminology sits at the intersections of these literatures rather than apart from them, which is why we draw on evidence from across the social sciences in what follows. But what drives this crisis?
## Questionable Research
One of the earlier discussed practices that distort scientific progress is called publication bias. @rosenthalFileDrawerProblem1979 defines it as the preference for publishing positive over negative or inconclusive results. This, often institutionally driven bias, also called the 'file drawer problem', can occur at any stage in research [@kuhbergerPublicationBiasPsychology2014; @francoPublicationBiasSocial2014]. Contributing practices include selective reporting, where null findings or variables are omitted from analysis [@breznauDoesSociologyNeed2021] and the post-hoc adaptation of hypotheses [@gerberPublicationBiasEmpirical2008]. But @breznauObservingManyResearchers2022 don't see publication bias as the main driver of the huge variance in results. Instead, they emphasize the role of idiosyncratic researcher variability and the broader context of research practices, leads to the problem of science practices that might produce unreliable or invalid results: so-called questionable research practices (QRP).
One of the earlier discussed practices that distort scientific progress is called publication bias. @rosenthalFileDrawerProblem1979 defines it as the preference for publishing positive over negative or inconclusive results. This, often institutionally driven bias, also called the 'file drawer problem', can occur at any stage in research [@kuhbergerPublicationBiasPsychology2014; @francoPublicationBiasSocial2014]. Contributing practices include selective reporting, where null findings or variables are omitted from analysis [@breznauDoesSociologyNeed2021] and the post-hoc adaptation of hypotheses [@gerberPublicationBiasEmpirical2008]. But @breznauObservingManyResearchers2022 don't see publication bias as the main driver of the huge variance in results. Instead, they emphasize the role of RDOF and the broader context of research practices, leads to the problem of science practices that might produce unreliable or invalid results: so-called questionable research practices (QRP).
In their excellent manifesto for reproducible science, @munafoManifestoReproducibleScience2017 conceptionalize QRPs along the typical stages of empirical research. Common QRPs include HARKing or presenting an unexpected exploratory finding as a preplanned hypothesis, p-hacking or manipulating data or analysis to achieve a desired p-value and selective reporting, that is not reporting studies or variables that lack significant results. Other QRPs involve undisclosed data exclusion, stopping data collection when a desired result is found, or not reporting all conditions or measures used. These practices inflate false-positive rates and undermine research credibility [@auspurgAusmassUndRisikofaktoren2014; @breznauDoesSociologyNeed2021; @chinQuestionableResearchPractices2023].
Problematic practices involve the misuse of p-values, where researchers simply misinterpret the significance level as the likelihood of truth in their findings, leading to vast overconfidence in their results-that can also be a consequence of or lead to a failure to control for bias and poor quality control [@breznauDoesSociologyNeed2021; @munafoManifestoReproducibleScience2017]. Demographic, geographic or political biases and peer review limitations are more sources for error [@breznauDoesSociologyNeed2021; @grossmannOpenScienceReform2021]. Additionally, gendered penalties favor men publishing disproportionately more than women [@akbaritabarGenderPatternsPublication2021]. Misaligned institutional incentives, also accelerated by an intense competition for academic jobs, tenure and funding, lead to a so-called publish or perish culture [@smaldinoOpenScienceModified2019; @breznauDoesSociologyNeed2021].
A truth-incentivizing survey of over 2000 psychologists revealed a high prevalence of QRPs. Around 60% admitted to not reporting all dependent measures, 50% to selective reporting, and 30% to falsely claiming they predicted an unexpected finding. About 2% even confessed to data falsification [@johnMeasuringPrevalenceQuestionable2012a]. Criminology shows similar patterns, though the study of @chinQuestionableResearchPractices2023 recorded lower rates of admission, likely because its design lacked the truth-incentivizing mechanisms used by @johnMeasuringPrevalenceQuestionable2012a.
A truth-incentivizing survey of over 2000 psychologists revealed a high prevalence of QRPs. Roughly 60% admitted to not reporting all dependent measures, 50% to selective reporting, and 30% to falsely claiming they predicted an unexpected finding. About 2% even confessed to data falsification [@johnMeasuringPrevalenceQuestionable2012a]. A comparable study in Criminology by @chinQuestionableResearchPractices2023 recorded remarkably lower rates of admission, but this likely reflects methodological rather than substantive differences: the design lacked the truth-incentivizing mechanisms used by @johnMeasuringPrevalenceQuestionable2012a. Additionally, given the sensitivity of the topic and the probable underrepresentation of QRP-engaging researchers in voluntary surveys, both sets are best read as lower bounds on true prevalence.
All the above leads to the conclusion, that our institutions make refutation harder than confirmation. Open science (OS) is the design response, resetting defaults to transparency, pre-specification, and reproducibility. @munafoManifestoReproducibleScience2017 translate that philosophy into a lifecycle blueprint: blinding and preregistration, stronger methods training and independent oversight, open data, code and diversified peer review to harden reproducibility, evaluation and other measures. The central movement to address the above issues is the OS movement, devoting its effort to challenge publication bias, low statistical power, p-hacking, HARKing and other problems by increasing reproducibility and transparency [@grossmannOpenScienceReform2021].
All the above leads to the conclusion, that our institutions make refutation harder than confirmation. Open science (OS) is the design response, resetting defaults to transparency, pre-specification, and reproducibility. @munafoManifestoReproducibleScience2017 translate that philosophy into a lifecycle blueprint: blinding and PR, stronger methods training and independent oversight, open data, code and diversified peer review to harden reproducibility, evaluation and other measures. The central movement to address the above issues is the OS movement, devoting its effort to challenge publication bias, low statistical power, p-hacking, HARKing and other problems by increasing reproducibility and transparency [@grossmannOpenScienceReform2021].
## Open Science Practices
Following an extensive literature review, @vicente-saezOpenScienceNow2018a characterize OS using four differentias: transparency in communication, accessibility or searchability to all data and materials, sharing of everything with a commitment to do so and collaboration along a scientific, distributed global dialogue throughout all stages involved in science. They integrate these into a succinct definition: "Open Science is transparent and accessible knowledge that is shared and developed through collaborative networks" [@vicente-saezOpenScienceNow2018a, p. 434]. @banksAnswers18Questions2019 establish a broader definition of os that refers to many concepts, including scientific philosophies embodying communality and universalism, specific practices operationalizing these norms including os policies. A common ground is that *open* science and OSPs try to prevent research misconduct by simply increasing research transparency.
Following an extensive literature review, @vicente-saezOpenScienceNow2018a characterize OS using four differentias: transparency in communication, accessibility or searchability to all data and materials, sharing of everything with a commitment to do so and collaboration along a scientific, distributed global dialogue throughout all stages involved in science. They integrate these into a succinct definition: "Open Science is transparent and accessible knowledge that is shared and developed through collaborative networks" [@vicente-saezOpenScienceNow2018a, p. 434]. @banksAnswers18Questions2019 establish a broader definition of OS that refers to many concepts, including scientific philosophies embodying communality and universalism, specific practices operationalizing these norms including OS policies. A common ground is that *open* science and OSPs try to prevent research misconduct by simply increasing research transparency.
Building on these definitions, in line with the work of many other authors from diverse disciplines [e.g. @dienlinAgendaOpenScience2021 and @greenspanOpenSciencePractices2024], there are numerous practices that have been proposed to enact OS.
### Open Data and Open Materials
*Open data* and *open materials* enable replication by publishing all materials necessary to reproduce research in detail, finding errors, bias or simply support the results of the replicated work [@dienlinAgendaOpenScience2021].
*Open data* and *open materials* enable computational reproduction and, where new data are collected, replication, by publishing all materials necessary to reproduce research in detail, finding errors, bias or simply support the results of the reproduced work [@dienlinAgendaOpenScience2021].
**Open data** (OD) is defined as *the sharing of data that was collected, generated or obtained from a third party and processed to investigate the research question assessed in the publication*. Open materials are often shared alongside open data. To delineate a differentiated picture as sharing behavior for data and materials can be expected to differ due to for example privacy concerns, **open materials** (OM) are distinctively defined as *all research materials necessary to reproduce the reported results like notebooks, code or syntax, guides, protocols that can be shared digitally*. Both definitions closely follow the definitions given by the @americanpsychologicalassociationOpenScienceBadges.
**Open data** (OD) is defined as *the sharing of data that was collected, generated or obtained from a third party and processed to investigate the research question assessed in the publication*. Open materials are often shared alongside open data. To delineate a differentiated picture as sharing behavior for data and materials can be expected to differ due to for example privacy concerns, **open materials** (OM) are distinctively defined as *all research materials necessary to reproduce the reported results like notebooks, code or syntax, guides, protocols that can be shared digitally*. Both definitions closely follow the definitions given by the @americanpsychologicalassociationOpenScienceBadges. OD and OM are supported by a growing body of evidence that suggests they can improve the quality and impact of research.
OD and OM have several benefits. First, there is accumulating evidence that providing data alongside publications increases visibility and impact. Some estimates suggest around a 30% citation increase for papers that share data, and importantly, this advantage appears at least partly independent of JIF [@tennantAcademicEconomicSocietal2016; @banksAnswers18Questions2019]. Beyond citations, openly available datasets enable the exploration by others, supporting novel findings and exploratory, hypothesis-generating work [@piwowarSharingDetailedResearch2007; @piwowarStateOALargescale2018].
First, research increasingly confirms that providing data alongside publications increases visibility and impact. Some estimates suggest around a 30% citation increase for papers that share data, and importantly, this advantage appears at least partly independent of journal impact factor (JIF) [@tennantAcademicEconomicSocietal2016; @banksAnswers18Questions2019]. Beyond citations, openly available datasets enable the exploration by others, supporting novel findings and exploratory, hypothesis-generating work [@piwowarSharingDetailedResearch2007; @piwowarStateOALargescale2018].
Second, openness improves methodological rigor and documentation. Knowing that others will inspect our code, data, and decisions incentivizes clearer documentation, more careful workflows, and fewer statistical errors in final papers [@tennantAcademicEconomicSocietal2016; @banksAnswers18Questions2019]. This also promotes transparency about analytic choices and potential biases [@breznauDoesSociologyNeed2021].
Third, OD and OM reinforce field credibility. By allowing independent examination of methods and results, openness reduces the chance that findings are based on idiosyncratic decisions or unreported researcher degrees of freedom [@scogginsMeasuringTransparencySocial2024; @breznauObservingManyResearchers2022]. Multiple sources suggest that open practices reduce QRPs overall [@scogginsMeasuringTransparencySocial2024; @tennantAcademicEconomicSocietal2016; @munafoManifestoReproducibleScience2017].
Third, OD and OM reinforce field credibility. By allowing independent examination of methods and results, openness reduces the chance that findings are based on idiosyncratic decisions or unreported RDOF [@scogginsMeasuringTransparencySocial2024; @breznauObservingManyResearchers2022]. Multiple sources suggest that open practices reduce QRPs overall [@scogginsMeasuringTransparencySocial2024; @tennantAcademicEconomicSocietal2016; @munafoManifestoReproducibleScience2017].
Finally, both have economic and societal benefits, that are even more evident for open access (OA). They discourage redundant data collection, enabling cost savings that can be redirected to new research questions [@tennantAcademicEconomicSocietal2016; @piwowarSharingDetailedResearch2007]. At the same time, the public availability of data stimulates methodological innovation and cross-dataset syntheses that would otherwise remain infeasible [@piwowarStateOALargescale2018]. These dynamics amplify the academic, economic, and societal impact of research [@tennantAcademicEconomicSocietal2016].
@@ -103,27 +103,27 @@ Despite these gains, legitimate concerns persist among many researchers. With in
There are also method-specific hurdles. For qualitative research, transparency can be especially challenging when meaning-making is relational and context-dependent. Fieldnotes and transcripts may lose essential value once separated from the researcher and participants [@breznauDoesSociologyNeed2021; @freeseReplicationSocialScience2017]. These issues underscore that one-size-fits-all mandates are unlikely to succeed.
In short, many systemic and researcher-centric challenges cut across OSPs - and they will reappear in the discussion of preregistration that follows.
In short, many systemic and researcher-centric challenges cut across OSPs - and they will reappear in the discussion of PR that follows.
### Preregistration
A preregistration is a time-stamped plan for a study's hypotheses, design, and analysis, often made public. Its contents vary by method (e.g., hypotheses, sampling, interview guides, exclusion rules, analysis plans) [@loggPreregistrationWeighingCosts2021; @managoPreregistrationRegisteredReports2023; @americanpsychologicalassociationOpenScienceBadges].
A PR is a time-stamped plan for a study's hypotheses, design, and analysis, often made public. Its contents vary by method (e.g., hypotheses, sampling, interview guides, exclusion rules, analysis plans) [@loggPreregistrationWeighingCosts2021; @managoPreregistrationRegisteredReports2023; @americanpsychologicalassociationOpenScienceBadges].
Timestamping restrains HARKing by separating predictions from evidence, reducing the flexibility for post-hoc theorizing [@scogginsMeasuringTransparencySocial2024; @loggPreregistrationWeighingCosts2021]. More broadly, by committing ex ante, researcher degrees of freedom are narrowed. The analytic and design choices that otherwise enable selective reporting or specification searching are constrained, and any deviations become visible to readers and reviewers. The same logic limits p-hacking: when transformations, outlier rules, model families, covariates, and confirmatory contrasts are specified in advance, cherry-picking becomes less feasible because analytical decisions are made independently of the data. Preregistration also addresses structural issues of study quality. Declaring sample-size requirements upfront helps prevent underpowered designs at construction [@kuhbergerPublicationBiasPsychology2014; @grossmannOpenScienceReform2021]. We predefine theory, measures, and analyses, seek early input, and document choices so reviewers can vet them and avoid misinterpretation-strengthening credibility [ @evansImprovingEvidencebasedPractice2023; @sarafoglouSurveyHowPreregistration2022; @scogginsMeasuringTransparencySocial2024]. Preregistration helps separate confirmatory from exploratory work, reduces publication bias (e.g., via Registered Reports), and narrows researcher degrees of freedom [@simmonsFalsePositivePsychologyUndisclosed2011].
Timestamping restrains HARKing by separating predictions from evidence, reducing the flexibility for post-hoc theorizing [@scogginsMeasuringTransparencySocial2024; @loggPreregistrationWeighingCosts2021]. More broadly, by committing ex ante, RDOF are narrowed. The analytic and design choices that otherwise enable selective reporting or specification searching are constrained, and any deviations become visible to readers and reviewers. The same logic limits p-hacking: when transformations, outlier rules, model families, covariates, and confirmatory contrasts are specified in advance, cherry-picking becomes less feasible because analytical decisions are made independently of the data. PR also addresses structural issues of study quality. Declaring sample-size requirements upfront helps prevent underpowered designs at construction [@kuhbergerPublicationBiasPsychology2014; @grossmannOpenScienceReform2021]. We predefine theory, measures, and analyses, seek early input, and document choices so reviewers can vet them and avoid misinterpretation-strengthening credibility [@evansImprovingEvidencebasedPractice2023; @sarafoglouSurveyHowPreregistration2022; @scogginsMeasuringTransparencySocial2024]. PR helps separate confirmatory from exploratory work, reduces publication bias (e.g., via Registered Reports), and narrows RDOF [@simmonsFalsePositivePsychologyUndisclosed2011].
For this work, **preregistration** is defined as *the act of planning and documenting the hypotheses, study design, and analysis plan of a study before data is collected or even viewed. The documentation is typically time-stamped and made publicly available*.
For this work, **PR** is defined as *the act of planning and documenting the hypotheses, study design, and analysis plan of a study before data is collected or even viewed. The documentation is typically time-stamped and made publicly available*.
The Open Science movement, particularly preregistration, has been criticized for not providing tailored transparency practices for qualitative research and for importing a positivist framework that may not fit all traditions [@breznauDoesSociologyNeed2021]. Nevertheless, the core principle of transparency remains relevant: qualitative reports should contain enough information for another researcher to understand the logic and process behind the findings [@breznauDoesSociologyNeed2021]. In qualitative contexts, preregistration can focus on documenting guiding questions, sampling logic, coding frameworks, and decision trails while remaining compatible with iterative analysis.
The Open Science movement, particularly PR, has been criticized for not providing tailored transparency practices for qualitative research and for importing a positivist framework that may not fit all traditions [@breznauDoesSociologyNeed2021]. Nevertheless, the core principle of transparency remains relevant: qualitative reports should contain enough information for another researcher to understand the logic and process behind the findings [@breznauDoesSociologyNeed2021]. In qualitative contexts, PR can focus on documenting guiding questions, sampling logic, coding frameworks, and decision trails while remaining compatible with iterative analysis.
Frequently voiced concerns are about increasing work, thereby lengthening projects and restricting researcher's freedom by confining them to their predefined plan. However, preplanning simply reorders the workflow rather than creating extra work, potentially preventing costly redesigns or follow-up-studies. Additionally, this does not inhibit exploratory work as the goal is to provide clarity and transparency by distinguishing between preplanned analysis and those conducted after viewing the data. By moving the conceptual work upstream, preregistration clarifies claims, adds transparency to the decision process and strengthens credibility by marking plans and deviations [@loggPreregistrationWeighingCosts2021; @evansImprovingEvidencebasedPractice2023]. In-principle acceptance adds a guarantee to the upfront work, provided the approved plan is followed [@sarafoglouSurveyHowPreregistration2022; @banksAnswers18Questions2019].
Frequently voiced concerns are about increasing work, thereby lengthening projects and restricting researcher's freedom by confining them to their predefined plan. However, preplanning simply reorders the workflow rather than creating extra work, potentially preventing costly redesigns or follow-up studies. Additionally, this does not inhibit exploratory work as the goal is to provide clarity and transparency by distinguishing between preplanned analysis and those conducted after viewing the data. By moving the conceptual work upstream, PR clarifies claims, adds transparency to the decision process and strengthens credibility by marking plans and deviations [@loggPreregistrationWeighingCosts2021; @evansImprovingEvidencebasedPractice2023]. In-principle acceptance adds a guarantee to the upfront work, provided the approved plan is followed [@sarafoglouSurveyHowPreregistration2022; @banksAnswers18Questions2019].
In summary, preregistration does not constrain scientific creativity, it clarifies claims. By making the sequence of decisions explicit-what was planned, what changed, and why-we reduce bias, improve interpretability, and strengthen confidence in reported findings [@hardwickeReducingBiasIncreasing2023].
In summary, PR does not constrain scientific creativity, it clarifies claims. By making the sequence of decisions explicit-what was planned, what changed, and why-we reduce bias, improve interpretability, and strengthen confidence in reported findings [@hardwickeReducingBiasIncreasing2023].
### Open Access {#sec-open-access}
**Open access** is a key OSP, defined as making research freely available online to anyone, as opposed to requiring payment via journal subscriptions [@banksAnswers18Questions2019, @breznauDoesSociologyNeed2021]. The Budapest OA Initiative defines OA as being free to read and reuse for lawful purposes, including text and data mining [@BOAI2002]. A simpler, broad definition is the lawful free availability of a research publication on the internet which will be used here.
**Open access** is a key OSP, defined as making research freely available online to anyone, as opposed to requiring payment via journal subscriptions [@banksAnswers18Questions2019; @breznauDoesSociologyNeed2021]. The Budapest OA Initiative defines OA as being free to read and reuse for lawful purposes, including text and data mining [@BOAI2002]. A simpler, broad definition is the lawful free availability of a research publication on the internet which will be used here.
OA publishing offers several benefits. It increases accessibility and equity, as anyone with an internet connection can reach an OA article, potentially reducing inequalities for those at underfunded institutions [@banksAnswers18Questions2019]. There is a significant OA citation advantage, as OA articles are cited more frequently than closed-access publications. This preference is now considered a form of research bias known as "FUTON" (full text on the net) bias [@piwowarStateOALargescale2018, @wentzVisibilityResearchFUTON2002; @piwowarSharingDetailedResearch2007]. OA also improves research quality by reducing the suppression of null findings [@francoPublicationBiasSocial2014] and enabling large-scale text and data mining [@tennantAcademicEconomicSocietal2016]. Furthermore, it accelerates equitable access, helping to bridge the global North-South divide, and enhances public accountability for publicly funded research [@tennantAcademicEconomicSocietal2016].
OA publishing offers several benefits. It increases accessibility and equity, as anyone with an internet connection can reach an OA article, potentially reducing inequalities for those at underfunded institutions [@banksAnswers18Questions2019]. There is a significant OA citation advantage, as OA articles are cited more frequently than closed-access publications. This preference is now considered a form of research bias known as "FUTON" (full text on the net) bias [@piwowarStateOALargescale2018; @wentzVisibilityResearchFUTON2002; @piwowarSharingDetailedResearch2007]. OA also improves research quality by reducing the suppression of null findings [@francoPublicationBiasSocial2014] and enabling large-scale text and data mining [@tennantAcademicEconomicSocietal2016]. Furthermore, it accelerates equitable access, helping to bridge the global North-South divide, and enhances public accountability for publicly funded research [@tennantAcademicEconomicSocietal2016].
Despite its benefits, OA faces challenges. Some newer or smaller Gold OA journals are perceived as less prestigious [@piwowarStateOALargescale2018], and concerns about "predatory publishers" have been mistakenly linked with OA [@tennantAcademicEconomicSocietal2016]. Article processing charges (APCs) can be a barrier for authors, particularly in low- and middle-income countries [@banksAnswers18Questions2019; @breznauDoesSociologyNeed2021], though roughly 70% of peer-reviewed OA journals are fee-free, and many offer waivers [@tennantAcademicEconomicSocietal2016; @breznauDoesSociologyNeed2021]. Publishers may also be hesitant to adopt OA due to concerns about losing subscription revenue. While OA promotes transparency, it cannot solve issues like QRPs or underpowered studies on its own, if incentives continue to reward quantity over quality [@grossmannOpenScienceReform2021; @banksAnswers18Questions2019].
@@ -131,19 +131,17 @@ Despite its benefits, OA faces challenges. Some newer or smaller Gold OA journal
A focused literature review on adoption produced limited evidence as we still know surprisingly little about how often OSPs are actually used in criminology and legal psychology. The evidence is fragmented, method-dependent, and sometimes contradictory - so estimates of prevalence are shaky, even as enthusiasm for OSPs is high and QRPs might appear common.
Self-reports suggest high OSP familiarity - but they co-exist with widespread QRPs and are vulnerable to bias. In the study conducted by @chinQuestionableResearchPractices2023, 89% of respondents said they had used at least one OSP, yet 87% also admitted at least one QRP, and some serious QRPs (e.g., hiding known problems) were non-trivial. Survey data indicate that about 25% of researchers across fields have preregistered a study, with higher uptake in psychology (50-60%) and lower prevalence in sociology (~30%) [@fergusonSurveyOpenScience2023a]. Another survey in the field similarly estimated preregistration use at 45% (42-49%) [@chinQuestionableResearchPractices2023]. The reported prevalence of OD varies widely across disciplines. Survey data suggest that more than 60% of researchers report having posted data or code, with higher rates in psychology (>50%) compared to sociology (~35%) [@fergusonSurveyOpenScience2023a]. The prevalence of OM sharing is more limited compared to OD and access. Survey results indicate that 43% (40-47%) of researchers report providing access to their research materials [@chinQuestionableResearchPractices2023]. Few or no journals require data sharing in the field, coupled with rare preregistration and a tiny share of replication studies [@pridemoreReplicationCriminologySocial2018].
Self-reports suggest high OSP familiarity - but they co-exist with widespread QRPs and are vulnerable to bias. In the study conducted by @chinQuestionableResearchPractices2023, 89% of respondents said they had used at least one OSP, yet 87% also admitted at least one QRP, and some serious QRPs (e.g., hiding known problems) were non-trivial. Survey data indicate that about 25% of researchers across fields have preregistered a study, with higher uptake in psychology (50-60%) and lower prevalence in sociology (~30%) [@fergusonSurveyOpenScience2023a]. Another survey in the field similarly estimated PR use at 45% (42-49%) [@chinQuestionableResearchPractices2023]. The reported prevalence of OD varies widely across disciplines. Survey data suggest that more than 60% of researchers report having posted data or code, with higher rates in psychology (>50%) compared to sociology (~35%) [@fergusonSurveyOpenScience2023a]. The prevalence of OM sharing is more limited compared to OD and access. Survey results indicate that 43% (40-47%) of researchers report providing access to their research materials [@chinQuestionableResearchPractices2023]. Few or no journals require data sharing in the field, coupled with rare PR and a tiny share of replication studies [@pridemoreReplicationCriminologySocial2018].
In their survey at the Netherlands Institute for the Study of Crime and Law Enforcement, @moneva2025attitudes find broadly positive attitudes but divergent views by method and career stage, and a long list of cultural, structural, legal/privacy, and cost barriers. @fessingerStateOpenScience2025 also shows strong approval (88% positive) and some experience (58% tried at least one OSP), but routine adoption looks limited (only 44% even hold a repository account). In contrast, an assessment of social science studies between 2014 and 2017 found no preregistered studies at all [@hardwickeEmpiricalAssessmentTransparency2020].
Article audits show far lower OSP uptake than surveys, implying either nondisclosure or overestimation. @greenspanOpenSciencePractices2024 coded 722 articles (2018-2022) across five leading journals and found OM in about a third of papers, but \<10% with OD, \<2% with open code or preregistration, and no upward trend.
Article audits show far lower OSP uptake than surveys, implying either nondisclosure or overestimation. @greenspanOpenSciencePractices2024 coded 722 articles (2018-2022) across five leading journals and found OM in about a third of papers, but \<10% with OD, \<2% with open code or PR, and no upward trend.
Put together, we have: (a) structural signals that transparency norms aren't yet embedded, (b) surveys that likely overstate or at least poorly calibrate actual practice; (c) parallel evidence from legal psychology that approval is high but practical barriers keep routine use patchy and (d) little to no evidence of actual os practice, opposed to plain opinion.
The applied nature of the research in this field means fragile findings can drive high-stakes policy and practice. Single studies have shaped policing responses (e.g., the Minnesota Domestic Violence study by @shermanSpecificDeterrentEffects1984) only to be refuted by later replications, underscoring the risks of acting on unverified results [@mcneeleyReplicationCriminologyNecessary2015]. The relative youth of criminology and incentives that privilege novelty further heighten the need for systematic replication. To enable it, we should adopt measures [@mcneeleyReplicationCriminologyNecessary2015]. Given how little is known about the prevalence of OSPs in the field and the indicators we see for widespread QRPs, there is a strong case for prioritizing replication-and thereby a need to take stock.
Put together, we have: (a) structural signals that transparency norms aren't yet embedded, (b) surveys that likely overstate or at least poorly calibrate actual practice, (c) parallel evidence from legal psychology that approval is high but practical barriers keep routine use patchy and (d) little to no evidence of actual OS practice, opposed to plain opinion. The applied nature of the research in this field means that fragile findings can drive high-stakes policy and practice. Single studies, like the Minnesota Domestic Violence study by @shermanSpecificDeterrentEffects1984, have shaped policing responses only to be refuted by later replications, emphasizing the risks of acting on unverified results [@mcneeleyReplicationCriminologyNecessary2015]. The relative youth of criminology and incentives that privilege novelty further increase the need for systematic replication. To enable it, we should adopt measures [@mcneeleyReplicationCriminologyNecessary2015]. Given how little is known about the prevalence of OSPs in the field and the indicators we see for widespread QRPs, there is a strong case for prioritizing replication - and thereby a need to take stock.
# Data and Method
The aim of this work is to compile a sample of publications in the fields of criminology and legal psychology, classify it as either statistical inference (SI) publications or non-SI publications and further examine the former to assess whether any of the OSPs under consideration are used: preregistration, OD, OM, or OA. OA results are reported as secondary, descriptive analyses to benchmark open-science adoption. The presented OSPs will be operationalized and a text-classification pipeline (keyword dictionaries and machine-learning models) will be used to detect them. OA status will be determined using publicly available metadata, given the expected high reliability of information on OA. The fine-tuned models are validated against a hand-coded sample that is extended using a large-language-model (LLM, ChatGPT 4o & ChatGPT 5o), with the product of both being then used to train classifier models that will classifiy the analytical sample to estimate true prevalences of OSPs. The full research process is illustrated in @fig-flowchart-pipeline.
The aim of this work is to compile a sample of publications in the fields of criminology and legal psychology, classify it as either statistical inference (SI) publications or non-SI publications and further examine the former to assess whether any of the OSPs under consideration are used: PR, OD, OM, or OA. OA results are reported as secondary, descriptive analyses to benchmark open-science adoption. The presented OSPs will be operationalized and a text-classification pipeline (keyword dictionaries and machine-learning models) will be used to detect them. OA status will be determined using publicly available metadata, given the expected high reliability of information on OA. The fine-tuned models are validated against a hand-coded sample that is extended using a large-language-model (LLM, ChatGPT 4o & ChatGPT 5o), with the product of both being then used to train classifier models that will classifiy the analytical sample to estimate true prevalences of OSPs. Machine learning models learn patterns from a training set, a labeled subset of articles and are then tested on a validation set, a separate labeled subset withheld during training, to assess whether the learned patterns generalize rather than merely memorize the training examples [@robertsCrossvalidationStrategiesData2017]. The full research process is illustrated in @fig-flowchart-pipeline. A thorough description or discussion of the methods, including the sampling procedure, data collection, and classification pipeline is provided in the methodological report. The following sections provide a summary of the key aspects of the methods.
Full-text data for training the machine learning classification models will be collected with a web application developed specifically for this project. Since software development is not the focus of this work, details of the app's architecture will not be discussed here. A brief description of the application, along with screenshots, is provided in the supplementary material.
@@ -151,7 +149,7 @@ This work is necessarily scoped by time and resources. It shall therefore be tre
## Population
The scope of this work encompasses all publications from the top 100 journals classified under "Criminology & Penology" or the journals that are categorized as "Law" (which might also include sociologically or psychologically driven quantitative studies) and "Psychology, Multidisciplinary", ranked by the 2023 JIF according to Clarivate's Journal Citation Reports [@clarivateJournalImpactFactor2023] that rely on SI. Publication metadata were retrieved via the Crossref API. While Crossref provides extensive coverage, it is not exhaustive, and prior work has shown that missing records are often systematic rather than random [@delgado-quirosWhyAreThese2024; @hausteinWhenArticleActually2015]. Using multiple bibliographic sources (e.g., Scopus, Web of Science) would reduce this bias [@gerasimovComparisonDatasetsCitation2024; @delgado-quirosWhyAreThese2024], but this was not feasible within the scope of this pilot. Consequently, the study population is restricted to articles indexed in Crossref from the selected top 100 journals.
The scope of this work encompasses all publications from the top 100 journals classified under "Criminology & Penology", the journals that are categorized as "Law" and "Psychology, Multidisciplinary", ranked by the 2023 JIF according to Clarivate's Journal Citation Reports (JCR) [@clarivateJournalImpactFactor2023] that rely on SI. Law journals were included because Clarivate's "Law" category contains a substantial body of legal psychology research relevant to our scope as for example shown in the analyses of journals by @gonzalez-salaCaracterizacionPsicologiaJuridica2017. Publication metadata were retrieved via the Crossref API. While Crossref provides extensive coverage, it is not exhaustive, and prior work has shown that missing records are often systematic rather than random [@delgado-quirosWhyAreThese2024; @hausteinWhenArticleActually2015]. Using multiple bibliographic sources (e.g., Scopus, Web of Science) would reduce this bias [@gerasimovComparisonDatasetsCitation2024; @delgado-quirosWhyAreThese2024], but this was not feasible within the scope of this pilot. Consequently, the study population is restricted to articles indexed in Crossref from the selected top 100 journals.
As the population is restricted to publications that make SIs, this concept has to be clearly defined mostly in line with @scogginsMeasuringTransparencySocial2024, as works that rely on data, statistical analysis and experiments. @scogginsMeasuringTransparencySocial2024 restricted further on only experiments, which was deemed not necessary as all assessed OSPs are suitable to be used and should be used in not only experiments, but also in works assessing second-hand data or alike [@akkerPreregistrationSecondaryData2021; @westonRecommendationsIncreasingTransparency2019]. Thereby, descriptive, correlational, comparative and other non-purely theoretical research was included.
@@ -168,8 +166,8 @@ The sample size was determined by a precision-based calculation to ensure a $\pm
```{mermaid}
%%| label: fig-flowchart-pipeline
%%| fig-cap: "A flowchart of the full research process. All steps described are further explained in the methodologic report and the supplements."
%%| fig-width: 4.2
%%| fig-height: 8
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A["Population
Top 100 JIF journals 2013-2023"]
@@ -199,36 +197,35 @@ Random Forest / XGBoost
TF-IDF keyword features"]
H1["Full-text retrieval
HTML / PDF · custom web app"]
HTML / PDF, scraped"]
H2["Full-text retrieval
HTML / PDF · custom web app"]
HTML / PDF, scraped"]
I["SI classifier applied to Sample B"]
I2["SI papers identification
Fine-tuned SI classifier
n = 1,763 with usable full text"]
OA["OA classified from metadata
using Crossref, Web of Science, Scopus
Covers full Sample B"]
using Crossref, Web of Science, Scopus"]
J["OSP training subset
n = 352 · drawn from SI papers in Sample B
Balanced by year
n = 352, from SI papers in Sample B
manual & LLM labelling"]
KOD["OD classifier
RF / XGBoost · grid-tuned"]
RF / XGBoost"]
KOM["OM classifier
RF / XGBoost · grid-tuned"]
RF / XGBoost"]
KPR["PR classifier
RF / XGBoost"]
L["Prevalence estimates, Post-stratified by year, adjusted for misclassification"]
KPR["Preregistration classifier
RF / XGBoost · grid-tuned"]
L["Design-based prevalence estimates, Post-stratified by year, adjusted for misclassification"]
A --> B
B -- Training/Testing Sample --> D
@@ -242,9 +239,11 @@ J --> KOD & KOM & KPR
KOD & KOM & KPR -- Applied to all SI papers --> L
OA --> L
J --> L
L ~~~ invisible[ ]
style invisible fill:none,stroke:none,color:none
```
First, Sample A, a random sample of up around 500 publications was manually classified to train the initial SI classifier. This step also helped estimate the effort for subsequent tasks. Next, an independent Sample B was drawn, stratified by year, thereby addressing problems in cross-validation and the non-independence of residuals assumptions of many machine-learning models [@robertsCrossvalidationStrategiesData2017].
First, Sample A, a random sample of up around 500 publications was manually classified to train the initial SI classifier. This step also helped estimate the effort for subsequent tasks. Next, an independent Sample B was drawn, stratified by year, thereby addressing problems in cross-validation (resampling of the sample to multiple smaller training and validation subsets) and the non-independence of residuals assumptions of many machine-learning models [@robertsCrossvalidationStrategiesData2017].
The SI classifier was then used to analyze and classify all publications in Sample B. From the identified SI papers in Sample B, a balanced dataset was randomly sampled to create a training set for the OSP classifiers. Finally, these trained OSP classifiers were applied to the entire analytical Sample B. While a publisher or journal-based stratification for the full sample would have been ideal, it was not feasible due to the limited number of available full texts.
@@ -258,7 +257,7 @@ Before the full text data could be collected, some steps were necessary in order
#| echo: false
#| results: asis
#| tbl-cap: Cases Dropped from all Publications Obtained
#| label: tbl-cases
#| label: tbl-01-cases
tbl <- read_csv("data/tbl-sample-case-drops.csv")
@@ -325,10 +324,10 @@ if (isTRUE(debug_mode)) {
}
```
The data obtained necessitated multiple transformations. All transformations are reported in the respective section in the methodological report. Publications were filtered by the resulting date variable to limit the population to the defined time interval. To reduce SI coding efforts, simple keyword-lists were used to reduce the number of publications by matching titles. Missing values were assessed, checks were processed for language, @tbl-cases shows that from an initial number of 95042 publications, all steps resulted in a final publication count of 40,860. It is important to note here that several improvements were implemented here but not processed. More details can be found in the provided materials.
The data obtained necessitated multiple transformations. All transformations are reported in the respective section in the methodological report as stated in the data availability statement. Publications were filtered by the resulting date variable to limit the population to the defined time interval. To reduce manual SI coding efforts, simple keyword-lists were used to reduce the number of publications by matching titles. Missing values were assessed and language filters were processed. @tbl-01-cases shows that from an initial number of 95042 publications, all steps resulted in a final publication count of 40,860. Note that, owing to caching difficulties, several planned filtering refinements did not execute at this stage - a deviation transparently documented in the methodological report. The resulting population is therefore somewhat larger than originally intended, but the surplus publications were removed by subsequent filtering steps. Further details are available in the provided materials.
```{r}
#| label: tbl-cases2
#| label: tbl-02-cases2
#| tbl-cap: Cases Dropped from Analytical Sample
tbl2 <- read_csv("data/tbl-sample-case-drops-stattraining-final.csv")
tbl_cases2 <- tbl2 %>%
@@ -517,27 +516,29 @@ The final analytical sample is made up of 4265 publications. The OS prevalence c
### Full Text Retrieval
The initial approach to gathering full texts, which used Zotero to translate DOIs as per Scoggins and Robertson, was unreliable across multiple attempts and software versions. Due to the unsuitability of existing software tools, be it for technical or legal reasons, a custom web application was developed.
The initial approach to gathering full texts, which used Zotero to translate DOIs as per @scogginsMeasuringTransparencySocial2024, was unreliable across multiple attempts and software versions. Due to the unsuitability of existing software tools, be it for technical or legal reasons, a custom web application was developed.
Downloading the analytical sample was mostly successful, though some publisher protections caused dropouts. Due to time constraints, additional more optimized runs were not feasible. Documents under 1,000 words were considered non-full-text papers. However, shorter HTML texts were retained for potential keyword matching. Text quality assessment (Flesch-Index) and word count identified missing full texts [@benoitQuantedaPackageQuantitative2018]. Full texts were downloaded for Independent Sample A and the Analytical Sample from which Sample B was drawn. The resulting dropouts were expected to have been implicitly handled by post-stratification, but publisher-level weighting was planned and considered but infeasible due to sparse cells that would have produced unstable weights. Post-stratification was conducted by year only, which does not correct publisher- or journal-specific dropouts. Future, non-piloting iterations should add publisher-level adjustment.
## Classification Tasks and Methods
This section will present a brief summary of all methods used to classify the variables of interest. A thorough discussion of the decisions taken, the full descriptions and specifications of the models used as well as the preprocessing steps can be found in the supplied materials. A thorough discussion of all decisions taken, the full descriptions and specifications of the models used as well as the preprocessing steps can be found in the replication materials available in the osf repository.
This section will present a brief summary of all methods used to classify the variables of interest. A thorough discussion of the decisions taken, the full descriptions and specifications of the models used as well as the preprocessing steps can be found in the supplied materials. A thorough discussion of all decisions taken, the full descriptions and specifications of the models used as well as the preprocessing steps can be found in the reproduction materials available in the OSF repository.
Since most existing classification approaches considered were deemed unsuitable for this scope (e.g., @kimResearchPaperClassification2019; @sanguansatFeatureMatricizationDocument2012; @jandotInteractiveSemanticFeaturing2016), this work instead relies on Random-Forest and XGBoost-models trained on a manually and LLM coded subset of publications as LLMs have shown good performance on similar classification tasks [@buntValidatingUseLarge2025; @zhaoAdvancingSingleMultitask2024].
For each task, OSP-specific document-feature-matrices using term frequencies or TF-IDF of keyword sets, partly adapted from @scogginsMeasuringTransparencySocial2024, were constructed.
For each task, OSP-specific document-feature-matrices using term frequencies or term-frequency inverse-document-frequency (TF-IDF[^8]) of keyword sets, partly adapted from @scogginsMeasuringTransparencySocial2024, were constructed.
First, a strict dichotomous operationalization of "SI" or not SI, as well as of the OSPs was synthesized and documented in a short coding manual. A subset of Sample A was coded by hand, followed by a ChatGPT-based labelling of the fulltext. On a random subsample, agreement after reconciliation was high ($\kappa$ $\approx$ .83), so combined manual/LLM labels served as training and test data for the ML classifiers. A similar approach was used for Sample B. Each OSP classifier was tuned on all possible combinations of different feature sets and model.
First, a strict dichotomous operationalization of "SI" or "not SI", as well as of the OSPs was synthesized and documented in a short coding manual. A subset of Sample A was coded by hand, followed by a ChatGPT-based labelling of the fulltext. On a random subsample, agreement of both after reconciliation was high ($\kappa$ $\approx$ .83), so combined manual/LLM labels (classification result variable values) served as training and validation data for the ML classifiers. A similar approach was used for Sample B. Each OSP classifier was tuned on all possible combinations of different feature sets and model.
Given time constraints and the pilot nature of the study, preprocessing and evaluation were optimized for the OSP classifier only, not for the SI classifier. The more rigorous workflow applied to OSP - designed to handle high computational demands and substantial class imbalance - would likely also have improved SI performance, but was not pursued because SI results were already satisfactory, as documented in the provided material. Furthermore, journal-level adoption of OSPs was originally intended to be assessed using the Transparency and Openness Promotion Factor [@nosekPromotingOpenResearch2015]. However, as the available sample sizes were insufficient for journal-level analyses, these were not carried out.
Given time constraints and the pilot nature of the study, preprocessing and evaluation were optimized for the OSP classifier only, not for the SI classifier. The more rigorous workflow applied to OSP - designed to handle high computational demands and substantial class imbalance (unequal distribution of the outcome levels) - would likely also have improved SI performance, but was not pursued because SI results were already satisfactory, as documented in the provided material. Furthermore, journal-level adoption of OSPs was originally intended to be assessed using the Transparency and Openness Promotion Factor [@nosekPromotingOpenResearch2015]. However, as the available sample sizes were insufficient for journal-level analyses, these were not carried out.
[^8]: Term frequency (TF) is the count of a term in a document, while inverse document frequency (IDF) is the logarithm of the total number of documents divided by the number of documents containing the term. TF-IDF is calculated as TF multiplied by IDF, giving higher weight to terms that are frequent in a document but rare across the corpus [@sang-woonResearchPaperClassification2019; @ramosUsingTFIDFDetermine2003].
## Analysis
The research was deliberately designed to study open-science practices via supervised classifiers rather than relying exclusively on metadata. This choice prioritized scalability and the potential to capture practice signals that metadata may miss, at the cost of managing model error and class imbalance. Given the exploratory character of the work, the analyses were not pre-defined, only data collection, sampling, and the model-training strategy were specified in advance. Concerns about classifier interpretability informed the evaluation strategy [@gilpinExplainingExplanationsOverview2018].
Estimates for OSPs are domain-estimates among SI papers (see @tbl-cases2) drawn from a year-stratified random sample, beta-method CIs are based on design-based variance. Design-weights were applied post-stratified to frame-by-year totals with finite-population corrections. All OSP estimates are domain estimates for SI papers using design-based inference. Design corrected 95% confidence intervals are computed with the beta method (Clopper-Pearson) transformation which provides better coverage for low-prevalences than Wald intervals [@agrestiIntroductionCategoricalData2007].
Estimates for OSPs are domain-estimates among SI papers (see @tbl-02-cases2) drawn from a year-stratified random sample, beta-method CIs are based on design-based variance. Design-weights were applied post-stratified to frame-by-year totals with finite-population corrections. All OSP estimates are domain estimates for SI papers using design-based inference. Design corrected 95% confidence intervals are computed with the beta method (Clopper-Pearson) transformation which provides better coverage for low-prevalences than Wald intervals [@agrestiIntroductionCategoricalData2007].
Results generalize to the keyword-filtered data. With $n=1,763$ SI papers, SI-domain CIs are wider than the planned $\pm$ 1.5 pp. Because some SI papers may have been excluded by the screening, OSP levels for all SI papers in the full corpus of 90k publications may differ. An audit of excluded records could quantify the coverage and enable adjustment but was not conducted here.
@@ -547,106 +548,20 @@ Data is reported per year. As per year data given the very low prevalences is ex
# Results & Discussion
Two research questions were formulated: $RQ_1$ on the prevalence of OD and OM among statistical-inference (SI) publications, and $RQ_2$ on the prevalence of preregistration. After extensive model development, validation, calibration, thresholding, and misclassification adjustment, prevalences for OD, OM, and Preregistration were too low for the ML classifiers to yield interpretable, adjusted estimates.
```{r}
#| tbl-cap: Sample Characteristics by Statistical Inference Status
#| label: tbl-sample-char
#| tbl-pos: H
df <- qs_read(file_sample_analysis)
population <- qs_read(file_meta_final)
tbl_sample_desc <- df %>% mutate(
journal_category = case_when(
journal_category == "PSYCHOLOGY, MULTIDISCIPLINARY" ~ "A",
journal_category == "LAW" ~ "B",
journal_category == "CRIMINOLOGY & PENOLOGY" ~ "C"
)) %>%
tbl_summary(
include = c(is_open_access, is_open_data, is_open_materials, is_prereg, txt_source, txt_only_abstract, journal_category, journal_jif_quartile, txt_count, txt_flesch, journal_x2023_jif),
by = is_statistical,
label = list(
is_open_access = "Open Access",
is_open_data = "Open Data",
is_open_materials = "Open Materials",
is_prereg = "Preregistration",
txt_source = "Text Source",
txt_only_abstract = "Only Abstract",
journal_category = "Journal Category",
journal_jif_quartile = "JIF Quartile",
txt_count = "Count: Words",
txt_flesch = "Flesch Score",
journal_x2023_jif = "JIF (2023)",
is_statistical = "Statistical Inference"
),
statistic = list(
all_continuous() ~ "{mean} ({sd})",
all_categorical() ~ "{n} / {N} ({p}%)"
)
) %>%
add_p(
include = c(txt_only_abstract, txt_source, txt_count, txt_flesch, journal_x2023_jif),
test = list(
txt_only_abstract ~ "fisher.test",
txt_source ~ "chisq.test",
txt_count ~ "wilcox.test",
txt_flesch ~ "wilcox.test",
journal_x2023_jif ~ "wilcox.test"
),
pvalue_fun = label_style_pvalue(digits = 3)
) %>%
add_overall() %>%
modify_header(label ~ "**Variable**") %>%
modify_spanning_header(c("stat_1", "stat_2") ~ "**Statistical Inference**")%>%
modify_footnote_body(
footnote = "A: Psychology, Multidisciplinary; B: Law; C: Criminology & Penology",
columns = "label",
rows = variable == "journal_category"
)
if(output_format == "pdf/tex") {
tbl_sample_desc %>%
as_gt() %>%
tab_options(
table.font.size = gt::px(12),
latex.use_longtable = TRUE
)
} else if(output_format == "docx") {
tbl_sample_desc %>%
as_flex_table() %>%
set_table_properties(width = 1) %>%
theme_booktabs(bold_header = TRUE) %>%
align(align = "center", part = "all") %>%
fontsize(size = 11, part = "header") %>%
fontsize(size = 8, part = "body") %>%
width(5, 2.34, unit = "cm") %>%
width(2:4, 3.5, unit = "cm") %>%
autofit() %>%
height_all(height = .2)
} else {
}
if (isTRUE(debug_mode)) {
debug_info[[knitr::opts_current$get("label")]] <-
if (knitr::is_html_output()) "HTML" else "LaTeX"
}
```
Two research questions were formulated: $RQ_1$ on the prevalence of OD and OM among statistical-inference (SI) publications, and $RQ_2$ on the prevalence of PR. After extensive model development, validation, calibration, thresholding, and misclassification adjustment, prevalences for OD, OM, and PR were too low for the ML classifiers to yield interpretable, adjusted estimates.
The ML classifiers trained on GPT labels inherit GPT's strengths and the data's sparsity. For the relatively small 20% validation set coded by GPT, the open-science practice classifiers are less precise and less reliable than the Statistical-Inference classifier. In contrast, a question that was not originally foregrounded proved answerable: the prevalence and trajectory of OA among SI publications, measured from metadata with high reliability, show clear increases over time.
Before misclassification adjustment, design-based prevalences were estimated among SI papers with 95% CIs. For outcomes identified by the ML classifiers (OD, OM, Preregistration), these reflect survey-design uncertainty only.
Before misclassification adjustment, design-based prevalences were estimated among SI papers with 95% CIs. For outcomes identified by the ML classifiers (OD, OM, PR), these reflect survey-design uncertainty only.
@fig-osp-adoption shows a steady rise in OA from \~20% in 2013 to \~50% in 2023, while the other practices suffer from extremely low counts; for some years (e.g., 2013 OD; 2016 Preregistration) estimates were not possible.
@tbl-osp-prev-overall confirms low prevalences across the full period: OA $40.9\%$ (38.8-43.1), OM $4.3\%$ (3.4-5.3), Preregistration $3.6\%$ (2.8-4.5), and OD $2.2\%$ (1.6-2.9).
@tbl-03-osp-prev-overall confirms low prevalences across the full period with OA being the only exception, estimated to be used in $40.9\%$ (38.8-43.1 CI) across the population. OA, OM and OD are all estimated to be available for only less than five percent of the publications.
```{r}
#| fig-cap: OSP Adoption Over Time, among statistical inference papers (design-weighted)
#| label: fig-osp-adoption
#| fig-pos: H
#| fig-width: 7
#| tbl-cap: Overall Prevalence of open science practices among Statistical Inference Papers (Design-Weighted to Frame-by-Year Totals)
#| label: tbl-03-osp-prev-overall
df <- qs_read(file_sample_analysis)
population <- qs_read(file_meta_final)
# ensure that types match
df <- df %>% mutate(published_year = as.integer(published_year))
@@ -704,6 +619,91 @@ vars <- c(
"is_open_access_bin" = "Open Access"
)
overall_results_list <- lapply(names(vars), function(var_name) {
# Create the formula for the specific variable
form <- as.formula(paste0("~", var_name))
# Calculate the proportion and CI on the entire des_stat object
# again, use method = "beta" for robustness
est <- svyciprop(form, design = des_stat, method = "beta", na.rm = TRUE)
# Extract the proportion and confidence interval
p_est <- as.numeric(coef(est))
ci <- as.numeric(confint(est))
# Return a clean tibble with the results
tibble(
osp = paste0(vars[var_name], ""), # Creates the label, e.g., "Prereg (SI)"
p = p_est,
p_low = ci[1],
p_upp = ci[2]
)
})
# Combine the list of results into a single data frame
overall_osp_si_raw <- bind_rows(overall_results_list)
# Apply the final formatting to match your original code
overall_osp_si <- overall_osp_si_raw %>%
mutate(`Prevalence` = sprintf("%.1f%% (%.1f-%.1f)", 100 * p, 100 * p_low, 100 * p_upp)) %>%
select(osp, `Prevalence`)
# Print the final result
overall_osp_si <- overall_osp_si %>%
rename(
OSP = osp
) %>%
arrange(desc(`Prevalence`))
tbl_overall_osp_si <- overall_osp_si %>%
kbl(
format = 'latex',
longtable = TRUE,
booktabs = TRUE,
escape = T,
) %>% # add footnote
column_spec(1, width = '3cm')%>%
kable_styling(
position = "center",
latex_options = "hold_position",
full_width = FALSE) %>%
kableExtra::footnote(
general = "Prevalence estimates in statistical inference publications using design-weights per year (95% CI)",
general_title = "Note:",
footnote_as_chunk = T,
threeparttable = T
)
if(output_format == "pdf/tex") {
print(tbl_overall_osp_si)
} else if(output_format == "docx") {
overall_osp_si %>%
flextable() %>%
set_table_properties(width = 1, layout = "autofit") %>%
theme_booktabs(bold_header = TRUE) %>%
align(align = "center", part = "all") %>%
fontsize(size = 11, part = "header") %>%
fontsize(size = 10, part = "body") %>%
add_footer_lines(values = c("Note: Prevalence estimates in statistical inference publications using design-weights per year (95% CI)"))
} else {
}
if (isTRUE(debug_mode)) {
debug_info[[knitr::opts_current$get("label")]] <-
if (knitr::is_html_output()) "HTML" else "LaTeX"
}
```
@fig-osp-adoption shows a steady rise in OA from \~20% in 2013 to \~50% in 2023, while the other practices suffer from extremely low counts; for some years (e.g., 2013 OD; 2016 PR) estimates were not possible.
```{r}
#| fig-cap: OSP Adoption Over Time, among statistical inference papers (design-weighted)
#| label: fig-osp-adoption
#| fig-pos: H
#| fig-width: 7
# Loop through each variable, run svyby, and collect results in a list
results_list <- lapply(names(vars), function(var_name) {
@@ -797,80 +797,81 @@ if (isTRUE(debug_mode)) {
}
```
In parallel, @tbl-sample-char suggests systematic differences between SI and non-SI papers: distributions of text sources differ (likely reflecting publisher effects or text-quality variation), abstracts-only are more common among non-SI items, word counts are higher for SI papers, journal impact is higher, and OA appears more common. Several contrasts are statistically significant (many $p < .001$), but these should be treated as descriptive given unmodeled multilevel variance and field composition.
In parallel, @tbl-04-sample-char suggests systematic differences between SI and non-SI papers: distributions of text sources differ (likely reflecting publisher effects or text-quality variation), abstracts-only are more common among non-SI items, word counts are higher for SI papers, journal impact is higher, and OA appears more common. Several contrasts are statistically significant (many $p < .001$), but these should be treated as descriptive given unmodeled multilevel variance and field composition.
```{r}
#| tbl-cap: Overall Prevalence of Open Science Practices among Statistical Inference Papers (Design-Weighted to Frame-by-Year Totals)
#| label: tbl-osp-prev-overall
#| tbl-cap: Sample Characteristics by Statistical Inference Status
#| label: tbl-04-sample-char
#| tbl-pos: H
overall_results_list <- lapply(names(vars), function(var_name) {
# Create the formula for the specific variable
form <- as.formula(paste0("~", var_name))
# Calculate the proportion and CI on the entire des_stat object
# again, use method = "beta" for robustness
est <- svyciprop(form, design = des_stat, method = "beta", na.rm = TRUE)
# Extract the proportion and confidence interval
p_est <- as.numeric(coef(est))
ci <- as.numeric(confint(est))
# Return a clean tibble with the results
tibble(
osp = paste0(vars[var_name], ""), # Creates the label, e.g., "Prereg (SI)"
p = p_est,
p_low = ci[1],
p_upp = ci[2]
)
})
# Combine the list of results into a single data frame
overall_osp_si_raw <- bind_rows(overall_results_list)
# Apply the final formatting to match your original code
overall_osp_si <- overall_osp_si_raw %>%
mutate(`Prevalence` = sprintf("%.1f%% (%.1f-%.1f)", 100 * p, 100 * p_low, 100 * p_upp)) %>%
select(osp, `Prevalence`)
# Print the final result
overall_osp_si <- overall_osp_si %>%
rename(
OSP = osp
tbl_sample_desc <- df %>% mutate(
journal_category = case_when(
journal_category == "PSYCHOLOGY, MULTIDISCIPLINARY" ~ "A",
journal_category == "LAW" ~ "B",
journal_category == "CRIMINOLOGY & PENOLOGY" ~ "C"
)) %>%
tbl_summary(
include = c(is_open_access, is_open_data, is_open_materials, is_prereg, txt_source, txt_only_abstract, journal_category, journal_jif_quartile, txt_count, txt_flesch, journal_x2023_jif),
by = is_statistical,
label = list(
is_open_access = "Open Access",
is_open_data = "Open Data",
is_open_materials = "Open Materials",
is_prereg = "Preregistration",
txt_source = "Text Source",
txt_only_abstract = "Only Abstract",
journal_category = "Journal Category",
journal_jif_quartile = "JIF Quartile",
txt_count = "Count: Words",
txt_flesch = "Flesch Score",
journal_x2023_jif = "JIF (2023)",
is_statistical = "Statistical Inference"
),
statistic = list(
all_continuous() ~ "{mean} ({sd})",
all_categorical() ~ "{n} / {N} ({p}%)"
),
missing = "no"
) %>%
arrange(desc(`Prevalence`))
tbl_overall_osp_si <- overall_osp_si %>%
kbl(
format = 'latex',
longtable = TRUE,
booktabs = TRUE,
escape = T,
) %>% # add footnote
column_spec(1, width = '3cm')%>%
kable_styling(
position = "center",
latex_options = "hold_position",
full_width = FALSE) %>%
kableExtra::footnote(
general = "Prevalence estimates in statistical inference publications using design-weights per year (95% CI)",
general_title = "Note:",
footnote_as_chunk = T,
threeparttable = T
)
add_p(
include = c(txt_only_abstract, txt_source, txt_count, txt_flesch, journal_x2023_jif),
test = list(
txt_only_abstract ~ "fisher.test",
txt_source ~ "chisq.test",
txt_count ~ "wilcox.test",
txt_flesch ~ "wilcox.test",
journal_x2023_jif ~ "wilcox.test"
),
pvalue_fun = label_style_pvalue(digits = 3)
) %>%
add_overall() %>%
modify_header(label ~ "**Variable**") %>%
modify_spanning_header(c("stat_1", "stat_2") ~ "**Statistical Inference**")%>%
modify_footnote_body(
footnote = "A: Psychology, Multidisciplinary; B: Law; C: Criminology & Penology",
columns = "label",
rows = variable == "journal_category"
)
if(output_format == "pdf/tex") {
print(tbl_overall_osp_si)
tbl_sample_desc %>%
as_gt() %>%
tab_options(
table.font.size = gt::px(12),
latex.use_longtable = TRUE
)
} else if(output_format == "docx") {
overall_osp_si %>%
flextable() %>%
set_table_properties(width = 1, layout = "autofit") %>%
tbl_sample_desc %>%
as_flex_table() %>%
set_table_properties(width = 1) %>%
theme_booktabs(bold_header = TRUE) %>%
align(align = "center", part = "all") %>%
fontsize(size = 11, part = "header") %>%
fontsize(size = 10, part = "body") %>%
set_caption(caption = "Note: Prevalence estimates in statistical inference publications using design-weights per year (95% CI)")
fontsize(size = 8, part = "body") %>%
width(5, 2.34, unit = "cm") %>%
width(2:4, 3.5, unit = "cm") %>%
autofit() %>%
height_all(height = .2)
} else {
}
@@ -880,11 +881,11 @@ if (isTRUE(debug_mode)) {
}
```
In @tbl-osp-prev, adjustments were applied using sensitivity and specificity from the ML-validation analysis in [@liuQuantitativeBiasAnalysis2023]. Under extreme rarity, adjustments become unstable: intervals widen dramatically (approaching $[0,1]$) or yield boundary/negative estimates when specificity is insufficient relative to prevalence. For OD, the false-positive rate ($1-\text{Sp} \approx 12.7\%$) exceeds the observed prevalence ($2.2\%$), pushing adjusted points below zero. For OM, low sensitivity ($\text{Se} = 0.20$) and tiny validation counts produce near-uninformative intervals. Given these constraints, the adjusted values can be interpreted as sensitivity ranges rather than confirmatory estimates. Any substantive claims should thereby rather be based on design-based estimates and on OA (measured from metadata).
In @tbl-05-osp-prev, adjustments were applied using sensitivity and specificity from the ML-validation analysis in [@liuQuantitativeBiasAnalysis2023]. Under extreme rarity, adjustments become unstable: intervals widen dramatically (approaching $[0,1]$) or yield boundary/negative estimates when specificity is insufficient relative to prevalence. For OD, the false-positive rate ($1-\text{Sp} \approx 12.7\%$) exceeds the observed prevalence ($2.2\%$), pushing adjusted points below zero. For OM, low sensitivity ($\text{Se} = 0.20$) and tiny validation counts produce near-uninformative intervals. Given these constraints, the adjusted values can be interpreted as sensitivity ranges rather than confirmatory estimates. Any substantive claims should thereby rather be based on design-based estimates and on OA (measured from metadata).
```{r}
#| tbl-cap: Observed and Adjusted Prevalence of Open Science Practices among Statistical Inference Papers
#| label: tbl-osp-prev
#| tbl-cap: Observed and Adjusted Prevalence of open science practices among Statistical Inference Papers
#| label: tbl-05-osp-prev
# https://influentialpoints.com/Training/estimating_true_prevalence.htm
# https://academic.oup.com/ije/article/52/3/942/6982613?login=false
@@ -1048,7 +1049,7 @@ if (isTRUE(debug_mode)) {
}
```
Earlier differences in text sources suggest heterogeneity by journal, thereby implicating also publisher variance [@scogginsMeasuringTransparencySocial2024]. @fig-osp-time-by-publisher visualizes OA shares over time for the 12 most prolific publishers in the sample (listed in the caption). Leveraging larger $n$, the author fit simple OLS trends to annual OA proportions. The four most prolific publishers show clear increases. Four publishers do not: Oxford University Press, Emerald, ASCE, and MDPI. MDPI remains at 100% OA, Emerald at 0% in this sample; ASCE shows an apparent decline consistent with limited observations; Oxford University Press is relatively stable. All observed increases are highly statistically significant. Future work should use models designed for proportions (e.g., binomial GLMs) and, ideally, hierarchical pooling across publishers and years.
Earlier differences in text sources suggest heterogeneity by journal, thereby implicating also publisher variance [@scogginsMeasuringTransparencySocial2024]. @fig-osp-time-by-publisher visualizes OA shares over time for the 12 most prolific publishers in the sample (listed in the caption). Leveraging larger $n$, the author fit simple ordinary least squares (OLS) regression trends to annual OA proportions. The four most prolific publishers show clear increases. Four publishers do not: Oxford University Press, Emerald, ASCE, and MDPI. MDPI remains at 100% OA, Emerald at 0% in this sample; ASCE shows an apparent decline consistent with limited observations; Oxford University Press is relatively stable. All observed increases are highly statistically significant. Future work should use models designed for proportions (e.g., binomial generalized linear models, GLMs) and, ideally, hierarchical pooling across publishers and years.
```{=latex}
\footnotesize
@@ -1201,25 +1202,25 @@ if (isTRUE(debug_mode)) {
\normalsize
```
This study was deliberately scoped as a pilot, which constrained coverage, precision, and tooling. The population assessed was limited to SI papers from the top 100 JCR journals in criminology and legal psychology and to Crossref metadata, so venue and index biases remain. The 2013-2023 window omits the most recent changes. Keyword screening did not fully exclude non-target items, and a Quarto "freeze" configuration led to using print over online dates in some cases. Full-text retrieval was partial and legally bounded to TDM-permitted publishers; short documents (<1,000 words) were treated as missing full text, risking misclassification.
This study was deliberately scoped as a pilot, which constrained coverage, precision, and tooling. The population assessed was limited to SI papers from the top 100 JCR journals in criminology and legal psychology and to Crossref metadata, so venue and index biases remain. The 2013-2023 window omits the most recent changes. Keyword screening did not fully exclude non-target items, and a Quarto "freeze" configuration led to using print over online dates in some cases. Full-text retrieval was partial and legally bounded to text and data mining permitted publishers, short documents (<1,000 words) were treated as missing full text, risking misclassification.
Measurement and modeling challenges were substantial. SI/OSP labels were trained on a small, single-coder hand set plus GPT assistance. Severe class imbalance for OSPs, few validation positives, and upsampling inflated nominal accuracy while depressing stability. Misclassification adjustments (Rogan-Gladen) became unstable at very low prevalences, and some OA trend analyses used simple OLS rather than binomial/GLM approaches.
In the methodological report, comparing basic text characteristics between OSP-labeled papers and non-OSP papers revealed non-independence (e.g., differences in word count, Flesch score, and text source), despite the assumption that such features should not vary with true OSP status. This pattern indicates likely misclassification and/or model leakage, with classifiers picking up irrelevant proxies (publisher templates, document length) rather than OSP content.
In the methodological report, comparing basic text characteristics between OSP-labeled papers and non-OSP papers revealed non-independence (e.g., differences in word count, Flesch score, and text source), despite the assumption that such features (variables) should not vary with true OSP status. This pattern indicates likely misclassification and/or model leakage, with classifiers picking up irrelevant proxies (publisher templates, document length) rather than OSP content.
I therefore propose a series of recommendations for future iterations, that should expand bibliographic metadata sources (Crossref + Scopus and Web of Science) and further audit screened-out records to assess selection, operationalizations with sharper rules more close to the constructs defined by e.g. OSF, employ multi-coder assessment, and quantify inter-rater-reliability on a larger training data base OR classify leveraging ChatGPT as implied by the very accurate precisions evident here and replace OLS with binomial GLMs or hierarchical models for proportions. On the technicals side, a more stringent Quarto setup should be used, with simplified modular code based on a refined version of the codebase used here. The downloader should be improved in terms of a more homogeneous extraction logic by including the HTML and PDF full-text extraction in the pre-processing pipeline, making the whole process more transparent, reproducible and less error-prone. Finally, the sample size should be increased substantially, ideally to the full population of SI papers in the frame, to improve precision and enable analysis on journal level.
Despite of all the limitations, there are main substantive implications: OSP prevalence signals in SI papers-especially preregistration and OM-are rare enough that model-based estimation is fragile at this scale, whereas OA, measured from metadata, shows a clear upward trend approaching roughly half of SI outputs by 2023. Methodologically, GPT proves to be a promising primary coder for a scaled follow-up, and the pipeline developed here provides a reproducible, yet improvable foundation for a larger, better-powered study.
Despite of all the limitations, there are main substantive implications: OSP prevalence signals in SI papers-especially PR and OM-are rare enough that model-based estimation is fragile at this scale, whereas OA, measured from metadata, shows a clear upward trend approaching roughly half of SI outputs by 2023. Methodologically, GPT proves to be a promising primary coder for a scaled follow-up, and the pipeline developed here provides a reproducible, yet improvable foundation for a larger, better-powered study.
# Conclusion
The replication crisis has intensified the examination of research practices and accelerated the push for transparency and openness. This study contributes by mapping the adoption of open-science practices (OSP) within criminology and legal psychology, establishing a baseline for future efforts. The evidence indicates meaningful progress in availability-most clearly in OA-yet massive, persistent gaps in reproducibility, particularly for OD, OM, and preregistration.
The replication crisis has intensified the examination of research practices and accelerated the push for transparency and openness. This study contributes by mapping the adoption of OSPs within criminology and legal psychology, establishing a baseline for future efforts. The evidence indicates meaningful progress in availability-most clearly in OA-yet massive, persistent gaps in reproducibility, particularly for OD, OM, and PR.
Two decades ago, @ioannidisWhyMostPublished2005 argued that the credibility of findings are closely tied to statistical power, field-specific protocols, and careful attention to pre-study odds. In other words, simply assessing p-values in a rather mechanistic manner is insufficient @collingStatisticalInferenceReplication2021. In that spirit, this work emphasizes measurement, validation, and transparency over nominal statistical "wins," offering an initial, field-specific picture of where credibility can be strengthened and how to get there.
Two decades ago, @ioannidisWhyMostPublished2005 argued that the credibility of findings are closely tied to statistical power, field-specific protocols, and careful attention to pre-study odds. In other words, simply assessing p-values in a rather mechanistic manner is insufficient [@collingStatisticalInferenceReplication2021]. In that spirit, this work emphasizes measurement, validation, and transparency over nominal statistical "wins," offering an initial, field-specific picture of where credibility can be strengthened and how to get there.
Methodologically, the study shows that GPT-assisted coding can be accurate and scalable for detecting OSPs, while downstream ML classifiers struggle under extreme class imbalance-a limitation that complicates misclassification-adjusted prevalence estimation. Still, the pipeline built here demonstrates a path toward for larger, confirmatory follow-ups.
This work discussed the replication crisis, its implications for criminology and legal psychology, and how OSPs can help to address some of the issues that have been raised. While the last decades the wording "crisis" framed the discussion in a rather negative light, recent work suggests an upward trend in OSPs which accelerates the transition towards more credible research-moving "from crisis to credibility" @korbmacherReplicationCrisisHas2023. Awareness and adoption of open practices are growing [@grossmannReasonsCautiousOptimism2021], institutions are adapting norms and incentives [@smaldinoOpenScienceModified2019]. Even though the results of this study indicate that there is still a long way to go, the upward trend in OA and the presence of OM and preregistration in some papers are encouraging signs
This work discussed the replication crisis, its implications for criminology and legal psychology, and how OSPs can help to address some of the issues that have been raised. While the last decades the wording "crisis" framed the discussion in a rather negative light, recent work suggests an upward trend in OSPs which accelerates the transition towards more credible research-moving "from crisis to credibility" [@korbmacherReplicationCrisisHas2023]. Awareness and adoption of open practices are growing [@grossmannReasonsCautiousOptimism2021], institutions are adapting norms and incentives [@smaldinoOpenScienceModified2019]. Even though the results of this study indicate that there is still a long way to go, the upward trend in OA and the presence of OM and PR in some papers are encouraging signs
To make sure, that our results are robust, reliable and credible, this work shall be seen as a call for an open, cumulative, and collaborative research culture. Accordingly, the author invites direct reproduction and incremental improvement of this pipeline-via open sharing of data, code, prompts, and labeling protocols-so the analysis can be stress-tested, recalibrated, and strengthened.