mischbeck/ThesisProposal
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Finalizes Proposal

Browse Source
This commit is contained in:
Michael Beck 2025-01-08 11:56:51 +01:00
parent 22989cf064
commit d2cf9c9f51
2 changed files with 60 additions and 52 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

85
ResearchProposal.md
View File

@ -2,12 +2,12 @@
documentclass: article documentclass: article
author: author:
- Michael Beck - Michael Beck
title: "A Systematic Review of Open Science Practices in the Studies of Crime" title: "A Systematic Review of Open Science Practices in the Studies of Crime"
description: "" description: ""
subtitle: "Research Proposal" subtitle: "Research Proposal"
date: "2024-12-19" date: "2025-01-08"
lang: en-US lang: en-US
toc: true toc: true
@ -36,34 +36,34 @@ header-includes: |
\setcounter{page}{-1} \setcounter{page}{-1}
include-before: | include-before: |
\newcommand{\scellL}[1]{% \newcommand{\scellL}[1]{%
\parbox{4.2cm}{\raggedright\leftskip=1em\hskip-1em#1} \parbox{4.2cm}{\raggedright\leftskip=1em\hskip-1em#1}
} }
\newcommand{\scell}[1]{% \newcommand{\scell}[1]{%
\parbox{3cm}{ \parbox{3cm}{
\begin{center}#1 \begin{center}#1
\end{center}
\vspace{1mm}
}
}
\newcommand{\scellB}[1]{%
\parbox{2.5cm}{
\begin{center}#1
\end{center}
\vspace{1mm}
}
}
\begin{center}
\vfill
Supervisor: \\
\textbf{Dr. Alexander Trinidad} \\
\vspace{1cm}
\hrule
\end{center} \end{center}
\thispagestyle{empty} \vspace{1mm}
\newpage }
\thispagestyle{empty} }
\newcommand{\scellB}[1]{%
\parbox{2.5cm}{
\begin{center}#1
\end{center}
\vspace{1mm}
}
}
\begin{center}
\vfill
Master's thesis \\
Supervisor: \\
\textbf{Dr. Alexander Trinidad} \\
\vspace{1cm}
\hrule
\end{center}
\thispagestyle{empty}
\newpage
\thispagestyle{empty}
--- ---
\vfill \vfill
@ -87,11 +87,11 @@ To challenge the biases and to support the possibility of these "repetitions" or
@banksAnswers18Questions2019 establish a definition of open science as a broad term that refers to many concepts including scientific philosophies embodying communality and universalism, specific practices operationalizing these norms including open science policies like sharing of data and analytic files, redefinition of confidence thresholds, pre-registration of studies and analytical plans, engagement in replication studies, removal of pay-walls, incentive systems to encourage the above practices and even specific citation standards. This typology is in line with the work of many other authors from diverse disciplines [e.g. @dienlinAgendaOpenScience2021; and @greenspanOpenSciencePractices2024]. The ongoing debate of the last decades were especially focused on two open science practices. @banksAnswers18Questions2019 establish a definition of open science as a broad term that refers to many concepts including scientific philosophies embodying communality and universalism, specific practices operationalizing these norms including open science policies like sharing of data and analytic files, redefinition of confidence thresholds, pre-registration of studies and analytical plans, engagement in replication studies, removal of pay-walls, incentive systems to encourage the above practices and even specific citation standards. This typology is in line with the work of many other authors from diverse disciplines [e.g. @dienlinAgendaOpenScience2021; and @greenspanOpenSciencePractices2024]. The ongoing debate of the last decades were especially focused on two open science practices.
First, the **publishing of materials, data and code** or *open data* that enables replication of studies. Replication thereby makes it possible to assess the pursued research in detail, find errors, bias or simply support the results of the replicated work [@dienlinAgendaOpenScience2021]. While many researchers see challenges in the publication of their data and materials due to a potentially higher workload, legal concerns or just lack of interest, many of these concerns could be ruled out by streamlined processes or institutional support [@freeseAdvancesTransparencyReproducibility2022; @freeseReplicationStandardsQuantitative2007]. As open data reduces p-hacking, facilitates new research by enabling reproduction, reveals mistakes in the analytical code and enables a diffusion of knowledge on the research process, it seems that many scientists, journals and other institutions start to adopt open data in their research to an increasing extent [@dienlinAgendaOpenScience2021; @finkReplicationCodeAvailability; @freeseAdvancesTransparencyReproducibility2022; @zenk-moltgenFactorsInfluencingData2018; @matternWhyAcademicsUndershare2024]. First, the **publishing of materials, data and code** or _open data_ that enables replication of studies. Replication thereby makes it possible to assess the pursued research in detail, find errors, bias or simply support the results of the replicated work [@dienlinAgendaOpenScience2021]. While many researchers see challenges in the publication of their data and materials due to a potentially higher workload, legal concerns or just lack of interest, many of these concerns could be ruled out by streamlined processes or institutional support [@freeseAdvancesTransparencyReproducibility2022; @freeseReplicationStandardsQuantitative2007]. As open data reduces p-hacking, facilitates new research by enabling reproduction, reveals mistakes in the analytical code and enables a diffusion of knowledge on the research process, it seems that many scientists, journals and other institutions start to adopt open data in their research to an increasing extent [@dienlinAgendaOpenScience2021; @finkReplicationCodeAvailability; @freeseAdvancesTransparencyReproducibility2022; @zenk-moltgenFactorsInfluencingData2018; @matternWhyAcademicsUndershare2024].
Second, **preregistration** involves thoroughly outlining and documenting research plans and their rationale in a repository. These plans can be made publicly accessible when the researcher decides to share them. The specifics of preregistration can vary based on the research type and may encompass elements such as hypotheses, sampling strategies, interview guides, exclusion criteria, study design, and analysis plans [@managoPreregistrationRegisteredReports2023]. Within this definition, a preregistration shall not prevent exploratory research. Deviations from the research plan are still allowed but have to be communicated transparently [@managoPreregistrationRegisteredReports2023; @nosekRegisteredReports2014]. Preregistration impacts research in multiple ways: it helps performing exploratory and confirmatory research independently, protects against publication bias as journals typically commit to publish registered research and counters "researchers' degrees of freedom" in data analysis by reducing overfitting through cherry-picking, variable swapping, flexible model selection and subsampling [@mertensPreregistrationAnalysesPreexisting2019; @FalsePositivePsychologyUndisclosed]. This minimizes the risk of bias by promoting decision-making that is independent of outcomes. It also enhances transparency, allowing others to evaluate the potential for bias and adjust their confidence in the research findings accordingly [@hardwickeReducingBiasIncreasing2023]. Second, **preregistration** involves thoroughly outlining and documenting research plans and their rationale in a repository. These plans can be made publicly accessible when the researcher decides to share them. The specifics of preregistration can vary based on the research type and may encompass elements such as hypotheses, sampling strategies, interview guides, exclusion criteria, study design, and analysis plans [@managoPreregistrationRegisteredReports2023]. Within this definition, a preregistration shall not prevent exploratory research. Deviations from the research plan are still allowed but have to be communicated transparently [@managoPreregistrationRegisteredReports2023; @nosekRegisteredReports2014]. Preregistration impacts research in multiple ways: it helps performing exploratory and confirmatory research independently, protects against publication bias as journals typically commit to publish registered research and counters "researchers' degrees of freedom" in data analysis by reducing overfitting through cherry-picking, variable swapping, flexible model selection and subsampling [@mertensPreregistrationAnalysesPreexisting2019; @FalsePositivePsychologyUndisclosed]. This minimizes the risk of bias by promoting decision-making that is independent of outcomes. It also enhances transparency, allowing others to evaluate the potential for bias and adjust their confidence in the research findings accordingly [@hardwickeReducingBiasIncreasing2023].
My initial plan for my master's thesis was to study the effect of open science practices on reported effect sizes in published papers. During my initial literature review, it appeared to me that there were very few publications that used pre-registration in data-driven criminology. Instead of assessing effect sizes, this raised the question how open science practices have been adapted within criminology. I therefore intend, motivated by the expected positive impact of open science practices and in line with the research of @scogginsMeasuringTransparencySocial2024, to assess the two research questions: My initial plan for my master's thesis was to study the effect of open science practices on reported effect sizes in published papers. During my initial literature review, it appeared to me that there were very few publications that used pre-registration in data-driven Criminology and Legal Psychology. Instead of assessing effect sizes, this raised the question how open science practices have been adapted within criminology. I therefore intend, motivated by the expected positive impact of open science practices and in line with the research of @scogginsMeasuringTransparencySocial2024, to assess the two research questions in Criminology and Legal Psychology:
> $RQ_1$: What proportion of papers that rely on statistical inference make their data and code public? > $RQ_1$: What proportion of papers that rely on statistical inference make their data and code public?
@ -99,7 +99,7 @@ My initial plan for my master's thesis was to study the effect of open science p
@scogginsMeasuringTransparencySocial2024 did an extensive analysis of nearly 100,000 publications in political science and international relations. They observed an increasing use of preregistration and open data, with levels still being relatively low. The extensive research not only revealed the current state of open science in political science, but also generated rich data to perform further meta research. @scogginsMeasuringTransparencySocial2024 did an extensive analysis of nearly 100,000 publications in political science and international relations. They observed an increasing use of preregistration and open data, with levels still being relatively low. The extensive research not only revealed the current state of open science in political science, but also generated rich data to perform further meta research.
I intend to apply similar methods in the field of criminology and criminal psychology: gather data about papers in a subset of criminology journals, classify those papers by application of open source practices using sophisticated machine learning methods and explore the patterns over time to take stock of research practices in the disciplines. In the following section I describe the intended data collection and research methods that are highly based on @scogginsMeasuringTransparencySocial2024 research. I intend to apply similar methods in the field of Criminology and Legal Psychology: gather data about papers in a subset of Criminology and Legal Psychology journals, classify those papers by application of open source practices using sophisticated machine learning methods and explore the patterns over time to take stock of research practices in the disciplines. In the following section I describe the intended data collection and research methods that are highly based on @scogginsMeasuringTransparencySocial2024 research.
# Data and Method # Data and Method
@ -107,27 +107,24 @@ The study will focus on papers in criminal psychology that use data and statisti
## Data Collection ## Data Collection
The process of data collection will closely follow @scogginsMeasuringTransparencySocial2024 and begin with identifying relevant journals in criminal psychology. I will consult the Clarivate Journal Citation Report to obtain a comprehensive list of journals within the fields by filtering for the top 100 journals. The Transparency-and-Openness-Promotion-Factor[^4] (TOP-Factor) according to @nosekPromotingOpenResearch2015 will be used to then assess the journal's admission of open science practices and by including it in the journal dataset. Once the relevant journals are identified, I will use APIs such as Crossref, Scopus, and Web of Science to download metadata for all papers published between 2013 to 2023. The process of data collection will closely follow @scogginsMeasuringTransparencySocial2024 and begin with identifying relevant journals in criminal psychology. I will consult the Clarivate Journal Citation Report to obtain a comprehensive list of journals within the fields by filtering for the top 100 journals. The Transparency-and-Openness-Promotion-Factor[^4] (TOP-Factor) according to @nosekPromotingOpenResearch2015 will be used to then assess the journal's admission of open science practices and by including it in the journal dataset. Once the relevant journals are identified, I will use APIs such as Crossref, Scopus, and Web of Science to download metadata for all papers published between 2013 to 2023.
After obtaining the metadata, I will proceed to download the full-text versions of the identified papers. Whenever possible, I will prioritize downloading HTML versions of the papers due to their structured format, which simplifies subsequent text extraction. For papers that are not available in HTML, I will consider downloading full-text PDFs. Tools such as PyPaperBot or others[^1] can facilitate this process, although I will strictly stick to ethical and legal guidelines, avoiding unauthorized sources like Sci-Hub or Anna's Archive and only using sources that are either included in my institutions campus license or available via open access. If access to full-text papers becomes a limiting factor, I will assess alternative strategies such as collaborating with institutional libraries to request specific papers or identifying open-access repositories that may provide supplementary resources. Non-available texts will be considered with their own category in the later analysis. Once all available full-text papers are collected, I will preprocess the data by converting HTML and PDF files into plain text format using tools such as SciPDF Parser or others[^2]. This preprocessing step ensures that the text is in a standardized format suitable for analysis. After obtaining the metadata, I will proceed to download the full-text versions of the identified papers. Whenever possible, I will prioritize downloading HTML versions of the papers due to their structured format, which simplifies subsequent text extraction. For papers that are not available in HTML, I will consider downloading full-text PDFs. Tools such as PyPaperBot or others[^1] can facilitate this process, although I will strictly stick to ethical and legal guidelines, avoiding unauthorized sources like Sci-Hub or Anna's Archive and only using sources that are either included in my institutions campus license or available via open access. If access to full-text papers becomes a limiting factor, I will assess alternative strategies such as collaborating with institutional libraries to request specific papers or identifying open-access repositories that may provide supplementary resources. Non-available texts will be considered with their own category in the later analysis. Once all available full-text papers are collected, I will preprocess the data by converting HTML and PDF files into plain text format using tools such as SciPDF Parser or others[^2]. This preprocessing step ensures that the text is in a standardized format suitable for analysis.
The proposed data collection is resource-intensive but serves multiple purposes. However, resource constraints could pose challenges, such as limited access to computational tools, DDoS-protection[^3], API-rate limits or delays in obtaining full-text papers. To mitigate these risks, I plan to prioritize scalable data collection methods, limit data collection to a manageable extent and use existing institutional resources, including library services and open-access repositories. Additionally, I will implement efficient preprocessing workflows ensuring that the project remains feasible within the given timeline and resources. The proposed data collection is resource-intensive but serves multiple purposes. However, resource constraints could pose challenges, such as limited access to computational tools, DDoS-protection[^3], API-rate limits or delays in obtaining full-text papers. To mitigate these risks, I plan to prioritize scalable data collection methods, limit data collection to a manageable extent and use existing institutional resources, including library services and open-access repositories. Additionally, I will implement efficient preprocessing workflows ensuring that the project remains feasible within the given timeline and resources.
[^1]: [ferru97/PyPaperBot](https://github.com/ferru97/PyPaperBot), [monk1337/resp](https://github.com/monk1337/resp) [^1]: [ferru97/PyPaperBot](https://github.com/ferru97/PyPaperBot), [monk1337/resp](https://github.com/monk1337/resp)
[^2]: [GitHub - titipata/scipdf_parser](https://github.com/titipata/scipdf_parser), [GitHub - aaronsw/html2text](https://github.com/aaronsw/html2text), [html2text · PyPI](https://pypi.org/project/html2text/), [GitHub - jsvine/pdfplumber](https://github.com/jsvine/pdfplumber), [GitHub - cat-lemonade/PDFDataExtractor](https://github.com/cat-lemonade/PDFDataExtractor/tree/main), [GitHub - euske/pdfminer](https://github.com/euske/pdfminer)
[^2]: [GitHub - titipata/scipdf\_parser](https://github.com/titipata/scipdf_parser), [GitHub - aaronsw/html2text](https://github.com/aaronsw/html2text), [html2text · PyPI](https://pypi.org/project/html2text/), [GitHub - jsvine/pdfplumber](https://github.com/jsvine/pdfplumber), [GitHub - cat-lemonade/PDFDataExtractor](https://github.com/cat-lemonade/PDFDataExtractor/tree/main), [GitHub - euske/pdfminer](https://github.com/euske/pdfminer)
[^3]: DDoS: Distributed Denial of Service, see @wangDDoSAttackProtection2015. [^3]: DDoS: Distributed Denial of Service, see @wangDDoSAttackProtection2015.
[^4]: The TOP-Factor according to @nosekRegisteredReports2014 is a score that assesses the admission of open science practices can be obtained from [topfactor.org](https://topfactor.org/journals). [^4]: The TOP-Factor according to @nosekRegisteredReports2014 is a score that assesses the admission of open science practices can be obtained from [topfactor.org](https://topfactor.org/journals).
## Classification ## Classification
The classification process will begin with operationalizing the key open science practices that I aim to study. This involves the definition of clear criteria for identifying papers that fall into the categories I plan to classify: Papers that use statistical inference, papers that applied preregistration, papers that applied open data practices, papers that offer open materials and papers that are available via open access. The classification process will begin with operationalizing the key open science practices that I aim to study. This involves the definition of clear criteria for identifying papers that fall into the categories I plan to classify: Papers that use statistical inference, papers that applied preregistration, papers that applied open data practices, papers that offer open materials and papers that are available via open access.
Classification of open access papers will be performed using the available metadata. The other classes will be identified using machine learning models trained on a preclassified training dataset. The models will categorize papers using generated document feature matrices (DFM's) in line with @scogginsMeasuringTransparencySocial2024. Classification of open access papers will be performed using the available metadata. The other classes will be identified using machine learning models trained on a preclassified training dataset. The models will categorize papers using generated document feature matrices (DFM's) in line with @scogginsMeasuringTransparencySocial2024.
To train machine learning models capable of classifying the papers, I will manually categorize a subset of papers. The sample size will be determined using weighted fitting of learning curves according to @figueroaPredictingSampleSize2012 which need an initial hand-coded sample size of 100-200. To ensure the representativeness of this subset, I will sample papers proportionally from different journals, publication years, and subfields within criminology. The stratified sampling approach will help mitigate biases and ensure that the training data reflects the diversity of the overall dataset. If the necessary sample size exceeds my time constraints, I will try to use clustering based text classification to extend the training sample [@zengCBCClusteringBased2003]. To train machine learning models capable of classifying the papers, I will manually categorize a subset of papers. The sample size will be determined using weighted fitting of learning curves according to @figueroaPredictingSampleSize2012 which need an initial hand-coded sample size of 100-200. To ensure the representativeness of this subset, I will sample papers proportionally from different journals, publication years, and subfields within Criminology and Legal Psychology. The stratified sampling approach will help mitigate biases and ensure that the training data reflects the diversity of the overall dataset. If the necessary sample size exceeds my time constraints, I will try to use clustering based text classification to extend the training sample [@zengCBCClusteringBased2003] and will also consider the use of large language models like ScienceOS for the generation of the training data by using such a model to preclassify papers.
The sampled subset will serve as a "labeled" dataset for supervised learning. Different classification methods were considered but deemed as not suitable for the task as those were either found to be designed for document topic classification or too time intense for a master's thesis [e.g. @kimResearchPaperClassification2019; @sanguansatFeatureMatricizationDocument2012; @jandotInteractiveSemanticFeaturing2016]. The sampled subset will serve as a "labeled" dataset for supervised learning. Different classification methods were considered but deemed as not suitable for the task as those were either found to be designed for document topic classification or too time intense for a master's thesis [e.g. @kimResearchPaperClassification2019; @sanguansatFeatureMatricizationDocument2012; @jandotInteractiveSemanticFeaturing2016].
@ -139,7 +136,7 @@ The automated classification will enable me to categorize a large amount papers
## Analysis ## Analysis
In the analysis phase of the research, an exploratory analysis will be conducted to explore temporal trends in the adoption of open science practices over the past decade. This involves comparing the adoption rates of practices such as pre-registration, open data, open materials, and open access across the disciplines of sociology and criminology, as well as among different journals. The goal is to identify possible differences or similarities in how these practices have been embraced over time. This evaluation aims to uncover insights into the methodological rigor and transparency within the fields, providing a comprehensive understanding of the current landscape and potential areas for improvement in research practices. By building on the methods developed by @scogginsMeasuringTransparencySocial2024, I hope to generate data and insights that will support future efforts to promote transparency and reproducibility in criminal psychology. In the analysis phase of the research, an exploratory analysis will be conducted to explore temporal trends in the adoption of open science practices over the past decade. This involves comparing the adoption rates of practices such as pre-registration, open data, open materials, and open access across the disciplines of Criminology and Legal Psychology, as well as among different journals. The goal is to identify possible differences or similarities in how these practices have been embraced over time. This evaluation aims to uncover insights into the methodological rigor and transparency within the fields, providing a comprehensive understanding of the current landscape and potential areas for improvement in research practices. By building on the methods developed by @scogginsMeasuringTransparencySocial2024, I hope to generate data and insights that will support future efforts to promote transparency and reproducibility in criminal psychology.
\newpage \newpage
@ -148,7 +145,6 @@ In the analysis phase of the research, an exploratory analysis will be conducted
::: {#refs} ::: {#refs}
::: :::
\newpage \newpage
Eigenständigkeitserklärung Eigenständigkeitserklärung
@ -158,7 +154,6 @@ Hiermit versichere ich, dass ich die vorliegende Arbeit selbstständig und ohne
\vspace{5cm} \vspace{5cm}
\begin{tabular}{@{}p{4in}@{}} \begin{tabular}{@{}p{4in}@{}}
\hrulefill \\ \hrulefill \\
Michael Beck, 08.08.2024 \\ Michael Beck, 08.08.2024 \\
\end{tabular} \end{tabular}

13
lit.bib
View File

@ -2182,6 +2182,12 @@
file = {/home/michi/Zotero/storage/V555Q9F6/Claesen et al. - 2021 - Comparing dream to reality an assessment of adherence of the first generation of preregistered stud.pdf} file = {/home/michi/Zotero/storage/V555Q9F6/Claesen et al. - 2021 - Comparing dream to reality an assessment of adherence of the first generation of preregistered stud.pdf}
} }
@misc{clarivate2023JournalImpact,
title = {2023 {{Journal Impact Factor}}},
author = {{Clarivate}},
urldate = {2024-12-19}
}
@misc{cnnCNNcomReclusiveLinux, @misc{cnnCNNcomReclusiveLinux,
title = {{{CNN}}.Com - {{Reclusive Linux}} Founder Opens up - {{May}} 18, 2006}, title = {{{CNN}}.Com - {{Reclusive Linux}} Founder Opens up - {{May}} 18, 2006},
author = {{CNN}}, author = {{CNN}},
@ -7008,6 +7014,13 @@
file = {/home/michi/Zotero/storage/Z3KW2L59/Miric et al. - 2019 - Protecting their digital assets The use of formal.pdf;/home/michi/Zotero/storage/PDPQB7E6/S0048733319300204.html} file = {/home/michi/Zotero/storage/Z3KW2L59/Miric et al. - 2019 - Protecting their digital assets The use of formal.pdf;/home/michi/Zotero/storage/PDPQB7E6/S0048733319300204.html}
} }
@misc{MischbeckOpenScienceReviewOpenScienceReview,
title = {Mischbeck/{{OpenScienceReview}} - {{OpenScienceReview}} - {{Gitea}}},
urldate = {2024-12-19},
howpublished = {https://git.mischbeck.de/mischbeck/OpenScienceReview},
file = {/home/michi/Zotero/storage/D9M82ZBY/OpenScienceReview.html}
}
@article{moffittMaleAntisocialBehaviour2018, @article{moffittMaleAntisocialBehaviour2018,
title = {Male Antisocial Behaviour in Adolescence and Beyond}, title = {Male Antisocial Behaviour in Adolescence and Beyond},
author = {Moffitt, Terrie E.}, author = {Moffitt, Terrie E.},