A Study of Computational Reproducibility Using URLs Linking to Open Access Datasets and Software

Document Type

Conference Paper

Publication Date




Publication Title

WWW'22: Companion Proceedings of the Web Conference 2022



Conference Name

WWW'22: The ACM Web Conference 2022, April 25-29, 2022, Virtual Event, Lyon France


Datasets and software packages are considered important resources that can be used for replicating computational experiments. With the advocacy of Open Science and the growing interest of investigating reproducibility of scientific claims, including URLs linking to publicly available datasets and software packages has become an institutionalized part of research publications. In this preliminary study, we investigated the disciplinary dependency and chronological trends of including open access datasets and software (OADS) in electronic theses and dissertations (ETDs), based on a hybrid classifier called OADSClassifier, consisting of a heuristic and a supervised learning model. The classifier achieves the best F1 of 0.92. We found that the inclusion of OADS-URLs exhibited a strong disciplinary dependence and the fraction of ETDs containing OADS-URLs has been gradually increasing over the past 20 years. We developed and share a ground truth corpus consisting of 500 manually labeled sentences containing URLs from scientific papers. The dataset and source code are available at https://github.com/lamps-lab/oadsclassifier.


© 2022 Copyright held by the owner/authors.

Link included in accordance with publisher policy.

Original Publication Citation

Salsabil, L., Wu, J., Choudhury, M. H., Ingram, W. A., Fox, E. A., Rajtmajer, S. M., & Giles, C. L. (2022). In Frédérique Laforest, Raphaël Troncy, Lionel Médini, & Ivan Herman (Eds.), WWW'22: Companion Proceedings of the Web Conference 2022 (pp. 784-788). Association for Computing Machinery. https://doi.org/10.1145/3487553.3524658


0000-0002-6162-2896 (Salsabil), 0000-0003-0173-4463 (Wu), 0000-0002-9318-8844 (Choudhury)