Document Type

Conference Paper

Publication Date

2024

DOI

10.1145/3641525.3663628

Publication Title

Proceedings of the 2nd ACM Conference on Reproducibility and Replicability

Pages

96-100

Conference Name

ACM REP '24: 2nd ACM Conference on Reproducibility and Replicability, 18-20 June 2024, Rennes, France

Abstract

The iterative character of work in machine learning (ML) and artificial intelligence (AI) and reliance on comparisons against benchmark datasets emphasize the importance of reproducibility in that literature. Yet, resource constraints and inadequate documentation can make running replications particularly challenging. Our work explores the potential of using downstream citation contexts as a signal of reproducibility. We introduce a sentiment analysis framework applied to citation contexts from papers involved in Machine Learning Reproducibility Challenges in order to interpret the positive or negative outcomes of reproduction attempts. Our contributions include training classifiers for reproducibility-related contexts and sentiment analysis, and exploring correlations between citation context sentiment and reproducibility scores. Study data, software, and an artifact appendix are publicly available at https://github.com/lamps-lab/ccair-ai-reproducibility.

Rights

© 2024 Copyright held by the owner/authors.

This work is licensed under a Creative Commons Attribution-NonCommercial International 4.0 License.

Original Publication Citation

Obadage, R. R., Rajtmajer, S. M., & Wu, J. (2024) SHORT: Can citations tell us about a paper's reproducibility? A case study of machine learning papers. In Proceedings of the 2nd ACM Conference on Reproducibility and Replicability (pp. 96-100). Association for Computing Machinery. https://doi.org/10.1145/3641525.3663628

ORCID

0000-0003-1593-4052 (Obadage), 0000-0003-0173-4463 (Wu)

Download

Share

COinS