Document Type
Article
Publication Date
2026
DOI
10.1145/3770855.3819003
Publication Title
KDD '26: Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2
Pages
11693-11704
Conference Name
32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 9-13 August 2026, Jeju Island, Republic of Korea
Abstract
The literature has witnessed an emerging interest in developing and evaluating AI agents for automated assessment of research claims in scientific papers. Existing benchmarks focus primarily on the computational aspect of this task, testing agents' ability to reproduce or replicate research outcomes when having access to the code and data. This setting, while foundational, (1) fails to capture the inconsistent availability of new data for replication as opposed to reproduction, and (2) lacks ground-truth diversity by focusing exclusively on fully reproducible or replicable papers, thereby failing to evaluate an agent's ability to identify non-replicable research. Furthermore, most benchmarks only evaluate the final reproducibility or replicability outcomes without an evaluation of the process. In response, we introduce ReplicatorBench, an end-to-end benchmark, including human-verified replicable and non-replicable research claims in social and behavioral sciences, for evaluating AI agents in research replication across three stages: (1) extraction of relevant information and retrieval of replication data; (2) design and execution of computational experiments; and (3) interpretation of replication results, allowing a test of AI agents' capability to mimic the activities of human replicators in real world. To set a baseline of AI agents' capability, we develop ReplicatorAgent, an agentic framework equipped with necessary tools like web search and iterative interaction with sandboxed environments, to accomplish tasks in ReplicatorBench. We evaluate ReplicatorAgent across four underlying large language models (LLMs), as well as different design choices of programming language and levels of code access. Our findings reveal that while current LLM agents are capable of effectively designing and executing computational experiments, they struggle with retrieving new data, necessary to replicate a claim. All code and data are publicly available at: https://github.com/CenterForOpenScience/llm-benchmarking.
Rights
© 2026 Copyright held by the owner/authors.
This work is licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) License.
Original Publication Citation
Nguyen, B., Soós, D., Ma, Q., Obadage, R. R., Ranjan, Z., Koneru, S., Errington, T. M., Nematova, S., Rajtmajer, S., Wu, J., & Jiang, M. (2026). ReplicatorBench: Benchmarking LLM agents for replicability in social and behavioral sciences. In, Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 11693-11704). Association for Computing Machinery. https://doi.org/10.1145/3770855.3819003
Repository Citation
Nguyen, B., Soós, D., Ma, Q., Obadage, R. R., Ranjan, Z., Koneru, S., Errington, T. M., Nematova, S., Rajtmajer, S., Wu, J., & Jiang, M. (2026). ReplicatorBench: Benchmarking LLM agents for replicability in social and behavioral sciences. In, Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 11693-11704). Association for Computing Machinery. https://doi.org/10.1145/3770855.3819003
ORCID
0000-0002-7089-6354 (Soos), 0000-0003-1593-4052 (Obadage), 0000-0003-0173-4463 (Wu)