Document Type

Conference Paper

Publication Date




Publication Title

DocEng '22: Proceedings of the 22nd ACM Symposium on Document Engineering


16 (4 pp.)

Conference Name

ACM Symposium on Document Engineering 2022 (DocEng '22), September 20-23, 2022, San Jose, California


Recently, the Allen Institute for Artificial Intelligence released the Semantic Scholar Open Research Corpus (S2ORC), one of the largest open-access scholarly big datasets with more than 130 million scholarly paper records. S2ORC contains a significant portion of automatically generated metadata. The metadata quality could impact downstream tasks such as citation analysis, citation prediction, and link analysis. In this project, we assess the document linking quality and estimate the document conflation rate for the S2ORC dataset. Using semi-automatically curated ground truth corpora, we estimated that the overall document linking quality is high, with 92.6% of documents correctly linking to six major databases, but the linking quality varies depending on subject domains. The document conflation rate is around 2.6%, meaning that about 97.4% of documents are unique. We further quantitatively compared three near-duplicate detection methods using the ground truth created from S2ORC. The experiments indicated that locality-sensitive hashing was the best method in terms of effectiveness and scalability, achieving high performance (F1=0.960) and a much reduced runtime. Our code and data are available at


© 2022 Copyright held by the owner/authors.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Original Publication Citation

Wu, J., Hiltabrand, R., Soós, D., & Giles, C. L. (2022). Scholarly big data quality assessment: A case study of document linking and conflation with S2ORC. In DocEng '22: Proceedings of the 22nd ACM Symposium on Document Engineering (Article 16, pp.1-4). Association for Computing Machinery.


0000-0003-0173-4463 (Wu)