Document Type

Conference Paper

Publication Date

2025

DOI

10.1145/3720553.3746674

Publication Title

HT '25: Proceedings of the 36th ACM Conference on Hypertext and Social Media

Pages

22-27

Conference Name

HT '25: 36th ACM Conference on Hypertext and Social Media, September 15-18, 2025, Chicago, IL

Abstract

Science news is increasingly important in connecting scientists and the public by sharing discoveries and innovations. With the rise of large language models (LLMs), there is potential to automate science news creation, but concerns exist about the quality of LLM-generated news versus human-written news. This paper explores whether LLMs can outperform humans in distinguishing between human-written and LLM-generated news. Inspired by the Chain-of-Thought prompting method, we designed a simple yet effective variant called Guided Few-shot (GFS), which encodes the characteristics of news of two types with examples. Our experiments indicated that GFS with just a single example effectively boosted the performance of all LLMs and made open-weight LLMs achieve or exceed the performance of humans and commercial LLMs. The code and data are available in our repository at https://github.com/lamps-lab/sanews.

Rights

© 2025 Copyright held by the owner/authors.

This work is licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) License.

Original Publication Citation

Soos, D., Jiang, M., & Wu, J. (2025). Can LLMs beat humans on discerning human-written and LLM-generated science news? In Y. Zheng, L. Boratto, C. Hargood, & D. Lee (Eds.), HT '25: Proceedings of the 36th ACM Conference on Hypertext and Social Media (pp.22-27). Association for Computing Machinery. https://doi.org/10.1145/3720553.3746674

ORCID

0000-0002-7089-6354 (Soos), 0000-0003-0173-4463 (Wu)

Download

Share

COinS