Document Type

Article

Publication Date

2026

DOI

10.54531/FCQK4553

Publication Title

Journal of Healthcare Simulation

Volume

Advance online publication

Pages

6 pp.

Abstract

Background

Artificial intelligence (AI) has become increasingly embedded in research workflows. Large language models (LLMs) are being used to code segments of text, organise codes into themes and interpret patterns within contexts. Recent comparisons between human and AI analyses demonstrate up to 80% thematic overlap, yet humans consistently exhibit deeper interpretive integration and contextual understanding. This study assesses whether experienced researchers can distinguish between entirely human-generated and AI-generated qualitative content analyses of a simulation debriefing.

Methods

We conducted a qualitative descriptive study comparing human-generated qualitative content analysis (QCA) with ChatGPT-4o-generated QCA using a single focus group transcript on emotion management during debriefing of an emotion-triggering simulation case. First, a human research team performed QCA following Schreier’s framework to identify themes. The same transcript was then analysed by ChatGPT-4o using a prompt aligned with Schreier’s QCA approach. Second, both outputs were standardised into matched text formats to control for structural and stylistic variation. Third, de-identified versions of each analysis (human and AI) were provided to a separate group of experienced reviewers, who completed two rounds of open-ended review. Finally, we conducted a QCA of the reviewers’ responses to examine how expert researchers characterise and differentiate AI- and human-generated qualitative analyses.

Results

Only three out of five experienced researchers were able to distinguish between AI-generated and human-generated QCA. The reviewers who correctly identified the AI analysis noted that AI generated incomplete quotes and missed specific details. Reviewers’ preferences for a given analysis did not correlate with accurate identification of human-generated vs AI-generated analysis. Three out of five reviewers preferred the AI analysis.

Discussion

To our knowledge this is the first study specific to simulation debriefing. Two out of five reviewers were not able to differentiate between human and AI-generated outputs, suggesting that confident identification of AI-generated content may reflect bias more than accuracy, and raising concerns about the fairness of informal AI-detection practices. A combined approach in which humans perform an initial analysis that is then followed by subsequent refinement using AI-based processes may represent the most productive path forward.

Rights

© The Authors 2026. 

This article is distributed under the terms of the Creative Commons Attribution-Share Alike 4.0 International (CC BY-SA 4.0) License, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original authors and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Data Availability

Article states: "None declared."

Original Publication Citation

Lucas, A. T., Ramos, J., Bajwa, M., Calhoun, A., Scerbo, M. W., & Palaganas, J. C. (2026). Can an experienced qualitative researcher distinguish AI from human qualitative content analysis? Journal of Healthcare Simulation. Advance online publication. https://doi.org/10.54531/FCQK4553 

ORCID

0000-0002-0498-3222 (Scerbo)

Additional Files
fcqk4553-supplementary_1.pdf (65 kB)
Supplemental Digital Content 1
articles_content-1786084204796-f7d711ff-4e3f-4270-8613-f8772e3a2f00_assets_fcqk4553-supplementary_2.docx (17 kB)
Supplemental Digital Content 2
Download

Share

COinS