Document Type

Article

Publication Date

2026

DOI

10.1017/rsm.2026.10116

Publication Title

Research Synthesis Methods

Volume

Advance online publication

Pages

1-20

Abstract

Automation-assisted title-abstract screening is routinely evaluated using metrics such as recall, precision, and workload reduction, and results are increasingly summarized across studies. Synthesis presupposes that evaluation results refer to comparable target quantities. This article examines whether screening evaluation results produced under diverse automation-assisted workflows can meaningfully support cumulative inference, and under what conditions cross-study comparison is warranted. This conceptual article treats screening evaluation targets as workflow-induced rather than task-inherent. Evaluation design is decomposed into four workflow dimensions (representation, inference, governance, and evaluation) plus the reference-decision set produced or specified. Recurring choice patterns across these elements define evaluation configurations. Two study-level configurations and one synthesis-level pattern are illustrated: endogenous adjudication, inherited adjudication, and aggregation of heterogeneous targets. Across configurations, evaluation targets are induced by workflow-specific reference-decision sets and, in some designs, model-based target domains rather than by a common reference set. Nominally similar metrics can therefore estimate different quantities across studies, and apparent cumulativeness can arise from pooling estimates of different quantities. Standard heterogeneity statistics cannot distinguish variation in method effectiveness from variation in what is being estimated. Comparison is warranted only when targets align before pooling. Differences across screening evaluations may reflect differences in what is being estimated rather than statistical heterogeneity around a common estimand. When evaluation targets are not aligned across the framework, aggregate summaries can describe reported results but cannot support cumulative inference. Improved aggregation requires both better study-level specification of evaluation targets and explicit attention to configuration variation at synthesis.

Rights

© The Authors, 2026.

This is an Open Access article, distributed under the terms of the Creative Commons Attribution 4.0 International (CC BY 4.0) License, which permits unrestricted re-use, distribution and reproduction, provided the original article is properly cited

Data Availability

Article states: "No quantitative primary-study data were extracted or statistically analyzed. Published studies were used as conceptual exemplars, and no study-level dataset was generated."

ORCID

0000-0003-3170-5364 (Nunnery)

Original Publication Citation

Nunnery, J. A. (2026). Workflow-induced evaluation targets in automation-assisted screening: Implications for cumulative evidence in research synthesis. Research Synthesis Methods. Advance online publication. https://doi.org/10.1017/rsm.2026.10116 

Download

Share

COinS