Center for Secure and Intelligent Critical Systems (CSICS) Publications
Towards Reliable Lung Cancer Prediction: A Hybrid Framework for Noise Reduction and Uncertainty Control
ORCID
0000-0001-9199-2479 (Rana), 0000-0002-8789-0610 (Shetty)
Document Type
Conference Paper
Publication Date
2025
DOI
10.1609/aaaiss.v7i1.36932
Publication Title
Proceedings of the AAAI Symposium Series
Volume
7
Issue
1
Pages
558-565
Conference Name
2025 AAAI Fall Symposium Series, November 6-8, 2025, Arlington, Virginia
Abstract
Uncertainty remains a critical challenge in healthcare AI, since predictive errors can directly compromise patient safety and undermine trust. Structured clinical datasets in healthcare are frequently characterized by heterogeneous acquisition protocols, incomplete records, and inconsistent or noisy encodings. This inflates aleatoric uncertainty and weakens calibration. These challenges are exemplified in lung cancer risk modeling, where small cohorts, variable collection practices, and limited feature quality make the problem especially acute. Significant advances in uncertainty quantification (UQ) have been achieved in imaging and signal processing through Bayesian inference, evidential learning, and robust architectural designs. In contrast, tabular clinical datasets remain a critical yet underexplored domain. Addressing this gap requires methods that are lightweight, certifiable, and effective on noisy datasets without relying on large models or data. Considering this challenges, we propose a frequency-aware hybrid representation that combines Principal Component Analysis (PCA) with the Discrete Cosine Transform (DCT). Using mutual information (MI)–based feature ordering, the framework suppresses high-frequency artifacts while preserving discriminative structure. As the framework was applied to a publicly available lung cancer dataset, it demonstrated an accuracy improvement from 98.1% to 99.7%, reduced Negative Log-Likelihood (NLL) by 82% from 5.25% to 0.94%, lowered aleatoric uncertainty from 10.50% to 3.35% (68% reduction), and preserved AUROC at 99%. We evaluated the framework across three publicly available lung cancer datasets where it demonstrated a reduction in aleatoric uncertainty by 7% on an average, confirming generalizability. The Wilcoxon signed-rank test confirms that the results are statistically significant. This work shows that part of the ‘irreducible’ variability is actually compressible noise, thereby facilitating more reliable and uncertainty-aware AI for healthcare.
Rights
© 2023, Association for the Advancement of Artificial Intelligence. All rights reserved.
"In the Returned Rights section of the AAAI copyright form, authors are specifically granted back the right to use their own papers for noncommercial uses, such as inclusion in their dissertations or the right to deposit their own papers in their institutional repositories, provided there is proper attribution. The published version is not available for posting outside the AAAI Digital Library."
Original Publication Citation
Pal, S., Roy, S., Rana, P., Banerjee, A., Majumder, K., & Shetty, S. (2025). Towards reliable lung cancer prediction: A hybrid framework for noise reduction and uncertainty control. Proceedings of the AAAI Symposium Series, 7(1), 558-565. https://doi.org/10.1609/aaaiss.v7i1.36932
Repository Citation
Pal, S., Roy, S., Rana, P., Banerjee, A., Majumder, K., & Shetty, S. (2025). Towards reliable lung cancer prediction: A hybrid framework for noise reduction and uncertainty control. Proceedings of the AAAI Symposium Series, 7(1), 558-565. https://doi.org/10.1609/aaaiss.v7i1.36932