Frozen External Validation of a Calibrated, Uncertainty-Aware Swin Transformer for Thyroid Nodule Malignancy Classification on Ultrasound

Frozen external validation of thyroid ultrasound AI

Abstract

Deep learning for thyroid ultrasound is usually reported through internal discrimination, yet independent validation shows that performance can deteriorate across institutions and scanner environments, and studies seldom evaluate calibration, transferred operating thresholds, uncertainty behaviour and referral utility together under a protocol frozen before external data are opened. This study aimed to quantify the internal-to-external change in discrimination, classification and calibration when a model, its temperature, thresholds and preprocessing are frozen before external evaluation. Methods. A public 3,115-image Kaggle archive underwent duplicate and label-conflict review, yielding 3,079 images in fixed Train (2,155), Validation (462) and internal Test (462) sets. Four baseline architectures and nine Swin-Tiny candidates were compared using Validation data only. The selected configuration used focal loss, Gaussian training noise (0.03), temperature scaling and 20-pass Monte Carlo (MC) Dropout. It was evaluated once internally, then on all 5,000 TN5000 images and the official 1,000-image Test subset without fine-tuning, recalibration or threshold reselection. Results. Internal Test AUROC was 0.879 (95% CI 0.848–0.909); sensitivity 0.827 and specificity 0.753 at threshold 0.50. Full external AUROC was 0.828 (95% CI 0.817–0.841), sensitivity 0.910 and specificity 0.529, producing 672 false positives among 1,426 benign images. The official Test subset yielded AUROC 0.825 and sensitivity 0.915. Exact deduplication had negligible effect (AUROC 0.827). At 70% coverage, referring the most uncertain cases raised accuracy to 0.857 and sensitivity to 0.951. Conclusion. Discrimination transferred but specificity did not, demonstrating that internal accuracy alone is insufficient to characterise transportability. The frozen, leakage-audited framework supports only human-supervised triage.