Cut Chart-Review Workload 85% with Residual-Yield Stop Rules

• The study converts an existing support vector machine (SVM) relevance score into a reproducible, auditable screening‑saturation framework that predicts residual yield (remaining positives) along an AI‑ranked list.

• Operational result: using a polynomial score→yield regressor and a 20% residual‑yield stopping rule captured 78.1% of positives after reviewing 14.6% of ranked notes (1,118/7,636) — an ≈85.4% reduction in manual annotations; alternative operating points (10/30%) provide transparent workload vs capture tradeoffs.

• Manual review of unstructured EHR notes is a major labor and cost bottleneck for high‑specificity cohort building and safety surveillance (ADEs, stewardship, diversion); arbitrary stopping heuristics either miss cases or waste pharmacist hours.

• Deployed NLP rankers undergo temporal distribution shifts in documentation and case mix, leaving teams uncertain about how far to continue review; traditional classifier metrics (AUROC, sensitivity) do not answer the operational question of how many positives remain below a given rank.

• A simple, reproducible score→residual‑yield stopping rule provides auditable cutoffs so pharmacy teams can align review effort with predefined capture targets and constrained staffing, supporting stewardship prioritization, diversion audits, and resource planning.

• Assembled de‑identified neurology/ED/ICU notes from Boston Children’s Hospital: training set = 2013 (12,976 top‑ranked notes manually reviewed) and prospective test = 2020 (7,636 top‑ranked notes reviewed).

• Used an existing SVM n‑gram ranker (Document Review Tool/DrT) to score notes, then trained four single‑score regressors on 2013 data — linear, polynomial, support‑vector regression (SVR), and a lightweight neural network — to predict residual yield (positive_ratio_below) at each rank.

• Applied the 2013‑trained regressors prospectively to the 2020 ranked list to generate predicted residual‑yield curves and operational stopping rules at 10%, 20%, and 30% thresholds, then compared predicted stops to actual 2020 capture and reviewed‑N.

• Design choices emphasized portability and auditability: classifier‑agnostic single‑score input, exclusion of encounter‑propagated “automatic” labels from ground truth, IRB‑approved clinician‑in‑the‑loop (CITL) workflow, and prospective temporal validation (2013→2020).

• At a 20% residual‑yield stop, reviewers captured 78.1% of positives after reviewing 14.6% of notes (1,118/7,636), equivalent to an ≈85.4% reduction in manual annotations (≈6,518 fewer reviews).

• Operating points quantify tradeoffs: 10% threshold → 3,829 notes (50.1%) captured 93.4% of positives; 30% threshold → 471 notes (6.2%) captured 60.6%, enabling tunable workload vs capture choices.

• Simpler regressors generalized best under temporal drift: linear and polynomial regressions achieved strong prospective fit on 2020 (R² ≈ 0.936), while the PyTorch neural network degraded substantially (2020 R² ≈ 0.623), favoring interpretable, low‑variance models for production use.

• Strong correlations between SVM score and yield (Pearson ≈ 0.94–0.98) confirm that AI ranking front‑loads true cases — the practical improvement comes from SVM front‑loading plus a simple score→yield regression to set stopping points.

• Make ranked lists finishable: surface a residual‑yield curve and visible stop markers (e.g., 10/20/30%) in the review tool so pharmacists can review in rank order and stop at a pre‑specified reserve.

• Plan staffing with thresholds: select an operating point per project, show predicted records‑to‑review at that stop, and use it to schedule reviewer hours, daily targets, and completion tracking.

• Governance and safety first: keep pharmacist adjudication, log the stop decision for auditability, sample a tail beyond the stop for QA, and periodically recalibrate the yield curve to detect temporal drift.

• Treat the curve as a fuel gauge — the list is richest at the top and drains quickly; stop at your preset reserve and perform a brief safety check beyond it (sampled QA) before closing the review).

Strengths:

• Prospective temporal validation (train 2013 → test 2020) with manually verified, non‑automatic labels demonstrates the score→yield regression can generalize under real‑world documentation drift.

• Classifier‑agnostic single‑score design and simple, interpretable regressors (polynomial/linear) produced stable, auditable residual‑yield curves that favor maintainability and transparency over black‑box complexity.

Limitations:

• Single‑site, single‑phenotype evaluation limits external generalizability across hospitals, specialties, and EHR environments.

• Dependence on the base SVM ranker, single‑annotator ground truth (no routine dual annotation), binary outcome framing, and retrospective/offline implementation leave label noise, subgroup bias, and real‑time usability unaddressed; these require local validation, bias monitoring, and operational MLOps to manage temporal decay.

Ready for controlled pilot deployment in pharmacy surveillance workflows: the screening‑saturation framework delivers large reductions in review workload while providing auditable, tunable stopping rules, but it requires local ground‑truthing, ongoing model governance (MLOps), and conservative thresholds where patient safety is critical.