EHR Logs Quantify Post-Alert CDS Time Tradeoffs

Predictive Model DevelopmentPredictive Process Outcomes

Authors:Chen, Jinying; Cutrona, Sarah L; Dharod, Ajay, et al.

Journal:Jamia Open

Vendor-agnostic topic modeling of EHR audit logs turns “CDS exhaust” into a Fitbit-like monitor, flagging post-alert workflow shifts within the 1-hour window (+1.3 to +2.2 minutes handling CDS, ~−2.9 minutes documentation). Pharmacy informatics and leaders gain scalable visibility to prioritize alert tuning or de-implementation, protect pharmacist verification throughput, and steer CDS governance without laborious surveys or opaque vendor metrics.

October 22, 2025

Quick Take

Topic modeling of EHR audit logs detected post‑CDS shifts in the 1‑hour after‑check‑in window: prevalence of CDS‑focused activities rose by 0.073 (~+1.30 minutes per encounter), CDS‑related activities rose by 0.098 (~+2.18 minutes), and EHR‑modification activities declined by 0.113 (~−2.92 minutes).

Pharmacy informatics teams can apply this vendor‑agnostic audit‑log + topic‑modeling approach to monitor medication‑related CDS for unintended workflow shifts and alert burden, then target tuning or de‑implementation accordingly.

Why it Matters

New electronic health record (EHR)‑embedded clinical decision support (CDS) can shift clinician attention at the point of care, producing interruptions and alert overload; measuring trade‑offs (for example, reduced chart review or documentation) is critical because they can affect patient safety and medication decisions.

Current monitoring approaches often rely on surveys, direct observation, or vendor metrics — methods that are labor‑intensive, opaque, and inconsistent across time or systems — leaving limited, scalable visibility into how CDS changes actual EHR work.

In hospital pharmacy, where medication safety and prescribing support alerts are central, transparent visibility into these shifts enables alert governance, targeted tuning or de‑implementation, and stewardship of scarce clinical time and CDS resources.

What They Did

Used de‑identified Epic EHR audit logs from five cancer clinics covering 12 months before and 12 months after implementing a tobacco‑support CDS; selected encounters for current smokers and constructed balanced cohorts.

Segmented logs into sessions (5‑minute idle threshold) and trained a structural topic model (STM) on 3,445 matched EHR sessions to infer high‑level EHR activities from micro‑level user actions.

Validated topics via expert review (four clinician/EHR experts), compared sessions with and without the alert (automatic checks), and performed supervised classification experiments for predictive validity.

Estimated encounter‑level activity prevalence and topic‑derived time‑on‑activity for the primary 1‑hour after‑check‑in window (and secondary 4‑hour before window) using propensity‑score matching (PSM) and inverse‑probability‑weighted regression adjustment (IPWRA).

What They Found

Within the 1‑hour after‑check‑in window, CDS‑focused activity prevalence increased by 0.073 (95% CI 0.066–0.079), a 39.8% relative rise, corresponding to approximately +1.30 minutes per encounter.

CDS‑related activities rose by 0.098 (95% CI 0.089–0.106), a 71.6% relative increase, ≈+2.18 minutes per encounter; concurrently, EHR modification prevalence fell by −0.113 (95% CI −0.124 to −0.102), a −39.9% relative decrease, ≈−2.92 minutes.

Reviewing patient‑data activities declined by −0.058 (95% CI −0.072 to −0.044), a 14.5% relative drop (~−0.56 minutes); activity‑level changes showed increases in CD1–CD4 (CDS handling and related tasks) and decreases in M1–M4 (EHR modification/documentation).

Operational implication for pharmacy: the observed per‑encounter shifts (CDS +1.30 and +2.18 min; modification/review −2.92 and −0.56 min) could affect pharmacist verification throughput, alert governance, and resource stewardship by reallocating verification and documentation effort.

Unclear mechanism: the analysis identifies signal and magnitude of change but does not isolate the exact causal drivers—further qualitative or experimental study is needed to determine what actually drove the observed improvements or trade‑offs.

Takeaways

Stand up a CDS monitoring pipeline: extract EHR audit logs (for example, Epic) for new best practice advisories (BPAs) and order sets, apply topic modeling, and display encounter‑level prevalence‑of‑activity for the 1‑hour post‑check‑in window in a dashboard.

For each go‑live, capture pre‑ and post‑baselines and match windows on clinic, encounter type, provider type, and window length to reduce confounding; involve clinical subject matter experts to label topics and confirm signals.

Operationalize response workflows: when monitoring shows more CDS handling accompanied by less chart review/modification, consider refining firing logic, streamlining associated documentation or order sets, providing targeted refresher training, or de‑implementing low‑value alerts.

Practical framing: treat the system like a “Fitbit for CDS” — audit logs show where minutes are going and pharmacists monitor trends. Preserve human oversight via CDS governance; do not auto‑change builds based solely on model output.

Strengths and Limitations

Strengths:

Transparent, vendor‑agnostic method using raw EHR audit logs combined with structural topic modeling, with topic number selected by coherence and exclusivity metrics.

Rigorous validation strategy: expert topic labeling, propensity‑score matching with IPWRA adjustment, predictive classification checks, and multiple sensitivity analyses.

Limitations:

Pre–post quasi‑experimental design limits causal inference; unmeasured confounding is possible and topic mixtures preclude attributing absolute time solely to the CDS intervention.

Audit‑log constraints: work done off‑EHR is unobserved; the 5‑minute idle rule and fixed 1‑hour after‑check‑in window approximate active time and may misclassify some activity; study data come from a single health system.

Bottom Line

Audit‑log topic modeling provides a scalable, vendor‑agnostic monitoring tool to detect CDS‑related workflow shifts; it is ready for pharmacy pilots to prioritize alert tuning and targeted de‑implementation while preserving governance and human review.