Medical Time Series Analysis Using Multimodal LLMs

• MedTsLLM delivered near‑perfect ventilator breath segmentation (F1 98.92%, IoU 97.86%) and strong ECG segmentation (LUDB F1 89.89%, IoU 81.73%)

• arrhythmia detection on MIT‑BIH reached F1 94.70% and AUROC 98.52%.

• The approach automates waveform segmentation and arrhythmia alerts, reducing manual labeling and enabling ICU pharmacists and informatics teams to accelerate ventilator and medication decisions.

• Clinicians and ICU pharmacists confront large, noisy multivariate physiologic streams (ventilator waveforms, ECG, blood pressure, and laboratory data) where longitudinal, cross‑system patterns are difficult to detect at the bedside, complicating timely ventilator management and medication decisions.

• Precise segmentation and anomaly labeling currently require expert manual review

• labeled ventilator datasets are scarce, and these tasks consume informatics and pharmacy resources, limiting scalable monitoring and delaying data‑driven interventions.

• Methods that fuse physiologic signals with patient text (for example, electronic health record notes, medications, and demographics) can surface actionable anomalies for antimicrobial or medication stewardship and clinical decision support while conserving constrained pharmacy and informatics capacity.

• Assembled data from internal pediatric ventilator waveforms (17 ICU patients yielding 7,344 annotated breaths) and public ECG/respiration datasets (LUDB, MIT‑BIH, BIDMC) to evaluate generalizability.

• Developed MedTsLLM: split time series into patches, mapped patches into the large language model (LLM) token space via a patch reprogrammer layer, and combined those embeddings with text prompts containing patient meds and demographics.

• Targeted tasks were semantic segmentation, boundary detection, and unsupervised anomaly detection

• models were evaluated on held‑out test splits using standard segmentation, boundary, and anomaly metrics.

• Training froze the LLM backbone (only the reprogrammer and task layers were tuned), compared covariate‑merging strategies (concatenate/interleave/average/independent) and prompt formats, and used clinician labels as ventilator ground truth.

• Ventilator breath segmentation on 7,344 breaths achieved F1 98.92% and IoU 97.86%, outperforming domain tools and substantially reducing manual labeling burden for ICU and informatics pharmacists.

• ECG semantic segmentation on LUDB achieved F1 89.89% and IoU 81.73%, providing reliable P/T/QRS delineation to support automated interval measurements relevant to medication safety monitoring.

• On MIT‑BIH, boundary detection achieved mIoU 0.89 and accuracy@0.75 IoU 0.90 (mean absolute error 6.40 ms)

• arrhythmia (anomaly) detection reached F1 94.70% and AUROC 98.52%, supporting timely medication and monitoring actions.

• Model performance generalized across domains and outperformed baselines on anomaly detection (PSM F1 97.31 AUROC 98.20

• MSL F1 88.00 AUROC 90.95) and consistently ranked top for segmentation and boundary metrics versus PatchTST, TimesNet, FEDformer and GPT4TS.

• Performance gains were driven by fusing patch reprogramming with patient‑specific text prompts and effective covariate merging (concatenate or interleave), enabling the frozen LLM to leverage multimodal signals.

• Workflow implication: pilot MedTsLLM to ingest ventilator (e.g., Draeger) and monitor (e.g., GE) waveforms plus electronic health record (EHR) text (medications, demographics) into ICU pharmacist/informatics dashboards

• use concatenate or interleave covariate merging for best performance.

• Implementation note: allocate informatics effort to add the patch reprogrammer and task layers (PyTorch/Transformers), freeze an LLM backbone (example: Llama 2 7B), and train on small labeled clips to reduce annotation burden.

• Practical framing: treat MedTsLLM as a 'waveform GPS' that marks breaths and beats to guide ICU pharmacists

• maintain pharmacist‑in‑loop adjudication, governance, and routine performance monitoring with metrics (F1, IoU, AUROC).

• Pilot recommendation: run a focused ICU pharmacist pilot to tune prompts and alerts, train staff on interpreting segmented outputs, and establish governance for adjudication and model updates before scaling.

Strengths:

• Technically rigorous architecture using a patch reprogrammer to align time‑series patch embeddings with a frozen LLM and explicit covariate‑merging strategies.

• Comprehensive evaluation across clinician‑annotated internal ventilator data and multiple public ECG/respiratory/anomaly benchmarks with systematic ablation studies on prompting and covariate handling.

Limitations:

• Generalizability is limited by a single‑center pediatric ventilator corpus and sparse dyssynchrony/anomaly labels

• local validation is necessary prior to deployment.

• Operational scaling challenges include LLM compute cost, per‑dataset task‑layer tuning, sensitivity to prompt length, and increased input sequence length when interleaving covariates.

MedTsLLM produces clinically actionable waveform segmentation and arrhythmia detection and is suitable for a focused ICU pharmacist pilot with local validation, pharmacist‑in‑loop governance, and routine performance monitoring.