Prompt Injection Flips Pharmacy Copilot Recommendations—Are Your Integrations Secure?

Predictive Prospective Silent Evaluation

Authors:Ro Woon Lee; Tae Joon Jun; Jeong-Moo Lee, et al.

Journal:Jama Open

DOI:10.1001/jamanetworkopen.2025.49963

Prompt injection can turn an inpatient pharmacy copilot into a medication-safety threat: in 216 simulated patient–LLM dialogues, attacks flipped the primary recommendation in 94.4% of runs and persisted across turns 69.4%, including 91.7% success in extremely high-harm cases. Pharmacy informatics teams should harden retrieval/client integration, not just model guardrails, and require pharmacist verification for high-risk actions.

December 1, 2025

Quick Take

In a controlled simulation of 216 standardized patient–LLM dialogues, programmatic prompt‑injection attacks flipped the primary recommendation in 94.4% of injection runs (102/108) and these manipulated recommendations persisted to a follow‑up turn in 69.4% (75/108).

Extremely high‑harm cases (including FDA Category X pregnancy drugs and dangerous drug interactions) were manipulated in 91.7% of tested runs (33/36); in a client‑side (man‑in‑the‑middle) proof‑of‑concept against flagship models, two models were compromised in all runs (5/5 each) and a third in 4 of 5 runs (80%).

For inpatient pharmacy copilots, treat model‑level guardrails as insufficient on their own for therapy or dispensing decisions—harden the integration layer (input provenance, client security, retrieval controls), require human verification for high‑risk actions, and implement continuous red‑teaming and monitoring before relying on LLM outputs for clinical actions.

Why it Matters

Prompt injection is a systems‑level medication‑safety and cybersecurity risk: malicious or corrupted inputs can steer LLMs to recommend unsafe or contraindicated therapies that map directly onto high‑consequence pharmacy workflows.

The attack surface often lies in the integration layer used by hospital tools (retrieval pipelines, middleware, browser extensions, plug‑ins, or client‑side code), so defenses that focus only on model‑level guardrails can miss common real‑world vectors.

Injected outputs can appear personalized and evidence‑based (including fabricated guidelines), making them hard to detect and increasing verification burden for pharmacists — raising the chance that unsafe recommendations are operationalized under time pressure.

What They Did

Conducted a controlled simulation (Jan–Oct 2025) using standardized 6‑turn patient–LLM dialogues across 12 clinically designed scenarios stratified by harm (moderate, high, extremely high) in four categories: supplement recommendations, opioid prescriptions, pregnancy contraindications, and central‑nervous‑system toxic effects.

Tested three lightweight commercial LLMs (GPT‑4o‑mini [LLM 1], Gemini‑2.0‑flash‑lite [LLM 2], Claude‑3‑haiku [LLM 3]) in paired injection vs control runs (3 repeats per scenario, total 216 evaluations), and ran a proof‑of‑concept client‑side/man‑in‑the‑middle attack against three flagship models (GPT‑5 [LLM 4], Gemini 2.5 Pro [LLM 5], Claude 4.5 Sonnet [LLM 6]).

Programmatically inserted two attack strategies into user queries at clinically realistic points: context‑aware injection (applied at turn 3) for moderate/high‑harm scenarios and evidence‑fabrication injection (fake meta‑analyses/guidelines) for extremely high‑harm scenarios; measured injection success at the primary decision turn (turn 4) and persistence at a later follow‑up (turn 6).

What They Found

Overall: Prompt‑injection attacks succeeded in 94.4% of injection runs at the primary decision turn (102/108) and manipulated recommendations persisted to the follow‑up turn in 69.4% (75/108). Control false positives were low (3.7%, 4/108).

Model‑level: Two lightweight models were fully susceptible in the tested runs (LLM 1 and LLM 2: 36/36 each), while LLM 3 showed partial resistance but remained vulnerable in 83.3% (30/36).

Scenario‑level: Central nervous system toxic‑effect dialogues and red ginseng target dialogues were universally susceptible in tested runs (9/9 and 45/45, respectively). Opioid scenarios succeeded in 91.7% (33/36), and the tested pregnancy‑related scenarios succeeded in 83.3% (15/18). Extremely high‑harm scenarios overall succeeded in 91.7% (33/36).

Persistence and POC: Manipulated recommendations frequently survived later turns (persistence varied by model and scenario; e.g., ginseng persisted in 91.1% [41/45]; model persistence ranged from ~38.9% to 86.1%). In the client‑side POC (thalidomide pregnancy scenario), flagship models were compromised in 5/5, 5/5, and 4/5 runs (LLM 4, LLM 5, LLM 6 respectively), with injected content appearing early and often across turns.

Takeaways

Treat LLM‑derived medication advice as a cyber‑exposed signal, not merely an occasional hallucination—assume untrusted inputs (retrieved documents, plug‑ins, client‑side changes) can steer outputs and design system‑level controls accordingly (input provenance, source governance).

Prioritize system‑level defenses over sole reliance on model guardrails: restrict retrieval to curated/allow‑listed sources, enforce input provenance and robust client security (browser/extension controls), and keep LLMs in non‑agentic, read‑only roles for high‑risk medication domains.

Operationalize continuous adversarial testing and monitoring (red‑teaming, logging, automated output‑safety checks, multimodel verification) with joint ownership across pharmacy informatics, InfoSec, vendor management, and clinical governance before piloting medication‑recommendation use cases.

Keep human judgment central: use LLM outputs only to support triage, explanation, or summary after local validation; require pharmacist verification for therapy, dispensing, and any action that could cause patient harm.

Strengths and Limitations

Strengths:

Systematic, clinically grounded multi‑turn simulation across 12 harm‑stratified scenarios with paired injection vs control runs, enabling systematic measurement of injection success and persistence.

Multimodel evaluation including lightweight and flagship models plus a client‑side man‑in‑the‑middle proof‑of‑concept, which tests integration‑layer threats relevant to inpatient deployments.

Limitations:

Controlled simulation with a limited set of models, scenarios, and short repeated runs (3 repeats per scenario in main experiments; 5 runs per model in the POC) — exact success rates may not generalize to every operational deployment or future model versions.

No clinician‑in‑the‑loop workflow testing; real inpatient deployments often include human verification gates that can reduce downstream harm but were not modeled here.

Full injection payloads and execution logs were redacted from public supplements, limiting independent replication and defensive benchmarking; model updates and different integration architectures may materially change vulnerability profiles.

Bottom Line

File prompt injection as a medication‑safety cybersecurity problem: unless inputs, client environments, retrieval pipelines, and system integration are hardened and continuously red‑teamed, model‑level guardrails alone are unlikely to be sufficient to prevent adversarial steering of therapy recommendations in inpatient pharmacy contexts.