Rethinking Pharmacy AI Exams: The New Resident

• A proprietary large language model (LLM) adaptation (AMIE, based on Gemini 2.5 Pro) outperformed oncology trainees on a 60-case synthetic breast cancer vignette set but did not exceed attending oncologists; reported rates of harmful recommendations were low.

• Pharmacy implication: treat deployed AI as a supervised, fast-learning 'digital resident' — shift from one-time validation to continuous oversight (interactive interrogation, sandbox experiential learning, and real‑world continuous learning) to manage new safety and equity risks.

• One-time, question-and-answer benchmarks miss the context-dependent medication decisions pharmacists make; as AI systems gain experiential learning, static predeployment testing will not reveal hidden reasoning failures or misalignment with care goals.

• Pharmacy workflows require reviewing disorganized data, coordinating care, and navigating trade-offs (for example, additional testing versus timely treatment, payer denials, and goals-of-care conflicts); without interrogation and sandbox testing, AI may induce uncritical adoption and scale unvetted practices.

• Operationally, this requires funded stewardship, secure near‑real‑time research sandboxes, interoperable data pipelines, and dedicated resources for continuous clinical decision support (CDS) governance and oversight.

• Described a case study running a proprietary large language model (LLM)–based system (AMIE — Articulate Medical Intelligence Explorer, adapted from Gemini 2.5 Pro) on 60 synthetic breast cancer vignettes and compared its recommendations with those from oncology trainees and attending oncologists.

• Proposed a three‑pillar evaluation framework: interactive interrogation (oral board–style dialogue), sandbox experiential learning (high‑fidelity simulated patient trajectories), and real‑world continuous learning (postdeployment monitoring and iterative improvement).

• Called for secure, near‑real‑time research sandboxes using de‑identified historical data, open APIs, and clinician‑in‑the‑loop oversight to allow safe experiential testing.

• Argued for shifting evaluation from one‑time tests to iterative, experience‑based validation that aligns with clinical workflows.

• The AMIE/Gemini 2.5 Pro adaptation outperformed oncology trainees on the 60‑case synthetic vignette set, with low reported rates of harmful recommendations, but it did not outperform attending oncologists.

• The editorial highlights gaps in current benchmarks: clinicians spend only about 40% of their time at the bedside, while top LLMs score roughly 25% on broad 'Humanity’s Last Exam' benchmarks, demonstrating that question‑and‑answer tests can overstate narrow task performance and miss clinical judgment.

• Authors recommend replacing one‑time validation with the three‑pillar approach — interactive interrogation, sandbox experiential learning, and real‑world continuous learning — to enable continuous, encounter‑level assessment.

• Pharmacy implication: regard deployed AI as a 'digital resident' that will evolve through agentic self‑reflection, internet access, and experiential learning, requiring iterative updates and continuous oversight tied to medication outcomes.

• Expect AI tools to behave less like static dosing calculators and more like fast‑learning trainees whose recommendations evolve with experience — requiring ongoing clinical supervision rather than blind trust.

• Evaluation will increasingly resemble an oral board: pharmacists must probe the AI’s reasoning, ask it to revise plans with new data (e.g., labs, genomics), and challenge contradictions before acting.

• Meaningful deployment depends on rich, interoperable data and sandbox‑style testing where models can 'practice' on realistic trajectories without harming patients.

• For pharmacists, oversight resembles a continuous morbidity‑and‑mortality process: every encounter can improve the system, but human judgment must filter which lessons are safe to scale.

Strengths:

• Presents a clear, actionable three‑pillar blueprint — interactive interrogation, sandbox experiential learning, and real‑world continuous learning — for safer, human‑aligned clinical AI.

• Grounds recommendations in recent clinical AI evidence and system realities, emphasizing modern data infrastructure, open APIs, and clinician–engineer partnerships.

Limitations:

• This is a conceptual editorial; the proposed 'Humanity’s Next Medical Exam' framework and research sandboxes are not yet empirically tested or fully operationalized.

• Illustrative examples rely largely on a single synthetic oncology vignette study and high‑level scenarios, leaving questions about scalability and implementation across diverse pharmacy settings unanswered.

The editorial calls for shifting pharmacy AI governance from one‑time predeployment validation to continuous, sandboxed 'digital resident' oversight — operationalizing interactive interrogation, sandbox experiential learning, and real‑world continuous learning before scaling clinical deployment.