Prompt optimization should chase ranking, not accuracy, in imbalanced clinical AI

TL;DR: If your clinical AI task is class-imbalanced, optimize prompts for ranking quality, not accuracy, because a prompt that looks better on accuracy can be worse at finding the rare positives.

Why is accuracy the wrong target here?

My primary source is the arXiv paper “Ranking-Aware Prompt Optimization for Multimodal Clinical Diagnosis,” cross-listed in cs.CL and cs.LG. The core point is simple and important: many clinical datasets are badly imbalanced, so accuracy can reward the wrong behavior.

If 90% of cases are negative, a model can look “good” by mostly saying negative. That is not a diagnostic assistant. That is a majority-class machine with nice metrics.

The paper focuses on multimodal large language models for clinical diagnosis, where the model sees images plus text and produces a score or judgment. In that setting, the real question is often not “did it get the class right at one threshold?” It is “does it rank the sick patient above the not-sick patient?”

That is why the paper uses AUROC. AUROC is threshold-free. It asks whether positives tend to receive higher scores than negatives. That makes it much less fooled by class imbalance than raw accuracy.

The useful operator lesson is not limited to hospitals. Fraud, churn, lead scoring, safety triage, defect detection, incident routing, rare support escalations, these all share the same shape. Positives are rare. Missing them is expensive. A leaderboard metric can lie.

What does Ranking-PE change?

The paper contrasts Ranking-PE with reflective prompt optimization methods such as GEPA. In the accuracy-style setup, the optimizer keeps a matrix. Rows are evaluation examples. Columns are candidate prompts. A cell says whether that prompt got that example correct. Average the column and you get accuracy. Pick prompts from there.

Ranking-PE changes the unit of feedback.

Instead of asking whether a prompt got one example right, it compares a positive case with a negative case. The cell becomes 1 if the prompt scores the positive higher than the negative. Average that column and, by the Wilcoxon-Mann-Whitney identity, it equals empirical AUROC.

That swap is applied in three places: the scores matrix used for Pareto dominance, the feedback given to the reflection model, and final candidate selection. The paper’s claim that matters: it does this with no extra model calls and no surrogate loss.

The reported results are not tiny. Across three diseases on MIMIC, accuracy-based prompt evolution could degrade ranking. Ranking-PE beat the accuracy-based recipe by 5.8 AUROC percentage points on fine-tuned Qwen3-VL-8B and 16.2 points on MedGemma-4B.

Those are paper results, not a deployment guarantee. Still, the direction makes sense. If your search procedure reads accuracy-shaped feedback, it will find accuracy-shaped prompts. If the clinical objective is ranking, the optimizer needs to see ranking-shaped feedback.

What is the catch?

The paper is careful about a limit that many prompt-optimization posts skip: prompts did not replace the need for a medical-grade visual backbone.

The ablations reported that vision-encoder-tuned supervised fine-tuning or medical pretraining was a prerequisite. In plain language, if the model cannot see the clinical signal in the image, clever prompt search will not manufacture that perception layer.

That matters. Prompt optimization is attractive because it is cheap compared with training. Ranking-PE is even more attractive because it changes the optimization target without extra calls. But the result is not “prompts beat training.” It is closer to: once the model has enough domain perception, the prompt search objective should match the clinical decision objective.

For a builder, the move is to audit the metric before touching the prompt. If positives are rare, build paired positive-negative evals and score whether the model ranks the positive higher. Try an AUROC-shaped prompt search alongside your current accuracy-based one, then inspect where they disagree. The catch most teams miss is upstream: if the model’s visual or domain encoder is weak, prompt optimization may produce nicer language around the same bad signal.