An LLM judge prefers whichever answer you show it first

If you ask a model to pick the better of two answers, it has a real preference for the one presented first. Swap the order and a meaningful share of verdicts flip. The MT-Bench and Chatbot Arena paper documents this alongside verbosity bias, where judges favor longer answers, and self-enhancement bias, where a model favors its own outputs.

This matters more than it sounds. If you are running an eval where variant A is always in the first slot and variant B is always in the second, some fraction of your reported win rate is just slot assignment. You will ship the change and wonder why the offline number never showed up in production.

What I do now:

• Every pairwise comparison runs twice, once in each order. If the two runs disagree, that pair is a tie, not a coin flip you get to keep.
• Track the disagreement rate as its own metric. It tells you how much of your eval is signal. A judge that contradicts itself on a third of pairs is not measuring quality.
• Watch length. If the winner is reliably the longer answer, control for it before believing the result.

The broader lesson is that an LLM judge is an instrument, and instruments need calibrating before you trust a reading. The temptation is to treat the judge as ground truth because it produces confident prose. It produces confident prose either way.