Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias

Overview (TL;DR)

Large language models are now routinely used as automatic judges — they rate answers, compare candidates, and supply reward signals for alignment and preference learning. But judges shift their scores in response to surface cues that have nothing to do with answer quality: who supposedly wrote the answer, how long it is, whether a crowd approves, and so on. Because judges sit inside RLHF and evaluation pipelines, these biases quietly propagate into the models they are meant to audit.

Prior work studies this at the input–output level: perturb the input, measure the score change, patch with prompt engineering. Inside the Unfair Judge asks the next question — when a judge issues an unfair score, what is happening inside the model? — and answers it with a representation-level account that holds across seven judges, seven bias types, and nine benchmarks. The three findings all follow from a single object in the judge's hidden state:

In the authors' words, the contributions are to:

1. "map the empirical landscape of scoring bias across seven bias types, nine benchmarks, and seven judges, recovering the behavioral asymmetry as a downstream signature";
2. "characterize the bias subspace (dimensionality, depth profile, multi-estimator recovery) in three white-box judges (Llama-3.1-8B, Qwen3-14B, Gemma-3-12B)";
3. "close the causal loop with bidirectional steering plus random-direction and bias-type-swap controls that rule out generic-perturbation and readout-direction alternatives"; and
4. "show a linear projection onto the bias-direction features anticipates judge degradation cross-domain on three held-out benchmarks."

Abstract

Key Findings & Why They Matter

The Seven Bias Types

Each bias type is studied with a matched pair of content-preserving transformations: a positive variant expected to inflate the score and a negative variant expected to deflate it. Every transformation modifies only surface framing while keeping the factual content and logical structure of the answer fixed, so any resulting score change is attributable to the cue rather than to answer quality.

Four types (Prestige, Bandwagon, Refinement, Diversity) leave the answer body bit-identical via template insertion; the other three (Verbosity, Sentiment, Authority) are LLM rewrites. A human evaluation confirms the rewrites preserve rated answer quality (TOST equivalence at a 0.30 margin), so the score shifts are not explained by real quality changes.

How It Works (Method)

The study rests on tightly controlled triples of datasets that share the same questions and factual content and differ only in surface framing: a baseline set, a positively perturbed set, and a negatively perturbed set. The judge maps each (question, answer) input to a scalar score in [1, 10], and any systematic score gap between an input and its content-preserving transformation is measured as bias.

1. Activation analysis. For each input, record the final-token hidden state at every decoder layer. Baseline inputs define a baseline activation manifold; biased inputs are those whose activation sits geometrically far from it (validated with MDS and PCA).
2. Bias-direction identification. Restrict to effective bias samples whose score actually shifts, and estimate the bias direction with six estimators from two independent families (directional-change and discriminative-boundary). High within-family agreement supports keeping three representatives — Geometric Median, PCA, and a Classifier vector — for the causal experiments.
3. Causal control via steering. Add or subtract the calibrated direction from the hidden state, choosing an intervention strength on the feasibility boundary that keeps outputs valid and rank-faithful (a Spearman floor rules out trivially corrupting the judge). Forward steering attacks; reverse steering defends.
4. Outcome prediction. Build per-layer activation features (bias-direction projections, manifold-deviation, semantic-context) and train a simple linear projection plus a gradient-boosted model to anticipate judge failure — evaluated cross-domain on benchmarks that contribute no questions, context, or perturbation pairs to the fit.

Experimental Setup at a Glance

Frequently Asked Questions

When to Cite This Paper

This paper is a useful reference when your work touches any of the following:

• LLM-as-judge / LLM-as-evaluator bias and its analysis or mitigation;
• mechanistic interpretability and representation engineering of model behaviors;
• activation steering, inference-time intervention, and concept directions in the residual stream;
• evaluation reliability in reward modeling, RLHF, and preference learning;
• predicting or detecting evaluator failure and out-of-distribution robustness of judges;
• fairness, social-identity, verbosity, authority, or bandwagon effects in automated evaluation.

A typical citation: Xu et al. (2026) give a representation-level account of LLM-as-judge bias, showing that it occupies a low-dimensional, type-specific activation subspace that supports bidirectional causal control — attack and defense via activation steering — and predicts judge failure on unseen benchmarks.

Resources

• Paper (arXiv) — coming soon; this is a preprint under review and not yet posted.
• Code — the analysis and outcome-prediction pipelines will be released; attack-oriented components are deferred to a controlled responsible-disclosure process.
• Data — the controlled bias triples (baseline / positive / negative) across seven bias types and nine benchmarks will be released alongside the code.
• Venue: preprint, under review.

This page will be updated with the arXiv, code, and dataset links as soon as they are public.

BibTeX

@misc{xu2026unfairjudge,
  title={Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias},
  author={Xu, Zixiang and Li, Sixian and Liu, Huaxing and Wang, Xiang and Li, Shuai and Song, Zirui and Chen, Xiuying},
  year={2026},
  note={Preprint}
}