Contrastive decoding is an LLM inference method that improves generation quality by combining scores from two models (or two modes of one model): a strong “expert” model and a weaker “amateur” model that penalizes generic continuations. It often yields more specific, less repetitive text without retraining.
What is Contrastive Decoding?
Instead of selecting the next token purely from the expert model, contrastive decoding scores tokens using a difference like logP_expert − α·logP_amateur. The expert keeps the output coherent, while the amateur acts as a commonness detector: tokens preferred by both models are often generic, so the subtraction down-weights them. The method works best when the models are compatible and the expert is meaningfully better.
Where it’s used and why it matters
Contrastive decoding is used in assistants and summarizers where greedy decoding feels bland. It’s attractive because it’s deployment-time: you can tune α and roll it out per endpoint. The main drawback is extra compute and latency from running two models per step.
Examples of Contrastive Decoding in Practice
- Product Q&A: reduce boilerplate and increase specificity.
- Summarization: avoid generic openers and repetitions.
- Chat assistants: improve variety while keeping coherence.
FAQs
How is this different from temperature or top-p? Those reshape the expert distribution; contrastive decoding changes scoring by subtracting an amateur preference.
Do I need two models? Typically yes, though the amateur can be a smaller or cheaper variant.
When can it hurt? With a mismatched or overly weak amateur model, or when latency budgets are tight.
How can I test it? Implement a two-model decoding loop, sweep α, and measure repetition/factuality on a small eval set.
