Which 2023 method aligns language models directly from preference data with a simple classification-style loss, removing the separate reward model and reinforcement-learning loop used by RLHF?

Show answer

Direct Preference Optimization (DPO)

Direct Preference Optimization reparameterizes the RLHF objective so the optimal policy can be trained directly on preference pairs, with no explicit reward model or PPO loop. This makes alignment training markedly simpler and more stable while targeting the same preference objective.

Also asked as:

What 2023 alignment method bypasses the need for an explicit reward model by reparameterizing the RLHF objective into a simple classification loss?

Would you remember this in 6 months?

Trivia Forge uses spaced repetition so facts stick. Free, no signup needed.

Turn on Machine Learning practice →

More Machine Learning trivia

Which activation function is defined by the formula f(x) = max(0, x)?
Which activation function is most commonly used in the final layer of a multi-class classification neural network to produce a probability distribution?
Which adversarial architecture consists of two networks, a Generator and a Discriminator, competing in a zero-sum game?
Which 1996 clustering algorithm is density-based and can identify outliers as 'noise' points?
Which clustering algorithm iteratively assigns points to the nearest cluster center and then updates the center based on the mean of assigned points?
Which dimensionality reduction technique minimizes the Kullback-Leibler divergence between similarity distributions in high and low-dimensional spaces?

See all 98 →