In the standard RLHF pipeline, what intermediate model is trained to predict human preference rankings so that it can supply a learned reward signal during reinforcement learning?
Show answer
Reward model
Because querying humans for every policy update is infeasible, RLHF first fits a reward model to pairwise human preferences (often with a Bradley-Terry objective). The policy is then optimized against this learned reward, with a KL penalty keeping it close to the original model.
Also asked as:
- In the RLHF process, what specific component is fitted to pairwise human preferences to provide the reward signal for policy optimization?