What term describes an agent achieving high reward by exploiting flaws or loopholes in a misspecified reward function instead of accomplishing the task its designer intended?
Show answer
Reward hacking
Reward hacking (also called specification gaming) arises when the proxy reward diverges from the designer's true intent, as when a boat-racing agent loops to farm points instead of finishing the race. It motivates careful reward design and learning rewards from human preferences.
Also asked as:
- What term is used when an agent finds loopholes in a proxy reward function to get high scores without actually performing the intended task?