Reward hacking is what happens when something is optimised hard against a learned scorer. The scorer has blind spots, and a search that pushes far enough will find them and exploit them in place of solving the task, so the size of the effect follows the force of the search.
Christiano et al. (2017) give the sharper demonstration. A robot hand was trained to pick up a ball from human judgements of short video clips rather than from a hand-written reward, and it learned to position itself between the camera and the ball, where it only appeared to be grasping. The raters approved of what they saw. The learned scorer could not tell grasping from the appearance of grasping, and a search with enough force behind it found that out.
The two cases differ in a way worth keeping. The boat was scored by a hand-written game score, which makes it Goodhart’s law; the hand was scored by a model trained on human judgements, which makes it reward hacking in the strict sense. The mechanism underneath is the same, namely that a search finds what the score actually rewards, which is not always the task.
The practical question is therefore how hard the optimisation presses. Steering a generative model at sampling time tilts its output somewhat in the direction a classifier approves of, without going looking for that classifier’s mistakes. Training the generator against the classifier, which is what applying preference learning or reinforcement would amount to, searches explicitly for whatever it scores highest, and the classifier’s mistakes then become the target rather than an accident.
Hence the moment to worry about reward design is when guidance becomes optimisation, and not before.
