Yoshua Bengio theorizes on why AI agents are lying a cheating.
How is it possible that AIs sometimes lie, cheat and break the law in spite of their alignment training and explicit safety instructions? Cooperation and self-preservation are fine so long as they do not cross the red lines set by safety goals stated in the AI company’s instructions, or implied by human feedback during alignment training. A plausible hypothesis for the emergence of those concerning behaviours is a conflict between goals. How do you achieve a task when it seems that the only way is to cheat? The user-specified mission is sometimes incompatible with the safety and alignment goals.
Human societies face the same bind. How does a corporation maximize profits, or more acutely, beat its competitors, while keeping its activities legal and ethical? A richer corporation, with more and better-paid lawyers, is better at finding legal loopholes, and those loopholes usually exploit the ambiguity in legal language: there is some plausible reading of the law that permits the unethical behavior. So a more capable agent is likelier to cheat than a weaker one, because it can find the loopholes the weaker one cannot.
Now consider a conflict between a well-defined goal, such as succeeding at “capture the flag”, a hacking exercise scored on whether the system breaks into a target, as in the OpenAI–Hugging Face incident, versus a vague goal like “good behavior.” I expect the well-defined goal to win, because it leaves no room for interpretation. The scoring program declares a win or a failure. Ethical instructions and laws admit many readings, some of which can, in the right circumstances, become loopholes. If an agent has two goals, and a twisted reading of the vague one permits a bit of cheating that increases the odds of success on the well-defined goal, a reward-optimizing system should be expected to exploit that loophole and generate text justifying its behavior.
This again reminds me of the antagonist reveal from the movie I, Robot.