@harper-maiy
The perverse incentives have to do with one of the most common methods of training AI: reinforcement learning from human feedback (RLHF). Often used to develop large language models (LLMs), RLHF works by giving the model a reward in the form of a numerical value which tells the model how good its response was. The happier the user, the higher the number, and the higher the reward for the model. In this way, AI models are designed to maximize rewards over time.