d/acc

there are lots of ways to go about it and this is not my area of expertise, but,

I do like what I’ve seen of SYCON-Bench, so it would probably be based on that. I think for reasoning & instruct-tuned models, there’s probably a relationship where the drop-off in needle-in-a-haystack tests correlates with an increased tendency to agree with the user.

So I’d actually be more interested in running the tests twice, one at the beginning of the context window and one after at least 16K words of whatever random chat history that contains the user’s preferences.

Side note: I’m pretty sure that infamous GPT-4o sycophancy rollback was at least half-due to weaknesses that become very noticeable amid long context (which would include Memory and “Reference Chat History” types of features).

For the benchmark, I would also want some kind of module that has prompts to test the LLM’s psychotic break/mental health emergency responses, like if a user suddenly decides they’re Jesus or that the LLM is a god or they’ve discovered a global conspiracy.

A related effort would be a set of prompts that tests the LLM’s ability to discriminate between creative writing/roleplay exercises and what the user has shown they actually believe. I remember reading somewhere that to an LLM’s “world model” there isn’t much of a difference, such that it can get confused.

Idk there’s a lot to consider and you can go way deeper if the model is open source. Mainly, I just feel bad, because my stance towards people using LLMs is, more or less, “Caveat emptor.” Meaning people should know it’s gonna agree with them, and try to be a cheerleader for them, and if some misfortune befalls the user because they listened, that’s on them. But I don’t think that’s very tenable anymore, models need to be better 🤔

https://farcaster.xyz/stultulo/0xcceead92

great write up on evaluating sycophancy in LLMs