Smart people in 2024: obviously the 00s-era AI misalignment story ("we tell AGI to end suffering, it kills everyone, no more suffering") is too naive, we have a smarter version

2025: coder tells AGI to remove bugs, it deletes all code, no more bugs

https://x.com/alex_xiong_/status/1942442882402771178

Seems to me there should be two different bots, pitted against each other: one does the code and a different one does the testing.

The first gets rewarded for code that passes the tests, the second gets rewarded for finding bugs in the code / deviations from spec. Adversarial AI.

Founder, The Roots of Progress (rootsofprogress.org). On book-writing sabbatical, trying not to check social media including DMs

I can see in the future a kind of testing where its a bot that is always running in CI, where it does pen testing, and trying different things. It's sort of a different kind of chaos monkey. Testing is an endless latent space that needs to be tested. And as the code changes, the intricate paths that one could take to do 'coverage' will also change. That means that while the 'coding' agent will be require initially to do a lot of work, the testing needs to be significantly more substantial as the code grows - and will pretty much take all the tokens as the codebase grows over time.