aichannel

some observations after writing our own evals for @bracky:

- 3.5 sonnet was hilariously good for it's time
- 3.7 sonnet was hilariously bad, on all pretty much all metrics
- 3.5 haiku is very underrated, would love to see a v4
- 2.5 flash is **almost** perfect, prob the best single pass model you can get. not great at multi-turn
- openai has the most selection, but no clear winners really (in small to mid size)

If you could see the way Max set us up to evaluate Bracky's performance when powered by different AI models, you would be way more bullish.

Setting up smart evals is so critical for performance. With models improving exponentially, it positions us to always be powering bracky with the best-in-class brain.

helping /bracky, playing /bracket reading /greatbooks