@eirrann.eth

BIP 🧵 My MoE AI model was stuck in Hermes. I went model shopping and found an unexpected VRAM bottleneck: I'd been too cautious with it. The GPU must fit both the model and its working memory. My 24 GB RTX 3090 now runs a 27B model with 180k context.

1 reply

1 recast

2 reactions