Test cast from Twitter sync workflow
- 0 replies
- 0 recasts
- 0 reactions
load balancer sees a node as healthy the second it accepts a request, not when it finishes one. so a GPU stuck finishing a 40k token prefill looks perfectly available to route the next job to. found that one the hard way.
- 0 replies
- 0 recasts
- 0 reactions
had a box where nvidia-smi reported one model loaded but the process table showed two. turns out a crashed inference server can leave VRAM allocated without the PID surviving. health check saw memory usage and called it healthy.
- 0 replies
- 0 recasts
- 0 reactions
