Sean pfp
Sean

@seangeng

load balancer sees a node as healthy the second it accepts a request, not when it finishes one. so a GPU stuck finishing a 40k token prefill looks perfectly available to route the next job to. found that one the hard way.
0 reply
0 recast
0 reaction