v on Farcaster

Varun Srinivasan pfp

Varun Srinivasan

Here's a deep dive into what caused this very painful outage. Tap into the thread for details. When you link a token to a cast like $DEGEN, it creates an embed. Our server hydrates this embed with data like the image, token creator and stores it in our database so we can show it to you quickly. When we launched token news, we added a lot of info to the token object like the casts that were included in the news. We didn't realize that our hydrator was including all the news and news casts into this object every time we created a token link. We recently ended up in a recursive state because some of the casts in the token news had token links themselves. Now a cast with token link would trigger our server to include the news object, which contained casts, some of which contained the same token link and so on..... These casts would quickly balloon in size to 5 to 10 MB each. Our feed generator tries to fetch all the casts you want to read, order them and then compress them and put them into redis. This compression made the CPU on the feed workers stall. It got very bad very fast where as soon as a worker would come up, it would start picking jobs off the queue and stall immediately. We couldn't even SSH into the box to figure out what was going on. The way we ended up handling it is shutting off various parts of the feed generation until we could figure out a few problematic lines of code. We also scaled back the processor and forced it to run very slowly so we could get into the boxes and profile things, and both of these threads eventually led to us finding the culprits and fixing them.

31 replies

33 recasts

294 reactions

Varun Srinivasan pfp

Varun Srinivasan

Just kidding, I forgot I didn't need to thread.

3 replies

1 recast

43 reactions

pugson (Farcaster Professional) pfp

pugson (Farcaster Professional)

that’s crazy. only redis was affected or did any of your databases bubble up with recursive data as well?

1 reply

0 recast

3 reactions

derek pfp

just curious: are the interfaces GQL? in my experience very easy to have a recursion issue

1 reply

0 recast

0 reaction

Nabil Abdellaoui pfp

Nabil Abdellaoui

Checks all the boxes of nasty bugs: - Can't see it in staging - Needs very specific data to reproduce - Doesn't really error, silently bubbles up - Explodes everything Special bonus if it triggered a pager duty while the dev on-call is drunk.

0 reply

0 recast

8 reactions

Blinky Stitt pfp

@flashprofits.eth

I'm a big fan of using cpuset so that apps never run on core 0. That way ssh always has some room.

0 reply

0 recast

6 reactions

bertwurst pfp

1 reply

0 recast

11 reactions

Hi, I’m Ashley pfp

Hi, I’m Ashley

♾️♾️♾️♾️♾️♾️♾️ sheesh! Recursive and deadlock bugs are two of the most difficult bugs to detect true on-call nightmare. If you’ve got a huge codebase, I feel bad for whoever was on call 😵‍💫

0 reply

0 recast

1 reaction

Aaron Ho φ pfp

add circuit breakers!

0 reply

0 recast

0 reaction

Rakshita Philip pfp

Rakshita Philip

i don’t understand any of it, but i’m happy for you

1 reply

0 recast

7 reactions

Colin Charles pfp

Thank you for writing this blameless incident report!

0 reply

0 recast

2 reactions

MajorTom327 pfp

@majortom327.eth

Interesting! Good postmortem! Yeah, that would have been hard to debug. I don't know if you could potentially debug locally. If so, using bisect would have been a good tool to find which lines are having issues! But handling this kind of time is hard as everybody is running to fix it fast. It's easy to say what would have been easy to do after the issue was solved. Good job!

0 reply

0 recast

0 reaction

dCommunity - Home of AvenueD pfp

dCommunity - Home of AvenueD

Nice job to the team 👊 That is not ordinary and is made worse with the public facing nature. Glad you guys withstood the pressure 😌

0 reply

0 recast

0 reaction

TobyJaguar pfp

0 reply

0 recast

0 reaction

ohm. pfp

Somebody said, "add circuit breakers" 😭😭😭😂😂😂

0 reply

0 recast

0 reaction

Babooun pfp

Great communication, love the transparency ! Do you use real data for testing ? Like having recorded 24 hours of real prod events you dynamically replay on staging to test new release.

0 reply

0 recast

0 reaction

InsideTheSim 🎩🍪 pfp

InsideTheSim 🎩🍪

@insidethesim.eth

Oooh, that’s insidious. Good job pulling that apart and finding the source.

0 reply

0 recast

0 reaction

UNIBROS 🔥 pfp

anyway we can get the cached token images refreshed? Mine has been showing the wrong image. Can’t get in touch w/ anyone on the team

0 reply

0 recast

0 reaction

sandymariposa pfp

R we supposed to be jet lagged now?

0 reply

0 recast

0 reaction

Biggydaddy.eth pfp

I am just here to read, everything seems gibberish to me 😩😩

0 reply

0 recast

0 reaction