🦒 pfp
🦒
@srijan.eth
IMPORTANT — this morning a bug was encountered on @noiceapp which had led to sending out ~20k unintended tips some older farcaster posts got tipped by mistake due to how we recently changed our event ingestion infra. around 750 users were affected with ~$650 worth tips sent out. why/how it happened: this got triggered after us internally migrating farcaster event ingestion infra from neynar's webhooks to their kafka based stream for reliability reasons. tl;dr: • farcaster emits reaction.deleted events when users hit storage limits • our system misinterpreted these as reaction.created due to missing prior context • that triggered real tips on older casts as our idempotency logic didn’t catch it what we’ve done: • fixed the event type parsing and strengthened idempotency checks • reimbursed all affected users with 2× USDC of the USD value in the tokens originally deducted • personally following up in DMs of affected users with a quick rundown issue is patched, reimbursements are complete and we’ve added safeguards to prevent this in the future. appreciate who helped debug this 🙏
22 replies
8 recasts
134 reactions
🌈 YON pfp
🌈 YON
@yonfrula
thank you for the transparency, glad it's fixed!
1 reply
1 recast
13 reactions
Kasra Rahjerdi pfp
Kasra Rahjerdi
@jc4p
this is a way of handling stuff like this that i haven’t seen anyone else do, i feel like most would sweep this under a rug, major props to you for helping nudge us all in the direction of legitimacy!
1 reply
0 recast
12 reactions
Veiva pfp
Veiva
@veiva
Mistakes happen, especially during infra changes, but the way this was handled (2× reimbursements, personal follow-ups) shows strong accountability
0 reply
0 recast
3 reactions
frederick pfp
frederick
@sgniwder
ty for the update!
0 reply
0 recast
1 reaction
MJC pfp
MJC
@mjc716
interesting insights into fc architecture that will help me on some things, ty
0 reply
0 recast
1 reaction
noice pfp
noice
@noicebot
https://app.noice.so/?castHash=0xb6bc6234a3eb518782fc552c52010b476846aecc
0 reply
0 recast
1 reaction
keccak256 pfp
keccak256
@keccak0256
Gg I was not affected ig
0 reply
0 recast
1 reaction
J'Mariano pfp
J'Mariano
@cryptfi-mariano
I am glad to hear that it's all fixed now
0 reply
0 recast
1 reaction
Varun Deshpande pfp
Varun Deshpande
@thebitsian
great work man. well handled.
0 reply
0 recast
1 reaction
chalinosghost pfp
chalinosghost
@chalinosghost.eth
ok banger Problem was publicly addressed and fixed to prevent future bugs. Nice work, love to see transparency in the projects i follow.
0 reply
0 recast
1 reaction
Yash Agarwal pfp
Yash Agarwal
@yashhsm
💪
0 reply
0 recast
0 reaction
lightmode pfp
lightmode
@lightmode
Luckily it was fixed.
0 reply
0 recast
0 reaction
Squirtle0x pfp
Squirtle0x
@squirtle0x.eth
Great postmortem, bugs happen but the response to a bug is more important
0 reply
0 recast
0 reaction
noga (is a no-go) pfp
noga (is a no-go)
@nogatenzin
well done post, good exaple of a founder addressing the issue & educating from it. would love a vid version of this someday
0 reply
0 recast
0 reaction
Rico 🎩 pfp
Rico 🎩
@ricoxy
Honesty is the best policy. Keep up the good work.
0 reply
0 recast
0 reaction
Emem pfp
Emem
@manempowered
We appreciate the transparency and accountability. Good thing it's all properly handled
0 reply
0 recast
0 reaction
Kate 🧾 pfp
Kate 🧾
@k8
noice transparency.
0 reply
0 recast
0 reaction
Vinay Débrou ⚙️ pfp
Vinay Débrou ⚙️
@vinaydebrou.eth
good you caught it fast and fixed it. kudos👏🏻
0 reply
0 recast
0 reaction
Emmy Walka pfp
Emmy Walka
@emmywalka
Respek
0 reply
0 recast
0 reaction