A quick overview of how @akshaan built Trending Topics on Farcaster. 

1. Casts are selected and preprocessed daily. 
2. Model categorizes thousands of casts with a prompt. 
3. Topics are post-processed, merged and winners are picked

It costs about $1/day and runs in < 1 hour.

Technowatermelon. Elder Millenial. Building Farcaster. 

nf.td/varun

Casts are fed into a model with a prompt to provide topics. A condensed version of our prompt is below. 

We do about 2 million input tokens a day processing casts, and it usually takes ~30 minutes to fetch all the related topics for each cast. 

Models are very good at figuring out subtle context. They can tell that a post is about basketball even with an oblique reference to a team.

Topics aren't very clean when they come out. For example, you'll get "Pacers NBA finals" and "Indiana Pacers" which should be the same topic. 

A post-processing step removes filler words, merges topics with stemming and prioritizes topics with high uniqueness (TF-IDF score).

This produces a very small set of unique topics.

Are trending casts easily callable via api?

Each day, a few thousand of the best performing casts are selected and pre-processed. 

All of the text content is turned into structured JSON that can be fed into the model as a prompt. 

Right now, we're classifying 1-5% of all casts.

Very cool. 

Appreciate you sharing this! Would love to see more of this behind the scenes content.

why daily instead of every few hours? because there’s still a human in the loop?

Good work - curiously - this seems like a very batch operation and you only get "trending" on a per day basis. Do we not want to be more granular to get more trending topics, e.g. I'm quite possibly sure that during the F1, and livecaster usage, 50 people casting about it could be a trend over the ~2 hours?

Why do they clean up the topics after being chosen by the model?

Pretty cool. Which model are you using?

Are you parallelizing the requests? Just < 1h for a few thousands still seems slow. This should take a few mins max

Ok banger. I appreciate the depth and transparency. What an efficient way to achieve, in my opinion, one of the best features added in the host of others

Throughout yesterday I expected to see eid in the trending section but I didn’t.. can you explain why please?

love this development, noticed it yesterday and I was like noiceee

Hey Varun, can you share the metrics for picking the "best performing casts" and how do you guys select casts for the trending tab?