Data

Just released the largest open dataset of Farcaster threads with embeddings!

📊 24.3M high-quality threads
🔍 512-dim Voyager embeddings (f32)
✨ Spam-filtered & engagement-ranked
📅 Complete Farcaster history to May 2025

Perfect for semantic search, clustering, recommendation systems & social analysis

🤗

Just released the largest open dataset of Farcaster threads with embeddings!

📊 24.3M high-quality threads
🔍 512-dim Voyager embeddings (f32)
✨ Spam-filtered & engagement-ranked
📅 Complete Farcaster history to May 2025

Perfect for semantic search, clustering, recommendation systems & social analysis

🤗 https://huggingface.co/datasets/shoni/farcaster

Blockchain data analyst. Interested in data science and blockchain.

Analysis-
 https://paragraph.xyz/@yesyes,

https://dune.com/yesyes

i dabble with ai using open identity data from crypto https://alexpaden.tech

couple notes:
v1 only includes 1 layer deep of quote recursion 
V1 does not include url/image/video descriptions in text

Designing and building AI systems

https://agents.gladio.ai

Thanks for sharing this breakdown! The world needs more step by steps from builders

Sure I see these as the most granular useful piece of data in analysis, so I’ll be shifting to the next building block of user based content

I get all fc data from Neynar parquet service then use a set of custom Postgres procedures/tables to format/query/generate embeddings (it’s a near live pipeline)

The embeddings are float32, I use pgvector which doesn’t support int8 or I would have done that. You can easily convert the float32 to float16 for storage/query savings.

I run everything on my Mac studio but the goal is to create advanced live apis that enable other builds/remove custom memory from the stack

The embeddings themselves probably $50, ai tools building the procedures was probably $200, Neynar parquet is about $300/mo and Mac studio $10k which runs the Postgres server/pipeline/etc.

The whole nonspam dataset is about 2B tokens

This is very cool! Would you mind sharing more about the process?

- potential applications in mind
- how you ingested the data
- various embeddings and other data mangling done
- cloud vs local usage 
- total cost to generate this

please use it if you can think of a reason and let me know if you have suggestions— I’ve already thought of a few improvements for my formatting technique before being embedded that I’ll test. 

Going to start testing clusters over the next few days and see what happens

Ty🙏

that’s really cool, i really wish i had time to play with this