Data

Just released the largest open dataset of Farcaster threads with embeddings!

📊 24.3M high-quality threads
🔍 512-dim Voyager embeddings (f32)
✨ Spam-filtered & engagement-ranked
📅 Complete Farcaster history to May 2025

Perfect for semantic search, clustering, recommendation systems & social analysis

🤗 https://huggingface.co/datasets/shoni/farcaster

yesterday I did LSH deduplication on sanitized thread data and surprisingly only 2-3% of threads fit into this bucket (images not considered)— I expected 10-20% gms though that doesn’t make much sense in retrospect

i dabble with ai using open identity data from crypto https://alexpaden.tech

https://mattilyra.github.io/2017/05/23/document-deduplication-with-lsh.html