Data

Okay, I’m going to backtrack a bit while I repopulate the casts table in my DB using the updated Neynar v3 pipeline.

About a month ago, I said I’d release an embedding dataset for all non-spam Farcaster users. Midway through, I decided I wanted richer embeddings—URL/image/video inclusion.

That was the wrong mindset. I should’ve just released the open-source dataset of thread summaries.

The v2 Neynar casts table pipeline has been down for two days, but the old tables still exist, and the thread pipeline is ready (tables, procedures, queries, code). I’ll run it on text-only to generate the ~45M embeddings needed for release.

In hindsight, not doing this sooner was a clear mistake from a startup release perspective. The data, queries, and procedures were already built—I just overengineered and shipped nothing.

Even now, this is likely one of the most detailed thread embedding datasets, and certainly the most advanced open-source one.

Sorry for the delay.

Regarding embeddings, I had a minor issue with threads to include quotes in the text, I resolved it yesterday and reran on already processed rows. I think the open-source embedding dataset v1 (creative commons attribution 4.0 international (CC BY 4.0)) will be available tomorrow for Hugging Face upload. 

Generating the embeddings themselves today which is the final step and should only take a few hours in batch parallel

i dabble with ai using open identity data from crypto https://alexpaden.tech

Jobs not done? Thought you resolved it and reran it