We're starting to think about a new sync model for Farcaster. 

The current system works but is unlikely to scale up another 10x. Here's our articulation of the problem we want to go after.

https://storage.googleapis.com/gweb-research2023-media/pubtools/1974.pdf

https://github.com/cockroachdb/cockroach/blob/master/docs/design.md

There are some incredible things that Spanner and Cockroachdb does for globally distributed databases - the kv rocksdb design that the hubs do are similar to what cockroachdb does actually maybe can borrow more ideas from them. Incredible people over there.

Technowatermelon. Elder Millenial. Building Farcaster. 

nf.td/varun

🗽based; 

building skeet.build 🟠;
building @buoy 🛟; 
building check.supply 🏦;

spanner and cockroachdb rely on trusted components – spanner has massive time synchronization techniques to get intra-server time as precisely aligned as possible, which does nothing in the adversarial distributed consensus category. cockroachdb uses a significantly narrowed version of rocksdb (pebble) that they layer on top of. I've had a decent amount of experience now (unfortunately) dealing with pebble's shortcomings, to the point I had to vendor a fork of it just to get it to behave the way i needed for another decentralized protocol.

Cockroachdb folks use a slightly different technique that’s a bit more forward looking called Hybrid Logical Clock to get around spanner’s atomic clocks.

Agreed it’s a different problem, but when it comes to global replication I’d say there’s some learnings

BDFL of Quilibrium, Eng @ Farcaster, ex-Coinbase, always opinionated, never hydrated

/quilibrium

just getting back to the problem at hand - i thought you all were looking into partitioning the data into smaller segments so that you have more independent pieces. I vaguely recall you all playing around with this idea. then if hubs go out of sync smaller chunks can be updated and sync'ed more efficiently

100% – the distributed adversarial conditions that exist in trustless timestamping are a bit more challenging, but there are some strong approaches: obviously blockchain (w/r/t proof of work) was the first idea that solved it, but since then there's been approaches like incorporating verifiable delay functions to serve as a lamport clock timestamping basis. It's still not perfect, but definitely doable. Hubs have _less_ of a timestamping issue and more of a distributed consensus problem with CRDTs, in that the span of CRDTs being consumed in a given sync period are too large to reconcile across all parties in a timely basis. The "all parties" is the sticking point – because the moment any hub(s) become "preferred" it turns into a credible neutrality problem.

Cockroachdb folks use a slightly different technique that’s a bit more forward looking called Hybrid Logical Clock to get around spanner’s atomic clocks.

Agreed it’s a different problem, but when it comes to global replication I’d say there’s some learnings

 https://cse.buffalo.edu/tech-reports/2014-04.pdf