hey if you're wondering where the public data dump for April is, I need help figuring out the implications of my decisions with the Snapchain setup -- would love any ideas.

Short of it: Snapchain prefers a 24/7 running instance, can't afford to run a heavy box 24/7, stuck on filtering 90GB of data on 8GB of RAM

have you thought about Redis to store what you to access under a ttl renewed by any criteria?

@christopher has a great solution with this (where you can like, do extra filtering at the redis consumption layer) https://github.com/officialunofficial/waypoint -- my main issue rn stems from doing lifetime dumps instead of incremental dumps, if i had a queue of the last day's likes or something it would be a lot easier

future proof dev. joying life of bugs, phd on hot hot fix in prod

yeahhh that's very smart. i just... don't need persistent synchronized data.. i just want a monthly dump, but maybe that's the best move

That's how Waypoint and the Redis Streams work. We build the last day's worth of messages, then you create a specific consumer group (likes-consumer-group) and then provision N number of consumers to burn it down pub-sub style.

yeah basically the issue is one time i wrote code to pull everything at once and filter it and save it and now i want to re-run that code instead of writing smarter code now that i already have all the data up until a month ago saved online

I’m might not getting the prob exactly, but getting deltas for what you exactly need to update helps the read writes.
Redis is very useful for instant memory, being also distributed