hey if you're wondering where the public data dump for April is, I need help figuring out the implications of my decisions with the Snapchain setup -- would love any ideas.

Short of it: Snapchain prefers a 24/7 running instance, can't afford to run a heavy box 24/7, stuck on filtering 90GB of data on 8GB of RAM

have you thought about Redis to store what you to access under a ttl renewed by any criteria?

@christopher has a great solution with this (where you can like, do extra filtering at the redis consumption layer) https://github.com/officialunofficial/waypoint -- my main issue rn stems from doing lifetime dumps instead of incremental dumps, if i had a queue of the last day's likes or something it would be a lot easier

future proof dev. joying life of bugs, phd on hot hot fix in prod

I’m might not getting the prob exactly, but getting deltas for what you exactly need to update helps the read writes.
Redis is very useful for instant memory, being also distributed

yeah basically the issue is one time i wrote code to pull everything at once and filter it and save it and now i want to re-run that code instead of writing smarter code now that i already have all the data up until a month ago saved online