this is pretty insane 

the tek is so gud now that data paradigms are being inverted 

you would never think storing something as video would be *more* efficient

this project stores millions of text chunks inside a video file (mp4)

then runs sub-second semantic search on it
- no vector DB, no servers
- uses 10x less RAM  &amp; storage
- no internet required

it's called Memvid and it just broke my brain https://t.co/Ic3eRCfynp

Ilias Ism

this is pretty insane 

the tek is so gud now that data paradigms are being inverted 

you would never think storing something as video would be *more* efficient 

https://x.com/illyism/status/1930888284877521249?s=46

app developer /spotlight | MediaFi & Attention Markets

They don't have to respond, its the entire point of information theory!

There's a fundamental limit to how much you can compress information. The entire concept of word vectors was to compress the space of all possible words into a vector space that's finite.

Its already lossy, so if you tell me you found and improvement on an already lossy compression, I already know your probabilistic data retrieval is more lossy than the original lossy compression.

https://en.m.wikipedia.org/wiki/Lossy_compression

would be curious to hear how they would respond to that!

tbh we're (me) all probably a little too credulous when we see this stuff on the feed

This doesn't smell right. The storage of text documents should be the most efficient because the data footprint of text files is basically binary already. If the image is RGB codes, then those are IEEE floating points across a matrix.

I'm not exactly saying this might not be faster in retrieval, but I am saying I'd like to see data on what exactly is the criteria on efficiency. Maybe you're gaining storage space in video codec compression, but it doesn't seem like this should work without loss purely on information theory.

The tradeoff for the efficiency *has* to be probabilistic retrieval.

it’s prob not the right choice for almost any use case yet but it’s cool that it can be credibly considered

Color me skeptical but this feels like a lot of unnecessary additional overhead compared to just using a vector database.