programming

Have any of you had experience running SMCI? If so would you do it again?

Nope.

Scrapping web and running big data workload ?

@percs Leader, Coder, Product person and building simple solutions for tough problems. Enjoy Summer, Board Games, Games. My opinions here.

Assuredly. In possession of 10TB at the moment.

I would use parquet to reduce storage. streaming data to queries would be interesting challenge.

best of luck !

what's the advantage of parquet? i'm already compressing with zfs and lz4.

OK. I'm familiar with column / row. Not really useful for my scenario. Thanks for the suggestion tho!

If I have ton of columns in table and when while reading if I read only few columns then Parquet is useful. It will retrieve from column format storage and will decompress only required "cell" ( row x column ). You can use LZ4 with Parquet to compress. Parquet is useful for fast scanning and compression both.

You might be unpacking full LZ4 to read the bytes, if so parquet can help. You can consider
 * Parquet
 * Avro 
Rather storing data in DB and scanning all table columns ( row ) unnecessarily.