Content
@
https://opensea.io/collection/dev-21
0 reply
0 recast
2 reactions
Varun Srinivasan
@v
Anyone here worked on an eng team that processed more than 100M background jobs /day? Starting to run into issues with our setup. Looking for ideas on what people typically turn to at this scale.
24 replies
17 recasts
122 reactions
Ethspresso ๐๐ต๐ฉ
@ethspresso.eth
Airflow + K8s on bare metal is serving us well. At smaller scale than this though, something like 100k+ daily jobs of varying size and runtime. We use this for everything from ML training jobs to data transformations and 3rd party integrations. As @jacek mentioned, streaming is a good idea, for certain use-cases.
1 reply
0 recast
1 reaction
Samuel ใ
@samuellhuber.eth
regarding Airflow in K8s at Scale here's Shopify doing it https://shopify.engineering/lessons-learned-apache-airflow-scale
0 reply
0 recast
1 reaction
