Python

If you’re into distributed data check out this article about Open AI’s setup. It’s really cool for global scale. I took a ton of notes. The Custom Kafka Flink Connector is pretty solid given Flink’s default connector assumes only one Kafka cluster. I’m super impressed. 

https://open.substack.com/pub/bytebytego/p/how-o

X-posting to Python channel because the article describes how Open-AI is investing in Python. 


Flink’s core is built in Java and Scala.
Most production jobs were originally written in those languages, but since Python has become the de facto language for data and ML, OpenAI is investing heavily in using PyFlink.

Python is leveraged for:

- Streaming data transformations and SQL-like queries, making it feel closer to working with a database for researchers.

- Advanced use cases like streaming joins (a huge pain I am familiar with) and asynchronous I/O, developed jointly by OpenAI and the Flink open-source community.

- Python wrappers over Java, bridging performance gaps and managing communication overhead since Flink’s core runs on the JVM.

Travel, History, Data & Distributed Systems

Crypto since 2016

55 countries

Polymath, Software Engineer & Nomad

Ex: Alchemy, Salesforce

Cellframe Token

Cast about coding, life in 🏴󠁧󠁢󠁷󠁬󠁳󠁿 South Wales 🏴󠁧󠁢󠁷󠁬󠁳󠁿 and general nonsense. Chair of the UK Python Association.

🤔 That’s a great question. The article focuses on OpenAI’s technical contributions, but I do hope they eventually use the $50 million fund (pledge for community support) to support community-based Python organizations.

I wonder how big a donation they've made to the PSF as a result.

My guess is it's exactly the same amount we've seen come into the UK Python Association.