devops

any data engineering/infrastructure folks here? i want to use Terraform for data pipelining but i have only used Google Cloud Composer for IaC

hey y’all! seeking quick feedback below ⬇️

cloud engineer by day, blockchain dev by night || Developer DAO member

Got it, Terraform is great for that, I’ve created a few example repos that way myself. Also look at Pulumi, it’s similar to TF but focuses on providing libraries for other languages (JS, Go, etc) instead of having a dedicated language like TF. TF is more widely used but Pulumi can be less friction for devs to use.

cool! yeah, just making sure it's worth investing more focus into. GCP does so much on its own stack. and mostly open source contributions and a common IaC template so that others can replicate the work. i want to publish

I’m a devops engineer who works closely with a data engineering team and know they have a few workflows set up in TF. Definitely possible, it just depends on what specifically you’re wanting to build and what benefits you’re hoping to gain from TF

Yes! I'm available and willing to chat / pair. Hopefully I can help out a little 🙏

yep! will DM and setup time next week if you’re interest in learning more?

Terraform is pretty good! I think the cross cloud functionality is a bit overrated, but otherwise, works well when you set up smaller state files.

Is there something specific, re: Terraform and data pipelining, that you are curious about?