Launching Trek, an open source web content extraction library built in Rust!

A core part of our work is to understand any link's content on the Internet. And that also means extracting metadata quickly so users can get context, e.g. in a feed.

We're building from @kepano's work on Defuddle and then some to do this.

Trek also compiles into WASM, enabling anyone to extract content data in a clean, decluttered way in your TS/JS project.

It leverages lol_html from Cloudflare to stream HTML in for content extraction instead of building the entire page as a normal scraper would and "trekking" the DOM. This means it's really fast and more importantly memory efficient.

Check out the playground here: https://officialunofficial.github.io/trek/

Docs:

making /obsidian • writing stephango.com

Launching Trek, an open source web content extraction library built in Rust!

A core part of our work is to understand any link's content on the Internet. And that also means extracting metadata quickly so users can get context, e.g. in a feed.

We're building from @kepano's work on Defuddle and then some to do this.

Trek also compiles into WASM, enabling anyone to extract content data in a clean, decluttered way in your TS/JS project.

It leverages lol_html from Cloudflare to stream HTML in for content extraction instead of building the entire page as a normal scraper would and "trekking" the DOM. This means it's really fast and more importantly memory efficient.

Check out the playground here: https://officialunofficial.github.io/trek/

Docs: https://github.com/officialunofficial/trek

It's in the playground but I moved too fast, broke things trying to release/announce this morning 😄 

Feel free to contribute/PR! Would be a good way for Opencast to fetch metadata without a backend.

This is sweet! Why not proxy the fetching on the playground to prevent CORS issues?

Playground is here, oops.

https://officialunofficial.github.io/trek/playground/

🔍 ensdata.net ✦ 𝙽𝙾𝚆: design engineer @family ✦ 𝙿𝚁𝙴𝚅: @rainbow 🌈 ✦ 𝙱𝚄𝙸𝙻𝙳𝙸𝙽𝙶: @vision

Oh sick yeah, Trek can power this. We’re also prioritizing in JSON Schema over OG/twitter: tags.

https://json-schema.org/

dang y’all are cooking with the open source libraries!! looks very cool