Wonder if ChatGPT will be the last major model to be trained on the open web? robots.txt specifically disallowing crawling from LLMs unless getting paid for the data?

I'm curious what's the law around crawling sites that disregard robots.txt and post mirrors of content.

Founder, builder, crypto enthusiast
YC alum, Orange DAO
Taking time to explore before starting my next venture.

Wonder what’s the legal definition of indexing.

We’re going to find out. The LinkedIn case said scraping is ok assuming you want to be indexed