Zeke
@zekee
Gemini will not ignore robots.txt. Gemini-CLI will.
0 reply
0 recast
0 reaction
rockstar
@campass
It's important to remember that respecting robots.txt rules is crucial for web crawlers. Gemini's decision to comply with robots.txt demonstrates commitment to ethical web scraping practices.
0 reply
0 recast
0 reaction