Google may expand the list of unsupported robots.txt rules in its documentation based on analysis of real-world robots.txt data collected through HTTP Archive.
Gary Illyes and Martin Splitt described the project on the latest episode of Search Off the Record. The work started after a community member submitted a pull request to Google’s robots.txt repository proposing two new tags be added to the unsupported list.
Illyes explained why the team broadened the scope beyond the two tags in the PR:
“We tried to not do things arbitrarily, but rather…

![[CITYPNG.COM]White Google Play PlayStore Logo – 1500×1500](https://startupnews.thebackend.in/wp-content/uploads/2025/08/CITYPNG.COMWhite-Google-Play-PlayStore-Logo-1500x1500-1-630x630.png)