robotstxtr

Project information

Project ID: 85027340

Engine-aware robots.txt policy for R with Google's matcher algorithm.

R rstats r-package robots-txt robots-exclu... rfc9309 crawler web-crawling googlebot seo technical-seo

301 Commits
1 Branch
4 Tags

README
MIT License
CHANGELOG
CONTRIBUTING
GitLab Pages

Created on

August 01, 2026