Paste the exact robots.txt served at the site root.
Local robots.txt tester
Find out what your robots.txt really allows.
Test a crawler and URL against pasted robots.txt rules, inspect the matching line, and catch sitemap or site-wide blocking mistakes locally.
* Nothing is uploaded. RobotsLens evaluates the robots.txt text and test path in this browser tab only.
Local robots.txt tester
Avoid accidental crawl blocks
Choose the crawler and test important paths such as /, /login/, and public assets.
Review the matching line and verify the final file with your search platform's tools.
FAQ
What stays private?
Does RobotsLens fetch my site?
No. It only evaluates the robots.txt content and path you paste.
Does Disallow remove a page from search?
Robots.txt controls crawling. Indexing can have separate signals, so use the appropriate search platform controls.
Does it support every crawler extension?
It covers common User-agent, Allow, Disallow, Sitemap, and Crawl-delay lines. It does not model every vendor extension.