DNS
Resolvers can return different cached records depending on geography, TTL and propagation state.
Network & Web Tools
Pro diagnostic reportRobots.txt Tester Online Free
Test common robots.txt Allow and Disallow rules against a crawler and URL path without contacting a website.
Diagnose
Run a bounded public check.
Understand
See a health score and explanation.
Act
Follow prioritized recommendations.
Test a robots.txt rule locally
Paste robots.txt content, choose a crawler and test a URL path. No website is contacted.
Test result
Allowed
- Path
- /admin/help/article
- Matching rule
- allow: /admin/help/
- Parsed groups
- 1
- Matching rules
- 2
This is a practical tester for common Allow and Disallow patterns. Search engines can apply additional interpretation rules, so verify important changes in the relevant search engine console.
About this tool
This local tester helps website owners and developers review crawler directives before publishing them. It parses user-agent groups, compares matching path rules and reports the winning directive. It also explains an important SEO distinction: a valid robots.txt file can limit crawling, but it does not guarantee removal from search results and must never be used as a security mechanism.
Common use cases
- ✓ Check a planned robots.txt update before deployment.
- ✓ Troubleshoot why a crawler may avoid a URL path.
- ✓ Compare a named crawler group with the wildcard group.
- ✓ Review whether sitemap and crawl directives support the same indexing plan.
How to use it
- Paste the complete robots.txt content.
- Enter the crawler user-agent name you want to simulate.
- Enter a full URL or path from the same site.
- Review the winning rule, then verify the live file with the relevant search-engine tools.
Important interpretation notes
- • robots.txt controls compliant crawler access; it is not a password, firewall or authorization system.
- • Disallowing a URL does not guarantee that the URL disappears from an index.
- • Blocking CSS or JavaScript required for rendering can prevent search engines from understanding a page correctly.
- • Rules are evaluated by path specificity, and crawler support for non-standard directives can vary.
Professional interpretation guide
Understand and verify your robots.txt tester results
Use the result as a focused diagnostic signal. Verify important findings before changing DNS, hosting, security, caching or application configuration.
What the result tells you
- ✓ Matching rules indicate whether the supplied path is crawlable for the selected user agent.
- ✓ Specific user-agent groups can override wildcard guidance.
- ✓ The result does not determine indexing eligibility or protect content from access.
How to verify it
- Test important paths with the intended crawler name.
- Compare with the live robots.txt file.
- Use the relevant search-engine inspection tools before major changes.
Network diagnostics
Interpret results in the correct protocol layer
Network tools observe different layers: DNS resolves names, TCP establishes connections, TLS authenticates encrypted sessions and HTTP carries requests, redirects, headers and caching rules. A successful result at one layer does not prove that the complete website or application is healthy.
TLS
Certificate validity, hostname matching, trust chains and protocol support are separate checks.
HTTP
Status codes, redirect chains, headers and caching directives describe application behaviour.
Measurement context
Latency and reachability vary by location, network path, browser and time.
What the operation can change
- A cached DNS answer may differ from an authoritative nameserver response.
- A 200 status code can still return the wrong content or an application error page.
- A valid certificate does not prove the website itself is trustworthy.
- Security-header tools report configuration, not complete vulnerability coverage.
Domain-specific verification
- Repeat tests from another network or resolver when location matters.
- Check the final URL and every redirect hop.
- Confirm timestamps, certificate names and expiry dates.
- Use browser developer tools and server logs for production incidents.
Common mistakes to avoid
- • Using robots.txt to protect confidential content.
- • Blocking CSS or JavaScript needed for rendering.
- • Confusing crawl control with removal from an index.
Privacy and safe use
Text analysis is performed in the browser. When fetching a live file, the public URL is requested.
Limitations
- • Crawler implementations and support for non-standard directives can differ.
- • A robots.txt rule is a crawl directive, not an indexing directive or access-control mechanism.
Continue the diagnostic workflow
Combine related checks instead of relying on one isolated result.
Related Network & Web Tools
Frequently asked questions
What is robots.txt?
robots.txt is a public text file at a site's root that gives crawl instructions to search-engine and other automated user agents.
Does robots.txt prevent indexing?
Not reliably. It controls crawling, not indexing. A blocked URL can still appear in search results when search engines discover it through links or other sources.
Does robots.txt protect private pages?
No. The file is public and compliant crawlers may follow it, but it is not authentication or access control. Protect confidential content with server-side authorization.
What happens when Allow and Disallow both match?
For major search crawlers, the most specific matching path generally wins. When matching rules have equal length, Allow is commonly preferred.
Can crawler implementations differ?
Yes. Major search engines broadly follow the Robots Exclusion Protocol, but unsupported extensions and edge cases can be interpreted differently.
Should blocked URLs be included in a sitemap?
Usually no. A sitemap should primarily list canonical URLs that you want search engines to crawl and consider for indexing.