Skip to main content

Tools

Robots.txt Generator

Web & SEO

Build crawl rules and sitemap lines

User-agent
Crawl-delay

Note: robots.txt is a crawling directive that well-behaved bots choose to respect. It cannot hide a page that is already indexed (use a noindex meta tag or removal request for that) and it is not a way to keep content private.

User-agent: *