FREE TOOL
Robots.txt Generator
Visually configure search engine crawler rules 鈥?generate standard robots.txt files in one click
Visually configure search engine crawler rules 鈥?generate standard robots.txt files in one click
Robots.txt is a text file placed in the root directory of a website that tells search engine crawlers which pages they can and cannot crawl. It's the fundamental way websites communicate with search engines and an essential part of SEO optimization. Properly configured robots.txt helps search engines crawl your site more efficiently.
New site launch: quickly generate a standard robots.txt file. SEO optimization: control crawl scope to concentrate ranking weight. Admin protection: block crawlers from admin and sensitive pages. Bandwidth savings: block crawling of images, CSS, JS and other static resources. Multi-site management: quickly generate configs for different websites.
Select the crawler (User-agent) to configure 鈥?default is all crawlers. Add Disallow (block) or Allow (permit) rules and enter paths. Optionally configure crawl delay and Sitemap URL. The robots.txt content is generated in real-time on the right. Click "Copy" or "Download" to get the file and upload it to your website root directory.
Robots.txt must be placed in the root directory of your website with the lowercase filename robots.txt. For example: https://www.example.com/robots.txt. Search engine crawlers automatically visit this address to get crawl rules.
Disallow tells crawlers not to crawl specified paths. Allow tells crawlers they can crawl specified paths. Allow takes priority over Disallow, often used for "block entire directory but allow one file" scenarios. E.g., Disallow: /admin/ but Allow: /admin/login.php.
No. Robots.txt tells crawlers not to crawl, but if other pages link to those pages, search engines may still index their URLs. To completely prevent indexing, use the noindex meta tag or X-Robots-Tag response header instead.
Crawl-delay tells crawlers how many seconds to wait between each fetch, reducing server load. However, Googlebot does not support this directive 鈥?Google crawl speed must be set in Search Console. Other crawlers like Bing and Yandex do support it.
Yes. * matches any sequence of characters, $ matches the end of a URL. For example, Disallow: /*.pdf$ blocks all PDF files, Disallow: /?* blocks URLs with query parameters. Note that not all crawlers fully support wildcards.