The robots.txt file is one of the oldest and most fundamental tools in a webmaster's toolkit, yet it remains one of the most frequently misconfigured. A single misplaced directive can block Googlebot from crawling your entire site. As part of our technical SEO services, we audit every client's robots.txt file before moving to any other optimization -- because if crawlers cannot reach your pages, nothing else matters.
This guide covers everything you need to know about robots.txt: the syntax, the directives, the common mistakes, and the CMS-specific configurations that catch most site owners off guard.
What Is Robots.txt?
Robots.txt is a plain text file located at the root of your domain (https://www.example.com/robots.txt) that communicates crawling instructions to web robots (bots). It follows the Robots Exclusion Protocol, a standard that most major search engine bots (Googlebot, Bingbot, etc.) respect.
Critical distinction: Robots.txt controls crawling, not indexing. A Disallow rule tells bots not to crawl a URL, but it does not prevent that URL from appearing in search results if other signals (like external links) point to it. To prevent indexing, you need a noindex directive.
Robots.txt Syntax
A robots.txt file consists of one or more groups, each targeting a specific user-agent. Here is the basic structure:
User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /admin/public/
Sitemap: https://www.example.com/sitemap.xml
Core Directives
User-agent: Specifies which bot the following rules apply to. Use * to target all bots, or specify a bot by name.
User-agent: * # All bots
User-agent: Googlebot # Google's crawler only
User-agent: Bingbot # Bing's crawler only
Disallow: Tells the specified bot not to crawl URLs matching the given path.
Disallow: /private/ # Blocks /private/ and all URLs beneath it
Disallow: /search # Blocks /search, /search-results, /searching, etc.
Disallow: /page?sort= # Blocks URLs with the sort parameter
Allow: Overrides a Disallow for a more specific path. This is particularly useful within a blocked directory.
Disallow: /resources/
Allow: /resources/guides/ # This subdirectory is crawlable
Sitemap: Declares the location of your XML sitemap. This can appear anywhere in the file, but convention places it at the bottom.
Sitemap: https://www.example.com/sitemap.xml
Sitemap: https://www.example.com/sitemap-news.xml
Pattern Matching
Google and Bing support two pattern-matching characters:
*(asterisk): Matches any sequence of characters.Disallow: /*.pdf$ # Blocks all URLs ending in .pdf Disallow: /*/feed/ # Blocks feed URLs in any directory$(dollar sign): Anchors the pattern to the end of the URL.Disallow: /category$ # Blocks /category but allows /category/seo/
Disallow vs. Noindex: Understanding the Difference
This is the single most common source of confusion in robots.txt management.
| Disallow (robots.txt) | Noindex (meta tag or HTTP header) | |
|---|---|---|
| Function | Prevents crawling | Prevents indexing |
| Bot can access page? | No | Yes (it must crawl the page to see the noindex directive) |
| Page can appear in search results? | Possibly, if external links point to it | No |
| Link equity passes? | No (bot cannot discover outbound links) | Depends on accompanying nofollow directive |
The trap: If you Disallow a URL in robots.txt AND add a noindex tag to the page's HTML, Google may never see the noindex tag because it cannot crawl the page. The URL could still appear in search results (as a "URL only" listing). To reliably deindex a page, it must be crawlable so the bot can read the noindex directive.
Best practice: Use robots.txt to manage crawl budget by blocking URLs you do not want crawled (internal search results, faceted navigation, admin areas). Use noindex to prevent specific pages from appearing in search results.
Common Robots.txt Mistakes
1. Blocking CSS and JavaScript Files
Google needs to render your pages to evaluate content and layout. If your robots.txt blocks CSS or JS files, Google cannot render the page, which can hurt your rankings and trigger mobile usability errors.
# BAD -- blocks rendering resources
Disallow: /wp-content/themes/
Disallow: /wp-includes/
# GOOD -- allow all rendering resources
Allow: /wp-content/themes/*.css
Allow: /wp-content/themes/*.js
Allow: /wp-includes/*.js
2. Leaving Staging Rules in Production
A staging site typically blocks all crawlers:
User-agent: *
Disallow: /
If this rule makes it to production during deployment, your entire site becomes invisible to search engines. Always verify robots.txt as part of your launch checklist.
3. Blocking Entire Directories Unintentionally
# Intended to block /search/ (internal search)
Disallow: /search
# Actually blocks /search, /search-results, /search-engine-optimization, etc.
Add a trailing slash to block only the directory:
Disallow: /search/
4. Conflicting Rules Across User-Agent Groups
If you have both a wildcard group and a Googlebot-specific group, Google uses only the most specific group. It does not merge them.
User-agent: *
Disallow: /private/
User-agent: Googlebot
Allow: /
In this case, Googlebot follows only the Googlebot group, which allows everything -- including /private/. The wildcard group is ignored entirely for Googlebot.
CMS-Specific Robots.txt Examples
WordPress
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-login.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?s=
Disallow: /*?p=
Disallow: /tag/
Sitemap: https://www.example.com/sitemap_index.xml
Shopify
Shopify auto-generates a robots.txt that blocks checkout, cart, and admin paths. Since 2021, you can customize it through a robots.txt.liquid template in your theme.
Static Sites / Custom Applications
User-agent: *
Disallow: /api/
Disallow: /internal/
Disallow: /staging/
Sitemap: https://www.example.com/sitemap.xml
Testing Your Robots.txt
Before deploying changes, always test:
- Google Search Console Robots.txt Tester (under Settings > Crawling > robots.txt): Enter a URL to see if it is blocked or allowed by your current rules.
- URL Inspection Tool: Shows whether Google could crawl the URL and how it was rendered.
- Third-party tools: Screaming Frog, Merkle's robots.txt tester, and Yoast's online robots.txt validator.
Test both positive (URLs that should be crawlable) and negative (URLs that should be blocked) cases.
Managing Crawl Budget with Robots.txt
For large sites with hundreds of thousands of URLs, robots.txt is a blunt but effective tool for crawl budget optimization. Common candidates for blocking:
- Internal search result pages.
- Faceted navigation with parameter combinations.
- Print-friendly page versions.
- Calendar views generating infinite URL spaces.
- User-generated content areas with low value.
Pair robots.txt blocking with proper internal link management -- do not link to URLs you have blocked, as this wastes crawl equity on links that lead nowhere.
FAQ
What happens if I do not have a robots.txt file?
Search engines will crawl your entire site without restrictions. This is fine for small sites with no pages that need to be blocked. However, having a robots.txt -- even a minimal one with just a Sitemap directive -- is considered best practice.
Can I use robots.txt to hide sensitive content?
No. Robots.txt is publicly accessible -- anyone can view it at yourdomain.com/robots.txt. Do not use it to hide confidential URLs. Use server-side authentication (password protection, IP restrictions) for sensitive content. Using Disallow actually advertises the existence of those paths.
Does robots.txt affect Google Ads or social media crawlers?
Google Ads uses a separate bot (AdsBot-Google) that does not always respect robots.txt rules targeting User-agent: *. You need a specific User-agent: AdsBot-Google rule to control it. Social media crawlers (Facebook's crawler, Twitter's bot) also use their own user agents and may ignore generic rules.
How quickly do robots.txt changes take effect?
Google caches your robots.txt and refreshes it approximately once per day, though the frequency varies. After making changes, you can request a recrawl via Search Console. Expect the change to be reflected within 24-48 hours for most sites.
Make Sure Your Robots.txt Is Working for You
A misconfigured robots.txt file can silently undermine your entire SEO effort. If you are not confident in your current configuration, Order an SEO Audit and we will review your robots.txt alongside your full technical SEO setup.