Robots.txt

Robots.txt is a simple text file that lives at the root of your website. It tells search engine crawlers which parts of your site they can and cannot request. On WooCommerce or Shopify, it helps you steer bots away from low-value URLs like cart and checkout pages. Still, it controls crawling, not indexing, so it is not a reliable way to hide a page.

---

Key Takeaways

• Crawl control, not index control: Robots.txt tells bots what to fetch. Yet a blocked URL can still appear in search results.
• It lives at the root: The file must sit at yoursite.com/robots.txt, and each subdomain needs its own copy.
• Common WooCommerce mistake: Blocking /wp-content/ can stop Google from rendering your pages the right way.
• Sitemap signpost: You can declare your sitemap inside robots.txt to help crawlers find your key pages faster.

---

Understanding Robots.txt

What Robots.txt Actually Does

Think of robots.txt like a polite sign on a shop door. It asks visitors to skip certain aisles, but it cannot lock anything. Well-behaved crawlers like Googlebot read it first and follow the rules. Bad bots, however, can ignore it completely.

The file uses simple directives. “User-agent” names the crawler, and “Disallow” lists paths you want skipped. An “Allow” line can then carve out an exception inside a blocked folder. You can also add a “Sitemap” line pointing to your XML sitemap.

Modern robots.txt also supports pattern matching. An asterisk acts as a wildcard for any run of characters. A dollar sign marks the end of a URL. Together, these symbols let you block tricky patterns without listing every single URL by hand.

Search engines cache the file to avoid fetching it constantly. In practice, Google generally caches robots.txt for up to 24 hours. It also enforces a file size limit of 500 kibibytes, and content past that is ignored. The rules follow an official standard, RFC 9309, which requires a parsing limit of at least 500 KiB.

Crawl Control Is Not Index Control

This is the caveat that trips up most store owners. Robots.txt stops a bot from reading a page, but it does not remove that page from the index. As a result, if other sites link to a blocked URL, Google can still list it. That can happen through normal internal linking or external backlinks.

You will often spot this problem in Google Search Console. It flags such pages with the status “Indexed, though blocked by robots.txt.” In other words, Google found the URL but could not read it. The fix is to allow the crawl, then apply a proper noindex.

To truly keep a page out of search, you need a different tool. A meta robots noindex tag or a canonical tag handles indexing directly. We break down that difference in the Versus section below.

How To Create And Edit It

Creating the file is refreshingly low-tech. You open a plain text editor and save it as robots.txt. Then you upload it to your site’s root folder. On WooCommerce, many SEO plugins let you edit it right from the dashboard.

After any edit, always validate the result. A single misplaced Disallow can hide key pages for weeks. So change the file slowly, and test each rule as you go. It is also smart to keep a backup of the last working version.

Common WooCommerce Robots.txt Mistakes

The biggest mistake is blocking /wp-content/. Google needs your CSS, images, and scripts to render pages correctly. Blocking that folder can hurt how Google sees your layout, which matters for JavaScript SEO.

Another slip is disallowing cart and checkout URLs carelessly. These pages add little SEO value, so many stores block them. However, be careful not to block resources those pages share with product pages. When in doubt, test any change before you deploy it live.

A third mistake is treating robots.txt as a security tool. The file is public, so anyone can read it at your domain. Listing private paths there simply advertises them to curious visitors. Instead, keep sensitive areas behind real authentication.

---

A Hypothetical E-commerce Example

The Setup

Imagine a mid-sized WooCommerce store called Fernway Home Goods. It sells around 2,000 products and runs on a busy shared server. In practice, the team notices Googlebot spending crawl time on faceted filter URLs. These are color and price sort links that spin up thousands of near-duplicate pages. That waste eats into the budget meant for real listings.

To fix this, they add a Disallow rule for the filter parameters in robots.txt. They also add a Sitemap line pointing to their product sitemap. As a result, crawlers soon spend more time on real product and category pages. Their crawl load starts flowing to pages that actually earn traffic.

First, though, they test the new rules before going live. Google Search Console offers a robots.txt report for exactly this. It flags syntax errors and shows which URLs a rule blocks. A quick check here prevents an accidental site-wide block.

The Crawl-Versus-Index Lesson

Here is where the crawl-versus-index lesson lands. One filtered URL had picked up a few external links. Even after the Disallow, that URL still showed up in search with a bare title. Because Google could not crawl it, it never saw a cleaner signal.

So Fernway takes a second step. First, they remove the Disallow rule from the file. Then they add a noindex tag to those filter pages. Once Google recrawls and reads the noindex, the pages finally drop from results.

The payoff is real and lasting. Fernway now spends its crawl budget where it counts. Meanwhile, the noindex quietly clears out the duplicate pages. In the end, one file change fixed crawling, and one tag fixed indexing.

---

Robots.txt Vs. Meta Robots / Noindex

These two tools look similar but do opposite jobs. Robots.txt blocks crawling at the door. By contrast, a meta robots noindex tag controls indexing after the page is read.

Here is the key trap to avoid. If you block a page in robots.txt, Google never sees a noindex tag on it. So the two rules can quietly cancel each other out. The safe order is always crawl first, then decide whether to index.

• Use robots.txt when: You want to save crawl budget or keep bots out of scripts and admin paths.
• Use noindex when: You need a page fully removed from search results, like thin or duplicate pages.
• Never combine them: Blocking crawl hides the noindex, so a page can linger in the index anyway.

---

The Pros And Cons

The Pros

• Saves crawl budget: It keeps bots off low-value URLs, so they focus on pages that drive revenue.
• Simple to set up: It is a plain text file, so most owners can edit it without a developer.
• Points to your sitemap: One extra line helps crawlers discover your important pages much faster.

The Cons

• No index control: A blocked page can still rank, which surprises many store owners.
• Easy to break: One stray slash can block your entire site from search engines.
• Public by design: Anyone can read it, so it is never a place for secrets.

---

Frequently Asked Questions

Where should the robots.txt file be located?

It must sit at the root of your domain, at yoursite.com/robots.txt. Each subdomain and protocol needs its own separate file. A file placed in a subfolder will simply be ignored by crawlers. So a staging subdomain, for example, needs its very own copy.

Does robots.txt remove a page from Google?

No, it only blocks crawling, not indexing. A blocked page can still appear in results if other pages link to it. To remove a page, use a noindex tag instead, and let Google crawl it first. This gap is the single most misunderstood part of the file.

Should I block AI crawlers in robots.txt?

You can, since many AI bots respect the file. In fact, some stores now pair it with an llms.txt file for AI guidance. Choose based on whether you want your content used by AI models. Still, not every bot obeys the rules, so treat it as a request.

---

The Bottom Line

Robots.txt is a small file with outsized influence over how search engines crawl your store. Treat it as a traffic director for bots, not a lock on your content. Most stores only need a short, careful file, reviewed after each redesign. Get the crawl-versus-index difference right, and you protect both your crawl budget and your rankings.