Robots.txt SEO Guide

# Robots.txt SEO Guide

Robots.txt is a small file with the power to accidentally block an entire site from search engines. It deserves more attention than most sites give it.

## What Robots.txt Actually Does

Robots.txt tells crawlers which parts of a site they are allowed to request. It does not remove pages from Google’s index by itself, and it does not stop a page from being indexed if other pages link to it, it only tells crawlers not to fetch it. That distinction trips up a lot of site owners who assume blocking a page in robots.txt is the same as deindexing it.

## Common Mistakes

– **A blanket Disallow: / left over from staging.** Sites moved from a staging environment sometimes launch with the staging robots.txt still blocking the entire site.
– **Blocking CSS and JavaScript files.** Blocking the resources a page needs to render can prevent Google from properly understanding the page’s layout and content.
– **Assuming robots.txt hides a page from search results.** A blocked page can still appear in search results without a description if other sites link to it. Use a noindex tag instead if the goal is actually keeping a page out of search results.
– **No sitemap reference.** Robots.txt is a common place to point crawlers to the XML sitemap, and leaving that out is a missed, easy signal.

## How to Check It

Search Console’s robots.txt tester shows exactly what is being blocked and lets you test specific URLs against the current rules before making changes.

## Next Step

If you are not sure what your robots.txt file is actually blocking, [get in touch](/technical-seo-services/) for a direct check.