Robots.txt SEO Guide for Law Firm Websites

Arslan SEO Insights tells law firms that robots.txt is a small text file that tells search engine crawlers which parts of a website they are allowed to request, and getting it wrong can accidentally hide entire sections of a site, including practice area pages that should be bringing in cases.

It does not remove pages from Google's index by itself, and it cannot stop a page from appearing in search results if other sites link to it. Understanding that difference is the first step to using it correctly.

What Robots.txt Actually Does

Robots.txt lives at a predictable web address on every site, usually something like arslanseoinsights.com/robots.txt, and it is one of the first things a search engine crawler checks when it visits a site.

It contains simple rules telling crawlers which folders or pages they are allowed to crawl and which ones they should avoid.

That word, crawl, matters a lot. Robots.txt controls whether a crawler is allowed to fetch and read a page. It does not control whether that page can appear in Google's search results.

These are two different systems, and confusing them is the single most common mistake law firm website owners make with this file.

Here is why the distinction matters in practice.

If a page is blocked in robots.txt but other websites, or other pages on your own site, link to it, Google can still know the page exists through those links, and can still show it in search results, just without a description, since Google was never allowed to actually read the page's content to generate one.

This often confuses site owners who see a blocked page showing up in search results and assume the robots.txt rule failed. It did not fail. It did exactly what it was supposed to do, which is prevent crawling, not prevent appearing in search results altogether.

If you actually want to keep a page out of search results entirely, the correct tool is a noindex meta tag placed directly on that page, not a robots.txt block.

This is worth repeating because it trips up so many people: robots.txt blocks crawling, noindex blocks indexing, and they solve two different problems.

Why This File Matters More Than It Looks Like It Should

A law firm website, especially a personal injury or mass tort firm site with many practice area, case type, and city pages, depends on search engines being able to freely crawl all of that content.

A single misconfigured line in robots.txt can block an entire folder, meaning every page inside it becomes invisible to search engine crawlers in one shot.

This is not a hypothetical risk. It happens regularly, usually without anyone noticing until traffic to an entire section of the site quietly disappears over several weeks.

Because the failure is silent, meaning there is no error message, no broken page, nothing visibly wrong when you look at the site in a browser, robots.txt problems can go unnoticed for a long time.

A firm might spend months wondering why its truck accident page never seems to rank, never realizing the entire directory containing that page has been blocked from crawling since a site update six months earlier.

Common Mistakes Found on Law Firm Sites

A blanket Disallow: / left over from staging. When a website is being built or redesigned, developers commonly add a robots.txt rule that blocks the entire site from being crawled, to prevent an unfinished staging version of the site from accidentally getting indexed.

The problem happens when that same staging robots.txt file gets carried over to the live site during launch, and nobody remembers to remove it.

The result is a fully built, live law firm website that search engines are told, in effect, to stay away from entirely.

This is one of the most damaging and most common technical mistakes found on law firm sites, and it is also one of the easiest to fix once found, which is exactly why checking for it should be a standard first step in any technical review.

Blocking CSS and JavaScript files. Some older or overly cautious robots.txt configurations block folders containing a site's styling and script files, thinking these are unnecessary for crawlers to see.

In reality, modern search engines need to load these files to understand how a page actually looks and functions, particularly for mobile rendering.

Blocking them can prevent a search engine from properly understanding whether a page is mobile-friendly and how its content is actually laid out, which can hurt rankings even though the page's text content itself was never blocked.

Assuming robots.txt hides a page from search results. As covered above, a blocked page can still appear in search results, just without a proper title and description, if other pages link to it.

This becomes a real problem when a firm tries to use robots.txt to hide something sensitive, like an old case results page that references outcomes the firm no longer wants featured, or a page built for an internal testing purpose.

If the goal is to truly keep something out of search results, a noindex tag on the page itself is the correct tool, not a robots.txt disallow rule.

No sitemap reference. Robots.txt is a standard, easy place to point crawlers directly to a site's XML sitemap, which lists out every important page on the site in one place.

Leaving this out is not a catastrophic error, since search engines can usually still find a sitemap submitted through Search Console.

But it is a small, free signal that costs nothing to include and can help crawlers, especially newer or less familiar ones, find and prioritize a site's pages more efficiently.

This matters more on a larger law firm site with many practice area, case type, and location pages, where efficient crawling has a bigger impact.

Blocking parameter-based URLs too aggressively. Some site configurations generate URLs with tracking parameters, filters, or session identifiers attached, and an overly broad robots.txt rule meant to block these can accidentally catch real, important URLs that happen to share part of the same pattern.

This is a more advanced mistake but worth checking for on sites using certain page builders or e-commerce-style filtering on case type listings.

Conflicting rules for different crawlers. Robots.txt can specify different rules for different crawlers, like Googlebot versus Bingbot versus an AI crawler like GPTBot.

Sometimes a rule meant to apply narrowly to one crawler accidentally gets applied more broadly, or a general rule at the top of the file conflicts with a more specific rule further down in ways the site owner did not intend.

Understanding how these rules are read in order, with more specific rules generally taking priority, matters when a site has more than a simple, uniform rule set.

How Robots.txt Rules Are Actually Structured

A basic robots.txt file for a law firm site might look something like this in structure: a line specifying which crawler the following rules apply to, followed by one or more lines disallowing specific folders, and a line pointing to the sitemap.

A rule disallowing an admin or internal folder, like the WordPress dashboard folder on a site built with WordPress, is normal and expected, since there is no reason a search engine needs to crawl a site's login or admin area.

The mistake is when a rule meant to be narrow, like blocking just the admin folder, is written broadly enough to accidentally catch a real content folder, like a practice area or blog folder, especially if folder names share a common prefix.

A Real Example of What This Costs a Firm

Picture a personal injury firm that redesigned its website and moved to a new hosting platform. During the build, the development team worked on a staging URL with a robots.txt file set to Disallow: / so search engines would not accidentally index the unfinished site.

When the redesign went live, the same robots.txt file was copied over along with everything else, and that one line stayed in place.

For the next several months, the firm's rankings on every practice area page slowly dropped as Google, respecting the new instruction, stopped crawling the site and gradually removed previously indexed pages from its results as it periodically rechecked them and found them newly blocked.

The firm's marketing team noticed a decline in calls and assumed the market had simply gotten more competitive, or that a Google algorithm update had hurt them, since nothing on the visible site looked broken.

Nobody thought to check a hidden text file, because most people do not know it exists.

The fix, once found, took about five minutes: removing the single Disallow: / line and resubmitting the sitemap through Search Console. But the recovery was not instant.

It took several weeks for Google to recrawl the site and begin restoring indexed pages, and a few more weeks after that for rankings and traffic to climb back toward where they had been before the redesign.

This is a real pattern seen across law firm sites that go through redesigns or migrations without a technical check immediately afterward, and it shows why this one small file deserves a specific, deliberate check at exactly those moments.

Robots.txt and AI Crawlers

As AI tools like ChatGPT and Perplexity increasingly pull information from the web to answer questions, some site owners have started adding rules to their robots.txt file specifically for AI crawlers like GPTBot or PerplexityBot.

Whether to allow or block these crawlers is a legitimate business decision for a firm to make, since being cited by an AI answer engine can bring visibility and referral traffic, but it does mean giving that crawler access to your content.

There is no universally correct answer here, but it is worth making that decision deliberately rather than by accident, since a robots.txt rule left in from years ago may be blocking or allowing these newer crawlers without anyone having made an active choice about it.

How to Check What Your Robots.txt Is Actually Doing

The most direct way to see your current robots.txt file is to visit yourdomain.com/robots.txt in a browser, which shows you the exact, live rules currently in effect.

From there, Google Search Console includes a URL inspection tool that lets you test specific, real URLs from your site, like a specific practice area page, against the current robots.txt rules, showing you clearly whether that exact page is currently blocked or allowed.

This is a far more reliable check than just reading the file yourself and guessing how the rules apply, especially on a site with several different rules stacked together.

It is worth running this check any time a site goes through a redesign, a migration to a new hosting platform, or a platform change, since these are the exact moments when a leftover staging rule or an accidental blanket block tends to get introduced.

A quick check immediately after a site launch or migration can catch a problem within days instead of months.

What a Good Robots.txt Setup Looks Like for a Law Firm Site

A well-configured robots.txt file for a personal injury or mass tort firm site generally allows crawling of every practice area, case type, city, and blog page, since these are exactly the pages meant to attract search traffic.

It disallows crawling of genuinely internal areas, like an admin login folder or an internal search results page that generates duplicate, low-value URLs. It includes a clear reference to the site's XML sitemap.

And it does not attempt to use disallow rules as a substitute for noindex tags on pages the firm actually wants excluded from search results.

Robots.txt Versus Meta Robots Tags Versus the X-Robots-Tag Header

Law firm site owners often mix up three different tools that all sound similar but do different jobs. Robots.txt, as covered throughout this guide, controls crawling at the site level, meaning whether a search engine is allowed to fetch a page at all.

A meta robots tag is a small piece of code placed inside an individual page's own code that tells search engines whether that specific page should be indexed and whether links on it should be followed.

An X-Robots-Tag is a similar instruction, but delivered as part of the server's response rather than inside the page itself, which is mainly useful for non-HTML files like PDFs, such as an old case results PDF a firm might want kept out of search results.

The practical difference matters because these tools do not always work together the way people assume.

A page blocked in robots.txt cannot be crawled, which means a search engine cannot even see a noindex meta tag placed on that page, since it never gets to read the page's code in the first place.

This creates a real trap: a site owner who blocks a page in robots.txt and also adds a noindex tag to it, thinking that is extra safe, may actually cause the page to linger in search results longer, because Google cannot crawl the page to discover the noindex instruction that would have told it to fully drop the page.

If you truly want a page out of search results, the correct approach is to allow crawling and use a noindex tag, not to block crawling and hope the page disappears.

A Simple, Correct Example Structure

Without getting into code formatting that does not translate well outside a technical document, the logical structure of a solid law firm robots.txt file follows a simple pattern:

A general rule allowing all crawlers to access the entire site by default, a small number of specific disallow rules limited to genuinely internal areas like the admin login folder or internal search result pages, and a line pointing to the sitemap's full web address.

That is usually all a law firm site needs. Complexity in a robots.txt file is rarely a good sign. The simpler and more deliberate the file, the less likely it is to contain an accidental mistake blocking something important.

A Practical Checklist for Reviewing Your Firm's Robots.txt

A few minutes spent walking through these checks can catch most of the problems covered above:

• Visit yourdomain.com/robots.txt directly in a browser and read through every line. Does anything block an entire folder you recognize as containing real content, like /practice-areas/ or /blog/?
• Look for a line reading Disallow: / on its own, which blocks the entire site. This should almost never appear on a live site.
• Confirm there is a line pointing to your sitemap's full address.
• Use Search Console's URL inspection tool to test a handful of your most important pages, like your top three practice area pages, and confirm each one shows as allowed for crawling.
• Check the file again immediately after any site redesign, migration, or hosting change, since these are the most common moments for a staging rule to accidentally carry over.
• If you use a page builder or SEO plugin that auto-generates robots.txt rules, review what it produces rather than assuming the default settings are correct for your specific site structure.
• After robots.txt is confirmed clean, check for other crawlability issues that can block Google just as effectively even when robots.txt itself is fine.

Next Step

If you are not sure what your firm's robots.txt file is actually blocking, or whether it is quietly hiding pages that should be bringing in cases, get a free audit for a direct check.

You can also read more about our broader law firm SEO work or see how technical issues connect to real results in our case studies.