Robots.txt for Hospital Sites: What to Block, What Never to Block and AI Crawler Rules

This post is for the marketing lead or web manager at a hospital, clinic group or health tech company who has been asked two questions in the same week: "is robots.txt blocking anything it shouldn't?" and "should we block ChatGPT and Gemini?" It gives a working file structure, a list of what to block and what never to block, and a decision table for AI crawlers.

Hospital sites are unusually prone to robots.txt mistakes. They carry faceted doctor directories, appointment flows, patient portals, staging copies and old microsites, and the file is often written once by a developer and never revisited. A single misplaced line can remove a whole department from search. A missing one can let a crawler spend its visits on thousands of filter combinations.

One point first, because it changes how you use everything else here. Robots.txt controls crawling, not access and not indexing. Google says plainly that a robots.txt file is not a mechanism for keeping a page out of Google. Anything genuinely private needs authentication.

What robots.txt can and cannot do

The format is standardised in RFC 9309 and Google documents how it interprets the file. Four things matter for a hospital site.

• It is advisory. Well-behaved crawlers follow it. Scrapers and attackers do not. Never list a sensitive path in it as though that protects the path, because the file is public and lists exactly where to look.
• It blocks fetching, not indexing. A blocked URL that is linked from elsewhere can still appear in results as a bare URL with no description. If you want a page out of the index, allow crawling and use a robots meta tag or X-Robots-Tag header with noindex. Google's page on blocking indexing explains why the page must be crawlable for the noindex to be seen.
• It is per host. The file at the root of one hostname does not apply to a subdomain. A hospital with a separate booking subdomain needs a separate file there.
• Google reads only certain fields. User-agent, allow, disallow and sitemap are the ones that matter. Do not rely on crawl-delay for Googlebot.

The file also has a size cap. Google enforces a limit of 500 KiB and ignores content after it, which is rarely a problem unless a file has been generated from a long list of paths.

What to block on a hospital site

The goal is to spare crawlers pages that have no value in search and that multiply without limit. Typical candidates:

• Faceted and parameter URLs. A doctor directory filtered by city, specialty, language, gender and availability can produce thousands of near-identical URLs. A pattern such as Disallow: /*?gender= or Disallow: /*&sort= keeps crawlers out of the combinations. Block specific parameters, not every query string, or you will block legitimate pages like pagination.
• Internal search results. Disallow: /search/ or the equivalent for your CMS.
• Cart, checkout and account flows, if you sell health checkups or products online.
• Booking step URLs that exist only to hold a session, such as intermediate steps of a multi-page booking form.
• Staging, preview and test paths, though a staging site should be behind a login rather than relying on robots.txt.
• Print versions and duplicate renders where the canonical page already exists.
• Thank-you and confirmation pages, though noindex is the better tool if they might be linked.

Test each pattern before shipping. Google's robots.txt guide explains the wildcard and end-of-URL characters, and Search Console shows the file it last fetched. Wildcards are easy to over-match: Disallow: /doctors also blocks /doctors-near-me and /doctors-day-health-camp because matching is by prefix.

What never to block

These are the lines that cause the most damage when they appear in a hospital's file.

• CSS and JavaScript. Google needs these to render pages and judge layout and mobile usability. Its JavaScript SEO basics explain that blocked resources can stop rendering from matching what users see. Older advice to block /wp-includes/ or /assets/ should be removed from any inherited file.
• Images and video on pages you want to appear in image and video results.
• Your XML sitemaps. The sitemap location should be listed with a Sitemap line, and the sitemap URL must not itself be disallowed.
• Entire language or location folders during a migration. A Disallow: / left over from a staging launch is the classic failure and takes a site out of crawling within days.
• Pages you want noindexed. Block and noindex together defeat each other.
• Doctor, department and treatment pages, even those that feel thin. Improve or merge them instead.

A launch-day check is cheap: open the live file at the root, confirm it is not the staging version, and fetch two key URLs in Search Console's URL Inspection to confirm they are allowed.

Patient portals and security

Patient portals, report downloads and teleconsultation rooms often sit on paths like /portal/ or a subdomain. Blocking them in robots.txt is reasonable to stop crawlers wasting visits, but it is not security. Authentication, session handling and access controls are what protect the data. If you operate in markets where health-data rules apply, treat the file as hygiene and not as a control. Our article on secure healthcare site design goes through where real protections belong.

Also avoid putting patient-related paths in the file in a way that reveals structure. A line like Disallow: /reports/patient-id/ tells the world where reports live. Use opaque paths and rely on authentication.

AI crawler rules

The question now comes up in every board meeting. The honest answer is that there are three different purposes, and vendors publish separate tokens for them.

The vendor documents are the source of truth, and they change, so check them before editing:

• OpenAI's crawler overview describes OAI-SearchBot and GPTBot as independent settings, so a site can allow the search bot and disallow the training bot. It recommends allowing OAI-SearchBot if you want to appear in ChatGPT search results.
• Google states that Google-Extended is a robots.txt token controlling whether content crawled by Google may be used for training Gemini models and for grounding in Gemini Apps and Vertex AI. It says Google-Extended does not affect inclusion in Google Search or act as a ranking signal. Blocking it does not take you out of search.
• Perplexity's crawler page says PerplexityBot is designed to surface and link websites in search results and is not used to crawl content for foundation models. It also says Perplexity-User generally ignores robots.txt because a user requested the fetch.

For a hospital, the useful distinction is between visibility and training. Patients increasingly ask assistants for recommendations, and being retrievable by search-oriented crawlers is part of being cited at all. Our guide to AI search optimisation covers that side. Training opt-outs are a legal and brand decision: some groups prefer to keep clinical content out of training sets, others do not mind.

A reasonable default for most hospitals is to allow search and retrieval agents, make a deliberate decision on training tokens, and review quarterly. An example structure, described rather than pasted as code:

1. A default group for all crawlers (User-agent: *) with the parameter and internal-search blocks.
2. Separate named groups for each AI token you want to treat differently, each with its own allow and disallow lines.
3. A Sitemap line pointing to your index sitemap.

Remember that a crawler follows the most specific group matching its name, and ignores the generic group if a specific one exists. If you create a group for GPTBot, repeat any generic blocks you still want applied to it.

Common failure modes

• Wildcard over-match that blocks a service line (Disallow: /ivf blocking /ivf-cost-india).
• Environment leakage. A production deploy overwrites the live file with the staging one that disallows everything.
• CDN or firewall blocks. The file allows a crawler, but a bot-protection rule returns 403. The file is not the problem, and the vendor docs list IP ranges to allow.
• Multiple files. The CMS serves one file and a plugin adds another, and different layers disagree.
• Subdomain gaps. The main site is clean while a booking or telehealth subdomain has no file, or a stray one.
• Fixing 404s by blocking them. A blocked URL is not removed from the index. Return the correct status code.

Review checklist

• Open the live file on every hostname you operate. Confirm it is the intended version.
• List every Disallow line, state what it is for, and test each against five real URLs it should and should not match.
• Confirm CSS, JS, images and sitemaps are fetchable.
• Confirm no page that carries a noindex is also disallowed.
• Decide, in writing, your stance on each AI token, and who owns the review.
• Check server logs for the crawlers you allowed, to confirm they are actually arriving.
• Check Search Console's crawl stats and page indexing report after any change.
• Keep the file in version control so changes have an author and a date.

If you would like an outside review of crawl rules alongside the rest of your site's technical health, a technical SEO audit covers this file, the sitemaps and the rendering checks together. The technical SEO glossary entry is a quick reference for terms used above.