Skip to main content
NewNMC’s advertising guidelines for doctors are in force from 6 Oct 2026. See what to change, and get a free compliance auditNew NMC advertising rules for doctors: what to change
AI Search

Identifying AI Referral Traffic in Server Logs and Search Console

A technical guide to separating AI crawlers, user-initiated fetches and human referrals in server logs, GA4 and Search Console, and an honest list of what cannot be attributed.

Identifying AI Referral Traffic in Server Logs and Search Console

Marketing teams now get a monthly question: how much traffic do we get from AI? The honest answer is that some of it is visible, some is misfiled under direct or organic, and some cannot be seen at all. This article is for analytics owners, web developers and marketing leads at healthcare organisations who can read server logs or work with someone who can. It shows which signals are reliable, which are partial and what to stop claiming.

It is deliberately separate from the question of what AI says about you. For the wider effect of generative search on healthcare traffic, see AI Overviews and healthcare SEO. Here we concentrate on identification.

Three different things called "AI traffic"

Keeping these apart prevents most reporting errors.

  1. Crawler requests. Automated systems fetch your pages to train a model, build a search index or retrieve content. They are not visitors.
  2. User-initiated fetches. A person asks an assistant a question, and the assistant fetches your page in real time to answer. The request comes from the vendor's infrastructure, not from the person's browser.
  3. Human referrals. A person clicks a link in an AI answer and lands on your site, usually carrying a referrer or a tracking parameter.

Logs show all three. Analytics tags, which run in the browser, show only the third, and only when the browser supplies enough information.

Read the vendors' documentation first

Vendors publish user-agent names, and they have different purposes. As documented at the time of writing:

  • OpenAI's bots documentation lists OAI-SearchBot, which surfaces sites in ChatGPT search features; GPTBot, which crawls content that may be used to train models; and ChatGPT-User, which handles user-initiated actions and, because a user triggers it, may not follow robots.txt rules.
  • Perplexity's bots guide describes PerplexityBot, used to surface and link sites in search results, and Perplexity-User, which supports user actions and generally ignores robots.txt because a user requested the fetch.
  • Anthropic's crawler help page names three bots: ClaudeBot for training content, Claude-User for user-directed retrieval and Claude-SearchBot for improving search results.
  • Google says that AI features in Search use no dedicated crawler. Googlebot governs access for Search, including AI features, and Google-Extended is a separate control for some other Google AI uses.

The purposes matter for policy. Blocking a training crawler does not remove you from a vendor's search results, and blocking a search crawler can. Decide separately for each, with legal and clinical communications input, and re-read the vendor pages before you act, because these names and behaviours change.

Finding crawlers in server logs

Pull a thirty-day access log from your server, CDN or reverse proxy. If the site runs behind a CDN, use the CDN logs, since the origin may only see the CDN's addresses.

  1. Filter the user-agent field for the bot names above. Match on the name token, not the full string, because version text changes.
  2. Count requests per bot per day and the top fifty URLs per bot.
  3. Check response codes. A high share of 403, 404 or 5xx responses to a search bot means your content may not be retrievable.
  4. Compare against your sitemap. Which service and doctor pages are never fetched?
  5. Verify identity. User-agent strings can be faked. Where a vendor publishes IP ranges or verification methods, check that requests claiming to be that bot come from them, and treat the rest as unverified.

A hypothetical illustration: a hospital's log shows ChatGPT-User fetching one doctor-profile page repeatedly on the same afternoon. That suggests someone, or several people, asked an assistant a question that involved that doctor. It does not tell you who, what they asked or whether they later booked. It is a sign that the page is a retrieval source, which is useful for content planning.

Finding human referrals

When a person clicks a citation, the visit may arrive with a referrer such as the assistant's domain, or with a tracking parameter appended to the link. Do not assume the exact strings. Look in your own data.

In GA4:

  1. Open the traffic acquisition report with session source and medium.
  2. Search sources for the names of the assistants you care about, and for unfamiliar sources that appear after you test.
  3. Create a custom channel group, for example "AI assistants", with rules that match the source values you actually observe. Keep the rule list in a document and revisit it quarterly, since new products and domains appear.
  4. Break out landing pages. AI referrals tend to land on deep informational pages, not the homepage.

In server logs, filter the referrer header for the same domains, and inspect the query string of the landing URL for source parameters. Logs catch visits that blocked scripts hide from GA4, so compare the two counts. A large gap tells you how much your analytics is missing.

What Search Console can and cannot show

Google states that sites shown in AI Overviews and AI Mode count toward overall search traffic and appear in the Performance report under the Web search type. In other words, clicks from Google's AI features are folded into your ordinary Search Console numbers. At the time of writing, that documentation does not describe a separate breakout for them, so you cannot isolate them by filter. Check the page again before telling stakeholders this, in case Google has added a report.

You can still infer. Look at query types: long, conversational queries are more likely to trigger AI features. Track impressions and click-through for that query group against a short-keyword group over time, and annotate the dates when AI features appeared in your results. That is inference, and should be labelled as such.

Want this applied to your practice? Get a free ai search audit.

Book my free audit

What cannot be attributed

State these limits in every report.

  • Impressions inside AI answers with no click. Nobody can see them on your side.
  • Copy-pasted URLs and typed names. A patient who reads your name in an answer and searches for it later appears as branded organic. Treat that indirectly, through branded-query trends.
  • App and in-app browser traffic. Some clicks arrive with no referrer and show as direct.
  • Mobile apps that open links through a redirect. The referrer may belong to an intermediary.
  • Attribution to a specific prompt. No log tells you what the person asked.
  • Calls and WhatsApp conversions that start from an AI answer. Without a tracked number or link, these are invisible.

A test you can run in an afternoon

Create a list of fifty representative queries and run them on the assistants your patients use. For each answer that cites your site, click the citation from a normal browser and note, in a sheet: assistant, device, browser, whether the page loaded, and what appears in your analytics and server log within a few minutes. This tells you which assistants send a referrer, which add parameters, and which leave nothing. It also shows whether pages cited by assistants actually respond quickly and without errors.

Repeat after any site migration or CDN change. Do the test from a mobile device on mobile data too, because in-app browsers behave differently from desktop browsers.

Segment before you summarise

A single total hides the useful detail. Once you have an AI assistants channel group, break it down three ways.

  • By landing page type. Separate doctor profiles, condition pages, service pages, cost pages and the blog. Healthcare referrals that land on a cost or comparison page behave differently from those landing on an explainer, and the content team needs to know which pages assistants send people to.
  • By device and country. Medical tourism teams should see whether referrals from outside India reach international patient pages or the Indian service pages. A mismatch is a content gap.
  • By downstream action. Tie sessions to the events that matter: appointment form submitted, click to call, WhatsApp click. Volumes will be small, so report counts and look at the pages involved, not conversion rates built on a handful of sessions.

Then compare the pages assistants fetch in the logs with the pages humans arrive on. If a page is fetched often by user-initiated bots but rarely clicked, the assistant is probably summarising it without sending the reader on. That page may need a clearer reason to visit, such as a booking step or a tool that cannot be summarised.

Failure modes

  • Blocking by accident. A firewall or bot-protection rule blocks search crawlers while allowing everything else. Check the response codes for each bot.
  • Treating user-agent counts as visitors. Ten thousand requests from a training crawler are not ten thousand readers.
  • Overbuilt regexes. A channel rule that matches loosely, for example any source containing a short string, will misclassify unrelated traffic. Test rules on historical data.
  • Dropping tracking on redirects. Marketing short links that strip parameters hide the source.
  • Reporting AI traffic as total AI influence. It is a lower bound.

Reporting wording that holds up

Use language such as: "Identified AI assistant referrals: N sessions, a lower bound. Crawler activity: the following bots fetched the following sections. Google AI features: included in Web search in Search Console and not separable." That sets expectations honestly.

AiLLMz organises citations and publication workflows that support visibility across AI search and LLMs; the log and analytics steps here stay the same whichever tools you use. For the tracking set-up, see our analytics tracking service and SEO reporting service, and the AI search optimisation service for what to do with the findings.

Where to start

  • Export thirty days of CDN or server logs and count requests for each named bot.
  • Check that your search-relevant bots receive success responses on your service pages.
  • Build an "AI assistants" channel group in GA4 from the sources you observe.
  • Run the fifty-query click test and record what each assistant passes on.
  • Add the limits list to your monthly report template.
  • Review vendor bot documentation every quarter.
Filed underAI referral trafficChatGPT referral traffic GA4AI crawler user agentsGPTBot OAI-SearchBot ChatGPT-UserSearch Console AI Mode traffic

About the author

Co-Founder & CTO · Gurugram, India

Arush builds the systems the rest of the company runs on — the patient-acquisition pipelines, the AI chatbots, the analytics that tie a single click to a booked appointment. If a campaign reports a number, it's because something his team built is tracking it.

Free guide · 15 pages

The Patient Acquisition Blueprint

A 90-day plan from first search to booked appointment.

The Pioneers Brief — one essay, three benchmarks, one teardown, every Thursday. Subscribe free →

Free ai search audit

Your next chapter starts with a conversation.

Three quick questions. We come prepared with a plan for your specialty and city.

  • A 30-minute call with a healthcare specialist
  • Your top three growth opportunities, in writing
  • Honest advice — even if that means not hiring us
  • No obligation and no hard sell
You’ll speak with a senior strategist
Consultations are hosted by Arush Thapar’s team.
Step 1 of 3Takes 30 seconds

What do you want to improve first?

Details stay privateNo obligationReply within 1 working day

Client portfolio · SEO & search visibility

Work relevant to identifying ai referral traffic in server logs and search.

Explore client work →