This post is for the marketing lead or SEO of a small clinic or practice whose website runs on shared hosting, and who has never opened an access log. Search Console tells you what Google chose to report. The server log tells you what actually requested your pages: every bot and every human, with the URL, the time, the status code and the user-agent string.
You will not need a dedicated log platform. A month of logs from a clinic site is small enough for a spreadsheet or a short script. The method below covers getting the files, separating real crawlers from impostors, reading the status-code mix, and measuring how often each type of page is crawled. We describe the method rather than report figures from any one site, because the useful numbers are the ones from your own logs.
A caution before you start. Access logs contain IP addresses, and under most privacy policies those are personal data. Work on a copy, keep it on a restricted drive, do not paste raw lines into shared documents or chats, and delete the working copy when you are done.
Step 1: Get 30 days of raw logs
On shared hosting, access logs are usually available from the control panel (look for "Raw access", "Access logs" or "Metrics") or by FTP/SFTP from a logs directory. Three practical problems recur.
- Retention. Many hosts keep only a few days of raw logs, rotating and compressing older ones. If you can only get a week, start a scheduled download now and analyse again in a month.
- Format. Most hosts write the "combined" format: client IP, timestamp, request line, status, bytes, referrer and user agent. Check the first line. If the host adds response time or the host header, note which column it is.
- Proxy or CDN in front. If the site sits behind a CDN, the IP in the origin log may be the CDN's. Look for a forwarded-for field, or use the CDN's own logs.
Concatenate the files in date order and keep one working file. Confirm that the first and last timestamps cover the 30 days you expect.
Step 2: Filter by user agent, then verify
Start by splitting traffic by user-agent string: Googlebot, Bingbot, known AI crawlers, other bots, and everything else. The user-agent string alone proves nothing, because anyone can send "Googlebot" in a header. Verification matters because fake crawlers are common, and counting them as Google will skew every conclusion.
Googlebot. Google's verifying Googlebot guide gives two methods. The manual one: run a reverse DNS lookup on the IP, confirm that the hostname ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that hostname and confirm it returns the original IP. The automatic one: match the IP against the published JSON files of crawler ranges (common crawlers, special-case crawlers and user-triggered fetchers are separate files). Match against the range lists for bulk work, and use reverse DNS for spot checks.
Bingbot. Microsoft documents how to verify Bingbot and publishes an IP list; apply the same two-way logic.
AI crawlers. OpenAI's crawler documentation lists three separate agents with different purposes and each has its own published IP list: GPTBot, used for content that may be used to train its models; OAI-SearchBot, which surfaces sites in ChatGPT search; and ChatGPT-User, which fetches a page when a person's request in ChatGPT triggers it. The document states that ChatGPT-User is not used for automatic crawling and that robots.txt rules may not apply to it. These differences matter when you decide what to allow. Other operators, including Anthropic and Perplexity, publish their own user-agent names and documentation; check each operator's page rather than relying on a list from a blog, including this one.
Anything that claims a verified crawler's name but fails verification goes into a separate "impostor" bucket. Keep it out of the totals.
Step 3: Read the status-code mix
For each verified crawler, count requests by status code. Group them as 200, 301/302/307/308, 304, 404/410, 403/401, 429 and 5xx. What each group tells you:
- 200 on the pages you care about is the healthy baseline.
- A large share of 301s means crawlers are following old URLs. Look at the Location values. Internal links, sitemap entries or old campaign URLs may still point at redirected addresses, and each hop costs a request.
- 304 (Not Modified) is good. It means the server answered a conditional request and Google reused its copy. If you never see 304s, the server may not be sending the headers that make them possible.
- 404 and 410 from a crawler that discovered the URL by following a link means something on your site or another site links to it. Group them by pattern, such as old WordPress paths, removed doctor pages or mistyped campaign URLs.
- 403, 401 and 429 to a verified crawler suggest a firewall, security plugin or rate limiter is blocking the bot. This is a frequent cause of "Googlebot cannot access" warnings on shared hosting.
- 5xx means the server failed. Check the timestamps; clusters usually line up with backups, plugin updates or traffic spikes.
Do the same split for the impostor bucket. A flood of fake Googlebot 404s on admin or login paths is a security sign, not an SEO one, and your host can block it.
