Is Your Web Host Blocking AI Search? Technical Checklist

Is Your Web Host Blocking AI Search? A Technical Checklist for Visibility in ChatGPT, Perplexity and Google AI Overviews

Is Your Web Host Blocking AI Search_ A Technical Checklist for Visibility in ChatGPT, Perplexity and Google AI Overviews

More and more people start their research in an AI assistant instead of a search box. They ask ChatGPT to compare project management tools, ask Perplexity which hosting plan suits a small online store, or read the AI Overview at the top of a Google results page. When those systems build an answer, they recommend and cite the sources they were able to read.

Most advice on Generative Engine Optimization (GEO) focuses on content: clear answers, structured pages, expert authors. That advice only works if the AI systems can reach your pages in the first place. In the GEO audits we run at TRYSEO, a B2B SEO and GEO agency based in Germany, this access check comes first, because it is often decided below the content layer: in your robots.txt file, your CDN settings, your firewall and the limits of your hosting plan.

This checklist walks through the technical points that decide whether AI crawlers can access your website, how to test each one, and what to change if something is blocking them.

Why your hosting setup now affects AI visibility

Classic SEO has trained site owners to make sure Googlebot can crawl their pages. AI search adds a new group of crawlers with different names, different purposes and different technical behavior. Many security tools, CDNs and hosting providers treat these crawlers as unwanted bot traffic by default, because some of them generate heavy load and many site owners do not want their content used for model training.

The result is that a website can rank well on Google and still be close to invisible in ChatGPT or Perplexity, without anyone on the team having made that decision. The block happens silently in a firewall rule, a plugin setting or a CDN toggle.

Know which crawlers you are dealing with

AI companies run separate crawlers for separate jobs. The distinction matters, because you may want to allow some and block others.

CompanyUser agentPurpose
OpenAIGPTBotCollects content for model training
OpenAIOAI-SearchBotIndexes pages for search results in ChatGPT
OpenAIChatGPT-UserFetches pages when a user’s request triggers a visit
AnthropicClaudeBotCollects content for model training
AnthropicClaude-SearchBot, Claude-UserSearch indexing and user-triggered fetches for Claude
PerplexityPerplexityBotIndexes pages for Perplexity answers
PerplexityPerplexity-UserFetches pages during a user’s query
GoogleGooglebotRegular Search crawling, which also feeds AI Overviews and AI Mode
GoogleGoogle-ExtendedA robots.txt token that controls use of your content for Gemini models, not a separate crawler
MicrosoftBingbotBing Search crawling, which also feeds Microsoft Copilot

Two points are easy to miss. First, AI Overviews in Google Search are fed by the regular Googlebot, so blocking Google-Extended does not remove you from them. Google’s crawler documentation confirms that the Google-Extended token does not affect inclusion or ranking in Search. Blocking Googlebot, on the other hand, removes you from Google entirely. Second, the search and user-triggered crawlers are the ones that decide whether you get cited in live answers. Blocking a training crawler such as GPTBot is a legitimate business decision. Blocking OAI-SearchBot or PerplexityBot by accident means giving up visibility you probably wanted. Crawler names change over time, so check each provider’s current documentation before you write rules.

1. Check robots.txt, including the one your CMS generates

Open yourdomain.com/robots.txt in a browser and read it line by line. Look for user agent groups that name AI crawlers with a Disallow rule, and for a broad block such as “User-agent: *” followed by “Disallow: /” that someone left in place after a staging launch.

On WordPress and similar systems, robots.txt is often generated dynamically by an SEO or security plugin. Some of these plugins now offer a single switch to block AI bots, and it may have been turned on during setup. If the file you see in the browser does not match what you expected, find the plugin or setting that writes it.

A configuration that blocks model training but keeps search and user-triggered access open looks like this:

User-agent: GPTBot

Disallow: /

 

User-agent: OAI-SearchBot

Allow: /

 

User-agent: ChatGPT-User

Allow: /

 

User-agent: PerplexityBot

Allow: /

Whether you block training crawlers at all is your decision. The important part is that the decision is deliberate.

2. Check your CDN and firewall bot rules

This is where most accidental blocks happen, because these rules sit outside the website, in the firewall or the content delivery network (CDN), and are rarely reviewed by the people responsible for marketing.

Cloudflare, for example, introduced settings to block AI crawlers and began blocking them by default for new domains in 2025. Its bot management features can also challenge or block automated traffic in general. Other CDNs and web application firewalls offer similar options, and some managed hosting providers apply bot filtering at the server level without showing it prominently in the dashboard.

Go through these places:

  • The bot and AI crawler settings in your CDN dashboard, including any “block AI bots” or “AI crawl control” options.
  • Custom firewall rules that block or challenge requests by user agent, country or ASN. AI crawlers often run from cloud provider IP ranges that broad rules catch.
  • Security plugins such as WordPress firewalls, which may rate limit or block unknown bots.
  • Server-level bot protection offered by your hosting provider.

A JavaScript challenge or CAPTCHA page counts as a block. Crawlers do not solve challenges, so a page that shows a “checking your browser” screen to bots is effectively unavailable to them.

3. Review rate limits and server capacity

Some AI crawlers request many pages in a short time. On shared hosting with tight resource limits, this can trigger rate limiting, 429 (Too Many Requests) responses or 503 (Service Unavailable) errors. A crawler that keeps receiving errors will come back less often or stop requesting those pages.

If your logs show frequent 429 or 503 responses to AI crawlers, you have three options: raise the limits for verified crawlers, add caching so repeated requests are served cheaply, or move to a plan with more resources. Blocking the crawler entirely solves the load problem but also removes you from the answers that crawler feeds.

4. Keep server response times low

User-triggered fetchers such as ChatGPT-User and Perplexity-User retrieve pages while a person is waiting for an answer. These systems work with tight time budgets, and a page that takes several seconds to respond is more likely to be skipped in favor of a faster source.

Time to first byte (TTFB) is the number to watch here. Full-page caching, a CDN with edge caching for HTML, an up-to-date PHP version and a database that is not overloaded all help. These are the same improvements that help human visitors and Core Web Vitals, so the work pays off twice.

5. Serve your main content in the HTML

Googlebot renders JavaScript. Most AI crawlers do not, or do so only in limited ways. If your product descriptions, pricing tables or articles are loaded by JavaScript after the page opens, an AI crawler may see an empty template.

To test this, view the page source (not the rendered page in developer tools) and search for a sentence from your main content. If it is missing from the source, the content depends on client-side rendering. Server-side rendering, static generation or pre-rendering for bots solves the problem. For sites built on single-page application frameworks, this is often the single biggest barrier to AI visibility.

6. Clean up status codes, redirects and canonicals

The basics of technical SEO apply to AI crawlers as well:

  • Important pages should return a 200 status code directly, not after a chain of redirects.
  • Redirects should be 301s that point to the final URL in one step.
  • Canonical tags should point to the version of the page you want cited.
  • Pages you want to appear in answers should not carry a noindex tag. Some AI search systems rely on traditional search indexes, so a page excluded from Google or Bing has fewer ways to reach AI answers.

This is especially important after a hosting migration or a website relaunch, when redirect rules and server configurations change and errors are easy to introduce.

7. Add structured data, and treat llms.txt as optional

Structured data in Schema.org format helps machines understand what a page is about: an organization, a product, an article, an FAQ. Organization markup with consistent company name, address and links to official profiles makes it easier for AI systems to connect your website with your brand.

You may also have come across llms.txt, a proposed file that lists a site’s most important pages in a format meant for language models. It is cheap to add, but none of the major AI providers has confirmed that its systems use it. It does not replace any of the checks above.

How to verify what AI crawlers actually see

Configuration settings tell you what should happen. Server logs tell you what does happen. Most hosting control panels give you access to raw access logs, and that is where the answer lives.

Search the logs for the user agents from the table above and check three things: whether the crawlers visit at all, which status codes they receive, and which pages they request. A crawler that only ever receives 403 responses is being blocked somewhere. A crawler that never appears may be blocked at the CDN level before the request reaches your server, in which case your CDN’s own logs or analytics will show it.

For a quick manual test, request a page with a crawler’s user agent from the command line:

curl -I -A “OAI-SearchBot” https://yourdomain.com/

A 200 response is a good sign. Keep in mind that some firewalls verify crawlers by IP address, so a request from your own machine may be treated differently from one coming from the real crawler. The major providers publish their crawler IP ranges, which you can use both to verify genuine crawler traffic in your logs and to allow it in firewall rules without opening the door to bots that only fake the user agent.

The checklist

  1. Decide which AI crawlers you want to allow, separating training crawlers from search and user-triggered crawlers.
  2. Read your live robots.txt and find the plugin or setting that generates it.
  3. Review AI bot settings, firewall rules and challenges in your CDN.
  4. Check security plugins and server-level bot protection at your hosting provider.
  5. Look for 429 and 503 responses to crawlers and adjust rate limits or caching.
  6. Measure TTFB on your key pages and add caching where it is slow.
  7. Confirm that your main content appears in the HTML source.
  8. Fix redirect chains, wrong canonicals and unintended noindex tags.
  9. Add Organization and page-level structured data.
  10. Check server and CDN logs every month to confirm the crawlers are getting 200 responses.

Conclusion

AI assistants can only recommend what they can read. Before investing in content for AI search, make sure your hosting stack is not turning those systems away at the door. In most cases, the fix is a setting in a dashboard, not a rebuild. Once the technical access is in place, the content work of answering questions clearly, showing expertise and earning mentions on trusted sites has a chance to show results in ChatGPT, Perplexity and Google’s AI features.

About the author

Renan Ausec is an SEO and GEO Specialist at TRYSEO, a B2B SEO and GEO agency based in Magdeburg, Germany. He works on technical SEO, content and visibility in AI search for companies in Germany and Brazil.

Is Your Host Keeping Its Uptime Promise? How to Measure It Yourself

Almost every hosting plan advertises an uptime figure. It is usually 99.9%, sometimes 99.95% or 99.99%, and it sits on the pricing page next t...
4 min read
Walter Akolo
Walter Akolo
Hosting Expert

What American expats should know about foreign bank accounts when moving overseas? 

Moving abroad brings exciting opportunities, but navigating the financial landscape as a US citizen living overseas requires careful planning ...
5 min read
Walter Akolo
Walter Akolo
Hosting Expert

How to Build a CRM: Build Your Own With AI or No-Code in 8 Steps

Learning how to build a CRM starts with one question: what does your sales team need that ready-made CRM software doesn't give you? If the ans...
11 min read
Matthew Ellis
Matthew Ellis
Web Developer

How Long Does It Take to Make an App? Months vs. Hours, Explained

How long does it take to make an app? Development agencies often estimate 2 to 4 months for a simple app and 7 to 12 months or more for a comp...
8 min read
Matthew Ellis
Matthew Ellis
Web Developer
Click to go to the top of the page
Go To Top
HostAdvice.com provides professional web hosting reviews fully independent of any other entity. Our reviews are unbiased, honest, and apply the same evaluation standards to all those reviewed. While monetary compensation is received from a few of the companies listed on this site, compensation of services and products have no influence on the direction or conclusions of our reviews. Nor does the compensation influence our rankings for certain host companies. This compensation covers account purchasing costs, testing costs and royalties paid to reviewers.