
More and more people start their research in an AI assistant instead of a search box. They ask ChatGPT to compare project management tools, ask Perplexity which hosting plan suits a small online store, or read the AI Overview at the top of a Google results page. When those systems build an answer, they recommend and cite the sources they were able to read.
Most advice on Generative Engine Optimization (GEO) focuses on content: clear answers, structured pages, expert authors. That advice only works if the AI systems can reach your pages in the first place. In the GEO audits we run at TRYSEO, a B2B SEO and GEO agency based in Germany, this access check comes first, because it is often decided below the content layer: in your robots.txt file, your CDN settings, your firewall and the limits of your hosting plan.
This checklist walks through the technical points that decide whether AI crawlers can access your website, how to test each one, and what to change if something is blocking them.
Why your hosting setup now affects AI visibility
Classic SEO has trained site owners to make sure Googlebot can crawl their pages. AI search adds a new group of crawlers with different names, different purposes and different technical behavior. Many security tools, CDNs and hosting providers treat these crawlers as unwanted bot traffic by default, because some of them generate heavy load and many site owners do not want their content used for model training.
The result is that a website can rank well on Google and still be close to invisible in ChatGPT or Perplexity, without anyone on the team having made that decision. The block happens silently in a firewall rule, a plugin setting or a CDN toggle.
Know which crawlers you are dealing with
AI companies run separate crawlers for separate jobs. The distinction matters, because you may want to allow some and block others.
| Company | User agent | Purpose |
| OpenAI | GPTBot | Collects content for model training |
| OpenAI | OAI-SearchBot | Indexes pages for search results in ChatGPT |
| OpenAI | ChatGPT-User | Fetches pages when a user’s request triggers a visit |
| Anthropic | ClaudeBot | Collects content for model training |
| Anthropic | Claude-SearchBot, Claude-User | Search indexing and user-triggered fetches for Claude |
| Perplexity | PerplexityBot | Indexes pages for Perplexity answers |
| Perplexity | Perplexity-User | Fetches pages during a user’s query |
| Googlebot | Regular Search crawling, which also feeds AI Overviews and AI Mode | |
| Google-Extended | A robots.txt token that controls use of your content for Gemini models, not a separate crawler | |
| Microsoft | Bingbot | Bing Search crawling, which also feeds Microsoft Copilot |
Two points are easy to miss. First, AI Overviews in Google Search are fed by the regular Googlebot, so blocking Google-Extended does not remove you from them. Google’s crawler documentation confirms that the Google-Extended token does not affect inclusion or ranking in Search. Blocking Googlebot, on the other hand, removes you from Google entirely. Second, the search and user-triggered crawlers are the ones that decide whether you get cited in live answers. Blocking a training crawler such as GPTBot is a legitimate business decision. Blocking OAI-SearchBot or PerplexityBot by accident means giving up visibility you probably wanted. Crawler names change over time, so check each provider’s current documentation before you write rules.
1. Check robots.txt, including the one your CMS generates
Open yourdomain.com/robots.txt in a browser and read it line by line. Look for user agent groups that name AI crawlers with a Disallow rule, and for a broad block such as “User-agent: *” followed by “Disallow: /” that someone left in place after a staging launch.
On WordPress and similar systems, robots.txt is often generated dynamically by an SEO or security plugin. Some of these plugins now offer a single switch to block AI bots, and it may have been turned on during setup. If the file you see in the browser does not match what you expected, find the plugin or setting that writes it.
A configuration that blocks model training but keeps search and user-triggered access open looks like this:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
Whether you block training crawlers at all is your decision. The important part is that the decision is deliberate.
2. Check your CDN and firewall bot rules
This is where most accidental blocks happen, because these rules sit outside the website, in the firewall or the content delivery network (CDN), and are rarely reviewed by the people responsible for marketing.
Cloudflare, for example, introduced settings to block AI crawlers and began blocking them by default for new domains in 2025. Its bot management features can also challenge or block automated traffic in general. Other CDNs and web application firewalls offer similar options, and some managed hosting providers apply bot filtering at the server level without showing it prominently in the dashboard.
Go through these places:
- The bot and AI crawler settings in your CDN dashboard, including any “block AI bots” or “AI crawl control” options.
- Custom firewall rules that block or challenge requests by user agent, country or ASN. AI crawlers often run from cloud provider IP ranges that broad rules catch.
- Security plugins such as WordPress firewalls, which may rate limit or block unknown bots.
- Server-level bot protection offered by your hosting provider.
A JavaScript challenge or CAPTCHA page counts as a block. Crawlers do not solve challenges, so a page that shows a “checking your browser” screen to bots is effectively unavailable to them.
3. Review rate limits and server capacity
Some AI crawlers request many pages in a short time. On shared hosting with tight resource limits, this can trigger rate limiting, 429 (Too Many Requests) responses or 503 (Service Unavailable) errors. A crawler that keeps receiving errors will come back less often or stop requesting those pages.
If your logs show frequent 429 or 503 responses to AI crawlers, you have three options: raise the limits for verified crawlers, add caching so repeated requests are served cheaply, or move to a plan with more resources. Blocking the crawler entirely solves the load problem but also removes you from the answers that crawler feeds.
4. Keep server response times low
User-triggered fetchers such as ChatGPT-User and Perplexity-User retrieve pages while a person is waiting for an answer. These systems work with tight time budgets, and a page that takes several seconds to respond is more likely to be skipped in favor of a faster source.
Time to first byte (TTFB) is the number to watch here. Full-page caching, a CDN with edge caching for HTML, an up-to-date PHP version and a database that is not overloaded all help. These are the same improvements that help human visitors and Core Web Vitals, so the work pays off twice.
5. Serve your main content in the HTML
Googlebot renders JavaScript. Most AI crawlers do not, or do so only in limited ways. If your product descriptions, pricing tables or articles are loaded by JavaScript after the page opens, an AI crawler may see an empty template.
To test this, view the page source (not the rendered page in developer tools) and search for a sentence from your main content. If it is missing from the source, the content depends on client-side rendering. Server-side rendering, static generation or pre-rendering for bots solves the problem. For sites built on single-page application frameworks, this is often the single biggest barrier to AI visibility.
6. Clean up status codes, redirects and canonicals
The basics of technical SEO apply to AI crawlers as well:
- Important pages should return a 200 status code directly, not after a chain of redirects.
- Redirects should be 301s that point to the final URL in one step.
- Canonical tags should point to the version of the page you want cited.
- Pages you want to appear in answers should not carry a noindex tag. Some AI search systems rely on traditional search indexes, so a page excluded from Google or Bing has fewer ways to reach AI answers.
This is especially important after a hosting migration or a website relaunch, when redirect rules and server configurations change and errors are easy to introduce.
7. Add structured data, and treat llms.txt as optional
Structured data in Schema.org format helps machines understand what a page is about: an organization, a product, an article, an FAQ. Organization markup with consistent company name, address and links to official profiles makes it easier for AI systems to connect your website with your brand.
You may also have come across llms.txt, a proposed file that lists a site’s most important pages in a format meant for language models. It is cheap to add, but none of the major AI providers has confirmed that its systems use it. It does not replace any of the checks above.
How to verify what AI crawlers actually see
Configuration settings tell you what should happen. Server logs tell you what does happen. Most hosting control panels give you access to raw access logs, and that is where the answer lives.
Search the logs for the user agents from the table above and check three things: whether the crawlers visit at all, which status codes they receive, and which pages they request. A crawler that only ever receives 403 responses is being blocked somewhere. A crawler that never appears may be blocked at the CDN level before the request reaches your server, in which case your CDN’s own logs or analytics will show it.
For a quick manual test, request a page with a crawler’s user agent from the command line:
curl -I -A “OAI-SearchBot” https://yourdomain.com/
A 200 response is a good sign. Keep in mind that some firewalls verify crawlers by IP address, so a request from your own machine may be treated differently from one coming from the real crawler. The major providers publish their crawler IP ranges, which you can use both to verify genuine crawler traffic in your logs and to allow it in firewall rules without opening the door to bots that only fake the user agent.
The checklist
- Decide which AI crawlers you want to allow, separating training crawlers from search and user-triggered crawlers.
- Read your live robots.txt and find the plugin or setting that generates it.
- Review AI bot settings, firewall rules and challenges in your CDN.
- Check security plugins and server-level bot protection at your hosting provider.
- Look for 429 and 503 responses to crawlers and adjust rate limits or caching.
- Measure TTFB on your key pages and add caching where it is slow.
- Confirm that your main content appears in the HTML source.
- Fix redirect chains, wrong canonicals and unintended noindex tags.
- Add Organization and page-level structured data.
- Check server and CDN logs every month to confirm the crawlers are getting 200 responses.
Conclusion
AI assistants can only recommend what they can read. Before investing in content for AI search, make sure your hosting stack is not turning those systems away at the door. In most cases, the fix is a setting in a dashboard, not a rebuild. Once the technical access is in place, the content work of answering questions clearly, showing expertise and earning mentions on trusted sites has a chance to show results in ChatGPT, Perplexity and Google’s AI features.
About the author
Renan Ausec is an SEO and GEO Specialist at TRYSEO, a B2B SEO and GEO agency based in Magdeburg, Germany. He works on technical SEO, content and visibility in AI search for companies in Germany and Brazil.
