The quickest way is to open yourwebsite.co.uk/robots.txt in your browser. If you see the line Disallow: / under the name of an AI bot, such as OAI-SearchBot, Claude-SearchBot or PerplexityBot, that bot is blocked. The same line under User-agent: * blocks every bot that isn't named separately. It isn't the only check though. A firewall or security setting can turn AI bots away even when that file says they're welcome.

If AI bots can't read your website, AI tools have less to go on when your customers ask about businesses like yours. Blocking is also easy to do by accident, because your site still works fine for people.

This guide shows you how to check, what each bot does and what to change. The first steps need no technical skills. We make Midwaver, an AI visibility tracker that flags anything on your site that stops AI tools reading it. If AI search is new to you, start with What Is AI Search and Why Should Your Business Care?

Why it matters if AI bots can't read your website

AI search tools build their answers from web pages they've been able to read. If a tool can't get into a page, that page can't be used as a source. That's our reading of how these tools work. For ChatGPT, OpenAI says it outright: sites that opt out of its search bot "will not be shown in ChatGPT search answers".

Blocking can easily be accidental. A developer adds a rule to keep unwanted bots out. A security tool is switched on with strict settings. A setting from when the site was being built is never changed back. Nobody notices, because people can still visit the site.

The main AI bots and what they do

AI companies use more than one bot each. The bots do different jobs. That matters because blocking the wrong one can hide you from AI search. Blocking another may make no difference to it at all.

• Search bots. These find pages to use in AI search answers. OpenAI's is OAI-SearchBot, Anthropic's is Claude-SearchBot and Perplexity's is PerplexityBot. OpenAI says to allow OAI-SearchBot if you want to appear in ChatGPT search. Anthropic says disabling Claude-SearchBot may reduce your visibility and accuracy in Claude's search results. Perplexity says to allow PerplexityBot so your site appears in its search results.

• Training bots. These collect content that may be used to train AI models. The main ones are GPTBot (OpenAI) and ClaudeBot (Anthropic). Google-Extended is a related control for Google's Gemini. Google says it doesn't affect a site's inclusion in Google Search.

• User-triggered bots. These visit a page when a person asks an AI tool a question. They include ChatGPT-User, Claude-User and Perplexity-User. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person starts the visit. Perplexity says its user fetcher generally ignores robots.txt. Anthropic says disabling Claude-User may reduce your visibility for searches a person directs.

• Googlebot. This is Google's own crawler. Google says a page has to be indexed and eligible to be shown in Google Search with a snippet to be a supporting link in AI Overviews and AI Mode. The wider picture is in GEO vs SEO: What's the Difference?

How to check your robots.txt file

robots.txt is a plain text file that tells bots which parts of your website they may visit. Anyone can read it, so you can check yours in a minute.

1. Open the file. Type your website address into your browser followed by /robots.txt. If you get a "page not found" message, you don't have one. That means no rules are blocking anything.

2. Find the User-agent lines. Each group of rules starts with a User-agent line naming the bot it applies to. User-agent: * means every bot. A name such as GPTBot means that bot only.

3. Read the Disallow lines under it. Disallow: / means the whole site is off limits to that bot. A Disallow line pointing to one folder blocks only that folder. A Disallow line with nothing after it blocks nothing.

4. Look for the search bots. Check for OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot. If any of them has Disallow: / under it, that bot can't read your site.

Some website builders add rules for you. Wix has a robots.txt editor in its SEO tools. Its guide shows how to block AI crawlers there. On WordPress, look under Settings then Reading for the option that discourages search engines from indexing your site. If it's ticked, untick it unless the site is still being built.

If you're unsure what you're reading, send the file to your web developer. Ask whether OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot are allowed to read your public pages.

Other ways your website can block AI bots

robots.txt only controls bots that choose to follow it. Other things can turn bots away without touching that file.

• Firewalls and security tools. Services that protect your site, such as Cloudflare, can block bots. Cloudflare now lets every customer set rules for search, agent and training bots separately. Since 15 September 2026, new domains on Cloudflare block training and agent bots on pages that show ads, while search bots stay allowed. Check what your settings are. OpenAI and Perplexity publish lists of the IP addresses their bots use, so a firewall can be set to let them in.

• Blocking by IP address. Anthropic warns that blocking IP addresses may not reliably opt you out. It can also stop its bots from reading your robots.txt file.

• Text hidden behind scripts. A study by Vercel and MERJ in December 2024 found that GPTBot, ClaudeBot and PerplexityBot didn't run JavaScript. So text that only appears after a page's scripts load may be invisible to them. Google's Gemini uses Googlebot's systems, which can run scripts. That study is nearly two years old, so tools may have changed. To test your own page, right-click it, choose View page source and search for a sentence from the page. If it isn't there, bots that don't run scripts may not see it.

• Logins and bot checks. Pages behind a login can't be read. "Are you human?" checks can also turn bots away.

• Tags that hide a page from search. A noindex tag tells Google to leave a page out of search. Google says a page has to be indexed and eligible to be shown to be a supporting link in its AI features.

A developer can test all of this in a few minutes by requesting your pages using each bot's name. If the site answers with "forbidden" or "access denied", often shown as a 403 error, that bot is being turned away.

What to allow and what to block

This part is our view rather than a rule. If you want AI tools to recommend you, we'd allow the search bots and the user-triggered bots. Whether to allow training bots is a separate decision. Some businesses don't want their content used to train AI models. That's a fair choice. Blocking training bots such as GPTBot and ClaudeBot isn't the same as blocking AI search, as long as the search bots are still allowed. We haven't found evidence that allowing training bots makes a business more likely to be recommended.

Be careful with any tool that offers one switch to "block all AI bots". It can catch the search bots as well. Cloudflare notes that bots that do both searching and training are affected when training is blocked.

How to fix it

1. Decide which bots you want. Use the section above as a guide.

2. Edit your robots.txt file. Remove the Disallow: / lines for the bots you want to let in. Your website builder or developer can do this.

3. Check your firewall and security settings. Make sure the search bots aren't being blocked there. Ask your host or developer if you're unsure.

4. Wait and check again. OpenAI says it can take about 24 hours for its systems to adjust after you update robots.txt. Then run the checks above again.

5. Look at the rest of your site. Access is only the start. The basics of being recommended are in How to Get Your Business Recommended by ChatGPT.

Find out how you show up

Letting AI bots in gets them to your door. It doesn't decide whether they recommend you. The next question is what AI actually says when your customers ask about businesses like yours.

Midwaver asks ChatGPT, Perplexity and Google AI Overviews the questions your customers ask. It then shows you the answers. You can see how often you're mentioned and how that's trending, which questions bring you up, which competitors are named instead, where your website is cited and which sources AI tools rely on. It also flags anything on your site that stops AI tools reading it.

Midwaver can't make an AI tool recommend you and no tool can. It shows you where you stand so you can tell whether what you change makes a difference.

Search is changing. Your customers used to search on Google or Bing and get a list of links. Now they’re asking AI for recommendations. Is your business being recommended? Start your free 7 day trial with Midwaver. See what AI says about your business, who’s winning in your industry and how to get mentioned more. No credit card needed, no obligation.