How to check if GPTBot and other AI crawlers are blocked in robots.txt
Published October 23, 2026 · 5 min read
Open yourdomain.com/robots.txt and look for “User-agent” lines that name AI crawlers, followed by “Disallow: /”. For ChatGPT's search answers the crawler that matters most is OAI-SearchBot: OpenAI says sites that block it will not appear in ChatGPT search answers, while GPTBot is the separate crawler for training.
Step 1: open your robots.txt
Type your domain followed by /robots.txt into a browser. If you get a “not found” page, nothing is blocked by robots.txt (though a firewall or bot-protection service can still block crawlers).
Step 2: look for these crawler names
A block looks like a “User-agent:” line with the crawler's name followed by “Disallow: /”. A line “User-agent: *” with “Disallow: /” blocks every crawler that respects robots.txt.
| User-agent | Who runs it | What it is for |
|---|---|---|
| OAI-SearchBot | OpenAI | Finds pages for ChatGPT's search answers. Blocked sites do not appear in those answers. |
| GPTBot | OpenAI | Collects content that may be used to train OpenAI's models. Blocking it is a separate choice from search. |
| ChatGPT-User | OpenAI | Visits a page when a user asks ChatGPT to. Not used to decide search inclusion. |
| Googlebot | Google Search. Google's AI features in Search work from pages Google can crawl and index. |
Step 3: check what else stands in the way
- Bot protection or a firewall that challenges unknown visitors can stop crawlers even when robots.txt allows them.
- Pages whose text only appears after JavaScript runs may look empty to crawlers that do not run JavaScript.
- A “noindex” tag tells search engines not to index the page.
A free test
Seroq's free site check reads your robots.txt and homepage and runs 19 fixed tests, including whether AI crawlers are allowed, whether your text is in the page source and whether your company facts are in structured data. It takes seconds and needs no account.
Questions people also ask
- If I block GPTBot, will ChatGPT stop recommending me?
- According to OpenAI, GPTBot and OAI-SearchBot are controlled separately: blocking GPTBot signals that your content should not be used for training, while OAI-SearchBot decides whether you can appear in ChatGPT search answers.
- How long does a change take?
- OpenAI says robots.txt changes for its search crawler can take about 24 hours to take effect.
Sources
See what AI says about your brand
Start with the free check: it shows in seconds whether AI crawlers can read your site. A full Seroq report asks the AI engines your buyers' questions and shows who gets chosen first.