How to Check if ChatGPT Can See Your Website (and Fix It if It Can't)

The quick check: open ChatGPT with search on and ask "What does yourdomain.com do?" If it returns real details from your pages, ChatGPT can see you. If it says it can't access the site — or invents an answer — you're blocked. Confirm at the crawler level: curl -A "GPTBot/1.2" -I https://yourdomain.com should return 200.

Ask ChatGPT about your product category and watch who gets cited. If it's never you, the failure lives in exactly one of three places: access (the crawler can't reach you), extraction (it reaches you but sees nothing), or grounding (it sees you but won't cite you). This guide tests all three in under ten minutes — by hand, or in one automated AEO scan.

TL;DR

• Test it directly: ask ChatGPT (search mode) and Perplexity "What does yourdomain.com do?" A vague, refused, or uncited answer means you aren't readable.
• Three OpenAI agents, three jobs: GPTBot (training), OAI-SearchBot (the ChatGPT Search index), ChatGPT-User (fetches a link on demand when a user asks).
• Blocking GPTBot does not remove you from ChatGPT Search — that index belongs to OAI-SearchBot.
• Four blockers cause nearly every case: robots.txt Disallow: /, WAF and bot-fight rules, client-side rendering, and geo-blocks or login walls.
• Google-Extended governs Gemini training and grounding only. It has no effect on Google Search inclusion or ranking.

How to test your site in ChatGPT and Perplexity right now

No tooling required, and nothing to install. Run all three tests — they fail in different ways, and the difference is the diagnosis.

1. ChatGPT, with search enabled. Paste this exactly, swapping in your domain:

What does yourdomain.com do? Cite the pages you used.

Then force a live fetch with a specific URL:

Summarize the pricing page at https://yourdomain.com/pricing in three bullets.

Because that second prompt names a URL, ChatGPT fetches it on demand as ChatGPT-User instead of reading its search index. Useful signal — but a URL-paste that works does not prove OAI-SearchBot has you indexed. Always run the first prompt too.

2. Perplexity. Same prompts, unchanged. Perplexity searches on every query, so it's the cleanest read on whether PerplexityBot has actually indexed you:

What does yourdomain.com do? List your sources.

3. A branded-versus-generic pair. Ask by name, then ask by category:

What is [Your Product] and who is it for?
What are the best tools for [your category]?

What a good result looks like

• Specific facts that only your site states — your real pricing tiers, your feature names, your positioning line.
• Citations pointing at your domain, and ideally at more than just the homepage.
• The category question surfaces you next to competitors, not only the branded one.

What a bad result looks like

• "I can't access that website" or "the site appears to be blocked" — a hard access failure. Go to Step 1.
• A confident answer with no citation to your domain — it's paraphrasing your Crunchbase, LinkedIn, or a directory listing, not your site.
• Generic filler like "a platform for businesses" — the crawler reached you and got an empty shell. Go to Step 3.
• Details that are one redesign out of date — you're being answered from training data, not a live fetch.
• Present on the branded question, absent from the category one — access is fine, grounding isn't. Go to Step 4.

AI crawler user agents: the full reference table

Every major answer engine announces itself with a distinct user agent, and the split matters: within a single company, one agent feeds model training and a different one feeds the search index you actually want to appear in.

The two most useful facts in that table: GPTBot is training-only, and Google-Extended is not a crawler at all — it's an opt-out switch for Gemini, with zero bearing on whether Google indexes or ranks you.

A copy-paste robots.txt that welcomes answer engines

Drop this at https://yourdomain.com/robots.txt and edit the private paths at the bottom. It allows every search-time and user-initiated agent while opting out of training-only crawlers:

# Answer engines — allow these if you want to be cited
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: meta-webindexer
Allow: /

User-agent: MistralAI-Index
Allow: /

User-agent: MistralAI-User
Allow: /

# Training-only crawlers — delete these groups if you're happy to be trained on
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: MistralAI-Training
Disallow: /

# Everyone else
User-agent: *
Disallow: /admin/
Disallow: /api/
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Two things to understand before you paste it:

• Blocking training crawlers is a trade-off, not a free win. It keeps your content out of future model weights — which also means the model has no baked-in knowledge of you to fall back on when live search doesn't fire.
• robots.txt is per-agent, not per-company. Blocking GPTBot says nothing about OAI-SearchBot, and blocking ClaudeBot says nothing about Claude-SearchBot. That separation is the entire point of the split tokens — use it deliberately.

Step 1: Check robots.txt for AI crawler blocks

Open https://yourdomain.com/robots.txt and read it top to bottom. Three things to look for:

1. A blanket User-agent: * with Disallow: /. The single most common cause, and almost always accidental — a staging config that shipped, a framework default, or a "coming soon" file nobody deleted. It blocks every well-behaved crawler in the table above.
2. Named AI agents under a Disallow: /. Usually inherited from a "privacy-focused" robots.txt template copied in 2024, when the prevailing advice was to block anything with Bot in the name.
3. No robots.txt at all. A missing file means allow-all, so it isn't a blocker — but it's also the only sitemap-discovery channel crawlers have, so add one.

Also confirm the file returns 200 with a plain-text content type. A single-page app that serves its shell at every path will hand crawlers an HTML page where robots.txt should be — which parses as no rules and no Sitemap: line:

curl -sI https://yourdomain.com/robots.txt | head -3

The fix: delete the block, or scope it to paths you genuinely want private. Blocking AI crawlers is a legitimate choice — but it should be a decision, not an inherited accident.

Step 2: Check your WAF and bot protection

robots.txt is only half of access. Cloudflare, Vercel's firewall, AWS WAF, Fastly, and most bot-protection products ship one-click "block AI bots" controls — and some are enabled by default.

Test every crawler in one pass:

for ua in "GPTBot/1.2" "OAI-SearchBot/1.0" "ClaudeBot/1.0" "Claude-SearchBot/1.0" "PerplexityBot/1.0"; do
  printf '%-22s %s\n' "$ua" "$(curl -s -o /dev/null -w '%{http_code}' -A "$ua" https://yourdomain.com)"
done

Anything that isn't 200 — or a redirect to your canonical URL — is a block. 403 is an outright deny, 429 is rate limiting, and a 200 that's only a few hundred bytes of "Checking your browser…" is a JavaScript challenge, which crawlers cannot solve.

On Cloudflare specifically, three separate settings can each cause this, and they live in different places:

• Bot Fight Mode / Super Bot Fight Mode (Security → Bots). Bot Fight Mode challenges non-verified automated traffic indiscriminately. Super Bot Fight Mode lets you set Verified Bots to Allow — do that, because OpenAI, Anthropic, Perplexity, Google, and Meta all publish IP ranges and are verifiable.
• The AI-crawler blocking toggle (Security → Bots, labelled "Block AI bots" or "AI Scrapers and Crawlers" depending on your dashboard). One switch, whole category. If it's on, you opted out of answer engines — check that you meant to.
• Custom WAF rules and rate limits (Security → WAF). Rules written against cf.client.bot or user-agent string matches often predate the search-time crawlers and catch them by accident.

The fix: allowlist by verified-bot category rather than by user-agent string. A UA string can be spoofed by anyone; verified-bot matching checks the request against the vendor's published IP ranges, so the real crawler gets in without opening a hole for impersonators. Then re-run the loop above and confirm 200 across the board.

Step 3: Check what the crawler actually sees

Most AI crawlers do not execute JavaScript. They take whatever HTML your server returns and parse it. If your site is client-side rendered, that HTML is often an empty <div id="root"></div> and a script tag.

Test the raw response — no browser, no rendering:

curl -s https://yourdomain.com | grep -i "your headline text"

No match means no answer engine has ever read that sentence. Two more worth running:

# How much content is actually in the HTML? A few KB or less is a shell.
curl -s https://yourdomain.com | wc -c

# Do the title and description survive without JS?
curl -s https://yourdomain.com | grep -iE "<title>|<meta name=\"description\""

This failure hides behind good Google rankings, because Googlebot renders JavaScript and most AI crawlers don't. You can sit on page one of Google and be completely invisible to ChatGPT at the same time.

The fix: server-side render or statically generate every page you want cited. Next.js, Astro, SvelteKit, and Remix all do this by default — but client-only useEffect data fetching, auth-gated shells, and "loading…" skeletons still leak into marketing pages constantly. Pure-SPA builders are the worst case: why AI engines can't find your Lovable site walks through the prerendering fix for that specific stack.

The four blockers that stop AI crawlers

Almost every "ChatGPT can't see my site" case is one of these four. They produce different symptoms, so the symptom tells you which one you have.

The last row is the one people miss, so it's worth spelling out.

Geo-blocking. Crawlers fetch from their vendor's own infrastructure, largely US datacenter ranges. If you geo-restrict to one country, block datacenter ASNs wholesale, or serve a consent interstitial to EU IPs before any content renders, the crawler gets the interstitial and nothing else. Test with curl from a VPS in another region, or check your firewall's country and ASN rules against the crawler IP ranges each vendor publishes.

Login walls and soft paywalls. Middleware that redirects unauthenticated requests to /login redirects crawlers too — and if the matcher is broad, it can swallow your marketing pages along with your app. Same story for content that exists in the HTML but sits under a "sign up to read" overlay: the crawler is served text your visitors can't see, which is exactly the mismatch engines discount.

One more, less dramatic: redirect chains. http → https → www → trailing slash is four hops, and a canonical tag pointing at a URL that 404s is worse. Crawlers give up sooner than browsers do.

All four are what an AEO scanner automates — the same probes, run per engine, so you see which specific bot gets stopped where.

Step 4: Check whether your content is worth citing

Access gets you read. Structure gets you quoted. Answer engines lift passages, so audit your key pages for:

• A plain definition near the top — "X is Y" in the first hundred words
• Question-style headings with direct answers underneath
• Lists and tables — the most liftable content formats
• Schema.org markup — Article, FAQPage, and Organization schema give models machine-readable grounding
• Trust signals — visible author, publish date, and an about page

The full playbook is in our AEO explainer, and the structural overlap with classic search is covered in SEO vs AEO.

Step 5: Automate it — because this breaks silently

Everything above can regress with one robots.txt edit, one WAF update, or one framework migration — and nothing will alert you. There is no Search Console for ChatGPT.

CheckVibe's AEO scanner runs all of these tests automatically:

• Per-engine access matrix — we probe your site as ChatGPT, Claude, Perplexity, Google AI, Copilot, Meta AI, and Mistral crawlers and show exactly who gets in and who gets blocked
• 46 AEO checks across access, extractability, readability, structured data, and trust signals
• 68 SEO checks in the same pass — because classic indexability feeds both
• A fix prompt on every finding — paste into Claude, Cursor, or Windsurf, redeploy, rescan

Run a free scan and you'll know in about a minute whether the AI engines your customers use can actually see you.

FAQ

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot is OpenAI's training crawler — blocking it keeps your content out of future model training and nothing else. ChatGPT's search feature is served by a different agent, OAI-SearchBot, with on-demand fetches handled by ChatGPT-User. Leave those two allowed and you can still be surfaced and linked inside ChatGPT answers. The real cost of blocking GPTBot is subtler: the model carries no baked-in knowledge of you for the times search doesn't fire.

Why does ChatGPT say it can't access my website?

Four causes, in rough order of likelihood: robots.txt disallows the fetching agent; your WAF or Cloudflare Bot Fight Mode returns a 403 or a JavaScript challenge to non-browser traffic; the page sits behind a login or is geo-restricted away from the crawler's datacenter IPs; or the URL lands in a redirect loop. Reproduce it in one line — curl -A "OAI-SearchBot/1.0" -I https://yourdomain.com — and whatever status code comes back is your answer.

Does JavaScript rendering stop AI crawlers?

Usually, yes. Most AI crawlers fetch your HTML and parse it without running scripts, so anything injected client-side simply isn't there for them. Googlebot does render JavaScript, which is why a site can rank well on Google and stay invisible to ChatGPT and Perplexity. Check it with curl -s https://yourdomain.com | grep "your headline" — if your text isn't in the raw response, no answer engine has ever read it.

How do I know if ChatGPT has crawled my site?

Check your server logs for the user agents GPTBot, OAI-SearchBot, or ChatGPT-User with OpenAI's published IP ranges. No hits in weeks usually means you're blocked at robots.txt or WAF level — or your site has too little linked, extractable content to attract crawls.

Why does my site rank on Google but never get cited by AI?

Because Googlebot renders JavaScript and is whitelisted by every WAF, while AI crawlers mostly don't render JS and are frequently blocked by default bot rules. Your site can be page one on Google and a blank 403 to GPTBot simultaneously. Test each engine separately.

Should I block AI crawlers instead?

If you sell content itself (journalism, research, courses), blocking training crawlers can be rational. For products and services, blocking answer engines mostly means your competitors get recommended instead of you. You can also split the difference: allow search-time crawlers like OAI-SearchBot while blocking training-time crawlers like GPTBot.

What is llms.txt and do I need it?

llms.txt is an emerging convention — a markdown file at /llms.txt that gives language models a curated index of your site's most important content. It's cheap to add and several tools already read it. It complements rather than replaces robots.txt: robots.txt controls access, llms.txt aids understanding.

How long until AI engines pick up my fixes?

Search-time crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) re-fetch popular pages within days, so an access fix can show up that fast. Training-data inclusion moves on model-release timescales — months. Re-run the ChatGPT and Perplexity prompts about a week after you deploy the fix; if the answer is still wrong, you have a second blocker, not a patience problem.