Your real estate website can look perfectly normal and still block the automated visitors used by AI search. When that happens, your neighbourhood pages, market updates, buyer guides and seller advice may never be considered for an answer.
Different agents, different purposes
AI services use distinct automated agents for different purposes. Some, including GPTBot and ClaudeBot, may collect public content for model development. Others, including OAI-SearchBot, Claude-SearchBot and PerplexityBot, help discover current pages for search. User-directed agents may also retrieve a page when someone asks a specific question. These controls are not always the same: robots.txt can allow one agent and block another, while CDN bot protection, web application firewalls and rate limits can override those instructions and block access without the site owner's knowledge.
In Plain English
AI search systems use automated visitors to find and retrieve public webpages. Your site can allow or refuse those visitors, and the settings are usually hidden in places most business owners never see.
Here is the part that surprises people: there are different kinds of AI visitor, and they do different jobs.
One kind collects material that may help train future models. Some businesses choose to block that, and it can be a reasonable choice.
Another kind helps AI search find current information or retrieves a page when a user asks a specific question. If that visitor is blocked, your useful real estate content may never be considered for the answer.
A lot of websites block both by accident. Security plugins, CDN settings and “block all bots” rules can treat every automated visitor as a threat. The result is a website that looks open to buyers and sellers but is quietly closed to AI.
Three things worth checking this week
Your robots.txt file
This file gives instructions to automated visitors. A broad rule can block more AI tools than intended.
Your CDN or firewall rules
Website security can override robots.txt and stop an AI visitor before it ever reaches the page.
Your main page content
Important information should not depend entirely on JavaScript, sliders or clicks. Some automated visitors may receive an incomplete version of the page.
If you are not sure how to check these items, that is normal. Most real estate agents will never work with these settings directly. The important point is simple: good content cannot be selected if AI cannot reach it.
Which AI visitors are which?
This section is optional. The names below explain why a single “allow AI” or “block AI” switch can be misleading.
- OpenAI separates
OAI-SearchBot, which supports ChatGPT Search, fromGPTBot, which may collect content for model improvement.ChatGPT-Useris used for certain user-initiated visits, and robots.txt may not apply in the same way. - Anthropic publishes
ClaudeBotfor model development,Claude-SearchBotfor search, andClaude-Userfor user-directed retrieval. - Perplexity publishes
PerplexityBotfor search results andPerplexity-Userfor user-requested page visits. Perplexity says these settings work independently. - Google handles AI features in Search through Googlebot.
Google-Extendedis a separate control for some other Google AI training and grounding uses; it is not the crawler that determines access to AI Overviews or AI Mode.
Robots.txt is only one layer. CDN, hosting and firewall rules can still block an agent that appears to be allowed.
Official references: OpenAI crawlers · Anthropic crawlers · Perplexity crawlers · Google AI features and websites · Cloudflare WAF guidance
The Takeaway
Your goal is not to allow every bot. It is to make an intentional choice and avoid accidentally blocking the AI search tools that could discover your public real estate content.
Can AI Actually Read Your Pages?
Getting in is not the same as being understood.
Leave A Comment