How AI Assistants Read Your Website: Crawlers, robots.txt, Structured Data and llms.txt Explained

AI assistants read your website through named crawlers that fetch the page’s HTML, and each company publishes which crawler controls what: OpenAI’s OAI-SearchBot decides whether a site can appear in ChatGPT search, PerplexityBot surfaces sites in Perplexity, and Google’s AI Overviews use the normal Googlebot index with “no additional technical requirements”. So a site is AI-readable when robots.txt lets the search crawlers in, the key facts sit in plain HTML text, and the structured data matches what the page says. llms.txt is an unofficial proposal, not a requirement, and Google says no AI-specific files are needed.

What actually happens when an AI assistant looks at a website?

Picture a customer asking an assistant: “Is there a dog groomer in Didsbury that does hand-stripping, and what does it cost?” For a question about the present, most assistants do not rely on memory. They run a search, pick a few promising pages, fetch them, read the text and write an answer, often with links.

That gives three moments where your site can drop out. The assistant’s search index may never have stored your page, because its crawler was blocked. The fetch may fail or return little text, because the content is inside an image or only appears after scripts run. Or the page may be read but not trusted, because its facts are vague or contradict other sources. This article is about the first two, which are the technical ones. Our separate guide on getting recommended by AI assistants covers reputation, reviews and trust.

Which crawlers do AI assistants use, and what does each one do?

Each company identifies its software by a name, called a user agent, in its own documentation. The important point is that one company can run several crawlers with different jobs, and blocking one is not the same as blocking another.

Both OpenAI and Perplexity say robots.txt changes take about 24 hours to take effect. OpenAI also adds utm_source=chatgpt.com to links it sends, so you can see visits from ChatGPT in your analytics.

How should robots.txt be set up for AI search?

robots.txt is a plain text file at the root of your site, at yoursite.com/robots.txt. Crawlers that follow the standard read it before fetching pages. For a local business that wants to be found, a simple, open file is usually right: allow everything except private areas, and point to the sitemap.

A typical open file has three lines: “User-agent: *”, then “Disallow: /admin/” (or whatever your private area is), then “Sitemap: https://yoursite.com/sitemap.xml”. With that, every well-behaved crawler, including the AI search crawlers in the table, may read your public pages.

If you have decided you do not want your content used for model training, you can add a block for that crawler only, for example “User-agent: GPTBot” followed by “Disallow: /”. That is a different decision from search visibility. The mistake to avoid is blocking OAI-SearchBot or PerplexityBot while wanting to appear in their answers.

One more trap: robots.txt is not the only gate. Some hosting firewalls and security services block unfamiliar bots by default, before robots.txt is ever read. If your file looks open but you never appear as a source, ask your host whether AI crawlers are being challenged or blocked at the network level.

Why does plain HTML text matter so much?

A crawler fetches the page’s code. Whatever is written there as text is easy to read. Whatever lives elsewhere may be missed. Google can render JavaScript, though it treats rendering as an extra step. Not every other crawler documents the same ability, so the safe assumption is that key facts should not depend on scripts.

Common places small business sites hide their most useful facts:

• 1A price list or menu uploaded only as a photo or a PDF.
• 2Opening hours inside a decorative image or a social media embed.
• 3Service details inside tabs, sliders or pop-ups that load only when clicked.
• 4Contact details inside a map widget, with no written address on the page.
• 5Whole pages assembled by scripts after loading, with an almost empty page in the raw code.

A quick test: open your page, choose “View page source” in the browser, and search for your phone number, your main price and the name of your area. If they are not in the source, a crawler that does not run scripts will not see them either. Google’s own guidance for AI features says the same in plainer terms: keep important content available in text form.

Does structured data help AI assistants understand a business?

Structured data is a small block of code, usually in a format called JSON-LD, that labels facts on the page: this is the business name, this is the address, these are the opening hours. For local businesses, Google’s LocalBusiness documentation lists name and address as required, and recommends fields such as telephone, website, opening hours, price range and a map location given to at least five decimal places.

Two things to be precise about. First, Google says AI Overviews and AI Mode need no special schema markup; ordinary eligibility for Search is enough. Second, Google’s general guidance is that structured data should match the visible text. Markup that claims opening hours the page does not show, or reviews that do not exist on the page, is a problem rather than a shortcut. Think of structured data as a tidy label on information you already state, not a way to say extra things.

Are FAQ pages still useful now that FAQ rich results are gone?

Google stopped showing FAQ rich results in Search from 7 May 2026, after limiting them to government and health sites in 2023. So FAQ markup no longer produces the expandable questions under a search listing. Google also notes that unused structured data does not cause problems, so there is no need to rush to remove existing markup.

The visible FAQ text is a different matter. A page that states “Do you cover Chorlton? Yes, and there is no travel charge within 5 miles” is exactly the kind of short, self-contained answer that matches how people phrase questions to assistants. Nobody publishes proof that assistants prefer FAQ pages, so treat this as reasoning rather than a guarantee. But it costs little, and it helps human visitors too.

What is llms.txt, and do I need one?

llms.txt is a plain text file, placed at the root of a site, that summarises the site and links to its key pages in a format meant for language models. It was proposed by Jeremy Howard in September 2024. Its own website calls it “a proposal” that is “open for community input”. It is not a standard from any standards body.

Google’s documentation for AI features says you do not need any AI-specific text files to appear in AI Overviews or AI Mode. No major assistant we know of has published a commitment to read llms.txt as part of choosing sources. So you do not need one. If your provider adds it at no cost, it does no harm. If anyone charges you for “llms.txt optimisation”, you are paying for something with no documented effect.

A 10-point AI-readability check for your own site

• 1Open yoursite.com/robots.txt and confirm it does not disallow everything.
• 2Confirm OAI-SearchBot, PerplexityBot, Googlebot and Bingbot are not blocked by name.
• 3Ask your host whether its firewall blocks AI or unknown crawlers.
• 4View page source on the home page and find your phone number, area and a price in the text.
• 5Check that menus, price lists and hours exist as text, not only as images or PDFs.
• 6Confirm there is no “noindex” on pages you want found.
• 7Check LocalBusiness structured data with Google’s Rich Results Test, and compare it with the visible page.
• 8Make sure each service has its own page with a clear title and heading.
• 9Add or update a visible FAQ written from real customer questions.
• 10Search Google and Bing for your business name and check both have indexed your current pages.

What Adyft Sites includes

This is the part where we describe our own approach, so you can compare it with any provider. Adyft Sites are set up to be found on Google for your area and services and to be readable by AI assistants. Because we write every page from your answers, prices, hours and areas are written as text rather than left in photos or PDFs, and the sites are built to be fast on phones. The FAQ, Contact and Privacy pages are free on every plan. If you want to know how a specific technical item from the check above is handled for your site, ask us and we will tell you plainly.

None of that guarantees any assistant will name you; no one can promise that honestly. It removes the technical reasons it would not. Starter is $1,499 plus $69 a month, with every page written for you and live in under seven days. If you already have a site you like, the 10-point check above is free to run, and fixing what it finds may be all you need.

Sources: Google, AI features and your website — https://developers.google.com/search/docs/appearance/ai-features · OpenAI, overview of crawlers — https://developers.openai.com/api/docs/bots · OpenAI, publishers and developers FAQ — https://help.openai.com/en/articles/12627856-publishers-and-developers-faq · Perplexity, crawlers — https://docs.perplexity.ai/guides/bots · llms.txt proposal — https://llmstxt.org/ · Google, LocalBusiness structured data — https://developers.google.com/search/docs/appearance/structured-data/local-business · Google, FAQPage structured data — https://developers.google.com/search/docs/appearance/structured-data/faqpage