A website that AI can read: what it means for design and build

A page that looks full in a browser can reach a machine empty. It is not a coding mistake or a design mistake. It follows from decisions made at the start of the design, and nobody makes them with bad intent.

What we know about it

In December 2024 Vercel and the agency MERJ published measurements from their network: in one month they recorded 569 million requests from GPTBot and 370 million from the Anthropic crawler. None of the major AI crawlers ran JavaScript. They did download the scripts, but downloading and running are not the same thing.

The exception is Googlebot, which has a rendering service and also feeds Gemini. The others do not. This is data from one network, not a law, and crawler behaviour can change without notice. As a basis for decisions it is enough: what is not in the HTML from the server does not exist for most machines.

Four decisions that empty a page

• Text inside an image. A headline set in graphics keeps exact typography and looks better. A machine sees only a file.
• Content in tabs and accordions loaded after a click. Specifications, ingredients, shipping. Exemplary order in the browser, nothing in the HTML from the server.
• Text that appears only as you scroll. The animation is an effect, not a technique, but it is usually built so that a script inserts the paragraph into the page.
• A page built on video where the whole message runs in the video. Underneath it a two-word headline and a button remain.

None of those four is bad design in itself. We have designed three of them ourselves and we stand by them. The difference is whether the same message also exists in text that arrives from the server.

A good website looks right to a person and makes sense to a machine. These are not two opposing things, they are just thought about in a different order.

It does not mean building boring websites

The usual reaction is that this strips a site down to bare text. It does not. Animation, video and graphic typography can all stay. The point is to have a layer underneath them carrying the same message in plain text: the headline, the first paragraph, and for a product the price and availability.

In practice it means one extra question in the design. Wherever an image, a video or motion carries the message, we ask where the same thing is written in words. If there is no answer, either the text is added or we decide deliberately that the page does not matter.

How to check it in a minute

Open the source of your page with Ctrl+U and try to find a sentence of your own content in it. If it is not there, the machine does not see it either. Faster is the check of what AI sees of a site: you enter an address and see how many words of the page survive without scripts running, and how much of that is the menu and the footer.

It is covered in more detail in the article why a page full of text can be empty for AI, together with the sources for that measurement.

What follows for a brief

When you commission a new website, it is one sentence in the brief: the content we care about has to be in the HTML from the server. It sounds technical, but it is decided in the design, not in the coding. A layout built on gradually revealing content is then either built differently or given a text layer alongside it.

On the websites we build it is the default. The technical side is handled by the development arm of our group; we deal with what comes earlier: whether the page carries its message in words at all, or only in pictures.

A good-looking page that a machine reads as empty is not work well done. It is work where one question never came up.