/ Research · AI search · October 2026
Is the IBEX 35 ready for AI search?
I read the robots.txt of Spain’s 35 largest listed companies, then knocked on every front door pretending to be Chrome, Googlebot, GPTBot, ClaudeBot and friends. Same machine, same IP, only the name changes. To tell an AI block from a generic bot block, I invented a control group: PlaceBot, the placebo bot.
What is not on this page: nothing exploitable. Which bots a public page lets in is visible to any visitor. A certificate-chain finding from 4 October did not reproduce on 6 October, so it was dropped.
/ In short
The numbers
/ The carousel
The LinkedIn carousel
The LinkedIn version. Click to enlarge.
/ 01 · The basics
What “ready for AI” means
It is not having an llms.txt. It is classic SEO: a bot can get in, read the page without JavaScript, understand it and find it. Plus a conscious decision about AI bots, which come in two kinds:
| Kind | Reads your site to | Examples | If you block it |
|---|---|---|---|
| Training | learn (train models) | GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider | Not used for training. A legitimate choice. |
| Search | cite you when someone asks | OAI-SearchBot, Claude-SearchBot, PerplexityBot | You disappear from those answers. |
| User | open a page a person asked for | ChatGPT-User, Claude-User, Perplexity-User | That person cannot read you from the chat. OpenAI notes robots.txt may not apply. |
Google-Extended is not a crawler: it is a robots.txt token for Gemini training and does not affect Google Search or AI Overviews, which use regular Googlebot. Block Googlebot or Bingbot and you also leave the AI features built on them.
/ 02 · What they say
robots.txt: the sign on the door
robots.txt is a polite note, read with the RFC 9309 rules: if a bot has no group of its own, the User-agent: * group applies. Inditex has none (it redirects to the home page), so 34 files.
| Company | Site | Names AI bots | Blocks | llms.txt | JSON-LD | Words no JS / JS |
|---|---|---|---|---|---|---|
| Banco Santander | www.santander.com | · | · | no | · | 1,326 / 986 |
| Inditex | www.inditex.com | no robots.txt | · | no | · | 371 / 339 |
| Iberdrola | www.iberdrola.com | · | · | yes | · | 0 / 819 |
| BBVA | www.bbva.com | · | · | no | 4 | 1,640 / 1,030 |
| CaixaBank | www.caixabank.com | · | · | no | 2 | 820 / 883 |
| ArcelorMittal | corporate.arcelormittal.com | · | · | no | · | 992 / 729 |
| Endesa | www.endesa.com | · | · | yes | · | 3,566 / 1,534 |
| Aena | www.aena.es | · | · | no | · | 1,491 / 678 |
| Ferrovial | www.ferrovial.com | · | · | no | 1 | 754 / 158 |
| Repsol | www.repsol.com | · | · | yes | 3 | 1,640 / 796 |
| Naturgy | www.naturgy.com | · | · | no | 6 | 1,428 / 787 |
| ACS | www.grupoacs.com | · | · | no | · | 1,098 / 853 |
| IAG | www.iairgroup.com | · | · | no | · | 452 / 381 |
| Amadeus | amadeus.com | · | · | yes | · | 1,595 / 1,164 |
| Telefónica | www.telefonica.com | · | · | no | 4 | 944 / 723 |
| Banco Sabadell | www.grupbancsabadell.com | · | · | no | · | 1,112 / 399 |
| Cellnex | www.cellnex.com | · | · | no | 6 | 2,993 / 1,310 |
| Bankinter | www.bankinter.com | ChatGPT-User GPTBot | · | yes | 4 | 8,963 / 747 |
| Mapfre | www.mapfre.com | · | · | no | 4 | 2,512 / 943 |
| Acciona | www.acciona.com | · | · | yes | · | 2,178 / 410 |
| Indra | www.indragroup.com | Bytespider ClaudeBot ChatGPT-User | Bytespider | no | 3 | 823 / 780 |
| Puig | www.puig.com | · | · | no | 27 | 256 / 313 |
| Unicaja | www.unicajabanco.es | · | · | no | 1 | 1,431 / 1,361 |
| Redeia | www.redeia.com | · | · | no | · | 1,785 / 733 |
| Merlin Properties | www.merlinproperties.com | · | · | yes | 5 | 341 / 218 |
| Acciona Energía | www.acciona-energia.com | · | · | no | · | 509 / 260 |
| Grifols | www.grifols.com | · | · | no | · | 1,616 / 1,061 |
| Logista | www.logista.com | ChatGPT-User GPTBot Google-Extended ClaudeBot PerplexityBot | · | yes | · | 543 / 483 |
| Enagás | www.enagas.es | · | · | yes | 3 | 1,243 / 398 |
| Acerinox | acerinox.com | · | · | no | · | 859 / 112 |
| Fluidra | www.fluidra.com | · | · | no | · | 0 / 1,543 |
| Sacyr | sacyr.com | · | · | no | 1 | 1,310 / 854 |
| Colonial | www.colonial-sfl.com | · | · | no | · | 385 / 303 |
| Rovi | rovi.es | · | · | no | · | 5,866 / 583 |
| Solaria | solariaenergia.com | · | · | no | · | 917 / 908 |
/ 03 · What they do
The door: same client, nine names
robots.txt is a request; the firewall is what lets a bot in or not. Each site was requested with the same client and IP, changing only the User-Agent. Doubtful cases ran twice in random order with 5-second pauses.
ok 200 with content · verif. 200 but a verification or near-empty page · 403 forbidden · 429 too many requests · empty: not tested. “Chrome” is the same client claiming to be Chrome, not a real browser.
| Company | CDN | Chrome | Googlebot | Bingbot | GPTBot | ClaudeBot | OAI-SearchBot | ChatGPT-User | PerplexityBot | PlaceBot |
|---|---|---|---|---|---|---|---|---|---|---|
| Banco Santander | Akamai + Imperva | ok | ok | ok | ok | ok | ok | ok | ok | |
| Inditex | Akamai | ok | ok | ok | ok | ok | ok | ok | ok | |
| Iberdrola | Akamai | verif. | 403 | 403 | 403 | 403 | 403 | 403 | 403 | 403 |
| BBVA | Akamai | ok | ok | ok | ok | ok | ok | ok | ok | |
| CaixaBank | Cloudflare | ok | 403 | 403 | ok | ok | ok | ok | ok | ok |
| ArcelorMittal | Cloudflare | 403 | 403 | 403 | 403 | 403 | 403 | 403 | 403 | |
| Endesa | Imperva | ok | ok | ok | ok | ok | ok | ok | ok | |
| Aena | Akamai | ok | ok | ok | ok | ok | ok | ok | ok | |
| Ferrovial | none seen | ok | ok | ok | 403 | ok | ok | 403 | ok | ok |
| Repsol | none seen | ok | ok | ok | ok | ok | ok | ok | ok | |
| Naturgy | Radware | verif. | verif. | verif. | verif. | verif. | verif. | verif. | verif. | verif. |
| ACS | none seen | ok | ok | ok | ok | ok | ok | ok | ok | |
| IAG | Cloudflare | ok | ok | ok | ok | ok | ok | ok | ok | |
| Amadeus | Imperva | ok | ok | ok | ok | ok | ok | ok | verif. | |
| Telefónica | Cloudflare | ok | ok | ok | ok | ok | ok | ok | ok | |
| Banco Sabadell | Akamai | ok | ok | 403 | 403 | 403 | 403 | ok | 403 | 403 |
| Cellnex | Imperva | ok | ok | ok | ok | ok | ok | ok | ok | |
| Bankinter | Cloudflare | 403 | 403 | 403 | 403 | 403 | 403 | 403 | ok | |
| Mapfre | none seen | ok | ok | ok | ok | ok | ok | ok | ok | |
| Acciona | none seen | ok | 403 | 403 | 403 | 403 | 403 | 403 | 403 | ok |
| Indra | Cloudflare | ok | ok | ok | ok | ok | ok | ok | ok | |
| Puig | Cloudflare | 403 | 403 | 403 | 403 | 403 | 403 | 403 | ok | |
| Unicaja | Akamai | ok | 403 | 403 | 403 | 403 | 403 | 403 | 403 | ok |
| Redeia | Imperva | ok | ok | ok | ok | ok | ok | ok | ok | |
| Merlin Properties | none seen | ok | ok | ok | ok | ok | ok | ok | ok | |
| Acciona Energía | none seen | ok | 403 | 403 | 403 | 403 | 403 | 403 | 403 | ok |
| Grifols | F5 | ok | ok | ok | ok | ok | ok | ok | ok | |
| Logista | none seen | ok | ok | ok | ok | ok | ok | ok | ok | |
| Enagás | none seen | ok | ok | ok | ok | ok | ok | ok | ok | |
| Acerinox | Radware | verif. | ok | ok | ok | ok | ok | ok | ok | |
| Fluidra | Imperva | verif. | verif. | verif. | verif. | verif. | verif. | verif. | verif. | |
| Sacyr | none seen | ok | 403 | ok | 403 | ok | ok | ok | ok | ok |
| Colonial | Cloudflare | ok | ok | ok | 403 | 403 | ok | ok | ok | ok |
| Rovi | none seen | ok | ok | ok | ok | ok | ok | ok | ok | |
| Solaria | none seen | ok | 429 | 429 | 429 | 429 | 429 | 429 | 429 | ok |
/ 04 · The placebo
PlaceBot, the control group
In medicine the control group takes a sugar pill. Here the drug is the bot’s name. PlaceBot is an honest, unknown bot: Mozilla/5.0 (compatible; PlaceBot/1.0; +https://www.linkedin.com/in/hache-cervera). Nobody has a rule for it.
If GPTBot is blocked and PlaceBot too, the site blocks any bot. If PlaceBot gets in but a fake Googlebot does not, the site blocks impersonators. Only if both get in is the rule really about that AI. Without the placebo, all three look like “they block AI”.
The twist: Bankinter and Puig block the fake Chrome and let PlaceBot in with full content. Lying about who you are closes more doors than being a nobody.
The rules look at the exact User-Agent: at Colonial, a bare “GPTBot” gets in; the full official string gets 403.
/ 05 · Findings
What stood out
- Iberdrola, the paradox. The biggest llms.txt in the index (2.4 MB, 7,291 links), and a door that returns 403 to every bot name, placebo included. Without JavaScript its home page has 0 words. Generic anti-bot, not anti-AI, but it says one thing with the file and another with the guard.
- Colonial does it by the book, silently. Blocks training bots (GPTBot, ClaudeBot), lets search bots in (OAI-SearchBot, ChatGPT-User, PerplexityBot). All in Cloudflare; robots.txt says nothing.
- Ferrovial probably filters “GPT”. GPTBot and ChatGPT-User fall; OAI-SearchBot passes. A hypothesis.
- Acciona, Acciona Energía, Unicaja. Every famous bot is blocked, the fake Googlebot too, and PlaceBot gets in: anti-impersonation, not anti-AI.
- Sabadell and Solaria. Sabadell blocks Bingbot and most “…Bot/” strings. Solaria answers 429 to any known bot from the first request.
- Bankinter’s llms.txt describes bankintercard, its consumer-credit subsidiary, not Bankinter.
- Indra is the only one blocking a bot in robots.txt: Bytespider (ByteDance). Logista names five AI bots to invite them.
/ 06 · llms.txt
llms.txt is a 2024 proposal
Proposed by Jeremy Howard in September 2024: a short Markdown file at /llms.txt with a title, a summary and key links, meant to fit in a model’s context. Google says you do not need “new machine readable files, AI text files, or markup” to appear in AI Overviews or AI Mode, and John Mueller has said Google does not use it. In a corporate site it is a curiosity, not a score.
/ 07 · Checklist
Is a site ready for AI search?
| Check | How |
|---|---|
| Get in (sign) | robots.txt does not block Googlebot, Bingbot or OAI-SearchBot, Claude-SearchBot, PerplexityBot. |
| Get in (guard) | curl.exe -s -o NUL -w "%{http_code}" -A "<full bot User-Agent>" https://site/, then the same with an invented bot. 403 versus 200 means a rule by name. |
| Read | DevTools, Disable JavaScript, reload. The content must still be there. |
| Understand | Clear titles and headings, answers in text, JSON-LD. |
| Find | Sitemap, internal links, indexed in Google and Bing. |
| Decide | Allow or block training bots knowingly; it does not change search visibility. |
| Measure | AI referrals in analytics, bot hits in server logs. |
/ 08 · Method
Security layers, and how I handled them
| Layer | What it does | How it was handled |
|---|---|---|
| CDN | Serves and filters traffic | Logged the provider, never read as intent |
| Bot management | Scores every visit | Counted words, not only status codes |
| Verified bots | Lets real crawlers in by IP | The limit: I test the name, not the IP |
| Anti-impersonation | Blocks fake famous bots | The placebo exposes it |
| Client fingerprint | Checks a “Chrome” behaves like Chrome | The fake Chrome is a baseline, not a user |
| JS challenge | Asks to run JavaScript first | Marked “verif.”, not an AI block |
| Rate limit | Slows fast clients | Pauses and random order |
Between 15 and 35 requests per site over two days, home page only. The limit: real crawlers come from verified IP ranges and may get a different answer. This shows that some rules look at the bot’s name, not what the real GPTBot sees.
Sources: Google: AI features and your website · OpenAI crawlers · llms.txt proposal · RFC 9309, robots.txt
/ Next
I check before I publish.
I wrote the scripts with help from Claude. The control groups, the second rounds and the findings I threw away were my call. If your team needs that kind of checking, email me.