Google said on its Search Central blog on October 30, 2026 that it plans to support a shared AI training control in robots.txt. The idea is simple: one line tells every participating AI company whether it may train models on your content, instead of a separate rule for each crawler. Microsoft announced the same day that Bing will adopt the shared training signal and add a separate setting for AI answers.

Search Engine Roundtable reported that OpenAI and Anthropic have signed on too. The control comes from the AI Preferences (AIPREF) working group at the Internet Engineering Task Force, the body that sets many internet standards.

Our view: this is good news for site owners, and there’s no rush. Google is aiming for 2027 and the syntax is still a draft. What you should do now is decide what you want, because training and AI answers are now two separate choices, and most small businesses will want different answers to each.

What changed

Google’s post, written by Search Advocate John Mueller, explains that websites will be able to add a shared “policy token” to robots.txt. If a site uses that token to opt out of training and doesn’t mention Google-Extended, Google plans to treat it as if the site had blocked Google-Extended. Google-Extended is the AI training control Google has offered since 2023.

A few details from Google’s announcement:

  • Existing company-specific rules keep working. You can set one broad rule and still set a different rule for Google alone.
  • Sites that allow Google-Extended can appear in Gemini app answers through grounding and receive clicks from them.
  • Google says the approach matches the EU’s Code of Practice for general-purpose AI models, which treats robots.txt as the main way publishers state machine-readable preferences.
  • Google aims to support the control in 2027, alongside other AI providers.

Microsoft’s Bing blog goes further. It splits AI use into two independent settings:

  • AI Training: whether your content may train generative models. Set it for a whole site or folder in robots.txt, or page by page with a meta tag or HTTP header.
  • AI Grounding: whether your content may inform AI answers in Bing, Copilot and Microsoft 365. This one is page level only. Options are Allow, Disallow, or a new middle setting that limits use to Microsoft’s own products and excludes the grounding services Microsoft sells to other companies.

Both default to Allow. Bing says neither setting changes eligibility for regular Bing search results. It also warns that blocking grounding will reduce your visibility in AI experiences.

Bing starts honoring the new settings on February 1, 2027. They replace the older NoArchive and NoCache tags for AI use. If your site relies on those tags to keep content out of AI, Bing says you need to publish the new preferences by March 1, 2027, and record a temporary grounding preference in Bing Webmaster Tools, which opens on December 15, 2026.

What it means for your business

For a typical service business, store or local company, the two choices point in different directions.

AI answers are where buyers now ask questions. When someone asks Copilot or Gemini for a plumber, an accountant or a product comparison, the answer is built from pages the system is allowed to use. Blocking grounding takes you out of that pool. For a business that wants leads, we’d almost never block it.

Training is a judgment call. Allowing training doesn’t send you traffic. Bing says plainly that neither setting promises inclusion, traffic or payment. If your site holds original research, paid content or material you license, opting out of training is reasonable. For a services site with standard pages about what you do, the content has little resale value, and we don’t see much downside in leaving it open.

Google Search is a separate matter. Google’s crawler documentation says Google-Extended doesn’t affect inclusion or ranking in Google Search, so blocking training with Google won’t touch your organic rankings. AI Overviews in Search are governed by Search’s own controls, which this announcement doesn’t change.

The quiet risk is old settings nobody remembers adding. It’s common to find robots.txt files that block AI crawlers because a plugin or a previous developer added rules in 2023. Some sites also carry noarchive tags set by an SEO or security plugin. Under Bing’s new rules, those old tags stop counting for AI after March 1, 2027, which could open up content an owner meant to protect, or keep blocking content an owner never meant to hide.

What we’d do this week

  1. Read your robots.txt. Open yoursite.com/robots.txt and list every rule that names an AI crawler, such as GPTBot, Google-Extended, ClaudeBot or CCBot. For each one, find out why it’s there.
  2. Check for noarchive and nocache. Look at the page source of your homepage and a key service page, and check your SEO plugin settings. If you rely on these tags to block AI use, put March 1, 2027 in your calendar.
  3. Decide training and answers separately. Write down two answers: may AI companies train on our content, and may AI assistants cite our pages in answers. For most businesses that want leads, the second answer is yes.
  4. Don’t add the new token yet. The syntax is an Internet-Draft and Google hasn’t published final documentation. Bing’s guide shows the current draft format, and we’ll update our clients’ files once Google confirms it.
  5. Keep your Bing Webmaster Tools account current. Bing’s temporary grounding setting lives there, and its AI Performance report shows where your pages are cited.

If you’d like us to review your robots.txt and AI settings as part of a technical SEO check, start with our free SEO audit. For more on earning citations, read our guide on how to rank in AI Overviews or see our SEO services. The primary source is Google’s Search Central announcement, with details in Bing’s announcement.