# makerportal.ai — crawler posture
#
# POSTURE: ALLOW ALL AI CRAWLERS. Decided by the site owner on 2026-08-06 and
# recorded here rather than inferred from the wildcard. This is a deliberate
# position, not the residue of never having chosen one: MakerPortal publishes
# computed answers and wants them quoted, cited and linked by assistants as well
# as by search engines, and every page and markdown twin carries its own
# canonical URL and attribution line for exactly that purpose.
#
# The file was rewritten on 2026-08-05 to say so agent by agent. Before that the
# same permission was in force by default — `User-agent: *` / `Allow: /` at the
# top means every agent below already had access, including the ones that were
# never named — so the 2026-08-05/06 change is one of EXPLICITNESS and date, not
# of permission. A vendor that reads only its own token now sees a decision, and
# the next change here is a diff rather than an archaeology exercise.
#
# TO REVERSE IT, edit the blocks, not this comment: one `Allow: /` becomes
# `Disallow: /` per agent you are excluding, and the annotation above each block
# says what that agent actually does. The three Anthropic tokens and the three
# OpenAI tokens are three different behaviours each; blocking one is not
# blocking the others.
#
# The 2026-08-05 pass added the tokens that had fallen through the wildcard:
# OAI-SearchBot, Claude-User and Claude-SearchBot are the current OpenAI and
# Anthropic tokens (the file previously named only the DEPRECATED `Anthropic-ai`
# alongside ClaudeBot), plus the Google, Meta, Amazon, ByteDance, Mistral and
# Common Crawl peers.
#
# THE THREE ANTHROPIC TOKENS ARE THREE DIFFERENT THINGS and are listed
# separately on purpose: ClaudeBot crawls for training corpora, Claude-User
# fetches a page because a person asked about it in a conversation, and
# Claude-SearchBot indexes for search results. Blocking one is not blocking the
# others. The same split holds for GPTBot / ChatGPT-User / OAI-SearchBot.
#
# What this file cannot do: it is a request, not an enforcement, and it says
# nothing about training rights. The site's position on citation and reuse is
# stated on the pages themselves and in the markdown twins.

User-agent: *
Allow: /

# ── OpenAI ───────────────────────────────────────────────────────────────────
# GPTBot: training crawler. ChatGPT-User: user-initiated fetch.
# OAI-SearchBot: search indexing — added 2026-08-05.
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

# ── Anthropic ────────────────────────────────────────────────────────────────
# ClaudeBot: crawler. Claude-User: user-initiated fetch. Claude-SearchBot:
# search indexing. The last two were added 2026-08-05. `Anthropic-ai` and
# `Claude-Web` are deprecated tokens, kept so an old client is not surprised.
User-agent: ClaudeBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Anthropic-ai
Allow: /

User-agent: Claude-Web
Allow: /

# ── Google ───────────────────────────────────────────────────────────────────
# Google-Extended governs Gemini/Vertex grounding; it does not affect Search
# ranking either way. Googlebot itself is covered by the wildcard above.
User-agent: Google-Extended
Allow: /

User-agent: Google-CloudVertexBot
Allow: /

# ── Apple ────────────────────────────────────────────────────────────────────
User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Allow: /

# ── Perplexity ───────────────────────────────────────────────────────────────
User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

# ── Meta ─────────────────────────────────────────────────────────────────────
User-agent: Meta-ExternalAgent
Allow: /

User-agent: Meta-ExternalFetcher
Allow: /

User-agent: FacebookBot
Allow: /

# ── Microsoft / Bing ─────────────────────────────────────────────────────────
User-agent: bingbot
Allow: /

# ── Amazon, ByteDance, Mistral, Cohere, You.com, Common Crawl ────────────────
User-agent: Amazonbot
Allow: /

User-agent: Bytespider
Allow: /

User-agent: MistralAI-User
Allow: /

User-agent: cohere-ai
Allow: /

User-agent: cohere-training-data-crawler
Allow: /

User-agent: YouBot
Allow: /

User-agent: CCBot
Allow: /

# ── Disallowed: commercial SEO scrapers ──────────────────────────────────────
# Not an AI-posture decision. These crawl hard, attribute nothing, and resell
# the result; the site gets no citation and no reader from either.
User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: MJ12bot
Disallow: /

Sitemap: https://makerportal.ai/sitemap.xml
Host: https://makerportal.ai