# robots.txt — fluentive.app
#
# TWO deliberate choices here. Both exist because a redundant "allow" once
# silently cancelled every rule in this file.
#
# 1) ONE group. Under the robots.txt standard a crawler obeys only the group
#    that best matches its own name and ignores every other group, including
#    "User-agent: *". This file used to carry a named group per bot (Googlebot,
#    Bingbot, GPTBot, ClaudeBot, ...) that said nothing but "Allow: /", so those
#    crawlers matched their own group and ignored every Disallow. The groups
#    were removed; the bots we welcome are listed as a comment at the bottom.
#
# 2) NO blanket "Allow: /". Full access is already the default for anything not
#    disallowed, so it grants nothing. Google and Bing resolve conflicts by
#    longest match, where "Disallow: /cdn-cgi/" correctly beats "Allow: /" — but
#    simpler crawlers, including several AI crawlers, use first-match-wins. To
#    those, a leading "Allow: /" matches every URL and kills every Disallow
#    below it. Do not add it back.
#
# If a specific crawler ever needs different rules, give it its own group AND
# repeat every Disallow from this one inside it, or it will crawl those paths.

User-agent: *

# Private app areas — no public content, nothing to index
Disallow: /app/
Disallow: /dashboard/
Disallow: /auth/
Disallow: /login/
Disallow: /register/
Disallow: /api/
Disallow: /internal/

# Cloudflare internal endpoints. Email Address Obfuscation rewrites plain-text
# addresses into /cdn-cgi/l/email-protection links, which 404 for crawlers.
# Blocking /cdn-cgi/ is Cloudflare's own documented recommendation.
# If Cloudflare Images is ever enabled, add "Allow: /cdn-cgi/image/" above this
# line — the longer path wins over the broader Disallow.
Disallow: /cdn-cgi/

# Crawlers explicitly welcome here — search, AI search and AI training alike.
# Listed for the record only; everything not disallowed above is already open.
#   Search      Googlebot, Bingbot, BingPreview, Applebot
#   AI search   OAI-SearchBot, PerplexityBot, Claude-SearchBot
#   AI crawl    GPTBot, ClaudeBot, anthropic-ai, Google-Extended,
#               FacebookBot, cohere-ai, CCBot

# --- Google ---
User-agent: Googlebot
Allow: /

# Controls Gemini / AI training usage
User-agent: Google-Extended
Allow: /

# --- OpenAI ---
User-agent: GPTBot
Allow: /

# OpenAI search indexer (separate from GPTBot)
User-agent: OAI-SearchBot
Allow: /

# --- Anthropic ---
User-agent: ClaudeBot
Allow: /

# Alternative Anthropic crawler name
User-agent: anthropic-ai
Allow: /

# --- Perplexity ---
User-agent: PerplexityBot
Allow: /

# --- Microsoft / Bing (Copilot) ---
User-agent: Bingbot
Allow: /
User-agent: BingPreview
Allow: /

# --- Apple ---
User-agent: Applebot
Allow: /

# --- Meta (optional / future AI use) ---
User-agent: FacebookBot
Allow: /

# --- Cohere ---
User-agent: cohere-ai
Allow: /

# --- Common Crawl (used by many LLM training pipelines) ---
User-agent: CCBot
Allow: /

Sitemap: https://fluentive.app/sitemap.xml