Letting Claude Make a Rails Site AI-Native

Over the past four days, Claude Code and I turned smarts.md from a smart-contract docs site built for humans into one built for AI agents.

Those two things sound like the same site. They're not. A site for humans optimizes for readability, search, and load time. A site for AI agents has to answer a different set of questions:

• Can AI crawlers find you at all, or does your default robots.txt lock them out?
• Can an AI skip HTML rendering and pull clean content directly?
• Can an AI call your tools without having to "read the docs" first?
• Does every page tell an AI exactly how to reference and query it?

Here are the six things I had Claude build. Each is independently useful, and you can drop each into your own site on its own.

1. Explicitly welcome AI crawlers in robots.txt

Most sites inherit a robots.txt from some template that implicitly denies any User-Agent it doesn't know. But after 2025, many AI crawlers are new User-Agents — GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended, Google-Extended, CCBot, meta-externalagent, and more. If you want them in, you have to Allow them explicitly.

The smarts robots.txt opens with a policy note, then lists User-agent + Allow: / for 12 crawlers covering OpenAI, Anthropic, Perplexity, Google Gemini, Apple, Common Crawl, Meta, and Cohere. The stance is blunt: this site exists to be read by AI.

2. llms.txt — a menu for LLMs

llms.txt is a convention proposed by llmstxt.org: drop a concise Markdown file at the site root that tells an LLM what this site is, how to use it, and where the key entry points are. It's your site's elevator pitch.

The smarts llms.txt is about 60 lines:

• A one-liner positioning statement (Live docs for every verified smart contract)
• How to use it from an AI agent (MCP endpoint, one-line setup command)
• URL patterns (/{chain}/{address} vs. friendly slug)
• MCP tools with one-sentence descriptions
• A curated contract list (giving the LLM "default answers")
• Scope and limits (which chains aren't supported, caching policy)

The insight: LLMs pay in tokens to read your docs, so they won't crawl the whole site. Hand them a distilled index and their hit rate goes up sharply.

3. .well-known/mcp.json — bet early

The MCP protocol hasn't standardized a discovery convention like /.well-known/openid-configuration yet. No public client crawls a fixed well-known path today.

But publishing the manifest is 30 lines:

def well_known_mcp
  response.set_header("Cache-Control", "public, max-age=3600")
  response.set_header("Access-Control-Allow-Origin", "*")
  render json: {
    name: "smarts",
    version: "0.1.0",
    description: "...",
    protocol_version: "2024-11-05",
    transports: [{ type: "sse", endpoint: "https://smarts.md/mcp/sse" }],
    capabilities: { tools: true, resources: false, prompts: false },
    tools: MCP_TOOLS.map { |t| { name: t[:name], description: t[:blurb] } }
  }
end

The moment a discovery convention lands, I don't have to change a line. And it's self-documentation for developers — anyone curious about my MCP can just curl smarts.md/.well-known/mcp.json.

The trick: the tools array is derived from the MCP_TOOLS constant, single source of truth. An existing structural test forces every app/tools/*_tool.rb to appear in that constant, so the manifest inherits the "new tool won't be forgotten" guarantee for free.

4. A .md variant: a WebFetch-friendly distillation of every page

This one's my favorite. Claude Code's WebFetch tool has to clean and extract content from arbitrary HTML — that's expensive in tokens and unreliable. But what if you hand it .md directly?

Rails' respond_to supports this natively:

# config/routes.rb
get ":slug(.:format)", to: "contracts#show",
    constraints: { slug: ContractSlugs::ROUTE_PATTERN, format: /html|md/ }

https://smarts.md/usdc-eth returns HTML; https://smarts.md/usdc-eth.md returns a 40-line Markdown: address, chain, classification, MCP endpoint, how to query via an AI agent, source links.

# USD Coin on Ethereum

- **Address:** `0xa0b8...`
- **Chain:** Ethereum
- **Classification:** ERC-20 Token

## Query via AI agent

- **MCP endpoint:** `https://smarts.md/mcp/sse`
- **Reference:** `usdc-eth`
- **Sample prompt:** "Tell me the current state of usdc-eth"

One fetch, all the structured content an AI agent wants, no DOM parser needed. Claude knocked this out fast — one controller, two .md.erb templates, (.:format) in routes, done.

5. An MCP card on every page: let users point their AI here

This one's the reverse discovery: some of my visitors are already using Claude Code / Cursor / Windsurf. They land on a contract page, and the next thing they want is "let my AI take a look at this."

So every contract page renders a card with:

• Reference: the curated slug (e.g. usdc-eth) or chain/address, with a copy button
• Sample prompt: "Tell me the current state of usdc-eth", with a copy button
• No AI wired up yet?: a pointer to mcp.smarts.md for the one-line setup

One click of friction between "I'm looking at this contract" and "I'm asking my AI about it." The card uses a DaisyUI card and a generic Stimulus copy controller — 50 lines of erb. A regression test locks down the data-copy-text-value attributes so a future refactor can't silently paste the wrong string onto users' clipboards.

6. Classic SEO, with schema.org done right

OpenGraph, Twitter Card, JSON-LD — these are the traditional search and social-card layer. Worth doing, because AI crawlers read them too.

The interesting choice is how JSON-LD should describe a smart contract. Claude went with a WebPage wrapping a SoftwareApplication:

{
  "@type": "WebPage",
  "about": {
    "@type": "SoftwareApplication",
    "name": "USD Coin",
    "applicationCategory": "SmartContract",
    "operatingSystem": "Ethereum",
    "identifier": "0xa0b8..."
  }
}

operatingSystem gets the chain name, identifier gets the address, additionalType gets the classification (e.g. "Uniswap V3 Pool"). schema.org has no dedicated SmartContract type, but SoftwareApplication with these fields is enough for an LLM to understand "this is a contract on Ethereum with a specific address."

The OG image is a 1200×630 summary_large_image; Twitter Card, breadcrumbs, softwareVersion, and license fields round it out.

How much work is all this?

The six items above came out of seven PRs over four days:

PR	Files	Size
feat/ai-crawler-discovery	2	~126 loc
feat/well-known-mcp-manifest	4	~129 loc
feat/markdown-contract-pages	6	~298 loc
feat/contract-mcp-card	3	~126 loc
feat/seo-meta-tags	8	~292 loc
feat/seo-enrichments	8	~189 loc
feat/og-card	—	—

Each is an independent PR with its own tests and commit message. That cadence is the highest-leverage thing about working with Claude — every PR is small enough to review at a glance, but stacked together the site grew a full AI surface area.

A copy-paste checklist

If you have a content-shaped site and want it AI-friendly, do this in order:

1. Allow the major AI crawlers explicitly in robots.txt
2. Write a /llms.txt (a distilled menu)
3. Add a .md variant to your core content pages
4. If you run an MCP server, publish /.well-known/mcp.json
5. Put an "ask your AI about this page" card on content pages
6. Fill in JSON-LD / OpenGraph with schema.org types that match your content

Most SEO engineers won't tell you the first five — they're not in the traditional SEO playbook. But from four days of data on my end, the AI traffic curve and the human traffic curve are two completely independent series. You have to serve both.