AI Agent Workflows with MCP

html2rss includes a native MCP server so agents can scrape pages, capture YAML, test, and ship RSS without parsing CLI stdout.

Why MCP

• Tools and schemas are discovered by the client.
• Every result is an envelope: ok, next_step, guidance, payload.
• html2rss://runtime reports whether Botasaurus is configured on this MCP process (botasaurus_configured only).

Full tool schemas: MCP Server Reference.

Client setup

Invoke through your version manager so Ruby and gem bins resolve correctly.

Cursor

~/.cursor/mcp.json or .cursor/mcp.json:

{
  "mcpServers": {
    "html2rss": {
      "command": "mise",
      "args": ["exec", "--", "html2rss", "mcp"],
      "env": {
        "BOTASAURUS_SCRAPER_URL": "http://127.0.0.1:4010"
      }
    }
  }
}

For asdf, use "command": "asdf" and "args": ["exec", "html2rss", "mcp"].

Claude Desktop

Same server block in Claude’s desktop config (~/Library/Application Support/Claude/claude_desktop_config.json on macOS).

Put BOTASAURUS_SCRAPER_URL in the MCP env block — a shell export does not reach the MCP process. Confirm with html2rss://runtime.

Golden loops

Articles now: scrape (or batch_scrape). strategy: "auto" already falls back to Botasaurus when configured — do not retry with explicit faraday after auto. Empty items can still be success; follow next_step.

Durable YAML: optional inspect → recon → capture → test → apply.

• capture returns draft YAML in payload.yaml.
• For html2rss-configs destinations, add directory.topics and explicit channel title / url.
• test asserts live extraction; then apply ships RSS (isError on zero items).

Config already in hand: validate → test → apply.

Weak scrape/capture → inspect. When alternates warrant it, follow next_step to recon.

html2rss-config skill

For curated configs in the html2rss-configs repo, use the repo skill at .agents/skills/html2rss-config/SKILL.md. It routes new / repair / expand modes and points at:

Helper	Role
scripts/add_config	Create from a URL or file
scripts/batch_recon	Parallel BUILD/DEFER/DROP ledger
scripts/check_config	CLI validate + test
scripts/probe_rss	Native RSS via recon

YAML lives under lib/html2rss/configs/<domain>/. Quality gates stay in that repo’s AGENTS.md.

Botasaurus for agents

Start the companion scraper (from html2rss or your web Compose stack), then point the MCP env at it:

# From the html2rss gem repo:
docker compose -f docker-compose.botasaurus.yml up -d

Typical URL: http://127.0.0.1:4010.

Daemon logs

html2rss mcp uses stdout for JSON-RPC and logs to stderr. Set LOG_LEVEL=debug for more detail or LOG_LEVEL=warn to quiet.