Skip to content
html2rss logo - Turn any website into RSShtml2rss

Capturing Feed Configs

html2rss capture analyzes a page through the auto-source pipeline and prints a reusable feed config with derived CSS selectors. Use it when you want a first draft faster than hand-writing selectors from scratch.

When to Capture

Section titled “When to Capture”

Use capture when:

  • you are writing a new custom feed and want a starting YAML
  • auto-source finds articles, but you want durable selectors for review
  • an AI agent or MCP client should produce a config before validate / apply

Prefer automatic feed generation when you only need items now and do not need a saved config.

CLI

Section titled “CLI”

Print YAML to stdout:

Terminal window
html2rss capture https://example.com/articles
# Pin Botasaurus for JS-rendered listings
BOTASAURUS_SCRAPER_URL="http://localhost:4010" \
html2rss capture https://example.com/articles --strategy botasaurus
# Hint the item card when auto detection is weak
html2rss capture https://example.com/articles --items_selector ".post-card"
# Analyze a local HTML fixture
html2rss capture https://example.com/articles --input ./page.html
# Save the draft
html2rss capture https://example.com/articles > my-feed.yml

Common options:

  • --strategy — auto, faraday, botasaurus, or local_file (default auto)
  • --items_selector — CSS selector hint for item cards
  • --limit — maximum articles kept while deriving selectors
  • --max-redirects / --max-requests — request budget overrides
  • --input — local HTML file (sets local_file strategy)

See the CLI reference for the full flag list.

Ruby API

Section titled “Ruby API”
require 'html2rss'
require 'yaml'
# Derive a config hash (:channel and :selectors)
config = Html2rss.capture('https://example.com/articles')
# Pin strategy or provide an items hint
config = Html2rss.capture(
  'https://spa-site.com',
  strategy: :botasaurus,
  items_selector: '.article-card'
)
# Serialize with string keys (same wire form as hand-written YAML)
File.write(
  'my-feed.yml',
  YAML.dump(Html2rss::HashUtil.deep_stringify_keys(config))
)
# Use immediately
feed = Html2rss.feed(config)

How It Works

Section titled “How It Works”
  1. Request — fetches the page with the chosen strategy
  2. Discover — runs AutoSource to extract articles
  3. Analyze — normalizes the page into an SST document and maps segment positions back to articles
  4. Derive — builds CSS selectors from SST tag paths for items, title, link, and description
  5. Assemble — returns a config hash ready for YAML or Html2rss.feed

Capture segment discovery currently uses the list Segmenter strategy only (not AutoSource cluster/semantic heuristics). When the draft is weak, pass --items_selector or refine selectors by hand.

What Capture Derives

Section titled “What Capture Derives”

Capture focuses on:

  • channel.url (and related channel defaults)
  • selectors.items
  • selectors.title
  • selectors.url (derived href selector)
  • selectors.description when a distinct description root exists

It does not invent author, published_at, categories, or enclosure selectors. Add those manually when the page exposes them reliably.

Description is omitted when it would resolve to the invalid CSS selector . (item root equals description root).

Section titled “Recommended Follow-Up”
  1. Validate: html2rss validate my-feed.yml
  2. Render: html2rss feed my-feed.yml
  3. Tighten selectors, strategy, or request.botasaurus options if needed
  4. For Feed Directory contributions, add directory.topics (see Creating Custom Feeds)
Section titled “Related”