Skip to content
html2rss logo - Turn any website into RSShtml2rss

Capturing Feed Configs

html2rss capture analyzes a page through the auto-source pipeline and prints a reusable YAML draft: an items selector plus enhance: true. At feed-build time, enhance: true fills missing article fields from each matched card.

When to Capture

Section titled “When to Capture”

Use capture when:

  • you are writing a new custom feed and want a starting YAML
  • auto-source finds articles, but you want durable selectors for review
  • an AI agent or MCP client should produce a config before validate / apply

Prefer automatic feed generation when you only need items now and do not need a saved config.

CLI

Section titled “CLI”

Print YAML to stdout:

Terminal window
html2rss capture https://example.com/articles
# Pin Botasaurus for JS-rendered listings
BOTASAURUS_SCRAPER_URL="http://localhost:4010" \
html2rss capture https://example.com/articles --strategy botasaurus
# Hint the item card when auto detection is weak
html2rss capture https://example.com/articles --items_selector ".post-card"
# Analyze a local HTML fixture
html2rss capture https://example.com/articles --input ./page.html
# Quality JSON on stderr; YAML stays on stdout
html2rss capture https://example.com/articles --explain
# Save the draft
html2rss capture https://example.com/articles > my-feed.yml

Common options:

  • --strategy — auto, faraday, botasaurus, or local_file (default auto)
  • --items_selector — CSS selector hint for item cards
  • --limit — maximum articles kept while deriving selectors (default 25)
  • --max-redirects / --max-requests — request budget overrides
  • --input — local HTML file (sets local_file strategy)
  • --explain — print capture quality JSON on stderr (articles_count, channel_title, has_selectors, segment_strategy, selected_strategy, admission_drops); YAML stays on stdout

See the CLI reference for the full flag list.

Ruby API

Section titled “Ruby API”
require 'html2rss'
# Derive a config hash (channel + items selector with enhance: true)
config = Html2rss.capture('https://example.com/articles')
# Pin strategy or provide an items hint
config = Html2rss.capture(
  'https://spa-site.com',
  strategy: :botasaurus,
  items_selector: '.article-card'
)
File.write('my-feed.yml', Html2rss::Config.to_yaml(config))
feed = Html2rss.feed(config)

How It Works

Section titled “How It Works”
  1. Request — FeedPipeline (AutoFallback when :auto)
  2. Discover — AutoSource extracts admitted articles
  3. Segment — SST Segmenter strategies :list → :cluster → :semantic
  4. Gate — emit an items selector only when enough articles match
  5. Assemble — { items: { selector:, enhance: true } } plus channel. When AutoFallback selects a concrete transport (or you pin one), Capture stamps strategy: into the YAML so later html2rss apply / Html2rss.feed replay the same transport.

When the quality gate fails, selectors are omitted (has_selectors: false) rather than inventing attribute selectors. Hint with --items_selector or refine by hand.

What Capture Derives

Section titled “What Capture Derives”

Capture focuses on:

  • channel.url (and related channel defaults)
  • selectors.items with enhance: true

It does not invent per-field title/url/description selectors, author, published_at, categories, or enclosure selectors. enhance: true fills missing article fields via the HTML article extractor at feed-build time.

MCP capture returns that YAML in payload.yaml. validate / apply accept the YAML string (XOR a config hash). apply is isError when the feed has zero items (payload.item_count).

Section titled “Recommended Follow-Up”
  1. Validate: html2rss validate my-feed.yml
  2. Render: html2rss apply my-feed.yml
  3. Tighten the items selector, strategy, or request.botasaurus options if needed
  4. For Feed Directory contributions, add directory.topics and keep enhance: true unless chrome leaks (see Creating Custom Feeds)
Section titled “Related”