Auto Source

The auto_source scraper automatically finds items on a page, so you don’t have to specify CSS selectors.

To enable it, add auto_source: {} to your configuration:

channel:
  url: https://example.com
auto_source: {}

How It Works

One fetch, then scrapers run in tiers. Within a tier, results merge. The pipeline stops when Cleanup has admitted enough articles (limit, default 25).

1. Native syndication: native_feed — discovers same-origin RSS/Atom (rel=alternate plus common feed paths) and parses them before HTML heuristics. Direct feed entry URLs (Content-Type or body sniff) skip HTML AutoSource and parse immediately.
2. In-page structured: schema, microdata, microformats2, json_state, xhr_articles
3. Follow-up IO: wordpress_api, sitemap, meta_oembed
4. Heuristic: semantic_html
5. Heuristic: html — skipped when an earlier tier already admitted at least one clean article

What each scraper does:

• native_feed: Prefer native RSS/Atom when present. Tallies appear as AutoSource::NativeFeed under --explain.
• schema: Parses <script type="json/ld"> tags containing structured data (e.g., Schema.org), extracting authors and publishers.
• microdata: Extracts HTML Microdata annotations (itemscope itemtype).
• microformats2: Parses Microformats2 h-entry markup, extracting p-name, e-content, u-url, dt-published, p-author, p-category, and u-photo / u-featured media.
• json_state: Walks in-page JSON (<script type="application/json">, window.__NEXT_DATA__, window.__NUXT__, window.STATE) for arrays with title/url pairs.
• xhr_articles: Reuses JSON XHR/fetch bodies captured during a Botasaurus browser scrape (no extra HTTP). Empty for Faraday and Botasaurus HTTP-request tiers.
• wordpress_api: Detects <link rel="https://api.w.org/"> and pulls posts from the REST API. See WordPress API.
• sitemap: Locates XML sitemaps (<link rel="sitemap">, /sitemap.xml, or /robots.txt), filtering by priority and recency, with Google News tags (<news:news>).
• meta_oembed: OpenGraph/Twitter meta tags plus JSON oEmbed (<link rel="alternate" type="application/json+oembed">).
• semantic_html: Semantic HTML5 (<article>, <main>, <section>), with fallback clustering for anchorless lists.
• html: Repeated item structure (list / cluster / semantic segmenter). Last resort only.

After scrapers, Cleanup is the admission gate: merge, dedupe, hard-exclude non-article destinations, drop junk titles, trim to limit. Scoring ranks and demotes; it does not refill limit after Cleanup.

Drop reasons (counts) land on Html2rss::Status#admission_drops. Print them with html2rss auto URL --explain (JSON on stderr; the feed stays on stdout). Prefer Html2rss.auto_feed_result in Ruby when you need that status object.

auto_source also prepends kicker/teaser labels to titles when present, and discovers channel-level author and image (favicon, touch-icon, or OpenGraph images) from the host page.

Ruby limit: and CLI --limit cap kept articles (default 25).

json_state / xhr_articles: both need discoverable arrays of hashes with clear title and url fields. Minified or obfuscated payloads, heavily encoded values, or blobs that require executing embedded functions are ignored. xhr_articles only sees JSON bodies captured during the current Botasaurus browser scrape (capped by the scrape API).

wordpress_api: depends on a public WordPress REST API root. The scraper fetches post records directly; it does not resolve category names or featured media metadata.

Entry URL resolution

Under strategy: auto with auto_source and no custom selectors, a weak homepage or hub extract can trigger entry URL resolution before Botasaurus escalation. The pipeline probes same-origin listing and feed candidates, then sticky-rewrites the effective scrape URL only when a retry extract yields items.

Resolution runs when enabled, the page is not a blocked surface, and any of:

• fewer than 3 admitted articles
• a weak surface (high_entropy_surface, app_shell, unsupported_surface)
• at least half of the articles came from native_feed

Inspect results with html2rss auto URL --explain or Html2rss::Status#to_h:

Field	Meaning
entry_url	Pasted / channel URL
scrape_url	Effective fetch URL after resolution
entry_resolution	{ applied, probe_count, reason } plus optional winner_score

entry_resolution.applied: true means a winner was selected. The sticky rewrite still requires a successful retry extract.

Defaults: enabled: true, max_probes: 5. Set enabled: false to keep the pasted URL only.

Fine-Tuning

Scraper Options

channel:
  url: https://example.com
auto_source:
  scraper:
    native_feed:
      enabled: true # default: true
    wordpress_api:
      enabled: false # default: true
    sitemap:
      enabled: true # default: true
      min_priority: 0.3 # default: 0.3
      max_age_days: 30 # default: 30
    meta_oembed:
      enabled: true # default: true
    schema:
      enabled: false # default: true
    microdata:
      enabled: true # default: true
    microformats2:
      enabled: true # default: true
    semantic_html:
      enabled: true # default: true
      fallback_anchorless: true # default: true
    json_state:
      enabled: false # default: true
    xhr_articles:
      enabled: true # default: true; uses Botasaurus-captured XHR JSON only
    html:
      enabled: true
      minimum_selector_frequency: 3 # default: 2
      use_top_selectors: 3 # default: 5
      fallback_anchorless: true # default: true

Entry Resolution Options

channel:
  url: https://example.com
auto_source:
  entry_resolution:
    enabled: true # default: true
    max_probes: 5 # default: 5; must be > 0

Cleanup Options

channel:
  url: https://example.com
auto_source:
  cleanup:
    keep_different_domain: false # default: true
    min_words_title: 4 # default: 3

---

For detailed documentation on the Ruby API, see the official YARD documentation.