Auto Source
The auto_source scraper automatically finds items on a page, so you don’t have to specify CSS selectors.
To enable it, add auto_source: {} to your configuration:
channel: url: https://example.comauto_source: {}How It Works
Section titled “How It Works”One fetch, then scrapers run in tiers. Within a tier, results merge. The pipeline stops when Cleanup has admitted enough articles (limit, default 25).
- Native syndication:
native_feed— discovers the site’s own RSS/Atom (rel=alternateplus common feed paths), or a feed the page advertises on another host, and parses it before HTML heuristics (see Native feeds). Direct feed entry URLs (Content-Type or body sniff) skip HTML AutoSource and parse immediately. - In-page structured:
schema,microdata,microformats2,json_state,xhr_articles - Follow-up IO:
wordpress_api,sitemap,meta_oembed - Heuristic:
semantic_html - Heuristic:
html— skipped when an earlier tier already admitted at least one clean article
What each scraper does:
native_feed: Prefer native RSS/Atom when present, including enclosures, durations and channel facts the feed states. Tallies appear asAutoSource::NativeFeedunder--explain.schema: Parses<script type="json/ld">tags containing structured data (e.g., Schema.org), extracting authors and publishers.microdata: Extracts HTML Microdata annotations (itemscope itemtype).microformats2: Parses Microformats2h-entrymarkup, extractingp-name,e-content,u-url,dt-published,p-author,p-category, andu-photo/u-featuredmedia.json_state: Walks in-page JSON (<script type="application/json">,window.__NEXT_DATA__,window.__NUXT__,window.STATE) for arrays withtitle/urlpairs.xhr_articles: Reuses JSON XHR/fetch bodies captured during browser execution (no extra request). It is empty when the selected fetch path does not capture browser responses.wordpress_api: Detects<link rel="https://api.w.org/">and pulls posts from the REST API. See WordPress API.sitemap: Locates XML sitemaps (<link rel="sitemap">,/sitemap.xml, or/robots.txt), filtering by priority and recency, with Google News tags (<news:news>).meta_oembed: OpenGraph/Twitter meta tags plus JSON oEmbed (<link rel="alternate" type="application/json+oembed">).semantic_html: Semantic HTML5 (<article>,<main>,<section>).json_document: When the response itself is JSON, runs alone (no HTML tiers): maps JSON Feed documents directly, otherwise walks the JSON fortitle/urlarrays.html: Repeated item structure (list / cluster / semantic segmenter). Last resort only.
After scrapers, Cleanup is the admission gate: merge, dedupe, hard-exclude non-article destinations, drop junk titles, trim to limit. Scoring ranks and demotes; it does not refill limit after Cleanup.
Drop reasons (counts) land on Html2rss::Status#admission_drops. Print them with html2rss scrape URL --explain (JSON on stderr; the feed stays on stdout). Prefer Html2rss.auto_feed_result in Ruby when you need that status object.
auto_source also prepends kicker/teaser labels to titles when present, and discovers channel-level author and image (favicon, touch-icon, or OpenGraph images) from the host page.
Ruby limit: and CLI --limit cap kept articles (default 25).
json_state / xhr_articles: both need discoverable arrays of hashes with clear title and url fields. Minified or obfuscated payloads, heavily encoded values, or blobs that require executing embedded functions are ignored. xhr_articles only sees JSON bodies captured during the current Botasaurus browser scrape (capped by the scrape API).
wordpress_api: depends on a public WordPress REST API root. The scraper fetches post records directly; it does not resolve category names or featured media metadata.
Native feeds
Section titled “Native feeds”The site’s own feed comes first. native_feed reads <link rel="alternate"> in the page head, then probes common feed paths on the same host.
A feed on another host (for example a podcast platform) is followed only when the page itself advertises it with <link rel="alternate">. Such a request still goes through the network guard and the request budget. An advertised http feed is fetched as https, and skipped if that fails. Common-path guesses never leave the page’s host.
Which feed wins. On a podcast page, audio-looking feeds are tried first. A page counts as a podcast page when podcast, episode or audio appears in its host, path or <title>, its og:type is music.* or video.*, or its JSON-LD names a PodcastSeries, PodcastEpisode or AudioObject. On any other page, audio feeds rank after the common-path guesses, so a news page that advertises only its podcast keeps its news feed or its page items.
Which items are kept.
- Items of a followed feed may link to the feed’s own domain as well as the page’s (a platform’s episode pages).
- An item without a link uses its
http(s)guid, else its audio/video file URL. An item with none of these is skipped. - An item whose URL is its own audio/video file is kept even when the file is hosted on a separate media host.
- Atom entries with no
alternatelink (or a link withoutrel) are dropped. - When the page repeats a feed item, the feed’s enclosure and image win.
A followed feed also supplies the channel’s author, itunes:category and explicit flag. Precedence: your channel config, then the feed, then the page’s <meta name="author">.
Media enclosures
Section titled “Media enclosures”auto_source attaches playable audio and video to items so podcast and video apps can use the feed. It reads these sources:
- Schema.org:
AudioObject,VideoObject,PodcastEpisodeandClipitems (standalone, in@graph, or nested asassociatedMedia,audioorvideoof an article). The file iscontentUrl;encodingFormat,contentSizeanddurationsupply type, length and duration.embedUrlis a player only: never an enclosure and never the item link. A media object with a player but no file is still an item, without an enclosure. - HTML:
<audio>and<video>(srcor<source>) and links to audio/video files (also.pdf,.zip,.tar.gz,.tgz) inside an item’s card. Links in page chrome such as navigation are ignored. A<video poster>becomes the item image when the card has no other image. - Feeds: RSS
<enclosure>,media:content(also inmedia:group), Atomrel="enclosure"links anditunes:duration.
Only absolute http(s) URLs count; data:, blob:, javascript:, ftp: and mailto: references are dropped.
What becomes the RSS enclosure: the item’s first audio or video file, else its first downloadable file (such as a PDF or ZIP). Watch pages, embed or player pages, stream playlists (HLS, DASH), Flash players and <iframe> embeds are never enclosures. html2rss does not fetch an embed page, so a card with only an embedded player stays an item without an enclosure.
Length and duration come only from what the source states (Schema.org contentSize and duration, a feed’s length, fileSize, duration or itunes:duration). An unknown length is 0; html2rss sends no HEAD requests. Duration is kept on audio and video only.
JSON Feed lists every downloadable file as an attachment, with duration_in_seconds when known and size_in_bytes only when positive.
Podcast tags
Section titled “Podcast tags”An RSS feed gets itunes: tags only when more than half of its items have an audio or video enclosure.
- Channel:
itunes:image,itunes:author, anditunes:categoryanditunes:explicitwhen known. - Items:
itunes:durationanditunes:image, when known.
Category and explicit are taken from a followed native feed only; they are never invented.
Each item description shows at most one media element: the audio/video enclosure as a player, else the item image. Nothing is added when the description already contains that file. A PDF is never rendered into the description.
Entry URL resolution
Section titled “Entry URL resolution”Under strategy: auto with auto_source and no custom selectors, a weak homepage or hub extract can trigger entry URL resolution before Botasaurus escalation. The pipeline probes same-origin listing and feed candidates, then sticky-rewrites the effective scrape URL only when a retry extract yields items.
Resolution runs when enabled, the page is not a blocked surface, and any of:
- fewer than 3 admitted articles
- a weak surface (
high_entropy_surface,app_shell,unsupported_surface) - at least half of the articles came from
native_feed
Inspect results with html2rss scrape URL --explain or Html2rss::Status#to_h:
| Field | Meaning |
|---|---|
entry_url |
Pasted / channel URL |
scrape |
Effective fetch URL after resolution |
entry_resolution |
{ applied, probe_count, reason } plus optional winner_score |
entry_resolution.applied: true means a winner was selected. The sticky rewrite still requires a successful retry extract.
Defaults: enabled: true, max_probes: 5. Set enabled: false to keep the pasted URL only.
Fine-Tuning
Section titled “Fine-Tuning”Scraper Options
Section titled “Scraper Options”channel: url: https://example.comauto_source: scraper: native_feed: enabled: true # default: true wordpress_api: enabled: false # default: true sitemap: enabled: true # default: true min_priority: 0.3 # default: 0.3 max_age_days: 30 # default: 30 meta_oembed: enabled: true # default: true schema: enabled: false # default: true microdata: enabled: true # default: true microformats2: enabled: true # default: true semantic_html: enabled: true # default: true json_state: enabled: false # default: true json_document: enabled: true # default: true; JSON response bodies only xhr_articles: enabled: true # default: true; uses Botasaurus-captured XHR JSON only html: enabled: true # default: true minimum_selector_frequency: 3 # default: 2 use_top_selectors: 3 # default: 5Unknown scraper names or options fail validation with an error naming the key, so a typo or a removed option (such as fallback_anchorless) is reported instead of silently ignored.
Entry Resolution Options
Section titled “Entry Resolution Options”channel: url: https://example.comauto_source: entry_resolution: enabled: true # default: true max_probes: 5 # default: 5; must be > 0Cleanup Options
Section titled “Cleanup Options”channel: url: https://example.comauto_source: cleanup: keep_different_domain: true # default: false keep_sponsored: true # default: falseFor detailed documentation on the Ruby API, see the official YARD documentation.
