Skip to content
html2rss logo - Turn any website into RSShtml2rss

Strategy

The strategy key defines how html2rss fetches a website’s content.

  • auto (default): Chooses from the available fetch paths based on scrape results. Use this for normal scraping.
  • default: Makes a direct request without executing JavaScript. Use it only when you need a fixed request mode.
  • botasaurus: Uses the companion scraper for browser rendering and browser-specific controls. Requires BOTASAURUS_SCRAPER_URL.
  • local_file: Reads HTML content directly from a local file on disk without making network requests.

strategy is a top-level config key. Request-specific controls live under request.

auto is the recommended interface: html2rss evaluates the scrape result and can use the configured companion scraper when needed. Pin a concrete strategy only for diagnosis, reproducibility, or strategy-specific controls.

auto (default)

Section titled “auto (default)”

When the config uses auto_source (no custom selectors), each strategy attempt:

  1. Runs AutoSource scrapers, starting with native_feed for same-origin RSS/Atom (direct feed URLs parse without HTML AutoSource).
  2. May run entry URL resolution on a weak homepage/hub extract — probe same-origin listing/feed candidates and sticky-rewrite the scrape URL only when the retry yields items — before escalating to the next strategy.

Gates, defaults, and Status fields: Auto Source — Entry URL resolution.

Auto fallback shares one request budget across all strategy attempts. The baseline reserves a NativeFeed follow-up slot and, when entry resolution is enabled, max_probes probe slots plus one retry GET. Raise request.max_requests (or --max-requests) when builds exhaust the budget.

Under auto, Html2rss.feed_result(...).status exposes scrape telemetry: selected_strategy, attempt_count, strategy_attempts, and when resolution ran entry_url, scrape_url, and entry_resolution (see Managing Feed Configs). Strategy selection is also visible at LOG_LEVEL=info (hidden at the default LOG_LEVEL=warn).

default

Section titled “default”

default is the fixed direct-request mode. It does not execute JavaScript. Keep auto unless you are diagnosing fetch behavior or must prevent browser rendering.

When no custom User-Agent header is supplied in headers, html2rss sends html2rss/<VERSION>. The shared request policy enforces redirect, request, timeout, and private-network safeguards and accepts supported compressed HTML responses.

botasaurus

Section titled “botasaurus”

botasaurus delegates page fetching to the companion scrape API. auto can use it when configured; pin strategy: botasaurus only when you need browser-specific behavior.

Requirements:

  • BOTASAURUS_SCRAPER_URL set to your Botasaurus scrape API base URL (for example http://localhost:4010)
  • pin strategy: botasaurus only when you need to force browser rendering or configure the controls below

html2rss enforces local request policy preflight and a feed-build timeout budget. The Botasaurus scrape API splits total scrape time (queue, boot, navigate, wait) from work time (navigate, selector wait, and scroll after the browser is ready). Feed YAML wait_timeout_seconds is validated against the work cap (1..30 by default); the gem forwards request.botasaurus options to the scrape API without shrinking retries or waits to fit a remaining feed budget. Transport timeout for POST /scrape is the lesser of the remaining feed-build budget and BOTASAURUS_SCRAPE_TIMEOUT_SECONDS plus a small buffer (default 45s scrape total, 47s transport cap).

When the scrape API reports a timed-out stage, Html2rss::RequestService::RequestTimedOut may include timeout_phase (queue, boot, or work) and the same value in the exception message (timeout_phase=<stage>). Transport-hop timeouts leave timeout_phase unset.

During the browser execution tier, the Botasaurus scrape API always captures JSON XHR/fetch response bodies (xhr_responses). html2rss forwards them as Response#captured_responses so AutoSource xhr_articles can extract articles without extra requests. The HTTP-request tier returns an empty capture list. Caps and filtering live in the scrape API (see its README); article-likeness filtering is a client concern.

The client speaks OpenAPI 2.0 ScrapeSuccess / ScrapeError envelopes. Success requires an html string. Extra request.botasaurus keys fail YAML validation.

Configuration

Section titled “Configuration”
channel:
  url: "https://example.com"
strategy: botasaurus
auto_source: {}
request:
  botasaurus:
    navigation_mode: auto
    max_retries: 2
    headless: false

Supported request.botasaurus options (unknown keys are rejected):

  • execution_mode (auto, request, browser; scrape-API default auto when omitted)
  • navigation_mode (auto, get, google_get, google_get_bypass, organic_get; default auto)
  • max_retries (0..3; omit to use the scrape-API default of 2)
  • wait_for_selector (string)
  • wait_timeout_seconds (integer 1..30; omit to use the scrape-API default of 15; counts against post-boot work budget on the scraper, not browser boot)
  • scroll (boolean)
  • block_images (boolean)
  • block_images_and_css (boolean)
  • block_trackers (boolean)
  • wait_for_complete_page_load (boolean)
  • headless (boolean, default false)
  • proxy (string)
  • user_agent (string)
  • window_size ({ width: 1920, height: 1080 } — both positive integers, required together)
  • lang (string, for example en-US)
  • headers (hash, forwarded to the Botasaurus scrape API alongside config-level headers)
  • cookies (hash)

Request Payload Shape

Section titled “Request Payload Shape”

Example scrape-API payload shape:

{
  "url": "https://example.com",
  "navigation_mode": "auto",
  "max_retries": 2,
  "headless": false
}

Command-Line Usage

Section titled “Command-Line Usage”
Terminal window
BOTASAURUS_SCRAPER_URL="http://localhost:4010" \
html2rss scrape https://example.com/updates --strategy botasaurus
BOTASAURUS_SCRAPER_URL="http://localhost:4010" \
html2rss apply my_config.yml --strategy botasaurus

local_file

Section titled “local_file”

local_file parses content directly from a local file on disk. It is designed for offline testing, local fixtures, and CLI --input workflows.

Because no remote network connection is established, local_file skips remote network guards and SSRF preflight checks.

Configuration

Section titled “Configuration”
strategy: local_file
request:
  local_file_path: "tmp/page.html"
channel:
  url: "https://example.com/articles"
selectors:
  items:
    selector: ".article"
  title:
    selector: "h2"
  url:
    selector: "a"
    extractor: "href"

Request Structure

Section titled “Request Structure”

Use this split consistently:

  • strategy: selects auto, default, botasaurus, or local_file
  • headers: top-level headers shared by remote strategies (defaults to html2rss/<VERSION> User-Agent)
  • request.max_redirects: redirect limit for the request session (default: 5)
  • request.max_requests: total request budget for the whole feed build (default: 10)
  • request.total_timeout_seconds: maximum total wall-clock budget for the entire feed build
  • request.local_file_path: file path for local_file strategy
  • request.botasaurus.*: Botasaurus-only options

Example:

strategy: botasaurus
headers:
  User-Agent: "Mozilla/5.0 (compatible; html2rss/1.0)"
request:
  max_redirects: 5
  max_requests: 6
  total_timeout_seconds: 45
  botasaurus:
    navigation_mode: google_get_bypass
    wait_timeout_seconds: 15
channel:
  url: "https://example.com/app"
selectors:
  items:
    selector: ".article"
  title:
    selector: "h2"
  url:
    selector: "a"
    extractor: "href"

For detailed documentation on the Ruby API, see the official YARD documentation.