Skip to content
html2rss logo - Turn any website into RSShtml2rss

Strategy

The strategy key defines how html2rss fetches a website’s content.

  • auto (default): Tries concrete strategies in order: faraday -> botasaurus.
  • faraday: Makes a direct HTTP request. It is fast but does not execute JavaScript.
  • botasaurus: Delegates fetching to a Botasaurus scrape API. Included in the auto chain; requires BOTASAURUS_SCRAPER_URL when that tier runs (or when you pin strategy: botasaurus).
  • local_file: Reads HTML content directly from a local file on disk without making network requests.

strategy is a top-level config key. Request-specific controls live under request.

auto falls back to botasaurus when faraday errors or extracts zero items. Pin a concrete strategy when you need a specific transport (for example, forcing botasaurus directly).

auto (default)

Section titled “auto (default)”

The default strategy chain is:

faraday -> botasaurus

When the config uses auto_source (no custom selectors), each strategy attempt:

  1. Runs AutoSource scrapers, starting with native_feed for same-origin RSS/Atom (direct feed URLs parse without HTML AutoSource).
  2. May run entry URL resolution on a weak homepage/hub extract — probe same-origin listing/feed candidates and sticky-rewrite the scrape URL only when the retry yields items — before escalating to the next strategy.

Gates, defaults, and Status fields: Auto Source — Entry URL resolution.

Auto fallback shares one request budget across all strategy attempts. The baseline reserves a NativeFeed follow-up slot and, when entry resolution is enabled, max_probes probe slots plus one retry GET. Raise request.max_requests (or --max-requests) when builds exhaust the budget.

Under auto, Html2rss.feed_result(...).status exposes scrape telemetry: selected_strategy, attempt_count, strategy_attempts, and when resolution ran entry_url, scrape_url, and entry_resolution (see Managing Feed Configs). Auto fallback decisions are also visible at LOG_LEVEL=info (hidden at the default LOG_LEVEL=warn).

faraday

Section titled “faraday”

faraday is the standard static HTTP transport. When no custom User-Agent header is supplied in headers, html2rss sends html2rss/<VERSION> by default.

HTML is accepted when Content-Type includes text/html, or when the body sniffs as HTML (<!DOCTYPE html / <html) and the response is not JSON. Gzip, deflate, and brotli bodies are decoded even when Content-Encoding is missing; unlabeled brotli is tried for octet-stream.

botasaurus

Section titled “botasaurus”

botasaurus delegates page fetching to a Botasaurus scrape API endpoint. It runs as the second tier of auto, or when you pin strategy: botasaurus.

Requirements:

  • BOTASAURUS_SCRAPER_URL set to your Botasaurus scrape API base URL (for example http://localhost:4010)
  • pin strategy: botasaurus when you want to skip Faraday and force this transport directly

html2rss enforces local request policy preflight and a feed-build timeout budget. The Botasaurus scrape API splits total scrape time (queue, boot, navigate, wait) from work time (navigate, selector wait, and scroll after the browser is ready). Feed YAML wait_timeout_seconds is validated against the work cap (1..30 by default); the gem forwards request.botasaurus options to the scrape API without shrinking retries or waits to fit a remaining feed budget. Faraday transport timeout for POST /scrape is the lesser of the remaining feed-build budget and BOTASAURUS_SCRAPE_TIMEOUT_SECONDS plus a small buffer (default 45s scrape total, 47s transport cap).

When the scrape API reports a timed-out stage, Html2rss::RequestService::RequestTimedOut may include timeout_phase (queue, boot, or work) and the same value in the exception message (timeout_phase=<stage>). Transport-hop timeouts leave timeout_phase unset.

During the browser execution tier, the Botasaurus scrape API always captures JSON XHR/fetch response bodies (xhr_responses). html2rss forwards them as Response#captured_responses so AutoSource xhr_articles can extract articles without extra requests. The HTTP-request tier returns an empty capture list. Caps and filtering live in the scrape API (see its README); article-likeness filtering is a client concern.

The client speaks OpenAPI 2.0 ScrapeSuccess / ScrapeError envelopes. Success requires an html string. Extra request.botasaurus keys fail YAML validation.

Configuration

Section titled “Configuration”
channel:
  url: "https://example.com"
strategy: botasaurus
auto_source: {}
request:
  botasaurus:
    navigation_mode: auto
    max_retries: 2
    headless: false

Supported request.botasaurus options (unknown keys are rejected):

  • execution_mode (auto, request, browser; scrape-API default auto when omitted)
  • navigation_mode (auto, get, google_get, google_get_bypass, organic_get; default auto)
  • max_retries (0..3; omit to use the scrape-API default of 2)
  • wait_for_selector (string)
  • wait_timeout_seconds (integer 1..30; omit to use the scrape-API default of 15; counts against post-boot work budget on the scraper, not browser boot)
  • scroll (boolean)
  • block_images (boolean)
  • block_images_and_css (boolean)
  • block_trackers (boolean)
  • wait_for_complete_page_load (boolean)
  • headless (boolean, default false)
  • proxy (string)
  • user_agent (string)
  • window_size ({ width: 1920, height: 1080 } — both positive integers, required together)
  • lang (string, for example en-US)
  • headers (hash, forwarded to the Botasaurus scrape API alongside config-level headers)
  • cookies (hash)

Request Payload Shape

Section titled “Request Payload Shape”

Example scrape-API payload shape:

{
  "url": "https://example.com",
  "navigation_mode": "auto",
  "max_retries": 2,
  "headless": false
}

Command-Line Usage

Section titled “Command-Line Usage”
Terminal window
BOTASAURUS_SCRAPER_URL="http://localhost:4010" \
html2rss auto https://example.com/updates --strategy botasaurus
BOTASAURUS_SCRAPER_URL="http://localhost:4010" \
html2rss feed my_config.yml --strategy botasaurus

local_file

Section titled “local_file”

local_file parses content directly from a local file on disk. It is designed for offline testing, local fixtures, and CLI --input workflows.

Because no remote network connection is established, local_file skips remote network guards and SSRF preflight checks.

Configuration

Section titled “Configuration”
strategy: local_file
request:
  local_file_path: "tmp/page.html"
channel:
  url: "https://example.com/articles"
selectors:
  items:
    selector: ".article"
  title:
    selector: "h2"
  url:
    selector: "a"
    extractor: "href"

Request Structure

Section titled “Request Structure”

Use this split consistently:

  • strategy: selects auto, faraday, botasaurus, or local_file
  • headers: top-level headers shared by remote strategies (defaults to html2rss/<VERSION> User-Agent)
  • request.max_redirects: redirect limit for the request session (default: 5)
  • request.max_requests: total request budget for the whole feed build (default: 10)
  • request.total_timeout_seconds: maximum total wall-clock budget for the entire feed build
  • request.local_file_path: file path for local_file strategy
  • request.botasaurus.*: Botasaurus-only options

Example:

strategy: botasaurus
headers:
  User-Agent: "Mozilla/5.0 (compatible; html2rss/1.0)"
request:
  max_redirects: 5
  max_requests: 6
  total_timeout_seconds: 45
  botasaurus:
    navigation_mode: google_get_bypass
    wait_timeout_seconds: 15
channel:
  url: "https://example.com/app"
selectors:
  items:
    selector: ".article"
  title:
    selector: "h2"
  url:
    selector: "a"
    extractor: "href"

For detailed documentation on the Ruby API, see the official YARD documentation.