Skip to content
html2rss logo - Turn any website into RSShtml2rss

Handling Dynamic Content

Some websites load their content dynamically using JavaScript. Static fetch paths may not see this content reliably.

Solution

Section titled “Solution”

Keep the default strategy: auto and configure BOTASAURUS_SCRAPER_URL. html2rss can then choose an available fetch path based on the scrape result, which handles many JavaScript-rendered listing pages without per-feed strategy configuration.

If a site still needs browser-specific navigation, waits, or scrolling, pin strategy: botasaurus and configure those advanced controls under request.botasaurus:

strategy: botasaurus
request:
  max_redirects: 5
  max_requests: 6
  botasaurus:
    navigation_mode: google_get_bypass
    wait_for_selector: ".article-card"
    scroll: true
channel:
  url: https://example.com/app
selectors:
  items:
    selector: .article-card
  title:
    selector: h2
  url:
    selector: a
    extractor: href

When to Use Botasaurus Extraction

Section titled “When to Use Botasaurus Extraction”

Botasaurus extraction is ideal when:

  • Content loads after page load - JavaScript fetches data dynamically
  • Single Page Applications (SPAs) - React, Vue, Angular apps
  • Anti-bot interstitials - Cloudflare, Google search referrers, or challenge pages
  • Scroll-triggered content - Infinite scroll lists (scroll: true)

Botasaurus Request Controls

Section titled “Botasaurus Request Controls”

Configure browser actions under request.botasaurus:

Wait for Elements

Section titled “Wait for Elements”
strategy: botasaurus
request:
  botasaurus:
    wait_for_selector: ".content-loaded"
    wait_timeout_seconds: 15

wait_timeout_seconds applies to selector wait on the scraper after the browser is ready. Slow Chrome or Xvfb boot consumes the scrape total wall (SCRAPE_TIMEOUT_SECONDS, default 45s), not this work budget (SCRAPE_WORK_TIMEOUT_SECONDS, default 30s).

Scroll for Infinite Content

Section titled “Scroll for Infinite Content”
strategy: botasaurus
request:
  botasaurus:
    scroll: true

Anti-Bot Bypass Navigation Modes

Section titled “Anti-Bot Bypass Navigation Modes”
strategy: botasaurus
request:
  botasaurus:
    navigation_mode: google_get_bypass
    max_retries: 2

JSON Loaded Over XHR

Section titled “JSON Loaded Over XHR”

Browser-rendered fetches can pass captured JSON XHR/fetch bodies to AutoSource xhr_articles automatically (enabled by default). With the companion scraper configured, keep auto for SPA listing pages that hydrate article lists over the network; pin botasaurus only when you need its browser-specific controls. See Auto Source and Strategy.

Performance Considerations

Section titled “Performance Considerations”

Browser-based extraction uses more resources than static HTTP fetching because it:

  • Renders pages in an automated Chrome instance
  • Executes JavaScript and handles DOM events
  • Manages browser pools and network emulation

Keep auto for normal use. Pin a concrete strategy only when diagnosing a site or configuring browser-specific behavior. See the Strategy Reference for details.

Section titled “Related Topics”