Skip to content
html2rss logo - Turn any website into RSShtml2rss

Handling Dynamic Content

Some websites load their content dynamically using JavaScript. Static fetch paths may not see this content reliably.

Solution

Section titled “Solution”

Default strategy: auto automatically tries faraday first, then falls back to botasaurus when BOTASAURUS_SCRAPER_URL is configured. This handles many JS-rendered listing pages without needing custom configuration.

When a site requires browser rendering or anti-bot bypass by default, you can explicitly set strategy: botasaurus and configure request controls under request.botasaurus:

strategy: botasaurus
request:
  max_redirects: 5
  max_requests: 6
  botasaurus:
    navigation_mode: google_get_bypass
    wait_for_selector: ".article-card"
    scroll: true
channel:
  url: https://example.com/app
selectors:
  items:
    selector: .article-card
  title:
    selector: h2
  url:
    selector: a
    extractor: href

When to Use Botasaurus Extraction

Section titled “When to Use Botasaurus Extraction”

Botasaurus extraction is ideal when:

  • Content loads after page load - JavaScript fetches data dynamically
  • Single Page Applications (SPAs) - React, Vue, Angular apps
  • Anti-bot interstitials - Cloudflare, Google search referrers, or challenge pages
  • Scroll-triggered content - Infinite scroll lists (scroll: true)

Botasaurus Request Controls

Section titled “Botasaurus Request Controls”

Configure browser actions under request.botasaurus:

Wait for Elements

Section titled “Wait for Elements”
strategy: botasaurus
request:
  botasaurus:
    wait_for_selector: ".content-loaded"
    wait_timeout_seconds: 15

Scroll for Infinite Content

Section titled “Scroll for Infinite Content”
strategy: botasaurus
request:
  botasaurus:
    scroll: true

Anti-Bot Bypass Navigation Modes

Section titled “Anti-Bot Bypass Navigation Modes”
strategy: botasaurus
request:
  botasaurus:
    navigation_mode: google_get_bypass
    max_retries: 2

JSON Loaded Over XHR

Section titled “JSON Loaded Over XHR”

When Botasaurus uses the browser tier, captured JSON XHR/fetch bodies feed AutoSource xhr_articles automatically (enabled by default). Prefer strategy: botasaurus (or auto with BOTASAURUS_SCRAPER_URL) for SPA listing pages that hydrate article lists over the network rather than embedding them in HTML. See Auto Source and Strategy.

Performance Considerations

Section titled “Performance Considerations”

Browser-based extraction uses more resources than static HTTP fetching because it:

  • Renders pages in an automated Chrome instance
  • Executes JavaScript and handles DOM events
  • Manages browser pools and network emulation

Use static HTTP fetching (faraday) for static content, and lean on auto or explicit botasaurus strategies when browser rendering is required. See the Strategy Reference for details.

Section titled “Related Topics”