Skip to content
html2rss logo - Turn any website into RSS html2rss

Handling Dynamic Content

Some websites load their content dynamically using JavaScript. Static fetch paths may not see this content reliably.

Solution

Section titled “Solution”

Use a browser-based extraction strategy when JavaScript-heavy pages do not work with default static fetching.

browserless is common for this workflow, and botasaurus is an alternate browser-based strategy when you run a Botasaurus scrape API.

Keep the strategy at the top level and put request-specific options under request:

strategy: browserless
request:
  max_redirects: 5
  max_requests: 6
  browserless:
    preload:
      wait_after_ms: 5000
channel:
  url: https://example.com/app
selectors:
  items:
    selector: .article
  title:
    selector: h2
  url:
    selector: a
    extractor: href

When to Use Browser-Based Extraction

Section titled “When to Use Browser-Based Extraction”

A browser-based extraction strategy is necessary when:

  • Content loads after page load - JavaScript fetches data from APIs
  • Single Page Applications (SPAs) - React, Vue, Angular apps
  • Infinite scroll - Content loads as you scroll
  • Dynamic forms - Content changes based on user interaction

Preload Actions

Section titled “Preload Actions”

For dynamic sites, rendering once is often not enough. Use request.browserless.preload to wait, click, or scroll before the HTML snapshot is taken.

Wait Before Capturing Dynamic Content

Section titled “Wait Before Capturing Dynamic Content”
strategy: browserless
request:
  browserless:
    preload:
      wait_after_ms: 4000

Click “Load More” Buttons

Section titled “Click “Load More” Buttons”
strategy: browserless
request:
  browserless:
    preload:
      wait_after_ms: 3000
      click_selectors:
        - selector: ".load-more"
          max_clicks: 3
          wait_after_ms: 250

Scroll Infinite Lists

Section titled “Scroll Infinite Lists”
strategy: browserless
request:
  browserless:
    preload:
      scroll_down:
        iterations: 5
        wait_after_ms: 200
      wait_after_ms: 2500

These preload steps can be combined in a single config when a site needs several interactions before all items appear.

Performance Considerations

Section titled “Performance Considerations”

Browser-based extraction is slower than default static HTTP fetching because it:

  • Launches a headless Chrome browser
  • Renders the full page with JavaScript
  • Takes more memory and CPU resources

Use static HTTP fetching for static content and switch to browser-based extraction when needed. See the Strategy Reference for concrete transports, defaults, and environment requirements.

Section titled “Related Topics”