Selectors

The selectors scraper gives you fine-grained control over content extraction using CSS selectors.

A valid RSS item requires at least a title or a description.

Basic Configuration

At a minimum, you need an items selector to define the list of articles and a title selector for the article titles.

channel:
  url: "https://example.com"
selectors:
  items:
    selector: ".article"
  title:
    selector: "h1"

Automatic Item Enhancement

With enhance: true (the default), html2rss fills missing fields on each matched item from the card HTML: title, url, description, published_at, image, categories, and enclosures. Explicit selectors still win when they produce a value.

Description leftover text drops listing chrome (CTA lines such as “read more”, date-shaped chips, type chips, title echoes, section names). When the items selector is a heading or wrapping link with no date or description, extraction can climb to the parent card — it aborts if that parent looks like a listing (multiple headings or article links).

Capture emits items + enhance: true only. Keep enhance: true unless page chrome leaks into items.

selectors:
  items:
    selector: ".article"
    enhance: true # default: true

Item Ordering

You can control the order of items in your feed:

selectors:
  items:
    selector: ".article"
    order: "reverse" # Reverse the order of items (newest first)

Available options:

• "reverse": Reverses the order of items (useful when the website shows oldest items first)
• Default: Items appear in the order they are found on the page

Paginated Feeds

Configure selectors.items.pagination to fetch items across multiple pages. html2rss supports several pagination strategies to handle different website architectures.

Common Options

• strategy: The pagination strategy name (default: rel_next).
• max_pages: The total page budget for the item selector chain, including the initial page (default: 5).

If you specify pagination as a simple integer (e.g., pagination: 5), it defaults to the rel_next strategy with the specified max_pages.

---

Strategies

1. rel_next (Default)

Follows standard HTML next page relationships via <link rel="next"> or <a rel="next"> tags.

selectors:
  items:
    selector: "article"
    pagination: 5 # or { strategy: rel_next, max_pages: 5 }

2. custom_selector

Uses a custom CSS or XPath selector to locate the link pointing to the next page. It reads the element’s href attribute or text content as the next URL.

• selector (String, required): The CSS selector or XPath expression for the next-page element.

selectors:
  items:
    selector: "article"
    pagination:
      strategy: custom_selector
      selector: "a.next-page-link"
      max_pages: 5

3. url_template

Increments a page number in a query parameter or replaces a {page} token inside the URL template.

• param (String, default: page): The name of the query parameter representing the page number.
• start_page (Integer, default: 1): The initial page number.
• step (Integer, default: 1): The step value to increment by for each subsequent page.

If the initial URL contains the {page} placeholder, html2rss replaces {page} with the calculated page number. Otherwise, it appends/replaces the query parameter specified by param.

selectors:
  items:
    selector: "article"
    pagination:
      strategy: url_template
      param: "page"
      start_page: 1
      step: 1
      max_pages: 5

4. offset

Increments a numeric offset in a query parameter or replaces an {offset} token inside the URL template.

• param (String, default: offset): The name of the query parameter representing the offset.
• start_offset (Integer, default: 0): The initial offset value.
• increment (Integer, default: 20): The offset value to add for each subsequent page.

If the initial URL contains the {offset} placeholder, html2rss replaces {offset} with the calculated offset. Otherwise, it appends/replaces the query parameter specified by param.

selectors:
  items:
    selector: "article"
    pagination:
      strategy: offset
      param: "offset"
      start_offset: 0
      increment: 20
      max_pages: 5

5. json_cursor

Digs next-page URLs or cursor tokens directly from JSON response payloads.

Requires configuring either cursor_path or next_url_path:

• cursor_path (String): Dot-notation path to the cursor value in the JSON payload (e.g., meta.next_cursor). When present, html2rss appends/updates the query parameter specified by param (defaults to cursor) with the cursor value.
• next_url_path (String): Dot-notation path to the next page URL in the JSON payload (e.g., links.next).
• param (String, default: cursor): The query parameter name used with cursor_path.

selectors:
  items:
    selector: "items"
    pagination:
      strategy: json_cursor
      cursor_path: "meta.next_cursor" # or next_url_path: "links.next"
      param: "cursor"
      max_pages: 5

---

Behavior & Request Budgeting

• Auto-budgeting: The request budget (max_requests) is automatically calculated and adjusted based on your max_pages configuration so you do not need to manually configure request.max_requests under normal circumstances.
• System ceiling: The absolute request limit is capped at 10 requests per build. If max_pages exceeds the system request ceiling, it will be clamped to the ceiling, and a warning will be logged.
• Stop conditions: Pagination stops automatically if:
  • The configured max_pages is reached.
  • The request budget is exhausted.
  • A page yields 0 new items.
  • An already visited URL is encountered (loop protection).
  • A network or extraction error occurs.
• Request safeguards: All request safeguards apply to pagination (such as redirect limits, timeout boundaries, private-network denial, and response-size limits).

RSS 2.0 Selectors

While you can define any named selector, only the following are used in the final RSS feed:

RSS 2.0 Tag	html2rss Name	Notes
title	title
description	description
link	url
author	author
category	categories
guid	guid
enclosure	enclosure
pubDate	published_at
comments	comments	⚠️ Not currently implemented

Selector Options

Each selector can be configured with the following options:

Name	Description
selector	The CSS selector for the target element.
extractor	The extractor to use for this selector.
attribute	The attribute name (required for attribute extractor).
static	The static value (required for static extractor).
post_process	A list of post-processors to apply to the value.

Extractors

Extractors define how to get the value from a selected element.

• text: The inner text of the element (default).
• html: The outer HTML of the element.
• href: The value of the href attribute.
• attribute: The value of a specified attribute.
• static: A static value.

Post-Processors

Post-processors manipulate the extracted value.

• gsub: Performs a global substitution on a string.
• html_to_markdown: Converts HTML to Markdown.
• markdown_to_html: Converts Markdown to HTML.
• parse_time: Parses a string into a Time object.
• parse_uri: Resolves a relative URL against channel.url and returns the normalized URL string.
• sanitize_html: Sanitizes HTML to prevent security vulnerabilities.
• substring: Extracts a substring from a string.
• template: Creates a new string from a template and other selector values. Use %{self} for the current selector value.

Always use the sanitize_html post-processor for any HTML content to prevent security risks.

Advanced Usage

Categories

To add categories to an item, provide a list of selector names to the categories selector.

selectors:
  genre:
    selector: ".genre"
  branch:
    selector: ".branch"
  categories:
    - genre
    - branch

Custom GUID

To create a custom GUID for an item, provide a list of selector names to the guid selector.

selectors:
  title:
    selector: "h1"
  url:
    selector: "a"
    extractor: "href"
  guid:
    - url

Enclosures

Use the enclosure selector to attach media (audio, video, or other non-image files) to an item. The selector is wired into each article and rendered into the feed.

RSS <enclosure> uses the first non-image enclosure. Images stay on the item description (and on JSON Feed image / attachments) — they are not promoted into RSS <enclosure>. For podcast/media RSS, select a non-image resource with enclosure.

selectors:
  items:
    selector: ".post"
  title:
    selector: "h2"
  enclosure:
    selector: "audio"
    extractor: "attribute"
    attribute: "src"
    content_type: "audio/mp3"

---

For detailed documentation on the Ruby API, see the official YARD documentation.