Skip to content
html2rss logo - Turn any website into RSS html2rss

Custom HTTP Requests

Some websites require custom HTTP headers, authentication, or specific request configurations to access their content. html2rss makes it easy to customize your requests to handle these scenarios.

When You Need Custom Headers

Section titled “When You Need Custom Headers”

You might need custom HTTP requests when:

  • APIs require authentication (Bearer tokens, API keys)
  • Websites block default user agents (need to appear as a real browser)
  • Content is behind login (session cookies, authorization headers)
  • Rate limiting (custom headers to identify your requests)
  • Content negotiation (specific Accept headers for different formats)

Basic Configuration

Section titled “Basic Configuration”

Add a headers section to your feed configuration:

headers:
  User-Agent: "Mozilla/5.0 (compatible; html2rss/1.0)"
  Authorization: "Bearer YOUR_API_TOKEN"
  Accept: "application/json"
channel:
  url: https://api.example.com/posts
selectors:
  items:
    selector: ".post"
  title:
    selector: "h2"

Common Use Cases

Section titled “Common Use Cases”

API Authentication

Section titled “API Authentication”

Many APIs require authentication tokens:

headers:
  Authorization: "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."
  X-API-Key: "your-api-key-here"

User Agent Spoofing

Section titled “User Agent Spoofing”

Some websites block requests that don’t look like real browsers:

headers:
  User-Agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
  Accept: "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
  Accept-Language: "en-US,en;q=0.5"
  Accept-Encoding: "gzip, deflate"

Content Type Negotiation

Section titled “Content Type Negotiation”

Request specific content types:

headers:
  Accept: "application/json"  # For JSON APIs
  Accept: "text/html"         # For HTML content
  Accept: "application/rss+xml"  # For RSS feeds

Custom API Headers

Section titled “Custom API Headers”

Some APIs require specific headers:

headers:
  X-Requested-With: "XMLHttpRequest"
  X-Custom-Header: "your-value"
  Content-Type: "application/json"

Dynamic Headers

Section titled “Dynamic Headers”

You can use dynamic parameters in headers for runtime values:

headers:
  Authorization: "Bearer {{api_token}}"
  X-User-ID: "{{user_id}}"

See our Dynamic Parameters guide for more details.

Testing Your Headers

Section titled “Testing Your Headers”

Test your configuration to ensure headers work correctly:

Terminal window
# Test with curl first
curl -H "Authorization: Bearer YOUR_TOKEN" https://api.example.com/posts


# Then test with html2rss
html2rss feed your-config.yml

Troubleshooting

Section titled “Troubleshooting”

Common Issues

Section titled “Common Issues”
  • 401 Unauthorized: Check your authentication headers
  • 403 Forbidden: Verify API keys and permissions
  • 429 Too Many Requests: Add rate limiting or different user agents
  • Empty responses: Some APIs require specific Accept headers

Debug Tips

Section titled “Debug Tips”
  1. Use browser developer tools to see what headers successful requests use
  2. Test with curl before configuring html2rss
  3. Check API documentation for required headers
  4. Enable debug logging to see what headers are being sent

Advanced Examples

Section titled “Advanced Examples”

GitHub API

Section titled “GitHub API”
headers:
  Authorization: "token YOUR_GITHUB_TOKEN"
  Accept: "application/vnd.github.v3+json"
  User-Agent: "html2rss/1.0"
channel:
  url: https://api.github.com/repos/owner/repo/issues

Reddit API

Section titled “Reddit API”
headers:
  User-Agent: "html2rss/1.0 by your-username"
  Accept: "application/json"
channel:
  url: https://www.reddit.com/r/programming.json
Section titled “Related Topics”

Need More Help?

Section titled “Need More Help?”