Generic extraction strategies

Sources persist a JSON strategy. Runtime behavior is determined only by this configuration—there are no hostname-specific parsers.

HTML

{
  "mode": "html",
  "request_url": "https://example.org/events",
  "method": "GET",
  "item_selector": ".event-card",
  "title_selector": ".title",
  "date_selector": ".date",
  "time_selector": ".time",
  "link_selector": "a.details",
  "link_attribute": "href",
  "description_selector": ".summary",
  "location_selector": ".venue",
  "date_format": ""
}

Selectors may read attributes using title_attribute, date_attribute, time_attribute, link_attribute, description_attribute, and location_attribute. Use :scope to read an attribute from the event container itself.

JSON API

{
  "mode": "json",
  "request_url": "https://example.org/api/sessions",
  "method": "GET",
  "items_path": "results.events",
  "title_path": "name",
  "date_path": "startDate",
  "end_path": "endDate",
  "description_path": "description",
  "location_path": "venue.address",
  "link_path": "url",
  "disabled_path": "disabled"
}

Paths use dot notation. Numeric path components index arrays.

Joining metadata

{
  "mode": "json",
  "request_url": "https://example.org/api/sessions",
  "items_path": "",
  "date_path": "utcDate",
  "link_path": "link",
  "join_item_key": "movieId",
  "join_title_path": "name",
  "join_description_path": "summary",
  "join_duration_path": "duration",
  "join": {
    "request_url": "https://example.org/api/movies",
    "method": "GET",
    "items_path": "",
    "key_path": "id"
  }
}

JSON containing HTML fragments

{
  "mode": "html_fragment",
  "request_url": "https://example.org/ajax",
  "method": "POST",
  "body": {"action": "events", "venue": "123"},
  "html_path": "html",
  "item_selector": ".event-card",
  "title_selector": ".title",
  "date_selector": ".date",
  "link_selector": "a",
  "link_attribute": "href",
  "page_parameter": "paged",
  "page_start": 1,
  "max_pages": 10,
  "total_pages_path": "pages_total"
}

Safety limits

Request URLs still pass public-network SSRF validation.
Redirect destinations are validated for normal HTML fetches.
Responses default to a 5 MB limit.
Pagination is capped at 20 pages.
Browser-compatible headers are global; strategy headers may add endpoint-specific Accept or Referer values.

2.4 KiB Raw Blame History