96 lines
2.4 KiB
Markdown
96 lines
2.4 KiB
Markdown
# Generic extraction strategies
|
|
|
|
Sources persist a JSON strategy. Runtime behavior is determined only by this configuration—there are no hostname-specific parsers.
|
|
|
|
## HTML
|
|
|
|
```json
|
|
{
|
|
"mode": "html",
|
|
"request_url": "https://example.org/events",
|
|
"method": "GET",
|
|
"item_selector": ".event-card",
|
|
"title_selector": ".title",
|
|
"date_selector": ".date",
|
|
"time_selector": ".time",
|
|
"link_selector": "a.details",
|
|
"link_attribute": "href",
|
|
"description_selector": ".summary",
|
|
"location_selector": ".venue",
|
|
"date_format": ""
|
|
}
|
|
```
|
|
|
|
Selectors may read attributes using `title_attribute`, `date_attribute`, `time_attribute`, `link_attribute`, `description_attribute`, and `location_attribute`. Use `:scope` to read an attribute from the event container itself.
|
|
|
|
## JSON API
|
|
|
|
```json
|
|
{
|
|
"mode": "json",
|
|
"request_url": "https://example.org/api/sessions",
|
|
"method": "GET",
|
|
"items_path": "results.events",
|
|
"title_path": "name",
|
|
"date_path": "startDate",
|
|
"end_path": "endDate",
|
|
"description_path": "description",
|
|
"location_path": "venue.address",
|
|
"link_path": "url",
|
|
"disabled_path": "disabled"
|
|
}
|
|
```
|
|
|
|
Paths use dot notation. Numeric path components index arrays.
|
|
|
|
### Joining metadata
|
|
|
|
```json
|
|
{
|
|
"mode": "json",
|
|
"request_url": "https://example.org/api/sessions",
|
|
"items_path": "",
|
|
"date_path": "utcDate",
|
|
"link_path": "link",
|
|
"join_item_key": "movieId",
|
|
"join_title_path": "name",
|
|
"join_description_path": "summary",
|
|
"join_duration_path": "duration",
|
|
"join": {
|
|
"request_url": "https://example.org/api/movies",
|
|
"method": "GET",
|
|
"items_path": "",
|
|
"key_path": "id"
|
|
}
|
|
}
|
|
```
|
|
|
|
## JSON containing HTML fragments
|
|
|
|
```json
|
|
{
|
|
"mode": "html_fragment",
|
|
"request_url": "https://example.org/ajax",
|
|
"method": "POST",
|
|
"body": {"action": "events", "venue": "123"},
|
|
"html_path": "html",
|
|
"item_selector": ".event-card",
|
|
"title_selector": ".title",
|
|
"date_selector": ".date",
|
|
"link_selector": "a",
|
|
"link_attribute": "href",
|
|
"page_parameter": "paged",
|
|
"page_start": 1,
|
|
"max_pages": 10,
|
|
"total_pages_path": "pages_total"
|
|
}
|
|
```
|
|
|
|
## Safety limits
|
|
|
|
- Request URLs still pass public-network SSRF validation.
|
|
- Redirect destinations are validated for normal HTML fetches.
|
|
- Responses default to a 5 MB limit.
|
|
- Pagination is capped at 20 pages.
|
|
- Browser-compatible headers are global; strategy headers may add endpoint-specific `Accept` or `Referer` values.
|