45 lines
2.3 KiB
Markdown
45 lines
2.3 KiB
Markdown
# What2Do
|
|
|
|
A lightweight, local-first event aggregator. It crawls configured venue pages with CSS selectors or schema.org JSON-LD, normalizes events, and provides date/location/radius search.
|
|
|
|
No LLM, ML, computer vision, browser automation, JavaScript build process, PostGIS, Redis, or external database is required.
|
|
|
|
## Run
|
|
|
|
```sh
|
|
docker compose up --build
|
|
```
|
|
|
|
Open http://localhost:8000. The source admin is at http://localhost:8000/sources.
|
|
|
|
Two example sources are installed automatically:
|
|
|
|
- Dicey Riley's recurring weekly specials
|
|
- La La La's dated event list
|
|
|
|
Click **Crawl now** beside each source after startup. Crawls then run on each source's configured schedule while the app is running.
|
|
|
|
## Extraction model
|
|
|
|
Rules are relative to a repeating event container:
|
|
|
|
- Event container, title, date, time, link, description, and location are CSS selectors.
|
|
- Date parsing accepts a Python `strptime` format or flexible text parsing.
|
|
- `json-ld` as the event container reads schema.org `Event` entries without CSS rules.
|
|
- Weekly pages can create the next 12 occurrences from weekday labels.
|
|
- The source's venue address is the fallback when event HTML omits a location.
|
|
|
|
Use **Test extraction** to preview up to 20 records before saving. Static server-rendered HTML is intentionally used; sites that only render events with JavaScript are not supported in this low-overhead version.
|
|
|
|
## Location search
|
|
|
|
Addresses are geocoded and cached using OpenStreetMap's public Nominatim service. The search field also accepts `latitude,longitude` directly, and every source has optional coordinate fields for deterministic/offline venue positioning. Set a real contact in `NOMINATIM_USER_AGENT` before public/shared use and follow the [Nominatim usage policy](https://operations.osmfoundation.org/policies/nominatim/). For heavier use, configure or self-host a compatible geocoder.
|
|
|
|
## Operational notes
|
|
|
|
- SQLite data is persisted in the `what2do-data` Docker volume.
|
|
- Fetches are limited to 5 MB and public HTTP(S) destinations to reduce SSRF risk.
|
|
- The crawler does not bypass authentication or anti-bot systems.
|
|
- Review each site's terms and `robots.txt`; only crawl sources you are permitted to index.
|
|
- The MVP has no authentication. Bind it to localhost or place it behind an authenticated reverse proxy before exposing crawler administration to a network.
|