# What2Do

A lightweight, local-first event aggregator. It crawls configured venue pages with CSS selectors or schema.org JSON-LD, normalizes events, and provides date/location/radius search.

No LLM, ML, computer vision, browser automation, JavaScript build process, PostGIS, Redis, or external database is required.

## Run

```sh
docker compose up --build
```

Open http://localhost:8000. The source admin is at http://localhost:8000/sources.

Two example sources are installed automatically:

- Dicey Riley's recurring weekly specials
- La La La's dated event list

Click **Crawl now** beside each source after startup. Crawls then run on each source's configured schedule while the app is running.

## Extraction model

Rules are relative to a repeating event container:

- Event container, title, date, time, link, description, and location are CSS selectors.
- Date parsing accepts a Python `strptime` format or flexible text parsing.
- `json-ld` as the event container reads schema.org `Event` entries without CSS rules.
- Weekly pages can create the next 12 occurrences from weekday labels.
- The source's venue address is the fallback when event HTML omits a location.

Use **Test extraction** to preview up to 20 records before saving. Static server-rendered HTML is intentionally used; sites that only render events with JavaScript are not supported in this low-overhead version.

## Location search

Addresses are geocoded and cached using OpenStreetMap's public Nominatim service. The search field also accepts `latitude,longitude` directly, and every source has optional coordinate fields for deterministic/offline venue positioning. Set a real contact in `NOMINATIM_USER_AGENT` before public/shared use and follow the [Nominatim usage policy](https://operations.osmfoundation.org/policies/nominatim/). For heavier use, configure or self-host a compatible geocoder.

## Tests

The approved 50-source Illawarra corpus is stored in `tests/fixtures/illawarra_sources.json`. Normal tests are fixture-based and do not access live websites:

```sh
docker compose --profile test run --rm test
```

Detailed per-test outcomes, durations, failure traces, and totals are exported to `test-results/latest.txt` after every run. The file is replaced on the next run and is ignored by Git.

Run optional live reachability checks separately (results depend on third-party availability and access policies):

```sh
docker compose --profile test run --rm test pytest -m network tests/integration
```

## Operational notes

- SQLite data is persisted in the `what2do-data` Docker volume.
- Fetches are limited to 5 MB and public HTTP(S) destinations to reduce SSRF risk.
- The crawler does not bypass authentication or anti-bot systems.
- Review each site's terms and `robots.txt`; only crawl sources you are permitted to index.
- The MVP has no authentication. Bind it to localhost or place it behind an authenticated reverse proxy before exposing crawler administration to a network.