83 lines
4.0 KiB
Markdown
83 lines
4.0 KiB
Markdown
# What2Do
|
|
|
|
A lightweight, local-first event aggregator. It crawls configured venue pages with CSS selectors or schema.org JSON-LD, normalizes events, and provides date/location/radius search.
|
|
|
|
No LLM, ML, computer vision, browser automation, JavaScript build process, PostGIS, Redis, or external database is required.
|
|
|
|
## Run
|
|
|
|
```sh
|
|
docker compose up --build
|
|
```
|
|
|
|
Open http://localhost:8000. The source admin is at http://localhost:8000/sources.
|
|
|
|
Two example sources are installed automatically:
|
|
|
|
- Dicey Riley's recurring weekly specials
|
|
- La La La's dated event list
|
|
|
|
Click **Crawl now** beside each source after startup. Crawls then run on each source's configured schedule while the app is running.
|
|
|
|
## Extraction model
|
|
|
|
Rules are relative to a repeating event container:
|
|
|
|
- Event container, title, date, time, link, description, and location are CSS selectors.
|
|
- Date parsing accepts a Python `strptime` format or flexible text parsing.
|
|
- `json-ld` as the event container reads schema.org `Event` entries without CSS rules.
|
|
- Weekly pages can create the next 12 occurrences from weekday labels.
|
|
- The source's venue address is the fallback when event HTML omits a location.
|
|
|
|
Use **Test extraction** to preview up to 20 records before saving. Static server-rendered HTML is intentionally used; sites that only render events with JavaScript are not supported in this low-overhead version.
|
|
|
|
## Location search
|
|
|
|
Addresses are geocoded and cached using OpenStreetMap's public Nominatim service. The search field also accepts `latitude,longitude` directly, and every source has optional coordinate fields for deterministic/offline venue positioning. Set a real contact in `NOMINATIM_USER_AGENT` before public/shared use and follow the [Nominatim usage policy](https://operations.osmfoundation.org/policies/nominatim/). For heavier use, configure or self-host a compatible geocoder.
|
|
|
|
## Tests
|
|
|
|
The approved 50-source Illawarra corpus is stored in `tests/fixtures/illawarra_sources.json`. Normal tests are fixture-based and do not access live websites:
|
|
|
|
```sh
|
|
docker compose --profile test run --rm test
|
|
```
|
|
|
|
Detailed per-test outcomes, durations, failure traces, and totals are exported to `test-results/latest.txt` after every run. The file is replaced on the next run and is ignored by Git.
|
|
|
|
### Cached source extraction corpus
|
|
|
|
Caching is manual so normal tests never contact source websites. The cache command requests each manifest URL once, waits two seconds between requests, skips already cached documents, and records fetch metadata in `tests/fixtures/source_cache/index.json`:
|
|
|
|
```sh
|
|
docker compose --profile test run --rm cache-sources
|
|
```
|
|
|
|
Use `--force` only when intentionally refreshing every document. You can trial a smaller batch with `--limit`, for example:
|
|
|
|
```sh
|
|
docker compose --profile test run --rm cache-sources python scripts/cache_source_documents.py --delay 2 --limit 5
|
|
```
|
|
|
|
After caching, run automatic extraction entirely offline:
|
|
|
|
```sh
|
|
docker compose --profile test run --rm test sh -c "pip install --no-cache-dir pytest==8.4.2 pytest-cov==6.2.1 && python -m pytest tests/unit/test_cached_source_extraction.py -m cached_sources -vv -o addopts='' --extraction-results=/test-results/source-extraction.txt"
|
|
```
|
|
|
|
Per-source event counts and extraction failures are written to `test-results/source-extraction.txt`. Cached website documents and generated reports remain local and are ignored by Git.
|
|
|
|
Run optional live reachability checks separately (results depend on third-party availability and access policies):
|
|
|
|
```sh
|
|
docker compose --profile test run --rm test pytest -m network tests/integration
|
|
```
|
|
|
|
## Operational notes
|
|
|
|
- SQLite data is persisted in the `what2do-data` Docker volume.
|
|
- Fetches are limited to 5 MB and public HTTP(S) destinations to reduce SSRF risk.
|
|
- The crawler does not bypass authentication or anti-bot systems.
|
|
- Review each site's terms and `robots.txt`; only crawl sources you are permitted to index.
|
|
- The MVP has no authentication. Bind it to localhost or place it behind an authenticated reverse proxy before exposing crawler administration to a network.
|