What2Do

A lightweight, local-first event aggregator. It crawls configured venue pages with CSS selectors or schema.org JSON-LD, normalizes events, and provides date/location/radius search.

No LLM, ML, computer vision, browser automation, JavaScript build process, PostGIS, Redis, or external database is required.

Run

docker compose up --build

Open http://localhost:8000. The source admin is at http://localhost:8000/sources.

Two example sources are installed automatically:

• Dicey Riley's recurring weekly specials
• La La La's dated event list

Click Crawl now beside each source after startup. Crawls then run on each source's configured schedule while the app is running.

Extraction model

Rules are relative to a repeating event container:

• Event container, title, date, time, link, description, and location are CSS selectors.
• Date parsing accepts a Python strptime format or flexible text parsing.
• json-ld as the event container reads schema.org Event entries without CSS rules.
• Weekly pages can create the next 12 occurrences from weekday labels.
• The source's venue address is the fallback when event HTML omits a location.

Use Test extraction to preview up to 20 records before saving. Static server-rendered HTML is intentionally used; sites that only render events with JavaScript are not supported in this low-overhead version.

Location search

Addresses are geocoded and cached using OpenStreetMap's public Nominatim service. The search field also accepts latitude,longitude directly, and every source has optional coordinate fields for deterministic/offline venue positioning. Set a real contact in NOMINATIM_USER_AGENT before public/shared use and follow the Nominatim usage policy. For heavier use, configure or self-host a compatible geocoder.

Tests

The approved 50-source Illawarra corpus is stored in tests/fixtures/illawarra_sources.json. Normal tests are fixture-based and do not access live websites:

docker compose --profile test run --rm test

Detailed per-test outcomes, durations, failure traces, and totals are exported to test-results/latest.txt after every run. The file is replaced on the next run and is ignored by Git.

Cached source extraction corpus

Caching is manual so normal tests never contact source websites. The cache command requests each manifest URL once, waits two seconds between requests, skips already cached documents, and records fetch metadata in tests/fixtures/source_cache/index.json:

docker compose --profile test run --rm cache-sources

Use --force only when intentionally refreshing every document. You can trial a smaller batch with --limit, for example:

docker compose --profile test run --rm cache-sources python scripts/cache_source_documents.py --delay 2 --limit 5

After caching, run automatic extraction entirely offline:

docker compose --profile test run --rm test sh -c "pip install --no-cache-dir pytest==8.4.2 pytest-cov==6.2.1 && python -m pytest tests/unit/test_cached_source_extraction.py -m cached_sources -vv -o addopts='' --extraction-results=/test-results/source-extraction.txt"

Per-source event counts and extraction failures are written to test-results/source-extraction.txt. Cached website documents and generated reports remain local and are ignored by Git.

Run optional live reachability checks separately (results depend on third-party availability and access policies):

docker compose --profile test run --rm test pytest -m network tests/integration

Operational notes

• SQLite data is persisted in the what2do-data Docker volume.
• Fetches are limited to 5 MB and public HTTP(S) destinations to reduce SSRF risk.
• The crawler does not bypass authentication or anti-bot systems.
• Review each site's terms and robots.txt; only crawl sources you are permitted to index.
• The MVP has no authentication. Bind it to localhost or place it behind an authenticated reverse proxy before exposing crawler administration to a network.