commit 8267064bc1249c73ea160190cf7479b210c99daf Author: zahkcb@gmail.com Date: Mon Sep 7 11:52:41 2026 +1000 initial push diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 0000000..dfa7562 --- /dev/null +++ b/Dockerfile @@ -0,0 +1,10 @@ +FROM python:3.13-slim +ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 +WORKDIR /app +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt +COPY app ./app +COPY README.md . +RUN mkdir -p /data +EXPOSE 8000 +CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"] diff --git a/README.md b/README.md new file mode 100644 index 0000000..bb6775b --- /dev/null +++ b/README.md @@ -0,0 +1,44 @@ +# What2Do + +A lightweight, local-first event aggregator. It crawls configured venue pages with CSS selectors or schema.org JSON-LD, normalizes events, and provides date/location/radius search. + +No LLM, ML, computer vision, browser automation, JavaScript build process, PostGIS, Redis, or external database is required. + +## Run + +```sh +docker compose up --build +``` + +Open http://localhost:8000. The source admin is at http://localhost:8000/sources. + +Two example sources are installed automatically: + +- Dicey Riley's recurring weekly specials +- La La La's dated event list + +Click **Crawl now** beside each source after startup. Crawls then run on each source's configured schedule while the app is running. + +## Extraction model + +Rules are relative to a repeating event container: + +- Event container, title, date, time, link, description, and location are CSS selectors. +- Date parsing accepts a Python `strptime` format or flexible text parsing. +- `json-ld` as the event container reads schema.org `Event` entries without CSS rules. +- Weekly pages can create the next 12 occurrences from weekday labels. +- The source's venue address is the fallback when event HTML omits a location. + +Use **Test extraction** to preview up to 20 records before saving. Static server-rendered HTML is intentionally used; sites that only render events with JavaScript are not supported in this low-overhead version. + +## Location search + +Addresses are geocoded and cached using OpenStreetMap's public Nominatim service. The search field also accepts `latitude,longitude` directly, and every source has optional coordinate fields for deterministic/offline venue positioning. Set a real contact in `NOMINATIM_USER_AGENT` before public/shared use and follow the [Nominatim usage policy](https://operations.osmfoundation.org/policies/nominatim/). For heavier use, configure or self-host a compatible geocoder. + +## Operational notes + +- SQLite data is persisted in the `what2do-data` Docker volume. +- Fetches are limited to 5 MB and public HTTP(S) destinations to reduce SSRF risk. +- The crawler does not bypass authentication or anti-bot systems. +- Review each site's terms and `robots.txt`; only crawl sources you are permitted to index. +- The MVP has no authentication. Bind it to localhost or place it behind an authenticated reverse proxy before exposing crawler administration to a network. diff --git a/STRATEGIES.md b/STRATEGIES.md new file mode 100644 index 0000000..d7ea29f --- /dev/null +++ b/STRATEGIES.md @@ -0,0 +1,95 @@ +# Generic extraction strategies + +Sources persist a JSON strategy. Runtime behavior is determined only by this configuration—there are no hostname-specific parsers. + +## HTML + +```json +{ + "mode": "html", + "request_url": "https://example.org/events", + "method": "GET", + "item_selector": ".event-card", + "title_selector": ".title", + "date_selector": ".date", + "time_selector": ".time", + "link_selector": "a.details", + "link_attribute": "href", + "description_selector": ".summary", + "location_selector": ".venue", + "date_format": "" +} +``` + +Selectors may read attributes using `title_attribute`, `date_attribute`, `time_attribute`, `link_attribute`, `description_attribute`, and `location_attribute`. Use `:scope` to read an attribute from the event container itself. + +## JSON API + +```json +{ + "mode": "json", + "request_url": "https://example.org/api/sessions", + "method": "GET", + "items_path": "results.events", + "title_path": "name", + "date_path": "startDate", + "end_path": "endDate", + "description_path": "description", + "location_path": "venue.address", + "link_path": "url", + "disabled_path": "disabled" +} +``` + +Paths use dot notation. Numeric path components index arrays. + +### Joining metadata + +```json +{ + "mode": "json", + "request_url": "https://example.org/api/sessions", + "items_path": "", + "date_path": "utcDate", + "link_path": "link", + "join_item_key": "movieId", + "join_title_path": "name", + "join_description_path": "summary", + "join_duration_path": "duration", + "join": { + "request_url": "https://example.org/api/movies", + "method": "GET", + "items_path": "", + "key_path": "id" + } +} +``` + +## JSON containing HTML fragments + +```json +{ + "mode": "html_fragment", + "request_url": "https://example.org/ajax", + "method": "POST", + "body": {"action": "events", "venue": "123"}, + "html_path": "html", + "item_selector": ".event-card", + "title_selector": ".title", + "date_selector": ".date", + "link_selector": "a", + "link_attribute": "href", + "page_parameter": "paged", + "page_start": 1, + "max_pages": 10, + "total_pages_path": "pages_total" +} +``` + +## Safety limits + +- Request URLs still pass public-network SSRF validation. +- Redirect destinations are validated for normal HTML fetches. +- Responses default to a 5 MB limit. +- Pagination is capped at 20 pages. +- Browser-compatible headers are global; strategy headers may add endpoint-specific `Accept` or `Referer` values. diff --git a/app/__init__.py b/app/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/app/__pycache__/__init__.cpython-314.pyc b/app/__pycache__/__init__.cpython-314.pyc new file mode 100644 index 0000000..8fb6cdf Binary files /dev/null and b/app/__pycache__/__init__.cpython-314.pyc differ diff --git a/app/__pycache__/crawler.cpython-314.pyc b/app/__pycache__/crawler.cpython-314.pyc new file mode 100644 index 0000000..102872f Binary files /dev/null and b/app/__pycache__/crawler.cpython-314.pyc differ diff --git a/app/__pycache__/db.cpython-314.pyc b/app/__pycache__/db.cpython-314.pyc new file mode 100644 index 0000000..4cd34e5 Binary files /dev/null and b/app/__pycache__/db.cpython-314.pyc differ diff --git a/app/__pycache__/main.cpython-314.pyc b/app/__pycache__/main.cpython-314.pyc new file mode 100644 index 0000000..9062bd6 Binary files /dev/null and b/app/__pycache__/main.cpython-314.pyc differ diff --git a/app/__pycache__/strategies.cpython-314.pyc b/app/__pycache__/strategies.cpython-314.pyc new file mode 100644 index 0000000..b692db4 Binary files /dev/null and b/app/__pycache__/strategies.cpython-314.pyc differ diff --git a/app/crawler.py b/app/crawler.py new file mode 100644 index 0000000..c02aa43 --- /dev/null +++ b/app/crawler.py @@ -0,0 +1,287 @@ +import hashlib +import ipaddress +import json +import re +import socket +from dataclasses import dataclass +from datetime import datetime, timedelta +from urllib.parse import urljoin, urlparse +from zoneinfo import ZoneInfo + +import httpx +from bs4 import BeautifulSoup +from dateutil import parser as date_parser + +USER_AGENT = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36" +BROWSER_HEADERS = { + "User-Agent": USER_AGENT, + "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8", + "Accept-Language": "en-AU,en;q=0.9", + "Cache-Control": "no-cache", + "DNT": "1", + "From": "what2do local event index", +} +SOCIAL_WIDGET_MARKERS = ("instagram", "sbi_", "facebook", "social-feed", "tiktok") + +@dataclass +class ExtractedEvent: + title: str + starts_at: datetime + ends_at: datetime | None = None + description: str = "" + url: str = "" + address: str = "" + +class CrawlError(Exception): + pass + +def validate_public_url(url: str) -> None: + parsed = urlparse(url) + if parsed.scheme not in {"http", "https"} or not parsed.hostname: + raise CrawlError("Only absolute http:// and https:// URLs are allowed.") + try: + infos = socket.getaddrinfo(parsed.hostname, parsed.port or (443 if parsed.scheme == "https" else 80)) + except socket.gaierror as exc: + raise CrawlError(f"Host cannot be resolved: {exc}") from exc + for info in infos: + ip = ipaddress.ip_address(info[4][0]) + if not ip.is_global: + raise CrawlError("URLs resolving to private, loopback, link-local, or reserved addresses are blocked.") + +def _http_client(timeout: int = 20) -> httpx.Client: + return httpx.Client(follow_redirects=True, timeout=timeout, headers=BROWSER_HEADERS) + +def fetch_html(url: str) -> str: + validate_public_url(url) + with _http_client() as client: + response = client.get(url) + response.raise_for_status() + validate_public_url(str(response.url)) + content_type = response.headers.get("content-type", "") + if "html" not in content_type.lower(): + raise CrawlError(f"Expected HTML but received {content_type or 'an unknown content type'}.") + if len(response.content) > 5_000_000: + raise CrawlError("Page exceeds the 5 MB crawl limit.") + return response.text + +def _text(node, selector: str) -> str: + if not selector: + return "" + found = node.select_one(selector) + return found.get_text(" ", strip=True) if found else "" + +def _link(node, selector: str, base_url: str) -> str: + found = node.select_one(selector) if selector else None + target = found or node + for attribute in ("href", "data-permalink", "data-url", "data-href"): + value = target.get(attribute) + if value: + return urljoin(base_url, str(value)) + return base_url + +def _parse_date_range(value: str, source: dict, now: datetime) -> tuple[datetime, datetime | None]: + value = " ".join(value.replace("–", "-").replace("—", "-").split()) + same_month = re.fullmatch(r"(\d{1,2})\s*-\s*(\d{1,2})\s+([A-Za-z]+)\s+(\d{4})(?:\s+(.+))?", value) + cross_month = re.fullmatch(r"(\d{1,2})\s+([A-Za-z]+)\s*-\s*(\d{1,2})\s+([A-Za-z]+)\s+(\d{4})(?:\s+(.+))?", value) + if same_month: + start_day, end_day, month, year, time_text = same_month.groups() + suffix = f" {time_text}" if time_text else "" + start = _parse_date(f"{start_day} {month} {year}{suffix}", source, now) + end = _parse_date(f"{end_day} {month} {year}{suffix}", source, now) + return start, end + if cross_month: + start_day, start_month, end_day, end_month, year, time_text = cross_month.groups() + suffix = f" {time_text}" if time_text else "" + start = _parse_date(f"{start_day} {start_month} {year}{suffix}", source, now) + end = _parse_date(f"{end_day} {end_month} {year}{suffix}", source, now) + return start, end + return _parse_date(value, source, now), None + +def _parse_date(value: str, source: dict, now: datetime) -> datetime: + value = " ".join(value.split()) + tz = ZoneInfo(source.get("timezone") or "UTC") + fmt = source.get("date_format", "").strip() + if fmt: + candidates = [fmt] + # Event lists commonly omit a year. Resolve to the next occurrence. + for candidate in candidates: + parsed = datetime.strptime(value, candidate).replace(tzinfo=tz) + if "%Y" not in candidate and "%y" not in candidate: + parsed = parsed.replace(year=now.year) + if parsed < now - timedelta(days=2): + parsed = parsed.replace(year=now.year + 1) + return parsed + if not re.search(r"\b\d{1,4}\b", value): + raise ValueError("Date text must contain a numeric day or year") + parsed = date_parser.parse(value, fuzzy=True, default=now.replace(tzinfo=None), tzinfos={"INFO": None, "IPAC": None, "IMAX": None}) + if parsed.tzinfo is None: + parsed = parsed.replace(tzinfo=tz) + if not now.year - 1 <= parsed.year <= now.year + 10: + raise ValueError("Parsed year is outside the plausible event range") + return parsed.replace(microsecond=0) + +def _extract_json_ld(soup: BeautifulSoup, base_url: str, source: dict, now: datetime) -> list[ExtractedEvent]: + result = [] + def walk(value): + if isinstance(value, list): + for item in value: yield from walk(item) + elif isinstance(value, dict): + if value.get("@type") == "Event" or "Event" in (value.get("@type") or []): + yield value + for key in ("@graph", "itemListElement"): + if key in value: yield from walk(value[key]) + for script in soup.select('script[type="application/ld+json"]'): + try: + data = json.loads(script.string or script.get_text()) + except (json.JSONDecodeError, TypeError): + continue + for event in walk(data): + location = event.get("location") or {} + address = location.get("address", "") if isinstance(location, dict) else "" + if isinstance(address, dict): + address = ", ".join(str(address.get(k, "")) for k in ("streetAddress", "addressLocality", "addressRegion", "postalCode") if address.get(k)) + try: + start = _parse_date(str(event["startDate"]), source, now) + end = _parse_date(str(event["endDate"]), source, now) if event.get("endDate") else None + except (KeyError, ValueError, OverflowError): + continue + result.append(ExtractedEvent(title=str(event.get("name", "Untitled event")), starts_at=start, ends_at=end, description=str(event.get("description", "")), url=urljoin(base_url, str(event.get("url", base_url))), address=str(address))) + return result + + +def _weekly_dates(day_text: str, now: datetime, weeks: int = 12) -> list[datetime]: + names = {name.lower(): i for i, name in enumerate(("Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"))} + match = next(((name, idx) for name, idx in names.items() if name in day_text.lower()), None) + if not match: + return [] + delta = (match[1] - now.weekday()) % 7 + first = (now + timedelta(days=delta)).replace(hour=0, minute=0, second=0, microsecond=0) + return [first + timedelta(weeks=i) for i in range(weeks)] + +def _selector_for(node) -> str: + if node.get("id"): + return f"#{node['id']}" + classes = [value for value in node.get("class", []) if not value.startswith(("elementor-element-", "post-"))] + return node.name + "".join(f".{value}" for value in classes[:2]) + +def detect_rules(html: str, url: str, timezone: str = "Australia/Sydney") -> tuple[dict, list[ExtractedEvent], str]: + soup = BeautifulSoup(html, "html.parser") + now = datetime.now(ZoneInfo(timezone)) + json_source = {"url": url, "timezone": timezone, "item_selector": "json-ld", "title_selector": "", "date_selector": "", "time_selector": "", "link_selector": "", "description_selector": "", "location_selector": "", "date_format": "", "recurring_weekly": 0} + json_events = extract_events(html, json_source) + if json_events: + return json_source, json_events, "Found schema.org JSON-LD Event data." + + date_pattern = re.compile(r"(?:\b(?:mon|tue|wed|thu|fri|sat|sun)(?:day)?\b|\b\d{1,2}[\s/.-](?:\d{1,2}|jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)|\b(?:jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)[a-z]*\s+\d{1,2})", re.I) + candidates = [] + container_selectors = ("article", "li", ".event", ".events", "[class*='event']", "[class*='Event']", ".event-list__item", ".item", "[class*='item']", ".wpupg-item", ".wpupg-item.wpupg-type-event", "[data-permalink]", ".wp-block-post", ".elementor-loop-item", ".tribe-events-calendar-list__event-row", ".em-event", ".mec-event-article") + for selector in container_selectors: + try: nodes = soup.select(selector) + except Exception: continue + if not 2 <= len(nodes) <= 300: + continue + nodes = [node for node in nodes if not any(marker in " ".join(str(value) for value in (node.get("class") or [])).casefold() for marker in SOCIAL_WIDGET_MARKERS)] + if len(nodes) < 2: continue + dated = [node for node in nodes if date_pattern.search(node.get_text(" ", strip=True))] + titled = [node for node in dated if node.find(["h1", "h2", "h3", "h4", "a"]) or node.select_one("[class*='title'], [itemprop='name']")] + score = len(titled) * 3 + len(dated) - max(0, len(nodes) - len(dated)) + if len(titled) >= 2: + candidates.append((score, selector, titled)) + if not candidates: + raise CrawlError("No repeated event-like HTML containers were detected. Try JSON-LD or configure CSS selectors manually.") + trials = [] + for _, item_selector, nodes in sorted(candidates, reverse=True)[:8]: + title_selectors = [] + date_selectors = [] + link_selectors = [] + for sample in nodes[:8]: + for node in sample.select("h1, h2, h3, h4, a, [class*='title'], [itemprop='name']"): + selector = _selector_for(node) + if selector not in title_selectors: title_selectors.append(selector) + for node in sample.find_all(["time", "span", "div", "li", "p", "ul"]): + if date_pattern.search(node.get_text(" ", strip=True)): + selector = _selector_for(node) + if selector not in date_selectors: date_selectors.append(selector) + for node in sample.find_all("a"): + selector = _selector_for(node) + if selector not in link_selectors: link_selectors.append(selector) + if any(sample.get(attribute) for attribute in ("data-permalink", "data-url", "data-href")) and ":scope" not in link_selectors: + link_selectors.insert(0, ":scope") + time_selectors = [] + for sample in nodes[:8]: + for node in sample.find_all(["time", "span", "div", "p"]): + if re.search(r"\b\d{1,2}(?::\d{2})?\s*(?:am|pm)\b", node.get_text(" ", strip=True), re.IGNORECASE): + selector = _selector_for(node) + if selector not in time_selectors: time_selectors.append(selector) + for title_selector in title_selectors[:6]: + for date_selector in date_selectors[:10]: + for time_selector in [""] + time_selectors[:4]: + for recurring in (0, 1): + rules = {"url": url, "timezone": timezone, "item_selector": item_selector, "title_selector": title_selector, "date_selector": date_selector, "time_selector": time_selector, "link_selector": link_selectors[0] if link_selectors else "", "description_selector": "", "location_selector": "", "date_format": "", "recurring_weekly": recurring} + events = extract_events(html, rules) + unique_titles = len({event.title for event in events}) + explicit_dates = sum(1 for event in events if event.starts_at.second == 0 and event.starts_at.microsecond == 0 and now.year - 1 <= event.starts_at.year <= now.year + 10) + if events and unique_titles >= 2 and explicit_dates == len(events): + occurrence_penalty = 12 if recurring else 1 + trials.append((unique_titles * 10 + len(events) / occurrence_penalty, rules, events, len(nodes))) + if not trials: + raise CrawlError("Event-like containers were found, but no selector combination produced parseable events. Adjust selectors manually.") + _, rules, events, container_count = max(trials, key=lambda value: value[0]) + winning_nodes = soup.select(rules["item_selector"]) + sample = winning_nodes[0] if winning_nodes else None + if sample: + description_node = sample.find(["p", "summary"]) + location_node = next((node for node in sample.find_all(["span", "div", "p"]) if re.search(r"\b(?:venue|location|address|theatre|hall|hotel|club)\b", node.get_text(" ", strip=True), re.I)), None) + rules["description_selector"] = _selector_for(description_node) if description_node else "" + rules["location_selector"] = _selector_for(location_node) if location_node else "" + return rules, events, f"Guessed CSS selectors from {container_count} repeated event-like containers." + +def extract_events(html: str, source: dict) -> list[ExtractedEvent]: + soup = BeautifulSoup(html, "html.parser") + tz = ZoneInfo(source.get("timezone") or "UTC") + now = datetime.now(tz) + item_selector = source.get("item_selector", "").strip() + if not item_selector or item_selector == "json-ld": + json_events = _extract_json_ld(soup, source["url"], source, now) + if json_events or item_selector == "json-ld": + return json_events + events = [] + for item in soup.select(item_selector): + title = _text(item, source.get("title_selector", "")) + raw_date = _text(item, source.get("date_selector", "")) + raw_time = _text(item, source.get("time_selector", "")) + if not title or not raw_date: + continue + description = _text(item, source.get("description_selector", "")) + address = _text(item, source.get("location_selector", "")) + link = _link(item, source.get("link_selector", ""), source["url"]) + if source.get("recurring_weekly"): + for start in _weekly_dates(raw_date, now): + events.append(ExtractedEvent(title=title, starts_at=start, description=description, url=link, address=address)) + continue + try: + start, end = _parse_date_range(f"{raw_date} {raw_time}".strip(), source, now) + except (ValueError, OverflowError): + continue + events.append(ExtractedEvent(title=title, starts_at=start, ends_at=end, description=description, url=link, address=address)) + return events + +def infer_page_fields(html: str, url: str) -> dict: + soup = BeautifulSoup(html, "html.parser") + title = "" + og_site = soup.select_one('meta[property="og:site_name"]') + if og_site: + content = og_site.get("content") + title = str(content).strip() if content else "" + if not title and soup.title: + title = re.split(r"[|–—-]", soup.title.get_text(" ", strip=True))[0].strip() + heading = soup.find("h1") + source_name = heading.get_text(" ", strip=True) if heading else title or urlparse(url).hostname or "Event source" + text = soup.get_text(" ", strip=True) + address_match = re.search(r"\b\d{1,5}[A-Za-z/-]*\s+[A-Za-z][A-Za-z .'-]+(?:Street|St|Road|Rd|Lane|Ln|Avenue|Ave|Drive|Dr|Way|Place|Pl),?\s+[A-Za-z ]+\s+(?:NSW|VIC|QLD|SA|WA|TAS|NT|ACT)\s+\d{4}\b", text, re.I) + address = address_match.group(0).strip() if address_match else "" + return {"name": source_name, "venue_name": title or source_name, "venue_address": address} + +def external_key(event: ExtractedEvent) -> str: + value = f"{event.title.casefold()}|{event.starts_at.isoformat()}|{event.address.casefold()}" + return hashlib.sha256(value.encode()).hexdigest() diff --git a/app/db.py b/app/db.py new file mode 100644 index 0000000..a96841a --- /dev/null +++ b/app/db.py @@ -0,0 +1,93 @@ +import os +import sqlite3 +from contextlib import contextmanager +from pathlib import Path + +DB_PATH = os.getenv("DATABASE_PATH", "data/what2do.db") + +SCHEMA = """ +PRAGMA journal_mode=WAL; +PRAGMA foreign_keys=ON; +CREATE TABLE IF NOT EXISTS sources ( + id INTEGER PRIMARY KEY, + name TEXT NOT NULL, + url TEXT NOT NULL UNIQUE, + venue_name TEXT NOT NULL, + venue_address TEXT NOT NULL, + latitude REAL, + longitude REAL, + timezone TEXT NOT NULL DEFAULT 'Australia/Sydney', + strategy_json TEXT NOT NULL DEFAULT '{}', + item_selector TEXT NOT NULL, + title_selector TEXT NOT NULL, + date_selector TEXT NOT NULL, + time_selector TEXT NOT NULL DEFAULT '', + link_selector TEXT NOT NULL DEFAULT '', + description_selector TEXT NOT NULL DEFAULT '', + location_selector TEXT NOT NULL DEFAULT '', + date_format TEXT NOT NULL DEFAULT '', + recurring_weekly INTEGER NOT NULL DEFAULT 0, + enabled INTEGER NOT NULL DEFAULT 1, + crawl_interval_minutes INTEGER NOT NULL DEFAULT 360, + last_crawled_at TEXT, + last_status TEXT, + last_error TEXT, + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP +); +CREATE TABLE IF NOT EXISTS events ( + id INTEGER PRIMARY KEY, + source_id INTEGER NOT NULL REFERENCES sources(id) ON DELETE CASCADE, + external_key TEXT NOT NULL, + title TEXT NOT NULL, + description TEXT NOT NULL DEFAULT '', + starts_at TEXT NOT NULL, + ends_at TEXT, + venue_name TEXT NOT NULL, + address TEXT NOT NULL, + latitude REAL, + longitude REAL, + event_url TEXT NOT NULL, + hidden INTEGER NOT NULL DEFAULT 0, + last_seen_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + UNIQUE(source_id, external_key) +); +CREATE INDEX IF NOT EXISTS idx_events_starts_at ON events(starts_at); +CREATE INDEX IF NOT EXISTS idx_events_source ON events(source_id); +CREATE TABLE IF NOT EXISTS geocode_cache ( + query TEXT PRIMARY KEY, + latitude REAL, + longitude REAL, + display_name TEXT, + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP +); +""" + +@contextmanager +def connect(): + path = Path(DB_PATH) + path.parent.mkdir(parents=True, exist_ok=True) + db = sqlite3.connect(path) + db.row_factory = sqlite3.Row + db.execute("PRAGMA foreign_keys=ON") + try: + yield db + db.commit() + finally: + db.close() + +def init_db(): + with connect() as db: + db.executescript(SCHEMA) + event_columns = {row["name"] for row in db.execute("PRAGMA table_info(events)")} + if "hidden" not in event_columns: + db.execute("ALTER TABLE events ADD COLUMN hidden INTEGER NOT NULL DEFAULT 0") + source_columns = {row["name"] for row in db.execute("PRAGMA table_info(sources)")} + if "strategy_json" not in source_columns: + db.execute("ALTER TABLE sources ADD COLUMN strategy_json TEXT NOT NULL DEFAULT '{}'") + db.execute("""UPDATE sources SET strategy_json=json_object( + 'mode','html','request_url',url,'method','GET','response_type','html', + 'item_selector',item_selector,'title_selector',title_selector,'date_selector',date_selector, + 'time_selector',time_selector,'link_selector',link_selector,'description_selector',description_selector, + 'location_selector',location_selector,'date_format',date_format,'recurring_weekly',recurring_weekly + ) WHERE strategy_json='{}' OR strategy_json=''""") diff --git a/app/main.py b/app/main.py new file mode 100644 index 0000000..710d281 --- /dev/null +++ b/app/main.py @@ -0,0 +1,569 @@ +import asyncio +import json +import math +import os +import secrets +import threading +from calendar import Calendar +from contextlib import asynccontextmanager +from datetime import datetime, timedelta, timezone +from urllib.parse import urljoin + +import httpx +from fastapi import FastAPI, HTTPException, Query, Request +from fastapi.responses import HTMLResponse, JSONResponse, RedirectResponse +from fastapi.staticfiles import StaticFiles +from fastapi.templating import Jinja2Templates +from bs4 import BeautifulSoup + +from .crawler import CrawlError, _http_client, detect_rules, external_key, fetch_html, infer_page_fields +from .db import connect, init_db +from .strategies import execute_strategy, parse_strategy + +TEMPLATES = Jinja2Templates(directory="app/templates") +NOMINATIM_USER_AGENT = os.getenv("NOMINATIM_USER_AGENT", "what2do-local/0.1") +SCHEDULER_INTERVAL = int(os.getenv("CRAWL_INTERVAL_MINUTES", "60")) +PICKER_SNAPSHOTS = {} +PICKER_LOCK = threading.Lock() +PICKER_TTL = timedelta(minutes=15) +EVENT_TAGS = { + "music": ("music", "band", "dj", "album", "tour", "concert", "punk", "emo", "rock", "singer"), + "art": ("art", "gallery", "exhibition", "paint", "creative", "workshop"), + "comedy": ("comedy", "comedian", "stand-up", "standup"), + "trivia": ("trivia", "quiz"), + "film": ("film", "movie", "cinema", "screening"), + "food": ("food", "dinner", "lunch", "brunch", "schnitty", "steak", "surf & turf"), + "sports": ("sport", "football", "rugby", "darts", "poker", "race"), + "community": ("community", "market", "fundraiser", "family", "festival"), + "nightlife": ("party", "club", "night", "bingo", "jackpot"), +} + +SEEDS = [ + { + "name": "Dicey Riley's weekly specials", "url": "https://www.diceyrileyshotel.com.au/pub-specials/", + "venue_name": "Dicey Riley's Hotel", "venue_address": "329-333 Crown St, Wollongong NSW 2500, Australia", + "latitude": -34.42538, "longitude": 150.89375, + "item_selector": "article.weekly_specials", "title_selector": "h2.elementor-heading-title", + "date_selector": ".elementor-post-info__item--type-custom", "time_selector": "", "link_selector": "a", + "date_format": "", "recurring_weekly": 1, + }, + { + "name": "La La La's events", "url": "https://www.lalalas.com.au/all-events/", + "venue_name": "La La La's", "venue_address": "3 Globe Lane, Wollongong NSW 2500, Australia", + "latitude": -34.42569, "longitude": 150.89816, + "item_selector": ".slider_container > .items", "title_selector": ".feed_titles h2", + "date_selector": ".feed_dates ul", "time_selector": "", "link_selector": "a.slider_button", + "date_format": "%d %B %H:%M", "recurring_weekly": 0, + }, +] + +def source_values(form) -> dict: + raw_strategy = form.get("strategy_json", "{}").strip() or "{}" + try: strategy = json.loads(raw_strategy) + except json.JSONDecodeError as exc: raise ValueError(f"Strategy JSON is invalid: {exc.msg} at line {exc.lineno}") from exc + if strategy.get("mode", "html") == "html": + strategy.update({"mode": "html", "request_url": form.get("url", "").strip(), "method": strategy.get("method", "GET"), "response_type": "html", "item_selector": form.get("item_selector", "").strip(), "title_selector": form.get("title_selector", "").strip(), "date_selector": form.get("date_selector", "").strip(), "time_selector": form.get("time_selector", "").strip(), "link_selector": form.get("link_selector", "").strip(), "description_selector": form.get("description_selector", "").strip(), "location_selector": form.get("location_selector", "").strip(), "date_format": form.get("date_format", "").strip(), "recurring_weekly": 1 if form.get("recurring_weekly") else 0}) + return { + "name": form.get("name", "").strip(), "url": form.get("url", "").strip(), + "venue_name": form.get("venue_name", "").strip(), "venue_address": form.get("venue_address", "").strip(), + "latitude": float(form["latitude"]) if form.get("latitude") else None, + "longitude": float(form["longitude"]) if form.get("longitude") else None, + "timezone": form.get("timezone", "Australia/Sydney").strip(), "item_selector": form.get("item_selector", "").strip(), + "title_selector": form.get("title_selector", "").strip(), "date_selector": form.get("date_selector", "").strip(), + "time_selector": form.get("time_selector", "").strip(), "link_selector": form.get("link_selector", "").strip(), + "description_selector": form.get("description_selector", "").strip(), "location_selector": form.get("location_selector", "").strip(), + "date_format": form.get("date_format", "").strip(), "recurring_weekly": 1 if form.get("recurring_weekly") else 0, + "strategy_json": json.dumps(strategy), + "enabled": 1 if form.get("enabled") else 0, "crawl_interval_minutes": int(form.get("crawl_interval_minutes") or 360), + } + +def seed_sources(): + with connect() as db: + for source in SEEDS: + source_strategy = {"mode": "html", "request_url": source["url"], "method": "GET", "response_type": "html", "item_selector": source["item_selector"], "title_selector": source["title_selector"], "date_selector": source["date_selector"], "time_selector": source.get("time_selector", ""), "link_selector": source.get("link_selector", ""), "link_attribute": "href", "description_selector": "", "location_selector": "", "date_format": source.get("date_format", ""), "recurring_weekly": source.get("recurring_weekly", 0)} + fields = {**source, "strategy_json": json.dumps(source_strategy), "timezone": "Australia/Sydney", "time_selector": source.get("time_selector", ""), "link_selector": source.get("link_selector", ""), "description_selector": "", "location_selector": "", "enabled": 1, "crawl_interval_minutes": 360} + columns = ",".join(fields) + db.execute(f"INSERT OR IGNORE INTO sources ({columns}) VALUES ({','.join('?' for _ in fields)})", tuple(fields.values())) + db.execute("UPDATE sources SET latitude=COALESCE(latitude,?),longitude=COALESCE(longitude,?) WHERE url=?", (source.get("latitude"), source.get("longitude"), source["url"])) + db.execute("UPDATE events SET latitude=COALESCE(latitude,?),longitude=COALESCE(longitude,?) WHERE source_id=(SELECT id FROM sources WHERE url=?)", (source.get("latitude"), source.get("longitude"), source["url"])) + +def geocode(query: str): + if not query: + return None + try: + latitude, longitude = (float(value.strip()) for value in query.split(",", 1)) + if -90 <= latitude <= 90 and -180 <= longitude <= 180: + return latitude, longitude, query + except (ValueError, TypeError): + pass + cache_key = query.casefold() + with connect() as db: + cached = db.execute("SELECT latitude, longitude, display_name FROM geocode_cache WHERE query=?", (cache_key,)).fetchone() + if cached: + return tuple(cached) + parts = [part.strip() for part in query.split(",") if part.strip()] + candidates = [query] + if len(parts) > 1: + candidates.append(", ".join(parts[1:])) + result = [] + try: + for candidate in candidates: + response = httpx.get("https://nominatim.openstreetmap.org/search", params={"q": candidate, "format": "jsonv2", "limit": 1, "countrycodes": "au"}, headers={"User-Agent": NOMINATIM_USER_AGENT}, timeout=15) + response.raise_for_status() + result = response.json() + if result: + break + except (httpx.HTTPError, ValueError): + return None + if not result: + return None + value = (float(result[0]["lat"]), float(result[0]["lon"]), result[0].get("display_name", query)) + with connect() as db: + db.execute("INSERT OR REPLACE INTO geocode_cache(query,latitude,longitude,display_name) VALUES(?,?,?,?)", (cache_key, *value)) + return value + +def crawl_source(source_id: int): + with connect() as db: + row = db.execute("SELECT * FROM sources WHERE id=?", (source_id,)).fetchone() + if not row: + raise CrawlError("Source not found.") + source = dict(row) + try: + with _http_client(timeout=45) as client: + events = execute_strategy(source, client) + coords = (source["latitude"], source["longitude"]) + if None in coords: + point = geocode(source["venue_address"]) + coords = point[:2] if point else (None, None) + seen = [] + with connect() as db: + db.execute("UPDATE sources SET latitude=?,longitude=? WHERE id=?", (*coords, source_id)) + for event in events: + key = external_key(event) + seen.append(key) + address = event.address or source["venue_address"] + db.execute("""INSERT INTO events(source_id,external_key,title,description,starts_at,ends_at,venue_name,address,latitude,longitude,event_url) + VALUES(?,?,?,?,?,?,?,?,?,?,?) ON CONFLICT(source_id,external_key) DO UPDATE SET title=excluded.title,description=excluded.description, + starts_at=excluded.starts_at,ends_at=excluded.ends_at,venue_name=excluded.venue_name,address=excluded.address,latitude=excluded.latitude,longitude=excluded.longitude,event_url=excluded.event_url,last_seen_at=CURRENT_TIMESTAMP""", + (source_id, key, event.title, event.description, event.starts_at.astimezone(timezone.utc).isoformat(), event.ends_at.astimezone(timezone.utc).isoformat() if event.ends_at else None, source["venue_name"], address, *coords, event.url)) + if seen: + placeholders = ",".join("?" for _ in seen) + db.execute(f"DELETE FROM events WHERE source_id=? AND external_key NOT IN ({placeholders})", (source_id, *seen)) + db.execute("UPDATE sources SET last_crawled_at=CURRENT_TIMESTAMP,last_status=?,last_error=NULL WHERE id=?", (f"Found {len(events)} events", source_id)) + return len(events) + except Exception as exc: + with connect() as db: + db.execute("UPDATE sources SET last_crawled_at=CURRENT_TIMESTAMP,last_status='Failed',last_error=? WHERE id=?", (str(exc)[:1000], source_id)) + raise + +async def scheduler(): + while True: + await asyncio.sleep(max(1, SCHEDULER_INTERVAL) * 60) + with connect() as db: + rows = db.execute("SELECT id,last_crawled_at,crawl_interval_minutes FROM sources WHERE enabled=1").fetchall() + now = datetime.now(timezone.utc) + for row in rows: + last = datetime.fromisoformat(row["last_crawled_at"] + "+00:00") if row["last_crawled_at"] else None + if not last or now - last >= timedelta(minutes=row["crawl_interval_minutes"]): + try: await asyncio.to_thread(crawl_source, row["id"]) + except Exception: pass + +@asynccontextmanager +async def lifespan(app): + init_db(); seed_sources() + task = asyncio.create_task(scheduler()) + yield + task.cancel() + +app = FastAPI(title="What2Do", lifespan=lifespan) +app.mount("/static", StaticFiles(directory="app/static"), name="static") + +def classify_event(title: str, description: str) -> list[str]: + text = f"{title} {description}".casefold() + tags = [tag for tag, keywords in EVENT_TAGS.items() if any(keyword in text for keyword in keywords)] + return tags or ["other"] + +def distance_km(lat1, lon1, lat2, lon2): + p1, p2 = math.radians(lat1), math.radians(lat2) + dp, dl = math.radians(lat2-lat1), math.radians(lon2-lon1) + a = math.sin(dp/2)**2 + math.cos(p1)*math.cos(p2)*math.sin(dl/2)**2 + return 6371 * 2 * math.atan2(math.sqrt(a), math.sqrt(1-a)) + +@app.get("/", response_class=HTMLResponse) +def index(request: Request, location: str = "", latitude: float | None = None, longitude: float | None = None, start: str = "", end: str = "", radius: float = 25, tag: list[str] = Query(default=[]), term: str = ""): + point = (latitude, longitude, location) if latitude is not None and longitude is not None else (geocode(location) if location else None) + local_now = datetime.now().astimezone() + default_start = local_now.replace(day=1, hour=0, minute=0, second=0, microsecond=0) + if default_start.month == 12: + default_end = default_start.replace(year=default_start.year + 1, month=1) + else: + default_end = default_start.replace(month=default_start.month + 1) + if not start: + start = default_start.strftime("%Y-%m-%dT%H:%M") + if not end: + end = default_end.strftime("%Y-%m-%dT%H:%M") + try: start_at = datetime.fromisoformat(start).astimezone(timezone.utc) + except ValueError: + start_at = default_start.astimezone(timezone.utc) + start = default_start.strftime("%Y-%m-%dT%H:%M") + try: end_at = datetime.fromisoformat(end).astimezone(timezone.utc) + except ValueError: + end_at = default_end.astimezone(timezone.utc) + end = default_end.strftime("%Y-%m-%dT%H:%M") + with connect() as db: + rows = [dict(r) for r in db.execute("SELECT events.*,sources.name source_name FROM events JOIN sources ON sources.id=events.source_id WHERE events.hidden=0 AND starts_at < ? AND COALESCE(ends_at,starts_at) >= ? ORDER BY starts_at", (end_at.isoformat(), start_at.isoformat())).fetchall()] + for event in rows: + event["distance"] = distance_km(point[0], point[1], event["latitude"], event["longitude"]) if point and event["latitude"] is not None else None + local_start = datetime.fromisoformat(event["starts_at"]).astimezone() + event["display_time"] = local_start.strftime("%a %d %b %Y, %H:%M") + if event["ends_at"]: + local_end = datetime.fromisoformat(event["ends_at"]).astimezone() + event["display_time"] += f" – {local_end.strftime('%a %d %b %Y, %H:%M')}" + event["day_key"] = local_start.strftime("%Y-%m-%d") + event["end_day_key"] = datetime.fromisoformat(event["ends_at"]).astimezone().strftime("%Y-%m-%d") if event["ends_at"] else event["day_key"] + event["tags"] = classify_event(event["title"], event["description"]) + if point: rows = [event for event in rows if event["distance"] is not None and event["distance"] <= radius] + selected_tags = [value for value in tag if value in EVENT_TAGS or value == "other"] + if selected_tags: rows = [event for event in rows if set(selected_tags).intersection(event["tags"])] + if term: + needle = term.casefold() + rows = [event for event in rows if needle in f"{event['title']} {event['description']} {event['venue_name']} {event['address']}".casefold()] + calendar_start = start_at.astimezone() + month_days = list(Calendar(firstweekday=0).itermonthdates(calendar_start.year, calendar_start.month)) + day_counts = {} + for event in rows: + covered_day = datetime.fromisoformat(event["day_key"]).date() + final_day = datetime.fromisoformat(event["end_day_key"]).date() + while covered_day <= final_day: + key = covered_day.strftime("%Y-%m-%d") + day_counts[key] = day_counts.get(key, 0) + 1 + covered_day += timedelta(days=1) + calendar_days = [{"date": day, "key": day.strftime("%Y-%m-%d"), "count": day_counts.get(day.strftime("%Y-%m-%d"), 0), "in_month": day.month == calendar_start.month} for day in month_days] + map_events = [{"id": event["id"], "title": event["title"], "venue": event["venue_name"], "time": event["display_time"], "latitude": event["latitude"], "longitude": event["longitude"]} for event in rows if event["latitude"] is not None] + context = {"events": rows, "map_events": map_events, "calendar_days": calendar_days, "calendar_month": calendar_start.strftime("%B %Y"), "event_tags": EVENT_TAGS.keys(), "location": location, "latitude": latitude, "longitude": longitude, "start": start, "end": end, "radius": radius, "selected_tags": selected_tags, "term": term, "geocoded": point} + return TEMPLATES.TemplateResponse(request, "index.html", context) + +@app.get("/api/locations") +def location_suggestions(q: str = ""): + query = q.strip() + if len(query) < 3: + return [] + try: + response = httpx.get("https://photon.komoot.io/api/", params={"q": query, "limit": 10, "lang": "en"}, headers={"User-Agent": "what2do-local/0.1"}, timeout=10) + response.raise_for_status() + suggestions = [] + for feature in response.json().get("features", []): + properties = feature.get("properties", {}) + if properties.get("countrycode", "").upper() != "AU": + continue + coordinates = feature.get("geometry", {}).get("coordinates", []) + if len(coordinates) < 2: + continue + place = properties.get("name") or properties.get("city") or properties.get("district") or properties.get("county") + label = ", ".join(dict.fromkeys(value for value in (place, properties.get("city"), properties.get("state"), properties.get("postcode"), properties.get("country")) if value)) + suggestion = {"label": label, "latitude": float(coordinates[1]), "longitude": float(coordinates[0])} + if label and suggestion not in suggestions: + suggestions.append(suggestion) + return suggestions[:5] + except (httpx.HTTPError, ValueError, KeyError, TypeError): + return [] + +@app.get("/api/addresses") +def address_suggestions(q: str = ""): + query = q.strip() + if len(query) < 3: + return [] + try: + response = httpx.get("https://photon.komoot.io/api/", params={"q": query, "limit": 8, "lang": "en"}, headers={"User-Agent": "what2do-local/0.1"}, timeout=10) + response.raise_for_status() + results = [] + for feature in response.json().get("features", []): + properties = feature.get("properties", {}) + if properties.get("countrycode", "").upper() != "AU": continue + coordinates = feature.get("geometry", {}).get("coordinates", []) + if len(coordinates) < 2: continue + street = " ".join(value for value in (properties.get("housenumber"), properties.get("street")) if value) + label = ", ".join(dict.fromkeys(value for value in (properties.get("name"), street, properties.get("city"), properties.get("state"), properties.get("postcode"), properties.get("country")) if value)) + item = {"label": label, "latitude": float(coordinates[1]), "longitude": float(coordinates[0])} + if label and item not in results: results.append(item) + return results[:5] + except (httpx.HTTPError, ValueError, KeyError, TypeError): + return [] + +@app.get("/events/{event_id}", response_class=HTMLResponse) +def event_detail(request: Request, event_id: int): + with connect() as db: + row = db.execute("SELECT events.*,sources.name source_name,sources.url source_url FROM events JOIN sources ON sources.id=events.source_id WHERE events.id=? AND events.hidden=0", (event_id,)).fetchone() + if not row: + raise HTTPException(404) + event = dict(row) + local_start = datetime.fromisoformat(event["starts_at"]).astimezone() + event["display_date"] = local_start.strftime("%A, %d %B %Y") + event["display_time"] = local_start.strftime("%H:%M") + event["display_end"] = None + if event["ends_at"]: + local_end = datetime.fromisoformat(event["ends_at"]).astimezone() + event["display_end"] = local_end.strftime("%A, %d %B %Y at %H:%M") + event["canonical_url"] = str(request.url) + return TEMPLATES.TemplateResponse(request, "event.html", {"event": event}) + +@app.get("/sources", response_class=HTMLResponse) +def sources(request: Request): + with connect() as db: rows = db.execute("SELECT *, (SELECT count(*) FROM events WHERE source_id=sources.id) event_count FROM sources ORDER BY name").fetchall() + return TEMPLATES.TemplateResponse(request, "sources.html", {"sources": rows}) + +@app.get("/sources/new", response_class=HTMLResponse) +def new_source(request: Request): + return TEMPLATES.TemplateResponse(request, "source_form.html", {"source": {"enabled": 1, "timezone": "Australia/Sydney", "crawl_interval_minutes": 360, "strategy_json": "{}"}, "action": "/sources", "preview": None}) + +@app.get("/sources/{source_id}", response_class=HTMLResponse) +def edit_source(request: Request, source_id: int): + with connect() as db: source = db.execute("SELECT * FROM sources WHERE id=?", (source_id,)).fetchone() + if not source: raise HTTPException(404) + return TEMPLATES.TemplateResponse(request, "source_form.html", {"source": {**dict(source), "strategy_json": json.dumps(parse_strategy(source["strategy_json"]), indent=2)}, "action": f"/sources/{source_id}", "preview": None}) + +@app.post("/sources") +async def create_source(request: Request): + form = await request.form(); values = source_values(form) + with connect() as db: + columns = ",".join(values); cursor = db.execute(f"INSERT INTO sources ({columns}) VALUES ({','.join('?' for _ in values)})", tuple(values.values())) + return RedirectResponse(f"/sources/{cursor.lastrowid}", 303) + +@app.post("/sources/{source_id}") +async def update_source(request: Request, source_id: int): + form = await request.form(); values = source_values(form) + with connect() as db: db.execute(f"UPDATE sources SET {','.join(f'{key}=?' for key in values)} WHERE id=?", (*values.values(), source_id)) + return RedirectResponse(f"/sources/{source_id}", 303) + +@app.get("/sources/{source_id}/events", response_class=HTMLResponse) +def source_events(request: Request, source_id: int): + with connect() as db: + source = db.execute("SELECT * FROM sources WHERE id=?", (source_id,)).fetchone() + events = db.execute("SELECT * FROM events WHERE source_id=? ORDER BY starts_at", (source_id,)).fetchall() + if not source: raise HTTPException(404) + return TEMPLATES.TemplateResponse(request, "source_events.html", {"source": source, "events": events}) + +@app.get("/sources/{source_id}/events/{event_id}", response_class=HTMLResponse) +def edit_source_event(request: Request, source_id: int, event_id: int): + with connect() as db: + source = db.execute("SELECT * FROM sources WHERE id=?", (source_id,)).fetchone() + event = db.execute("SELECT * FROM events WHERE id=? AND source_id=?", (event_id, source_id)).fetchone() + if not source or not event: raise HTTPException(404) + return TEMPLATES.TemplateResponse(request, "event_form.html", {"source": source, "event": event}) + +@app.post("/sources/{source_id}/events/{event_id}") +async def update_source_event(request: Request, source_id: int, event_id: int): + form = await request.form() + values = (form.get("title", "").strip(), form.get("description", "").strip(), form.get("starts_at", "").strip(), form.get("ends_at", "").strip() or None, form.get("venue_name", "").strip(), form.get("address", "").strip(), float(form["latitude"]) if form.get("latitude") else None, float(form["longitude"]) if form.get("longitude") else None, form.get("event_url", "").strip(), event_id, source_id) + with connect() as db: + db.execute("UPDATE events SET title=?,description=?,starts_at=?,ends_at=?,venue_name=?,address=?,latitude=?,longitude=?,event_url=? WHERE id=? AND source_id=?", values) + return RedirectResponse(f"/sources/{source_id}/events", 303) + +@app.post("/sources/{source_id}/events/{event_id}/visibility") +def toggle_event_visibility(source_id: int, event_id: int): + with connect() as db: + db.execute("UPDATE events SET hidden=CASE hidden WHEN 1 THEN 0 ELSE 1 END WHERE id=? AND source_id=?", (event_id, source_id)) + return RedirectResponse(f"/sources/{source_id}/events", 303) + +@app.post("/sources/{source_id}/crawl") +def run_crawl(source_id: int): + try: crawl_source(source_id) + except Exception: pass + return RedirectResponse("/sources", 303) + +@app.post("/sources/{source_id}/delete") +def delete_source(source_id: int): + with connect() as db: db.execute("DELETE FROM sources WHERE id=?", (source_id,)) + return RedirectResponse("/sources", 303) + +def sanitize_picker_html(html: str, source_url: str) -> str: + soup = BeautifulSoup(html, "html.parser") + for tag in soup.select("script, iframe, frame, frameset, object, embed, form, input, button, textarea, select, base, meta[http-equiv]"): + tag.decompose() + for node in soup.find_all(True): + for attribute in list(node.attrs): + lowered = attribute.lower() + if lowered.startswith("on") or lowered in {"srcdoc", "integrity", "nonce", "formaction"}: + del node.attrs[attribute] + for attribute in ("src", "href", "poster", "action"): + value = node.get(attribute) + if not value: continue + value = str(value).strip() + if value.lower().startswith(("javascript:", "data:text/html", "vbscript:")): + del node.attrs[attribute] + elif attribute in {"src", "poster"}: + node.attrs[attribute] = urljoin(source_url, value) + elif attribute == "href" and node.name == "link" and "stylesheet" in (node.get("rel") or []): + node.attrs[attribute] = urljoin(source_url, value) + elif attribute in {"href", "action"}: + node.attrs[attribute] = "#" + node.attrs.pop("target", None) + head = soup.head + if not head: + head = soup.new_tag("head") + if soup.html: soup.html.insert(0, head) + else: soup.insert(0, head) + style = soup.new_tag("style") + style.string = "html,body{min-height:100%;background:white;color:#17211b} body{margin:12px} *{cursor:crosshair!important} img{max-width:100%;height:auto} .w2d-hover{outline:3px solid #dd5b32!important;outline-offset:2px!important}.w2d-picked{outline:4px solid #234c3c!important;outline-offset:2px!important}" + head.append(style) + script = soup.new_tag("script") + script.string = PICKER_SCRIPT + soup.body.append(script) if soup.body else soup.append(script) + return str(soup) + +PICKER_SCRIPT = r'''(() => { + let hovered = null, picked = null; + const esc = value => CSS.escape(value); + function stableClasses(element) { + return [...element.classList].filter(name => !name.startsWith('w2d-') && !/^elementor-element-[a-f0-9]+$/.test(name) && !/^post-\d+$/.test(name)).slice(0, 3); + } + function idPattern(id) { + const match = id.match(/^(.*?[-_])\d{3,}$/); + return match && match[1].length >= 3 ? `[id^="${CSS.escape(match[1])}"]` : `#${esc(id)}`; + } + function simpleSelector(element) { + if (element.id) return idPattern(element.id); + const classes = stableClasses(element); + return element.tagName.toLowerCase() + classes.map(name => `.${esc(name)}`).join(''); + } + function selectorFor(element) { + if (element.id) return idPattern(element.id); + const parts = []; + let node = element; + while (node && node.nodeType === 1 && node.tagName.toLowerCase() !== 'html') { + let part = node.tagName.toLowerCase(); + const classes = stableClasses(node).slice(0, 2); + if (classes.length) part += classes.map(name => `.${esc(name)}`).join(''); + const parent = node.parentElement; + if (parent && parent.querySelectorAll(`:scope > ${part}`).length > 1) { + const siblings = [...parent.children].filter(child => child.matches(part)); + part += `:nth-of-type(${[...parent.children].indexOf(node) + 1})`; + } + parts.unshift(part); + const candidate = parts.join(' > '); + try { if (document.querySelectorAll(candidate).length === 1 || parts.length >= 4) return candidate; } catch (_) {} + node = parent; + } + return parts.join(' > '); + } + function relativeSelector(element, container) { + if (element === container) return ':scope'; + const parts = []; + let node = element; + while (node && node !== container) { + let part = simpleSelector(node); + if (!node.id) { + const parent = node.parentElement; + if (parent) { + const matches = [...parent.children].filter(child => child.matches(part)); + if (matches.length > 1) part += `:nth-of-type(${[...parent.children].indexOf(node) + 1})`; + } + } + parts.unshift(part); node = node.parentElement; + } + return parts.join(' > '); + } + function repeatedAncestor(element) { + let node = element; + let best = null; + let depth = 0; + while (node && node.tagName && node.tagName.toLowerCase() !== 'body') { + const selector = simpleSelector(node); + let count = 0; + try { count = document.querySelectorAll(selector).length; } catch (_) {} + if (count >= 2 && count <= 300) { + const text = node.textContent.trim(); + const hasUsefulText = text.length >= 10; + const score = (hasUsefulText ? 30 : 0) + Math.min(count, 20) - depth * 4 - (node.querySelectorAll('*').length > 300 ? 20 : 0); + if (!best || score > best.score) best = { node, selector, count, score }; + } + node = node.parentElement; depth += 1; + } + return best; + } + document.addEventListener('mouseover', event => { + event.preventDefault(); event.stopPropagation(); + if (hovered) hovered.classList.remove('w2d-hover'); + hovered = event.target; hovered.classList.add('w2d-hover'); + }, true); + document.addEventListener('mouseout', event => { if (hovered) hovered.classList.remove('w2d-hover'); }, true); + document.addEventListener('click', event => { + event.preventDefault(); event.stopPropagation(); + if (picked) picked.classList.remove('w2d-picked'); + picked = event.target; picked.classList.add('w2d-picked'); + const selector = selectorFor(picked); + const repeated = repeatedAncestor(picked); + parent.postMessage({ type: 'what2do-selector', selector, selectorCount: document.querySelectorAll(selector).length, generalizedSelector: simpleSelector(picked), generalizedCount: document.querySelectorAll(simpleSelector(picked)).length, containerSelector: repeated?.selector || '', containerCount: repeated?.count || 0, relativeSelector: repeated ? relativeSelector(picked, repeated.node) : selector, tag: picked.tagName.toLowerCase(), text: picked.textContent.trim().slice(0, 160), href: picked.getAttribute('href') }, location.origin); + }, true); +})();''' + +@app.post("/picker/snapshots") +async def create_picker_snapshot(request: Request): + form = await request.form() + url = form.get("url", "").strip() + try: + sanitized = sanitize_picker_html(fetch_html(url), url) + except Exception as exc: + return JSONResponse({"error": str(exc)}, status_code=400) + token = secrets.token_urlsafe(24) + now = datetime.now(timezone.utc) + with PICKER_LOCK: + expired = [key for key, value in PICKER_SNAPSHOTS.items() if now - value["created_at"] > PICKER_TTL] + for key in expired: PICKER_SNAPSHOTS.pop(key, None) + PICKER_SNAPSHOTS[token] = {"html": sanitized, "created_at": now} + return {"preview_url": f"/picker/snapshots/{token}"} + +@app.get("/picker/snapshots/{token}", response_class=HTMLResponse) +def picker_snapshot(token: str): + with PICKER_LOCK: + snapshot = PICKER_SNAPSHOTS.get(token) + if not snapshot or datetime.now(timezone.utc) - snapshot["created_at"] > PICKER_TTL: + raise HTTPException(404, "Picker snapshot expired") + return HTMLResponse(snapshot["html"], headers={"Content-Security-Policy": "default-src 'none'; img-src https: http: data:; style-src 'unsafe-inline' https:; font-src https: data:; script-src 'unsafe-inline'; connect-src 'none'; frame-src 'none'; form-action 'none'; base-uri 'none'", "Cache-Control": "no-store", "X-Content-Type-Options": "nosniff"}) + +@app.post("/auto-detect", response_class=HTMLResponse) +async def auto_detect(request: Request): + form = await request.form() + url = form.get("url", "").strip() + source = { + "id": form.get("source_id"), "name": form.get("name", ""), "url": url, + "venue_name": form.get("venue_name", ""), "venue_address": form.get("venue_address", ""), + "latitude": form.get("latitude") or None, "longitude": form.get("longitude") or None, + "timezone": form.get("timezone", "Australia/Sydney"), "enabled": 1, + "crawl_interval_minutes": form.get("crawl_interval_minutes", 360), + } + try: + html = fetch_html(url) + rules, items, message = detect_rules(html, url, source["timezone"]) + strategy = {"mode": "html", "request_url": url, "method": "GET", "response_type": "html", **rules, "link_attribute": "href"} + source["strategy_json"] = json.dumps(strategy, indent=2) + with _http_client(timeout=45) as client: + strategy_items = execute_strategy({**source, "strategy_json": source["strategy_json"]}, client) + if strategy_items: + items = strategy_items + inferred = infer_page_fields(html, url) + for key, value in inferred.items(): + if value and not source.get(key): source[key] = value + source.update(rules) + if source.get("venue_address") and not source.get("latitude"): + point = geocode(source["venue_address"]) + if point: source["latitude"], source["longitude"] = point[:2] + preview_data = [{"title": item.title, "starts_at": item.starts_at.isoformat(), "ends_at": item.ends_at.isoformat() if item.ends_at else None, "url": item.url} for item in items[:20]] + error = None + except Exception as exc: + source.update({"item_selector": "", "title_selector": "", "date_selector": "", "time_selector": "", "link_selector": "", "description_selector": "", "location_selector": "", "date_format": "", "recurring_weekly": 0, "strategy_json": "{}"}) + preview_data, message, error = [], None, str(exc) + action = f"/sources/{source['id']}" if source.get("id") else "/sources" + return TEMPLATES.TemplateResponse(request, "source_form.html", {"source": source, "action": action, "preview": preview_data, "preview_error": error, "detect_message": message}) + +@app.post("/preview", response_class=HTMLResponse) +async def preview(request: Request): + form = await request.form(); values = source_values(form); values["url"] = form.get("url", "") + try: + with _http_client(timeout=45) as client: + items = execute_strategy(values, client)[:20] + preview_data = [{"title": item.title, "starts_at": item.starts_at.isoformat(), "ends_at": item.ends_at.isoformat() if item.ends_at else None, "url": item.url} for item in items] + error = None + except Exception as exc: preview_data, error = [], str(exc) + source_id = form.get("source_id") + return TEMPLATES.TemplateResponse(request, "source_form.html", {"source": values, "action": f"/sources/{source_id}" if source_id else "/sources", "preview": preview_data, "preview_error": error}) diff --git a/app/static/app.js b/app/static/app.js new file mode 100644 index 0000000..9bacf62 --- /dev/null +++ b/app/static/app.js @@ -0,0 +1,132 @@ +const locationInput = document.querySelector('#location-input'); +const latitudeInput = document.querySelector('#latitude-input'); +const longitudeInput = document.querySelector('#longitude-input'); +const options = document.querySelector('#location-options'); +let locationTimer; + +if (locationInput) { + locationInput.addEventListener('input', () => { + latitudeInput.value = ''; + longitudeInput.value = ''; + clearTimeout(locationTimer); + const query = locationInput.value.trim(); + if (query.length < 3) { options.hidden = true; return; } + locationTimer = setTimeout(async () => { + try { + const response = await fetch(`/api/locations?q=${encodeURIComponent(query)}`); + const places = await response.json(); + options.replaceChildren(); + places.forEach(place => { + const button = document.createElement('button'); + button.type = 'button'; + button.textContent = place.label; + button.addEventListener('click', () => { + locationInput.value = place.label; + latitudeInput.value = place.latitude; + longitudeInput.value = place.longitude; + options.hidden = true; + }); + options.append(button); + }); + if (places.length === 0) { + const message = document.createElement('div'); + message.className = 'location-message'; + message.textContent = 'No geographic locations found. Try a suburb, town, or postcode.'; + options.append(message); + } + options.hidden = false; + } catch (_) { options.hidden = true; } + }, 300); + }); + document.addEventListener('click', event => { if (!event.target.closest('.location-field')) options.hidden = true; }); +} + +const tagToggle = document.querySelector('#tag-toggle'); +const tagOptions = document.querySelector('#tag-options'); +if (tagToggle && tagOptions) { + tagToggle.addEventListener('click', () => { tagOptions.hidden = !tagOptions.hidden; }); + tagOptions.addEventListener('change', () => { + const count = tagOptions.querySelectorAll('input:checked').length; + tagToggle.textContent = count ? `${count} selected` : 'All types'; + }); + document.addEventListener('click', event => { + if (!event.target.closest('.tag-field')) tagOptions.hidden = true; + }); +} + +const tabs = document.querySelectorAll('.tab'); +let map; +tabs.forEach(tab => tab.addEventListener('click', () => { + tabs.forEach(item => item.classList.toggle('active', item === tab)); + document.querySelectorAll('.view-panel').forEach(panel => panel.classList.toggle('active', panel.id === tab.dataset.view)); + if (tab.dataset.view === 'map-view') initialiseMap(); +})); + +const eventRows = [...document.querySelectorAll('[data-event-start]')]; +const visibleCount = document.querySelector('#visible-event-count'); +const clearDay = document.querySelector('#clear-day'); +const dayEmpty = document.querySelector('#day-empty'); +const rangeStatus = document.querySelector('#range-status'); +let rangeStart = ''; +let rangeEnd = ''; + +function formatDay(dayKey) { + return new Date(`${dayKey}T00:00:00`).toLocaleDateString(undefined, { day: 'numeric', month: 'short' }); +} + +function filterToRange(startDay, endDay = startDay) { + let count = 0; + eventRows.forEach(row => { + row.hidden = Boolean(startDay) && (row.dataset.eventStart > endDay || row.dataset.eventEnd < startDay); + if (!row.hidden) count += 1; + }); + document.querySelectorAll('.calendar-day').forEach(day => { + const inRange = Boolean(startDay) && day.dataset.day >= startDay && day.dataset.day <= endDay; + day.classList.toggle('selected', inRange); + day.classList.toggle('range-start', day.dataset.day === startDay); + day.classList.toggle('range-end', day.dataset.day === endDay && endDay !== startDay); + }); + if (visibleCount) visibleCount.textContent = count; + if (clearDay) clearDay.hidden = !startDay; + if (dayEmpty) dayEmpty.hidden = count !== 0; +} + +document.querySelectorAll('.calendar-day').forEach(day => day.addEventListener('click', event => { + event.preventDefault(); + const clicked = day.dataset.day; + if (!rangeStart || rangeEnd) { + rangeStart = clicked; + rangeEnd = ''; + filterToRange(rangeStart); + if (rangeStatus) rangeStatus.textContent = `${formatDay(rangeStart)} selected · choose an end date`; + } else { + rangeEnd = clicked; + if (rangeEnd < rangeStart) [rangeStart, rangeEnd] = [rangeEnd, rangeStart]; + filterToRange(rangeStart, rangeEnd); + if (rangeStatus) rangeStatus.textContent = `${formatDay(rangeStart)} – ${formatDay(rangeEnd)}`; + document.querySelector('#event-list')?.scrollIntoView({ behavior: 'smooth', block: 'start' }); + } +})); +if (clearDay) clearDay.addEventListener('click', event => { + event.preventDefault(); rangeStart = ''; rangeEnd = ''; filterToRange(''); + if (rangeStatus) rangeStatus.textContent = 'Select a start date'; +}); + +function initialiseMap() { + if (map || !window.maplibregl) return; + const events = JSON.parse(document.querySelector('#map-events').textContent); + const center = events.length ? [events[0].longitude, events[0].latitude] : [150.8931, -34.4278]; + map = new maplibregl.Map({ container: 'events-map', style: { version: 8, sources: { osm: { type: 'raster', tiles: ['https://tile.openstreetmap.org/{z}/{x}/{y}.png'], tileSize: 256, attribution: '© OpenStreetMap contributors' } }, layers: [{ id: 'osm', type: 'raster', source: 'osm' }] }, center, zoom: 12 }); + map.addControl(new maplibregl.NavigationControl(), 'top-right'); + const bounds = new maplibregl.LngLatBounds(); + events.forEach(event => { + const popup = new maplibregl.Popup({ offset: 20 }).setHTML(`${escapeHtml(event.title)}
${escapeHtml(event.venue)}
View event`); + new maplibregl.Marker({ color: '#dd5b32' }).setLngLat([event.longitude, event.latitude]).setPopup(popup).addTo(map); + bounds.extend([event.longitude, event.latitude]); + }); + if (events.length > 1) map.fitBounds(bounds, { padding: 60, maxZoom: 14 }); +} + +function escapeHtml(value) { + const node = document.createElement('div'); node.textContent = value; return node.innerHTML; +} diff --git a/app/static/event.js b/app/static/event.js new file mode 100644 index 0000000..e9add51 --- /dev/null +++ b/app/static/event.js @@ -0,0 +1,16 @@ +const mapNode = document.querySelector('#event-map'); +if (mapNode && window.maplibregl) { + const longitude = Number(mapNode.dataset.longitude); + const latitude = Number(mapNode.dataset.latitude); + const map = new maplibregl.Map({ container: mapNode, style: { version: 8, sources: { osm: { type: 'raster', tiles: ['https://tile.openstreetmap.org/{z}/{x}/{y}.png'], tileSize: 256, attribution: '© OpenStreetMap contributors' } }, layers: [{ id: 'osm', type: 'raster', source: 'osm' }] }, center: [longitude, latitude], zoom: 15 }); + map.addControl(new maplibregl.NavigationControl(), 'top-right'); + new maplibregl.Marker({ color: '#dd5b32' }).setLngLat([longitude, latitude]).setPopup(new maplibregl.Popup({ offset: 20 }).setText(mapNode.dataset.title)).addTo(map); +} + +const shareButton = document.querySelector('#share-button'); +if (shareButton) shareButton.addEventListener('click', async () => { + try { + if (navigator.share) await navigator.share({ title: document.title, url: location.href }); + else { await navigator.clipboard.writeText(location.href); shareButton.textContent = 'Link copied'; } + } catch (_) {} +}); diff --git a/app/static/source.js b/app/static/source.js new file mode 100644 index 0000000..613f70e --- /dev/null +++ b/app/static/source.js @@ -0,0 +1,118 @@ +const pickerFrame = document.querySelector('#selector-picker'); +const loadPicker = document.querySelector('#load-picker'); +const pickerStatus = document.querySelector('#picker-status'); +const pickerSelection = document.querySelector('#picker-selection'); +const pickedSelector = document.querySelector('#picked-selector'); +const pickedText = document.querySelector('#picked-text'); +const pickedCount = document.querySelector('#picked-count'); +const pickedContainer = document.querySelector('#picked-container'); +const pickedRelative = document.querySelector('#picked-relative'); +const pickerTarget = document.querySelector('#picker-target'); +const applySelector = document.querySelector('#apply-selector'); +const applyPattern = document.querySelector('#apply-pattern'); +let currentSelection = null; + +if (loadPicker && pickerFrame) loadPicker.addEventListener('click', async () => { + const url = document.querySelector('input[name="url"]')?.value.trim(); + if (!url) { pickerStatus.textContent = 'Enter a page URL first.'; return; } + loadPicker.disabled = true; pickerStatus.textContent = 'Downloading and sanitizing page…'; + try { + const body = new FormData(); body.append('url', url); + const response = await fetch('/picker/snapshots', { method: 'POST', body }); + const result = await response.json(); + if (!response.ok) throw new Error(result.error || 'Unable to create preview'); + pickerFrame.src = result.preview_url; pickerFrame.hidden = false; + pickerStatus.textContent = 'Hover to highlight, then click an element.'; + } catch (error) { pickerStatus.textContent = error.message; } + finally { loadPicker.disabled = false; } +}); + +window.addEventListener('message', event => { + if (!pickerFrame || event.source !== pickerFrame.contentWindow || event.data?.type !== 'what2do-selector') return; + currentSelection = event.data; + pickedSelector.textContent = event.data.selector; + pickedCount.textContent = `${event.data.selectorCount} exact match${event.data.selectorCount === 1 ? '' : 'es'}; ${event.data.generalizedCount} similar elements`; + pickedContainer.textContent = event.data.containerSelector || 'No repeated ancestor detected'; + pickedRelative.textContent = event.data.containerSelector ? `${event.data.containerCount} containers · child: ${event.data.relativeSelector}` : ''; + pickedText.textContent = event.data.text || event.data.tag; + applyPattern.disabled = !event.data.containerSelector; + pickerSelection.hidden = false; +}); + +function updateSelectorField(target, selector) { + const input = document.querySelector(`[name="${target}"]`); + if (input) input.value = selector; + try { + const strategy = JSON.parse(strategyJson.value || '{}'); + if ((strategy.mode || 'html') === 'html') { + strategy[target] = selector; + strategyJson.value = JSON.stringify(strategy, null, 2); + } + } catch (_) {} +} + +if (applySelector) applySelector.addEventListener('click', () => { + if (!currentSelection) return; + updateSelectorField(pickerTarget.value, currentSelection.selector); + pickerStatus.textContent = `Applied exact selector to ${pickerTarget.options[pickerTarget.selectedIndex].text}.`; +}); + +if (applyPattern) applyPattern.addEventListener('click', () => { + if (!currentSelection?.containerSelector) return; + const target = pickerTarget.value; + updateSelectorField('item_selector', currentSelection.containerSelector); + updateSelectorField(target, target === 'item_selector' ? currentSelection.containerSelector : currentSelection.relativeSelector); + pickerStatus.textContent = `Matched ${currentSelection.containerCount} event containers and applied a relative ${pickerTarget.options[pickerTarget.selectedIndex].text.toLowerCase()} selector.`; +}); + +const strategyTemplates = { + html: { mode: 'html', request_url: 'https://example.org/events', method: 'GET', item_selector: '.event-card', title_selector: '.title', date_selector: '.date', time_selector: '.time', link_selector: 'a', link_attribute: 'href', description_selector: '.description', location_selector: '.venue', date_format: '' }, + json: { mode: 'json', request_url: 'https://example.org/api/events', method: 'GET', items_path: 'results', title_path: 'title', date_path: 'startDate', end_path: 'endDate', description_path: 'description', location_path: 'location.address', link_path: 'url', disabled_path: 'disabled' }, + json_join: { mode: 'json', request_url: 'https://example.org/api/sessions', method: 'GET', items_path: '', date_path: 'startDate', link_path: 'url', join_item_key: 'itemId', join_title_path: 'name', join_description_path: 'description', join_duration_path: 'duration', join: { request_url: 'https://example.org/api/items', method: 'GET', items_path: '', key_path: 'id' } }, + html_fragment: { mode: 'html_fragment', request_url: 'https://example.org/ajax', method: 'POST', body: { action: 'events' }, html_path: 'html', item_selector: '.event-card', title_selector: '.title', date_selector: '.date', link_selector: 'a', link_attribute: 'href', page_parameter: 'page', page_start: 1, max_pages: 10, total_pages_path: 'pages_total' } +}; +const strategyTemplate = document.querySelector('#strategy-template'); +const strategyJson = document.querySelector('#strategy-json'); +if (strategyTemplate && strategyJson) strategyTemplate.addEventListener('change', () => { + const template = strategyTemplates[strategyTemplate.value]; + if (template) strategyJson.value = JSON.stringify(template, null, 2); +}); + +const addressInput = document.querySelector('#venue-address'); +const addressOptions = document.querySelector('#address-options'); +const latitudeInput = document.querySelector('#venue-latitude'); +const longitudeInput = document.querySelector('#venue-longitude'); +let addressTimer; + +if (addressInput && addressOptions) { + addressInput.addEventListener('input', () => { + clearTimeout(addressTimer); + latitudeInput.value = ''; + longitudeInput.value = ''; + const query = addressInput.value.trim(); + if (query.length < 3) { addressOptions.hidden = true; return; } + addressTimer = setTimeout(async () => { + try { + const response = await fetch(`/api/addresses?q=${encodeURIComponent(query)}`); + const addresses = await response.json(); + addressOptions.replaceChildren(); + addresses.forEach(address => { + const button = document.createElement('button'); + button.type = 'button'; + button.textContent = address.label; + button.addEventListener('click', () => { + addressInput.value = address.label; + latitudeInput.value = address.latitude; + longitudeInput.value = address.longitude; + addressOptions.hidden = true; + }); + addressOptions.append(button); + }); + addressOptions.hidden = addresses.length === 0; + } catch (_) { addressOptions.hidden = true; } + }, 300); + }); + document.addEventListener('click', event => { + if (!event.target.closest('.address-field')) addressOptions.hidden = true; + }); +} diff --git a/app/static/style.css b/app/static/style.css new file mode 100644 index 0000000..853741b --- /dev/null +++ b/app/static/style.css @@ -0,0 +1 @@ +:root{--ink:#17211b;--muted:#657068;--paper:#f4f0e7;--card:#fffdf7;--accent:#dd5b32;--green:#234c3c;--line:#d9d4c8}*{box-sizing:border-box}body{margin:0;background:var(--paper);color:var(--ink);font:16px/1.5 system-ui,-apple-system,sans-serif}a{color:var(--green)}header{height:68px;padding:0 max(24px,calc((100% - 1120px)/2));display:flex;align-items:center;justify-content:space-between;border-bottom:1px solid var(--line)}nav{display:flex;gap:24px}nav a{text-decoration:none;font-weight:650}.brand{font:bold 24px Georgia,serif;text-decoration:none;color:var(--ink)}main{max-width:1120px;margin:auto;padding:48px 24px 90px}h1,h2,h3{font-family:Georgia,serif;line-height:1.1;margin:.25em 0}h1{font-size:clamp(2.5rem,6vw,5.5rem);max-width:850px}h2{font-size:2rem}.hero{padding:45px 0}.hero p:not(.eyebrow){font-size:1.2rem;color:var(--muted)}.eyebrow{font-size:.75rem;font-weight:800;letter-spacing:.16em;color:var(--accent)}.card{background:var(--card);border:1px solid var(--line);border-radius:12px;box-shadow:0 5px 18px #17211b0a}.search{display:grid;grid-template-columns:2fr 1.4fr 1.4fr .8fr auto;gap:12px;padding:18px;align-items:end}label{display:flex;flex-direction:column;gap:6px;font-size:.8rem;font-weight:750;position:relative}input,select{width:100%;border:1px solid var(--line);background:white;border-radius:7px;padding:12px;color:var(--ink);font:inherit}.suffix{position:absolute;right:12px;bottom:12px;color:var(--muted)}button,.button{border:0;border-radius:7px;padding:12px 18px;background:var(--green);color:white;text-decoration:none;font-weight:750;font:inherit;cursor:pointer;display:inline-block}.secondary{background:transparent;color:var(--green);border:1px solid var(--green)}.danger{background:#9d302c}.section-head{display:flex;justify-content:space-between;align-items:end;margin:52px 0 22px;gap:20px}.section-head h1{font-size:3.5rem}.section-head p{max-width:650px;color:var(--muted)}.events,.source-grid{display:grid;gap:14px}.event{padding:24px}.event h3{font-size:1.55rem}.event p{color:var(--muted)}.date{color:var(--accent);font-weight:800}.event-foot,.source-title,.actions{display:flex;align-items:center;justify-content:space-between;gap:12px;margin-top:22px}.pill{font-size:.75rem;background:#eee9de;padding:5px 9px;border-radius:99px}.good{background:#dce9df;color:#285039}.empty{text-align:center;padding:60px}.source{padding:24px}.source-stats{display:flex;gap:30px;flex-wrap:wrap;border-top:1px solid var(--line);border-bottom:1px solid var(--line);padding:15px 0;margin-top:18px}.source-stats>div{display:grid}.source-stats small{color:var(--muted)}.admin-events{display:grid;gap:12px}.admin-event{padding:22px;display:flex;align-items:center;justify-content:space-between;gap:20px}.admin-event.is-hidden{opacity:.6;border-style:dashed}.event-admin-form{padding:25px;display:grid;gap:18px}textarea{width:100%;border:1px solid var(--line);background:white;border-radius:7px;padding:12px;color:var(--ink);font:inherit;resize:vertical}.source-title{margin-top:0}.source-title a{word-break:break-all;font-size:.85rem}dl{display:flex;gap:40px;border-top:1px solid var(--line);border-bottom:1px solid var(--line);padding:15px 0}dt{color:var(--muted);font-size:.75rem}dd{margin:2px 0;font-weight:700}.actions{justify-content:flex-start}.actions form{margin:0}.notice{padding:12px 16px;border-radius:7px}.error{background:#f4deda;color:#792a27}.success{background:#dce9df;color:#285039}.url-detect-row{display:grid;grid-template-columns:minmax(0,1fr) auto;gap:8px}.url-detect-row .detect-submit{width:auto;white-space:nowrap;background:var(--green);color:white;border:0;font-weight:700}.address-field{position:relative}.picker-panel{grid-column:1/-1;padding:25px;display:grid;gap:14px}.picker-controls{display:flex;align-items:end;gap:10px}.picker-controls .field{min-width:220px}.selector-picker{width:100%;height:620px;border:1px solid var(--line);border-radius:8px;background:white}.picker-selection{display:grid;grid-template-columns:minmax(0,1fr) minmax(0,1fr);align-items:start;gap:14px;padding:14px;background:#eee9de;border-radius:8px}.picker-selection>div{display:grid;gap:5px}.picker-selection code{overflow-wrap:anywhere}.picker-selection span{color:var(--muted);font-size:.8rem;overflow:hidden;text-overflow:ellipsis}.picker-actions{display:flex!important;flex-direction:row;flex-wrap:wrap}.picker-actions button{padding:9px 12px}.rules-details{grid-column:1/-1;overflow:hidden}.rules-details>summary{cursor:pointer;padding:22px;font:700 1.5rem Georgia,serif;display:flex;justify-content:space-between;gap:15px}.rules-details>summary span{font:400 .8rem system-ui;color:var(--muted)}.rules-details[open]>summary{border-bottom:1px solid var(--line)}.sticky-actions input[type=submit]{width:auto;background:var(--green);color:white;border:0;font-weight:700;cursor:pointer}.sticky-actions input.primary{background:var(--accent)}.config-grid{display:grid;grid-template-columns:1fr 1fr;gap:18px;align-items:start}.form-section{padding:25px;display:grid;gap:17px}.form-heading{font:700 2rem/1.1 Georgia,serif;border-bottom:1px solid var(--line);padding-bottom:14px}.form-title{font:700 3.5rem/1.1 Georgia,serif;margin:.25em 0}.muted{color:var(--muted)}.field{display:flex;flex-direction:column;gap:6px;font-size:.8rem;font-weight:750}.two{display:grid;grid-template-columns:1fr 1fr;gap:14px}.check{flex-direction:row;align-items:center}.check input{width:auto}.form-section small{font-weight:400;color:var(--muted)}.sticky-actions{grid-column:1/-1;display:flex;justify-content:flex-end;gap:10px}.primary{background:var(--accent)}.preview{margin-top:24px;padding:25px}.preview div{display:flex;justify-content:space-between;padding:10px;border-top:1px solid var(--line);gap:20px}.preview span{font-size:.8rem;color:var(--muted);white-space:nowrap}footer{text-align:center;border-top:1px solid var(--line);padding:30px;color:var(--muted);font-size:.8rem}.hero-title{font:700 clamp(2.5rem,6vw,5.5rem)/1.05 Georgia,serif;max-width:850px;margin:.12em 0}.hero-copy{font-size:1.2rem;color:var(--muted)}.section-title{font:700 2rem/1.1 Georgia,serif;margin-top:5px}.location-field{position:relative}.location-options{position:absolute;z-index:20;top:100%;left:0;right:0;background:white;border:1px solid var(--line);border-radius:8px;box-shadow:0 12px 30px #17211b26;overflow:hidden}.location-options button{display:block;width:100%;text-align:left;border-radius:0;background:white;color:var(--ink);font-size:.85rem;border-bottom:1px solid var(--line)}.location-options button:hover{background:#eee9de}.location-message{padding:12px;color:var(--muted);font-size:.8rem;font-weight:400}.tag-field{position:relative}.tag-toggle{width:100%;text-align:left;background:white;color:var(--ink);border:1px solid var(--line);font-weight:400}.tag-options{position:absolute;z-index:19;top:100%;left:0;right:0;min-width:190px;background:white;border:1px solid var(--line);border-radius:8px;box-shadow:0 12px 30px #17211b26;padding:8px}.tag-option{display:flex;flex-direction:row;align-items:center;gap:8px;padding:7px;border-radius:5px;cursor:pointer}.tag-option:hover{background:#eee9de}.tag-option input{width:auto}.range-field{position:relative}.view-tabs{display:flex;gap:8px;margin-bottom:16px}.tab{background:transparent;color:var(--green);border:1px solid var(--green)}.tab.active{background:var(--green);color:white}.view-panel{display:none}.view-panel.active{display:block}.search-filters{grid-template-columns:1.8fr 1.35fr 1fr .65fr auto;position:relative}.date-calendar{overflow:hidden;margin-bottom:18px}.calendar{overflow:hidden}.calendar-heading{font:700 1.6rem Georgia,serif;padding:22px}.calendar-tools{display:flex;align-items:center;justify-content:space-between;gap:12px;padding:0 22px 14px;color:var(--muted);font-size:.8rem}.clear-day{background:transparent;color:var(--green);border:1px solid var(--green);padding:7px 10px;font-size:.75rem}.weekdays,.calendar-grid{display:grid;grid-template-columns:repeat(7,1fr)}.weekdays{border-top:1px solid var(--line);border-bottom:1px solid var(--line);padding:9px 0;color:var(--muted);font-size:.75rem;font-weight:700;text-align:center}.weekdays>input{display:block;min-width:0;border:0;border-radius:0;padding:0;background:transparent;color:var(--muted);font-size:inherit;font-weight:inherit;text-align:center;opacity:1}.calendar-day{min-width:0;min-height:82px;padding:8px;border:0;border-right:1px solid var(--line);border-bottom:1px solid var(--line);border-radius:0;background:transparent;color:var(--ink);text-align:left;display:flex;flex-direction:column;justify-content:space-between}.calendar-day:nth-child(7n){border-right:0}.calendar-day.outside{opacity:.35}.calendar-day.has-events{background:#fff4e8}.calendar-day.has-events:hover,.calendar-day.selected{background:#f4d6c6}.calendar-day.range-start,.calendar-day.range-end{box-shadow:inset 0 0 0 2px var(--accent)}.calendar-day b{font-size:.7rem;color:var(--accent)}.day-events{display:grid;gap:10px;margin-top:18px}.tag-list{display:flex;gap:6px;flex-wrap:wrap;margin-top:10px}[hidden]{display:none!important}.event-row{padding:18px 22px;text-decoration:none;color:var(--ink);display:flex;justify-content:space-between;align-items:center}.event-row:hover{border-color:var(--accent)}.event-title{font:700 1.25rem Georgia,serif;margin:5px 0}.map{height:580px}.back-link{display:inline-block;margin-bottom:34px}.event-detail{display:grid;grid-template-columns:1.1fr .9fr;gap:45px;align-items:start}.detail-title{font:700 clamp(2.5rem,5vw,4.8rem)/1.05 Georgia,serif;margin:10px 0}.detail-time{font:700 2rem Georgia,serif;color:var(--accent)}.description{font-size:1.1rem}.detail-card{padding:22px;margin:30px 0}.detail-field{display:grid;grid-template-columns:90px minmax(0,1fr);align-items:start;gap:14px;padding:13px 0;border-bottom:1px solid var(--line)}.detail-field:last-child{border-bottom:0}.detail-label{font-size:.75rem;color:var(--muted);font-weight:700;text-transform:uppercase;letter-spacing:.08em}.detail-value{font-weight:700;overflow-wrap:anywhere}.hint{font-size:.75rem;color:var(--muted);font-weight:400}.detail-actions{display:flex;gap:10px}.detail-map{height:520px;position:sticky;top:24px}@media(max-width:800px){.search,.search-filters,.config-grid,.event-detail,.url-detect-row,.picker-selection{grid-template-columns:1fr}.picker-controls{align-items:stretch;flex-direction:column}.selector-picker{height:480px}.two{grid-template-columns:1fr}.sticky-actions{grid-column:auto}.section-head{align-items:flex-start;flex-direction:column}.source-title,.event-foot,.admin-event{align-items:flex-start;flex-direction:column}dl{flex-wrap:wrap}.search button{width:100%}nav{gap:10px;font-size:.85rem}.calendar-day{min-height:72px;padding:7px}.calendar-day b{font-size:.6rem}.weekdays{font-size:.6rem}.map,.detail-map{height:420px}.event-detail{gap:25px}} diff --git a/app/strategies.py b/app/strategies.py new file mode 100644 index 0000000..dc56ef0 --- /dev/null +++ b/app/strategies.py @@ -0,0 +1,164 @@ +import json +from copy import deepcopy +from datetime import datetime, timedelta +from urllib.parse import urljoin +from zoneinfo import ZoneInfo + +from bs4 import BeautifulSoup + +from .crawler import BROWSER_HEADERS, CrawlError, ExtractedEvent, _extract_json_ld, _parse_date_range, _weekly_dates, validate_public_url + + +def parse_strategy(value) -> dict: + if isinstance(value, dict): + return deepcopy(value) + try: + return json.loads(value or "{}") + except (TypeError, json.JSONDecodeError): + return {} + + +def json_path(value, path: str, default=None): + if not path: + return value + current = value + for component in path.split("."): + if current is None: + return default + if isinstance(current, list): + try: current = current[int(component)] + except (ValueError, IndexError): return default + elif isinstance(current, dict): + current = current.get(component, default) + else: + return default + return current + + +def _value(node, selector: str, attribute: str = "") -> str: + if not selector: + return "" + target = node if selector == ":scope" else node.select_one(selector) + if not target: + return "" + if attribute: + return str(target.get(attribute, "")) + return target.get_text(" ", strip=True) + + +def _request(client, strategy: dict, page: int | None = None): + url = str(strategy.get("request_url") or strategy.get("url") or "") + validate_public_url(url) + method = strategy.get("method", "GET").upper() + params = deepcopy(strategy.get("params") or {}) + body = deepcopy(strategy.get("body") or {}) + page_parameter = strategy.get("page_parameter", "") + if page is not None and page_parameter: + (params if method == "GET" else body)[page_parameter] = str(page) + headers = {**BROWSER_HEADERS, **(strategy.get("headers") or {})} + response = client.request(method, url, params=params, data=body if method != "GET" else None, headers=headers) + response.raise_for_status() + if len(response.content) > int(strategy.get("max_bytes", 5_000_000)): + raise CrawlError("Strategy response exceeds configured size limit.") + return response + + +def _events_from_html(html: str, strategy: dict, source: dict) -> list[ExtractedEvent]: + soup = BeautifulSoup(html, "html.parser") + now = datetime.now(ZoneInfo(source.get("timezone") or "UTC")) + if strategy.get("item_selector") == "json-ld": + return _extract_json_ld(soup, source["url"], source, now) + events = [] + for item in soup.select(strategy.get("item_selector", "")): + title = _value(item, strategy.get("title_selector", ""), strategy.get("title_attribute", "")) + raw_date = _value(item, strategy.get("date_selector", ""), strategy.get("date_attribute", "")) + for old, new in (strategy.get("date_replacements") or {}).items(): raw_date = raw_date.replace(old, new) + raw_time = _value(item, strategy.get("time_selector", ""), strategy.get("time_attribute", "")) + if not title or not raw_date: continue + config = {**source, "date_format": strategy.get("date_format", "")} + if strategy.get("recurring_weekly"): + for start in _weekly_dates(raw_date, now): + events.append(ExtractedEvent(title=title, starts_at=start, description=_value(item, strategy.get("description_selector", ""), strategy.get("description_attribute", "")), url=urljoin(source["url"], _value(item, strategy.get("link_selector", ""), strategy.get("link_attribute", "href"))), address=_value(item, strategy.get("location_selector", ""), strategy.get("location_attribute", "")))) + continue + try: start, end = _parse_date_range(f"{raw_date} {raw_time}".strip(), config, now) + except (ValueError, OverflowError): continue + duration = strategy.get("duration_minutes") + if duration and not end: end = start + timedelta(minutes=int(duration)) + link = _value(item, strategy.get("link_selector", ""), strategy.get("link_attribute", "href")) + description = _value(item, strategy.get("description_selector", ""), strategy.get("description_attribute", "")) + location = _value(item, strategy.get("location_selector", ""), strategy.get("location_attribute", "")) + events.append(ExtractedEvent(title=title, starts_at=start, ends_at=end, description=description, url=urljoin(source["url"], link), address=location)) + return events + + +def _index_join(items: list, join_items: list, item_key: str, join_key: str) -> dict: + index = {} + for item in join_items: + keys = json_path(item, join_key, []) + if not isinstance(keys, list): keys = [keys] + for key in keys: + if key is not None: index[str(key)] = item + return index + + +def _events_from_json(payload, strategy: dict, source: dict, join_index: dict | None = None) -> list[ExtractedEvent]: + items = json_path(payload, strategy.get("items_path", ""), []) + if not isinstance(items, list): return [] + now = datetime.now(ZoneInfo(source.get("timezone") or "UTC")) + events = [] + for item in items: + if strategy.get("disabled_path") and json_path(item, strategy["disabled_path"]): continue + joined = join_index.get(str(json_path(item, strategy.get("join_item_key", "")))) if join_index else None + def field(name, default=""): + item_path = strategy.get(f"{name}_path", "") + join_path = strategy.get(f"join_{name}_path", "") + value = json_path(item, item_path, None) if item_path else None + if (value is None or value == "") and joined is not None and join_path: + value = json_path(joined, join_path, default) + return value if value is not None else default + title = str(field("title") or strategy.get("title_fallback", "Untitled event")) + date_value = str(field("date")) + if not date_value: continue + try: start, end = _parse_date_range(date_value, source, now) + except (ValueError, OverflowError): continue + end_value = field("end") + if end_value: + try: end, _ = _parse_date_range(str(end_value), source, now) + except (ValueError, OverflowError): pass + duration = field("duration") or strategy.get("duration_minutes") + if duration and not end: end = start + timedelta(minutes=int(duration)) + events.append(ExtractedEvent(title=title, starts_at=start, ends_at=end, description=str(field("description")), url=urljoin(source["url"], str(field("link"))), address=str(field("location")))) + return events + + +def execute_strategy(source: dict, client) -> list[ExtractedEvent]: + strategy = parse_strategy(source.get("strategy_json")) + mode = strategy.get("mode", "html") + if mode == "html": + response = _request(client, strategy) + return _events_from_html(response.text, strategy, source) + if mode not in {"json", "html_fragment"}: + raise CrawlError(f"Unsupported extraction mode: {mode}") + join_index = None + join = strategy.get("join") + if join: + join_response = _request(client, join) + join_payload = join_response.json() + join_items = json_path(join_payload, join.get("items_path", ""), []) + join_index = _index_join([], join_items if isinstance(join_items, list) else [], strategy.get("join_item_key", ""), join.get("key_path", "")) + events = [] + page = int(strategy.get("page_start", 1)) + max_pages = min(int(strategy.get("max_pages", 1)), 20) + for _ in range(max_pages): + response = _request(client, strategy, page) + payload = response.json() + if mode == "html_fragment": + fragment = json_path(payload, strategy.get("html_path", "html"), "") + page_events = _events_from_html(str(fragment), strategy, source) + else: + page_events = _events_from_json(payload, strategy, source, join_index) + events.extend(page_events) + total_pages = json_path(payload, strategy.get("total_pages_path", ""), None) + if not strategy.get("page_parameter") or not page_events or (total_pages is not None and page >= int(total_pages)): break + page += 1 + return events diff --git a/app/templates/base.html b/app/templates/base.html new file mode 100644 index 0000000..6fb1943 --- /dev/null +++ b/app/templates/base.html @@ -0,0 +1,5 @@ + + +{% block title %}What2Do{% endblock %}{% block head %}{% endblock %} +
What2Do
+
{% block content %}{% endblock %}
{% block scripts %}{% endblock %} diff --git a/app/templates/event.html b/app/templates/event.html new file mode 100644 index 0000000..f8e212a --- /dev/null +++ b/app/templates/event.html @@ -0,0 +1,22 @@ +{% extends "base.html" %} +{% block title %}{{ event.title }} · What2Do{% endblock %} +{% block head %}{% endblock %} +{% block content %} +← All events +
+
+
{{ event.display_date }}
+
{{ event.title }}
+
{{ event.display_time }}{% if event.display_end %} – {{ event.display_end }}{% endif %}
+ {% if event.description %}
{{ event.description }}
{% endif %} +
+
Venue
{{ event.venue_name }}
+
Address
{{ event.address }}
+
Source
Original event page ↗
+
+
Visit source
+
+
{% if event.latitude is not none %}
{% else %}
No map coordinates are available for this event.
{% endif %}
+
+{% endblock %} +{% block scripts %}{% endblock %} diff --git a/app/templates/event_form.html b/app/templates/event_form.html new file mode 100644 index 0000000..b341842 --- /dev/null +++ b/app/templates/event_form.html @@ -0,0 +1,6 @@ +{% extends "base.html" %} +{% block title %}Edit event · What2Do{% endblock %} +{% block content %} +
EDIT NORMALIZED EVENT
{{ event.title }}
Changes may be overwritten on the next crawl if the source still publishes different values.
← Source events
+
Title
Description
Starts at
Ends at
Venue name
Address
Latitude
Longitude
Event URL
+{% endblock %} diff --git a/app/templates/index.html b/app/templates/index.html new file mode 100644 index 0000000..1d8aa1c --- /dev/null +++ b/app/templates/index.html @@ -0,0 +1,31 @@ +{% extends "base.html" %} +{% block title %}Find events · What2Do{% endblock %} +{% block head %}{% endblock %} +{% block content %} +
LOCAL EVENT INDEX
Find something worth going out for.
Search events gathered from local venue websites.
+ +{% if location and not geocoded %}
That location could not be resolved. Select an autocomplete suggestion to apply distance filtering.
{% endif %} +
WHAT'S ON
{{ events|length }} events
+ Add source
+
+
+
+
Choose a range · {{ calendar_month }}
+
Select a start date
+
+
{% for day in calendar_days %}{% endfor %}
+
+
+ {% for event in events %}
{{ event.display_time }}
{{ event.title }}
{{ event.venue_name }} · {{ event.address }}
{% for event_tag in event.tags %}{{ event_tag }}{% endfor %}
View →
{% else %}
No events match these filters.
{% endfor %} + +
+
+
+ +{% endblock %} +{% block scripts %}{% endblock %} diff --git a/app/templates/source_events.html b/app/templates/source_events.html new file mode 100644 index 0000000..3b49c76 --- /dev/null +++ b/app/templates/source_events.html @@ -0,0 +1,6 @@ +{% extends "base.html" %} +{% block title %}{{ source.name }} events · What2Do{% endblock %} +{% block content %} +
SOURCE EVENTS
{{ source.name }}
Edit normalized events or hide them from public search and maps.
← Sources
+
{% for event in events %}
{{ event.starts_at }}{% if event.ends_at %} → {{ event.ends_at }}{% endif %}

{{ event.title }}

{{ event.venue_name }} · {{ event.address }}
{% if event.hidden %}Hidden{% endif %}
Edit
Public page
{% else %}
No events have been crawled for this source.
{% endfor %}
+{% endblock %} diff --git a/app/templates/source_form.html b/app/templates/source_form.html new file mode 100644 index 0000000..9935236 --- /dev/null +++ b/app/templates/source_form.html @@ -0,0 +1,27 @@ +{% extends "base.html" %} +{% block title %}Configure source · What2Do{% endblock %} +{% block content %} +
SOURCE CONFIGURATION
Configure website
Auto-detect standard HTML or configure a reusable extraction strategy.
← Sources
+{% if detect_message %}
{{ detect_message }} Review the strategy below before saving.
{% endif %} +{% if preview_error %}
Detection or preview failed: {{ preview_error }}
{% endif %} +
+
Website and venue
+
Source name
+
Page URL
+
Venue name
+
Venue address
+
Latitude
Longitude
+
Timezone
+
Visual selector picker
Loads a sanitized, non-interactive copy of the downloaded HTML. Dynamic JavaScript content may not appear.
+
Extraction strategy — advanced
+
HTML mode uses the convenient selector fields below. JSON and HTML-fragment modes use the generic JSON strategy directly.
+
Event container selector
Title selector
Date or range selector
Time selector
Link selector
Description selector
Location selector
Date format
+ +
Strategy template
Persisted strategy JSON
Supports request configuration, dotted JSON paths, CSS selectors and attributes, pagination, joins, and response limits. See STRATEGIES.md.
+
+
Scheduling
Crawl every (minutes)
+
{% if source.get('id') %}{% endif %}
+ +{% if preview is not none %}
Preview ({{ preview|length }})
{% for event in preview %}
{{ event.title }}{{ event.starts_at }}{% if event.ends_at %} → {{ event.ends_at }}{% endif %}
{% else %}

No events matched this strategy.

{% endfor %}
{% endif %} +{% endblock %} +{% block scripts %}{% endblock %} diff --git a/app/templates/sources.html b/app/templates/sources.html new file mode 100644 index 0000000..9c9e97c --- /dev/null +++ b/app/templates/sources.html @@ -0,0 +1,6 @@ +{% extends "base.html" %} +{% block title %}Sources · What2Do{% endblock %} +{% block content %} +
CRAWLER ADMIN
Sources
Configure crawlers and manage their normalized events.
+ Add website
+
{% for source in sources %}

{{ source.name }}

{{ source.url }}
{{ 'Enabled' if source.enabled else 'Paused' }}
Venuesmall>{{ source.venue_name }}
Events{{ source.event_count }}
Last result{{ source.last_status or 'Not crawled' }}
{% if source.last_error %}
{{ source.last_error }}
{% endif %}
ConfigureManage events
{% endfor %}
+{% endblock %} diff --git a/compose.yaml b/compose.yaml new file mode 100644 index 0000000..6a221a8 --- /dev/null +++ b/compose.yaml @@ -0,0 +1,16 @@ +services: + web: + build: . + ports: + - "8000:8000" + environment: + DATABASE_PATH: /data/what2do.db + PUBLIC_BASE_URL: http://localhost:8000 + CRAWL_INTERVAL_MINUTES: "60" + NOMINATIM_USER_AGENT: what2do-local/0.1 (set-a-contact@example.com) + volumes: + - what2do-data:/data + restart: unless-stopped + +volumes: + what2do-data: diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..77b8877 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,7 @@ +fastapi==0.116.1 +uvicorn==0.35.0 +httpx==0.28.1 +beautifulsoup4==4.13.4 +python-multipart==0.0.20 +jinja2==3.1.6 +python-dateutil==2.9.0.post0