# swehockey-schedule A deterministic crawler for the **Schedule and Results** table on [stats.swehockey.se](https://stats.swehockey.se) — one plain HTTP GET per season, no browser, no JavaScript, no waiting, and a byte-identical output for identical input. ```console $ go build -o swehockey . $ ./swehockey schedule -season 20961 -format csv | head -3 season_id,game_id,event_id,date,time,home,away,home_goals,away_goals,periods,spectators,venue 20961,90001001,1109941,2026-09-19,15:15,Frölunda HC,Växjö Lakers HC,5,4,1:1-1 2:2-1 3:2-2,12044,Scandinavium 20961,90001002,1109942,2026-09-19,15:15,HV 71,IF Malmö Redhawks,4,5,1:1-0 2:1-1 3:1-3 4:1-1,6800,Husqvarna Garden ``` ```console $ ./swehockey crawl -league 20961 # every season of the SHL, as NDJSON $ ./swehockey leagues # 119 competitions and their ids $ ./swehockey schedule -fixture testdata/schedule_20961.html # offline ``` ## How determinism is achieved The site's page is server-rendered: `curl` returns all 364 rows of the current season in one response. That removes the usual sources of flakiness (timers, `waitForSelector`, lazy rendering) and leaves exactly one risk — that the markup changes under you. The rest of the design is about that risk. 1. **One GET, no retries, no cookies, no state.** Same HTML in, same bytes out. The output contains only page-derived data: no fetch timestamp, no map iteration, fixed field order. `TestRenderIsPure` and the golden files enforce this. 2. **The table is found by its header signature**, not by a CSS path or a position in the document. Ads, navigation and layout changes cannot move it out from under the crawler. 3. **A declared contract per layout, not a pile of indices.** The site has shipped more than one version of this table, so each version is a profile in `layouts[]` (`parse.go`), identified by the header row it declares. An unknown shape fails with the header it actually found. 4. **Every field is shape-checked**, so a renamed column or a new cell is an error naming the game and the cell, never a silently empty column. 5. **Facts the site renders twice must agree.** In the modern layout the date is printed in a desktop cell *and* as a mobile `` group heading, and the start time appears in the tooltip span and in the mobile cell. The parser reads both and rejects the row if they disagree. Values the site prints once per group (the date heading, the round number) are carried forward, and the first row of a group must carry the value to inherit. 6. **Cross-field invariants**: the period scores must sum to the final score, a played game always has an event link and an unplayed one never does, game ids are unique, dates parse, venues are non-empty. 7. **Canonical order, imposed by us.** The site lists games in round order, and re-scheduled or awarded games keep the round they belong to, so the page's own date order is *not* monotonic (2011-12 has September games after October ones). The output is sorted by date, time and id, so it is a function of the set of rows rather than of the site's row order — `TestOrderIndependence` swaps two rows and requires identical bytes. 8. **Fixtures, not the network, in the tests.** Real HTML slices live in `testdata/` next to golden JSON. CI is offline and hermetic; a site change surfaces as a failing test with a readable message. `SWEHOCKEY_LIVE=1 go test -run TestLive` is the one opt-in test that talks to the site. 9. **Drift is a distinct failure mode**, not a crash: `*DriftError` and exit code 3. Callers can alert on it without parsing a stack trace. 10. **Idempotent cron.** `-state ` skips the run when the page body hash is unchanged, and `-check ` exits 3 when the output no longer matches a committed file, so an upstream change becomes a reviewable diff. ## The two layouts on the site today | profile | header | cells | seasons | | -------- | --------------------------------------------- | ----- | ------------------ | | `modern` | `Date Time Game Result … Spectators Venue` | 8 | 2018-19 onwards | | `legacy` | `Round Date Game Result … Spectators Venue` | 7 | up to 2017-18 | They differ in more than columns: - `modern` renders every fact twice, for a desktop and a mobile table, and prints the date only on the first row of each date. The time lives in `span.lnkTooltip[title]`, which is also the row's stable id. - `legacy` renders it once. The round is in its own column, the date and the time share one cell, and old team cells end with an `` note (`Eighth final 1`) that must not be swallowed into a team name. - A played game has 3, 4, 5 or 6 period scores depending on the era's rules, so the count is deliberately *not* pinned; the sum check is what protects the data. Verified on the whole SHL archive: **27 of 27 seasons, 2119 games, 1779 played**. ``` $ ./swehockey crawl -league 20961 -on-drift record | jq -s '[.[] | .games | length] | add' 2119 ``` ## Two implementations, one contract The crawler exists twice, in Go and in Python, and that is a feature: two independent parsers of the same page that have to agree. | | Go | Python | | --- | --- | --- | | entry point | `./swehockey schedule -season 20961` | `python3 -m swehockey.cli schedule --season 20961` | | tests | `go test ./...` (offline) | `python3 -m unittest discover -s tests -t .` (offline) | | live test | `SWEHOCKEY_LIVE=1 go test -run TestLive` | `SWEHOCKEY_LIVE=1 python3 -m unittest tests.test_swehockey.TestLive` | | dependencies | `golang.org/x/net/html` | none — the tree is built on `html.parser` | Both share `testdata/`: the Python suite asserts its output against the **Go golden files**, so a change in one implementation that the other does not share fails a test. Verified on the live page as well: ``` $ ./swehockey schedule -season 20961 -out go.json $ python3 -m swehockey.cli schedule --season 20961 --out py.json $ cmp go.json py.json # 364 games, 102536 bytes, identical $ ./swehockey schedule -season 20961 -format csv -out go.csv $ python3 -m swehockey.cli schedule --season 20961 --format csv --out py.csv $ cmp go.csv py.csv # 365 lines, identical $ python3 -m swehockey.cli crawl --league 20961 | jq -s '[.[].games|length]|add' 2119 # same as the Go crawl ``` Same layout profiles, same invariants, same canonical order, same exit codes (`0` ok, `1` error, `2` usage, `3` drift, `4` timeout). The only intentional difference is in the tree builder: Python treats the site's empty `` cells as empty elements instead of leaving them open the way a browser does, which is what makes the two trees agree. See `python/README.md`. ## Repository layout ``` main.go parse.go seasons.go fetch.go output.go the Go crawler parse_test.go the Go tests python/swehockey/ the Python crawler python/tests/ the Python tests testdata/ HTML fixtures and golden JSON, shared by both README.md this file ``` ## Commands | command | what it does | | --------- | ------------------------------------------------------------------- | | `schedule`| one season, found by `-season` (default 20961) | | `crawl` | every season of `-league`, in ascending id order, one document each | | `leagues` | the competitions linked from the front page | Flags: `-format json|csv|ndjson`, `-out`, `-fixture`, `-state`, `-check`, `-timeout`, `-quiet`, plus `-limit`, `-delay` and `-on-drift` for `crawl`. `crawl -on-drift record` emits `{"season_id":…,"error":…}` in place of a season that breaks the contract, so a bulk run neither dies on one bad page nor hides the hole. Exit codes: `0` ok, `1` error, `2` usage, `3` markup drift or golden mismatch, `4` timeout. ## Output ```json { "season_id": 20961, "layout": "modern", "title": "Schedule and Results", "last_update": "2026-09-26 22:41", "games": [ { "game_id": "90001001", "event_id": "1109941", "date": "2026-09-19", "time": "15:15", "home": "Frölunda HC", "away": "Växjö Lakers HC", "home_goals": 5, "away_goals": 4, "periods": [{"label": "1", "home": 1, "away": 1}], "spectators": 12044, "venue": "Scandinavium" } ] } ``` `game_id` is the site's own stable per-fixture id and is present for scheduled as well as played games; `event_id` comes from the `/Game/Events/` link and appears only once a game has been played. `home_goals` is `null`, not `0`, for a game that has not been played. `last_update` is the site's own clock, verbatim and unparsed, so it can be compared across runs without trusting a timezone conversion. ## Tests and fixtures ``` testdata/schedule_20961.html 2026-27, 3 date groups, OT + shootout, unplayed games testdata/schedule_legacy.html 2018-19, round carry-forward, a 6-period game testdata/schedule_legacy_playoffs.html 2013-14, notes in the team cells testdata/seasons_20961.html the season selector and league links testdata/*.json golden output for the fixtures above ``` `TestDrift` mutates a fixture once per way the page can change shape — renamed header, extra column, score that no longer matches the periods, a removed date group marker, a duplicated id, a missing timestamp, an empty table — and requires a named, actionable error for each. The fixtures are cut from real pages; the only thing the cutter changes is which `` elements it keeps, so the markup around them stays real, including the nested wrapper tables and the ad div. ## Known limits - Old seasons are pruned by the site: 2018-19 serves its last 45 games, 2024-25 only its 6 playoff games. The crawler reports exactly what the page contains. - A handful of season pages serve more slowly than the default 30s timeout (they stream a large table); `-timeout` covers it. - Rows with no result and no round appear in a few old pages; those are the reason `-on-drift record` exists.