swehockey-schedule

A deterministic crawler for the Schedule and Results table on stats.swehockey.se — one plain HTTP GET per season, no browser, no JavaScript, no waiting, and a byte-identical output for identical input.

$ go build -o swehockey .
$ ./swehockey schedule -season 20961 -format csv | head -3
season_id,game_id,event_id,date,time,home,away,home_goals,away_goals,periods,spectators,venue
20961,90001001,1109941,2026-09-19,15:15,Frölunda HC,Växjö Lakers HC,5,4,1:1-1 2:2-1 3:2-2,12044,Scandinavium
20961,90001002,1109942,2026-09-19,15:15,HV 71,IF Malmö Redhawks,4,5,1:1-0 2:1-1 3:1-3 4:1-1,6800,Husqvarna Garden

$ ./swehockey crawl -league 20961            # every season of the SHL, as NDJSON
$ ./swehockey leagues                        # 119 competitions and their ids
$ ./swehockey schedule -fixture testdata/schedule_20961.html   # offline

How determinism is achieved

The site's page is server-rendered: curl returns all 364 rows of the current season in one response. That removes the usual sources of flakiness (timers, waitForSelector, lazy rendering) and leaves exactly one risk — that the markup changes under you. The rest of the design is about that risk.

1. One GET, no retries, no cookies, no state. Same HTML in, same bytes out. The output contains only page-derived data: no fetch timestamp, no map iteration, fixed field order. TestRenderIsPure and the golden files enforce this.
2. The table is found by its header signature, not by a CSS path or a position in the document. Ads, navigation and layout changes cannot move it out from under the crawler.
3. A declared contract per layout, not a pile of indices. The site has shipped more than one version of this table, so each version is a profile in layouts[] (parse.go), identified by the header row it declares. An unknown shape fails with the header it actually found.
4. Every field is shape-checked, so a renamed column or a new cell is an error naming the game and the cell, never a silently empty column.
5. Facts the site renders twice must agree. In the modern layout the date is printed in a desktop cell and as a mobile <bold> group heading, and the start time appears in the tooltip span and in the mobile cell. The parser reads both and rejects the row if they disagree. Values the site prints once per group (the date heading, the round number) are carried forward, and the first row of a group must carry the value to inherit.
6. Cross-field invariants: the period scores must sum to the final score, a played game always has an event link and an unplayed one never does, game ids are unique, dates parse, venues are non-empty.
7. Canonical order, imposed by us. The site lists games in round order, and re-scheduled or awarded games keep the round they belong to, so the page's own date order is not monotonic (2011-12 has September games after October ones). The output is sorted by date, time and id, so it is a function of the set of rows rather than of the site's row order — TestOrderIndependence swaps two rows and requires identical bytes.
8. Fixtures, not the network, in the tests. Real HTML slices live in testdata/ next to golden JSON. CI is offline and hermetic; a site change surfaces as a failing test with a readable message. SWEHOCKEY_LIVE=1 go test -run TestLive is the one opt-in test that talks to the site.
9. Drift is a distinct failure mode, not a crash: *DriftError and exit code 3. Callers can alert on it without parsing a stack trace.
10. Idempotent cron. -state <file> skips the run when the page body hash is unchanged, and -check <golden> exits 3 when the output no longer matches a committed file, so an upstream change becomes a reviewable diff.

The two layouts on the site today

profile	header	cells	seasons
modern	Date Time Game Result … Spectators Venue	8	2018-19 onwards
legacy	Round Date Game Result … Spectators Venue	7	up to 2017-18

They differ in more than columns:

• modern renders every fact twice, for a desktop and a mobile table, and prints the date only on the first row of each date. The time lives in span.lnkTooltip[title], which is also the row's stable id.
• legacy renders it once. The round is in its own column, the date and the time share one cell, and old team cells end with an <i> note (Eighth final 1) that must not be swallowed into a team name.
• A played game has 3, 4, 5 or 6 period scores depending on the era's rules, so the count is deliberately not pinned; the sum check is what protects the data.

Verified on the whole SHL archive: 27 of 27 seasons, 2119 games, 1779 played.

$ ./swehockey crawl -league 20961 -on-drift record | jq -s '[.[] | .games | length] | add'
2119

Two implementations, one contract

The crawler exists twice, in Go and in Python, and that is a feature: two independent parsers of the same page that have to agree.

	Go	Python
entry point	./swehockey schedule -season 20961	python3 -m swehockey.cli schedule --season 20961
tests	go test ./... (offline)	python3 -m unittest discover -s tests -t . (offline)
live test	SWEHOCKEY_LIVE=1 go test -run TestLive	SWEHOCKEY_LIVE=1 python3 -m unittest tests.test_swehockey.TestLive
dependencies	golang.org/x/net/html	none — the tree is built on html.parser

Both share testdata/: the Python suite asserts its output against the Go golden files, so a change in one implementation that the other does not share fails a test. Verified on the live page as well:

$ ./swehockey  schedule -season 20961 -out go.json
$ python3 -m swehockey.cli schedule --season 20961 --out py.json
$ cmp go.json py.json                       # 364 games, 102536 bytes, identical
$ ./swehockey  schedule -season 20961 -format csv -out go.csv
$ python3 -m swehockey.cli schedule --season 20961 --format csv --out py.csv
$ cmp go.csv py.csv                         # 365 lines, identical
$ python3 -m swehockey.cli crawl --league 20961 | jq -s '[.[].games|length]|add'
2119                                          # same as the Go crawl

Same layout profiles, same invariants, same canonical order, same exit codes (0 ok, 1 error, 2 usage, 3 drift, 4 timeout). The only intentional difference is in the tree builder: Python treats the site's empty <th … /> cells as empty elements instead of leaving them open the way a browser does, which is what makes the two trees agree. See python/README.md.

Repository layout

main.go parse.go seasons.go fetch.go output.go   the Go crawler
parse_test.go                                     the Go tests
python/swehockey/                                 the Python crawler
python/tests/                                      the Python tests
testdata/                                         HTML fixtures and golden JSON, shared by both
README.md                                         this file

Commands

command	what it does
schedule	one season, found by -season (default 20961)
crawl	every season of -league, in ascending id order, one document each
leagues	the competitions linked from the front page

Flags: -format json|csv|ndjson, -out, -fixture, -state, -check, -timeout, -quiet, plus -limit, -delay and -on-drift for crawl.

crawl -on-drift record emits {"season_id":…,"error":…} in place of a season that breaks the contract, so a bulk run neither dies on one bad page nor hides the hole.

Exit codes: 0 ok, 1 error, 2 usage, 3 markup drift or golden mismatch, 4 timeout.

Output

{
  "season_id": 20961,
  "layout": "modern",
  "title": "Schedule and Results",
  "last_update": "2026-09-26 22:41",
  "games": [
    {
      "game_id": "90001001",
      "event_id": "1109941",
      "date": "2026-09-19",
      "time": "15:15",
      "home": "Frölunda HC",
      "away": "Växjö Lakers HC",
      "home_goals": 5,
      "away_goals": 4,
      "periods": [{"label": "1", "home": 1, "away": 1}],
      "spectators": 12044,
      "venue": "Scandinavium"
    }
  ]
}

game_id is the site's own stable per-fixture id and is present for scheduled as well as played games; event_id comes from the /Game/Events/<id> link and appears only once a game has been played. home_goals is null, not 0, for a game that has not been played. last_update is the site's own clock, verbatim and unparsed, so it can be compared across runs without trusting a timezone conversion.

Tests and fixtures

testdata/schedule_20961.html              2026-27, 3 date groups, OT + shootout, unplayed games
testdata/schedule_legacy.html             2018-19, round carry-forward, a 6-period game
testdata/schedule_legacy_playoffs.html    2013-14, <i>notes</i> in the team cells
testdata/seasons_20961.html               the season selector and league links
testdata/*.json                           golden output for the fixtures above

TestDrift mutates a fixture once per way the page can change shape — renamed header, extra column, score that no longer matches the periods, a removed date group marker, a duplicated id, a missing timestamp, an empty table — and requires a named, actionable error for each.

The fixtures are cut from real pages; the only thing the cutter changes is which <tr> elements it keeps, so the markup around them stays real, including the nested wrapper tables and the ad div.

Known limits

• Old seasons are pruned by the site: 2018-19 serves its last 45 games, 2024-25 only its 6 playoff games. The crawler reports exactly what the page contains.
• A handful of season pages serve more slowly than the default 30s timeout (they stream a large table); -timeout covers it.
• Rows with no result and no round appear in a few old pages; those are the reason -on-drift record exists.