Mats Tornberg
47e1811d9e
A second, independent implementation of the same contract, in the same repo: - swehockey/dom.py builds the tree on html.parser, so the variant has no runtime dependencies; the only HTML5 behaviour it depends on - stray <tr> landing in an implicit <tbody> - is handled explicitly by table_rows(). - parse.py mirrors the Go layout profiles, dual-render checks, carry-forward and invariants; a DriftError is raised in the same places with the same advice. - cli.py offers the same three commands, flags and exit codes as the Go main. - tests/ assert against the Go golden files, so the suite is also a cross-implementation check. 23 offline tests plus an opt-in live test. Verified against the live site: JSON and CSV are byte-identical to the Go output (364 games, 102536 bytes / 365 lines), and the full archive crawl gives the same 27 seasons, 2119 games and 1779 played. Also split the DriftError hint out in Go, so a golden mismatch no longer tells the reader to extend layouts[].
215 lines
10 KiB
Markdown
215 lines
10 KiB
Markdown
# swehockey-schedule
|
|
|
|
A deterministic crawler for the **Schedule and Results** table on
|
|
[stats.swehockey.se](https://stats.swehockey.se) — one plain HTTP GET per
|
|
season, no browser, no JavaScript, no waiting, and a byte-identical output for
|
|
identical input.
|
|
|
|
```console
|
|
$ go build -o swehockey .
|
|
$ ./swehockey schedule -season 20961 -format csv | head -3
|
|
season_id,game_id,event_id,date,time,home,away,home_goals,away_goals,periods,spectators,venue
|
|
20961,90001001,1109941,2026-09-19,15:15,Frölunda HC,Växjö Lakers HC,5,4,1:1-1 2:2-1 3:2-2,12044,Scandinavium
|
|
20961,90001002,1109942,2026-09-19,15:15,HV 71,IF Malmö Redhawks,4,5,1:1-0 2:1-1 3:1-3 4:1-1,6800,Husqvarna Garden
|
|
```
|
|
|
|
```console
|
|
$ ./swehockey crawl -league 20961 # every season of the SHL, as NDJSON
|
|
$ ./swehockey leagues # 119 competitions and their ids
|
|
$ ./swehockey schedule -fixture testdata/schedule_20961.html # offline
|
|
```
|
|
|
|
## How determinism is achieved
|
|
|
|
The site's page is server-rendered: `curl` returns all 364 rows of the current
|
|
season in one response. That removes the usual sources of flakiness (timers,
|
|
`waitForSelector`, lazy rendering) and leaves exactly one risk — that the
|
|
markup changes under you. The rest of the design is about that risk.
|
|
|
|
1. **One GET, no retries, no cookies, no state.** Same HTML in, same bytes out.
|
|
The output contains only page-derived data: no fetch timestamp, no map
|
|
iteration, fixed field order. `TestRenderIsPure` and the golden files enforce
|
|
this.
|
|
2. **The table is found by its header signature**, not by a CSS path or a
|
|
position in the document. Ads, navigation and layout changes cannot move it
|
|
out from under the crawler.
|
|
3. **A declared contract per layout, not a pile of indices.** The site has
|
|
shipped more than one version of this table, so each version is a profile in
|
|
`layouts[]` (`parse.go`), identified by the header row it declares. An
|
|
unknown shape fails with the header it actually found.
|
|
4. **Every field is shape-checked**, so a renamed column or a new cell is an
|
|
error naming the game and the cell, never a silently empty column.
|
|
5. **Facts the site renders twice must agree.** In the modern layout the date is
|
|
printed in a desktop cell *and* as a mobile `<bold>` group heading, and the
|
|
start time appears in the tooltip span and in the mobile cell. The parser
|
|
reads both and rejects the row if they disagree. Values the site prints once
|
|
per group (the date heading, the round number) are carried forward, and the
|
|
first row of a group must carry the value to inherit.
|
|
6. **Cross-field invariants**: the period scores must sum to the final score,
|
|
a played game always has an event link and an unplayed one never does,
|
|
game ids are unique, dates parse, venues are non-empty.
|
|
7. **Canonical order, imposed by us.** The site lists games in round order, and
|
|
re-scheduled or awarded games keep the round they belong to, so the page's
|
|
own date order is *not* monotonic (2011-12 has September games after October
|
|
ones). The output is sorted by date, time and id, so it is a function of the
|
|
set of rows rather than of the site's row order — `TestOrderIndependence`
|
|
swaps two rows and requires identical bytes.
|
|
8. **Fixtures, not the network, in the tests.** Real HTML slices live in
|
|
`testdata/` next to golden JSON. CI is offline and hermetic; a site change
|
|
surfaces as a failing test with a readable message. `SWEHOCKEY_LIVE=1 go
|
|
test -run TestLive` is the one opt-in test that talks to the site.
|
|
9. **Drift is a distinct failure mode**, not a crash: `*DriftError` and exit
|
|
code 3. Callers can alert on it without parsing a stack trace.
|
|
10. **Idempotent cron.** `-state <file>` skips the run when the page body hash
|
|
is unchanged, and `-check <golden>` exits 3 when the output no longer
|
|
matches a committed file, so an upstream change becomes a reviewable diff.
|
|
|
|
## The two layouts on the site today
|
|
|
|
| profile | header | cells | seasons |
|
|
| -------- | --------------------------------------------- | ----- | ------------------ |
|
|
| `modern` | `Date Time Game Result … Spectators Venue` | 8 | 2018-19 onwards |
|
|
| `legacy` | `Round Date Game Result … Spectators Venue` | 7 | up to 2017-18 |
|
|
|
|
They differ in more than columns:
|
|
|
|
- `modern` renders every fact twice, for a desktop and a mobile table, and
|
|
prints the date only on the first row of each date. The time lives in
|
|
`span.lnkTooltip[title]`, which is also the row's stable id.
|
|
- `legacy` renders it once. The round is in its own column, the date and the
|
|
time share one cell, and old team cells end with an `<i>` note
|
|
(`Eighth final 1`) that must not be swallowed into a team name.
|
|
- A played game has 3, 4, 5 or 6 period scores depending on the era's rules, so
|
|
the count is deliberately *not* pinned; the sum check is what protects the
|
|
data.
|
|
|
|
Verified on the whole SHL archive: **27 of 27 seasons, 2119 games, 1779 played**.
|
|
|
|
```
|
|
$ ./swehockey crawl -league 20961 -on-drift record | jq -s '[.[] | .games | length] | add'
|
|
2119
|
|
```
|
|
|
|
## Two implementations, one contract
|
|
|
|
The crawler exists twice, in Go and in Python, and that is a feature: two
|
|
independent parsers of the same page that have to agree.
|
|
|
|
| | Go | Python |
|
|
| --- | --- | --- |
|
|
| entry point | `./swehockey schedule -season 20961` | `python3 -m swehockey.cli schedule --season 20961` |
|
|
| tests | `go test ./...` (offline) | `python3 -m unittest discover -s tests -t .` (offline) |
|
|
| live test | `SWEHOCKEY_LIVE=1 go test -run TestLive` | `SWEHOCKEY_LIVE=1 python3 -m unittest tests.test_swehockey.TestLive` |
|
|
| dependencies | `golang.org/x/net/html` | none — the tree is built on `html.parser` |
|
|
|
|
Both share `testdata/`: the Python suite asserts its output against the **Go
|
|
golden files**, so a change in one implementation that the other does not share
|
|
fails a test. Verified on the live page as well:
|
|
|
|
```
|
|
$ ./swehockey schedule -season 20961 -out go.json
|
|
$ python3 -m swehockey.cli schedule --season 20961 --out py.json
|
|
$ cmp go.json py.json # 364 games, 102536 bytes, identical
|
|
$ ./swehockey schedule -season 20961 -format csv -out go.csv
|
|
$ python3 -m swehockey.cli schedule --season 20961 --format csv --out py.csv
|
|
$ cmp go.csv py.csv # 365 lines, identical
|
|
$ python3 -m swehockey.cli crawl --league 20961 | jq -s '[.[].games|length]|add'
|
|
2119 # same as the Go crawl
|
|
```
|
|
|
|
Same layout profiles, same invariants, same canonical order, same exit codes
|
|
(`0` ok, `1` error, `2` usage, `3` drift, `4` timeout). The only intentional
|
|
difference is in the tree builder: Python treats the site's empty `<th … />`
|
|
cells as empty elements instead of leaving them open the way a browser does,
|
|
which is what makes the two trees agree. See `python/README.md`.
|
|
|
|
## Repository layout
|
|
|
|
```
|
|
main.go parse.go seasons.go fetch.go output.go the Go crawler
|
|
parse_test.go the Go tests
|
|
python/swehockey/ the Python crawler
|
|
python/tests/ the Python tests
|
|
testdata/ HTML fixtures and golden JSON, shared by both
|
|
README.md this file
|
|
```
|
|
|
|
## Commands
|
|
|
|
| command | what it does |
|
|
| --------- | ------------------------------------------------------------------- |
|
|
| `schedule`| one season, found by `-season` (default 20961) |
|
|
| `crawl` | every season of `-league`, in ascending id order, one document each |
|
|
| `leagues` | the competitions linked from the front page |
|
|
|
|
Flags: `-format json|csv|ndjson`, `-out`, `-fixture`, `-state`, `-check`,
|
|
`-timeout`, `-quiet`, plus `-limit`, `-delay` and `-on-drift` for `crawl`.
|
|
|
|
`crawl -on-drift record` emits `{"season_id":…,"error":…}` in place of a season
|
|
that breaks the contract, so a bulk run neither dies on one bad page nor hides
|
|
the hole.
|
|
|
|
Exit codes: `0` ok, `1` error, `2` usage, `3` markup drift or golden mismatch,
|
|
`4` timeout.
|
|
|
|
## Output
|
|
|
|
```json
|
|
{
|
|
"season_id": 20961,
|
|
"layout": "modern",
|
|
"title": "Schedule and Results",
|
|
"last_update": "2026-09-26 22:41",
|
|
"games": [
|
|
{
|
|
"game_id": "90001001",
|
|
"event_id": "1109941",
|
|
"date": "2026-09-19",
|
|
"time": "15:15",
|
|
"home": "Frölunda HC",
|
|
"away": "Växjö Lakers HC",
|
|
"home_goals": 5,
|
|
"away_goals": 4,
|
|
"periods": [{"label": "1", "home": 1, "away": 1}],
|
|
"spectators": 12044,
|
|
"venue": "Scandinavium"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
`game_id` is the site's own stable per-fixture id and is present for scheduled
|
|
as well as played games; `event_id` comes from the `/Game/Events/<id>` link and
|
|
appears only once a game has been played. `home_goals` is `null`, not `0`, for a
|
|
game that has not been played. `last_update` is the site's own clock, verbatim
|
|
and unparsed, so it can be compared across runs without trusting a timezone
|
|
conversion.
|
|
|
|
## Tests and fixtures
|
|
|
|
```
|
|
testdata/schedule_20961.html 2026-27, 3 date groups, OT + shootout, unplayed games
|
|
testdata/schedule_legacy.html 2018-19, round carry-forward, a 6-period game
|
|
testdata/schedule_legacy_playoffs.html 2013-14, <i>notes</i> in the team cells
|
|
testdata/seasons_20961.html the season selector and league links
|
|
testdata/*.json golden output for the fixtures above
|
|
```
|
|
|
|
`TestDrift` mutates a fixture once per way the page can change shape — renamed
|
|
header, extra column, score that no longer matches the periods, a removed date
|
|
group marker, a duplicated id, a missing timestamp, an empty table — and
|
|
requires a named, actionable error for each.
|
|
|
|
The fixtures are cut from real pages; the only thing the cutter changes is which
|
|
`<tr>` elements it keeps, so the markup around them stays real, including the
|
|
nested wrapper tables and the ad div.
|
|
|
|
## Known limits
|
|
|
|
- Old seasons are pruned by the site: 2018-19 serves its last 45 games, 2024-25
|
|
only its 6 playoff games. The crawler reports exactly what the page contains.
|
|
- A handful of season pages serve more slowly than the default 30s timeout
|
|
(they stream a large table); `-timeout` covers it.
|
|
- Rows with no result and no round appear in a few old pages; those are the
|
|
reason `-on-drift record` exists.
|