Deterministic swehockey.se schedule crawler
One HTTP GET per season, no browser: the page is server-rendered, so every row arrives in one response and the extraction becomes a pure function of the HTML. - layouts[] declares one profile per version of the site's table (modern and legacy), each identified by its header row, so a new version of the page is a new profile plus a fixture instead of a guess. - The table is located by header signature, not by a CSS path. - Every cell is shape-checked, and facts the site renders twice (desktop and mobile date, time in the tooltip) must agree. - Cross-field invariants: periods sum to the score, ids are unique, played games always have an event link and unplayed ones never do. - Output is sorted by date, time and id, so it does not inherit the site's row order, and it carries no fetch timestamp: same HTML, same bytes. - Drift is a typed error and exit code 3, with the header it actually found. - Tests run on committed HTML fixtures and golden JSON; TestLive is opt-in. Verified over the full SHL archive: 27/27 seasons, 2119 games, 1779 played.
This commit is contained in:
Mats Tornberg committed
commit
6b701819ca
20 files changed
+2876
No files matched your search
@@ -0,0 +1,170 @@
|
||||
# swehockey-schedule
|
||||
|
||||
A deterministic crawler for the **Schedule and Results** table on
|
||||
[stats.swehockey.se](https://stats.swehockey.se) — one plain HTTP GET per
|
||||
season, no browser, no JavaScript, no waiting, and a byte-identical output for
|
||||
identical input.
|
||||
|
||||
```console
|
||||
$ go build -o swehockey .
|
||||
$ ./swehockey schedule -season 20961 -format csv | head -3
|
||||
season_id,game_id,event_id,date,time,home,away,home_goals,away_goals,periods,spectators,venue
|
||||
20961,90001001,1109941,2026-09-19,15:15,Frölunda HC,Växjö Lakers HC,5,4,1:1-1 2:2-1 3:2-2,12044,Scandinavium
|
||||
20961,90001002,1109942,2026-09-19,15:15,HV 71,IF Malmö Redhawks,4,5,1:1-0 2:1-1 3:1-3 4:1-1,6800,Husqvarna Garden
|
||||
```
|
||||
|
||||
```console
|
||||
$ ./swehockey crawl -league 20961 # every season of the SHL, as NDJSON
|
||||
$ ./swehockey leagues # 119 competitions and their ids
|
||||
$ ./swehockey schedule -fixture testdata/schedule_20961.html # offline
|
||||
```
|
||||
|
||||
## How determinism is achieved
|
||||
|
||||
The site's page is server-rendered: `curl` returns all 364 rows of the current
|
||||
season in one response. That removes the usual sources of flakiness (timers,
|
||||
`waitForSelector`, lazy rendering) and leaves exactly one risk — that the
|
||||
markup changes under you. The rest of the design is about that risk.
|
||||
|
||||
1. **One GET, no retries, no cookies, no state.** Same HTML in, same bytes out.
|
||||
The output contains only page-derived data: no fetch timestamp, no map
|
||||
iteration, fixed field order. `TestRenderIsPure` and the golden files enforce
|
||||
this.
|
||||
2. **The table is found by its header signature**, not by a CSS path or a
|
||||
position in the document. Ads, navigation and layout changes cannot move it
|
||||
out from under the crawler.
|
||||
3. **A declared contract per layout, not a pile of indices.** The site has
|
||||
shipped more than one version of this table, so each version is a profile in
|
||||
`layouts[]` (`parse.go`), identified by the header row it declares. An
|
||||
unknown shape fails with the header it actually found.
|
||||
4. **Every field is shape-checked**, so a renamed column or a new cell is an
|
||||
error naming the game and the cell, never a silently empty column.
|
||||
5. **Facts the site renders twice must agree.** In the modern layout the date is
|
||||
printed in a desktop cell *and* as a mobile `<bold>` group heading, and the
|
||||
start time appears in the tooltip span and in the mobile cell. The parser
|
||||
reads both and rejects the row if they disagree. Values the site prints once
|
||||
per group (the date heading, the round number) are carried forward, and the
|
||||
first row of a group must carry the value to inherit.
|
||||
6. **Cross-field invariants**: the period scores must sum to the final score,
|
||||
a played game always has an event link and an unplayed one never does,
|
||||
game ids are unique, dates parse, venues are non-empty.
|
||||
7. **Canonical order, imposed by us.** The site lists games in round order, and
|
||||
re-scheduled or awarded games keep the round they belong to, so the page's
|
||||
own date order is *not* monotonic (2011-12 has September games after October
|
||||
ones). The output is sorted by date, time and id, so it is a function of the
|
||||
set of rows rather than of the site's row order — `TestOrderIndependence`
|
||||
swaps two rows and requires identical bytes.
|
||||
8. **Fixtures, not the network, in the tests.** Real HTML slices live in
|
||||
`testdata/` next to golden JSON. CI is offline and hermetic; a site change
|
||||
surfaces as a failing test with a readable message. `SWEHOCKEY_LIVE=1 go
|
||||
test -run TestLive` is the one opt-in test that talks to the site.
|
||||
9. **Drift is a distinct failure mode**, not a crash: `*DriftError` and exit
|
||||
code 3. Callers can alert on it without parsing a stack trace.
|
||||
10. **Idempotent cron.** `-state <file>` skips the run when the page body hash
|
||||
is unchanged, and `-check <golden>` exits 3 when the output no longer
|
||||
matches a committed file, so an upstream change becomes a reviewable diff.
|
||||
|
||||
## The two layouts on the site today
|
||||
|
||||
| profile | header | cells | seasons |
|
||||
| -------- | --------------------------------------------- | ----- | ------------------ |
|
||||
| `modern` | `Date Time Game Result … Spectators Venue` | 8 | 2018-19 onwards |
|
||||
| `legacy` | `Round Date Game Result … Spectators Venue` | 7 | up to 2017-18 |
|
||||
|
||||
They differ in more than columns:
|
||||
|
||||
- `modern` renders every fact twice, for a desktop and a mobile table, and
|
||||
prints the date only on the first row of each date. The time lives in
|
||||
`span.lnkTooltip[title]`, which is also the row's stable id.
|
||||
- `legacy` renders it once. The round is in its own column, the date and the
|
||||
time share one cell, and old team cells end with an `<i>` note
|
||||
(`Eighth final 1`) that must not be swallowed into a team name.
|
||||
- A played game has 3, 4, 5 or 6 period scores depending on the era's rules, so
|
||||
the count is deliberately *not* pinned; the sum check is what protects the
|
||||
data.
|
||||
|
||||
Verified on the whole SHL archive: **27 of 27 seasons, 2119 games, 1779 played**.
|
||||
|
||||
```
|
||||
$ ./swehockey crawl -league 20961 -on-drift record | jq -s '[.[] | .games | length] | add'
|
||||
2119
|
||||
```
|
||||
|
||||
## Commands
|
||||
|
||||
| command | what it does |
|
||||
| --------- | ------------------------------------------------------------------- |
|
||||
| `schedule`| one season, found by `-season` (default 20961) |
|
||||
| `crawl` | every season of `-league`, in ascending id order, one document each |
|
||||
| `leagues` | the competitions linked from the front page |
|
||||
|
||||
Flags: `-format json|csv|ndjson`, `-out`, `-fixture`, `-state`, `-check`,
|
||||
`-timeout`, `-quiet`, plus `-limit`, `-delay` and `-on-drift` for `crawl`.
|
||||
|
||||
`crawl -on-drift record` emits `{"season_id":…,"error":…}` in place of a season
|
||||
that breaks the contract, so a bulk run neither dies on one bad page nor hides
|
||||
the hole.
|
||||
|
||||
Exit codes: `0` ok, `1` error, `2` usage, `3` markup drift or golden mismatch,
|
||||
`4` timeout.
|
||||
|
||||
## Output
|
||||
|
||||
```json
|
||||
{
|
||||
"season_id": 20961,
|
||||
"layout": "modern",
|
||||
"title": "Schedule and Results",
|
||||
"last_update": "2026-09-26 22:41",
|
||||
"games": [
|
||||
{
|
||||
"game_id": "90001001",
|
||||
"event_id": "1109941",
|
||||
"date": "2026-09-19",
|
||||
"time": "15:15",
|
||||
"home": "Frölunda HC",
|
||||
"away": "Växjö Lakers HC",
|
||||
"home_goals": 5,
|
||||
"away_goals": 4,
|
||||
"periods": [{"label": "1", "home": 1, "away": 1}],
|
||||
"spectators": 12044,
|
||||
"venue": "Scandinavium"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`game_id` is the site's own stable per-fixture id and is present for scheduled
|
||||
as well as played games; `event_id` comes from the `/Game/Events/<id>` link and
|
||||
appears only once a game has been played. `home_goals` is `null`, not `0`, for a
|
||||
game that has not been played. `last_update` is the site's own clock, verbatim
|
||||
and unparsed, so it can be compared across runs without trusting a timezone
|
||||
conversion.
|
||||
|
||||
## Tests and fixtures
|
||||
|
||||
```
|
||||
testdata/schedule_20961.html 2026-27, 3 date groups, OT + shootout, unplayed games
|
||||
testdata/schedule_legacy.html 2018-19, round carry-forward, a 6-period game
|
||||
testdata/schedule_legacy_playoffs.html 2013-14, <i>notes</i> in the team cells
|
||||
testdata/seasons_20961.html the season selector and league links
|
||||
testdata/*.json golden output for the fixtures above
|
||||
```
|
||||
|
||||
`TestDrift` mutates a fixture once per way the page can change shape — renamed
|
||||
header, extra column, score that no longer matches the periods, a removed date
|
||||
group marker, a duplicated id, a missing timestamp, an empty table — and
|
||||
requires a named, actionable error for each.
|
||||
|
||||
The fixtures are cut from real pages; the only thing the cutter changes is which
|
||||
`<tr>` elements it keeps, so the markup around them stays real, including the
|
||||
nested wrapper tables and the ad div.
|
||||
|
||||
## Known limits
|
||||
|
||||
- Old seasons are pruned by the site: 2018-19 serves its last 45 games, 2024-25
|
||||
only its 6 playoff games. The crawler reports exactly what the page contains.
|
||||
- A handful of season pages serve more slowly than the default 30s timeout
|
||||
(they stream a large table); `-timeout` covers it.
|
||||
- Rows with no result and no round appear in a few old pages; those are the
|
||||
reason `-on-drift record` exists.
|
||||
Reference in new issue
Block a user