swehockey/README.md

# swehockey-schedule

A deterministic crawler for the **Schedule and Results** table on
[stats.swehockey.se](https://stats.swehockey.se) — one plain HTTP GET per
season, no browser, no JavaScript, no waiting, and a byte-identical output for
identical input.

```console
$ go build -o swehockey .
$ ./swehockey schedule -season 20961 -format csv | head -3
season_id,game_id,event_id,date,time,home,away,home_goals,away_goals,periods,spectators,venue
20961,90001001,1109941,2026-09-19,15:15,Frölunda HC,Växjö Lakers HC,5,4,1:1-1 2:2-1 3:2-2,12044,Scandinavium
20961,90001002,1109942,2026-09-19,15:15,HV 71,IF Malmö Redhawks,4,5,1:1-0 2:1-1 3:1-3 4:1-1,6800,Husqvarna Garden
```

```console
$ ./swehockey crawl -league 20961            # every season of the SHL, as NDJSON
$ ./swehockey leagues                        # 119 competitions and their ids
$ ./swehockey schedule -fixture testdata/schedule_20961.html   # offline
```

## How determinism is achieved

The site's page is server-rendered: `curl` returns all 364 rows of the current
season in one response. That removes the usual sources of flakiness (timers,
`waitForSelector`, lazy rendering) and leaves exactly one risk — that the
markup changes under you. The rest of the design is about that risk.

1. **One GET, no retries, no cookies, no state.** Same HTML in, same bytes out.
   The output contains only page-derived data: no fetch timestamp, no map
   iteration, fixed field order. `TestRenderIsPure` and the golden files enforce
   this.
2. **The table is found by its header signature**, not by a CSS path or a
   position in the document. Ads, navigation and layout changes cannot move it
   out from under the crawler.
3. **A declared contract per layout, not a pile of indices.** The site has
   shipped more than one version of this table, so each version is a profile in
   `layouts[]` (`parse.go`), identified by the header row it declares. An
   unknown shape fails with the header it actually found.
4. **Every field is shape-checked**, so a renamed column or a new cell is an
   error naming the game and the cell, never a silently empty column.
5. **Facts the site renders twice must agree.** In the modern layout the date is
   printed in a desktop cell *and* as a mobile `<bold>` group heading, and the
   start time appears in the tooltip span and in the mobile cell. The parser
   reads both and rejects the row if they disagree. Values the site prints once
   per group (the date heading, the round number) are carried forward, and the
   first row of a group must carry the value to inherit.
6. **Cross-field invariants**: the period scores must sum to the final score,
   a played game always has an event link and an unplayed one never does,
   game ids are unique, dates parse, venues are non-empty.
7. **Canonical order, imposed by us.** The site lists games in round order, and
   re-scheduled or awarded games keep the round they belong to, so the page's
   own date order is *not* monotonic (2011-12 has September games after October
   ones). The output is sorted by date, time and id, so it is a function of the
   set of rows rather than of the site's row order — `TestOrderIndependence`
   swaps two rows and requires identical bytes.
8. **Fixtures, not the network, in the tests.** Real HTML slices live in
   `testdata/` next to golden JSON. CI is offline and hermetic; a site change
   surfaces as a failing test with a readable message. `SWEHOCKEY_LIVE=1 go
   test -run TestLive` is the one opt-in test that talks to the site.
9. **Drift is a distinct failure mode**, not a crash: `*DriftError` and exit
   code 3. Callers can alert on it without parsing a stack trace.
10. **Idempotent cron.** `-state <file>` skips the run when the page body hash
    is unchanged, and `-check <golden>` exits 3 when the output no longer
    matches a committed file, so an upstream change becomes a reviewable diff.

## The two layouts on the site today

| profile  | header                                        | cells | seasons            |
| -------- | --------------------------------------------- | ----- | ------------------ |
| `modern` | `Date Time Game Result … Spectators Venue`      | 8     | 2018-19 onwards    |
| `legacy` | `Round Date Game Result … Spectators Venue`     | 7     | up to 2017-18      |

They differ in more than columns:

- `modern` renders every fact twice, for a desktop and a mobile table, and
  prints the date only on the first row of each date. The time lives in
  `span.lnkTooltip[title]`, which is also the row's stable id.
- `legacy` renders it once. The round is in its own column, the date and the
  time share one cell, and old team cells end with an `<i>` note
  (`Eighth final 1`) that must not be swallowed into a team name.
- A played game has 3, 4, 5 or 6 period scores depending on the era's rules, so
  the count is deliberately *not* pinned; the sum check is what protects the
  data.

Verified on the whole SHL archive: **27 of 27 seasons, 2119 games, 1779 played**.

```
$ ./swehockey crawl -league 20961 -on-drift record | jq -s '[.[] | .games | length] | add'
2119
```

## Commands

| command   | what it does                                                        |
| --------- | ------------------------------------------------------------------- |
| `schedule`| one season, found by `-season` (default 20961)                       |
| `crawl`   | every season of `-league`, in ascending id order, one document each   |
| `leagues` | the competitions linked from the front page                           |

Flags: `-format json|csv|ndjson`, `-out`, `-fixture`, `-state`, `-check`,
`-timeout`, `-quiet`, plus `-limit`, `-delay` and `-on-drift` for `crawl`.

`crawl -on-drift record` emits `{"season_id":…,"error":…}` in place of a season
that breaks the contract, so a bulk run neither dies on one bad page nor hides
the hole.

Exit codes: `0` ok, `1` error, `2` usage, `3` markup drift or golden mismatch,
`4` timeout.

## Output

```json
{
  "season_id": 20961,
  "layout": "modern",
  "title": "Schedule and Results",
  "last_update": "2026-09-26 22:41",
  "games": [
    {
      "game_id": "90001001",
      "event_id": "1109941",
      "date": "2026-09-19",
      "time": "15:15",
      "home": "Frölunda HC",
      "away": "Växjö Lakers HC",
      "home_goals": 5,
      "away_goals": 4,
      "periods": [{"label": "1", "home": 1, "away": 1}],
      "spectators": 12044,
      "venue": "Scandinavium"
    }
  ]
}
```

`game_id` is the site's own stable per-fixture id and is present for scheduled
as well as played games; `event_id` comes from the `/Game/Events/<id>` link and
appears only once a game has been played. `home_goals` is `null`, not `0`, for a
game that has not been played. `last_update` is the site's own clock, verbatim
and unparsed, so it can be compared across runs without trusting a timezone
conversion.

## Tests and fixtures

```
testdata/schedule_20961.html              2026-27, 3 date groups, OT + shootout, unplayed games
testdata/schedule_legacy.html             2018-19, round carry-forward, a 6-period game
testdata/schedule_legacy_playoffs.html    2013-14, <i>notes</i> in the team cells
testdata/seasons_20961.html               the season selector and league links
testdata/*.json                           golden output for the fixtures above
```

`TestDrift` mutates a fixture once per way the page can change shape — renamed
header, extra column, score that no longer matches the periods, a removed date
group marker, a duplicated id, a missing timestamp, an empty table — and
requires a named, actionable error for each.

The fixtures are cut from real pages; the only thing the cutter changes is which
`<tr>` elements it keeps, so the markup around them stays real, including the
nested wrapper tables and the ad div.

## Known limits

- Old seasons are pruned by the site: 2018-19 serves its last 45 games, 2024-25
  only its 6 playoff games. The crawler reports exactly what the page contains.
- A handful of season pages serve more slowly than the default 30s timeout
  (they stream a large table); `-timeout` covers it.
- Rows with no result and no round appear in a few old pages; those are the
  reason `-on-drift record` exists.