Mats Tornberg
47e1811d9e
A second, independent implementation of the same contract, in the same repo: - swehockey/dom.py builds the tree on html.parser, so the variant has no runtime dependencies; the only HTML5 behaviour it depends on - stray <tr> landing in an implicit <tbody> - is handled explicitly by table_rows(). - parse.py mirrors the Go layout profiles, dual-render checks, carry-forward and invariants; a DriftError is raised in the same places with the same advice. - cli.py offers the same three commands, flags and exit codes as the Go main. - tests/ assert against the Go golden files, so the suite is also a cross-implementation check. 23 offline tests plus an opt-in live test. Verified against the live site: JSON and CSV are byte-identical to the Go output (364 games, 102536 bytes / 365 lines), and the full archive crawl gives the same 27 seasons, 2119 games and 1779 played. Also split the DriftError hint out in Go, so a golden mismatch no longer tells the reader to extend layouts[].
71 lines
3.5 KiB
Markdown
71 lines
3.5 KiB
Markdown
# swehockey-schedule (Python)
|
|
|
|
The Python twin of the Go crawler in the parent directory. Same layout profiles,
|
|
same invariants, same canonical order — and, on purpose, **the same bytes**.
|
|
|
|
```console
|
|
$ python3 -m swehockey.cli schedule --season 20961 --format csv | head -2
|
|
season_id,game_id,event_id,date,time,home,away,home_goals,away_goals,periods,spectators,venue
|
|
20961,90001001,1109941,2026-09-19,15:15,Frölunda HC,Växjö Lakers HC,5,4,1:1-1 2:2-1 3:2-2,12044,Scandinavium
|
|
|
|
$ python3 -m swehockey.cli crawl --league 20961 # every season, as NDJSON
|
|
$ python3 -m swehockey.cli leagues # 119 competitions
|
|
$ python3 -m unittest discover -s tests -t . # 23 offline tests
|
|
$ SWEHOCKEY_LIVE=1 python3 -m unittest discover -s tests -t . # + the site
|
|
```
|
|
|
|
With `pip install -e .` the same CLI is on `$PATH` as `swehockey-schedule`.
|
|
|
|
## Why it can agree byte for byte
|
|
|
|
- Same `LAYOUTS` profiles, keyed on the header row; same per-field shape
|
|
checks; same dual-render agreement for the modern layout; same carry-forward
|
|
for the date heading and the round number; same invariants (periods sum to
|
|
the score, ids unique, played ⇔ event link).
|
|
- Same canonical sort by `(date, time, game_id)`, because the site orders rows
|
|
by round and puts re-scheduled games where their round says.
|
|
- The output carries only page-derived data — no fetch timestamp, no unordered
|
|
iteration, fixed key order — and `json.dumps(..., indent=2, ensure_ascii=False)`
|
|
is byte-compatible with Go's `json.Encoder`.
|
|
- The tests read the **Go golden files** in `../testdata`, so the suite doubles
|
|
as a cross-implementation check.
|
|
|
|
Verified on the live page, not just on fixtures:
|
|
|
|
```
|
|
$ python3 -m swehockey.cli schedule --season 20961 --out py.json
|
|
$ ../swehockey schedule -season 20961 -out go.json
|
|
$ cmp py.json go.json && echo identical
|
|
identical # 364 games, 102536 bytes
|
|
# and the same for CSV: 365 identical lines
|
|
# and for the whole archive: 27 seasons, 2119 games, 1779 played
|
|
```
|
|
|
|
## No runtime dependencies
|
|
|
|
The tree is built by a ~120-line builder on `html.parser`, because the only part
|
|
of the HTML5 tree construction algorithm this crawler relies on is that stray
|
|
`<tr>` elements land in an implicit `<tbody>` — which `table_rows()` handles
|
|
explicitly, exactly as in Go.
|
|
|
|
One deliberate difference from a browser: the site writes empty cells as
|
|
`<th class="…" />`, and HTML5 ignores that slash, so a real browser leaves the
|
|
cell open and swallows its siblings. This builder treats the tag as an empty
|
|
element, which is what the author meant and what keeps `child_elements()` honest.
|
|
That is also why the Go and Python trees agree on the header row (7 `th`, 6
|
|
non-empty labels) even though the Go parser closes the tag implicitly.
|
|
|
|
## Modules
|
|
|
|
| module | what it does |
|
|
| ----------------- | --------------------------------------------------------- |
|
|
| `dom.py` | the HTML tree builder and the node helpers |
|
|
| `parse.py` | layout profiles, the game reader, the invariants |
|
|
| `seasons.py` | league and season discovery |
|
|
| `fetch.py` | one GET per page, charset guard, skip-unchanged state file |
|
|
| `output.py` | JSON, NDJSON, CSV, golden-file checking |
|
|
| `cli.py` | `schedule`, `crawl`, `leagues` and the exit codes |
|
|
|
|
Exit codes match the Go CLI: `0` ok, `1` error, `2` usage, `3` markup drift or
|
|
golden mismatch, `4` timeout.
|