Add the Python variant, agreeing with Go byte for byte
A second, independent implementation of the same contract, in the same repo: - swehockey/dom.py builds the tree on html.parser, so the variant has no runtime dependencies; the only HTML5 behaviour it depends on - stray <tr> landing in an implicit <tbody> - is handled explicitly by table_rows(). - parse.py mirrors the Go layout profiles, dual-render checks, carry-forward and invariants; a DriftError is raised in the same places with the same advice. - cli.py offers the same three commands, flags and exit codes as the Go main. - tests/ assert against the Go golden files, so the suite is also a cross-implementation check. 23 offline tests plus an opt-in live test. Verified against the live site: JSON and CSV are byte-identical to the Go output (364 games, 102536 bytes / 365 lines), and the full archive crawl gives the same 27 seasons, 2119 games and 1779 played. Also split the DriftError hint out in Go, so a golden mismatch no longer tells the reader to extend layouts[].
This commit is contained in:
1 parent
6b701819ca
commit
47e1811d9e
15 files changed
+1821
-3
No files matched your search
@@ -90,6 +90,50 @@ $ ./swehockey crawl -league 20961 -on-drift record | jq -s '[.[] | .games | leng
|
||||
2119
|
||||
```
|
||||
|
||||
## Two implementations, one contract
|
||||
|
||||
The crawler exists twice, in Go and in Python, and that is a feature: two
|
||||
independent parsers of the same page that have to agree.
|
||||
|
||||
| | Go | Python |
|
||||
| --- | --- | --- |
|
||||
| entry point | `./swehockey schedule -season 20961` | `python3 -m swehockey.cli schedule --season 20961` |
|
||||
| tests | `go test ./...` (offline) | `python3 -m unittest discover -s tests -t .` (offline) |
|
||||
| live test | `SWEHOCKEY_LIVE=1 go test -run TestLive` | `SWEHOCKEY_LIVE=1 python3 -m unittest tests.test_swehockey.TestLive` |
|
||||
| dependencies | `golang.org/x/net/html` | none — the tree is built on `html.parser` |
|
||||
|
||||
Both share `testdata/`: the Python suite asserts its output against the **Go
|
||||
golden files**, so a change in one implementation that the other does not share
|
||||
fails a test. Verified on the live page as well:
|
||||
|
||||
```
|
||||
$ ./swehockey schedule -season 20961 -out go.json
|
||||
$ python3 -m swehockey.cli schedule --season 20961 --out py.json
|
||||
$ cmp go.json py.json # 364 games, 102536 bytes, identical
|
||||
$ ./swehockey schedule -season 20961 -format csv -out go.csv
|
||||
$ python3 -m swehockey.cli schedule --season 20961 --format csv --out py.csv
|
||||
$ cmp go.csv py.csv # 365 lines, identical
|
||||
$ python3 -m swehockey.cli crawl --league 20961 | jq -s '[.[].games|length]|add'
|
||||
2119 # same as the Go crawl
|
||||
```
|
||||
|
||||
Same layout profiles, same invariants, same canonical order, same exit codes
|
||||
(`0` ok, `1` error, `2` usage, `3` drift, `4` timeout). The only intentional
|
||||
difference is in the tree builder: Python treats the site's empty `<th … />`
|
||||
cells as empty elements instead of leaving them open the way a browser does,
|
||||
which is what makes the two trees agree. See `python/README.md`.
|
||||
|
||||
## Repository layout
|
||||
|
||||
```
|
||||
main.go parse.go seasons.go fetch.go output.go the Go crawler
|
||||
parse_test.go the Go tests
|
||||
python/swehockey/ the Python crawler
|
||||
python/tests/ the Python tests
|
||||
testdata/ HTML fixtures and golden JSON, shared by both
|
||||
README.md this file
|
||||
```
|
||||
|
||||
## Commands
|
||||
|
||||
| command | what it does |
|
||||
|
||||
Reference in new issue
Block a user