Fill developer, publisher, year and description from Wikipedia

The library carried titles, systems and genres but almost nothing else:
developer was 3.8% filled, publisher 2.9%, description 0%. These are columns
the app has always had and never been able to populate.

enrich_metadata.py reads them off the same articles the cover fetcher
locates. Only empty fields are touched unless --overwrite is given.

  year         79%  -> 96%
  developer   3.8%  -> 95%
  publisher   2.9%  -> 96%
  description   0%  -> 96%

Parsing infoboxes needed several guards, each found by checking output
rather than trusting the first pass:

  * "Infobox video game" is a substring of "Infobox video game series", so
    the loose test resolved Banjo-Kazooie to the series overview. Now
    rejected, which also fixes the cover fetcher's article resolution.
  * An article spans every release and its date block leads with the
    original, so year is only filled when the article covers that platform.
    Otherwise a DS port inherits the SNES original's year.
  * A search hit that neither covers the platform nor closely matches the
    title is discarded: "Dragon Ball Z Budokai" surfaces "Shin Budokai", a
    different game on a different console. Left blank instead.
  * Values are grouped under bold platform headings, tagged with region
    codes, annotated with the platform in parentheses, and wrapped in
    templates whose named parameters leak through. Each of those read as
    the developer or publisher before being handled.

Developer, publisher and year are facts and written verbatim. Descriptions
are article summaries under CC BY-SA, stored with an attribution line.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-04 12:34:55 -04:00
co-authored by Claude Opus 5
parent 130921cf89
commit 771b34bb4b
7 changed files with 503 additions and 9 deletions
+48 -2
View File
@@ -77,7 +77,7 @@ cd backend && dotnet build # 0 warnings expected
### Cover art
`tools/cover-art/fetch_art.py` fills in box art, pushing each image through the
`tools/library/fetch_art.py` fills in box art, pushing each image through the
app's own `POST /api/images` so it gets the same validation and WebP re-encoding
as a manual upload. Standard library only — no virtualenv, and neither source
needs an account.
@@ -95,7 +95,7 @@ It tries two sources in order:
one per second and cached; the API returns 429 if pushed harder.
```bash
cd tools/cover-art
cd tools/library
python3 fetch_art.py --password '...' --dry-run # report matches, change nothing
python3 fetch_art.py --password '...' # download and attach
python3 fetch_art.py --password '...' --overwrite # also replace existing art
@@ -123,6 +123,52 @@ Fix the system field and re-run with `--overwrite` to correct them.
Art is publisher copyright. Fetching it for a private collection is ordinary
practice for library software; redistributing it is a different question.
### Metadata enrichment
`tools/library/enrich_metadata.py` fills developer, publisher, year and
description from the same Wikipedia articles the cover fetcher locates. Only
empty fields are touched unless `--overwrite` is passed — anything typed by hand
outranks anything derived here.
```bash
cd tools/library
python3 enrich_metadata.py --password '...' --dry-run
python3 enrich_metadata.py --password '...'
python3 enrich_metadata.py --password '...' --fields developer,publisher
```
Coverage went from this to this:
| Field | Before | After |
| --- | --- | --- |
| year | 79% | 96% |
| developer | 3.8% | 95% |
| publisher | 2.9% | 96% |
| description | 0% | 96% |
Developer, publisher and year are facts, written verbatim. Descriptions are
article summaries, which are CC BY-SA, so each is stored with an attribution
line naming the source article.
Parsing an infobox is messier than it looks, and the guards matter:
- **Series articles are rejected.** A substring test for `Infobox video game`
also matches `Infobox video game series`, which resolved Banjo-Kazooie to the
series overview instead of the 1998 game.
- **Year is only filled when the article covers that platform.** An article
spans every release, and its date block leads with the original — so a DS
port would otherwise be dated to the SNES original.
- **A search hit that neither covers the platform nor closely matches the title
is discarded.** Our "Dragon Ball Z Budokai" surfaces "Dragon Ball Z: Shin
Budokai", a different game on a different console. A blank field beats a
confidently wrong one.
- Platform headings (`'''PlayStation'''`), region codes (`JP`, `NA`), trailing
platform annotations (`Rare (N64)`) and named template parameters (`title=`)
are all stripped, since each one otherwise reads as the value itself.
Four games have no usable article: a typo'd title (`Brett Hull Hocky 95`),
`Dragon Ball Z Budokai`, and two niche releases.
### Database changes
```bash