Fill developer, publisher, year and description from Wikipedia
The library carried titles, systems and genres but almost nothing else:
developer was 3.8% filled, publisher 2.9%, description 0%. These are columns
the app has always had and never been able to populate.
enrich_metadata.py reads them off the same articles the cover fetcher
locates. Only empty fields are touched unless --overwrite is given.
year 79% -> 96%
developer 3.8% -> 95%
publisher 2.9% -> 96%
description 0% -> 96%
Parsing infoboxes needed several guards, each found by checking output
rather than trusting the first pass:
* "Infobox video game" is a substring of "Infobox video game series", so
the loose test resolved Banjo-Kazooie to the series overview. Now
rejected, which also fixes the cover fetcher's article resolution.
* An article spans every release and its date block leads with the
original, so year is only filled when the article covers that platform.
Otherwise a DS port inherits the SNES original's year.
* A search hit that neither covers the platform nor closely matches the
title is discarded: "Dragon Ball Z Budokai" surfaces "Shin Budokai", a
different game on a different console. Left blank instead.
* Values are grouped under bold platform headings, tagged with region
codes, annotated with the platform in parentheses, and wrapped in
templates whose named parameters leak through. Each of those read as
the developer or publisher before being handled.
Developer, publisher and year are facts and written verbatim. Descriptions
are article summaries under CC BY-SA, stored with an attribution line.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -77,7 +77,7 @@ cd backend && dotnet build # 0 warnings expected
|
||||
|
||||
### Cover art
|
||||
|
||||
`tools/cover-art/fetch_art.py` fills in box art, pushing each image through the
|
||||
`tools/library/fetch_art.py` fills in box art, pushing each image through the
|
||||
app's own `POST /api/images` so it gets the same validation and WebP re-encoding
|
||||
as a manual upload. Standard library only — no virtualenv, and neither source
|
||||
needs an account.
|
||||
@@ -95,7 +95,7 @@ It tries two sources in order:
|
||||
one per second and cached; the API returns 429 if pushed harder.
|
||||
|
||||
```bash
|
||||
cd tools/cover-art
|
||||
cd tools/library
|
||||
python3 fetch_art.py --password '...' --dry-run # report matches, change nothing
|
||||
python3 fetch_art.py --password '...' # download and attach
|
||||
python3 fetch_art.py --password '...' --overwrite # also replace existing art
|
||||
@@ -123,6 +123,52 @@ Fix the system field and re-run with `--overwrite` to correct them.
|
||||
Art is publisher copyright. Fetching it for a private collection is ordinary
|
||||
practice for library software; redistributing it is a different question.
|
||||
|
||||
### Metadata enrichment
|
||||
|
||||
`tools/library/enrich_metadata.py` fills developer, publisher, year and
|
||||
description from the same Wikipedia articles the cover fetcher locates. Only
|
||||
empty fields are touched unless `--overwrite` is passed — anything typed by hand
|
||||
outranks anything derived here.
|
||||
|
||||
```bash
|
||||
cd tools/library
|
||||
python3 enrich_metadata.py --password '...' --dry-run
|
||||
python3 enrich_metadata.py --password '...'
|
||||
python3 enrich_metadata.py --password '...' --fields developer,publisher
|
||||
```
|
||||
|
||||
Coverage went from this to this:
|
||||
|
||||
| Field | Before | After |
|
||||
| --- | --- | --- |
|
||||
| year | 79% | 96% |
|
||||
| developer | 3.8% | 95% |
|
||||
| publisher | 2.9% | 96% |
|
||||
| description | 0% | 96% |
|
||||
|
||||
Developer, publisher and year are facts, written verbatim. Descriptions are
|
||||
article summaries, which are CC BY-SA, so each is stored with an attribution
|
||||
line naming the source article.
|
||||
|
||||
Parsing an infobox is messier than it looks, and the guards matter:
|
||||
|
||||
- **Series articles are rejected.** A substring test for `Infobox video game`
|
||||
also matches `Infobox video game series`, which resolved Banjo-Kazooie to the
|
||||
series overview instead of the 1998 game.
|
||||
- **Year is only filled when the article covers that platform.** An article
|
||||
spans every release, and its date block leads with the original — so a DS
|
||||
port would otherwise be dated to the SNES original.
|
||||
- **A search hit that neither covers the platform nor closely matches the title
|
||||
is discarded.** Our "Dragon Ball Z Budokai" surfaces "Dragon Ball Z: Shin
|
||||
Budokai", a different game on a different console. A blank field beats a
|
||||
confidently wrong one.
|
||||
- Platform headings (`'''PlayStation'''`), region codes (`JP`, `NA`), trailing
|
||||
platform annotations (`Rare (N64)`) and named template parameters (`title=`)
|
||||
are all stripped, since each one otherwise reads as the value itself.
|
||||
|
||||
Four games have no usable article: a typo'd title (`Brett Hull Hocky 95`),
|
||||
`Dragon Ball Z Budokai`, and two niche releases.
|
||||
|
||||
### Database changes
|
||||
|
||||
```bash
|
||||
|
||||
Reference in New Issue
Block a user