Sort_YM/ARCHITECTURE.md

[Русский](ARCHITECTURE.ru.md)

# Architecture

A Python console utility that reads Yandex Music liked tracks and sorts them into playlists named
`Genre — RU` / `Genre — INT`, using the unofficial `yandex-music` library.

## Structure

```
sort_ym/
  config.py     — reads config.toml (throttle, small_group_min, top_artists/top_albums,
                  cache/token paths)
  auth.py       — OAuth device-flow, token storage in .token
  ymclient.py   — Client construction, batching lists, retrying network errors
  fetch.py      — likes -> full Track -> cache/tracks.json (serialize_track); track artists ->
                  client.artists() -> cache/artist_genres.json (both fetched incrementally);
                  unique_artist_ids — unique artist ids from the track cache
  source.py     — ephemeral source: parses a link to an arbitrary playlist + loads its tracks/
                  artists into memory, never touches disk (see "--source" below)
  genres.py     — the real Yandex genre tree; classify_track — resolves track genre and artist
                  genres into a single sub-genre (root_id); ROOT_BUCKET — sub-genre -> 11 top-level buckets
  language.py   — hybrid language detection (lyrics API / genre hint / alphabet)
  lyrics.py     — text lyrics of RU-tracks: track_supplement().lyrics.full_lyrics ->
                  cache/lyrics_text.json (incrementally, atomic-write)
  analyze.py    — lyric analysis via local Ollama (/api/chat, think + JSON Schema) ->
                  cache/lyrics_analysis.json (written after each track)
  classify.py   — (sub-genre | top-level bucket, language) -> playlist name; with_label — source
                  label suffix for apply --source --label
  report.py     — two-pass report row building, writes out/report.csv, summaries
  apply.py      — playlist creation + idempotent track insertion
  digest.py     — aggregated library summary (top artists, genres, decades, top albums)
                  -> out/digest.md, fully offline without --source; Wording/playlist_wording —
                  digest text for likes vs an arbitrary playlist
  cli.py        — auth / fetch / report / apply / dedupe / digest commands; --source/--label
                  on report/apply/digest (see "Key decisions")

tests/
  test_genres.py — classify_track, dominant, bucket_for, genre tree
  test_language.py — language detection
  test_lyrics.py — lyrics text download, cache structure, resumability
  test_analyze.py — local Ollama integration, JSON Schema parsing, cache atomicity
  test_report.py — two-pass collapsing of small sub-genre groups
  test_fetch.py, test_apply.py — track/artist cache, apply idempotency
  test_digest.py — artist/album aggregation, long-tail collapsing, digest size bounds
  test_source.py — playlist URL parsing, matching client.tracks() response by id, handling
                  private/inaccessible playlists
  test_cli.py, test_config.py — --source/--label flag validation before any network access;
                  config.toml reading including [ollama] block

cache/  — API cache (in .gitignore)
out/    — reports (in .gitignore)
```

## Data flow

```
auth  → .token
fetch → users_likes_tracks() → client.tracks(batch) → cache/tracks.json (+ artist_ids)
      → client.artists(batch) over unique artist_ids → cache/artist_genres.json
report → genres.load_or_fetch_catalog() → cache/genre_catalog.json
       → language.fetch_api_languages() → cache/lyrics_lang.json
       → build_rows(): pass 1 — classify_track() per track (genre_raw + artist genres)
                        pass 2 — groups < small_group_min collapse into the top-level bucket
       → out/report.csv (dry-run, account unchanged)
apply --yes → users_playlists_create / users_playlists_insert_track
digest → genres.load_catalog() + language.load_lang_cache() (cache only, no network)
       → build_rows() → out/digest.md
lyrics  → ru_lyric_track_ids() → client.track_supplement() → cache/lyrics_text.json
analyze → cache/lyrics_text.json → Ollama /api/chat → cache/lyrics_analysis.json
digest  → + cache/lyrics_analysis.json → out/digest.md (aggregate block) + out/lyrics_digest.md

report/apply/digest --source <url> → source.parse_playlist_url() → (user_id, kind)
       → client.users_playlists(kind, user_id=...) → TrackShort[] → client.tracks(batch)
       → an in-memory dict (fetch.serialize_track shape, never written to disk)
       → client.artists(batch) → genres.load_or_fetch_catalog() (shared cache/genre_catalog.json)
       + language.load_lang_cache() (read-only) → build_rows()
       → out/report_source.csv | out/digest_source.md | apply → playlists on your own account
         (shared "Genre — RU/INT" or "... (label)" with --label)
```

## Key decisions

- **Track genre** is only available through the album (`Track.albums[0].genre`) — the track itself
  has no genre field, so it's one string per track, often generic (`rock`, `alternative`). Bucketing
  is done through the real `client.genres()` tree (parent → sub-genres), not a guessed slug list —
  the `ROOT_BUCKET` table in `genres.py` relies on confirmed ids, not assumptions. Genres without a
  match fall into the `other` bucket and are printed in `report`'s output for manual table extension.

- **Sub-genre classification (`genres.classify_track`)**: track genre is one string per album and
  often generic. An artist (`client.artists()`, a batch endpoint) can have several tags, more
  precise than a single album genre. `classify_track(genre_raw, artist_genre_lists, catalog)`
  resolves both signals into a single sub-genre (`root_id`, e.g. `punk`/`indie`/`allrock`) at the
  **track** level, not the artist level — so the same artist with tracks of different character ends
  up in different playlists:
  1. Signals agree (track genre is among the artist's genres) → use it — granular per track.
  2. Track has no genre → use the most frequent (`dominant`) genre among the track's artists.
  3. Artist(s) have no tags → use the track's genre.
  4. Conflict: the album tag is generic/"umbrella" (`GENERIC_SLUGS`: `rock`, `alternative`, `pop`,
     ... — Yandex's low-confidence default) → the artist's genre wins (`dominant` over the combined
     tags of all the track's artists, tie-break by order of appearance). The album tag is
     precise/specific (`numetal`, `dnb`, ...) → it wins — likely a genuine genre experiment by the
     artist rather than noise worth correcting.

  The result (`fine`) is collapsed in a two-pass process in `report.build_rows`: first the sizes of
  `(fine, lang)` groups are counted, then groups smaller than `config.small_group_min` move to the
  parent top-level bucket via `genres.coarse_of(fine)` (a thin wrapper over `ROOT_BUCKET`). The pass
  is single (not iterative) — `coarse_of` is deterministic from `fine`, so there's no re-counting of
  sizes or looping.

- **Language** is determined hybrid, in priority order:
  1. `client.track_supplement(track_id).lyrics.text_language` — the only field in the library with
     the actual lyrics language. It's only present in the deprecated `track_supplement` method, not
     in the newer `tracks_lyrics()` — that one returns the text but not the language. Used
     deliberately despite the library's own deprecation notice.
  2. A genre-slug hint (`rusrap`, `shanson`, `bard`, ...).
  3. An alphabet heuristic on the title/artist (Cyrillic vs Latin, `unicodedata`).
  4. Otherwise — `UNKNOWN` → a separate "Undetermined" playlist.

- **Lyric analysis (`lyrics.py`, `analyze.py`)**: by default only Russian-language tracks are
  analyzed — the prompt and the narrative fields (`summary`/`resonance`/`key_line`) are written
  in Russian and were validated against a Russian-lyrics pilot run; quality on other languages is
  unverified. `sort_ym lyrics --all-languages` lifts the filter and fetches/analyzes every track
  with available lyrics regardless of detected language (`lyrics.ru_lyric_track_ids(...,
  all_languages=True)`). Text fetching and analysis are **two separate commands** on purpose: the fetch is network-bound against the deprecated `track_supplement`,
  the analysis is a multi-hour local GPU job, and a prompt or model change must be re-runnable
  without touching Yandex again. `analyze` and `digest` construct no `Client` at all.

  The Ollama call (`/api/chat`) combines `think: true` with a full JSON Schema in `format`:
  reasoning goes to a separate `message.thinking` field (never cached) while only the final
  answer is grammar-constrained, so the model reasons freely and the result still parses.
  `num_predict: -1` — a token cap would truncate mid-JSON and turn a slow answer into an invalid
  one; the bound is the request timeout instead. `temperature: 0.65`, not 0: at 0 the narrative
  fields flatten out.

  `themes` are free-form but pattern-constrained to English snake_case
  (`^[a-z0-9][a-z0-9_-]{2,29}$`) — in the pilot, an unconstrained field drifted between Russian and
  English inside a single run, making the values impossible to aggregate; the pattern makes
  Cyrillic mechanically unrepresentable at decode time. They are English for the same reason
  genre slugs and `ROOT_BUCKET` keys are: they are aggregation keys. `mood` is a closed enum that
  deliberately excludes irony/sarcasm — that axis is `stance`, and mixing the two produced
  compound values like `"aggressive/triumphant"`.

  Resumability: the cache is written atomically after **every single track** (not batched by 20
  as in `language.py`) — inference costs tens of seconds, so a crash must cost at most one track.
  An entry is reused only when `(model, prompt_version, lyrics_hash)` all match, so a corrected
  lyrics text or a bumped prompt invalidates exactly what it should. A timeout or a broken
  response is stored as an `error` marker and the batch continues; error entries are never
  considered fresh, so the next run retries precisely those tracks.

- **Track identifiers**: a track has a composite `track_id` (`"id:album_id"`, used as the cache key
  and to check against tracks already in a playlist) and separate `id`/`album_id` (needed as
  separate parameters for `track_supplement`, `tracks_lyrics`, and `users_playlists_insert_track`).

- **`apply` idempotency**: before adding tracks to a playlist, its current contents are read
  (`users_playlists(kind)`), and tracks already present are skipped. Re-running `apply` is safe and
  creates no duplicates. Likes are never touched, nothing is ever deleted.

- **Throttling and resumability**: every network loop (fetching tracks, languages) pauses between
  requests (`config.toml`, `[throttle]`) and writes the cache atomically (temp file + rename) after
  each batch — an interrupted run can simply be restarted, and already-fetched data is not
  requested again.

- **Library digest (`digest.py`)**: the `digest` command's output is meant to be pasted into an LLM
  chat, so it's built as a set of aggregates with fixed ceilings rather than a per-track dump — on a
  ~2000-track library that's ~150 lines, regardless of whether the library has 500 tracks or 10000
  (genre sections are bounded by the size of `ROOT_BUCKET`/`BUCKET_LABELS`; top artists and top
  albums are bounded by the `[digest]` config, not the library size; anything past the top is
  collapsed into a single sentence with aggregate numbers). Artists are counted **by name**, not via
  `zip(artists, artist_ids)` — `fetch.serialize_track` puts every named artist into `artists` but only
  those that also have an id into `artist_ids`, so the two lists can differ in length for a given
  track. For the same reason, an artist's own genre tags (`artist_tags`) are only looked up from
  tracks where the two lists' lengths match — otherwise an artist could end up with someone else's
  tags. Rows from `report.build_rows()` are joined back to `cache/tracks.json` by the `(id,
  album_id)` pair taken from the cache's **values**, not by reconstructing the key as
  `f"{id}:{album_id}"` — the cache key is the original id from `likes.tracks_ids`, and for
  album-less tracks it doesn't match that format. The album's year and title (`year`,
  `album_title`) are read from the same `albums[0]` as `album_id` — otherwise the title could end up
  paired with the wrong id. Old cache entries are backfilled with the same mechanism used for
  `artist_ids`: the migration checks whether the `year` key is **present**, not whether its value is
  truthy, otherwise tracks with a legitimately unknown year would be re-fetched on every `fetch` run.
  The command is fully offline — it never constructs a `Client`, unlike `report`/`apply`.

- **Arbitrary playlist by link (`source.py`, `--source` on `report`/`apply`/`digest`)**: an
  external source is **ephemeral** — its data is fetched into memory on every run and never
  written to disk (`source.py` imports no `json` and never touches `cache_dir` at all). The one
  exception is the shared `cache/genre_catalog.json` (Yandex's genre tree — a reference dataset,
  not a specific playlist's history), reused as usual via `genres.load_or_fetch_catalog`. Writes
  always target your own account: a foreign `user_id` is only ever passed into
  `client.users_playlists(kind, user_id=...)` (a read); no call in `apply.py`
  (`users_playlists_create`, `users_playlists_insert_track`) accepts one. The `yandex_music`
  library has no URL parser at all — `source.parse_playlist_url` parses
  `https://music.yandex.ru/users/<login>/playlists/<kind>` (plus a best-effort `/playlists/<uuid>`
  form via `client.playlist(uuid)`) itself, with no external dependency. `Playlist.tracks` is a
  list of `TrackShort` (no genre/album data); full `Track` objects are fetched with the same batch
  method used for likes, `client.tracks(...)`, but the response is matched back to the requested
  id by value (`by_id.get(...)`), not a positional `zip` — a foreign playlist is noticeably more
  likely to contain unavailable tracks, and a shortened response under `zip` would silently mix up
  track data (`Playlist.fetch_tracks()` is a trap — it just re-calls `users_playlists` and returns
  the same short objects, so it isn't used). `UnauthorizedError` extends `YandexMusicError`
  directly, not `NetworkError`, so `ymclient.with_retries` never catches or retries it — it used to
  be unhandled anywhere in the project; it's now caught in `source._load_playlist` (a private or
  inaccessible playlist) and by a top-level net in `cli.main()` (an expired token on any command).
  Language for an external source is not refined via `track_supplement` (one uncached network
  request per track on every run) — it falls back to the existing `cache/lyrics_lang.json` (read
  only) plus the heuristics; a track whose language disagrees with what the precise check would
  have picked can land in the wrong language playlist under `apply --source` without `--label`.

## Storage

- `.token` — the account's OAuth token (not committed).
- `cache/tracks.json` — liked track metadata, including `artist_ids` (the track's artist ids),
  `album_title` and `year` (the album's title and release year, `Album.title`/`Album.year`; `year`
  can be `null`), plus `added_at` (when the track was liked, `TrackShort.timestamp`),
  `duration_ms`, `track_version`/`album_version`, `release_date`, `album_likes_count` - raw API
  fields not used for classification but available via `report --extra` (see below).
- `cache/artist_genres.json` — `{artist_id: {name, genres: [slug, ...], counts, ratings}}`, one or
  more genres per artist; `counts`/`ratings` are `Artist.counts`/`Artist.ratings`, same
  `report --extra`-only purpose as `added_at` above.
- `cache/genre_catalog.json` — a flat snapshot of Yandex's genre tree (id -> title, root_id).
- `cache/lyrics_lang.json` — lyrics language per track (id -> language code | null).
- `cache/lyrics_text.json` — lyrics text per track (numeric id -> text | null, RU tracks only).
- `cache/lyrics_analysis.json` — per-track analysis record: `{track_id: {model, prompt_version,
  lyrics_hash, mood, themes, emotional_arc, pov, tone, description, error?}}`.
- `out/lyrics_digest.md` — full per-track reading grouped by primary mood.
- `config.toml` — includes `[ollama]` block with `host`, `model`, `prompt_version`, `timeout`,
  `keep_alive`.
- `out/report.csv` — a preview of the playlist breakdown before `apply` (including the
  `fine_bucket` column — the sub-genre before any collapsing into a top-level bucket).
  `report --order playlist` keeps source track order instead of sorting by target playlist;
  individual `--timestamp`/`--duration`/`--version`/`--album`/`--artist` flags (combinable) add
  that group's columns, `--extra` adds every group at once. Flag -> CSV columns mapping is
  `report.EXTRA_GROUPS`; `report.resolve_extra_columns()` expands the selected groups into a
  column list for the CLI.
- `out/digest.md` — a compact library summary for pasting into an LLM chat (top artists, top
  albums, genre/language/decade breakdowns).
- `out/report_source.csv`, `out/digest_source.md` — the same reports for a `--source` run; a
  single overwritten "last external source" slot, no history of other playlists is kept.