[Русский](ARCHITECTURE.ru.md) # Architecture A Python console utility that reads Yandex Music liked tracks and sorts them into playlists named `Genre — RU` / `Genre — INT`, using the unofficial `yandex-music` library. ## Structure ``` sort_ym/ config.py — reads config.toml (throttle, small_group_min, top_artists/top_albums, cache/token paths) auth.py — OAuth device-flow, token storage in .token ymclient.py — Client construction, batching lists, retrying network errors fetch.py — likes -> full Track -> cache/tracks.json; track artists -> client.artists() -> cache/artist_genres.json (both fetched incrementally) genres.py — the real Yandex genre tree; classify_track — resolves track genre and artist genres into a single sub-genre (root_id); ROOT_BUCKET — sub-genre -> 11 top-level buckets language.py — hybrid language detection (lyrics API / genre hint / alphabet) classify.py — (sub-genre | top-level bucket, language) -> playlist name report.py — two-pass report row building, writes out/report.csv, summaries apply.py — playlist creation + idempotent track insertion digest.py — aggregated library summary (top artists, genres, decades, top albums) -> out/digest.md, fully offline cli.py — auth / fetch / report / apply / dedupe / digest commands tests/ test_genres.py — classify_track, dominant, bucket_for, genre tree test_language.py — language detection test_report.py — two-pass collapsing of small sub-genre groups test_fetch.py, test_apply.py — track/artist cache, apply idempotency test_digest.py — artist/album aggregation, long-tail collapsing, digest size bounds cache/ — API cache (in .gitignore) out/ — reports (in .gitignore) ``` ## Data flow ``` auth → .token fetch → users_likes_tracks() → client.tracks(batch) → cache/tracks.json (+ artist_ids) → client.artists(batch) over unique artist_ids → cache/artist_genres.json report → genres.load_or_fetch_catalog() → cache/genre_catalog.json → language.fetch_api_languages() → cache/lyrics_lang.json → build_rows(): pass 1 — classify_track() per track (genre_raw + artist genres) pass 2 — groups < small_group_min collapse into the top-level bucket → out/report.csv (dry-run, account unchanged) apply --yes → users_playlists_create / users_playlists_insert_track digest → genres.load_catalog() + language.load_lang_cache() (cache only, no network) → build_rows() → out/digest.md ``` ## Key decisions - **Track genre** is only available through the album (`Track.albums[0].genre`) — the track itself has no genre field, so it's one string per track, often generic (`rock`, `alternative`). Bucketing is done through the real `client.genres()` tree (parent → sub-genres), not a guessed slug list — the `ROOT_BUCKET` table in `genres.py` relies on confirmed ids, not assumptions. Genres without a match fall into the `other` bucket and are printed in `report`'s output for manual table extension. - **Sub-genre classification (`genres.classify_track`)**: track genre is one string per album and often generic. An artist (`client.artists()`, a batch endpoint) can have several tags, more precise than a single album genre. `classify_track(genre_raw, artist_genre_lists, catalog)` resolves both signals into a single sub-genre (`root_id`, e.g. `punk`/`indie`/`allrock`) at the **track** level, not the artist level — so the same artist with tracks of different character ends up in different playlists: 1. Signals agree (track genre is among the artist's genres) → use it — granular per track. 2. Track has no genre → use the most frequent (`dominant`) genre among the track's artists. 3. Artist(s) have no tags → use the track's genre. 4. Conflict: the album tag is generic/"umbrella" (`GENERIC_SLUGS`: `rock`, `alternative`, `pop`, ... — Yandex's low-confidence default) → the artist's genre wins (`dominant` over the combined tags of all the track's artists, tie-break by order of appearance). The album tag is precise/specific (`numetal`, `dnb`, ...) → it wins — likely a genuine genre experiment by the artist rather than noise worth correcting. The result (`fine`) is collapsed in a two-pass process in `report.build_rows`: first the sizes of `(fine, lang)` groups are counted, then groups smaller than `config.small_group_min` move to the parent top-level bucket via `genres.coarse_of(fine)` (a thin wrapper over `ROOT_BUCKET`). The pass is single (not iterative) — `coarse_of` is deterministic from `fine`, so there's no re-counting of sizes or looping. - **Language** is determined hybrid, in priority order: 1. `client.track_supplement(track_id).lyrics.text_language` — the only field in the library with the actual lyrics language. It's only present in the deprecated `track_supplement` method, not in the newer `tracks_lyrics()` — that one returns the text but not the language. Used deliberately despite the library's own deprecation notice. 2. A genre-slug hint (`rusrap`, `shanson`, `bard`, ...). 3. An alphabet heuristic on the title/artist (Cyrillic vs Latin, `unicodedata`). 4. Otherwise — `UNKNOWN` → a separate "Undetermined" playlist. - **Track identifiers**: a track has a composite `track_id` (`"id:album_id"`, used as the cache key and to check against tracks already in a playlist) and separate `id`/`album_id` (needed as separate parameters for `track_supplement`, `tracks_lyrics`, and `users_playlists_insert_track`). - **`apply` idempotency**: before adding tracks to a playlist, its current contents are read (`users_playlists(kind)`), and tracks already present are skipped. Re-running `apply` is safe and creates no duplicates. Likes are never touched, nothing is ever deleted. - **Throttling and resumability**: every network loop (fetching tracks, languages) pauses between requests (`config.toml`, `[throttle]`) and writes the cache atomically (temp file + rename) after each batch — an interrupted run can simply be restarted, and already-fetched data is not requested again. - **Library digest (`digest.py`)**: the `digest` command's output is meant to be pasted into an LLM chat, so it's built as a set of aggregates with fixed ceilings rather than a per-track dump — on a ~2000-track library that's ~150 lines, regardless of whether the library has 500 tracks or 10000 (genre sections are bounded by the size of `ROOT_BUCKET`/`BUCKET_LABELS`; top artists and top albums are bounded by the `[digest]` config, not the library size; anything past the top is collapsed into a single sentence with aggregate numbers). Artists are counted **by name**, not via `zip(artists, artist_ids)` — `fetch._serialize` puts every named artist into `artists` but only those that also have an id into `artist_ids`, so the two lists can differ in length for a given track. For the same reason, an artist's own genre tags (`artist_tags`) are only looked up from tracks where the two lists' lengths match — otherwise an artist could end up with someone else's tags. Rows from `report.build_rows()` are joined back to `cache/tracks.json` by the `(id, album_id)` pair taken from the cache's **values**, not by reconstructing the key as `f"{id}:{album_id}"` — the cache key is the original id from `likes.tracks_ids`, and for album-less tracks it doesn't match that format. The album's year and title (`year`, `album_title`) are read from the same `albums[0]` as `album_id` — otherwise the title could end up paired with the wrong id. Old cache entries are backfilled with the same mechanism used for `artist_ids`: the migration checks whether the `year` key is **present**, not whether its value is truthy, otherwise tracks with a legitimately unknown year would be re-fetched on every `fetch` run. The command is fully offline — it never constructs a `Client`, unlike `report`/`apply`. ## Storage - `.token` — the account's OAuth token (not committed). - `cache/tracks.json` — liked track metadata, including `artist_ids` (the track's artist ids), `album_title` and `year` (the album's title and release year, `Album.title`/`Album.year`; `year` can be `null`). - `cache/artist_genres.json` — `{artist_id: {name, genres: [slug, ...]}}`, one or more genres per artist. - `cache/genre_catalog.json` — a flat snapshot of Yandex's genre tree (id -> title, root_id). - `cache/lyrics_lang.json` — lyrics language per track (id -> language code | null). - `out/report.csv` — a preview of the playlist breakdown before `apply` (including the `fine_bucket` column — the sub-genre before any collapsing into a top-level bucket). - `out/digest.md` — a compact library summary for pasting into an LLM chat (top artists, top albums, genre/language/decade breakdowns).