0.2.79 - Slow api fixes

CLAUDE.md

@@ -869,6 +869,121 @@ Ten-second progress reports and per-keystroke searches are DEBUG on purpose.
 the admin rail — bump it with a meaningful server change; nothing else identifies which
 tree a container was deployed from.
 
+**A slow request says where its time went.** `internal/timing` is a per-request trace
+carried in the context by pointer, the `requestIdentity` arrangement, and every layer
+under the handler records against it: an `http.RoundTripper` wrapper on all eight upstream
+clients (Emby, both *arrs, Tracearr, MDBList, Bazarr, OpenSubtitles, the integrations
+dispatcher), a `pgx.QueryTracer` on the pool, and the cache's own `Get`/`Set`/`Delete`.
+`withLogging` attaches the result as a `breakdown` field once the request passes
+`slowRequestThreshold` — an operator override over `MEMBY_SLOW_REQUEST_THRESHOLD`, 500ms
+deployed — and the console gives it a heading of its own in the log drawer and a second
+line on the row. A duration column could only ever say that something was slow; this is
+what makes it actionable:
+
+```
+GET /v1/home  9.51s  breakdown=emby.favourites=7.80s fanout=7.81s rows=1.40s db=18ms encode=31ms miss=1
+```
+
+Things to preserve:
+
+- **It measures stages, not spans in a tree.** Most of what matters here is concurrent —
+  Home fans five Emby queries out at once — so a total of wall-clock spans would exceed
+  the request's own duration and mean nothing. What a stage reports is summed busy time
+  and a **call count**, and the count is the half that finds duplicate work: `emby=7.8s
+  ×14` on a page that should make two lookups is the finding whatever the seconds say.
+- **The transport times the body too**, not only `RoundTrip`. Headers arrive while several
+  hundred kilobytes of row JSON is still on the wire, so timing the round trip alone would
+  under-report exactly the largest responses.
+- **`timing.WithLabel` is what makes a fan-out legible.** Five concurrent Emby queries
+  under one `emby=` stage say the launcher waited on Emby without saying which of them it
+  waited on, which is the whole of the next question. Each home row, and each leg of the
+  season-finale chain, names its own stage. A label is only ever a finer name for a stage
+  the client would have recorded anyway, so its absence changes nothing.
+- **`elsewhere` is the duration less the *largest* stage**, never less their sum, and is
+  clamped at zero. It is deliberately not called "gateway processing": with overlapping
+  stages no such figure exists, and what this is instead is a lower bound on time nothing
+  accounted for.
+- **Everything is inert without a trace.** A scheduled sync, a health probe and every unit
+  test call the same helpers and pay one nil check.
+
+**And the fixes the trace found.** They are all one shape — work on the critical path that
+did not have to be there — and each is easy to reintroduce:
+
+- **The automatic My Shows check ran a hundred times per episode.** It hangs off playback
+  reports, one arrives every ten seconds, and its own guard (half watched) is true for
+  every one of those after the halfway mark — so the second half of every episode ran two
+  serial Emby lookups, a Sonarr catalogue read and a Postgres write per tick and threw all
+  but the first away at the durable insert. `followChecks.claim` is a map lookup that comes
+  *first*, before the feature check and long before Emby. Deduplicating on the insert was
+  correct and was never the problem; what it could not do is prevent the work leading up to
+  it.
+- **`InvalidateUser` walked the whole keyspace.** Redis SCAN's cost is a property of the
+  keyspace rather than of the pattern, so dropping one viewer's dozen cached rows meant
+  paging through every item lookup the household had ever cached — on the playback stop
+  report, which is the request a television makes as somebody presses Back out of a film.
+  `Cache.Set` now indexes user-scoped keys into `idx:u:<user>` in the same pipeline as the
+  write, and invalidation reads that set. The SCAN survives as the fallback for a user with
+  no index, which is the honest answer for a gateway upgraded with older keys still live.
+- **Stable metadata lived where invalidation could reach it.** A person's biography, a
+  film's extras and whether an episode closes its season carry no user data and cannot
+  change when somebody finishes an episode, yet all three sat under `u:<user>:` and were
+  destroyed several times an evening by playback stops — then re-read from Emby by the next
+  card anybody looked at. `cache.MetadataKey` is where they live now: outside the
+  invalidated namespace and unkeyed by viewer, so one lookup answers for the household. The
+  rule for putting something there is narrow and worth stating: **the value must contain
+  nothing derived from a viewer.** Anything carrying `UserData` belongs in `UserKey`, where
+  invalidation can reach it — which is why the *filmography* stayed behind while the
+  biography moved.
+- **Identical concurrent misses each asked upstream.** `s.cachedRead` / `buildCached`
+  (`coalesce.go`) put a `singleflight` between the cache miss and the upstream call, keyed
+  by the cache key. The log showed one person lookup answering in 2.2s, then 4.6s, then
+  7.7s within a few seconds; nothing was getting slower, three televisions had missed the
+  same key at the same moment and Emby was serving three copies of one expensive query.
+  The cache could not help — nothing had been written when the second and third arrived.
+  Two properties are load-bearing: **the shared work does not inherit the caller's
+  cancellation** (singleflight hands every joiner the first caller's result, so a D-pad
+  moving off a card would otherwise fail everybody queued behind it), and **a joiner is
+  counted** on the breakdown, because "slow" and "queued behind itself" are different
+  problems with different fixes. It is applied to the person biography and filmography,
+  related titles, extras, series episodes and item detail — every route the launcher warms
+  on focus and then asks for again on the press.
+- **Serial chains that were never chains.** The season-finale route fetched Sonarr's series
+  catalogue *after* both Emby lookups it does not depend on; the extras route made its two
+  lookups in turn; `filterRecommendationPermissions` walked its hundred-id batches one at a
+  time on the home tail. All three now run together, and the permission check keeps its
+  original refusal — one failed batch hides every candidate, because a partial answer is
+  not evidence that the rest is permitted.
+- **Home read its ranking evidence after the rows arrived.** `rankingContext` is seven
+  Postgres reads and `UserRowStats` an eighth, all of them functions of the viewer's id and
+  nothing else — so they ran with nothing else in flight, on the critical path, for answers
+  that were available before Emby was asked anything. They are issued *beside* the Emby
+  fan-out now (`rankingInputs`, `personalizeTitlesWith`), and the seven inside
+  `rankingContext` are issued together. The two that are genuinely a chain stay one, but
+  their *writes* are ordered rather than their reads: `ApplyOnboarding` and
+  `ApplyExplicitPreference` are not commutative and the ranking must never depend on which
+  query answered first.
+- **Two documents were queried per request and could not change per request.** The feature
+  policy is read from sixteen places plus `/v1/status`, which every open television polls
+  every ten seconds; the household completion scores are a six-month aggregate whose answer
+  is *the same for every viewer in the house*. Both are cached in memory — five seconds and
+  a minute — and the operator's own write invalidates the first outright, so the window is
+  "how long until another instance notices", not "how long until my change takes effect".
+  A failed household read returns the previous reading rather than caching an empty one.
+- **The connection pool was four.** `pgxpool`'s default is `max(4, GOMAXPROCS)`, which on
+  the NAS is fewer than one household's televisions — and a query waiting for a connection
+  is invisible, because the query time stays honest while the request is slow anyway.
+  `store.Open` floors it at 16 with 4 kept warm, and leaves alone any deployment that
+  stated `pool_max_conns` for itself.
+
+Two things the trace found and did **not** change, because both are behaviour rather than
+waste. `fieldsRow` asks Emby for `People` and `RecursiveItemCount`, which are the two
+expensive fields on a row query — but the weighted ranker scores person affinity from the
+first and a series card prints its episode count from the second, so dropping either is a
+feature decision. And the home cache key carries the device id, which multiplies the
+rebuild by the number of televisions; that is there because a recommendation's reason names
+"this TV", although the recommendations it decorates come from a cache that is not
+device-keyed. Both are worth revisiting deliberately.
+
 **Emby's own version rides the reachability probe**, and the television prints it beside the
 gateway's on Settings → About: `0.1.50 (4.10.0.21)`. `emby.Client.Ping` already asks
 `/System/Info/Public`, which carries `Version`, so reading it costs nothing — a second call