data-entry-app/TECH_DEBT.md

# Tech Debt Audit & Remediation Plan

> Status: **Plan / not yet started.** Audit performed 2026-06-03 against `main`.
> Context: app has been through 11 versions. Dev runs on **SQLite (Windows)**;
> production is mid-migration to **PostgreSQL**. Six modules: Mix Calculator,
> Product Costing, Editor, Throughput, Reporting, Settings.
>
> Decisions already taken:
> - **Migrations:** adopt **Alembic** (replaces the startup `create_all` + ad-hoc `ALTER` scheme).
> - **Approach:** full audit first (this document), then execute in phases.

---

## Findings, ranked by severity

### P0 — correctness / data integrity

#### P0.1 — Money is stored as `Float` everywhere
Every cost / price / margin column is SQLAlchemy `Float`:

- `backend/app/models/product_costing.py` — `cleaned_product_cost_per_kg`, `grading_cost_per_kg`,
  `bagging_cost_per_kg`, `cracking_cost_per_kg`, `bag_cost_per_unit`, `freight_cost_per_unit`,
  `finished_product_delivered_cost`, `distributor_price`, `wholesale_price`, `cost_per_kg`,
  `distributor_margin`, `wholesale_margin`, `cost`, …
- `backend/app/models/assumption.py` — `grading_cost`, `bagging_cost`, `cracking_cost`,
  `bag_cost`, `cost_per_unit`.
- `backend/app/models/mix.py` — `quantity_kg`.
- `backend/app/models/mix_calculator.py` — `batch_size_kg`, `total_bags`, `total_kg`,
  `product_unit_size_kg`, `required_kg`, `mix_percentage`.
- `backend/app/models/product.py` — `distributor_margin`, `wholesale_margin`, `quantity_kg`.

**Why it matters:** for a costing-and-pricing tool, binary floating point introduces rounding
drift in money. It is also a **SQLite ↔ Postgres divergence point** — SQLite stores loose floats,
Postgres `Numeric` is exact, so the same calculation can produce different stored/displayed values
across environments.

**Fix:** migrate money/quantity columns to `Numeric(12, 4)` (tune precision/scale per field) and use
`Decimal` in the calculation engine. Guard with the existing formula-parity tests.

#### P0.2 — Frontend silently shows mock data when the API fails
`frontend/src/lib/api.ts` → `fetchJson(path, fallback, ...)` returns the `fallback` (mock data) on
**any** fetch error:

- `api.ts:151` and `api.ts:158` — `return fallback;` on failure.
- Fallbacks are real mock datasets: `mockRawMaterials`, `mockCosts`, `mockProducts`, `mockMixes`,
  `mockScenarios`, `mockMixCalculatorOptions`, `mockMixCalculatorSessions`, `mockClientAccess` —
  imported from `$lib/mock` (`api.ts:4-13`, used at `api.ts:296+`).

**Why it matters:** a backend hiccup makes the UI display **fabricated prices/costs** with no error
shown to the user. In a pricing application this is the most dangerous item in this audit — a user
could quote or decide off fake numbers.

**Fix:** remove the mock-on-error fallback path. Surface real API errors in the UI (error/empty
states). Keep `mock.ts` for tests only.

#### P0.3 — Schema management is `create_all` + ad-hoc `ALTER`, no versioning
`backend/app/db/migrations.py` runs on **every startup** via `bootstrap_schema()`
(`backend/app/main.py:105`, inside `ensure_database_ready()`):

- `ensure_metadata_tables()` → `metadata.create_all()` for any missing tables.
- `ensure_tenant_columns()` → adds `tenant_id` to a hardcoded `TENANT_TABLES` list.
- `ensure_legacy_columns()` → a hand-maintained `_LEGACY_COLUMN_PATCHES` tuple of
  `ALTER TABLE ... ADD COLUMN` statements.
- `sync_tenant_ids()` → ~250 lines of near-identical `UPDATE` backfills.
- `sync_product_visibility()` → data backfill.

**Why it matters:** this can only **create tables and add columns**. It can never change a column
type, add an index / constraint / FK, drop a column, or do a NOT-NULL backfill in a controlled way.
A fresh Postgres gets the *current* model via `create_all`, while an upgraded SQLite has columns
bolted on by `ALTER` in whatever order/type they accreted — the two **drift apart silently**, and
SQLite's loose typing hides mismatches until production. Across 11 versions the only escape hatch
has been appending more manual patches (unbounded, fragile).

The one-shot SQLite→Postgres move (`deploy/migrate-to-postgres.sh`) uses
`SET session_replication_role` and manual sequence resets — fine for a single cutover, but not a
repeatable/testable migration path.

**Fix:** adopt **Alembic** (see Phase 1).

---

### P1 — maintainability

#### P1.1 — Copy-pasted helpers, no shared util
No shared formatting/number module. Duplicated implementations:

- `formatDate` — **9** files: `lib/components/ClientAccessWorkspace.svelte`,
  `lib/components/mix-calculator/MixCalculatorResultsPanel.svelte`,
  `lib/components/MixCalculatorPrintDocument.svelte`, `routes/+page.svelte`,
  `routes/admin/+page.svelte`, `routes/client-access/+page.svelte`,
  `routes/mix-calculator/+page.svelte`, `routes/raw-materials/+page.svelte`,
  `routes/throughput/+page.svelte`.
- `formatNumber` — **5** files: `lib/components/mix-calculator/MixCalculatorEditor.svelte`,
  `lib/components/mix-calculator/MixCalculatorResultsPanel.svelte`,
  `lib/components/MixCalculatorPrintDocument.svelte`, `routes/mix-calculator/+page.svelte`,
  `routes/throughput/+page.svelte`.
- `toNum` — **2** files: `routes/throughput/+page.svelte`, `routes/throughput/add/+page.svelte`.

**Symptom already hit:** the `toNum` `value.trim is not a function` bug (Svelte coerces
`<input type="number">` bindings to `number`/`null`, not `string`). Fixed in both files
2026-06-03, but this class of bug will recur until there is a single source of truth.

**Fix:** add `frontend/src/lib/format.ts` (`formatDate`, `formatNumber`, `formatCurrency`, `toNum`)
and replace the duplicates.

#### P1.2 — Monolith route files
Largest route components (LOC):

| File | LOC |
| --- | --- |
| `routes/+page.svelte` (dashboard) | 2238 |
| `routes/product-costing/+page.svelte` | 1557 |
| `routes/throughput/+page.svelte` | 1232 |
| `routes/editor/+page.svelte` | 1163 |
| `routes/raw-materials/+page.svelte` | 1062 |
| `routes/client-access/+page.svelte` | 851 |
| `routes/reporting/+page.svelte` | 518 |

**Why it matters:** hard to test, hard to change safely, encourages more copy-paste.

**Fix:** decompose incrementally — extract components and `+page.ts` load logic. Dashboard and
product-costing first.

#### P1.3 — `migrations.py` conflates DDL + data backfill
Schema DDL (`ensure_*`) and data backfill (`sync_*`) live in one module, including ~250 lines of
near-identical `UPDATE` blocks in `sync_tenant_ids()`. Alembic will absorb most of this into
versioned schema steps + explicit data-migration steps.

---

### P2 — hygiene

- **P2.1** — `backend/tests/_repro_throughput_post.py` is a debug repro, not a real test. Remove.
- **P2.2** — ~20 `TODO`/`FIXME`/legacy markers across `backend/app` and `frontend/src`. Triage and burn down.
- **P2.3** — `backend/app/seed.py` is 1325 LOC. Split by module.
- **P2.4** — `CLAUDE.md` still says "PostgreSQL recommended / SQLite acceptable only for prototype",
  stale vs the live Postgres migration. Refresh.

---

## P0.4 — Three overlapping authentication / user-type systems

The app has accreted **three separate auth systems**, with **two cookies**, **three login
endpoints**, **three role namespaces**, and two parallel permission models. They overlap awkwardly
and one of them is already dead in the UI. This is the single largest piece of structural debt in
the codebase.

### The three systems

**1. Internal / "lean" system** — `users` / `roles` / `permissions` / `role_permissions`
- Code: `app/core/access.py`, `app/models/access.py`, `app/api/access.py` (`/api/access/*`),
  `app/seed_access.py`.
- Per-user password hash (`User.password_hash`, PBKDF2). Role → permission keys
  (`view_raw_materials`, `edit_products`, …). Fail-closed `require_permission(...)` dependencies.
- Token carries `sub=INTERNAL_USER_SUBJECT`; session `role="internal"`.
- Tenant is **hardcoded** to a constant: `INTERNAL_USER_TENANT_ID = "hunter-premium-produce"`
  (`core/access.py:37`).
- Uses `CLIENT_AUTH_COOKIE`.
- **This is the actual primary login.** The root page (`routes/+page.svelte:57`) calls
  `api.internalLogin()` → `/api/access/login`.

**2. Client-portal system** — `client_accounts` / `client_users` / `client_feature_access` /
   `client_user_module_permissions` / `client_access_audit_events`
- Code: `app/models/client_access.py`, `app/services/client_access_service.py`,
  `app/api/auth.py` (`/api/auth/client/*`), `app/api/client_access.py`.
- **Multi-tenant** (`tenant_id` on every table), per-account feature flags, per-user module
  access levels (`none`/`view`/`edit`/`manage`), `client_role` in {superadmin, admin, viewer, …}.
- Session `role="client"`. Uses `CLIENT_AUTH_COOKIE`.
- **Authentication is broken-by-design: a single shared password.** `client_login` checks
  `payload.password != settings.client_password` (`api/auth.py:79`) — *one* password for *all*
  client users; the per-user record only supplies identity, not a credential.
- **The login UI is dead.** `api.clientLogin` is referenced only by `api.ts` (definition) and
  `api.test.ts` — no component calls it. The endpoints, tables, tenant plumbing, and the
  `require_client_session` / `module_access_map` path are all still live and still wired into the
  shared route dependencies.

**3. Admin system** — environment-variable single credential
- Code: `app/api/auth.py` (`/api/auth/admin/*`), `require_admin_session` in `app/api/deps.py`.
- No DB row. Authenticates against `settings.admin_email` / `settings.admin_password`.
  Session `role="admin"` → **blanket access** (`session.ts:105` `hasModuleAccess` returns `true`
  for admin; `require_admin_session` gates the admin-only routes).
- Uses a **second cookie**, `ADMIN_AUTH_COOKIE`.
- Drives the separate `/admin` + `/admin/client-access` UI (`routes/admin/+page.svelte` calls
  `api.adminLogin()`), which exists to manage client users / feature flags / Power BI preview —
  i.e. the "management behind the scenes" layer we no longer want.

### How they tangle

- **Frontend** routes by URL: `/admin*` → `AdminShell`, everything else → `ClientShell`
  (`routes/+layout.svelte:15,42-49`). Two `localStorage` session stores
  (`data-entry-app-client-session`, `data-entry-app-admin-session`) in `session.ts`.
- **Shared route deps bend to accept two token shapes.** `require_client_session` and
  `require_client_module_access` (`api/deps.py:97-184`) special-case `role=="internal"` to skip the
  `ClientUser` DB lookup and read permissions from a role-derived map, while still supporting
  `role=="client"`. `core/access.py:_PERMISSION_TO_MODULE_LEVEL` exists purely to translate the
  internal permission keys into the legacy client module/level shape so the same routes accept both.
- **Two permission models run in parallel**: role→permission-keys (internal) vs
  per-user module→access-level rows + per-account feature flags (client). `permissions_to_module_map`
  bridges them.
- **`tenant_id` is smeared across ~25 tables and ~25 backend files** (heaviest:
  `db/migrations.py` 70 refs, `seed.py` 52, `api/product_costing.py` 36, plus every service/model),
  for multi-tenancy we no longer want.

### Target architecture (per product direction)

> One login for everyone. User type `lean` = full access. `client` = its own permissions.
> No multi-tenant. No separate behind-the-scenes management app, except `lean` may have a few
> extra settings (e.g. change logo).

- **One login endpoint + one login page** for all users.
- **One user store**: keep `users` / `roles` / `permissions` / `role_permissions`. Everyone is a
  `User` with a role. Add a **`lean`** role = all permissions (full access, including the extra
  settings like logo). Define a **`client`** role with its own permission set. Operations etc. stay
  as additional roles.
- **One cookie**, one session shape, one frontend session store.
- **Remove multi-tenancy**: drop `tenant_id` from models/queries/migrations/seed; collapse to a
  single implicit tenant.
- **Retire the env-var admin login** and the **dead client-portal login** + its tables/service,
  folding any still-needed capability (e.g. managing users) into permission-gated routes inside the
  single app. `lean`-only settings (logo, etc.) become permission-gated, not a separate shell.

### Decoupling / migration approach (proposed)

1. **Confirm the dead path is dead** (done: `clientLogin` has no UI caller) and snapshot any client
   data worth keeping (`client_users`, module permissions) so it can be re-expressed as `users` +
   `roles` if needed.
2. **Unify on the `users`/`roles`/`permissions` model.** Introduce `lean` and `client` roles in
   `seed_access.py` with the right permission sets. Migrate any real client users into `users`.
3. **Single login**: make `/api/access/login` the only login; one cookie; one session store; one
   login page. Remove `/api/auth/admin/*`, `/api/auth/client/*`, `ADMIN_AUTH_COOKIE`, the
   admin/client localStorage split, and the `/admin*` shell routing (fold any surviving admin
   screens into permission-gated routes in the main app).
4. **Collapse the dual permission model**: drop `_PERMISSION_TO_MODULE_LEVEL` bridging and the
   `role=="internal"` / `role=="client"` special-casing in `deps.py`; every route depends on
   `require_permission(...)` (or a thin module-level wrapper) only.
5. **Drop multi-tenancy**: Alembic migration removing `tenant_id` columns (or leaving them nullable
   and unused first, then dropping), plus removing `tenant_id` filters from services/queries and the
   `sync_tenant_ids` backfill. **Sequence this on top of Phase 1 (Alembic)** so the column drops are
   versioned and run identically on SQLite and Postgres.
6. **Delete the client-portal subsystem** once nothing references it: `models/client_access.py`,
   `client_access_service.py`, `api/auth.py`, `api/client_access.py`, related schemas, and the
   `ClientShell`/`AdminShell` split.

### Risk notes

- This touches **authentication** — stage it carefully behind tests; do not delete the old endpoints
  until the unified login is proven in dev against both SQLite and (a Postgres copy of) prod.
- The shared password (`P0.4`/system 2) and the env-admin credential should be considered a
  **security cleanup**, not just structure: per-user hashed passwords for everyone is the target.
- Dropping `tenant_id` is irreversible data-wise — do it as a dedicated, reviewed Alembic step with
  a backup, after the login unification has settled.

---

## Remediation plan (phased)

### Phase 0 — Safety net (no behavior change)
- Add a schema-parity smoke test: fresh-SQLite `create_all` vs `Base.metadata` so later phases
  cannot silently drift.
- Remove `backend/tests/_repro_throughput_post.py`.

### Phase 1 — Adopt Alembic  *(foundation; most moving parts)*
- Add `alembic` to `backend/pyproject.toml`; `alembic init`.
- Wire `env.py` to read `DATABASE_URL` (via `app.core.config.settings`) and `Base.metadata`.
- **Critical for this setup:** enable `render_as_batch=True` so `ALTER` works on **SQLite (Windows dev)**;
  Postgres handles `ALTER` natively.
- Autogenerate a **`0001_baseline`** migration from current models.
- `alembic stamp 0001_baseline` on existing dev **and** prod DBs so they are recognized without rebuilding.
- Fold `_LEGACY_COLUMN_PATCHES`, `sync_tenant_ids`, and `sync_product_visibility` into versioned
  migrations (schema steps + explicit data-migration steps).
- Replace the startup `bootstrap_schema(...)` call (`main.py:105`) with `alembic upgrade head`
  (or an explicit deploy step).
- Update `deploy/migrate-to-postgres.sh` **Phase 5** to run `alembic upgrade head` instead of
  calling `bootstrap_schema`.

### Phase 2 — Money correctness
- `Float → Numeric(12, 4)` (tune per field) across the money/quantity columns listed in P0.1.
- Use `Decimal` in `services/costing_engine.py` and `services/product_costing_service.py`.
- Dedicated Alembic migration; guard with `tests/test_costing_engine.py` formula-parity tests.

### Phase 3 — Frontend shared utils
- New `frontend/src/lib/format.ts`: `formatDate`, `formatNumber`, `formatCurrency`, `toNum`.
- Replace the 9 / 5 / 2 duplicate implementations. Eliminates the `toNum`-style bug class.

### Phase 4 — Remove mock-on-error fallback  *(quick, high-value correctness fix)*
- Remove the `fallback` return path in `api.ts` `fetchJson`.
- Surface real API errors / empty states in the UI.
- Keep `mock.ts` for tests only.

### Phase 5 — Unify authentication & user types  *(addresses P0.4; large, cross-cutting)*
Sits on top of Phase 1 (Alembic) because the column drops must be versioned. Order within the phase:
1. Snapshot/migrate any real client users into `users` + `roles`; add `lean` and `client` roles in
   `seed_access.py`.
2. Single login: make `/api/access/login` the only login; one cookie; one session store; one login
   page. Remove `/api/auth/admin/*`, `/api/auth/client/*`, `ADMIN_AUTH_COOKIE`, and the `/admin*`
   shell split.
3. Collapse the dual permission model — every route on `require_permission(...)`; delete the
   `internal`/`client` special-casing and `_PERMISSION_TO_MODULE_LEVEL` bridge in `deps.py`/`access.py`.
4. Drop multi-tenancy (`tenant_id`) via a dedicated Alembic migration + query cleanup; remove
   `sync_tenant_ids`.
5. Delete the dead client-portal subsystem (`models/client_access.py`, `client_access_service.py`,
   `api/auth.py`, `api/client_access.py`, `AdminShell`).
6. `lean`-only extras (logo change, etc.) become permission-gated settings in the single app.

### Phase 6 — Decompose monolith routes
- Incrementally extract components + `+page.ts` load logic. Start with dashboard (`+page.svelte`)
  and product-costing.

### Phase 7 — Hygiene
- Burn down `TODO`/legacy markers.
- Split `seed.py` by module.
- Refresh `CLAUDE.md` DB guidance.

---

## Suggested sequencing note
Phase 1 (Alembic) is the foundation the SQLite-dev / Postgres-prod split most depends on. However,
**Phase 4 (mock-on-error)** is the scariest correctness bug and a ~20-minute fix — a reasonable
quick win to do first, before Phase 1.

## Progress log
- 2026-06-03 — Audit completed; plan written. `toNum` bug fixed in
  `routes/throughput/+page.svelte` and `routes/throughput/add/+page.svelte` (precursor to Phase 3).
- 2026-06-03 — Auth/user-type investigation added (P0.4 + Phase 5). Found three overlapping auth
  systems; the client-portal login (`/api/auth/client/login`, shared password) is already dead in
  the UI (`clientLogin` has no component caller). Target: single login, `lean`/`client` roles, no
  multi-tenant, no separate admin shell.
- 2026-06-04 — Phase 4 quick win started: removed production `api.ts` mock-on-error fallback so
  failed reads throw normalized API errors instead of returning fabricated mock pricing/costing data.
  Removed `backend/tests/_repro_throughput_post.py` debug repro file.
- 2026-06-04 — Phase 0 safety net started: added a fresh SQLite schema smoke test that checks
  model metadata tables and columns are created as declared.
- 2026-06-04 — Phase 3 shared utils started: added `frontend/src/lib/format.ts`, covered it with
  unit tests, and replaced the duplicated `toNum` helper plus the mix-calculator/throughput number
  and date formatters touched in recent work.