data-entry-app/DATABASE-DESIGN.MD

# Database Design

## Purpose

This app uses a relational database to support five main concerns:

1. Raw material pricing and unit conversion.
2. Mix definitions and mix costing.
3. Product-level formulas and product costing.
4. Mix calculator session history.
5. Access control for both internal users and client users.

The backend is written with SQLAlchemy models in `backend/app/models`. The schema is created automatically at startup, and lightweight migration/patch logic lives in `backend/app/db/migrations.py`.

## Design Principles

- `tenant_id` is the tenancy boundary for most business tables.
- Reference/master data is stored separately from transactional/session data.
- Product costing is built from raw materials -> formulas -> products -> outputs.
- The mix calculator now prefers product-specific ingredient formulas over the shared mix master.
- The database is designed to run on both SQLite locally and Postgres in production.

## High-Level Domains

### 1. Raw Materials

These tables store ingredients and their price history.

- `raw_materials`
  - One row per ingredient/raw material.
  - Stores name, supplier, unit of measure, `kg_per_unit`, status, and notes.
  - Example: `Hulled Oats`, `White French Millet`, `Pano`.

- `raw_material_price_versions`
  - One-to-many from `raw_materials`.
  - Stores `market_value`, `waste_percentage`, `effective_date`, and status.
  - Lets the system keep historical prices instead of overwriting one current value.

Relationship:

- `raw_materials.id` -> `raw_material_price_versions.raw_material_id`

### 2. Mix Master

These tables store shared mix definitions.

- `mixes`
  - One row per named mix.
  - Stores client name, mix name, version, status, and notes.
  - This is the shared mix/master-recipe layer.

- `mix_ingredients`
  - One-to-many from `mixes`.
  - One row per raw material inside a mix.
  - Stores `quantity_kg` for that mix.

Relationships:

- `mixes.id` -> `mix_ingredients.mix_id`
- `raw_materials.id` -> `mix_ingredients.raw_material_id`

Important note:

- This table is still used by mix master pages and as a fallback.
- It is no longer the primary source for mix calculator formulas when product-specific formulas exist.

### 3. Products

These tables describe saleable products and their formula rows.

- `products`
  - One row per sellable product/SKU.
  - Stores client name, product name, optional `item_id`, packaging/unit info, margins, and linked mix.
  - `mix_id` links the product to the shared mix master entry.

- `product_ingredients`
  - One-to-many from `products`.
  - One row per raw material required for that product’s formula.
  - Stores `quantity_kg`, `sort_order`, and optional notes.
  - This is now the key table for the mix calculator.

Relationships:

- `products.mix_id` -> `mixes.id`
- `products.id` -> `product_ingredients.product_id`
- `raw_materials.id` -> `product_ingredients.raw_material_id`

Why both `mix_ingredients` and `product_ingredients` exist:

- `mix_ingredients` represents a shared recipe.
- `product_ingredients` represents the actual formula used for a specific product.
- Multiple products can point at the same mix name but still require product-specific formula rows.
- This solves the workbook case where product labels like `Budgie Mix 20kg` map to a formula/mix name like `Hunter - Budgie Mix`.

### 4. Costing Assumptions

These tables hold non-ingredient costs used in product costing.

- `process_cost_rules`
  - Holds grading, bagging, and cracking costs by `process_name`.

- `packaging_cost_rules`
  - Holds bag cost by `sale_type`, `unit_of_measure`, and `own_bag`.

- `freight_cost_rules`
  - Holds freight cost by `sale_type` and `unit_of_measure`.

These tables are read during product cost calculation after ingredient cost has been resolved.

### 5. Scenarios and Stored Outputs

- `scenarios`
  - Named pricing/costing scenarios.
  - Stores `overrides` as JSON.

- `costing_results`
  - One-to-many from `scenarios`.
  - Stores calculated output per product for a scenario.
  - Includes prices, warnings, and calculation details as JSON.

Relationships:

- `scenarios.id` -> `costing_results.scenario_id`
- `products.id` -> `costing_results.product_id`

### 6. Mix Calculator Sessions

These tables store saved calculator runs.

- `mix_calculator_sessions`
  - Header row for a calculator run.
  - Stores product, mix, batch size, total bags, total kg, prepared by, and timestamps.

- `mix_calculator_session_lines`
  - One-to-many from `mix_calculator_sessions`.
  - Snapshot of the scaled ingredient rows shown to the user at save time.
  - Stores `required_kg`, `mix_percentage`, unit, and display name.

Relationships:

- `mix_calculator_sessions.id` -> `mix_calculator_session_lines.session_id`
- `products.id` -> `mix_calculator_sessions.product_id`
- `mixes.id` -> `mix_calculator_sessions.mix_id`

Important note:

- Session lines are denormalized snapshots.
- They are intentionally stored separately so historical saved runs do not change if product formulas are updated later.

### 7. Client Access / Tenant Administration

These tables manage customer-facing users and feature/module access.

- `client_accounts`
  - One row per client/tenant account.

- `client_users`
  - One-to-many from `client_accounts`.
  - Customer-side users tied to a client account.

- `client_feature_access`
  - One-to-many from `client_accounts`.
  - Feature flags per client account.

- `client_user_module_permissions`
  - One-to-many from `client_users`.
  - Module-level access levels per client user.

- `client_access_audit_events`
  - One-to-many from `client_accounts`.
  - Audit log for client-access changes.

Relationships:

- `client_accounts.id` -> `client_users.client_account_id`
- `client_accounts.id` -> `client_feature_access.client_account_id`
- `client_accounts.id` -> `client_access_audit_events.client_account_id`
- `client_users.id` -> `client_user_module_permissions.client_user_id`

### 8. Internal Access Control

These tables are for internal staff login and permissions.

- `users`
  - Internal users.
  - Stores per-user `password_hash`, role link, and active flag.

- `roles`
  - Named roles like `Admin`, `Operations`, `Full Access`.

- `permissions`
  - Atomic permission keys like `view_mix_calculator`.

- `role_permissions`
  - Many-to-many join table between roles and permissions.

Relationships:

- `roles.id` -> `users.role_id`
- `roles.id` <-> `permissions.id` through `role_permissions`

## Core Costing Flow

### Raw Material Cost

The system calculates ingredient cost from:

- `market_value`
- `waste_percentage`
- `kg_per_unit`

This produces:

- loss cost
- adjusted cost per unit
- cost per kg

### Mix Cost

There are now two formula sources:

1. Preferred: `product_ingredients`
2. Fallback: `mix_ingredients`

For mix calculator and product costing:

- if a product has rows in `product_ingredients`, use them
- otherwise use the linked shared mix from `mix_ingredients`

### Product Cost

Product cost is built from:

1. ingredient formula cost
2. process costs
3. packaging cost
4. freight cost
5. optional distributor / wholesale margin

## Workbook Import Design

The seed/import logic is in `backend/app/seed.py`.

There are now two workbook roles:

- Legacy costing workbook:
  - `C- Raw Products Costs`
  - `M - All`
  - `Product Cost - Price`
- Product-formula workbook:
  - `input_data/1.xlsx`
  - sheet `mix_quantites_per_client_per_pr`

### Current Import Behaviour

- Raw materials are seeded from the legacy costing workbook.
- Shared mixes are seeded from the legacy costing workbook.
- Products are seeded from the legacy costing workbook.
- Product-specific formulas are seeded from `mix_quantites_per_client_per_pr`.

### Formula Matching Rule

Workbook formula rows are attached to products using:

1. `(client_name, product.name)` if it matches directly.
2. `(client_name, product.mix.name)` if the workbook row uses the mix/formula name instead of the sellable product label.

This is important for cases like:

- workbook formula: `HunterBird / Hunter - Budgie Mix`
- product row: `HunterBird / Budgie Mix 20kg`

Both product SKUs can inherit the same formula through the linked mix name.

## Tenancy

Most business tables include `tenant_id`.

This includes:

- raw materials
- price versions
- mixes
- mix ingredients
- product ingredients
- products
- scenarios
- costing results
- mix calculator sessions and lines
- client-access tables
- assumption tables

Startup migration logic backfills `tenant_id` where possible by deriving it from related parent tables.

## Visibility Rules

The `products.visible` flag is used to hide client/product rows from normal UI paths.

Startup migration logic also auto-hides products for a configured list of client names in `backend/app/db/migrations.py`.

This means:

- rows can exist in the database
- but not be offered in normal mix calculator/product selection flows

## Transaction vs Reference Data

Reference/master data:

- `raw_materials`
- `raw_material_price_versions`
- `mixes`
- `mix_ingredients`
- `products`
- `product_ingredients`
- `process_cost_rules`
- `packaging_cost_rules`
- `freight_cost_rules`
- access-control tables

Transactional/snapshot data:

- `mix_calculator_sessions`
- `mix_calculator_session_lines`
- `scenarios`
- `costing_results`
- `client_access_audit_events`

## Important Constraints

- `mix_ingredients` is unique on `(mix_id, raw_material_id)`.
- `product_ingredients` is unique on `(product_id, raw_material_id)`.
- `client_users` is unique on `(client_account_id, email)`.
- `client_feature_access` is unique on `(client_account_id, feature_key)`.
- `client_user_module_permissions` is unique on `(client_user_id, module_key)`.
- `mix_calculator_sessions` is unique on `(tenant_id, session_number)`.

These constraints prevent duplicate ingredient or access rows within the same parent scope.

## Known Tradeoffs

- `RawMaterial.name` is globally unique, not tenant-scoped. That is simple for now, but stricter than a multi-tenant design usually wants.
- `Product.mix_id` is still required even though product-specific formulas now exist. That is useful for compatibility and navigation, but it means a product currently has both a shared mix link and potentially its own formula rows.
- Some calculation outputs are denormalized into session/result tables for stability and history.
- Migration logic is startup-driven and pragmatic rather than using a full migration framework like Alembic.

## Recommended Mental Model

Use this as the working model of the schema:

- `raw_materials` = ingredients
- `raw_material_price_versions` = ingredient pricing history
- `mixes` = shared recipe labels
- `mix_ingredients` = shared recipe lines
- `products` = saleable SKUs
- `product_ingredients` = actual formula for a SKU
- `mix_calculator_sessions` + `lines` = saved production calculations
- `scenarios` + `costing_results` = stored what-if pricing outputs
- `client_*` tables = client account access
- `users / roles / permissions` = internal staff access

## Files To Read Alongside This Document

- [backend/app/models/raw_material.py](/abs/path/C:/Users/mattc/data-entry-app/data-entry-app/backend/app/models/raw_material.py:1)
- [backend/app/models/mix.py](/abs/path/C:/Users/mattc/data-entry-app/data-entry-app/backend/app/models/mix.py:1)
- [backend/app/models/product.py](/abs/path/C:/Users/mattc/data-entry-app/data-entry-app/backend/app/models/product.py:1)
- [backend/app/models/mix_calculator.py](/abs/path/C:/Users/mattc/data-entry-app/data-entry-app/backend/app/models/mix_calculator.py:1)
- [backend/app/models/scenario.py](/abs/path/C:/Users/mattc/data-entry-app/data-entry-app/backend/app/models/scenario.py:1)
- [backend/app/models/assumption.py](/abs/path/C:/Users/mattc/data-entry-app/data-entry-app/backend/app/models/assumption.py:1)
- [backend/app/models/client_access.py](/abs/path/C:/Users/mattc/data-entry-app/data-entry-app/backend/app/models/client_access.py:1)
- [backend/app/models/access.py](/abs/path/C:/Users/mattc/data-entry-app/data-entry-app/backend/app/models/access.py:1)
- [backend/app/db/migrations.py](/abs/path/C:/Users/mattc/data-entry-app/data-entry-app/backend/app/db/migrations.py:1)
- [backend/app/seed.py](/abs/path/C:/Users/mattc/data-entry-app/data-entry-app/backend/app/seed.py:1)