v1

2026-04-18 07:23:55 +12:00
parent f210020772
commit 6d44e05de4
396 changed files with 75296 additions and 0 deletions
@@ -0,0 +1,434 @@
+# A/B Testing System
+
+This repository now includes an SSR-first, SEO-safe A/B testing system for the SvelteKit frontend and Python API backend.
+
+The backend is now the managed source of truth for live experiment definitions.
+
+## Production prerequisite
+
+The experiment system requires real SvelteKit SSR in production.
+
+- Run `npm install` in `frontend/` after pulling these changes.
+- The frontend is configured to use `@sveltejs/adapter-node` when installed.
+- Do not deploy the experiment system as a prerendered static build.
+
+## Goals
+
+- Server-side assignment only.
+- Durable assignment by cookie.
+- SSR HTML already contains the chosen variant.
+- No bot targeting or user-agent-based variant serving.
+- Stable canonical signals across variants.
+- Frontend sends analytics only; it does not decide variants after render.
+
+## Frontend flow
+
+### Request lifecycle
+
+1. `frontend/src/hooks.server.js` runs for every request.
+2. It calls `buildRequestExperimentContext()` in `frontend/src/lib/experiments/server.js`.
+3. The hook fetches the current experiment definitions from the backend using `frontend/src/lib/experiments/provider.server.js`.
+4. The hook reads or creates the anonymous session cookie:
+   - `gw_session_id`
+5. For each eligible backend-managed experiment:
+   - it reads the persisted experiment cookie if present
+   - otherwise it hashes `gw_session_id + experiment_key`
+   - it maps that hash into the configured traffic weights
+   - it persists the assigned variant in a cookie such as `exp_homepage_hero`
+6. The hook stores the resolved assignments in `event.locals`.
+7. `+layout.server.js` exposes assignments, active experiments, analytics settings, and SEO metadata to the app.
+8. Route server loads expose page-specific assignments:
+   - `frontend/src/routes/+page.server.js`
+   - `frontend/src/routes/our-pricing/+page.server.js`
+9. Page components render the chosen variant directly during SSR:
+   - `frontend/src/routes/Home-2.svelte`
+   - `frontend/src/routes/Pricing.svelte`
+
+### Cookies
+
+- `gw_session_id`
+  - purpose: stable anonymous session id used for deterministic assignment and experiment analytics
+  - scope: `path=/`
+  - flags: `HttpOnly`, `SameSite=Lax`, `Secure` on HTTPS
+  - expiry: 180 days
+- `exp_homepage_hero`
+  - purpose: persists the homepage hero assignment
+  - expiry: 90 days
+- `exp_pricing_cta`
+  - purpose: persists the pricing CTA assignment
+  - expiry: 90 days
+- `gw_analytics_consent`
+  - purpose: enables non-essential experiment analytics when set to `granted`
+
+### Backend-managed definitions
+
+Live definitions are read from:
+
+- `GET /api/experiments`
+
+The frontend uses a short-lived server cache and falls back to the local seed registry only if the backend cannot be reached.
+
+Relevant server-side env var:
+
+- `EXPERIMENTS_API_INTERNAL_URL`
+  - optional internal URL the SvelteKit server should use to fetch experiment definitions
+  - useful when the frontend and backend run on different internal hosts
+
+### Deterministic assignment
+
+Assignment is deterministic and stable for a session:
+
+- hash input: `gw_session_id + ":" + experiment_key`
+- hash function: FNV-1a style integer hash in `hashToBucket()`
+- bucket space: `0..9999`
+- allocation: weighted selection via `pickVariant()`
+
+If the assignment cookie already exists and contains a valid variant, the cookie wins.
+
+### Consent behavior
+
+Consent is split from assignment.
+
+- Variant assignment can stay enabled even without analytics consent.
+- Experiment analytics can be disabled until consent is granted.
+
+Relevant env vars:
+
+- `PUBLIC_EXPERIMENTS_ENABLED`
+  - `false` disables all experiments globally
+  - any other value leaves the system enabled
+- `PUBLIC_EXPERIMENT_ASSIGNMENT_COOKIE_MODE`
+  - `necessary` means assignment cookies are allowed without analytics consent
+  - `consent` means no experiment assignment cookie is set until consent is granted
+- `PUBLIC_EXPERIMENT_ANALYTICS_REQUIRES_CONSENT`
+  - `true` means no experiment impression/click/form/conversion events are sent until consent is granted
+
+Recommended production values:
+
+- `PUBLIC_EXPERIMENTS_ENABLED=true`
+- `PUBLIC_EXPERIMENT_ASSIGNMENT_COOKIE_MODE=necessary`
+- `PUBLIC_EXPERIMENT_ANALYTICS_REQUIRES_CONSENT=true`
+
+## SEO protection
+
+### What makes this safe
+
+- Assignment happens on the server before HTML is rendered.
+- The same SSR HTML is served to crawlers and normal users for a given assigned variant.
+- No user-agent detection is used for variant serving.
+- No Googlebot special casing exists in the SvelteKit layer.
+- Variants do not use separate URLs.
+- Canonical handling is done in `frontend/src/routes/+layout.svelte` and remains stable across variants.
+- The current experiments change presentation, CTA emphasis, and component framing, not the underlying subject of the page.
+
+### Canonical handling
+
+The canonical URL is generated in `frontend/src/lib/experiments/server.js`.
+
+- canonical source: request path only
+- query params are excluded from canonical output
+- variant does not affect canonical output
+
+### QA/debug query params
+
+Supported debug query params:
+
+- `exp_key`
+- `exp_variant`
+- `exp_until`
+- `exp_sig`
+
+If any of these are present:
+
+- canonical still points to the clean path
+- the layout outputs `meta name="robots" content="noindex, nofollow"`
+
+This prevents QA links from becoming indexable.
+
+### Signed QA override
+
+Production QA override format:
+
+- payload string: `${exp_key}:${exp_variant}:${exp_until}`
+- signature: `hex(HMAC_SHA256(EXPERIMENT_DEBUG_SECRET, payload))`
+
+Required params:
+
+- `exp_key`
+- `exp_variant`
+- `exp_until`
+- `exp_sig`
+
+Rules:
+
+- the signature must match
+- `exp_until` must be a future Unix timestamp in seconds
+- the variant must exist in the registry
+
+Example shape:
+
+```text
+/?exp_key=homepage_hero_test&exp_variant=tiny_gang_social_proof&exp_until=1770000000&exp_sig=<hex_hmac>
+```
+
+Local dev-only cookie override is also supported:
+
+- cookie name: `exp_debug_<experiment_key>`
+- example: `exp_debug_homepage_hero_test=tiny_gang_social_proof`
+- ignored outside localhost/127.0.0.1
+
+## Frontend analytics
+
+Experiment analytics live in `frontend/src/lib/experiments/client.js`.
+
+It sends:
+
+- `experiment_key`
+- `variant_key`
+- `session_id`
+- `path`
+- `event_name`
+- `timestamp`
+- `conversion_value`
+- `metadata`
+
+### Tracked events
+
+- `impression`
+  - auto-fired once per experiment variant per path per browser session
+- `cta_click`
+  - fired from experiment-controlled CTAs on the homepage and pricing page
+- `form_start`
+  - fired once when the Meet & Greet form is first engaged
+- `form_submit`
+  - fired on submit attempt
+- `conversion`
+  - fired after a successful Meet & Greet submission
+
+### Delivery behavior
+
+- `navigator.sendBeacon()` first
+- `fetch(..., { keepalive: true })` fallback
+- failures are swallowed so UX does not block
+
+## Backend API
+
+The Python experiment endpoints are implemented in `backend/app/routers/experiments.py`.
+
+### Public endpoints
+
+- `GET /api/experiments`
+  - returns current experiment definitions and variants
+- `POST /api/experiments/impression`
+- `POST /api/experiments/event`
+- `POST /api/experiments/conversion`
+
+### Admin endpoint
+
+- `GET /api/v1/experiments/results`
+  - optional query param: `experiment_key`
+  - auth required
+
+### Validation
+
+Payload validation is defined in `backend/app/schemas/experiments.py`.
+
+Checks include:
+
+- experiment/variant key format
+- session id format
+- path must start with `/`
+- allowed event names per endpoint
+- scalar-only metadata values
+- known experiment + variant validation against the backend registry
+
+### Bot filtering
+
+Analytics quality filtering only applies to the analytics endpoints.
+
+- bot-like user agents are dropped from experiment analytics ingestion
+- this is not used for page serving
+- dropped bot requests return `{"ok": true, "accepted": false}`
+
+### Rate limiting
+
+Each public experiment ingestion endpoint is limited to:
+
+- `30/minute`
+
+## Database model
+
+Migration:
+
+- `backend/alembic/versions/bd9f6a8b7c1d_add_experiments.py`
+
+Tables:
+
+- `experiments`
+- `experiment_variants`
+- `experiment_events`
+
+Important indexed fields on `experiment_events`:
+
+- `experiment_key`
+- `variant_key`
+- `session_id`
+- `user_id`
+- `path`
+- `event_type`
+- `created_at`
+- composite `(experiment_key, variant_key, created_at)`
+- composite `(session_id, created_at)`
+
+## Registry and sync
+
+Seed registry:
+
+- `backend/app/experiments/registry.py`
+
+Frontend fallback registry:
+
+- `frontend/src/lib/experiments/registry.js`
+
+Startup seed sync:
+
+- `backend/app/main.py` calls `sync_experiment_registry()` during app lifespan startup
+
+Important:
+
+- startup sync creates missing experiments and variants from code defaults
+- it does not overwrite existing admin-managed values
+- admin edits therefore persist across restarts
+
+## Backend management
+
+Admin APIs:
+
+- `GET /api/admin/experiments`
+- `GET /api/admin/experiments/{experiment_key}`
+- `PUT /api/admin/experiments/{experiment_key}`
+
+Admin UI:
+
+- `/admin/experiments`
+
+Editable fields:
+
+- `cookie_name`
+- `name`
+- `description`
+- `enabled`
+- `eligible_routes`
+- variant labels
+- variant allocations
+- control variant selection
+
+## Current experiments
+
+### `homepage_hero_test`
+
+- route: `/`
+- variants:
+  - `control`
+  - `tiny_gang_social_proof`
+- changes:
+  - hero headline
+  - supporting copy
+  - hero CTA order/text
+  - trust-pill wording
+
+### `pricing_cta_test`
+
+- route: `/our-pricing`
+- variants:
+  - `control`
+  - `meet_greet_emphasis`
+- changes:
+  - pricing card CTA label
+  - pricing CTA panel headline
+  - pricing CTA panel body
+  - pricing CTA panel button label
+
+## How to add a new experiment
+
+1. Add a seed definition to `backend/app/experiments/registry.py`.
+2. Optionally add a fallback definition to `frontend/src/lib/experiments/registry.js`.
+3. Start the backend so startup sync creates the DB rows.
+4. Adjust the live definition in `/admin/experiments` if needed.
+5. Expose the assignment in a route `+page.server.js` or `+layout.server.js`.
+6. Render the variant in SSR using the server-provided assignment.
+7. Only change presentation/CTA/component framing unless you are intentionally testing content that still preserves canonical meaning.
+8. Add `experimentClick` tracking to experiment-owned CTAs.
+9. If a form conversion should be attributed, make sure the page participates in `trackActiveExperiments()`.
+10. Add or update tests.
+
+## How to analyse results
+
+1. Query `GET /api/v1/experiments/results`.
+2. Compare variants on:
+   - impressions
+   - CTA clicks
+   - form starts
+   - form submits
+   - conversions
+   - unique sessions
+   - conversion rate
+   - conversion value total
+3. Sanity-check sample size before making rollout decisions.
+4. Review qualitative differences separately from SEO, because SEO is intentionally held constant by URL/canonical behavior.
+
+## How to disable experiments safely
+
+### Disable everything
+
+Set:
+
+```text
+PUBLIC_EXPERIMENTS_ENABLED=false
+```
+
+Effect:
+
+- new requests render control
+- no active experiments are tracked
+- canonical behavior is unchanged
+
+### Disable one experiment
+
+Set the experiment to disabled in `/admin/experiments`.
+
+Effect:
+
+- the page falls back to control
+- existing cookies are ignored because the backend-managed definition marks the experiment inactive
+
+### Stop analytics but keep assignment
+
+Set:
+
+```text
+PUBLIC_EXPERIMENT_ANALYTICS_REQUIRES_CONSENT=true
+```
+
+and do not grant `gw_analytics_consent`.
+
+Effect:
+
+- SSR variant assignment can continue
+- no experiment analytics events are sent
+
+## Verification status
+
+Verified locally:
+
+- backend experiment endpoints and related analytics tests:
+  - `python -m pytest backend/tests/test_experiments.py backend/tests/test_analytics.py backend/tests/test_analytics_ingest.py`
+
+Not verified locally:
+
+- frontend Vitest suite
+
+Reason:
+
+- `npm test` hit a local sandbox `spawn EPERM` restriction while loading Vitest config
+- escalation to rerun outside the sandbox was not approved in this session