gw/abtesting.md

# A/B Testing System

This repository now includes an SSR-first, SEO-safe A/B testing system for the SvelteKit frontend and Python API backend.

The backend is now the managed source of truth for live experiment definitions.

## Production prerequisite

The experiment system requires real SvelteKit SSR in production.

- Run `npm install` in `frontend/` after pulling these changes.
- The frontend is configured to use `@sveltejs/adapter-node` when installed.
- Do not deploy the experiment system as a prerendered static build.

## Goals

- Server-side assignment only.
- Durable assignment by cookie.
- SSR HTML already contains the chosen variant.
- No bot targeting or user-agent-based variant serving.
- Stable canonical signals across variants.
- Frontend sends analytics only; it does not decide variants after render.

## Frontend flow

### Request lifecycle

1. `frontend/src/hooks.server.js` runs for every request.
2. It calls `buildRequestExperimentContext()` in `frontend/src/lib/experiments/server.js`.
3. The hook fetches the current experiment definitions from the backend using `frontend/src/lib/experiments/provider.server.js`.
4. The hook reads or creates the anonymous session cookie:
   - `gw_session_id`
5. For each eligible backend-managed experiment:
   - it reads the persisted experiment cookie if present
   - otherwise it hashes `gw_session_id + experiment_key`
   - it maps that hash into the configured traffic weights
   - it persists the assigned variant in a cookie such as `exp_homepage_hero`
6. The hook stores the resolved assignments in `event.locals`.
7. `+layout.server.js` exposes assignments, active experiments, analytics settings, and SEO metadata to the app.
8. Route server loads expose page-specific assignments:
   - `frontend/src/routes/+page.server.js`
   - `frontend/src/routes/our-pricing/+page.server.js`
9. Page components render the chosen variant directly during SSR:
   - `frontend/src/routes/Home-2.svelte`
   - `frontend/src/routes/Pricing.svelte`

### Cookies

- `gw_session_id`
  - purpose: stable anonymous session id used for deterministic assignment and experiment analytics
  - scope: `path=/`
  - flags: `HttpOnly`, `SameSite=Lax`, `Secure` on HTTPS
  - expiry: 180 days
- `exp_homepage_hero`
  - purpose: persists the homepage hero assignment
  - expiry: 90 days
- `exp_pricing_cta`
  - purpose: persists the pricing CTA assignment
  - expiry: 90 days
- `gw_analytics_consent`
  - purpose: enables non-essential experiment analytics when set to `granted`

### Backend-managed definitions

Live definitions are read from:

- `GET /api/experiments`

The frontend uses a short-lived server cache and falls back to the local seed registry only if the backend cannot be reached.

Relevant server-side env var:

- `EXPERIMENTS_API_INTERNAL_URL`
  - optional internal URL the SvelteKit server should use to fetch experiment definitions
  - useful when the frontend and backend run on different internal hosts

### Deterministic assignment

Assignment is deterministic and stable for a session:

- hash input: `gw_session_id + ":" + experiment_key`
- hash function: FNV-1a style integer hash in `hashToBucket()`
- bucket space: `0..9999`
- allocation: weighted selection via `pickVariant()`

If the assignment cookie already exists and contains a valid variant, the cookie wins.

### Consent behavior

Consent is split from assignment.

- Variant assignment can stay enabled even without analytics consent.
- Experiment analytics can be disabled until consent is granted.

Relevant env vars:

- `PUBLIC_EXPERIMENTS_ENABLED`
  - `false` disables all experiments globally
  - any other value leaves the system enabled
- `PUBLIC_EXPERIMENT_ASSIGNMENT_COOKIE_MODE`
  - `necessary` means assignment cookies are allowed without analytics consent
  - `consent` means no experiment assignment cookie is set until consent is granted
- `PUBLIC_EXPERIMENT_ANALYTICS_REQUIRES_CONSENT`
  - `true` means no experiment impression/click/form/conversion events are sent until consent is granted

Recommended production values:

- `PUBLIC_EXPERIMENTS_ENABLED=true`
- `PUBLIC_EXPERIMENT_ASSIGNMENT_COOKIE_MODE=necessary`
- `PUBLIC_EXPERIMENT_ANALYTICS_REQUIRES_CONSENT=true`

## SEO protection

### What makes this safe

- Assignment happens on the server before HTML is rendered.
- The same SSR HTML is served to crawlers and normal users for a given assigned variant.
- No user-agent detection is used for variant serving.
- No Googlebot special casing exists in the SvelteKit layer.
- Variants do not use separate URLs.
- Canonical handling is done in `frontend/src/routes/+layout.svelte` and remains stable across variants.
- The current experiments change presentation, CTA emphasis, and component framing, not the underlying subject of the page.

### Canonical handling

The canonical URL is generated in `frontend/src/lib/experiments/server.js`.

- canonical source: request path only
- query params are excluded from canonical output
- variant does not affect canonical output

### QA/debug query params

Supported debug query params:

- `exp_key`
- `exp_variant`
- `exp_until`
- `exp_sig`

If any of these are present:

- canonical still points to the clean path
- the layout outputs `meta name="robots" content="noindex, nofollow"`

This prevents QA links from becoming indexable.

### Signed QA override

Production QA override format:

- payload string: `${exp_key}:${exp_variant}:${exp_until}`
- signature: `hex(HMAC_SHA256(EXPERIMENT_DEBUG_SECRET, payload))`

Required params:

- `exp_key`
- `exp_variant`
- `exp_until`
- `exp_sig`

Rules:

- the signature must match
- `exp_until` must be a future Unix timestamp in seconds
- the variant must exist in the registry

Example shape:

```text
/?exp_key=homepage_hero_test&exp_variant=tiny_gang_social_proof&exp_until=1770000000&exp_sig=<hex_hmac>
```

Local dev-only cookie override is also supported:

- cookie name: `exp_debug_<experiment_key>`
- example: `exp_debug_homepage_hero_test=tiny_gang_social_proof`
- ignored outside localhost/127.0.0.1

## Frontend analytics

Experiment analytics live in `frontend/src/lib/experiments/client.js`.

It sends:

- `experiment_key`
- `variant_key`
- `session_id`
- `path`
- `event_name`
- `timestamp`
- `conversion_value`
- `metadata`

### Tracked events

- `impression`
  - auto-fired once per experiment variant per path per browser session
- `cta_click`
  - fired from experiment-controlled CTAs on the homepage and pricing page
- `form_start`
  - fired once when the Meet & Greet form is first engaged
- `form_submit`
  - fired on submit attempt
- `conversion`
  - fired after a successful Meet & Greet submission

### Delivery behavior

- `navigator.sendBeacon()` first
- `fetch(..., { keepalive: true })` fallback
- failures are swallowed so UX does not block

## Backend API

The Python experiment endpoints are implemented in `backend/app/routers/experiments.py`.

### Public endpoints

- `GET /api/experiments`
  - returns current experiment definitions and variants
- `POST /api/experiments/impression`
- `POST /api/experiments/event`
- `POST /api/experiments/conversion`

### Admin endpoint

- `GET /api/v1/experiments/results`
  - optional query param: `experiment_key`
  - auth required

### Validation

Payload validation is defined in `backend/app/schemas/experiments.py`.

Checks include:

- experiment/variant key format
- session id format
- path must start with `/`
- allowed event names per endpoint
- scalar-only metadata values
- known experiment + variant validation against the backend registry

### Bot filtering

Analytics quality filtering only applies to the analytics endpoints.

- bot-like user agents are dropped from experiment analytics ingestion
- this is not used for page serving
- dropped bot requests return `{"ok": true, "accepted": false}`

### Rate limiting

Each public experiment ingestion endpoint is limited to:

- `30/minute`

## Database model

Migration:

- `backend/alembic/versions/bd9f6a8b7c1d_add_experiments.py`

Tables:

- `experiments`
- `experiment_variants`
- `experiment_events`

Important indexed fields on `experiment_events`:

- `experiment_key`
- `variant_key`
- `session_id`
- `user_id`
- `path`
- `event_type`
- `created_at`
- composite `(experiment_key, variant_key, created_at)`
- composite `(session_id, created_at)`

## Registry and sync

Seed registry:

- `backend/app/experiments/registry.py`

Frontend fallback registry:

- `frontend/src/lib/experiments/registry.js`

Startup seed sync:

- `backend/app/main.py` calls `sync_experiment_registry()` during app lifespan startup

Important:

- startup sync creates missing experiments and variants from code defaults
- it does not overwrite existing admin-managed values
- admin edits therefore persist across restarts

## Backend management

Admin APIs:

- `GET /api/admin/experiments`
- `GET /api/admin/experiments/{experiment_key}`
- `PUT /api/admin/experiments/{experiment_key}`

Admin UI:

- `/admin/experiments`

Editable fields:

- `cookie_name`
- `name`
- `description`
- `enabled`
- `eligible_routes`
- variant labels
- variant allocations
- control variant selection

## Current experiments

### `homepage_hero_test`

- route: `/`
- variants:
  - `control`
  - `tiny_gang_social_proof`
- changes:
  - hero headline
  - supporting copy
  - hero CTA order/text
  - trust-pill wording

### `pricing_cta_test`

- route: `/our-pricing`
- variants:
  - `control`
  - `meet_greet_emphasis`
- changes:
  - pricing card CTA label
  - pricing CTA panel headline
  - pricing CTA panel body
  - pricing CTA panel button label

## How to add a new experiment

1. Add a seed definition to `backend/app/experiments/registry.py`.
2. Optionally add a fallback definition to `frontend/src/lib/experiments/registry.js`.
3. Start the backend so startup sync creates the DB rows.
4. Adjust the live definition in `/admin/experiments` if needed.
5. Expose the assignment in a route `+page.server.js` or `+layout.server.js`.
6. Render the variant in SSR using the server-provided assignment.
7. Only change presentation/CTA/component framing unless you are intentionally testing content that still preserves canonical meaning.
8. Add `experimentClick` tracking to experiment-owned CTAs.
9. If a form conversion should be attributed, make sure the page participates in `trackActiveExperiments()`.
10. Add or update tests.

## How to analyse results

1. Query `GET /api/v1/experiments/results`.
2. Compare variants on:
   - impressions
   - CTA clicks
   - form starts
   - form submits
   - conversions
   - unique sessions
   - conversion rate
   - conversion value total
3. Sanity-check sample size before making rollout decisions.
4. Review qualitative differences separately from SEO, because SEO is intentionally held constant by URL/canonical behavior.

## How to disable experiments safely

### Disable everything

Set:

```text
PUBLIC_EXPERIMENTS_ENABLED=false
```

Effect:

- new requests render control
- no active experiments are tracked
- canonical behavior is unchanged

### Disable one experiment

Set the experiment to disabled in `/admin/experiments`.

Effect:

- the page falls back to control
- existing cookies are ignored because the backend-managed definition marks the experiment inactive

### Stop analytics but keep assignment

Set:

```text
PUBLIC_EXPERIMENT_ANALYTICS_REQUIRES_CONSENT=true
```

and do not grant `gw_analytics_consent`.

Effect:

- SSR variant assignment can continue
- no experiment analytics events are sent

## Verification status

Verified locally:

- backend experiment endpoints and related analytics tests:
  - `python -m pytest backend/tests/test_experiments.py backend/tests/test_analytics.py backend/tests/test_analytics_ingest.py`

Not verified locally:

- frontend Vitest suite

Reason:

- `npm test` hit a local sandbox `spawn EPERM` restriction while loading Vitest config
- escalation to rerun outside the sandbox was not approved in this session