gw/abtesting.md

A/B Testing System

This repository now includes an SSR-first, SEO-safe A/B testing system for the SvelteKit frontend and Python API backend.

The backend is now the managed source of truth for live experiment definitions.

Production prerequisite

The experiment system requires real SvelteKit SSR in production.

• Run npm install in frontend/ after pulling these changes.
• The frontend is configured to use @sveltejs/adapter-node when installed.
• Do not deploy the experiment system as a prerendered static build.

Goals

• Server-side assignment only.
• Durable assignment by cookie.
• SSR HTML already contains the chosen variant.
• No bot targeting or user-agent-based variant serving.
• Stable canonical signals across variants.
• Frontend sends analytics only; it does not decide variants after render.

Frontend flow

Request lifecycle

1. frontend/src/hooks.server.js runs for every request.
2. It calls buildRequestExperimentContext() in frontend/src/lib/experiments/server.js.
3. The hook fetches the current experiment definitions from the backend using frontend/src/lib/experiments/provider.server.js.
4. The hook reads or creates the anonymous session cookie:
   • gw_session_id
5. For each eligible backend-managed experiment:
   • it reads the persisted experiment cookie if present
   • otherwise it hashes gw_session_id + experiment_key
   • it maps that hash into the configured traffic weights
   • it persists the assigned variant in a cookie such as exp_homepage_hero
6. The hook stores the resolved assignments in event.locals.
7. +layout.server.js exposes assignments, active experiments, analytics settings, and SEO metadata to the app.
8. Route server loads expose page-specific assignments:
   • frontend/src/routes/+page.server.js
   • frontend/src/routes/our-pricing/+page.server.js
9. Page components render the chosen variant directly during SSR:
   • frontend/src/routes/Home-2.svelte
   • frontend/src/routes/Pricing.svelte

Cookies

• gw_session_id
  • purpose: stable anonymous session id used for deterministic assignment and experiment analytics
  • scope: path=/
  • flags: HttpOnly, SameSite=Lax, Secure on HTTPS
  • expiry: 180 days
• exp_homepage_hero
  • purpose: persists the homepage hero assignment
  • expiry: 90 days
• exp_pricing_cta
  • purpose: persists the pricing CTA assignment
  • expiry: 90 days
• gw_analytics_consent
  • purpose: enables non-essential experiment analytics when set to granted

Backend-managed definitions

Live definitions are read from:

• GET /api/experiments

The frontend uses a short-lived server cache and falls back to the local seed registry only if the backend cannot be reached.

Relevant server-side env var:

• EXPERIMENTS_API_INTERNAL_URL
  • optional internal URL the SvelteKit server should use to fetch experiment definitions
  • useful when the frontend and backend run on different internal hosts

Deterministic assignment

Assignment is deterministic and stable for a session:

• hash input: gw_session_id + ":" + experiment_key
• hash function: FNV-1a style integer hash in hashToBucket()
• bucket space: 0..9999
• allocation: weighted selection via pickVariant()

If the assignment cookie already exists and contains a valid variant, the cookie wins.

Consent behavior

Consent is split from assignment.

• Variant assignment can stay enabled even without analytics consent.
• Experiment analytics can be disabled until consent is granted.

Relevant env vars:

• PUBLIC_EXPERIMENTS_ENABLED
  • false disables all experiments globally
  • any other value leaves the system enabled
• PUBLIC_EXPERIMENT_ASSIGNMENT_COOKIE_MODE
  • necessary means assignment cookies are allowed without analytics consent
  • consent means no experiment assignment cookie is set until consent is granted
• PUBLIC_EXPERIMENT_ANALYTICS_REQUIRES_CONSENT
  • true means no experiment impression/click/form/conversion events are sent until consent is granted

Recommended production values:

• PUBLIC_EXPERIMENTS_ENABLED=true
• PUBLIC_EXPERIMENT_ASSIGNMENT_COOKIE_MODE=necessary
• PUBLIC_EXPERIMENT_ANALYTICS_REQUIRES_CONSENT=true

SEO protection

What makes this safe

• Assignment happens on the server before HTML is rendered.
• The same SSR HTML is served to crawlers and normal users for a given assigned variant.
• No user-agent detection is used for variant serving.
• No Googlebot special casing exists in the SvelteKit layer.
• Variants do not use separate URLs.
• Canonical handling is done in frontend/src/routes/+layout.svelte and remains stable across variants.
• The current experiments change presentation, CTA emphasis, and component framing, not the underlying subject of the page.

Canonical handling

The canonical URL is generated in frontend/src/lib/experiments/server.js.

• canonical source: request path only
• query params are excluded from canonical output
• variant does not affect canonical output

QA/debug query params

Supported debug query params:

• exp_key
• exp_variant
• exp_until
• exp_sig

If any of these are present:

• canonical still points to the clean path
• the layout outputs meta name="robots" content="noindex, nofollow"

This prevents QA links from becoming indexable.

Signed QA override

Production QA override format:

• payload string: ${exp_key}:${exp_variant}:${exp_until}
• signature: hex(HMAC_SHA256(EXPERIMENT_DEBUG_SECRET, payload))

Required params:

• exp_key
• exp_variant
• exp_until
• exp_sig

Rules:

• the signature must match
• exp_until must be a future Unix timestamp in seconds
• the variant must exist in the registry

Example shape:

/?exp_key=homepage_hero_test&exp_variant=tiny_gang_social_proof&exp_until=1770000000&exp_sig=<hex_hmac>

Local dev-only cookie override is also supported:

• cookie name: exp_debug_<experiment_key>
• example: exp_debug_homepage_hero_test=tiny_gang_social_proof
• ignored outside localhost/127.0.0.1

Frontend analytics

Experiment analytics live in frontend/src/lib/experiments/client.js.

It sends:

• experiment_key
• variant_key
• session_id
• path
• event_name
• timestamp
• conversion_value
• metadata

Tracked events

• impression
  • auto-fired once per experiment variant per path per browser session
• cta_click
  • fired from experiment-controlled CTAs on the homepage and pricing page
• form_start
  • fired once when the Meet & Greet form is first engaged
• form_submit
  • fired on submit attempt
• conversion
  • fired after a successful Meet & Greet submission

Delivery behavior

• navigator.sendBeacon() first
• fetch(..., { keepalive: true }) fallback
• failures are swallowed so UX does not block

Backend API

The Python experiment endpoints are implemented in backend/app/routers/experiments.py.

Public endpoints

• GET /api/experiments
  • returns current experiment definitions and variants
• POST /api/experiments/impression
• POST /api/experiments/event
• POST /api/experiments/conversion

Admin endpoint

• GET /api/v1/experiments/results
  • optional query param: experiment_key
  • auth required

Validation

Payload validation is defined in backend/app/schemas/experiments.py.

Checks include:

• experiment/variant key format
• session id format
• path must start with /
• allowed event names per endpoint
• scalar-only metadata values
• known experiment + variant validation against the backend registry

Bot filtering

Analytics quality filtering only applies to the analytics endpoints.

• bot-like user agents are dropped from experiment analytics ingestion
• this is not used for page serving
• dropped bot requests return {"ok": true, "accepted": false}

Rate limiting

Each public experiment ingestion endpoint is limited to:

• 30/minute

Database model

Migration:

• backend/alembic/versions/bd9f6a8b7c1d_add_experiments.py

Tables:

• experiments
• experiment_variants
• experiment_events

Important indexed fields on experiment_events:

• experiment_key
• variant_key
• session_id
• user_id
• path
• event_type
• created_at
• composite (experiment_key, variant_key, created_at)
• composite (session_id, created_at)

Registry and sync

Seed registry:

• backend/app/experiments/registry.py

Frontend fallback registry:

• frontend/src/lib/experiments/registry.js

Startup seed sync:

• backend/app/main.py calls sync_experiment_registry() during app lifespan startup

Important:

• startup sync creates missing experiments and variants from code defaults
• it does not overwrite existing admin-managed values
• admin edits therefore persist across restarts

Backend management

Admin APIs:

• GET /api/admin/experiments
• GET /api/admin/experiments/{experiment_key}
• PUT /api/admin/experiments/{experiment_key}

Admin UI:

• /admin/experiments

Editable fields:

• cookie_name
• name
• description
• enabled
• eligible_routes
• variant labels
• variant allocations
• control variant selection

Current experiments

homepage_hero_test

• route: /
• variants:
  • control
  • tiny_gang_social_proof
• changes:
  • hero headline
  • supporting copy
  • hero CTA order/text
  • trust-pill wording

pricing_cta_test

• route: /our-pricing
• variants:
  • control
  • meet_greet_emphasis
• changes:
  • pricing card CTA label
  • pricing CTA panel headline
  • pricing CTA panel body
  • pricing CTA panel button label

How to add a new experiment

1. Add a seed definition to backend/app/experiments/registry.py.
2. Optionally add a fallback definition to frontend/src/lib/experiments/registry.js.
3. Start the backend so startup sync creates the DB rows.
4. Adjust the live definition in /admin/experiments if needed.
5. Expose the assignment in a route +page.server.js or +layout.server.js.
6. Render the variant in SSR using the server-provided assignment.
7. Only change presentation/CTA/component framing unless you are intentionally testing content that still preserves canonical meaning.
8. Add experimentClick tracking to experiment-owned CTAs.
9. If a form conversion should be attributed, make sure the page participates in trackActiveExperiments().
10. Add or update tests.

How to analyse results

1. Query GET /api/v1/experiments/results.
2. Compare variants on:
   • impressions
   • CTA clicks
   • form starts
   • form submits
   • conversions
   • unique sessions
   • conversion rate
   • conversion value total
3. Sanity-check sample size before making rollout decisions.
4. Review qualitative differences separately from SEO, because SEO is intentionally held constant by URL/canonical behavior.

How to disable experiments safely

Disable everything

Set:

PUBLIC_EXPERIMENTS_ENABLED=false

Effect:

• new requests render control
• no active experiments are tracked
• canonical behavior is unchanged

Disable one experiment

Set the experiment to disabled in /admin/experiments.

Effect:

• the page falls back to control
• existing cookies are ignored because the backend-managed definition marks the experiment inactive

Stop analytics but keep assignment

Set:

PUBLIC_EXPERIMENT_ANALYTICS_REQUIRES_CONSENT=true

and do not grant gw_analytics_consent.

Effect:

• SSR variant assignment can continue
• no experiment analytics events are sent

Verification status

Verified locally:

• backend experiment endpoints and related analytics tests:
  • python -m pytest backend/tests/test_experiments.py backend/tests/test_analytics.py backend/tests/test_analytics_ingest.py

Not verified locally:

• frontend Vitest suite

Reason:

• npm test hit a local sandbox spawn EPERM restriction while loading Vitest config
• escalation to rerun outside the sandbox was not approved in this session