A/B Testing Guide

This guide covers the technical details of WittyForm's A/B testing system, including how variants are created and served, how traffic is distributed, the statistical methods used to evaluate results, and how to work with A/B tests programmatically. A/B testing requires the Enterprise plan.

Creating variants via the UI

Each A/B test starts with a base form (Variant A, the control) and one or more challenger variants. When you create a new variant from the A/B Test tab, WittyForm performs a deep clone of the current form configuration, including all fields, conditional logic rules, theme settings, and integrations. Each variant receives a unique identifier as a UUID (e.g., 3f29c1a4-9b21-4e7d-8c10-6a5f0b2d1e84).

Variants are independent after creation. Changes to Variant A do not propagate to Variant B and vice versa. You can modify any aspect of a variant: add or remove fields, change the question order, swap themes, or alter conditional logic.

A single test supports up to four variants (A, B, C, D). If you need to test more than four options, run sequential tests after concluding the current one.

Traffic distribution

Traffic is allocated using a weighted random distribution. When a new visitor loads the form, the server generates a random number between 0 and 1 and maps it to a variant based on the configured weights. For example, with a 70/30 split:

• Random value 0.00 to 0.69 → Variant A
• Random value 0.70 to 0.99 → Variant B

Once a visitor is assigned to a variant, the assignment is stored in browser localStorage under the key wf_ab_[formId] so they see the same variant on subsequent visits. The entry expires after 30 days or when the test ends, whichever comes first. In private/incognito mode where localStorage may be unavailable, the assignment falls back to a per-tab session.

Traffic allocation can be adjusted while a test is running. Changes take effect immediately for new visitors. Existing visitors retain their original assignment.

Statistical methods

WittyForm applies two complementary statistical methods to evaluate test results. Both run continuously as new data arrives.

Bayesian inference

Each variant's conversion rate is modeled as a Beta distribution with parameters α = successes + 1 and β = failures + 1 (using a uniform Beta(1,1) prior). The "probability of being best" is computed via Monte Carlo simulation: 10,000 samples are drawn from each variant's Beta distribution, and the fraction of simulations where a given variant has the highest value is reported as its win probability.

This approach provides an intuitive interpretation. A statement like "Variant B has a 94% probability of being the best" means that, given the observed data, there is a 94% chance that Variant B's true conversion rate is higher than all other variants.

Z-test for proportions

For each pair of variants, WittyForm computes a two-proportion Z-test to determine whether the difference in conversion rates is statistically significant. The test statistic is:

z = (p1 - p2) / sqrt(p_pool * (1 - p_pool) * (1/n1 + 1/n2))

where:
  p1, p2    = conversion rates for variants 1 and 2
  n1, n2    = sample sizes for variants 1 and 2
  p_pool    = (x1 + x2) / (n1 + n2)  (pooled proportion)
  x1, x2    = number of conversions for variants 1 and 2

The resulting z-score is converted to a two-tailed p-value. A result is flagged as statistically significant when p < 0.05 (95% confidence). The dashboard also displays 90% and 99% confidence levels for reference.

Minimum sample sizes

The minimum sample size required to detect a given effect depends on the baseline conversion rate and the minimum detectable effect (MDE) you want to identify. The following table shows recommended minimum visitors per variant for common scenarios:

* Duration estimates assume 500 to 1,000 visitors per day across all variants.

WittyForm displays a progress bar on the test dashboard showing how close you are to reaching the recommended sample size. Tests that are concluded before reaching the minimum sample size display a warning that results may not be reliable.

Interpreting p-values

The p-value tells you the probability of observing the measured difference (or a larger one) if the two variants were actually identical. A low p-value means the observed difference is unlikely to be due to chance alone.

• p < 0.01: Very strong evidence that the variants perform differently. You can be highly confident in the result.
• p < 0.05: Strong evidence. This is the standard threshold WittyForm uses to flag results as statistically significant.
• p < 0.10: Suggestive but not conclusive. Consider running the test longer to collect more data.
• p ≥ 0.10: No significant difference detected. The variants may perform similarly, or you may not have enough data yet.

Example test results

The following table shows a realistic A/B test comparing two form layouts after two weeks of data collection:

In this example, Variant B (the challenger) has a 29.00% conversion rate compared to Variant A's 24.98%. The p-value of 0.0015 is well below the 0.05 threshold, and the Bayesian analysis gives Variant B a 97.7% probability of being the better performer. This test has reached statistical significance and Variant B can be confidently declared the winner.

Tracking conversion events

By default, WittyForm tracks form completion (submission) as the conversion event. The A/B test dashboard shows completion rate per variant out of the box, with no extra instrumentation required.

For deeper funnel insight, every form submission is also delivered to your configured webhook endpoint with the active variant ID in the payload, so you can correlate downstream events (payment completed, account activated, etc.) against the variant in your own analytics tool.