Statistical Significance in F2P Games
Decide whether a product, marketing, or monetization test result is reliable enough to act on, without confusing noise, novelty, whale effects, or cohort maturity for truth.
Executive Summary
Use this guide after you have a specific test result and need to know whether to adopt, revert, extend, or call it inconclusive. First identify the data type and unit of analysis, then verify sample size, duration, sample-ratio integrity, cohort maturity, segmentation, and practical effect size. A low p-value is not enough if the test was underpowered, peeking-driven, whale-sensitive, or operationally trivial.
Reviewed May 20, 2026 against Transcend's unified F2P significance framework and NIST references for proportion testing and sample-size logic. All numeric sample sizes are planning heuristics; rerun calculations with the company's baseline, minimum detectable effect, desired power, alpha, and actual metric distribution.
MECE Boundary
This guide answers one question: can we trust a test result enough to act? It should not carry benchmark strategy, live KPI monitoring, or root-cause diagnosis.
| Question | Use This Surface | What It Decides |
|---|---|---|
| Is this experiment result reliable? | Statistical Significance | Sample adequacy, test validity, uncertainty, and action confidence. |
| Did a live KPI move enough to investigate? | Control Chart Diagnostics | Ongoing process monitoring and out-of-control signals. |
| Which ROAS lever broke first? | ROAS Diagnostic | Attribution, CPM, CTR, ARPU/AOV, conversion, and platform caveats. |
| What product or lifecycle cause should we fix? | Product Marketing Analysis | Retention, conversion, economy, paywall, lifecycle, and offer diagnosis. |
Frame the Test
Before looking at results, write down the decision, the minimum detectable effect, the metric owner, and the unit of analysis. Most bad reads start by asking the right statistical question of the wrong data.
| Question Type | Unit of Analysis | Recommended Method |
|---|---|---|
| Product A/B test | Randomized users | Two-proportion test for binary outcomes; Welch/bootstrap for skewed continuous metrics. |
| Offer or payer-conversion test | Users exposed to the offer | Test payer conversion first; treat ARPPU and LTV as whale-sensitive. |
| Marketing channel comparison | Users or cohorts, depending on export granularity | User-level comparison when possible; aggregated models with outlier checks when not. |
| Version impact / pre-post read | Cohorts or time periods | Interrupted time series, diff-in-diff, or regime analysis; do not call it a clean A/B test. |
| Guild, PvP, or social feature | Cluster, guild, region, or geo | Cluster-randomized or geo test because one user's treatment can affect another user. |
required sample = f(baseline rate, minimum detectable effect, alpha, power)
NIST frames proportion sample size around baseline rate, detectable change, significance level, and power. In Transcend work, default planning is 80% power and alpha 0.05; use higher power or alpha 0.01 for irreversible, high-cost, or investor-facing decisions.
Sample Size Guide
These are planning heuristics from the Transcend framework, not universal benchmarks. Replace the baseline and minimum detectable effect with company data before launch.
| Metric | Example Baseline / MDE | Planning Sample | Use Caution When |
|---|---|---|---|
| Tutorial completion | 50% baseline, 5pp lift | About 1,600 users per group | Assignment starts after tutorial exposure or excludes early crashers. |
| D1 retention | 40% baseline, 5pp lift | About 1,500 users per group | Day-of-week mix differs between variants. |
| D7 retention | 15% baseline, 3pp lift | About 2,800 users per group | The test has not run through complete weekly cycles. |
| Payer conversion | 3% baseline, 1pp lift | About 4,500 users per group | The payer count is below 100-200 total payers. |
| LTV / ARPU | 10% lift, typical F2P CV near 3 | About 14,000 users per group | A few whales can flip the mean. |
| Whale-heavy LTV | 10% lift, high CV near 6 | 50,000+ users per group | Bootstrap, cap, or sensitivity checks disagree. |
Data Type Gate
Do not apply user-level A/B logic to aggregated MMP exports. If individual users are already collapsed into daily or cohort rows, whale handling, SRM, and bootstrap options change.
| Gate | User-Level Data | Aggregated Cohort Data |
|---|---|---|
| Typical source | Experiment platform or internal analytics export. | AppsFlyer, Adjust, Singular, dashboard, or cohort CSV. |
| Whale handling | Cap individual spend, bootstrap users, remove top 0.1% for sensitivity. | Winsorize cohort ROAS, inspect Cook's Distance, rerun with high-influence rows removed. |
| Quality check | Sample Ratio Mismatch on assignment counts. | VIF, autocorrelation, outlier, maturity, and source-manifest checks. |
| Minimum sample logic | Users or payers per variant. | Cohorts or time periods; 30-90+ periods is often the gating issue. |
Integrity Checks
Run these before interpreting p-values. A test that fails an integrity gate can still produce a precise-looking number; the precision is just false confidence.
| Failure Mode | Why It Breaks the Read | Required Check |
|---|---|---|
| Sample Ratio Mismatch | Variant assignment is not random; missing users are likely non-random. | Run chi-square on assignment counts before any metric read. |
| Peeking | Stopping when p-value looks good inflates false positives. | Commit sample size and duration before launch, or use sequential methods. |
| Novelty effect | Early engagement can rise because the change is new, then decay. | Compare first cycle versus later cycle before adopting. |
| Cohort maturity mismatch | D30 from old cohorts is being compared with D14 from new cohorts. | Use the same maturity cutoff across variants and cohorts. |
| Multiple comparisons | Many variants or segments make one false winner likely. | Apply Bonferroni or FDR discipline before declaring a winner. |
| Whale sensitivity | One high spender can dominate ARPU, ROAS, or LTV. | Report capped, uncapped, and sensitivity views; prefer payer conversion when possible. |
| Segment reversal | Aggregate result masks iOS/Android, geo, channel, or tenure differences. | Check top segments before shipping globally. |
Decision Standard
The decision should combine statistical confidence, practical materiality, and test integrity. Do not ship a tiny effect because it is significant, and do not discard a meaningful but underpowered result when extending is cheap.
| Decision | Use When | Next Action |
|---|---|---|
| Adopt / Scale | Integrity checks pass, sample target is met, effect is statistically credible, and lift clears the business threshold. | Ship with monitoring and define the post-ship control chart. |
| Revert | The result is credibly negative or creates unacceptable downside in a protected segment. | Roll back and document the suspected mechanism. |
| Extend | Direction is meaningful, integrity checks pass, but power or maturity is not enough. | Continue to the pre-defined sample or maturity date. |
| Inconclusive | The result is underpowered, mixed, operationally trivial, or invalidated by a quality gate. | Stop, archive, and design a better test only if the decision still matters. |
| Accept Shift | The business intentionally changed regimes, audiences, economics, or strategy. | Document the new baseline and route future monitoring to control charts. |
Current Source References
| Reference | Use in This Guide |
|---|---|
| NIST: Sample Sizes Required for Proportions | Sample-size logic for baseline rate, detectable change, alpha, and power. |
| NIST: Difference of Proportion Hypothesis Test | Two-proportion hypothesis-test framing for binary outcomes. |
| NIST/SEMATECH Engineering Statistics Handbook | General applied-statistics reference and engineering-method context. |
skills/skills/STATISTICAL_SIGNIFICANCE_UNIFIED_FRAMEWORK.md |
Transcend Type A/Type B distinction, sample guidance, whale handling, SRM, and decision vocabulary. |
skills/skills/STATISTICAL_SIGNIFICANCE_F2P_GUIDE.md |
F2P-specific context, benchmark caveats, full-week rule, monetization-test caveats, and spend-to-learn framing. |
Final Rule
Trust the result only when the question, data type, unit, sample, maturity, segments, and effect size all line up. When they do not line up, the A+ answer is not a prettier p-value; it is a clearer decision to extend, redesign, or stop.