A/B Testing¶

Optimize your marketplace listing performance by comparing different content variants and measuring real user engagement.

Overview¶

A/B testing allows you to scientifically test different versions of your marketplace listing content to determine which performs better. By splitting traffic between variants and measuring key metrics, you can make data-driven decisions to improve your conversion rates.

Getting Started¶

Creating an A/B Test¶

Navigate to GTM Campaigns > A/B Testing in your dashboard
Click Create A/B Test
Configure your test settings:
Test Name: A descriptive name (e.g., "Title Optimization Test - March 2024")
Hypothesis: What you expect to learn from this test
Primary Metric: The main metric to determine the winner
Select 2-4 content variants to compare
Click Create A/B Test to start

Selecting Variants¶

You can test any content versions you've created for your marketplace listings. Select at least 2 variants (up to 4) to compare. Variants can include different:

Titles and taglines
Short descriptions
Feature highlights
Pricing presentation
Call-to-action messaging

Metrics¶

Click-Through Rate (CTR)¶

Formula: Clicks ÷ Impressions × 100

Measures how often viewers click on your listing when they see it in search results or category pages. A higher CTR indicates your listing is attracting more attention.

Conversion Rate¶

Formula: Subscriptions ÷ Page Views × 100

Measures how often viewers become paying customers after visiting your listing page. This is often the most important metric for revenue optimization.

Free Trials Started¶

Total number of trial signups attributed to each variant during the test period. Useful for products with a freemium or trial-based model.

Revenue¶

Total Monthly Recurring Revenue (MRR) generated by customers who converted during the test. This metric accounts for different pricing tiers and helps optimize for revenue rather than just volume.

Statistical Confidence¶

We use a two-tailed z-test to determine if differences between variants are statistically significant. Results are considered significant at the 95% confidence level.

What does 95% confidence mean?

When we say a result has 95% confidence, it means there's only a 5% probability that the observed difference occurred by random chance. This is the industry standard for making business decisions based on test results.

Confidence Indicators¶

Confidence Level	Indicator	Recommendation
Below 90%	Low	Continue testing
90-95%	Moderate	Consider extending test
95%+	High	Safe to declare winner
99%+	Very High	Strong statistical evidence

Best Practices¶

1. Test One Element at a Time¶

For clearer results, change only one element per test (title, description, or highlights). Testing multiple changes simultaneously makes it difficult to determine which change drove the improvement.

2. Run Tests Long Enough¶

Allow tests to run for at least 7-14 days to gather sufficient data. This ensures you capture:

Weekday vs. weekend traffic patterns
Enough conversions for statistical significance
Natural variation in user behavior

3. Wait for Statistical Confidence¶

Don't declare a winner until you've reached 95% statistical confidence. Early results can be misleading due to random variation.

4. Document Your Hypotheses¶

Keep records of what you tested and learned. This builds institutional knowledge and prevents repeating failed experiments.

5. Consider Sample Size¶

Ensure each variant receives enough traffic to generate meaningful data. Low-traffic listings may need longer test durations.

Interpreting Results¶

Reading the Results Dashboard¶

After your test has collected data, the results dashboard shows:

Variant Performance: Side-by-side comparison of each variant's metrics
Lift Percentage: How much better (or worse) each variant performed vs. baseline
Confidence Level: Statistical confidence that the difference is real
Sample Size: Number of impressions/views for each variant

Declaring a Winner¶

A variant can be declared the winner when:

It outperforms other variants on the primary metric
Statistical confidence reaches 95% or higher
Sufficient sample size has been collected

No Clear Winner¶

Sometimes tests conclude without a clear winner. This happens when:

Differences between variants are too small to be statistically significant
All variants perform similarly
Sample size was insufficient

In these cases, consider:

Running a longer test
Testing more dramatically different variations
Focusing on a different element to optimize

Common Use Cases¶

Title Optimization¶

Test different title structures:

Feature-focused vs. benefit-focused
Including specific metrics or social proof
Different keyword placements

Description Testing¶

Compare description approaches:

Technical vs. business-focused language
Short vs. detailed descriptions
Different value proposition emphasis

Pricing Presentation¶

Test how you present pricing:

Highlighting savings vs. showing full price
Monthly vs. annual pricing emphasis
Including comparison to alternatives

FAQ¶

How long should I run a test?¶

We recommend a minimum of 7 days, ideally 14 days, to capture different traffic patterns and reach statistical significance.

Can I run multiple tests simultaneously?¶

Yes, but only if the tests don't overlap (different products or completely separate audiences). Running overlapping tests can contaminate results.

What if my listing doesn't get much traffic?¶

Low-traffic listings need longer test durations. Consider running tests for 30+ days or focusing on higher-traffic listings first.

Can I edit variants during a test?¶

No. Editing variants during a test invalidates the results. If you need to make changes, end the current test and start a new one.

How do I implement the winning variant?¶

Once a winner is declared, you can publish it directly from the results page. The winning content will replace your current live listing.