Try for free

Audience Increments

Running A/B Comparison

There are three variables worth testing in a typical D2C A/B testing world:

  1. Audience increments:
    1. To a platform like Meta who likes to get a clear feed of audience signal from your site - anonymous and member traffic
    2. To a platform like Klaviyo who can provide member lift metrics
  2. A/B Testing:
    1. Independently test feeds by running two separate pixels side by side
    2. Do not use BAU as it creates bias
    3. Always use expanded windows for testing - 1-dc is not a great way test [ because you are already looking at in-market buyers who are already buying ]
  3. Capabilities [Product Specific]:
    1. Can you run custom segments to optimize against a goal?
    2. Can you enable the site personalization API?
    3. Etc.

Audience Increments

Tools like Meta + Klaviyo (that Spanx has deployed) already provide feedback loops for how to measure audience and so first A/B testing is measuring feed quality

Meta (Test 1)

Prerequisite: Ensure pixels are set up correctly and run for a minimum of 7 days.

Meta uses a feed quality metric called EMQ. This enables the customer to be able to measure the quality of the datafeed. More information can be found here.

How to Test?

  1. Leave the existing provider data feed running to a pixel.
  2. Create a new pixel to provide the data feed (e.g., Blotout).
  3. Measure the data feed quality scores within 2-3 days using Meta's snapshot.

Meta (Test 2)

Prerequisite: Ensure pixels are set up correctly and run for a minimum of 14 days.

If you want to run an expanded audience increment test; use the web pixel from each existing provider feed and create an audience for the past 7 days. You can measure reach using the data feeds as well.

How to Test?

  1. Take the existing provider and create a past 7-day look for PageView users.
  2. Replicate the same for the new pixel (e.g., powered by Blotout).

Klaviyo Test

Prerequisite: Ensure events are set up for each data feed for a minimum of 7 days (iOS cycle).

How to Test?

  1. Check event volume per feed.
  2. Check incrementality against each other (Segment but isolate one provider).
  3. Check against Checkouts, Carts, PDPs, and Sessions.

Performance Testing [TOF / Audience Quality]

Meta remains a good choice for performance testing; however, pre-existing pixels (BAU) come with bias.

How to Set Up?

  1. Set up a new A test pixel against the existing feed.
  2. Set up a new B test pixel against Blotout.
  3. Ensure both pixels are fed for 10 days (14 preferred).

How to Test?

  1. Create two new campaigns with the exact same setup.
  2. Ensure the only difference between the two campaigns are the A and B pixel.
  3. Split test A/B only; do not compete against BAU (isolate goals).
  4. Run it on a 7DC1DV to check attribution virtues.
  5. Look for ROAS/CAC metrics.

Other Variables

These are variables outside of audience expansion and better data quality.

VariableVendor?Impact
CIPA complianceYesCompliance
Custom Conversion EventsYesChannel Performance
Shopify Cart Server SideYes1% GMV incremental (Shopify limitation)
Personalization APIYesUse it to hyper-personalize Site
Channel TransferYesMove from KL to ATTN and don't miss a heartbeat