Measuring custom pages

Both centres report performance per alternative version. Both also quietly hide that reporting below a certain amount of traffic, which is exactly where most alternative versions actually sit.

On the locked side

You can see, per alternative version: how often it was seen, how many people walked in, and the resulting rate. But only once a version has crossed a small minimum number of first-time walk-ins. The reference name you gave it is what identifies it in your own reports.

That small minimum is the real practical constraint. With a genuinely large allowance of possible versions, it's easy to create ones that never cross it. So they quietly generate traffic, cost upkeep, and report absolutely nothing. A version below that line isn't a failed version; it's simply an unmeasured one, and from the outside those two look exactly the same.

Your reference names are your entire reporting system. They're how versions show up in your own reports. Naming them for the audience and the message, rather than just numbering them in sequence, is the whole difference between a report you can actually read and one that turns into an archaeology project.

On the other centre

An alternative listing's performance only shows up in reports once it's crossed some minimum amount of visitor traffic, and the centre itself notes that a listing aimed specifically at paid traffic typically pulls in fewer visitors than a broader campaign would, simply because it's limited to one specific outside ad network.

This centre's own listing reports split traffic by source (its own search, its own browsing, ads, and referrals) and separately report the actual words that led someone to a given listing.

The main measurement itself changed recently, shifting from counting genuine new visitors to counting a specific kind of click instead. Comparing any listing result from before that change to one measured after it is comparing two genuinely different events.

One centre's own headline figure, and what it's actually worth

The locked side states that shopkeepers see a real average improvement when sending people to an alternative version, describing it as a striking-sounding percentage increase over the ordinary conversion rate on a default page.

Worth taking seriously because the centre itself states it. Not because you should expect to personally hit that exact number. The centre publishes no method, no sample, and no time period behind that figure; the underlying baseline varies enormously by category and by market; and traffic sent to an alternative version is inherently self-selected, since anyone building one is already targeting a specific audience on purpose.

Use it as evidence that the underlying mechanism genuinely works. Don't use it as a personal target, and don't put it into a forecast.

Comparing two alternative versions is usually not a real test

This is the single most common reasoning mistake in this whole area.

Two alternative versions receiving traffic from two different sources differ in who's actually looking, not just what they see. If one version is fed by a search campaign and another by a display campaign, a difference in how well they convert tells you at least as much about the traffic as it does about the version itself.

These alternative versions can't be run through the centre's own proper randomised testing tool at all, so there's no built-in way to fairly split traffic between them.

To compare two versions meaningfully, hold the traffic source completely constant (the same campaign, the same audience, the same time period, simply split between the two versions) and even then, you're only measuring paid traffic, not your normal, unpaid visitors.

What to actually report

For each version: its reference name, which audience or campaign it serves, how often it was seen, how many people walked in, the resulting rate, and where that traffic actually came from. Without that last piece, the rate itself can't really be interpreted at all.

And state plainly whether the version actually crossed the reporting threshold. "No data yet" and "performing poorly" are two completely different findings, and they get mixed up constantly.

What tends to go wrong

Versions sitting below the reporting threshold. A small minimum of walk-ins on the locked side; an unstated amount on the other. They report nothing at all and quietly look like failures.

Comparing versions fed by different sources. Measures the traffic, not the version itself.

Sequentially numbered reference names. An unreadable report months later.

Comparing across the point where the measurement itself changed. Genuinely different events being counted.

Counting every walk-in through an alternative version as a brand-new visitor. Some of those people would have converted on the default page anyway, whether the alternative version existed or not.