Retainers / CRO & Experimentation
Most CRO is redesign with a nicer name.
An agency changes your button colour, revenue goes up 3% that month, and everyone agrees it worked. Nobody checks whether the sample was big enough to mean anything, or that you also ran a promotion. Real experimentation is dull, statistical and occasionally tells you your best idea lost — which is exactly why it compounds and opinion-led redesigns do not.
- What it is
- An ongoing conversion optimisation and A/B testing retainer for Shopify and Shopify Plus — a continuous programme of hypotheses, builds and tests, rather than a one-off audit or redesign.
- Who it's for
- Stores with enough traffic to test — roughly 25,000 sessions and 300+ monthly conversions per template you want to test on. Below that we recommend qualitative research instead, and we will say so.
- What runs monthly
- Two to four experiments live at a time, a maintained hypothesis backlog ranked by expected value, full build and QA of every variant, and a written readout of every result including the losers.
- How we call a result
- Pre-declared sample size, minimum detectable effect and stopping rule. No peeking, no stopping early because the line looks good on day three.
- Tooling
- Your existing stack where you have one. Otherwise we implement testing that does not add a render-blocking script or introduce flicker.
- Investment
- From $4,000 per month. Included from the Growth tier when bundled. Three-month initial term, then rolling monthly.
- What it is not
- Not a redesign, and not a guaranteed uplift. Anyone promising a specific conversion increase before seeing your data is selling you a number, not a programme.
Why most CRO fails
The problem is not the ideas. It is how they are judged.
Almost every failed CRO engagement fails the same way — the work is fine and the measurement is fiction.
How CRO usually gets done
Plausible, confident, and impossible to learn from
- Changes are shipped as a batch redesign, so nobody knows which one moved the number
- A test is stopped as soon as it looks like winning — the most common way to manufacture a false positive
- Results are read against last month, which also contained a promotion and a holiday
- No sample size was calculated, so the test never had the power to detect the effect it claimed
- Losers are quietly dropped from the report instead of being the most useful finding in it
- The programme is opinion-led — whoever is most senior in the meeting wins the argument
How we run it
Slower to declare, far harder to argue with
- One change per variant, so a result is attributable to a cause
- Sample size and stopping rule declared before launch, then honoured
- Concurrent control, so seasonality and promotions hit both arms equally
- Minimum detectable effect agreed up front — if the test cannot detect it, we do not run it
- Losers written up. A disproven assumption saves the next three experiments
- Backlog ranked by expected value, not by who suggested it
The cycle
Every experiment runs the same loop.
Built from analytics, session recordings and the audit backlog. Written as a falsifiable statement with an expected effect, not a suggestion.
Sample size calculated from your real baseline, then the variant built properly — no flicker, no layout shift, QA on real devices.
Live until the pre-declared sample is reached. Monitored for validity, never stopped early because it looks good.
Ship, kill or iterate — with the result written up either way, and what it taught us folded back into the backlog.
What we test
Where the money usually is.
Ranked roughly by how often the test wins on the stores we see. Button colour is not on the list.
Product page architecture
What appears above the fold, where price and shipping sit, how variants are chosen, and where trust content interrupts hesitation.
- Variant selector patterns
- Delivery promise placement
- Trust and returns positioning
Cart and checkout path
Drawer versus page, the moment shipping cost is revealed, express payment prominence, and how many decisions sit between intent and payment.
- Cost transparency timing
- Express wallet placement
- Step reduction
Collection and discovery
Filtering, sort defaults, how many products load before fatigue, and whether the grid answers the question the visitor arrived with.
- Facet design
- Sort defaults
- Load pattern
Merchandising and offer
Bundle framing, threshold messaging for free shipping, cross-sell placement and whether urgency helps or costs you trust.
- Threshold nudges
- Bundle presentation
- Cross-sell position
Copy and value proposition
Headline framing, benefit ordering, and how the brand answers the objection a visitor already had before arriving.
- Headline tests
- Objection handling
- Proof placement
Mobile interaction cost
Tap targets, sticky elements competing for the same space, and the interaction delay that makes a phone feel broken rather than slow.
- Sticky element conflicts
- Tap target sizing
- Input friction
Honest statistics
What has to be true for a result to be worth acting on.
These are the checks we apply before calling anything a win. Most reports you have been sent would fail at least two.
| Check | What it means | What happens if it is skipped |
|---|---|---|
| Powered | Sample size calculated from your baseline and the minimum effect worth detecting. | The test cannot see the effect it claims to have found. Most "wins" are noise. |
| Concurrent | Control and variant run at the same time, to the same traffic mix. | You are comparing March to April and calling the weather a conversion lift. |
| Pre-declared | Stopping rule and success metric agreed before launch. | Peeking until it looks good manufactures a false positive roughly one time in three. |
| Single-variable | One change per variant. | Something moved. Nobody can say what, so nothing is learned for next time. |
| Segment-checked | Result holds on mobile and desktop, new and returning. | A desktop win hides a mobile loss, and you ship a net negative. |
| Durable | Re-checked after ship for novelty decay. | The uplift evaporates in six weeks and nobody notices it left. |
We run a modest number of experiments well rather than a large number badly. On most stores that is two to four live at once — enough to keep learning, few enough that each is powered and clean.
Investment
Priced on the programme, not on hours.
CRO & Experimentation retainer
Hypothesis backlog, two to four live experiments, full variant build and QA, statistical readouts on every result including losers, and a monthly session to decide what runs next.
A starting point for scoping rather than a quote — it depends on traffic volume, how many templates are in scope and how much build each variant needs. Included from the Growth tier when bundled with another retainer.
Questions we get
Straight answers.
How much traffic do we need before testing is worth it?
Roughly 25,000 monthly sessions and 300+ conversions on the template you want to test. Below that, a test needs months to reach significance and the business will have changed underneath it. If you are under that threshold we will tell you on the first call and recommend qualitative research, analytics work or a straight rebuild instead — all of which are cheaper and will move your numbers more.
Can you guarantee a conversion rate increase?
No. Anyone quoting a specific uplift before seeing your data is selling a number rather than a programme. What we commit to is a defined number of properly powered experiments per month, honest readouts including the losers, and a compounding record of what is true about your customers. Over a year that reliably beats a redesign, but the honest answer to "how much" is that it depends on where you are starting.
What happens when a test loses?
It gets written up like any other result, because a disproven assumption is often worth more than a small win — it stops you building the next three things on top of something false. Roughly half to two-thirds of well-designed experiments do not produce a winner. Any agency reporting a much higher hit rate is either stopping tests early or not calculating sample sizes.
Do you use our testing tool or your own?
Yours where you have one that works. Where you do not, we implement testing that does not add a render-blocking script or cause the flicker that both damages the experience and biases the result. We would rather spend a week getting the implementation clean than run a year of tests through something that slows the page it is measuring.
How is this different from your CRO service page?
The CRO service is a fixed-scope project — an audit, a set of prioritised fixes, a rebuild of specific templates. This retainer is the ongoing programme that runs afterwards, or instead of it if your templates are already sound. Many brands do the project first and move onto the retainer once the obvious problems are gone.
Who builds the variants?
The same senior engineers who build our client storefronts. That matters more than it sounds — badly built variants introduce flicker, layout shift and mobile bugs that suppress the very metric you are measuring, which quietly turns winning ideas into losing tests.
Next step
Find out what is worth testing first.
The free CRO and tracking audit produces the initial hypothesis backlog — ranked by expected value, yours whether or not you take the retainer.
