Please do not unplug anything from where it is. It’s the CRO Roundtable Roundup!
Thanks to Iqbal Ali, Matt Beischel, Shiva Manjunath, Craig Sullivan, David Swinstead, and Dewi Williams for joining us. Want to get in on the action and talk with other cool CRO people? Then you should probably…
This Week's Roundtable Topic
Price Testing
Session 287: February 27, 2026
Here, have some conversation starters:
- What pricing signals should experiments measure to predict retention?
- A/B price tests without cohort analysis are misleading.
- How should pricing models adapt to seasonal demand shifts?
One Sentence Takeaway
I’m a busy person; give me the TL:DR
UI revamps fail when teams chase aesthetics or ego without defined problems, rigorous measurement, and incremental validation.
The Notes
What we discussed this week
- Why most UI revamps destroy more value than they create
- Redesigns bundle structural fixes with unnecessary aesthetic experimentation
- Previously strong elements often regress while weak elements improve
- Traffic distribution shifts distort template-level performance comparisons
- Teams misattribute overall conversion decline to isolated components
- Post-launch firefighting replaces disciplined incremental improvement
- Lack of defined problems leads to cosmetic rather than strategic change
- Measurement layers as insurance against redesign blindness
- Set up a comparable tracking layer across old and new experiences
- Map equivalent funnel steps, even when the architecture changes
- Compare ratios instead of absolute numbers to detect breakage
- Pair quantitative diagnostics with session replay validation
- Surface hidden drop-offs before declaring the redesign a failure
- Separating technical migration from UX ambition
- Rebuild for feature parity before including experiential upgrades
- Avoid bundling theme refreshes into infrastructure migrations
- Resist stakeholder pressure to “modernize” during backend rewrites
- Preserve clean before-and-after comparisons to avoid ambiguity
- Prevent technical teams from being blocked by aesthetic underperformance
- When a full redesign is actually justified
- Extremely low traffic limits the feasibility of granular testing
- Fundamentally broken experiences
- The illusion of predictability in experimentation
- Teams consistently overestimate their forecasting accuracy
- Survivorship bias inflates confidence in past predictions
- The most statistically likely outcome of any test is neutral impact
- Testing exists because human intuition routinely fails
- Confident narratives often mask fundamentally random outcomes
- Impact scoring frameworks encourage false precision
- Objective evidence beats off-the-shelf prioritization models
- Traffic volume is underweighted in decision-making
- Evidence strength should outrank subjective confidence scores
- Multiple converging data sources increase robustness
- External case studies provide weak and context-poor signals
- Prioritization should remain dynamic as new evidence emerges
- Problem-centric backlogs drive better strategic learning
- Group experiments by clearly defined user problems
- Apply weighting at the problem level before solution ranking
- Iterate within a cluster to compound contextual insight
- Communicate progress as problem reduction rather than isolated wins
- Rebalance focus as relative problem sizes shift over time
- Build meta-learning across related experiments
- Triangulation improves understanding of user friction
- Analytics reveals patterns without explaining motivation
- Qualitative research surfaces behavioral and emotional context
- Heatmaps and recordings expose interaction breakdowns
- Customer complaints provide consequence-level signals
The Quotes
Standout quips from this week
“The problem is that in any redesign, you make some things worse that were perfectly okay before.”
“You can't compare. Why is conversion down? It's death by a thousand cuts.”
“If I was really good at predicting things, why would I waste that ability calling A/B test results on websites?”
“Always collect your failures because they're useful to analyze.”
“Triangulation is like... you have three data points to help you focus in on a specific area versus the whole ocean.”
“One of my friends used to get into big trouble because he was nearly always the person going, 'hang on a minute, this is not a good idea.'”
Book Club
Relevant reads recommended this week
No Book Club this week, sorry!
CRO Link Digest
Useful and thought-provoking content shared this week
- AI just tried to steal from us – LinkedIn post about someone using AI generated images to fake product damage, by Scott Tannen
- What Is Jobs-to-be-Done? – Medium article about the Jobs-to-be-Done thought framework, by Tony Ulwick
- How to Frame Problems With the TOSCA Framework – Medium article explaining TOSCA (Target, Objective, Strategy, Capabilities, Actions) problem-solving strategy, by Alex Miguel Meyer
- David Swinstead’s Prioritization Framework – infographic, by David Swinstead
- The Novelty Effect Is Not Real – slideshow deck arguing against the existence of the novelty effect, by David Swinstead
Off-Topic Sidebars
Experimentation isn’t the only thing we talk about at the CRO Roundtable. There’s often a healthy dose of discussion on shared interests, personal passions, and hobbies.
No Off-Topic Sidebars this week, sorry!
Sidebar Shareables
Amusing sidebar content shared this week
No Sidebar Shareables this week, sorry!