If paying by check, please include payment stub with your payment. It’s the CRO Roundtable Roundup!
Thanks to Iqbal Ali, Matt Beischel, Slobodan Manić, Shiva Manjunath, Kevin McCarthy, Friedolin Stockmeier, Andrei Stoleriu, and Craig Sullivan for joining us. Want to get in on the action and talk with other cool CRO people? Then you should probably…
This Week's Roundtable Topic
Session 284: How to Deal With Low Traffic
- When traffic is low, which metrics or methods should guide your experiments?
- Research methods as alternatives to A/B testing
- Use simulations and priors, not significance, when traffic is scarce
One Sentence Takeaway
I’m a busy person; give me the TL:DR
Experimentation programs fail without aligned incentives, psychological safety, and organizational support for learning through failure.
The Notes
What we discussed this week
- Incentive Misalignment in Experimentation Programs
- Temporary experiment code provides little career signaling for engineers
- Experimentation work conflicts with traditional developer progression paths
- Senior leaders prefer personal attribution over experimental decision-making
- Career incentives discourage delegating decisions to testing programs
- Organizational rewards favor visible ownership over collective learning
- Psychological Safety as a Requirement for Experimentation
- Fear of failure suppresses experimentation despite stated leadership support
- Leadership tolerance for failure often fails to reach middle management
- Teams hesitate to test when job security feels unstable
- Learning velocity declines in low-trust organizational environments
- Experimentation requires permission to fail without personal consequence
- Organizational Change Through Experimentation Education
- Widespread experimentation adoption requires sustained internal advocacy
- Workshops alone rarely shift entrenched beliefs or biases
- Cultural change occurs slower than experimentation leaders expect
- Resistance often stems from prior success and professional ego
- Experimentation maturity depends on long-term internal enablement
- Ethics in Experimentation Practice
- Experimentation decisions raise ethical considerations beyond metrics
- Testing impact extends to user trust and organizational responsibility
- Ethical reflection often emerges through peer discussion
- Experimentation ethics benefit from open, cross-functional dialogue
- Skepticism Toward Exaggerated Experimentation Claims
- Publicized experiment results frequently lack methodological credibility
- Extreme uplift claims undermine trust in experimentation discipline
- Ambiguous metrics obscure actual experimental impact
- Simplistic test narratives misrepresent real-world complexity
The Quotes
Standout quips from this week
Book Club
Relevant reads recommended this week
No books this week, sorry!
CRO Link Digest
Useful and thought-provoking content shared this week
- The Schwartz Pyramid: guide to the 5 levels of customer awareness
blog article - How to find a Unicorn: Dunning-Kruger & the Competence Pyramid
LinkedIn article by Alan Marcus - Potemkin Stupidity
(PDF) academic study by Jerry Sheppard and Jesse Young - Stop building Potemkin Villages!
Medium article by Mark Dalgarno - I was forced to use AI until the day I was laid off
Medium article by Brian Merchant - The Bumper Pack of Forms Resource Optimisation
Shareable Google doc
Off-Topic Sidebars
Experimentation isn’t the only thing we talk about at the CRO Roundtable. There’s often a healthy dose of discussion on shared interests, personal passions, and hobbies.
- Conference Format Preferences
- Lower engagement during presentation-heavy sessions
- Higher energy in discussion-based formats
- Social Media Content Dynamics
- Engagement bait driving exaggerated positioning
- Algorithms rewarding provocation over accuracy
- Personal Brand Performance
- Intentional persona amplification for visibility
- Trade-offs between authenticity and reach
Sidebar Shareables
Amusing sidebar content shared this week
No Sidebar Shareables this week, sorry!