Don’t forget, you are awesome. It’s the CRO Roundtable Roundup!
Thanks to Iqbal Ali, Matt Beischel, Slobodan Manić, Shiva Manjunath, Craig Sullivan, and David Swinstead for joining us. Want to get in on the action and talk with other cool CRO people? Then you should probably…
One Sentence Takeaway
I’m a busy person; give me the TL:DR
Running experiments with clear hypotheses, pre-mortems, and risk-aware thresholds helps ensure valuable learning even from inconclusive tests.
The Notes
What we discussed this week
- Experiment Statistics: Power and Thresholds
- Underpowered tests often create false confidence in directional results
- Bayesian vs frequentist approaches to probability thresholds
- 60% vs 90% significance thresholds and cultural impacts
- Non-inferiority framing as a valid alternative to strict wins
- Hypotheses used to permit future decisions despite inconclusive results
- Expected loss considered better than p-values for de-risking
- The Importance of Doing Pre-Mortems
- Importance of writing hypotheses tied to measurable outcomes
- Pre-mortems encourage thinking through win, loss, or flat scenarios
- LLMs can be used to generate possible reasons for test failure
- Predictive bias leads teams to over-assume tests will be winners
- Pre-mortems help teams prepare iterative next steps
- Business Metric Trade-Offs
- Lowering AOV may reduce revenue if not offset by more orders
- Revenue is a product of both AOV and order volume
- Some businesses prefer higher AOV, others prefer more customers
- Trade-offs differ between e-commerce and SaaS models
- More customers increase long-term remarketing opportunities
- AI, Data Use, and Compliance
- Concerns about companies scraping data without consent
- Past leaks exposed sensitive chats and corporate plans
- Ambiguity in “make discoverable” wording creates dark pattern risks
- Companies are intentionally pushing boundaries of data privacy
- Legal disputes around AI training datasets were discussed
- Recruiters and employees weigh personal ethics in company choices
- Dealing with Bug Fixes in Testing Practices
- Some large winning tests came from fixing bugs
- A/B testing bugs highlighted hidden losses and drove improvements
- Automated testing could prevent costly experiment errors
Hey all you cool GRO People!
GRO Talks Live returns to the winter 2025 Experimentation Elite conference with a heaping helping of in-person roundtable sessions, and we want to see you there!
The Quotes
Standout quips from this week
Book Club
Relevant reads recommended this week
Empire of AI
CRO Link Digest
Useful and thought-provoking content shared this week
Off-Topic Sidebars
Experimentation isn’t the only thing we talk about at the CRO Roundtable. There’s often a healthy dose of discussion on shared interests, personal passions, and hobbies.
No off-topic sidebars this week, sorry!
Sidebar Shareables
Amusing sidebar content shared this week
No sidebar shareables this week, sorry!