If paying by check, please include payment stub with your payment. It’s the CRO Roundtable Roundup!
Thanks to Iqbal Ali, Matt Beischel, and David Swinstead for joining us. Want to get in on the action and talk with other cool CRO people? Then you should probably…
This Week's Roundtable Topic
Moving to the Product Operating Model
Session 292: April 3, 2026
Here, have some conversation starters:
- How do CRO metrics fit into a product operating model?
- Who should own experimentation decisions when product, analytics, and engineering disagree?
- What are the best ways to prioritize initiatives in a product operating model?
One Sentence Takeaway
I’m a busy person; give me the TL:DR
Predictive, pattern-based CRO approaches create an illusion of certainty that can mislead decision-making unless grounded in real experimentation and contextual understanding.
The Notes
What we discussed this week
- The uneasy tradeoff between predictive CRO models and real experimental rigor
- Skepticism around tools claiming “winning patterns” without access to underlying test result data
- Scraping page variations is fundamentally different from understanding statistical outcomes or causality
- Probabilistic framing masks weak or nonexistent empirical foundations
- Tension between speed and rigor when prioritizing tests using modeled assumptions
- Debate over whether such systems meaningfully outperform intuition or simply repackage it
- Lack of transparency erodes trust even when outputs appear directionally useful
- When stakeholder confidence matters more than methodological purity
- External vendors can legitimize ideas that internal teams struggle to get approved independently
- Ethical gray area of using weak evidence to unlock progress that might otherwise stall
- Question whether outcomes justify methods when business results improve
- Conversion rate goals as incentives distort meaningful measurement
- The problem with tying performance strictly to conversion rate without accounting for traffic quality
- Risk of optimizing toward superficial gains degrades long-term business value
- Lack of guardrails for lead quality or downstream impact in goal-setting frameworks
- Pushing back when organizational pressure creates arbitrary and unrealistic performance targets
- Metrics lose meaning once they become targets
- Patterns, best practices, and the myth of universal UX truths
- Challenge the assumption that common design patterns inherently improve usability
- Intuitiveness can be achieved without adhering to established conventions
- Break expectations to capture attention and differentiate experiences
- Patterns emerge from context rather than universal applicability
- Blindly copying patterns ignores user intent and business constraints
- Context collapse in CRO: why other companies’ test results rarely transfer
- Lack of visibility into why a change was implemented undermines its perceived success
- External factors like leadership decisions or technical constraints influence outcomes
- Criticism of industry case studies that inflate results without statistical rigor
- The importance of situational context over surface-level replication
- The blurred line between data-driven decisions and “vibes” in experimentation
- Many hypotheses originate from intuition rather than hard data
- Honest framing of intuition as a valid but often unacknowledged input
- Preference for transparency about uncertainty over false precision
- Structured testing is what ultimately validates either approach
- Agency versus in-house CRO: scaling execution without losing strategy
- Agencies provide execution bandwidth rather than purely strategic direction
- It’s Important to maintain ownership of experimentation strategy internally
- Collaboration models where agencies augment rather than replace internal thinking
- Stakeholder misinterpretation of agency involvement as strategy outsourcing
- Balancing operational scale with strategic clarity in hybrid models
- Career volatility and the challenge of narrating impact in uncertain environments
- Difficulty explaining strong performance outcomes alongside sudden layoffs
- Crafting coherent narratives around complex, sometimes contradictory experiences
- How to articulate your strategic contributions beyond surface-level metrics
- Emotional and professional impact of abrupt role changes
The Quotes
Standout quips from this week
"Do you want the real answer or crappy nicety?"
"You could just hack through a bunch of stuff and increase your conversion rate… where’s the guardrails for quality?"
"The test directions were stuff I probably would have done anyway."
"I hate black box stuff."
Book Club
Relevant reads recommended this week
No books this week, sorry!
CRO Link Digest
Useful and thought-provoking content shared this week
- Beyond Winning: Spotify’s Experiments with Learning Framework – Spotify Engineering blog article by Michael Bellato, Mårten Schultzberg, and Sebastian Ankargren
- MCP Demystified Part 1: Connecting Chat Interfaces to Real Tools (Starting With a Calculator) – Convert Experiences blog article by Iqbal Ali
- Brooks’s Law – Wikipedia article