Failure is not an option. It’s the CRO Roundtable Roundup!
Thanks to Matt Beischel, Shiva Manjunath, Craig Sullivan, and David Swinstead for joining us. Want to get in on the action and talk with other cool CRO people? Then you should probably…
One Sentence Takeaway
I’m a busy person; give me the TL:DR
Prioritize clear experimentation maturity: emphasize data quality, bias mitigation, and tailored prioritization over one-size scoring.
The Notes
What we discussed this week
- Intent framework for mapping data behaviors
- X-axis: data-informed versus data-driven behaviors
- Y-axis: good intention versus bad intention
- Plot specific behaviors across the framework
- Measurement quality problems caused by perverse incentives
- Example: bonus tied to bounce rate resisted fixes
- Data quality concerns from dirty instrumentation
- Checkbox implementation on tickets harms data
- You should audit and validate tracking data regularly
- Benchmarking experimentation maturity
- Break behaviors into tactical maturity questions
- Tailor maturity model to organizational context
- Integrating behavior scores into maturity models
- Prioritization frameworks, scoring, and voting mechanics
- Comparing frameworks: PIE, ICE, CXL, PXL, and bespoke builds
- Questioning the validity of numeric “probability of winning” estimates
- Use objective scoring whenever possible
- Live team voting tools
- Flicker as a subtractive prioritization factor
- The role of AI in predicting test outcomes
- It is more-or-less impossible to assign true win probability
- Misleading metrics
- Time-on-site is an unreliable engagement metric
- Use a heartbeat/event approach to measure actual focus
- Bounce-rate does not have a standard, technically consistent definition
- Sample-size and metric collection limitations
Hey all you cool GRO People!
GRO Talks Live returns to the winter 2025 Experimentation Elite conference with a heaping helping of in-person roundtable sessions, and we want to see you there!
The Quotes
Standout quips from this week
Book Club
Relevant reads recommended this week
A Mind at Play
CRO Link Digest
Useful and thought-provoking content shared this week
- Beyond Winning: Spotify’s Experiments with Learning Framework – blog article from Spotify R&D
- Measuring U – a mixed-methods research and software firm
- The Trust Thermocline Explains How Companies “Suddenly” Lose Customers And Employees – Medium article by Adam Fisher
- Measure Camp – an open, free-to-attend unconference, different to any other digital analytics conference
- MeasureFest – a conference track dedicated to web analytics, conversion rate optimisation & business intelligence
- Experiment Nation – a Global Community that seeks to give all those who are interested in experimentation from around the world a voice
Off-Topic Sidebars
Experimentation isn’t the only thing we talk about at the CRO Roundtable. There’s often a healthy dose of discussion on shared interests, personal passions, and hobbies.
- Frustration with HOA policies
- Restrictive HOA rules and enforcement practices
- Can they seriously put a lien on your property?
- Tensions between platform moderation and free expression
- Balancing safety, responsibility, and open discussion