Failure is not an option. It’s the CRO Roundtable Roundup!
Thanks to Matt Beischel, Shiva Manjunath, Craig Sullivan, and David Swinstead for joining us. Want to get in on the action and talk with other cool CRO people? Then you should probably…
One Sentence Takeaway
I’m a busy person; give me the TL:DR
Prioritize clear experimentation maturity: emphasize data quality, bias mitigation, and tailored prioritization over one-size scoring.
The Notes
What we discussed this week
- Intent framework for mapping data behaviors
- X-axis: data-informed versus data-driven behaviors
- Y-axis: good intention versus bad intention
- Plot specific behaviors across the framework
- Measurement quality problems caused by perverse incentives
- Example: bonus tied to bounce rate resisted fixes
- Data quality concerns from dirty instrumentation
- Checkbox implementation on tickets harms data
- You should audit and validate tracking data regularly
- Benchmarking experimentation maturity
- Break behaviors into tactical maturity questions
- Tailor maturity model to organizational context
- Integrating behavior scores into maturity models
- Prioritization frameworks, scoring, and voting mechanics
- Comparing frameworks: PIE, ICE, CXL, PXL, and bespoke builds
- Questioning the validity of numeric “probability of winning” estimates
- Use objective scoring whenever possible
- Live team voting tools
- Flicker as a subtractive prioritization factor
- The role of AI in predicting test outcomes
- It is more-or-less impossible to assign true win probability
- Misleading metrics
- Time-on-site is an unreliable engagement metric
- Use a heartbeat/event approach to measure actual focus
- Bounce-rate does not have a standard, technically consistent definition
- Sample-size and metric collection limitations
The Quotes
Standout quips from this week
“You could be data driven and be bad intentioned, which means there are bad things you could do.”
“My favorite one was some guys that wouldn't let me fix Google Analytics because their bonus was tied to the bounce rate on the site.”
"If you don't like your job, get a new one."
"Maturity models only work if they’re built for your actual organization, not someone else’s template."
Book Club
Relevant reads recommended this week
A Mind at Play
The story of Claude Shannon—one of the foremost intellects of the twentieth century and the architect of the Information Age, whose insights stand behind every computer built, email sent, video streamed, and webpage loaded.
CRO Link Digest
Useful and thought-provoking content shared this week
- Beyond Winning: Spotify’s Experiments with Learning Framework – blog article from Spotify R&D
- Measuring U – a mixed-methods research and software firm
- The Trust Thermocline Explains How Companies “Suddenly” Lose Customers And Employees – Medium article by Adam Fisher
- Measure Camp – an open, free-to-attend unconference, different to any other digital analytics conference
- MeasureFest – a conference track dedicated to web analytics, conversion rate optimisation & business intelligence
- Experiment Nation – a Global Community that seeks to give all those who are interested in experimentation from around the world a voice
Off-Topic Sidebars
Experimentation isn’t the only thing we talk about at the CRO Roundtable. There’s often a healthy dose of discussion on shared interests, personal passions, and hobbies.
- Frustration with HOA policies
- Restrictive HOA rules and enforcement practices
- Can they seriously put a lien on your property?
- Tensions between platform moderation and free expression
- Balancing safety, responsibility, and open discussion