High octane special. It’s the CRO Roundtable Roundup!
Thanks to Iqbal Ali, Matt Beischel, Slobodan Manić, Surjit Panda, and Gnanagurubaran Venkatesvaran for joining us. Want to get in on the action and talk with other cool CRO people? Then you should probably…
One Sentence Takeaway
I’m a busy person; give me the TL:DR
Prioritize experimentation quality over quantity by aligning tests with real business problems, clean data structure, and long-term value.
The Notes
What we discussed this week
- Data Visualization in Experimentation Platforms
- Building Airtable interfaces for experimentation dashboards
- Challenges with AI builder limitations and interface constraints
- Usefulness of multi-line graphs showing test status over time
- Public vs. client-only access to experimentation dashboards
- Interface visualization tailored for multiple audience types
- Exploring external graph libraries and custom dashboards
- Avoiding dashboards no one uses: build with purpose
- The Shift from CRO to GRO
- Rebranding “CRO Talks” to “GRO Talks” for broader appeal
- CRO as a term hijacked by hacky or superficial practices
- GRO = Growth, Research, and Optimization
- Avoiding the “quick win” mindset in experimentation
- AI Agents and Web Accessibility
- Only 30% success rate for most AI agents
- Major failure point: web browsing with OWL due to poor HTML structure
- Importance of semantic HTML and accessibility for AI compatibility
- Using clear headings, navigation roles, and not relying on
<div>s - Impact of duplicate IDs and poor markup on JS and CSS
- Accessibility benefits extend to screen readers and crawlers
- Experimentation Bloat & Program Metrics
- Setting quantity-of-output metrics can lead to poor test quality
- Perverse incentivization: splitting a single test into desktop/mobile to pad numbers
- Define metrics like “decisions made” or “value for effort”
- Use pre-mortems to define potential failure paths and metrics
- Measure experimentation value quarterly using lagging indicators
- Track impact on business metrics like revenue or CSAT
- What Makes a “Bullshit Test”
- No real hypothesis or business case
- Bundling defect fixes with experimental changes
- Lack of clear outcome measurement or segmentation
- Poor test design that fails to isolate variables
- Metrics should be tied to current program problems
- Testing needs formal training and community standards
- Designing for User Problems, Not Just Business Goals
- Prioritize and categorize problems, not just solution ideas
- Build test libraries around key user problem statements
- Meta-analysis on problem-based testing cohorts
- Beware of hidden UX drivers during redesigns
- Redesign is often for internal satisfaction, not users
- Website Redesigns and Testing Strategies
- Redesigns typically cause performance dips
- Radical visual changes confuse users, gradual changes to design elements are safer
- Testing should be phased or through MVPs
- Avoid top-down ego-driven changes from leadership
The Quotes
Standout quips from this week
“A new coat of paint doesn’t fix a rotting foundation.”
“CRO is dead; long live GRO Talks.”
“Redesigns are usually driven by ego or boredom, not user needs.”
“There are no bad questions.”
Book Club
Relevant reads recommended this week
- No books this week, sorry!
CRO Link Digest
Useful and thought-provoking content shared this week
- LingsCars.com – R.I.P. to the OG “website with terrible UI”
- OWL – GitHub readme for OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation
Off-Topic Sidebars
Experimentation isn’t the only thing we talk about at the CRO Roundtable. There’s often a healthy dose of discussion on shared interests, personal passions, and hobbies.
- Weather & Life in the UK versus US
- UK heatwave and lack of air conditioning
- Differences in Fahrenheit vs. Celsius
- Distances between US cities compared to European cities
- Video Game Systems Design
- World of Warcraft talent system redesigns each expansion
- Player attrition due to complex system overhauls
- When bugs become features
- Travel & International Conferences
- Visa issues for travel to UK and US
- Making plans for Costa Rica and London trips
- Surprise at seeing Spanish and Chinese in US cities
Sidebar Shareables
Amusing sidebar content shared this week
- Air Conditioning in the USA: More than Just a Luxury – graphs comparing death rates in major US and EU cities during heatwaves
- Bunnyhopping from the Programmer’s Perspective – blog article about the technical origins of “bunnyhopping” in FPS games, from Adrian Biagioli
- Skiing – wiki article explaining the origins of and process for the skiing movement method in the Tribes video game series
- The early versions of counter-strike had a famous bunny-hopping bug – forum thread at Ycombinator