If paying by check, please include payment stub with your payment. It’s the CRO Roundtable Roundup!
Thanks to Iqbal Ali, Matt Beischel, Jamie Levrant, Slobodan Manić, Craig Sullivan, and David Swinstead for joining us. Want to get in on the action and talk with other cool CRO people? Then you should probably…
One Sentence Takeaway
I’m a busy person; give me the TL:DR
The integration of AI in experimentation is limited by its inability to predict user behavior accurately.
The Notes
What we discussed this week
- AI in Experimentation
- AI is often misrepresented as a tool to replace developers
- Skepticism about AI’s ability to predict test results
- The concept of AGI is typically exaggerated and misleading; it’s chasing funding
- Challenges with Integrating AI Features into Existing Software
- Hype without delivering practical value
- Vendors promote AI to replace technical roles, which is unrealistic. In the end, someone will always have to do the work.
- AI features should focus on metrics, analysis, and ops; not coding and design
- Running Experiments During Busy Periods
- Misunderstood as being too risky
- Holidays, sales, peaks, etc. are an excellent time to test; it helps you prepare for the next one
- Proper Metrics are Essential for Effective Experimentation
- Experimentation should involve multiple metrics, not just conversion
- Synthetic Users are a Trap
- Synthetic users cannot accurately predict outcomes
- Not a replacement for real user testing
- Can be useful for QA, but not for predicting user behavior
- The idea of predicting test winners with synthetic users is flawed
Hey all you cool GRO People!
GRO Talks Live returns to the winter 2025 Experimentation Elite conference with a heaping helping of in-person roundtable sessions, and we want to see you there!
The Quotes
Standout quips from this week
Book Club
Relevant reads recommended this week
No Book Club this week, sorry!
CRO Link Digest
Useful and thought-provoking content shared this week
- OpenRouter – software platform that provides access to multiple LLMs through a single API
- How to Quit Spotify – Substack article about Spotify’s unethical business practices by Brian Merchant
Off-Topic Sidebars
Experimentation isn’t the only thing we talk about at the CRO Roundtable. There’s often a healthy dose of discussion on shared interests, personal passions, and hobbies.
- Thanksgiving
- Preparing two turkeys and a ham for 8 people
- Leftover turkey recipes
- Black Friday sales, specifically running shoes
Sidebar Shareables
Amusing sidebar content shared this week
No Sidebar Shareables this week, sorry!