Cross posted over @ Medium— Testing is the poker tell of the digital world.
During the online poker boom of the mid 2000’s, I was a professional poker player for nearly three years, living in Las Vegas and grinding out a living playing mostly Omaha Hi/Lo poker online and in person. While I was playing full-time, I read dozens of books and poker magazines, and became a student of the math behind poker as well as the psychology behind people’s motivations. I also wrote in a diary almost every day, and I still have hundreds of scenarios written down in a handful of notebooks that painstakingly analyzed hands and the players at the table. It was as cathartic as it was educational – a way to internalize lessons learned in an attempt to improve my game.
Since I “retired” from poker in mid 2007 and started to focus on digital strategy, I still find myself constantly referencing things I learned from poker, and probably one of the most relevant comparisons to poker is the process to optimize and refine a website through A/B and multivariate testing.
When people first start playing poker, they typically just play with gut instinct – a bunch of red cards means they could catch a flush – high pairs make their heart jump – an inside straight draws look good because they “feel it.” Unfortunately, that’s also how many people design and manage websites – gut instincts. They think slideshows on the homepage look modern – they feel like the sidebar needs more graphics — they think the text of a call to action “just sounds good” or they saw another website do something similar without understanding the context.
With both of these “gut instinct” approaches, some people can get lucky, but luck only lasts so-long, and if you don’t approach poker and website optimization from a data-driven perspective, you’re going to end up missing out on a lot of opportunities and making a lot of mistakes.
Another way to think about digital optimization (A/B, multivariate testing) is that it’s the poker tell of the digital world. It’s the tactic that makes you profitable – it’s the tactic that helps you beat your competitors – and it’s the tactic a sophisticated operator utilizes in order to iteratively improve their performance.
Gut instincts typically rely on too few data points
In poker, every person is unique, and every scenario is unique, so the outcome of one hand shouldn’t robotically dictate your strategy for future hands. Sometimes someone will be drunk, or spending money wildly, or your table image will be lose, and you could end up in a scenario where someone goes all-in on a pair of 2’s and tries to run you down – and then they could get lucky on the river. But if you always assumed that people with a pair of 2’s were going to beat you, you would never play optimally and it would dramatically hurt your opportunities to make money. Essentially, you can’t use one aberrant data point as the basis for your future strategy, but if you were to combine multiple data points, you may get a better sense for why something has occurred and whether or not it will be replicated. Was there something about that player with a pair of 2’s that showed why they were going to go all in? Is there a way to predict that behavior from another player? Could you see similar behavior from someone with any low pair? These hypotheses are the basis of tests that you can conduct to build up multiple data points to make more informed decisions.
For websites, every visitor is unique, each visitor has unique motivations for visiting your website, and every device and technology stack they are using is unique, so you can’t assume that the opinion or feedback from one person will apply universally. Sometimes someone will be willing to use their mobile phone to go through 10 steps to make a purchase, scrolling left and right on a page that is optimized for a desktop, and willing to spend extra minutes to setup and confirm an account via email. But if you used that single mobile user as any sort of decision-making data point, you would be making a big mistake by not seeing the countless other mobile visitors who abandoned your website before taking action. You could also possibly get additional data about that aberrant user – what were they purchasing that made them decide to go through those extra hoops in order to make their purchase? Did that person also select overnight shipping? Were they buying what appeared to be a present for someone else? Could you use that information to build easier “last minute” shopping options? It’s important to not heavily weigh one data point in order to make a site-wide decision, but you should take that one data point and dig deeper in order to test hypotheses about user behavior. Once you start thinking about user motivations, aberrant data, and site-wide trends, you can start to make informed decisions about website optimization. This process is also how you start to dig into mobile and tablet usage, which provides you with additional testing scenarios through segmentation and targeting.
Sometimes the data that is less obvious is more valuable
In poker, the most obvious data comes from people turning over their cards at the end of a hand – but some of the most valuable information can come from seeing how often people throw away their hands and in what circumstances. Does someone typically play a hand “under the gun” in first position? Does someone fold 9/10 hands? How often does player X go in a hand if player Y is already in a hand? The data from hand selection can tell you the baseline of cards someone is willing to play, whether they are loose or tight, and their overall understanding of strategy. Once you start to determine someone’s “abandonment score,” which poker legend Phil Helmuth broke down into a more simple classification of someone’s equivalent “poker animal” then you start to be able to make better decisions about why certain groups of people act the way they do. This can help you break down demographic information much quicker and make more informed decisions.
For websites, a lot of people look at the conversions (signups, sales, shares) and try to determine what kinds of people are coming through their funnel or engaging – but more often than not, you can actually get better information and more effectively optimize your website based on who is abandoning your website or funnel, what platform/tech stacks they are using, how they found your website, the point that they abandoned your process, and a wide range of behavior based on where they clicked and what they were viewing. And when it comes down to it, people who finish the conversion funnel or take an action are very similar to people who make it all the way to the end of a poker hand –there were likely things along the way that encouraged them to follow through – but there were also likely factors outside of your control making that happen. You need to learn why they made it through the funnel because there could be important data points there that you need to understand in order to replicate and increase the likelihood of conversions occurring again in the future, but one of the most important things you can learn is why certain people are not making it through the funnel (aka why someone is folding a hand early) – and then how can you take that information and improve your process to ensure that you increase conversions.
This process to optimize the people who “folded early” is done in large part through an effort known as segmentation and targeting – which is essentially the process of only taking one portion of your website visitors, for instance visitors who came from Facebook and are using a mobile phone or tablet, and then serving them up a specific website variation that is optimized just for them in order to test how that new version affects conversions. That effort of drilling down into segments and targeting specific versions is one of the most important parts of iterative development, and it’s the main reason that most website optimization experts believe that the process to optimize a website is never actually finished – it’s merely a long-term effort that requires someone to segment and target more variations to smaller and smaller subsets of your users. Think of the process like playing poker against the same players over and over again, and building more data points about how they play in unique scenarios, then applying that knowledge to other players.
Poker and website optimization is a game where seemingly small decisions lead to big profits swings
Poker is a game of margins where math reins, but every scenario will remain slightly different and there is always an opportunity to increase your profitability by using psychology and poker tells. In poker, there is a thing called pot odds – where you essentially calculate your chance of making a hand that you think will win a pot with a certain number of cards remaining to be shown, and compare that with the amount of money currently in the pot and the amount of money you’ll have to put in to see the remaining cards. Every good player calculates pot odds throughout a hand, especially when big decisions need to be made, but every great player also realizes that every scenario is different. Perhaps the math would tell you that in 99/100 scenarios you should pay $50 to possibly win $500 — but let’s say that in one specific scenario, your opponent is doing what we call “poker clack” – smacking their lips or doing something out of the ordinary (like eating Oreos) – and you decide to fold your hand to the “made hand” in order to save a little money. That type of specific “I know your tell” scenario is the ideal situation for a website optimization expert – if you can get to a point where you’ve optimized and refined your process to a science and are able to make a minor tweak in order to squeeze just a little bit more profitability, or save a little money, from each transaction, at the end of the year, you’re going to see huge gains.
When optimizing a website, many organizations end up seeing huge gains just from minor changes to language, imagery, processes or value propositions. Sometimes after an A/B test, an organization may realize that they need to completely restructure a web page by dramatically simplifying it, or that they need to make big changes to a sales or signup funnel, but a lot of the time, you’ll be testing small changes in order to squeeze out a small increase in the percentage of converted visitors. Let’s say for instance that you have 5,000 people purchasing a product that costs $25 on your website over a month, out of a total of 250,000 visitors. Let’s say you increased your conversion rate from its current 2% to 2.4% — that would increase your monthly sales from $125,000 to $150,000. This type of dramatic increase in profits wouldn’t be possible if you didn’t’ think about how that small $25 sale could be scaled through an optimized conversion of your entire user base, much the same way that someone may not realize that paying that extra $50 to see someone turn over cards you already knew they had would end up costing huge amounts of money over the long run.
Both poker and website optimization require lots of tests – and it should be fun!
When you’re sitting at a poker table or across the table at a boardroom discussing a website, the process shouldn’t be like pulling teeth. If you want to test something, like bluffing on the river when a blank hits, or if you want to test changing the color of a button, do it! You should approach both poker and website optimization with the mentality of “you should test that!” But the important next step is to learn from your mistakes, and to analyze the data that comes from your tests.
Another important lesson to learn from both poker and website optimization is that everyone is fallible – you may think that a certain bet at the poker table is smart, and end up losing your entire stack. Or you may think that changing the header on a signup page will increase conversions, and end up cutting conversions in half. But the important thing to remember is that you shouldn’t feel bad about conducting a test as long as you learn from it!
I would be remiss to note that there is another thing to consider when testing both at the poker table and with websites – don’t put all your eggs in one basket. You can bluff that river when a blank hits, but you may want to think twice before going all in on that bluff, and instead just bet a healthy chunk of your stack. Similarly, in website optimization, you should rarely test 100% of your traffic on the new variations – you may direct only 15-30% of your traffic to the tested variations and keep the remainder of the traffic going to the original version. This ensures that if you have a hugely unsuccessful test, you aren’t going to dramatically hurt the conversion performance of your website.
Finally, if you’ve never played poker or considered website optimization, there is no time like the present to dig in and just give it the old college try. Neither is as confusing or scary as your competitors would like you to think, and you might end up walking away with a much thicker billfold because of it.
Want to learn more about poker or website optimization?
Read more about testing and get some ideas for website optimization by checking out Dan Siroker and Pete Koomen’s new book A/B Testing: The Most Powerful Way to Turn Clicks Into Customers. Another great read is Chris Goward’s “You Should Test That!” or if you’re looking for a quicker synopsis on the benefits of website optimization and testing, check out Brian Christian’s April 2012 Wired article, “The A/B Test: Inside the Technology That’s Changing the Rules of Business.”
If you are just getting started in poker, you can’t go wrong with Super System by Doyle Brunson, Beat Texas Hold’Em by Tom McEvoy and Shane Smith, Hold’em Poker: for Advanced Players by David Sklansky and Mason Malmuth, High-Low Split Poker by Ray Zee, or of course the always classic Mike Caro poker tells videos.