{"id":9033,"date":"2017-12-08T17:07:50","date_gmt":"2017-12-08T17:07:50","guid":{"rendered":"http:\/\/intercom.com\/blog\/?p=9033"},"modified":"2020-07-30T12:57:06","modified_gmt":"2020-07-30T11:57:06","slug":"why-ab-tests-should-yield-more-than-results","status":"publish","type":"post","link":"https:\/\/www.intercom.com\/blog\/why-ab-tests-should-yield-more-than-results\/","title":{"rendered":"The golden rule of A\/B testing: look beyond validation"},"content":{"rendered":"<p class=\"opening_paragraph\">A\/B tests provide more than statistical validation of one execution over another. They can and should impact how your team prioritizes projects.<\/p>\n<p>Too often, teams use A\/B testing to validate bad ideas. <a href=\"https:\/\/www.intercom.com\/blog\/scale\/\" target=\"_blank\" rel=\"noopener noreferrer\">They make minor changes and hope the test will produce big wins.<\/a> But these tests can be counterproductive. Results that are a product of random variation (e.g. not statistically significant) yield unhelpful insights, and even good results are not guaranteed to hold true when your winning variable is shipped to a full audience.<\/p>\n<p>If you adhere to A\/B testing best practices and ask the right questions before running a test, you\u2019ll learn what types of changes are actually worth your time and focus on projects that produce meaningful insights.<\/p>\n<h2 id=\"the-impact-of-statistical-power\">The impact of statistical power<\/h2>\n<p>There\u2019s a lot <a href=\"https:\/\/blog.kissmetrics.com\/your-ab-tests-are-illusory\/\">written<\/a> on frequent A\/B testing mistakes. The most common error is calling tests too soon based on statistical significance. Sometimes your test result is significant because there\u2019s an actual effect, other times it\u2019s due to sheer noise. After all, a random sample is never going to be a perfect representation of the full population.<\/p>\n<p>In order to differentiate an effect from noise, you not only need statistical <em>significance<\/em> but also <a href=\"https:\/\/en.wikipedia.org\/wiki\/Statistical_power\">statistical <em>power<\/em><\/a>. If you have more statistical power you can be more certain the lift is actually real.<\/p>\n<p>To get enough power and run a test correctly, ask yourself:<\/p>\n<ol>\n<li>How much do you think the change will increase the associated key performance indicator (KPI)?<\/li>\n<li>Given this desired effect, how long will you need to run the test to get accurate results?<\/li>\n<li>Is it worth the wait?<\/li>\n<\/ol>\n<h3>1. How much do you think the change will increase the KPI?<\/h3>\n<p>Say you want to improve your signup flow. You have a list of ideas and are trying to decide which ones to work on. You\u2019re not happy with the UI, but a complete overhaul will take a month to design and build. On the other hand, you could try out a different color scheme, which won\u2019t take long to change. Your team hypothesizes that the complete redesign can boost conversion from 10% to 15% whereas the color scheme change may boost conversion from 10% to 11%.<\/p>\n<p><img decoding=\"async\" class=\"small\" src=\"https:\/\/intercom.com\/blog\/wp-content\/uploads\/2016\/05\/A-B_Testing_chart1.png\" alt=\"\" \/><\/p>\n<h3>2. Given this desired effect, how long will you need to run the test to get accurate results?<\/h3>\n<p>Take your answer to the question above and plug it into <a href=\"http:\/\/www.evanmiller.org\/ab-testing\/sample-size.html\">this sample size calculator<\/a>. Under the hood there\u2019s serious <a href=\"http:\/\/sphweb.bumc.bu.edu\/otlt\/MPH-Modules\/BS\/BS704_Power\/BS704_Power5.html\">statistics<\/a> involved, but the basic logic is smaller effects take longer to detect whereas larger effects will be obvious sooner. This is an important insight: detecting small changes is expensive. In our example, the color scheme change only takes two days to build, but we\u2019ll need 24x more data to test a 1% versus a 5% absolute lift.<\/p>\n<p><img decoding=\"async\" class=\"small\" src=\"https:\/\/intercom.com\/blog\/wp-content\/uploads\/2016\/05\/A-B_Testing_chart2.png\" alt=\"\" \/><\/p>\n<h3>3. Is it worth the wait?<\/h3>\n<p>With sample sizes in mind, you should consider how long a test would need to run to achieve reliable results. Say your signup flow gets 600 visits per day. The complete redesign will require two days to gather enough data while the color scheme change will take much longer. So the larger project takes 32 days to develop and test, while the smaller project takes 49. They both take a lot of time, but the complete redesign has more potential.<\/p>\n<p><img decoding=\"async\" class=\"small\" src=\"https:\/\/intercom.com\/blog\/wp-content\/uploads\/2016\/05\/A-B_Testing_chart3.png\" alt=\"\" \/><\/p>\n<h2 id=\"focus-on-projects-with-bigger-potential-upside\">Focus on projects with bigger potential upside<\/h2>\n<p>Late last year, we thought our signup flow could do better. The layout on our previous signup page (shown below as the Control) was not logically organized. The integration methods were not displayed in an order that was most likely to be relevant to new users. And since our signup flow doesn\u2019t see hundreds of thousands of visitors, we knew we had to test something that we thought was <em>much<\/em> better.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/intercom.com\/blog\/wp-content\/uploads\/2016\/05\/A-B_Testing_signuptest.png\" alt=\"\" \/><br \/>\nWe wanted our signup conversion to increase by 10%. And after getting enough data, we looked at the signup rates between versions. Disappointingly, the difference was small and not statistically significant. The test was a wash, but that\u2019s okay. We still decided to use the new version, because the signup experience was cleaner and put us in a better position to iterate and improve in the future.<\/p>\n<h2 id=\"what-this-approach-means-for-startups\">What this approach means for startups<\/h2>\n<p>For young companies, small optimization projects just don\u2019t make sense. Tests take a long time to run and distract you from working on projects that matter. You might see a slight uptick, but you\u2019re likely working towards a local maximum. To get on the path towards the global maximum, young companies need to make big changes.<\/p>\n<p><img decoding=\"async\" class=\"small\" src=\"https:\/\/intercom.com\/blog\/wp-content\/uploads\/2016\/05\/A-B_Testing_convmax.png\" alt=\"\" \/><\/p>\n<p>Larger companies have spent a lot of time working on flows and understanding their customers. They\u2019re better suited for small improvements because they have the traffic to conclude tests faster. Plus, a 1% improvement means a lot more when you have hundreds of thousands of visitors a day.<\/p>\n<p>Once you\u2019ve made several large tests, have a good understanding of your customers and have large traffic volume, you can work on small optimizations. Remember, not every A\/B test is going to yield the result you want. What\u2019s important is that you determine your improvement goal and account for how long you need to run the tests to get the right kind of results. Otherwise you risk spending a lot of resources only to get back a negative ROI.<\/p>\n\n","protected":false},"excerpt":{"rendered":"<p>A\/B tests provide more than statistical validation of one execution over another. They can and should impact how your team prioritizes projects. Too often, teams use A\/B testing to validate bad ideas. They make minor changes&hellip;<\/p>\n","protected":false},"author":185,"featured_media":14070,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"category":[5],"tags":[149,161,144],"coauthors":[365],"class_list":["post-9033","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-product-and-design","tag-growth","tag-product-development","tag-research"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.3 (Yoast SEO v27.3) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>How A\/B Testing Can Help Your Team Prioritize, Not Just Validate<\/title>\n<meta name=\"description\" content=\"A\/B tests provide more than statistical validation of one execution over another. They can and should impact how your team prioritizes projects.\u00a0\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.intercom.com\/blog\/why-ab-tests-should-yield-more-than-results\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"The golden rule of A\/B testing: look beyond validation\" \/>\n<meta property=\"og:description\" content=\"A\/B tests provide more than statistical validation of one execution over another. They can and should impact how your team prioritizes projects.\u00a0\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.intercom.com\/blog\/why-ab-tests-should-yield-more-than-results\/\" \/>\n<meta property=\"og:site_name\" content=\"The Intercom Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/intercominc\" \/>\n<meta property=\"article:published_time\" content=\"2017-12-08T17:07:50+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2020-07-30T11:57:06+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.intercom.com\/blog\/wp-content\/uploads\/2017\/12\/AB_Tests_That_Really_Validate.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1968\" \/>\n\t<meta property=\"og:image:height\" content=\"932\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Scott Kamino\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@scottkamino\" \/>\n<meta name=\"twitter:site\" content=\"@intercom\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Scott Kamino\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/why-ab-tests-should-yield-more-than-results\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/why-ab-tests-should-yield-more-than-results\\\/\"},\"author\":{\"name\":\"Scott Kamino\",\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/#\\\/schema\\\/person\\\/cef528bbedf4f7b72687aede0ca763a8\"},\"headline\":\"The golden rule of A\\\/B testing: look beyond validation\",\"datePublished\":\"2017-12-08T17:07:50+00:00\",\"dateModified\":\"2020-07-30T11:57:06+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/why-ab-tests-should-yield-more-than-results\\\/\"},\"wordCount\":898,\"publisher\":{\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/why-ab-tests-should-yield-more-than-results\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/wp-content\\\/uploads\\\/2017\\\/12\\\/AB_Tests_That_Really_Validate.jpg\",\"keywords\":[\"growth\",\"product development\",\"research\"],\"articleSection\":[\"Product &amp; Design\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/why-ab-tests-should-yield-more-than-results\\\/\",\"url\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/why-ab-tests-should-yield-more-than-results\\\/\",\"name\":\"How A\\\/B Testing Can Help Your Team Prioritize, Not Just Validate\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/why-ab-tests-should-yield-more-than-results\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/why-ab-tests-should-yield-more-than-results\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/wp-content\\\/uploads\\\/2017\\\/12\\\/AB_Tests_That_Really_Validate.jpg\",\"datePublished\":\"2017-12-08T17:07:50+00:00\",\"dateModified\":\"2020-07-30T11:57:06+00:00\",\"description\":\"A\\\/B tests provide more than statistical validation of one execution over another. They can and should impact how your team prioritizes projects.\u00a0\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.intercom.com\\\/blog\\\/why-ab-tests-should-yield-more-than-results\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/why-ab-tests-should-yield-more-than-results\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/wp-content\\\/uploads\\\/2017\\\/12\\\/AB_Tests_That_Really_Validate.jpg\",\"contentUrl\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/wp-content\\\/uploads\\\/2017\\\/12\\\/AB_Tests_That_Really_Validate.jpg\",\"width\":1968,\"height\":932},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/\",\"name\":\"The Intercom Blog\",\"description\":\"Articles and Podcasts on Customer Service, AI and Automation, Product, and more\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/#organization\",\"name\":\"The Intercom Blog\",\"url\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/08\\\/Intercom-logo-sq-black-trans.png\",\"contentUrl\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/08\\\/Intercom-logo-sq-black-trans.png\",\"width\":1000,\"height\":1000,\"caption\":\"The Intercom Blog\"},\"image\":{\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/intercominc\",\"https:\\\/\\\/x.com\\\/intercom\",\"https:\\\/\\\/www.instagram.com\\\/intercom\\\/\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/2491343\",\"https:\\\/\\\/www.pinterest.ie\\\/intercom\\\/\",\"https:\\\/\\\/www.youtube.com\\\/channel\\\/UCJG0MvLP03kyzzAkD-w98aQ\",\"https:\\\/\\\/en.wikipedia.org\\\/wiki\\\/Intercom_(company)\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/#\\\/schema\\\/person\\\/cef528bbedf4f7b72687aede0ca763a8\",\"name\":\"Scott Kamino\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/40a16e903b0912053c17fbb058ad0595e4c0bffb51b2931e515531183097ae25?s=96&d=mm&r=pga236738adea91753bf31a51f0268845d\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/40a16e903b0912053c17fbb058ad0595e4c0bffb51b2931e515531183097ae25?s=96&d=mm&r=pg\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/40a16e903b0912053c17fbb058ad0595e4c0bffb51b2931e515531183097ae25?s=96&d=mm&r=pg\",\"caption\":\"Scott Kamino\"},\"description\":\"Data Analyst (Growth) at Intercom\",\"sameAs\":[\"https:\\\/\\\/x.com\\\/scottkamino\"],\"url\":\"https:\\\/\\\/www.intercom.com\\\/blog\\\/author\\\/scottkamino\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"How A\/B Testing Can Help Your Team Prioritize, Not Just Validate","description":"A\/B tests provide more than statistical validation of one execution over another. They can and should impact how your team prioritizes projects.\u00a0","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.intercom.com\/blog\/why-ab-tests-should-yield-more-than-results\/","og_locale":"en_US","og_type":"article","og_title":"The golden rule of A\/B testing: look beyond validation","og_description":"A\/B tests provide more than statistical validation of one execution over another. They can and should impact how your team prioritizes projects.\u00a0","og_url":"https:\/\/www.intercom.com\/blog\/why-ab-tests-should-yield-more-than-results\/","og_site_name":"The Intercom Blog","article_publisher":"https:\/\/www.facebook.com\/intercominc","article_published_time":"2017-12-08T17:07:50+00:00","article_modified_time":"2020-07-30T11:57:06+00:00","og_image":[{"width":1968,"height":932,"url":"https:\/\/www.intercom.com\/blog\/wp-content\/uploads\/2017\/12\/AB_Tests_That_Really_Validate.jpg","type":"image\/jpeg"}],"author":"Scott Kamino","twitter_card":"summary_large_image","twitter_creator":"@scottkamino","twitter_site":"@intercom","twitter_misc":{"Written by":"Scott Kamino","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.intercom.com\/blog\/why-ab-tests-should-yield-more-than-results\/#article","isPartOf":{"@id":"https:\/\/www.intercom.com\/blog\/why-ab-tests-should-yield-more-than-results\/"},"author":{"name":"Scott Kamino","@id":"https:\/\/www.intercom.com\/blog\/#\/schema\/person\/cef528bbedf4f7b72687aede0ca763a8"},"headline":"The golden rule of A\/B testing: look beyond validation","datePublished":"2017-12-08T17:07:50+00:00","dateModified":"2020-07-30T11:57:06+00:00","mainEntityOfPage":{"@id":"https:\/\/www.intercom.com\/blog\/why-ab-tests-should-yield-more-than-results\/"},"wordCount":898,"publisher":{"@id":"https:\/\/www.intercom.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.intercom.com\/blog\/why-ab-tests-should-yield-more-than-results\/#primaryimage"},"thumbnailUrl":"https:\/\/www.intercom.com\/blog\/wp-content\/uploads\/2017\/12\/AB_Tests_That_Really_Validate.jpg","keywords":["growth","product development","research"],"articleSection":["Product &amp; Design"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.intercom.com\/blog\/why-ab-tests-should-yield-more-than-results\/","url":"https:\/\/www.intercom.com\/blog\/why-ab-tests-should-yield-more-than-results\/","name":"How A\/B Testing Can Help Your Team Prioritize, Not Just Validate","isPartOf":{"@id":"https:\/\/www.intercom.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.intercom.com\/blog\/why-ab-tests-should-yield-more-than-results\/#primaryimage"},"image":{"@id":"https:\/\/www.intercom.com\/blog\/why-ab-tests-should-yield-more-than-results\/#primaryimage"},"thumbnailUrl":"https:\/\/www.intercom.com\/blog\/wp-content\/uploads\/2017\/12\/AB_Tests_That_Really_Validate.jpg","datePublished":"2017-12-08T17:07:50+00:00","dateModified":"2020-07-30T11:57:06+00:00","description":"A\/B tests provide more than statistical validation of one execution over another. They can and should impact how your team prioritizes projects.\u00a0","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.intercom.com\/blog\/why-ab-tests-should-yield-more-than-results\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.intercom.com\/blog\/why-ab-tests-should-yield-more-than-results\/#primaryimage","url":"https:\/\/www.intercom.com\/blog\/wp-content\/uploads\/2017\/12\/AB_Tests_That_Really_Validate.jpg","contentUrl":"https:\/\/www.intercom.com\/blog\/wp-content\/uploads\/2017\/12\/AB_Tests_That_Really_Validate.jpg","width":1968,"height":932},{"@type":"WebSite","@id":"https:\/\/www.intercom.com\/blog\/#website","url":"https:\/\/www.intercom.com\/blog\/","name":"The Intercom Blog","description":"Articles and Podcasts on Customer Service, AI and Automation, Product, and more","publisher":{"@id":"https:\/\/www.intercom.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.intercom.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.intercom.com\/blog\/#organization","name":"The Intercom Blog","url":"https:\/\/www.intercom.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.intercom.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.intercom.com\/blog\/wp-content\/uploads\/2019\/08\/Intercom-logo-sq-black-trans.png","contentUrl":"https:\/\/www.intercom.com\/blog\/wp-content\/uploads\/2019\/08\/Intercom-logo-sq-black-trans.png","width":1000,"height":1000,"caption":"The Intercom Blog"},"image":{"@id":"https:\/\/www.intercom.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/intercominc","https:\/\/x.com\/intercom","https:\/\/www.instagram.com\/intercom\/","https:\/\/www.linkedin.com\/company\/2491343","https:\/\/www.pinterest.ie\/intercom\/","https:\/\/www.youtube.com\/channel\/UCJG0MvLP03kyzzAkD-w98aQ","https:\/\/en.wikipedia.org\/wiki\/Intercom_(company)"]},{"@type":"Person","@id":"https:\/\/www.intercom.com\/blog\/#\/schema\/person\/cef528bbedf4f7b72687aede0ca763a8","name":"Scott Kamino","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/40a16e903b0912053c17fbb058ad0595e4c0bffb51b2931e515531183097ae25?s=96&d=mm&r=pga236738adea91753bf31a51f0268845d","url":"https:\/\/secure.gravatar.com\/avatar\/40a16e903b0912053c17fbb058ad0595e4c0bffb51b2931e515531183097ae25?s=96&d=mm&r=pg","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/40a16e903b0912053c17fbb058ad0595e4c0bffb51b2931e515531183097ae25?s=96&d=mm&r=pg","caption":"Scott Kamino"},"description":"Data Analyst (Growth) at Intercom","sameAs":["https:\/\/x.com\/scottkamino"],"url":"https:\/\/www.intercom.com\/blog\/author\/scottkamino\/"}]}},"jetpack_featured_media_url":"https:\/\/www.intercom.com\/blog\/wp-content\/uploads\/2017\/12\/AB_Tests_That_Really_Validate.jpg","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.intercom.com\/blog\/wp-json\/wp\/v2\/posts\/9033","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.intercom.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.intercom.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.intercom.com\/blog\/wp-json\/wp\/v2\/users\/185"}],"replies":[{"embeddable":true,"href":"https:\/\/www.intercom.com\/blog\/wp-json\/wp\/v2\/comments?post=9033"}],"version-history":[{"count":0,"href":"https:\/\/www.intercom.com\/blog\/wp-json\/wp\/v2\/posts\/9033\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.intercom.com\/blog\/wp-json\/wp\/v2\/media\/14070"}],"wp:attachment":[{"href":"https:\/\/www.intercom.com\/blog\/wp-json\/wp\/v2\/media?parent=9033"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.intercom.com\/blog\/wp-json\/wp\/v2\/category?post=9033"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.intercom.com\/blog\/wp-json\/wp\/v2\/tags?post=9033"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.intercom.com\/blog\/wp-json\/wp\/v2\/coauthors?post=9033"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}