{"id":2807,"date":"2015-06-04t17:29:57","date_gmt":"2015-06-04t17:29:57","guid":{"rendered":"\/\/www.catharsisit.com\/?p=2807"},"modified":"2015-06-04t17:29:57","modified_gmt":"2015-06-04t17:29:57","slug":"optimized-student-happiness-using-ab-testing","status":"publish","type":"post","link":"\/\/www.catharsisit.com\/blog\/optimized-student-happiness-using-ab-testing\/","title":{"rendered":"how we optimized for student happiness using a\/b testing"},"content":{"rendered":"

magoosh is all about making sure our students are well educated and happy<\/a>. \u00a0but we\u2019re also a data-driven business that uses metrics to make decisions — vague notions of happiness are nice, but we want numbers!<\/p>\n

so this is the story of how we improved student happiness by a\/b testing changes to our product with the goal not of optimizing clicks or conversions or revenues, but of maximizing student happiness. \u00a0to start, though, i\u2019ll introduce the metric at hand: net promoter score.<\/p>\n

nps: our reliable referral indicator<\/b><\/h2>\n

net promoter score<\/a> is a metric that tells you, on the whole, how willing your customers are to promote your product. \u00a0customers are asked on a scale of 1-10 how likely they would be to recommend your product; 9s and 10s are considered \u201cpromoters\u201d, 7s and 8s are neutral, and anything below 6 is a \u201cdetractor.\u201d \u00a0your net promoter score is calculated by subtracting the number of detractors from the number of promoters and dividing by the total number of respondents. \u00a0as a result, nps is a percentage somewhere in the range of -100% (all detractors) to 100% (all promoters). \u00a0not to brag, but our nps is high. \u00a0really high. \u00a0apple high<\/a>.<\/p>\n

at magoosh, nps is one of the most important metrics we track — it helps us determine not only whether students like our customer service and user interface, but also how well our products prepare students for their exams. \u00a0and most importantly it has been a reliable leading indicator of growth in word-of-mouth referrals — our largest marketing channel. \u00a0when nps is high, students talk about magoosh and more people buy it!<\/p>\n

\"screen<\/a><\/p>\n

historically, we\u2019ve asked students the nps question after<\/i> they\u2019ve taken their exams (and, importantly, seen their final scores). \u00a0we do this because our products prepare students for tests, and, really, the proof is in the pudding. \u00a0you can\u2019t fully decide if you\u2019re willing to recommend magoosh for gre prep until you\u2019ve taken the real gre. \u00a0the downside is that it can take a while for us to see nps change in response to product changes. \u00a0since we\u2019re waiting until after students are done studying to survey them, it can take months between when a student sees a new feature and when she rates our product.<\/p>\n

our nps issue: mismatched expectations<\/b><\/h2>\n

because nps is such an important metric to our company, we take changes very seriously. \u00a0earlier this year we saw nps for our gmat product dip fairly significantly. \u00a0looking into why, we discovered that several passive and detractor students were complaining that they were getting lower scores on their real gmat than they did on their magoosh practice tests.<\/p>\n

\"screen<\/a><\/p>\n

our algorithm was telling students to expect one score, but, for some, their official reports were coming back lower — obviously a frustrating experience. \u00a0these students were still improving their scores significantly, but once you\u2019ve got a 750 in your mind, a 700 seems disappointing! \u00a0we determined that we needed to fix our score prediction algorithm to be more accurate, but we were left with a major concern: would an improved algorithm that displayed a lower predicted score be demoralizing for students? \u00a0which was worse for customer satisfaction — a lower predicted score while studying, or a disappointing final score after the exam?<\/p>\n

the challenge: could we optimize quickly for nps?<\/b><\/h2>\n

normally when we have questions about what works best for conversion or marketing, we run a quick a\/b test to determine what works best. \u00a0but nps was different — we\u2019d never a\/b tested for nps optimization before, and our nps collection survey only went to students after their exams. \u00a0it would be months before students who saw the changed algorithm took their exams and we got back nps data. \u00a0making a significant change without knowing how it would affect our word of mouth marketing was a big risk.<\/p>\n

our solution: \u00a0bring nps inside our product<\/b><\/h2>\n

we determined that in order to a\/b test the algorithm change, we needed a method for collecting nps data while students were still studying — not just waiting til the end of their exam. \u00a0we began using a third-party tool called wootric<\/a>, which allows us to ask the nps question in our product and analyze the data in real-time. \u00a0we then deployed the changed algorithm to half of our gmat students, and we could then match the \u201clikely-to-refer\u201d rating to students in the treatment and control groups. \u00a0suddenly nps had a new use case for us — as a powerful, agile product tool.<\/p>\n

\"screen<\/a><\/p>\n

it turned out that the improved algorithm did not affect student satisfaction while studying with magoosh — nps from both student groups was identical. \u00a0knowing this allowed us to roll the change out to all students more quickly. \u00a0we were also able to track the students in the a\/b test over time, and have seen that post-exam nps for students in the treatment group is a full nine points higher<\/b> than for the control.<\/p>\n

takeaways from a\/b testing for nps<\/b><\/h2>\n

1) \u00a0include current customers in your optimizable funnel<\/b><\/p>\n

our goal is always to provide our students with the best possible test prep experience. \u00a0but since we\u2019re not able to read minds, it\u2019s not always easy to know if what we\u2019re doing is actually providing a great experience. \u00a0it\u2019s easy to think of customer acquisition as a funnel, and to wrap our brains around how to a\/b test to optimize that funnel. \u00a0but what doesn\u2019t come easily (at least for most startups — and definitely not for magoosh, at first) is to think of current customers as part of an optimizable funnel too.<\/p>\n

2) \u00a0optimize your products for referrals<\/b><\/p>\n

if your business is built on recommendations and word-of-mouth, then you really can\u2019t afford not to optimize your products for referrals. \u00a0this process has helped us make sure that what we\u2019re doing is making a meaningful difference for students, and has provided us with a useful and repeatable framework for testing future features and products.<\/p>\n

3) \u00a0focus on agility<\/b><\/p>\n

shift your thinking on nps from a one-time transactional model to an ongoing and contextual model. \u00a0in-product nps tools available today like wootric can help you do this easily, as well as keep track of your a\/b test groups. \u00a0you can speed up decision making and keep your pulse on customer happiness.<\/p>\n

 <\/p>\n

read about it on wootric’s blog too!\u00a0a\/b testing to optimize net promoter score at magoosh<\/a><\/p>\n

 <\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

it isn’t just luck. our student success is based on a\/b testing as well as quality and engaging products.<\/p>\n","protected":false},"author":77,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[],"ppma_author":[503],"acf":[],"yoast_head":"\nhow we optimized for student happiness using a\/b testing<\/title>\n<meta name=\"description\" content=\"how we ran an experiment with nps tool wootric to maximize magoosh student happiness!\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"\/\/www.catharsisit.com\/blog\/optimized-student-happiness-using-ab-testing\/\" \/>\n<meta property=\"og:locale\" content=\"en_us\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"how we optimized for student happiness using a\/b testing\" \/>\n<meta property=\"og:description\" content=\"how we ran an experiment with nps tool wootric to maximize magoosh student happiness!\" \/>\n<meta property=\"og:url\" content=\"\/\/www.catharsisit.com\/blog\/optimized-student-happiness-using-ab-testing\/\" \/>\n<meta property=\"og:site_name\" content=\"magoosh 2022年足球世界杯举办地\" \/>\n<meta property=\"article:published_time\" content=\"2015-06-04t17:29:57+00:00\" \/>\n<meta property=\"og:image\" content=\"\/\/www.catharsisit.com\/wp-content\/uploads\/2015\/06\/screen-shot-2015-06-03-at-4.13.43-pm.png\" \/>\n<meta name=\"author\" content=\"peter poer\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"peter poer\" \/>\n\t<meta name=\"twitter:label2\" content=\"est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"webpage\",\"@id\":\"\/\/www.catharsisit.com\/blog\/optimized-student-happiness-using-ab-testing\/\",\"url\":\"\/\/www.catharsisit.com\/blog\/optimized-student-happiness-using-ab-testing\/\",\"name\":\"how we optimized for student happiness using a\/b testing\",\"ispartof\":{\"@id\":\"\/\/www.catharsisit.com\/blog\/#website\"},\"datepublished\":\"2015-06-04t17:29:57+00:00\",\"datemodified\":\"2015-06-04t17:29:57+00:00\",\"author\":{\"@id\":\"\/\/www.catharsisit.com\/blog\/#\/schema\/person\/12cb533a0e076340a06bd6d6fee439af\"},\"description\":\"how we ran an experiment with nps tool wootric to maximize magoosh student happiness!\",\"breadcrumb\":{\"@id\":\"\/\/www.catharsisit.com\/blog\/optimized-student-happiness-using-ab-testing\/#breadcrumb\"},\"inlanguage\":\"en-us\",\"potentialaction\":[{\"@type\":\"readaction\",\"target\":[\"\/\/www.catharsisit.com\/blog\/optimized-student-happiness-using-ab-testing\/\"]}]},{\"@type\":\"breadcrumblist\",\"@id\":\"\/\/www.catharsisit.com\/blog\/optimized-student-happiness-using-ab-testing\/#breadcrumb\",\"itemlistelement\":[{\"@type\":\"listitem\",\"position\":1,\"name\":\"home\",\"item\":\"\/\/www.catharsisit.com\/blog\/\"},{\"@type\":\"listitem\",\"position\":2,\"name\":\"how we optimized for student happiness using a\/b testing\"}]},{\"@type\":\"website\",\"@id\":\"\/\/www.catharsisit.com\/blog\/#website\",\"url\":\"\/\/www.catharsisit.com\/blog\/\",\"name\":\"magoosh 2022年足球世界杯举办地\",\"description\":\"\",\"potentialaction\":[{\"@type\":\"searchaction\",\"target\":{\"@type\":\"entrypoint\",\"urltemplate\":\"\/\/www.catharsisit.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inlanguage\":\"en-us\"},{\"@type\":\"person\",\"@id\":\"\/\/www.catharsisit.com\/blog\/#\/schema\/person\/12cb533a0e076340a06bd6d6fee439af\",\"name\":\"peter poer\",\"image\":{\"@type\":\"imageobject\",\"inlanguage\":\"en-us\",\"@id\":\"\/\/www.catharsisit.com\/blog\/#\/schema\/person\/image\/c870f550a32ea3b6b77e05721421a012\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/75ead203be18d17c1536c3e51d63d904?s=96&d=mm&r=g\",\"contenturl\":\"https:\/\/secure.gravatar.com\/avatar\/75ead203be18d17c1536c3e51d63d904?s=96&d=mm&r=g\",\"caption\":\"peter poer\"},\"description\":\"peter helps make sure magoosh students have the best possible content. a proud arizona wildcat and teach for america alum, he worked as an instructional coach before getting an mba at berkeley\u2019s haas school of business. he is passionate about student achievement and educational equity. also prime numbers. peter enjoys cooking, running (slowly), and going to bed comically early.\",\"url\":\"\/\/www.catharsisit.com\/blog\/author\/peter\/\"}]}<\/script>\n<!-- \/ yoast seo premium plugin. -->","yoast_head_json":{"title":"how we optimized for student happiness using a\/b testing","description":"how we ran an experiment with nps tool wootric to maximize magoosh student happiness!","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"\/\/www.catharsisit.com\/blog\/optimized-student-happiness-using-ab-testing\/","og_locale":"en_us","og_type":"article","og_title":"how we optimized for student happiness using a\/b testing","og_description":"how we ran an experiment with nps tool wootric to maximize magoosh student happiness!","og_url":"\/\/www.catharsisit.com\/blog\/optimized-student-happiness-using-ab-testing\/","og_site_name":"magoosh 2022年足球世界杯举办地","article_published_time":"2015-06-04t17:29:57+00:00","og_image":[{"url":"\/\/www.catharsisit.com\/wp-content\/uploads\/2015\/06\/screen-shot-2015-06-03-at-4.13.43-pm.png"}],"author":"peter poer","twitter_card":"summary_large_image","twitter_misc":{"written by":"peter poer","est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"webpage","@id":"\/\/www.catharsisit.com\/blog\/optimized-student-happiness-using-ab-testing\/","url":"\/\/www.catharsisit.com\/blog\/optimized-student-happiness-using-ab-testing\/","name":"how we optimized for student happiness using a\/b testing","ispartof":{"@id":"\/\/www.catharsisit.com\/blog\/#website"},"datepublished":"2015-06-04t17:29:57+00:00","datemodified":"2015-06-04t17:29:57+00:00","author":{"@id":"\/\/www.catharsisit.com\/blog\/#\/schema\/person\/12cb533a0e076340a06bd6d6fee439af"},"description":"how we ran an experiment with nps tool wootric to maximize magoosh student happiness!","breadcrumb":{"@id":"\/\/www.catharsisit.com\/blog\/optimized-student-happiness-using-ab-testing\/#breadcrumb"},"inlanguage":"en-us","potentialaction":[{"@type":"readaction","target":["\/\/www.catharsisit.com\/blog\/optimized-student-happiness-using-ab-testing\/"]}]},{"@type":"breadcrumblist","@id":"\/\/www.catharsisit.com\/blog\/optimized-student-happiness-using-ab-testing\/#breadcrumb","itemlistelement":[{"@type":"listitem","position":1,"name":"home","item":"\/\/www.catharsisit.com\/blog\/"},{"@type":"listitem","position":2,"name":"how we optimized for student happiness using a\/b testing"}]},{"@type":"website","@id":"\/\/www.catharsisit.com\/blog\/#website","url":"\/\/www.catharsisit.com\/blog\/","name":"magoosh 2022年足球世界杯举办地","description":"","potentialaction":[{"@type":"searchaction","target":{"@type":"entrypoint","urltemplate":"\/\/www.catharsisit.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inlanguage":"en-us"},{"@type":"person","@id":"\/\/www.catharsisit.com\/blog\/#\/schema\/person\/12cb533a0e076340a06bd6d6fee439af","name":"peter poer","image":{"@type":"imageobject","inlanguage":"en-us","@id":"\/\/www.catharsisit.com\/blog\/#\/schema\/person\/image\/c870f550a32ea3b6b77e05721421a012","url":"https:\/\/secure.gravatar.com\/avatar\/75ead203be18d17c1536c3e51d63d904?s=96&d=mm&r=g","contenturl":"https:\/\/secure.gravatar.com\/avatar\/75ead203be18d17c1536c3e51d63d904?s=96&d=mm&r=g","caption":"peter poer"},"description":"peter helps make sure magoosh students have the best possible content. a proud arizona wildcat and teach for america alum, he worked as an instructional coach before getting an mba at berkeley\u2019s haas school of business. he is passionate about student achievement and educational equity. also prime numbers. peter enjoys cooking, running (slowly), and going to bed comically early.","url":"\/\/www.catharsisit.com\/blog\/author\/peter\/"}]}},"authors":[{"term_id":503,"user_id":77,"is_guest":0,"slug":"peter","display_name":"peter poer","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/75ead203be18d17c1536c3e51d63d904?s=96&d=mm&r=g","user_url":"","last_name":"poer","first_name":"peter","description":"peter helps make sure magoosh students have the best possible content. a proud arizona wildcat and teach for america alum, he worked as an instructional coach before getting an mba at berkeley\u2019s haas school of business. he is passionate about student achievement and educational equity. also prime numbers. peter enjoys cooking, running (slowly), and going to bed comically early."}],"_links":{"self":[{"href":"\/\/www.catharsisit.com\/blog\/wp-json\/wp\/v2\/posts\/2807"}],"collection":[{"href":"\/\/www.catharsisit.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"\/\/www.catharsisit.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"\/\/www.catharsisit.com\/blog\/wp-json\/wp\/v2\/users\/77"}],"replies":[{"embeddable":true,"href":"\/\/www.catharsisit.com\/blog\/wp-json\/wp\/v2\/comments?post=2807"}],"version-history":[{"count":0,"href":"\/\/www.catharsisit.com\/blog\/wp-json\/wp\/v2\/posts\/2807\/revisions"}],"wp:attachment":[{"href":"\/\/www.catharsisit.com\/blog\/wp-json\/wp\/v2\/media?parent=2807"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"\/\/www.catharsisit.com\/blog\/wp-json\/wp\/v2\/categories?post=2807"},{"taxonomy":"post_tag","embeddable":true,"href":"\/\/www.catharsisit.com\/blog\/wp-json\/wp\/v2\/tags?post=2807"},{"taxonomy":"author","embeddable":true,"href":"\/\/www.catharsisit.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=2807"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}