over time, we’ve noticed there are some topics that almost everyone wants to know more about, so we’ve compiled the list of most common questions we get asked by candidates, along with the answers that will help you decide if magoosh is the right place for you.

 

table of contents

culture

• what is the culture like at magoosh?
• what are the best parts about working at magoosh?

diversity, equity, inclusion & belonging

• how does magoosh foster a diverse and inclusive environment?

growth opportunities

• are there opportunities to grow at magoosh?

hiring process & practices

• what’s the hiring process?
• do you provide work sponsorship?
• how do i prepare?
• do you actually read the cover letter?
• is the salary negotiable—why not?

mission & values

• can you share more about why your values are inequalities?
• in what ways does magoosh fulfill its mission to level the educational playing field?
• in what ways have you seen magoosh live out the core values?

org structure

• what is an area team?
• what is the org structure?

remote workforce

• what structures are in place to support part-time and remote employees?
• i’m on the east coast, do i have to work west coast hours?
• can part-time employees choose their own hours?

culture

what is the culture like at magoosh?

magoosh’s culture is a reflection of our core values, which, day-to-day, means we’re a friendly bunch who are always challenging ourselves to learn, change, and passionately do everything we can to make test prep accessible to as many students as possible. we prioritize communicating every step of the way, moving quickly, and wow-ing our students and each other as we go.

aside from being friendly, magooshers cheer each other on, are flexible and driven, and provide each other with the feedback we need to grow. we also like getting to know each other better and try to enable folks to be their authentic selves, and do that by having social events and virtual gatherings, like dei book club, games happy hour, holiday extravaganza, wine club, and more!

 
what are the best parts about working at magoosh?

some of the things that make magoosh a great place to work include making an impact and engaging in meaningful work, a genuine commitment to the mission shared by all magooshers, leadership that genuinely cares about people, flexible work schedules, friendly and passionate team members, and fun social events (even virtual ones!).

diversity, equity, inclusion & belonging

how does magoosh foster a diverse and inclusive environment?

this is a work in progress for us! we have done a lot of awesome work so far, like:

• standardizing our interview questions and rubrics (more info here)
• holding ourselves accountable by using the rooney rule
• ensuring part-time, full-time, remote, and in-office team members feel integrated into the magoosh culture
• developing a framework for employee resource groups (ergs), including bmat, aapi, lgbtqia, and minds of all kinds
• building out a pay equity framework (more info here)
• ensuring equitable benefits and perks across our employee groups
• a number of other key results around belonging, voice, inclusion and more that are owned by our director of deib, dei committee, and ergs for an even greater focus and impact

take a look at these blogs for more info about our diversity goals and how we examined deib at magoosh.

we know there is more we can do, and we are actively pushing ourselves to become a more diverse and inclusive organization.

growth opportunities

are there opportunities to grow at magoosh?

yes! there are a ton of opportunities to grow! in the past, we’ve had part-time team members become part-time team leads, part-time team members transition into full-time roles, full-time team members move to new departments (like marketing to engineering), and full-time team members move up to higher-level individual contributor or management roles.

we do periodic evaluations so that magooshers always know where they’re at and what they need to do in order to grow.

if your dreams and goals ultimately take you outside of magoosh, we will support you in that next step of your journey, too! and we are very connected with our magoosh “alumni.”

hiring process & practices

what’s the hiring process?

check out a detailed answer to this question in this blog.

do you provide work sponsorship?

being a small startup, we are unable to provide sponsorship and currently can only employ folks who are authorized to work and live in the u.s.

how do i prepare?

read nicole’s awesome blog on this exact topic here.

do you actually read the cover letter?

yes, we read every single one! after you apply for an open position on our careers page, we review your resume and cover letter. while reading the cover letter, we’re looking for additional signs, beyond what is captured in the resume, that you have the skills needed for the position and that you have a genuine passion for magoosh’s mission and interest in the role you’re applying for.

don’t have all of the requirements for the role? we still encourage you to apply (we especially encourage candidates from historically underrepresented backgrounds to apply). the cover letter is a great way to tell us why you’d be a strong candidate for the role, despite not having all of the qualifications.

is the salary negotiable—why not?

at magoosh we’re committed to pay equity. we understand that folks with certain identities are more likely to get raises or do better in negotiations than others. we want to ensure that magooshers who are doing the same job at the same level will have the same salaries. we combat pay inequity by making all salaries non-negotiable. if you’re interested in learning more, take a look at this blog post that digs into our approach to salaries!

mission & values

can you share more about why your values are inequalities?

our values help guide our decision-making and how we work, but it’s not to say that one way of doing things is better than another. as you’ve already noticed, we turned our values into inequalities—the left side represents the magoosh way, and the right side represents another good way, just not our way. to learn more about our core values, click here.

in what ways does magoosh fulfill its mission to level the educational playing field?

what differentiates magoosh from so many other companies is our mission. we create high-quality test prep products at an affordable price. test prep can cost hundreds, even thousands of dollars, making it impossible for lower-income students to afford. we want all students to have the opportunity to prepare for their standardized exams. on top of our products being incredibly affordable, we also offer a bunch of student discounts and sometimes offer our product for free to community-based organizations that serve some of our most marginalized communities.

in what ways have you seen magoosh live out the core values?

magoosh lives and breathes its core values. they are a part of our everyday workflows. they help guide our decision-making, help us assess candidates in our interview process and employees in their performance, and are used to ensure we continue to live out our mission of leveling the playing field.

a few of the core values a new magoosher would quickly notice are: our teammates are truly some of the friendliest folks around, we really value communication, our leaders take wow > profit seriously, and done > perfect is a way of life here.

during the uncertainties of 2020, magoosh lived out wow > profit in a big way, both for students by allowing flexibility with their memberships and employees by offering flexible/reduced work hours, a covid sick bank (ft & pt employees), and additional paid time off for rest/mental health every quarter (just to name a few).

org structure

what is an area team?

an area team is a cross-functional team that works to drive forward the success of a product. the different areas at magoosh are:

• gre/gmat (grat team)
• sat/act (high school)
• lsat/mcat (lcat)
• ielts/toefl (iefl)
• new products (where we’re incubating new product ideas!)

on an area team, you may find a couple of marketers, a product manager, a curriculum manager, and an engineer.

what is the org structure?

magoosh doesn’t have a heavy top-down structure, meaning we maintain lines of communication open across all levels of the organization and encourage input/questions/ideas from everyone. that said, we do have area and department leads. area leads are responsible for our different products (listed in the answer above) and department leads guide the strategic direction of the department.

the various departments at magoosh include:

• content
• data
• engineering
• marketing
• operations (business ops, hr, recruiting, deib)
• product/ux
• student help

remote workforce

what structures are in place to support part-time and remote employees?

we have done a lot of work to ensure part-time magooshers are just as integrated into the culture as full-time magooshers. our part-time folks receive benefits like sick time, paid time off, 401k, and being part of all the social events that happen at magoosh. we’ve also revamped some of the perks for our part-time teammates! you can find more information here.

before 2020, we were already developing a remote/work-from-home framework to provide our teammates with more flexibility, and the pandemic just put that timeline in fast-forward. throughout most of 2020, our entire team worked remotely, and we anticipate shifting to an office/remote blended work environment for the future (even post-pandemic). magoosh has been supportive by contributing toward home office equipment, offering flexible hours, keeping teams connected through tools like asana and slack, and hosting virtual social events.

i’m on the east coast, do i have to work west coast hours?

you do not have to work west coast hours! we are super flexible when it comes to work schedules, and we want you to work the hours that are best for you while allowing for the overlapping hours that are needed for meetings and check-ins with your manager and team. in general, our ft employees work about 4-5 overlapping hours per day.

can part-time employees choose their own hours?

the ability for our part-time employees to choose their own hours largely depends on the business needs of the team. some of our part-time employees have the flexibility to choose their own work hours, allowing for 2-3 overlapping hours per week for manager/team check-ins and project updates, while other part-time teammates have more set schedules, doing the bulk of their work during standard business hours multiple days per week. your manager will be able to share more role-specific details, but in general, we aim to balance flexibility and role needs.

the post magoosh candidate faq appeared first on magoosh 2022年足球世界杯举办地.

below is an overview our our general hiring process for any position. (you can also read about it in more detail in our full magoosh hiring process blog post.)

• application review
• questionnaire
• phone interview
• reference and background checks
• in-person interview

the engineering exercise comes at the questionnaire stage. we’ve sought to create an exercise that’s both representative of the skills a candidate would use at magoosh and helpful to us in assessing the candidate’s experience.

the questions

we ask candidates to complete two questions: a programming question and an architecture/design question. for the programming question, we give the candidate a file and ask them to write code that is ready for production and code review. for the architecture question, we describe a challenge that we’ve faced and ask the candidate to share how they would approach that challenge.

the first iteration of the exercise was far from perfect—we received a number of questions, and we’d see submissions we didn’t expect. it didn’t take us long to realize our instructions left something to be desired. over time, we’ve taken candidate feedback and edited our instructions to make the exercise more clear. for example, we previously didn’t explain that we wanted production-ready code, so some candidates solved the programming question by writing a quickly thrown-together script. we plan to continue improving the exercise as we get more feedback.

timing

originally, we estimated both questions would take a total of three hours to complete, so we asked candidates to set aside a three-hour window to complete the exercise. we received feedback both from candidates and from people we had hired that the short window created unrealistic stress, which is not what we wanted. based on that feedback, we now ask candidates to complete the exercise within 24 hours. that way, all candidates have the same amount of time but there isn’t an unrealistic time pressure to complete the exercise exactly within three hours.

grading

our most senior engineer, zack mayeda, grades the engineering exercises. with the help of others, he’s created a rubric and grades against the rubric so that each exercise is graded on the same criteria. our hiring team also anonymizes as much of the candidate’s information as possible before handing the exercise off to him, so he can evaluate the work with as little bias as possible.

testing our test

when designing our exercise, we knew it would be impossible to create a process that was 100% perfect, but we still wanted to run a test to see if it was at least directionally accurate (one of our core values is data > intuition). so, i asked two talented senior engineers from another company to complete the exercise and then put their submissions (stripped of their names and information) into the queue along with other candidates’ submissions. if those engineers scored poorly, then we could be fairly certain that there might be something wrong with our exercise or rubric. fortunately, they both scored very well, which gave us some confidence that our exercise was a useful competency test. (as an aside, zack was very excited about the candidates, until i told him that they weren’t actually applying…sorry, zack!) all this said, we still acknowledge that our exercise isn’t perfect and will probably generate some false negatives and false positives. again, we’re dedicated to continuing to improve the process based on feedback we receive.

feedback

if a candidate does well enough on the exercise, we move them forward to the phone stage of the interview process. during the phone interview we may ask questions about why they made certain decisions. if a candidate doesn’t do well on the exercise and doesn’t make it to the phone interview stage, we’ll let them know we’re pursuing other candidates and give them the option to receive feedback on their work. we know feedback makes us all better, so we offer it to all candidates—not just those who make it all the way through to the end of the interview process.

tl;dr

1. • at magoosh, we believe your engineering exercise should be representative of the work you’d do at magoosh
   • our exercise assesses multiple types of engineering work: coding + architecture
   • we give everyone the same time constraints and try not to make those time constraints too stressful
   • when grading the exercise, we focus on the work not the person
   • we give feedback to anyone who wants it
   • we’re constantly trying to improve our exercise

interested in working at magoosh? check out our open positions.

if you don’t see an open position that’s perfect for you, please submit your resume via the general application.

learn more about our philosophy and culture here.

the post behind the scenes: the magoosh engineering exercise appeared first on magoosh 2022年足球世界杯举办地.

in the past year, a number of companies i admire—including moz, treehouse, and buffer—have had to make layoffs. hoping to shed light on the issue, those companies chose to be open about their cuts and published articles here, here, and here about their struggles. three businesses making layoffs may not seem like a big deal, but they represent only a small portion of the layoffs that happened last year. and with most layoffs going unannounced, the truth is we still have only a small glimpse of how widespread the problem really is.

at magoosh, a startup based in the bay area, we’re vastly aware of the uptick in layoffs happening around us. but rather than shutting our eyes and waiting to see what happens, we’ve made it a priority to understand why cuts happen and actively work to avoid them. taking a cue from moz, treehouse, buffer, and others who have written on the subject, i’d like to share our own thoughts on layoffs: what we’ve learned about why they happen, how we approach risk-taking at magoosh as a result, and how we implemented our own layoff-preventing practices.

by having more open conversations about layoffs, it’s my hope that we can promote more understanding around them and collectively start making better business decisions at the intersection of risk-taking, growth, and hiring.

layoffs can happen for a number of reasons. every company is different, after all. so, let’s focus on one prevalent reason that has widely affected companies in the tech space.

startups, especially silicon valley venture-backed startups, are encouraged to chase growth. growth can take the form of more users, more revenue, or more of something else.  the hope is that high growth (often unprofitable growth) will lead to more funding which will lead to more growth and so on. at some point the company, usually after it has a very large user base, will focus on profit with the thought that high growth at first could lead to more long-term profit. but there’s a natural tension between growth (i.e. market share) and profit, and it’s a tension that has existed long before the dot-com bubble in the early 2000’s. in fact, the harvard business review published an article on the subject as far back as 1975.

vc firms have investors who are expecting a 6x return on their investment over a period of 10 years. in order to achieve that level of success, vcs need some of their companies to achieve massive outcomes (100x or higher), given that most of their portfolio companies will fail or return very little. because of this, vcs are incentivized to invest in startups that aim for big growth—startups that are willing to lose money month-over-month in order to grow the top line faster.  when founders take vc money, they are implicitly promising they will chase growth, not profit.

targeting growth (over profit) can be a smart approach for many companies, especially companies in a winner-take-most market. for example, facebook, twitter, snapchat, and the like would not have reached their levels of success if they focused on profit early on. however, most companies are not facebook, twitter, or snapchat. and most companies don’t realize what they’re getting into with the growth-before-profit approach. too many of these companies—energized by silicon valley’s seemingly free-flowing cash and pervasive “growth at all costs” message—end up chasing growth (to get that next round of funding), when in reality they should be focused on nailing down their value prop and developing a path to profitability.

the growth-focused approach can be painful when it doesn’t work—and for most startups it doesn’t. every founder anticipates some level of failed experimentation (“this product might fail, that’s fine”), but what they don’t usually think through is the people side of the product. in other words, when you raise a lot of money and hire a lot of people to build a product and then it fails, you’re not only faced with dropping a product, you’re faced with laying people off.

example: when failed experiments lead to layoffs

i’ve looked up to moz since the early days of magoosh (2011). i’ve read many blog posts, especially the ones by their founder rand, on how to approach content marketing and how to approach building a company.  moz and rand have played a big part in magoosh’s growth and my own growth.

so, in late 2016, i was sad to learn that moz was announcing layoffs. from their ceo, sarah bird:

this is the gut-wrenchingly painful part. the hardest part of my job is asking people who have put their hearts and souls into moz to part ways. to align the organization with this strategic shift, we will be asking about 28% of mozzers to leave. they are a part of the moz family and it is heartbreaking that they will not be working alongside us in the future.

moz had raised $29m to-date, a small early round, then $18m in 2012 and another $10m in 2016. while i don’t know the specifics, i suspect they raised the $18m round based on the success they had up to that point, and on a big future vision of becoming more than an seo company—eventually becoming the leading provider of tools for all inbound marketing. this was a big experiment and a big risk. and after four years, they realized the risk didn’t work out the way they hoped, which led to laying off 28% of their staff.

running a startup is a constant exercise in risk assessment. sometimes big risks pay off and sometimes, as in moz’s case, they don’t.  and the truth is that those layoffs—while difficult—were likely the right decision for moz in the long-term. moz’s experience is just one example of what many startups go through. it’s why many layoffs happen and will continue to happen. so, what can companies do? are layoffs an inevitable consequence of taking risks?

at magoosh, we approach business and risk-taking differently. we raised a small round of capital to get us off the ground at the beginning, and since then we’ve made it a point to spend only the money that we have, using funds from our most-successful product lines to expand into new product lines and markets. it’s made us a stronger company and it’s insulated us from the rising tide of layoffs throughout the bay area, while allowing us to still take calculated risks.

so, what do our business decisions look like in action? below, i’ve outlined some scenarios that many companies face. then, i explain how a vc-backed growth-focused company would approach each scenario and compare it to how we would approach that same scenario at magoosh.

scenario 1: launching one (or several) new product lines

magoosh is an online test prep company, and we periodically add prep products for new exams. for instance, we recently launched an mcat prep product for aspiring doctors, and we have plans to add more exams in the future. let’s see how the approach to adding exams differs.

vc-backed approach:

• identify the exams the company wants to pursue and focus on those that have very large market sizes.
• build a model for the growth trajectory of those exams and determine how much capital is required and do some light validation (e.g. surveys, etc.).
• launch and build as many exam products as possible, staying within the capital constraint, and hire people to work on those products.

magoosh approach:

• identify the exams that we want to pursue, focusing not only on those that have a sizeable market, but also those that could quickly be roi+.
• validate demand for these exams by building an audience first, before potentially building a product. this validation can take 6-9 months.
• only create products for the exams for which we successfully built an audience.

tradeoffs:
the vc-backed approach can lead to faster growth since the products are created earlier. if there’s a first-mover advantage, then the benefit of faster growth could be significant, and as we discussed before, faster growth can also help the company raise their next round of funding. however, if many of the new products are not successful, the vc-backed approach could lead to significant lost capital and could result in layoffs for the people hired onto some of the new teams.

scenario 2:  developing a new growth channel—outbound sales

at magoosh, we recently decided to experiment with outbound sales. we had seen some indication of market interest based on inbound group purchase requests and believed this channel had more potential. let’s see how the approach to experimenting with outbound sales could differ:

vc-backed approach:

• determine the potential of outbound sales and then hire an individual or several individuals to grow revenue, depending on capital available.
• continue pursuing outbound sales for 2-3 years even if it’s losing money, that is as long as there is some positive signal. if it pays off then it could pay off in a big way and grow top-line revenue in the meantime.

magoosh approach:

• identify outbound sales as an opportunity and potentially hire a contractor or intern to pursue it rather than hiring someone full-time.
• if we decide to hire someone, explicitly let them know that this is an experiment so there’s more risk in this role than there would be in other roles, and look for a more significant positive signal before hiring others onto that team.

tradeoffs:

if the outbound sales is successful, the vc-backed approach will almost certainly lead to faster growth. and some experiments need time in order to demonstrate a return, so the magoosh approach of hiring a contractor or intern may not provide enough time to assess whether outbound sales was successful. the magoosh approach, however, leads to lower capital expenditure and provides more clarity on the uncertainty of the future of the role. the vc-backed approach could lead to layoffs if the new channel is not successful.

in both of the scenarios above, the vc-backed approach can lead to faster growth, and if successful,  is likely better overall. our company’s approach protects against lost capital and layoffs, meaning it’s better in the failure case (which, in the world of start-up experiments, is the most likely outcome), as it can still lead to profitability. and even though our approach can mean slower growth, we think of it as a long-term approach worth carving out time for.

for what it’s worth, our approach hasn’t kept magoosh from attaining high-growth or profitability. we became cash-flow positive in 2012 and continue to grow quickly year-over-year, even achieving a top 100 place on the inc. 5000 in 2016.

perhaps, some might say, we could have achieved that more rapidly if we had employed a vc-backed approach to growth, but i believe making time to build a risk model to mitigate the chance of layoffs was the right move for us in the long run. and i believe if more vc-backed companies acted as if they were chasing profitability, even if in reality they were chasing growth, they could mitigate their own layoffs because they had at least created some artificial constraints for themselves. constraints are difficult to abide by, though, so at magoosh we’ve developed our own framework to keep us on track.

over the years, we’ve developed a framework for how we approach risks, especially as they relate to people. when a manager wants to hire for a new role, we go through the following questions:

• what’s the expected benefit of this role?  estimate which metrics will move, by how much, and by when.

• is the work experimental in nature? if yes, what happens if the experiment doesn’t work out?

• what is the time-frame for this role? what will this person be doing in 6 months, in 1 year, in 2 years? if the person’s work/project will end or will no longer be a company priority, what will we do? should we hire a contractor instead?

these questions help us understand the expectations and risks associated with each role. we then try to communicate these expectations and risks to each candidate (aligning with our value of communication > efficiency). we realize this approach can make it harder to fill positions. because we’re upfront about the potential risk in our temp and experimental roles—while they may not actually be that risky in reality—people might turn down those roles in favor of full-time positions elsewhere that they perceive as being less risky. we’re comfortable with that trade-off, though. if the experiment doesn’t work out for any reason, we know the person opted into the risk.

our approach has its own issues. while we try to take risks, our risks are generally smaller than those of the companies that have raised a lot of capital. this means we’re less likely to see the massive 100x successes. for now, it’s a tradeoff we’re comfortable with, as we believe it’s an approach that works best for our company and our team.

now that you’ve read all about our approach to risk-taking, layoffs, and hiring, i need to make one thing clear: we’re not so naive to think magoosh is immune from layoffs. we recognize that for any number of reasons—changes in the market, the end of the gre as we know it—could lead us to make our own tough cuts. that will always be the reality. but i believe the existence of layoffs—the fact that they will always happen in business—is not a good enough reason not to take steps to minimize their potential. if we didn’t at least try, we would not be doing right by our employees or our company.

i’ve shared what we’re doing; now i’d love to hear from you as well. if you’re using other tactics at your company, or disagree with any of those we’re using here, please leave a note for me in the comments. let’s continue to make this an open discussion, so that we can all—magoosh included—get better at understanding, preventing, and addressing layoffs.

thanks to aj shankar (everlaw), joel gascoigne (buffer), tawheed kader (toutapp, now marketo), andrew cronk (re-factor) and greg mcverry (review talent feedback) for reviewing and providing feedback on this article.

header image designed by mark thomas

the post growth, profit, and consequences: what founders should know about the rise in tech layoffs and what we’re doing to prevent them here appeared first on magoosh 2022年足球世界杯举办地.

writing about diversity and inclusion (d&i) makes me uncomfortable. i’ve been told a compelling article should be bold and opinionated, but the truth is, my beliefs are oftentimes loosely-held and nuanced. the topic of diversity is no exception. yet here i am writing an article about it.

why? because i believe it’s important for myself and other ceos to frequently step back and take a hard look at how we run our companies and to share how we’ve failed, how we’re working to improve, and what we’ve learned in the process.

i’m the first to admit our company is still very early in its diversity journey. in fact, the more i look into d&i initiatives, the more i realize there’s a lot we don’t know and a lot of work we still need to do. that said, i still want to share with you how we think about d&i at magoosh and what we’re doing to create a more diverse and inclusive team—even if it’s uncomfortable to talk about. i hope it might support another team’s roadmap to diversity, that it communicates to magoosh jobseekers that we value d&i, and that it inspires more open conversations in the tech industry about our journeys, struggles, and failures in striving toward a more diverse workforce.

i started actively thinking about diversity in early 2015 after attending a panel at the lean startup conference about diversity in tech led by eric ries (founder of the lean startup movement), freada kapor (one of our investors) and a few others. about 15 minutes into the panel, eric shared a story about his aha moment in d&i. his company had been failing in the “diversity department,” he told the audience, but he attributed it to a pipeline issue. to challenge this assumption, he and his team decided to run blind application reviews (in which he couldn’t see the name of applicants) and the results surprised eric. he realized he would move different candidates forward when doing a blind application review compared to traditional application review. in other words, bias was largely creeping into his hiring process and he hadn’t even realized it. his anecdote hit hard. sitting there listening quietly, i was overcome with a sinking feeling. i realized bias—my bias— was likely creeping into our process.

i went back to magoosh with a plan to change our hiring methods. we would implement new strategies and seek out diverse recruiting channels. no problem, i told myself, we have this diversity thing figured out. i no longer had to feel like my company was one of those companies, blinded by its own biases.  that’s when—confident i was close to solving our own diversity issues—i reached out to freada and asked where we should start posting our new open positions. she responded the same day.

“bhavin–i guess i don’t understand the disconnect here.  you seem to be asking for a simple list (i.e. silver bullet) of where to recruit so that you’d end up with more diversity.  if the issue were simply about recruiting somewhere differently, everyone would know about it and every tech company would be more diverse….

instead this is a much more complicated and nuanced problem.

there are dozens of methodologically rigorous studies that document even well-intentioned people harbor all kinds of racial, ethnic and gender bias, especially when it comes to stem fields.  if i just send you a list of schools (which are easy enough to find without my help), none of these biases will be addressed.  resumes with african american or women’s names will be unconsciously judged differently.

a real commitment to diversity is not a check-the-box exercise.  there are things to do on the recruiting side (e.g. what’s the language in the job description?), hiring side (e.g. where you post or do in-person recruiting), interviewing (e.g. do you do structured interviews or free form and how do they each stack up if you care about diversity?), evaluating candidates (e.g. does rank of school matter?) then onboarding.

this is not one email or one meeting…it’s a decision to evaluate your practices and your work environment so that you can recruit and retain diversity.”

in other words, i didn’t have diversity figured out. i was asking for a simple solution to a complex problem—a problem that would require me to start rethinking magoosh’s hiring initiatives from every angle, not by doing a quick check-the-box exercise.

freada graciously offered to come chat with me and our leadership team. she answered our questions, shared some research, and identified strategies that we could implement. after the meeting, we began holistically evaluating our hiring process. our team took a long, hard look at our process and saw we were doing some things well and some things poorly. we came to realize committing to diversity and inclusion would be an ongoing process that would require quite a bit of effort.

since then, we’ve committed to continuous improvement of our hiring process, and i want to share what we’re doing today to keep getting better. each item below is a work in progress—we know there’s more we can do, and we continue to think about how to create a fairer and better hiring process and a more inclusive culture. in aggregate, here’s what we’d aim to accomplish with the practices outline below: 1) to encourage candidates from diverse networks to apply to magoosh; 2) to decrease our own unconscious bias; and 3) to create a fair system for hiring and growth at magoosh. take a look and i welcome you to share your thoughts and questions at the end.

what we’re doing to encourage candidates from diverse networks to apply to magoosh

diversifying our applicant pool

many of our hiring managers, myself included, don’t have diverse networks. we’re currently working on building our networks, so we can share open positions, details about our company, and more to people from diverse backgrounds. for a few positions, we also experimented with the rooney rule, holding off on phone interviews until we hit certain target percentages for applicant pool, and we hope to use the rooney rule for all positions in the future.

creating inclusive job descriptions

when freada visited magoosh, she shared that women are less likely than men to apply to positions in which they don’t meet all the criteria. she also told us both women or men may be discouraged from applying based on certain words in the job description. we made a few changes:

• in our job postings, we now say “you have many of the following” instead of saying “requirements”. we also added a sentence that says, “please feel comfortable applying, even if you don’t meet all the requirements for the position.”
• our hiring managers and recruiters know to avoid certain words that might discourage certain candidates from applying (e.g. we’re looking for a “finance guy”)
• we run our job postings through textio, a site that provides you with feedback about language in your posting. it tells us how much of the language in our job descriptions is gendered, how much of it might discourage someone of a different race, etc., then gives us a score based on those factors. we target a score of 65+ for each posting.
• we added a sentence to the bottom of all our job descriptions that states, “we are an equal opportunity employer and value diversity at our company. we do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.”

what we’re doing to decrease our unconscious bias

conducting blind questionnaire review

we’d love to have a blind review process for all cover letters and resumes, but unfortunately, our applicant tracking system doesn’t support that yet. however, we have been experimenting with blind review for candidates’ questionnaires—a later step in our process. when possible, our recruiter provides the questionnaire responses to the hiring manager without the candidate’s name, so the hiring manager isn’t unintentionally biased in some way.

using rubrics for interviews and questionnaires

a few years ago, we were recruiting for a position that was bringing in a high volume of applications. multiple people were grading the same questionnaires, and they quickly realized there were a number of discrepancies between each assessment. one of the individuals decided to introduce a grading rubric to address the problem. the hypothesis: if they predefined the criteria they were looking for in the role, then assessed each questionnaire based on that criteria, they could lower the chance for bias to creep in. ideally, two graders reviewing the same responses and using the same rubric would reach the same conclusion. overall, it proved to be a successful strategy, so other magoosh hiring managers began using rubrics for questionnaires and for the in-person interview as well. now, grading on a rubric has become a standard part of how we evaluate candidates.

asking structured in-person interview questions

we do two culture/behavioral interviews for all in-office, full-time hires. we use the same set of questions for each candidate, and for a given open position, we’ll use the same set of interviewers. each interviewer then independently completes a scorecard based on our in-person interview rubric, and then we debrief. this approach ensures candidates are being asked the same questions, and it prevents one interviewer from biasing another one.

what we’re doing to create a fair system for hiring and growth

providing guidance on what to write in a cover letter

we’ve found that some people know how to write cover letters and others don’t. we’ve also realized that one’s ability to write a cover letter isn’t necessarily a predictor of how well that person would do in the role. in our application, we now provide guidance to all applicants about what we’re looking for in a cover letter to ensure each candidate knows exactly what we’re looking for.

creating a salary framework

a few years ago, i wrote about our approach to non-negotiable salary. our salary framework is an extension of that principle. in short, we aim to place each of our employees in a specific role (e.g. product manager) and at a specific level (e.g. level 3). there’s a fixed salary associated with each role and level. we hope this results in equal pay for equal work.

“stay” interviews

most people are familiar with exit interviews in which an hr person asks an employee for feedback after that employee has already decided to leave. in mid-2015, we decided to conduct “stay” interviews or, as we call them, check-ins. our hr manager meets with every employee in the company to get feedback—both positive and negative. she then shares company-wide and department-wide themes (without sharing individual names) with me and the other team leads. we believe that getting this feedback is helping us take steps to create a more inclusive culture.

it’s hard to quantify exactly what impact these changes have made, but i can say that in late 2014, we had 18 employees who were 30% women and 70% men. today, in early 2017, our team of 30 full-time employees is 60% women and 40% men, including two women on our five-person senior management team, and four women people managers out of a total seven people managers. that said, we are still very underrepresented in certain areas and have much progress to make. i hope that by committing to the practices outlined above, and by expanding into more, we can eventually achieve and sustain our d&i goals.

diversity and inclusion requires consistent and relentless focus. we’re still very early in our journey—we have a long way to go and a lot to learn, but i’m grateful that we started down this path and that our team is constantly thinking about how we can get better.

in early 2016, we signed the kapor founder’s commitment, in which we committed to a series of actions that support diversity and inclusion in our company. you can learn more about the commitment and how your own company can commit to diversity here.

now that i’m done sharing my thoughts, i’d love to hear from you. as i mentioned previously, we still have a ways to go in our diversity efforts. if you have any ideas about how we can continue to increase the number of underrepresented minorities at magoosh, let me know in the comments.

thanks to cat perez (healthsherpa), sydney thomas (precursor ventures), and the magoosh team for reviewing this article.

the post there’s no ‘silver bullet’ to increasing diversity, but here’s how we’re making progress. appeared first on magoosh 2022年足球世界杯举办地.

at magoosh, we value communication. to that end, we want you to know what to expect with our hiring process. the entire process can take as little as a 3 weeks or much longer depending on how quickly you respond, how quickly your references respond, and our internal schedule. along the way, we’ll do our best to keep you updated. and you should always feel free to reach out to us if you want to know where you stand in the process or what the hold up is! i’ve outlined our general process below.

in an effort to continue learning and growing, we will be experimenting with some process changes in hopes of creating a more streamlined and enjoyable experience for our candidates. for some positions, the phone interview stage may come before the take-home assessment (omitting the intro call stage altogether). 

step 1a: application

you apply for one of our open positions on our jobs page. we review your resume and cover letter, looking for signs that you have the experience needed for the position, and more importantly, that you’ve done a little homework about magoosh. we’ll typically get back to you within 2 weeks and hopefully much faster (unless the position says otherwise).

step 1b: intro call

this step only applies to select in-office positions. if we like what we see in your resume and cover letter, we’ll schedule an intro call with you, likely with the hiring manager or our recruiter. the purpose of the intro call is for us to see if you meet the core requirements for the role, and for you to learn more about magoosh and the position. before we ask you to tackle our questionnaire, we want to make sure you have a good understanding of what we’re looking for and that you’re still interested, hopefully excited! the call usually takes about 30 minutes.

step 2: take-home assessment

next, we’ll send you a take-home assessment with some sample tasks and questions related to the role and to magoosh. these take-home assessments can take an average of 1.5 to 3 hours, depending on the position. the assessment is also a great opportunity for you to get a sense of what type of work you might be doing in the role. if you’re not excited about completing the assessment, you likely won’t be excited about the position—and that’s definitely okay! just let us know that the position isn’t for you. hiring is about finding a mutual fit.

in an effort to treat candidates as fairly as possible, we grade each assessment anonymously against our rubric. this just means the hiring manager won’t know whose assessment they’re grading, and at this stage, you’re assessed solely on the quality of your work. we recognize the investment our candidates put in at this stage of our process. we value that investment and want to make sure we’re pulling our weight! we grade each assessment carefully and thoughtfully, often with more than one grader to ensure fairness. once you complete the assessment, we’ll typically respond within a week.

step 3: phone interview

assuming there’s a mutual fit after the assessment, we’ll move on to the phone interview (think of it as a mini-interview for both of us)! the phone interviews usually last about 60 minutes during which we’ll ask questions to gauge your fit for the role and how you might add to our culture and values. and you’ll also have the chance to ask us about the role and about magoosh. we’ve definitely had applicants decide magoosh isn’t for them after phone interviews, and that’s also okay! again, for us and ideally for you, this process is about finding a mutual fit. this is the last step in our process for remote positions.

step 4: in-person interview

this step only applies to in-office positions. you’ll typically have three to five interviews over a 4-to-6 hour period.

• one or two competency interviews: you’ll take on more sample tasks that you would do in the position. we also may ask you how you would think through some challenges we’re currently facing.
• one or two culture interviews: the purpose of these interviews is to see if we share the same values in how we approach work. here’s a guide that will help you prepare for our in-person culture interview.
• reverse interview: this is an opportunity for you to ask questions to another magoosher about what it’s like to work at magoosh (e.g. what are typical hours like? what do you like most about working here? what do you like least about working here?). we’ll try to pair you with someone who has either been in a similar position or has the same manager as you would have.

after the in-person interview, we’ll typically get back to you within 2 weeks, many times sooner.

step 5: reference and background checks

this step only applies to in-office positions. after the in-person interview, we do reference and background checks. this is the last step in our process for in-office positions. for the background check we want to make sure you are who you say you are. for reference checks, we want to talk with people who have worked with you before so we can learn more about you from someone else’s perspective. these take about 1 week. if we make you an offer after the reference and background checks, we’ll ask you to get back to us within 1-2 weeks.

for remote positions, the entire process takes about 2-3 weeks and for in-office positions, it’s about 5-7 weeks. we’ve also had applicants make it through much faster for both, so it really depends.

learn more about the team already working at magoosh:

team page

and find out why we love our jobs so much:

magoosh wins happiest company award!

the post magoosh hiring process appeared first on magoosh 2022年足球世界杯举办地.

many founders and vcs espouse the mantra hire fast, fire fast. there’s a sense of urgency to fill every open position, because every day a position is open is a day that work isn’t getting done. and if you make a hiring mistake, well, you can always fire that employee and try again with someone else. sounds like good business, right?

but there’s a problem: employees are people, not resources. they are people with lives, obligations, and financial needs. yes, you should focus on doing what’s best for your company, but you should also treat people with the respect they deserve.

an all-too-common anecdote (based on a real story)

saasco, a bay area startup,  just raised a $7m round of funding and is hiring fast. they found aaron, who lives in cleveland, oh, and who had been searching for software engineering jobs in the bay area. saasco interviewed aaron giving him a coding test and technical design question. they even flew him out so he could meet the team over lunch. all went well. aaron got the job and relocated with his wife and kids (relocation expenses paid). the change was disruptive and expensive (due to the high cost of living) for his family , but he had a high-paying job at well-funded startup, and he and his family would adjust.

six weeks into the role, the founder calls aaron into his office and fires him. aaron wasn’t delivering results. he was used to a slower pace with more structure, but saasco was a bit chaotic. employees needed to make progress without being guided; the company needed to hit their metrics to raise their next round of funding. aaron was struggling with the ambiguity and others were getting frustrated. the company paid aaron a week of severance, and the founder knew he made the right business decision to let aaron go.

was the company right in letting aaron go? probably. could they have handled the situation better? definitely. these types of occurrences happen all too often. this “churn and burn” culture is not something i (or magoosh) want to be part of. so how do we approach it?

hire slow

slow doesn’t mean what you might think it means.  when hiring, you should move candidates through the process as quickly as possible. and when you find someone who’s a good fit, you should move quickly and make an offer.

hiring slow means that you should be thorough. don’t make the hiring decision based on a snap judgement. create a structured process that tests for job and culture fit and fights against your biases. google’s senior vice president of people operations has written quite a bit on the topic.

a thorough process is fairer to potential employees. you’re doing the hard work of assessing fit up front, so that hopefully you won’t need to let them go. we’ve iterated on our process many times, and we’ll continue to do so. you can read more about our hiring process in detail here.

fire fast, but humanely

occasionally, you’ll need to let people go to protect your company and your other employees. after all, one person who’s not a fit can negatively affect everyone else. however, you can let people go and treat them with respect. here’s how:

1. share your stance on firing with employees before they join.

when new employees join companies, they often assume they have some level of job protection, maybe for a year—it’s an implicit understanding especially for those who have not worked in a fast-paced startup culture. but if you believe in firing fast, then you should explicitly let the new hire know about your approach. even better, let them know what percent of people have been let go in the first 3 months. let them opt into the risk of joining your company. don’t rely on the employee’s understanding of at-will employment. don’t lie by omission to get them to join.

2. communicate expectations clearly and regularly

after you hire someone, spend time crafting a 3-month plan with them. share what you expect them to accomplish and how you will measure their performance. if you don’t know all the specifics, then share your general expectations about performance and values. have weekly one-on-ones (and daily check-ins in the first week or two) to share feedback and to give them time to ask questions. your job is to help the new employee be successful, so early feedback is critical.

at magoosh, we’re now better at this then we used to be, but we still have a lot of room for improvement. (i’m sure many magooshers reading this didn’t have a clearly laid out 3-month plan when they joined—sorry!) we now try to do the following:

1. provide specific culture and competency feedback during weekly 1-on-1’s.
2. regularly (every 2 weeks or so) communicate how well the new employees are doing in their role.
3. solicit feedback from new employees about their experience at magoosh and what we can do to help them succeed.

3. offer a reasonable and well-defined severance. 

for any number of reasons, things may not work out. but remember, when you fire someone, it’s your fault. if you had a better hiring process, you could have identified that this person wasn’t well-suited for the role or the company. now the employee has to find another job, which takes time. we took inspiration from rand fishkin of moz and offer one month of severance (assuming there were no egregious behavior issues). think of severance as a “hiring penalty”:  you pay it because you didn’t invest enough time in the hiring process, and it’s your incentive to make the hiring process better.

here’s a link to a google doc we share with candidates during the hiring process which outlines how we approach the first few months, the intro period:

in the fast-paced startup culture — where every day matters — it’s easy for founders to hire and fire quickly. however, step back and evaluate the consequences of those actions. the people you hire are not disposable resources. sometimes a relationship between an employee and a company doesn’t work out, and that’s ok. let them go, but do so humanely and then improve your hiring process for the next hire.

read more by bhavin:

why we don’t negotiate salary and neither should you

how being named the happiest company made me less happy

the post hire slow, fire fast (but do it humanely) appeared first on magoosh 2022年足球世界杯举办地.

at magoosh, our culture and values define how we approach our work, interact with students and each other, and much more. contrary to popular belief, culture isn’t about whether or not you drink blue bottle coffee, are good at foosball, or like certain movies. a good set of values is a decision-making framework—values should help team members make decisions between two good options.

in true nerd fashion, we’ve turned our values into inequalities—we are test prep experts after all! the left side represents the magoosh way, and the right side represents another good way, just not our way. knowing what you aren’t is as important as knowing what you are.

done > perfect

• we have a bias toward action. we don’t delay for perfect tomorrow what can be done well today.
• we believe in the pareto principle. we try to achieve 80% of the impact with 20% of the effort.

data > intuition

• we run experiments to test ideas and gather data. we’re scientists!
• we collect feedback often since we don’t read minds…well
• we confirm (or disprove!) our intuitions with data.

passion > [something]

• we love what we do–helping students is too much fun to be considered work.
• we enjoy thinking about magoosh and our students whether at work or home.
• we help our teammates and students in times of need, even if we have to leave a dinner party to do it.

challenge > comfort

• we tackle tasks that we’ve never done before.
• we understand the best progress often comes from uncomfortable situations.
• we challenge ourselves to learn new skills—we are always students.

communication > efficiency

• we set clear expectations and don’t oversell to students, partners, and teammates.
• we let people know when we’ve completed a task, closing the “communication loop.”
• we follow up and send reminders–we don’t pass the buck.
• we value direct, person-to-person feedback and communication.

friendly > formal

• we respond to and talk about colleagues, customers, and partners in a friendly and positive tone.
• we take responsibility for our mistakes and don’t harp on the mistakes of others.
• we trust each other and don’t engage in politicking.

change > status quo

• we will always be a work in progress.
• we adapt to difficult and unplanned situations with a positive attitude.
• we constantly re-evaluate our priorities.

learning > knowing

• we independently seek out learning opportunities
• we have a growth mindset and believe that weaknesses can be turned into strengths
• we believe in potential and give magooshers opportunities to grow and change roles

accessible > exclusive

• we are available to help and support our students and teammates.
• we encourage ideas from everyone—the best ideas can come from anywhere!

wow > profit

• we wow customers and each other with service even at the expense of short-term profit.
• we go above and beyond in our work and never say, “it’s not my job.”
• we readily share and give credit.

_______

by no means is this our last and final list. it’s done but not perfect, and we know there’s room for change. however, these values help us collectively make hundreds of decisions every day. we not only trust each other to follow them, but we also hold each other accountable when we don’t. i’ve even been able to adopt some of our values in my personal life—i now make faster decisions (though still slower than many on the magoosh team…still working on that), and i’ve learned to embrace change and unplanned situations.

the post magoosh core values appeared first on magoosh 2022年足球世界杯举办地.

at first, i was stoked. it felt like all the hard work we put into our culture and values was paying off. people truly enjoyed coming work and felt great about the work they did. but over time, my excitement about the award has waned. every 4 weeks, i get an email where i’m asked to rate my own happiness. and while i enjoy what i do and why i do it, i do have rough days. we all do. but i’m hesitant to be honest because we have an image to preserve—we’re the “happiest company in education” and if we start being less happy that means something is wrong, right?

because of this mindset, i started realizing that i may rate my happiness at a 9 instead of a 7 (with 7 being how i might truly feel on a given day.) i’m letting the award and the image of happiness get in the way of how i’m truly feeling, and that’s not okay. i also started to wonder, “is something wrong with me if i’m the ceo and am rating my happiness at my company as a 7 out of 10?” when i see other anonymous ratings that are lower than usual, i’d also wonder if we were on a downward trajectory, going from the happiest company to just another company (as if that’s a bad thing).

reflecting, i’ve realized that i’m never going to be satisfied: not with our happiness, not with our growth, not with any aspect of the business. we’re doing well as a company, but we can always do better. the happiest company award created the sense that we’ve arrived. but we haven’t. we’ll never arrive. but we are constantly moving forward.

ben horowitz, partner at vc firm a16z and the author of the hard things about hard things, wrote about how the most difficult skill of a ceo is managing your own psychology. i think the first step to that is being honest with myself.

i’m now pushing myself to embrace my feelings. when i’m at a 7, i rate myself at a 7 and then figure out how i can improve that number rather than avoiding it. when it comes to culture and happiness, we should be using tinypulse to get honest feedback and to fix our “broken windows.” we should not be using tinypulse to win the award of happiest company.

last week tinypulse asked us the happiness question again. i saw a 6 (several actually), and i’m embracing it. a 6 doesn’t mean that the company has serious problems. it’s reminder that there’s room for improvement, which is actually a good thing. things will never be perfect—we’re always a work in progress, and that’s okay.

the post how being named the happiest company made me less happy appeared first on magoosh 2022年足球世界杯举办地.

we’ve written before about our support for the partnership, but it bears repeating:

magoosh is committed to making education more accessible, effective, and enjoyable, and we welcome anything that aligns with that mission. students everywhere —  regardless of economics, geography, or background — deserve access to high quality education and test prep.  but beyond a shared commitment to leveling the education playing field, there are several other reasons that magoosh welcomes khan academy’s new sat prep tools.

a shifting paradigm helps online education

when magoosh was founded, the idea of sitting down in front of a computer to study for a test was still pretty foreign.  most students who could afford test prep paid big bucks to go to test centers and take scheduled classes, while many more who couldn’t afford it bought or borrowed guidebooks and did their best alone.  having so much of the college board’s official material available online can only help bring more students into the (far cheaper) world of digital education.  for an online test prep company, more students studying online is a good thing!

khan academy’s offering will also help to erode the misconception that, when it comes to test prep, quality is correlated with cost.  at magoosh, we’ve known for a long time that it’s possible to help students achieve great score improvements without charging them through the nose, but we still encounter that perception.  khan academy’s free, high quality prep is yet another counterexample we can point to.

better official material makes for better products

anyone who works in test prep can tell you, when it comes to making the best possible practice questions, no one does it better than the test-makers themselves.  at magoosh, when we’re writing questions for our test prep products, we rely heavily on the official guides to inform how we work.  for the first time, the official stuff is available from the college board, en masse, for free, to everyone.

we love this for two reasons.

first, we have more inspiration to draw from, and a better roadmap for how to make our own test-like questions and passages.

but second, and more importantly, students can now have great examples of what the test actually looks like, and will be able to more easily evaluate test prep options against the college board’s standard. more examples of what great prep looks like means better informed students, which will help make our products shine against the competition.

raising expectations for what’s possible in test prep

perhaps the best reason we’re excited for the official study material from khan academy is that it forces everyone in the test prep game — including us! — to step it up.  having good lessons and questions simply won’t be enough to justify charging students, and all test prep providers are going to have to figure out how to create more value in order to ask students to pay.

magoosh’s offerings already go beyond questions and lessons — when students pay for our service, they also get access to our team of test-prep experts.  students can send in questions about anything in our product — lessons, questions, general study tips, or whatever else they need — and a tutor will get back to them as quickly as possible.  our team works hard, 24/7 to make sure that students have what they need.  what’s more, our product team is hard at work building new features and products to help students succeed.

in the end, the only way a business can survive is if its customers find its products valuable.  our students tell us that they love what we do. we’re continually humbled by their overwhelming support and we hope to keep earning that love.  this new free sat prep helps everyone — both students and test prep companies like magoosh — continue to push for bigger and better things.

the post why a test prep company supports khan academy’s free sat prep appeared first on magoosh 2022年足球世界杯举办地.

so this is the story of how we improved student happiness by a/b testing changes to our product with the goal not of optimizing clicks or conversions or revenues, but of maximizing student happiness.  to start, though, i’ll introduce the metric at hand: net promoter score.

nps: our reliable referral indicator

net promoter score is a metric that tells you, on the whole, how willing your customers are to promote your product.  customers are asked on a scale of 1-10 how likely they would be to recommend your product; 9s and 10s are considered “promoters”, 7s and 8s are neutral, and anything below 6 is a “detractor.”  your net promoter score is calculated by subtracting the number of detractors from the number of promoters and dividing by the total number of respondents.  as a result, nps is a percentage somewhere in the range of -100% (all detractors) to 100% (all promoters).  not to brag, but our nps is high.  really high.  apple high.

at magoosh, nps is one of the most important metrics we track — it helps us determine not only whether students like our customer service and user interface, but also how well our products prepare students for their exams.  and most importantly it has been a reliable leading indicator of growth in word-of-mouth referrals — our largest marketing channel.  when nps is high, students talk about magoosh and more people buy it!

historically, we’ve asked students the nps question after they’ve taken their exams (and, importantly, seen their final scores).  we do this because our products prepare students for tests, and, really, the proof is in the pudding.  you can’t fully decide if you’re willing to recommend magoosh for gre prep until you’ve taken the real gre.  the downside is that it can take a while for us to see nps change in response to product changes.  since we’re waiting until after students are done studying to survey them, it can take months between when a student sees a new feature and when she rates our product.

our nps issue: mismatched expectations

because nps is such an important metric to our company, we take changes very seriously.  earlier this year we saw nps for our gmat product dip fairly significantly.  looking into why, we discovered that several passive and detractor students were complaining that they were getting lower scores on their real gmat than they did on their magoosh practice tests.

our algorithm was telling students to expect one score, but, for some, their official reports were coming back lower — obviously a frustrating experience.  these students were still improving their scores significantly, but once you’ve got a 750 in your mind, a 700 seems disappointing!  we determined that we needed to fix our score prediction algorithm to be more accurate, but we were left with a major concern: would an improved algorithm that displayed a lower predicted score be demoralizing for students?  which was worse for customer satisfaction — a lower predicted score while studying, or a disappointing final score after the exam?

the challenge: could we optimize quickly for nps?

normally when we have questions about what works best for conversion or marketing, we run a quick a/b test to determine what works best.  but nps was different — we’d never a/b tested for nps optimization before, and our nps collection survey only went to students after their exams.  it would be months before students who saw the changed algorithm took their exams and we got back nps data.  making a significant change without knowing how it would affect our word of mouth marketing was a big risk.

our solution:  bring nps inside our product

we determined that in order to a/b test the algorithm change, we needed a method for collecting nps data while students were still studying — not just waiting til the end of their exam.  we began using a third-party tool called wootric, which allows us to ask the nps question in our product and analyze the data in real-time.  we then deployed the changed algorithm to half of our gmat students, and we could then match the “likely-to-refer” rating to students in the treatment and control groups.  suddenly nps had a new use case for us — as a powerful, agile product tool.

it turned out that the improved algorithm did not affect student satisfaction while studying with magoosh — nps from both student groups was identical.  knowing this allowed us to roll the change out to all students more quickly.  we were also able to track the students in the a/b test over time, and have seen that post-exam nps for students in the treatment group is a full nine points higher than for the control.

takeaways from a/b testing for nps

1)  include current customers in your optimizable funnel

our goal is always to provide our students with the best possible test prep experience.  but since we’re not able to read minds, it’s not always easy to know if what we’re doing is actually providing a great experience.  it’s easy to think of customer acquisition as a funnel, and to wrap our brains around how to a/b test to optimize that funnel.  but what doesn’t come easily (at least for most startups — and definitely not for magoosh, at first) is to think of current customers as part of an optimizable funnel too.

2)  optimize your products for referrals

if your business is built on recommendations and word-of-mouth, then you really can’t afford not to optimize your products for referrals.  this process has helped us make sure that what we’re doing is making a meaningful difference for students, and has provided us with a useful and repeatable framework for testing future features and products.

3)  focus on agility

shift your thinking on nps from a one-time transactional model to an ongoing and contextual model.  in-product nps tools available today like wootric can help you do this easily, as well as keep track of your a/b test groups.  you can speed up decision making and keep your pulse on customer happiness.

read about it on wootric’s blog too! a/b testing to optimize net promoter score at magoosh

the post how we optimized for student happiness using a/b testing appeared first on magoosh 2022年足球世界杯举办地.