july 24, 9:45am pdt
we think we’ve stopped the attack, but we’re continuing the monitor. the site should be working for most students and we’ll continue looking into the issues.
july 24, 9:15am pdt
unfortunately, we’re still experiencing issues and intermittent down time. it looks like someone is flooding our servers with “fake” traffic, also known as a dos attack. our team and partners are trying to determine how to stop the attack.
july 22, 12:00pm pdt
we had an unexpected outage today, and i’m truly sorry. we know how important your studies are and that many of you have upcoming tests or blocked off time today to study, and we let you down. we don’t have all the details, and we’re not 100% sure we’re in the clear, but i wanted to share a little bit about what happened.
first, here’s the timeline (all times pacific daylight time on saturday, july 22nd):
- 2:00am: our site became slow
- 5:30am: the site was entirely down
- 6:30am: the site was back up but very slow again and intermittently down
- 11:00am: the site was back up
we host our servers with amazon via amazon web services. we also use a company in australia to help us work with amazon. they monitor our servers and try to make sure everything’s running smoothly.
so what happened?
there was a problem with the server our database is on. our database is where we store all of your information, such as which questions you’ve answered, when your account expires, etc. every time you submit an answer, flip a flashcard, or take any other action on our site, we store that in our database so we can keep track of your progress and provide that information back to you. the database server had connectivity issues which ultimately impacted your ability to use the site. we have automated backups that should kick in when something like this happens, but unfortunately they didn’t work either. we don’t exactly know why the issue happened and why the backups didn’t work yet, but we’re looking into it.
our engineering team and our partners in australia decided the best course of action would be to completely replace our database and database server. this meant we’d need to take the site down entirely for about 15 minutes while we set up the new server and copied over the new database. we took this action around 10:45am and the site was back up by 11am.
ideally, the site would have never gone down, or if it did, we’d have identified the solution earlier. we’ll look into why this happened and see how we can prevent it in the future. i’m truly sorry for disrupting your studies, and if you want to share your thoughts or need a little extra time on your account or need anything else, let us know by emailing [email protected].
thanks,
bhavin
ceo, magoosh
leave a reply