Service Outage

Good morning everyone.

Hurricane Sandy blazed through NYC last night and unfortunately the aftermath includes some crucial network infrastructure that’s powering the appFigures platform.

Here’s what’s going on. We’ll keep updating this post with news as we get them:

First things first – Our team has made it safely through the worst parts last night. Winds are still blowing here but we’re told the storm is behind us.

On to business:

• Last night our data center lost power due to flooding in the area. The data center switched to generator power and everything was still up and running.
• 12:00am EST: Several upstream providers started experiencing generator failures which affected traffic from some ISPs. A large number of members were still able to access both the platform.
• 4:00am: We were able to bring the site and API back up with an infrastructure hack using one of the working peers to handle all traffic. The platform should now be accessible to all members.
• 7:00am: Our data center lost connectivity, most likely due to power failure. Because of flooding the operations team hasn’t been allowed back in to restore power.
• 7:45pm: We’ve just been informed that the data center has been re-opened and the operations team is working with the building’s maintenance crews to get the generator back up.
• 10/31 1:45am: We’re still waiting for the data center’s building maintenance crews to bring the main generator up and restore power to the data center.
• 3:10am: Generator power has been restored to our data center and servers are currently in the process of being started.
• 4:30am: All of our servers have been brought up and we’re currently testing their state to ensure they are ready to accept traffic. Stay tuned.
• 5:15am: We have restored service to all web and API nodes. You should now be able to access the site :]

  We’re working to bring up background services right now and will be sending out email reports for 10/29 shortly.

  We’re investigating the following issues:
  – Non working reviews (classic)
  – Non working reviews (beta)
  – Inability to sync with Google Play
  – Inability to sync with Amazon Appstore

• 6:10am: Bad news… It looks like connectivity was lost just now. We’re not sure what’s the cause just yet.
• 6:50am: We have confirmed that the loss of connectivity is due to generator failure. Engineers are on-site and checking the generator. Fingers crossed it’s something simple.
• 12:15pm: We’re being told a generator parts are being rushed down to the data center right now.
• 5:20pm: Power has been restored to the data center. Yay!

  We’re bringing up services and should have both the site and API running right now.

  Some areas of the site may take a little longer to come up as there are a few services that are still in the process of starting up.

  Keep in mind: Utility power still hasn’t been restored to the data center, so the site is literally running on diesel. Hopefully everything will keep going smoothly but we’re not completely out of the woods just yet.

• 11/3 8:00am: Utility power has been restored to our data center.

We’re really sorry about this everyone! In this day and age catastrophes like hurricane Sandy, while devastating, should not take a platform like ours down.

We really appreciate all the good words you guys are tweeting and posting to us and want you to know we’re doing the best we can to bring everything back up as soon as possible.

While down you can find us on twitter and app.net.