SPS Commerce System Issue Update

  • 4
  • Announcement
  • Updated 5 years ago
  • (Edited)
Archived and Closed

This conversation is no longer open for comments or replies and is no longer visible to community members.

Our primary data center has experienced a major power event.  Our data center provided has restored power, and SPS Commerce technology staff are working to bring systems back online.

SPS Commerce continues to investigate this outage.


We will provide updates on an hourly basis or sooner when we have more information.  Be sure to follow this post if you wish to receive emails as we update this post.
Photo of Paul

Paul, Alum

  • 5,714 Points 5k badge 2x thumb

Posted 5 years ago

  • 4
Photo of Linda Kinkead

Linda Kinkead

  • 60 Points
do you have any updates?  
Photo of Paul

Paul, Alum

  • 5,714 Points 5k badge 2x thumb
Hi Linda,

We are still working on this one. Unfortunately, I don't have any additional updates at this time. I will post here as soon as I get more information from our technology team. 
Photo of Farshad

Farshad

  • 96 Points 75 badge 2x thumb
is there any updates?
Photo of Paul

Paul, Alum

  • 5,714 Points 5k badge 2x thumb
We're still investigating the outage and working on recovering all business impacting applications as quickly as possible.
Photo of Farshad

Farshad

  • 96 Points 75 badge 2x thumb
is it possible to get a notification on when the system is back up.
Photo of Paul

Paul, Alum

  • 5,714 Points 5k badge 2x thumb
Yes, I can add your email to our distribution list for that.  You can also follow this page (in the upper right).  That will give you an email notification when I post here. I will be posting updates and resolution here as we work toward that.
Photo of Marie O.

Marie O.

  • 120 Points 100 badge 2x thumb
Do you have a new ETA?  our IT manager called this morning and they said 2 hours or so.  It's past that now.
Photo of Ron Bonnell

Ron Bonnell

  • 60 Points
We are following this page. Thank you for keeping us updated.
Photo of Paul

Paul, Alum

  • 5,714 Points 5k badge 2x thumb
Hi All,

Based on our internal conversations here, we are projecting an 11AM CST resolution (about 25 minutes from now).  This is later than originally anticipated.  If this time changes(either earlier or later) I will continue to update this thread.  Thank you for your continued patience on this issue.
Photo of Edgar Silva

Edgar Silva

  • 140 Points 100 badge 2x thumb
Thanks Paul for keeping us posted on this !!
Kind regards,
Edgar
Photo of Patricia Geremia

Patricia Geremia

  • 160 Points 100 badge 2x thumb
Is the 11am time frame still a projection? Meaning any minute now?
Photo of Justin DeMers

Justin DeMers

  • 80 Points 75 badge 2x thumb
I guess 11am meant we would have a pretty "our data site is down" page instead of a this website cannot be found page.
Photo of Paul

Paul, Alum

  • 5,714 Points 5k badge 2x thumb
Hi All, 

Here is the next update which I've received from the technology team.


While we expect certain applications to be restored earlier, we are revising full transaction processing and services to be available by 2PM CST.


Background Information:

At 10:13PM CST on Sunday, Sept. 21st SPS Commerce detected that it's primary data center was not accessible. At 10:17PM we received notification from our provider Sungard Availability Systems that there was a power incident in the Sungard St. Paul facility and they were investigating.  We dispatched engineers to go to the location to respond to the situation.

Upon investigation there was a major failure in the utility power coming into the facility that caused a failure of two of the four redundant UPS systems in the facility.  The failure physically damaged the utility power coming into the building which significantly impacted resolution time.  At 5:47AM today, Sungard reported that alternative power had been put in place and UPS, cooling and infrastructure services were coming back online. At 7:11AM, Sungard notified that all four UPS systems were back in normal operating mode.  SPS had already started the system startup procedures as of 6:48AM working through a multi-stage system startup process.

Upon resolution, a full incident report will be provided.
Photo of Rob Hansen

Rob Hansen

  • 62 Points
Thanks for the updates Paul!
Photo of Edward Blanco

Edward Blanco

  • 60 Points
Any Update?

Rio Home Fashions
Photo of Paul

Paul, Alum

  • 5,714 Points 5k badge 2x thumb
Hi All,

Our AS2 services have now been restored.  As these services are being brought up I will continue to update.  WebForms services have not fully been restored as of yet.
Photo of Farshad

Farshad

  • 96 Points 75 badge 2x thumb

thank you for the update Paul.

the sooner this is back up the sooner we can get the Monday rolling...

Photo of Patricia Geremia

Patricia Geremia

  • 160 Points 100 badge 2x thumb
Thanks Paul.
Photo of Tanya C Anzaldo

Tanya C Anzaldo

  • 60 Points
Thanks for the continued updates!

Tanya
CA Custom Fruits
Photo of Patricia Geremia

Patricia Geremia

  • 160 Points 100 badge 2x thumb
I'm all set. Thanks again for all your help.
Photo of Paul

Paul, Alum

  • 5,714 Points 5k badge 2x thumb
Hi All,

Full recovery time still remains 2PM, although, I was able to log into an account internally, but not able to access the inbox yet.  I cannot confirm it will be fully working for everyone, but please feel free to give it a try on your end.  I will continue updating as new information becomes available.
Photo of Patricia Geremia

Patricia Geremia

  • 160 Points 100 badge 2x thumb
Actually I have no orders but I'm not blaming the system. :)
Photo of Farshad

Farshad

  • 96 Points 75 badge 2x thumb
nice, I am able to access and fully functional on our end.
Thank you Paul for all the updates.
Photo of tiffany.mcelrath

tiffany.mcelrath

  • 70 Points
How can I get email notifications when the systems are back up? I am able to log in, but I have no orders in webforms. Do we have an updated ETA yet?
Photo of Paul

Paul, Alum

  • 5,714 Points 5k badge 2x thumb
Hi All,

Looks like WebForms should be working for everyone now.  Our systems team is continuing to make progress as services are brought back online.  I'll keep this thread updated as we get to complete resolution.  Please let me know if you are experiencing similar issues to this morning so they can be reported ASAP. Thank you all for your patience this morning and afternoon!
Photo of Matt Duranso

Matt Duranso

  • 214 Points 100 badge 2x thumb
It was working for a short while and now it's down again. Please let us know of any further updates.
Photo of Mark Mattison

Mark Mattison

  • 82 Points 75 badge 2x thumb
We're starting to get 997 warnings from our trade partners and expected PO's from this morning have not showed up yet. WebForms is still experiencing issues. We're integrated with the trade partners involved but I'm unable to verify documents using WebForms. How soon before you get "caught up" with sending out data and when should I start to worry that data may have been lost?
Photo of Paul

Paul, Alum

  • 5,714 Points 5k badge 2x thumb
Hello all,

We are now current on data flow and are working on recovering any data in error due to the outage. We expect this work to continue through the night.
Photo of Katrina Sorrentino

Katrina Sorrentino

  • 110 Points 100 badge 2x thumb
I am not current on outbound transactions.  I have 49 940 documents that have not been processed at all and another 26 that erred and are waiting to be reprocessed.  Also, the 997's for 940's sent yesterday have yet to be processed.
Photo of xhajizta saman

xhajizta saman

  • 60 Points
Hi Paul,

we have reviewed our account and sales data are still not uploaded on sps commerce so please can you check when the web site will be working correctly

This conversation is no longer open for comments or replies.