SPS Commerce System Issue Update

  • 4
  • Announcement
  • Updated 5 years ago
  • (Edited)
Archived and Closed

This conversation is no longer open for comments or replies and is no longer visible to community members.

Our primary data center has experienced a major power event.  Our data center provided has restored power, and SPS Commerce technology staff are working to bring systems back online.

SPS Commerce continues to investigate this outage.


We will provide updates on an hourly basis or sooner when we have more information.  Be sure to follow this post if you wish to receive emails as we update this post.
Photo of Paul

Paul, Alum

  • 5,714 Points 5k badge 2x thumb

Posted 5 years ago

  • 4
Photo of tiffany.mcelrath

tiffany.mcelrath

  • 70 Points
I still have no orders in Webforms for today. My customer stated they sent some this morning around 10am CST. Please advise.
Photo of Marie O.

Marie O.

  • 120 Points 100 badge 2x thumb
I do not have orders with today's date either.  I know I have 40+ PO's from Amazon that should be in today's transmissions.  Also if we can view our inbox for webforms, should I presume all documents are working properly now?  Thanks!!
Photo of Tracy

Tracy

  • 60 Points
My customer has also stated they've sent some POs over this morning during the outage. But I have yet to see them (it has been already half an hour since the website was up and running for me). Thanks.
Photo of Amy Tu

Amy Tu

  • 60 Points

I am yet to see orders from today as well.

Thanks,

Photo of Kim

Kim

  • 90 Points 75 badge 2x thumb
I also am waiting for multiple Amazon orders that are yet to be in Webforms.  I have 40 orders according to the Amazon website, yet only 14 are in SPS.
Please advise.
Photo of Paul

Paul, Alum

  • 5,714 Points 5k badge 2x thumb
Hi everyone,

I'll do a bit of recon on these missing orders.  I'd assume it to be related to the outage, but want to verify just in case.  I know the AS2 portion this morning delayed data (this would account for Amazon, Walmart, Target, Lowes).  I'll be reaching out to technology to get additional updates on the dataflow side of things, and dig into each of your accounts to verify nothing is preventing these orders from arriving.

Marie, you are correct that outside of the currently missing data, WebForms should work and the documents within WebForms should be working as they normally would.
Photo of Andrew

Andrew

  • 60 Points
Add us to the list of those whose orders have not rolled through yet. Looking for 26 orders that were sent last night all of which have the doc status of "waiting" Any update on when the backlog of docs will be cleared or do I need to ask our customer to re-roll them?
Photo of Paul

Paul, Alum

  • 5,714 Points 5k badge 2x thumb
One more quick update from the Technology Team,

As of now, service has been restored to the spscommerce.net customer portal.  WebForms is up, but not loading new documents, and sent documents are being temporarily held while the remaining data services are restored.
Photo of Geoff Smith

Geoff Smith

  • 60 Points
Ditech Incorporated hasn't received any of it's orders from Arvin Sango.
Photo of Marie O.

Marie O.

  • 120 Points 100 badge 2x thumb
I don't believe webforms is fully functional - it is not letting us save our work or print labels.  
Photo of Paul

Paul, Alum

  • 5,714 Points 5k badge 2x thumb
Separating to remove others from thread Please reference the new topic here: Not able to load shipping labels or save forms
Photo of Paul

Paul, Alum

  • 5,714 Points 5k badge 2x thumb
Hi All,

Below is the most recent update from the Technology team.

Update:
Application recover is continuing.  Estimated recover time is being modified to 3:30PM CST.  At this time, SPS is receiving customer data, but document transformation is not being completed.  Once all systems are operational, documents that were 'In Transit' last night and today will be processed.  This would include both inbound and outbound data.
Photo of Matt Duranso

Matt Duranso

  • 214 Points 100 badge 2x thumb
It is past 3:30 and now WebForms seems to be down completely again.  Another update would be appreciated, as we really need to be able to send ASNs before we leave for the day.
Photo of Paul

Paul, Alum

  • 5,714 Points 5k badge 2x thumb
Looks as though dataflow has been resumed as of 3:30 and will be slowly throttled (data will not all be sent at once, but rather released in batches).  Matt, I've noticed other customers who are recently having the same issue.  It appears to be intermittent, but definitely something I'll relay back up to the Tech team.  I'm going to log into your account and try from my end.
Photo of Kristin

Kristin

  • 60 Points
I am still waiting for 25 more Amazon orders, so far only 6 are in Webforms.  I need to receive them in MAPADOC so I can do PO Confirmations within 24 hours or Amazon will issue charge backs. 
Photo of Donna

Donna

  • 60 Points

Paul, do you have an ETA when all data should processed?  I am currently only seeing outbound 810's have been processed.  As of 4:25 CST, we have not received any inbound 850's.  When we only have 48 hours from the time a PO has been issued to process the order, schedule trucking and  order to be received by the customer,  losing 8 working hours (so far) is putting us at risk of non-compliance and a charge (to us) for each order not received by HD within 48 hours.   

Photo of Paul

Paul, Alum

  • 5,714 Points 5k badge 2x thumb
Hi All,

The backlog is still being worked through.  I wasn't provided with an ETA unfortunately, so I can't speak to exactly when that will be complete.  I do know they are actively working through that and monitoring the data as it goes through each phase of the system.  I wouldn't expect the backlog to take more than a few hours to be fully completed, but again, I don't have an exact time.  I will monitor this through the evening and continue to update here.
Photo of Jon Grant

Jon Grant

  • 60 Points
Just after 5 PM ET, we received many emails indicating that SPS Commerce was unable to successfully process the transactions that we sent.

Will SPS Commerce be automatically reprocessing these errors, or should I create a case for each transaction to ensure that it gets reprocessed?

Thank you in advance for your guidance.
Photo of Thomas Harbert

Thomas Harbert

  • 92 Points 75 badge 2x thumb
I received the same e-mails for our integrated outbound documents between 5:00 and 5:15om yesterday.   My question is the same.  Do I need to resubmit or is SPS going to reprocess it?
Photo of Paul

Paul, Alum

  • 5,714 Points 5k badge 2x thumb
Good Morning,

We are continuing to reconcile data in error due to the outage. Throughout this data resend, we will provide continual updates.  This will cover those erred documents which you are getting emails for.
Photo of Katrina Sorrentino

Katrina Sorrentino

  • 110 Points 100 badge 2x thumb
Ad Hoc Reporting is giving me this error: JBO-26061: Error while opening JDBC connection.  Is anyone else able to use Ad Hoc Reporting?
Photo of Paul

Paul, Alum

  • 5,714 Points 5k badge 2x thumb
Hi Katrina,

Are you still receiving this error?
Photo of Katrina Sorrentino

Katrina Sorrentino

  • 110 Points 100 badge 2x thumb
I'm not getting this error now, but the page is inoperable, i.e. none of the drop downs are working and if I just click Search without adding any parameters, it doesn't return any data.  I know there are lots of transactions from today.  this is on the EDI Documents Page
Photo of Katrina Sorrentino

Katrina Sorrentino

  • 110 Points 100 badge 2x thumb
Thank you for your response and assistance, Paul!
Photo of Paul

Paul, Alum

  • 5,714 Points 5k badge 2x thumb
Hi Katrina,

I've opened an incident here with technology to take a look into that Ad-hoc tool as that is happening across the board.  I'll update you with the progress on this one as I get more information.
Photo of Paul

Paul, Alum

  • 5,714 Points 5k badge 2x thumb
Just a quick heads up.  They are still actively looking into this one. I'll keep you posted as the status gets updated.
Photo of Paul

Paul, Alum

  • 5,714 Points 5k badge 2x thumb
Hi Katrina,

They've restarted the Ad-hoc Reporting service.  Can you give it a try now and let me know if you are still having issues?
Photo of darex acctsrec

darex acctsrec

  • 110 Points 100 badge 2x thumb
Has this issue been resolved? I'm thinking we are still experiencing a problem with viewing shipping web forms. I'm getting an error message: "

Sorry ...

Access to the requested resource has been denied at this time." Please help asap. We are at month end closing and I cannot bill anything without our shipping documents. Thank you.

Photo of Paul

Paul, Alum

  • 5,714 Points 5k badge 2x thumb
HI Darex, I'll work on that with you at this page. I believe that issue to be unrelated.

This conversation is no longer open for comments or replies.