Current service status

We are currently having issue with one of the storage nodes in our Xen-cluster. We are working on starting up servers again.

All customers affected by this issue have been contacted by email individually.

Här kan du ladda ner rapporten som går igenom händelseförloppet under störningen den 16-17:e Maj.

Information om driftavbrott 16:e Maj 2013

One of our storage nodes in our mailcluster is offline.
Some customers have problems reaching their mailboxes.
We are working on the problem, should be solved in about 50 minutes

Incoming mail will be queued and delivered when the storage node is up.

Due to a security problem in the linux kernel we are forced to upgrade some of our OpenVZ nodes.
We will upgrade all nodes with this problem to avoid your server to be compromised.
The upgrades will begin today and will probably take 24 - 48 hours to upgrade all nodes.
Your server will be restarted during this time and will not reachable for 5-10 minutes.
Your server will be issued a shutdown command. And when the upgrade is finnished your server will be started again.

We apologize for the problems this may cause you.

Please contact support@glesys.com if you have further questions.

• 14:47Some customers are still experiencing problems. We are working on the issue.
• 15:56Most of the problems are solved buy we are still having issues with some of the nodes. We are working on getting these nodes us as soon as possible.
• 16:52Two nodes are still affected by the problem. All these customers have been contacted by mail.
• 21:54Only one node who has a problem. Further information is sent by email to these customers.
• 05:24Only one node who has a problem. Customers are updated via email.

** 01:42
System reports SAN error.

** 01:54
Affected SAN storage restarted. Services Online

** 04:44
System reports exactly the same error on the XEN SAN cluster.

** 06:41
We have what we think is a hardware error on one of our SAN arrays.
New hardware currently beeing installed.

** 08:02
New hardware is in place and systems seems to be running stable again since one hour.

There was a problem with network internal network traffic between Falkenberg and Stockholm. There was a fiber cut on one of our connections. The other connection had packetloss after this.
We was forced to change MTU size to get the redundant path working again.
Problem is solved.

There was a problem with network traffic in Stockholm due to a router problem.
Traffic where automaticly rerouted.

** 23:27
We have currently a reported problem with storage affecting 20% of our VMware platform.
We are investigating the cause.

** 23:40
Problem is solved. Affected VM´s where restarted.

Our switch board is beeing upgraded affecting telepones in Falkenberg and Stockholm offices.
Please use support (a) glesys.se if you want to contact us during this hour.

One of the fibers connecting our Stockholm and Falkenberg datacenters has been cut.
This is not affecting any of our services at the moment since our redundancy path is still up.

• 16:00The fiber cut is at Kista, Stokholm.
• 19:07Fiber is up, Stokab has done with the fiber splicer.