[VeloCityMods]

This is bad. We're now just crashing repeatedly. This is likely one of the drives.

Ugh, so much work now. Dammit.

Damn, hopefully you get this fixed .

Best of luck!, Fixing the problem.

[Silent Uprising]

I am working on a server which may cause us to go offline for brief periods (10-30 minutes).

Thanks.

Pretty sure it was longer than 30 minutes. Must be pretty serious then since I checked HF this morning and it was down. Checked an hour later still down. Finally back up though. Any estimate of when this will be fixed? Or you don't know until you find the problem?

[Mαxim]

Pretty sure it was longer than 30 minutes.

Thank you Captain Obvious.

[Mi2414]

Thanks for keeping us posted Omni, glad to see you working hard man(:

[Omniscient]

We had an hour downtime this morning because of a kernel crash and a reboot had an issue with restarting one of the important services. I was actually at the gym when this happened and I left early to deal with it.

Now I have updated the server kernel and made some adjustments so that service restarts properly on a reboot. Seems to be working considering we've repeatedly crashed and it comes online once it boots up.

But unfortunately all this is leading to a drive failure which is really a total pain in the ass. We run RAID configuration so there is redundancy and I have backups but this is extremely inconvenient for me and consistently a lot of work.

I have to decide where the hardware failure is so network can take a look. Because if I don't narrow it down they'll take the server offline for much longer than if I narrow it down.

I also have to consider just scrubbing the server and building a new one. It might be faster doing that instead. Because hardware failure can plague a server for weeks.

I'm definitely monitoring this carefully and considering all options. Luckily I don't have anything particular planned personally so I can be available to deal with this as quickly as possible. But members should be aware that we may have a lot more downtime until this is resolved. We may even see extended downtimes of 24-48 hours if things go bad.

[Armada]

Explains the downtime.

Thanks for the update boss.

[Silent Uprising]

Thank you Captain Obvious.

I didn't want to be rude and say it was longer than 30 minutes. I also wanted to contribute to the thread and follow it up with a question to see when this issue will be fixed. I didn't mean to be "Captain Obvious". Just wanted to contribute to the thread. I apologize if that's how it came out as...

[Gary]

I didn't want to be rude and say it was longer than 30 minutes. I also wanted to contribute to the thread and follow it up with a question to see when this issue will be fixed. I didn't mean to be "Captain Obvious". Just wanted to contribute to the thread. I apologize if that's how it came out as...

Ignore Corporal Oblivious, no need to worry about him.

As for the thread, didn't even realize as I slept through it.
Probably is an SSD failing or memory exhaustion

[Omniscient]

Actually, looks like I have narrowed it down to one of the SSD drives in a RAID configuration. Sometimes, I really think it's cool being a sysadmin. :)

I'm going to continue monitoring the situation. If we crash again I'll be able to have enough information to ask the datacenter to replace the single drive. Since we're RAID it should be fairly easy to do with hopefully minimal downtime. But I'm going to wait to see if the crashes continue or if the changes I made will help. Almost surely the drive will need replacement but I'm so happy I actually found which one it is.

[Reece Morgan]

Actually, looks like I have narrowed it down to one of the SSD drives in a RAID configuration. Sometimes, I really think it's cool being a sysadmin. :)

I'm going to continue monitoring the situation. If we crash again I'll be able to have enough information to ask the datacenter to replace the single drive. Since we're RAID it should be fairly easy to do with hopefully minimal downtime. But I'm going to wait to see if the crashes continue or if the changes I made will help. Almost surely the drive will need replacement but I'm so happy I actually found which one it is.

Presumably your running RAID 1 (at minimum) so this isn't like a big data loss issue right?
EDIT: Didn't read your last post. You mentioned redundancy there -_-

[Omniscient]

I've started a ticket with the datacenter. Let's see if I can get the hardware changes scheduled. If we're lucky it's this drive. If not then I'll ask them to replace the RAM or Motherboard (maybe both).

If they won't make the changes or if the changes don't work then I rebuild this server during the week.

['Cynical]

Let's hope the hardware changes go smoothly & solves the issue.
Rather unfortunate if they don't; especially if you have to undergo a complete server rebuild.

GL my man.

[Angelus]

Yea... i was wondering what those random crashes were.

Thanks for getting right to it, shows your dedication as a great admin.

[SoulZeh]

Well good luck man, hopefully you don't have to rebuild but like you said it could probably be somewhat faster.
Also, just expedience a crash while posting this, doesn't seem to be for too long now though.

[Evanescent]

Hope all will be good after to update or move on for other server.