[Phire]

Thanks for letting us know, Happy Valentines Day.

[Leroy Jethro Gibbs]

I may just wait it out and plan to do a full server upgrade in a couple weeks. Really sucks because I was hoping this one would last at least 4-5 years and it's only been about 19 months. It's a lot of work to only last 2 years. Very time consuming.

I can get the datacenter to swap out the drives but I fear it will cause more problems because of the RAID configuration. Lots can go wrong. I'm probably better off with new hardware entirely. I also want to do a new configuration because I'm having some conflicts between services I run which cause other problems which I won't divulge. Being sysadmin does take a lot of my time. It's just work that rarely goes noticed.

I'm sure for now we'll be fine with a few crashes a day. Sucks but honestly, nothing I can do about this immediately.

Hell I don't mind the occasional crashes. Honestly I know how you feel being bitched to about the site fucking up. The IT of my hosting company deals with it all the time lol. Mostly from me but I've learned to understand the shit is very complex and super easy to fuck up.

[LePotato]

Don't worry, I love you really! Happy Valentines bby <3

[RepForums]

it did go offline for me alittle while ago for around 10mins, but not problem after that, hope you have a good valentines day, barring this., plus its going to happen with such a big forum, don't sweat it my dude.

[Nyxuo]

I'm sure I can only speak for myself but I really enjoy the availability and speed of HF daily. I'm sure it requires more work then I personally take into consideration. Hopefully you find the time to sort it all out with as minimal headache as possible. Thanks for updating us and thanks for all the work you put into the site.

[Patzan]

Nah that was last month: https://hackforums.net/showthread.php?tid=5524036

Lol.
And Omni just let us wait some time, I know you love HF and care about it but go and live the day.

[Omniscient]

Reviewing this very carefully. I'm in an odd positition.

1. Hope it resolves after running tools in SU mode to mark/fix bad sectors.

2. Try to disable the raid configuration and go with the single drive.

3. Attempt to replace the bad drive and rebuild the RAID.

4. Build a new server.


The entire morning I'm working on this. The good news is that I'm sort of caught up to where I was in knowledge when this happened initially. I can't claim to be a pro at RAID configurations. It's more of learn-as-you-go situation. RAID normally is a set-it-and-forget-it type of deal. You just pray nothing goes wrong. Sadly, we run a software RAID configuration which isn't nearly as easy to replace a single drive as it is with a hardware RAID config. There are risks to removing a drive, replacing it, then rebuilding from the one good drive that wasn't replaced. If anything goes wrong, I'm immediately forced to build a whole new server from scratch at that very moment. We would be down for maybe 18-48 hours and I'd be frantically working. That's less than ideal.

Likely in the next couple days I'll reboot server into SU mode from IPMI console and see what I can do to mark the bad sectors if I can find them. You can't run test well while actually being online. That may require 2-3 hours of actual downtime though.

If option 1 fails then I may attempt to do option 2 and disable the drive entirely. If only drive one in the RAID config is failing then I assume that by removing it from the server or just disabling it in the RAID configuration may fix our problems. Of course we'll lose redundancy but I do keep backups regularly and on a different drive and partition. This may be a good solution actually.

Option 3 scares me greatly. Really just don't have the balls to try that. So much can go wrong.

And in the end I may just go with option 4 and rent another server and rebuild it from scratch and then move services over. It's less painful than #3, something I've done multiple times, and we do get a hardware upgrade.

What I don't like is being forced to deal with this now. Just didn't want to stop doing what I was doing, which was finally finishing off the automation of paid stickies.

[SoulZeh]

Reviewing this very carefully. I'm in an odd positition.

1. Hope it resolves after running tools in SU mode to mark/fix bad sectors.

2. Try to disable the raid configuration and go with the single drive.

3. Attempt to replace the bad drive and rebuild the RAID.

4. Build a new server.


The entire morning I'm working on this. The good news is that I'm sort of caught up to where I was in knowledge when this happened initially. I can't claim to be a pro at RAID configurations. It's more of learn-as-you-go situation. RAID normally is a set-it-and-forget-it type of deal. You just pray nothing goes wrong. Sadly, we run a software RAID configuration which isn't nearly as easy to replace a single drive as it is with a hardware RAID config. There are risks to removing a drive, replacing it, then rebuilding from the one good drive that wasn't replaced. If anything goes wrong, I'm immediately forced to build a whole new server from scratch at that very moment. We would be down for maybe 18-48 hours and I'd be frantically working. That's less than ideal.

Likely in the next couple days I'll reboot server into SU mode from IPMI console and see what I can do to mark the bad sectors if I can find them. You can't run test well while actually being online. That may require 2-3 hours of actual downtime though.

If option 1 fails then I may attempt to do option 2 and disable the drive entirely. If only drive one in the RAID config is failing then I assume that by removing it from the server or just disabling it in the RAID configuration may fix our problems. Of course we'll lose redundancy but I do keep backups regularly and on a different drive and partition. This may be a good solution actually.

Option 3 scares me greatly. Really just don't have the balls to try that. So much can go wrong.

And in the end I may just go with option 4 and rent another server and rebuild it from scratch and then move services over. It's less painful than #3, something I've done multiple times, and we do get a hardware upgrade.

What I don't like is being forced to deal with this now. Just didn't want to stop doing what I was doing, which was finally finishing off the automation of paid stickies.

Hope it works out for you (US), I mean you've gotta think for the long run we'd be much better off, Maybe.. Hoping that a new server doesn't have the same issue or even worse while doing the transfer.

[DLT.]

Sucks that this pops up when you were on the brink of finishing the automation, sounds like my luck. Well I hope all goes well buddy, thanks for keeping us notified.

[Dredd]

Thanks for keeping us posted, at least you have a rough idea what it is.

---

Reviewing this very carefully. I'm in an odd positition.

1. Hope it resolves after running tools in SU mode to mark/fix bad sectors.

2. Try to disable the raid configuration and go with the single drive.

3. Attempt to replace the bad drive and rebuild the RAID.

4. Build a new server.


The entire morning I'm working on this. The good news is that I'm sort of caught up to where I was in knowledge when this happened initially. I can't claim to be a pro at RAID configurations. It's more of learn-as-you-go situation. RAID normally is a set-it-and-forget-it type of deal. You just pray nothing goes wrong. Sadly, we run a software RAID configuration which isn't nearly as easy to replace a single drive as it is with a hardware RAID config. There are risks to removing a drive, replacing it, then rebuilding from the one good drive that wasn't replaced. If anything goes wrong, I'm immediately forced to build a whole new server from scratch at that very moment. We would be down for maybe 18-48 hours and I'd be frantically working. That's less than ideal.

Likely in the next couple days I'll reboot server into SU mode from IPMI console and see what I can do to mark the bad sectors if I can find them. You can't run test well while actually being online. That may require 2-3 hours of actual downtime though.

If option 1 fails then I may attempt to do option 2 and disable the drive entirely. If only drive one in the RAID config is failing then I assume that by removing it from the server or just disabling it in the RAID configuration may fix our problems. Of course we'll lose redundancy but I do keep backups regularly and on a different drive and partition. This may be a good solution actually.

Option 3 scares me greatly. Really just don't have the balls to try that. So much can go wrong.

And in the end I may just go with option 4 and rent another server and rebuild it from scratch and then move services over. It's less painful than #3, something I've done multiple times, and we do get a hardware upgrade.

What I don't like is being forced to deal with this now. Just didn't want to stop doing what I was doing, which was finally finishing off the automation of paid stickies.

Just out of curiosity do you run the main HF server at your home?

[Vitani's Mate]

It is cool.
Thanks for the heads up.

You always seem to have hf in tip top shape.

[Omniscient]

Just out of curiosity do you run the main HF server at your home?

Of course not. I actually use multiple datacenters behind many firewall layers.

[AlphaIndustries]

Thanks for letting us know and trying to do the best for hf, I'm sure you find a way to fix this. Happy valentines day boss.

[Dredd]

Of course not. I actually use multiple datacenters behind many firewall layers.

Would've been surprised if you did, it's just that you said you was going to rebuild the server, I presumed you had access to it, maybe you use a shared rack somewhere?

[Evanescent]

It is always online when I visit HF.