+- Hack Forums (https://hackforums.net)
+-- Forum: Hack Forums Official Information (https://hackforums.net/forumdisplay.php?fid=1)
+--- Forum: Site News (https://hackforums.net/forumdisplay.php?fid=2)
+--- Thread: Part 2 Of The DBZ Saga (/showthread.php?tid=1112579)
Part 2 Of The DBZ Saga - Omniscient - Mar 8, 2011
OMG. You have got to be kidding me.
Most everyone read this thread:
https://hackforums.net/showthread.php?tid=1112217
We had a bad weekend with our DB server. That thread explains how our DB tables became corrupt and I had to revert to backup. In the thread I eventually explain how I'm getting a new SSD drive, backup NAS, and more ram for the server.
The NAS backup was installed Monday afternoon and I quickly threw a Monday 4am backup onto it. Worked great.
The SSD and ram were going to be added at 10pm PST. It was all scheduled and I was really happy about the new upgrades.
10pm came and the work began. I say we were down 30 minutes before server came back online. My next job was to mount new SSD drive then move the DB datadir to the new drive for the boost in performance.
That's where it began to go wrong. When I attempted a dump I had 2 tables go bad on me. One was a mod log table. The other was the posts table (uh oh). The mod logs I just truncated. I didn't need them anways. The rest of the tables backed up just fine. So I began to try and figure out how to fix my posts table.
Two hours later and not a hint of success I was forced to make the same decision I made yesterday. To revert to the Monday 4am backup. I was even scared it would not work. However I think the NAS saved us. The revert went fine and we're back online.
Fact is I think the drive we have is having hardware failure. I'm sure it's exacerbated by our extremely active website. But I was having I/O errors on just moving the raw mybb_posts.ibd file. It was a 5.3GB file but the mybb_moderator table was tiny.
How I'm going to deal with this is unknown. I'm hoping our new clean drive and NAS will keep us from losing data. But I'm not sure. If more problems continue I'll have to ask datacenter to replace this drive and rebuild this server. That's a potential nightmare for me. I just don't have the time to do that.
My next step is to create a strong backup scenario where the NAS is used twice daily.
I know all this sucks. We lose a day of posts then the next day we lose another 16 hours of posts. I'm rather pissed about it but I'm actually very tired at the moment and need some sleep.
I wish this had gone differently but it is what it is. I feel that if I didn't get that extra drive and NAS today we might have had a much bigger problem on our hands. Yes we lost some data. But some data is better than a lot of data.
So let's get past this and move on. I'm too tired to hold a grudge against the server.
EDIT: Made a video.
* - EatSleepCode - Mar 8, 2011
Don't worry Omni, thanks for updating the site! 👍
* - Рringles - Mar 8, 2011
yahoo open , that whas fast
* - T3h Hack3r - Mar 8, 2011
Glad to see everything is working and functional again, lets hope you don't run into anymore issues.
* - Live - Mar 8, 2011
Forums are up Omni, Gj! You should rly write a book about your happenings with HF server :D
* - christiano - Mar 8, 2011
I was very happy to see HF up again
Good job Omni
* - Flashnox™ - Mar 8, 2011
Good to hear we're back online :)
* - +Awesome - Mar 8, 2011
You work great bro..keep it up..!!
* - Akolt - Mar 8, 2011
Thank you for making us back online =)
* - qwerty156 - Mar 8, 2011
Yeah, i was worried the site was being DDoSed!
* - Cona - Mar 8, 2011
I still love you, Omni.
* - 101 - Mar 8, 2011
Seriously, when HF is down, I have nothing to do.
Glad it's all back now! :)
Thnx Omni! <3
* - Xer0bot - Mar 8, 2011
All my mail is gone . Wonderful .
* - Sweat Heart - Mar 8, 2011
If anything can go wrong, it will.
*Murphy's law
* - Ugleh - Mar 8, 2011
I think even if you couldn't revert back because of an upgrade you would of created some program or asked someone to create something to sift threw everything and convert it for you.
I remember when i was new to the whole sql thing and phpmyadmin wouldnt accept my gzip, i freaked out till i saw i could open the zip and find the sql file right there, saved me from a noob moment, that was my only ever sql problem haha.
but you are running many sites so i can see how a small problem turns out to a big problem.
Great job with the fix, hope no more ever problems.