Mar 8, 2011 03:16 AM
(This post was last modified: Mar 8, 2011 02:17 PM by Omniscient.)
OMG. You have got to be kidding me.
Most everyone read this thread:
https://hackforums.net/showthread.php?tid=1112217
We had a bad weekend with our DB server. That thread explains how our DB tables became corrupt and I had to revert to backup. In the thread I eventually explain how I'm getting a new SSD drive, backup NAS, and more ram for the server.
The NAS backup was installed Monday afternoon and I quickly threw a Monday 4am backup onto it. Worked great.
The SSD and ram were going to be added at 10pm PST. It was all scheduled and I was really happy about the new upgrades.
10pm came and the work began. I say we were down 30 minutes before server came back online. My next job was to mount new SSD drive then move the DB datadir to the new drive for the boost in performance.
That's where it began to go wrong. When I attempted a dump I had 2 tables go bad on me. One was a mod log table. The other was the posts table (uh oh). The mod logs I just truncated. I didn't need them anways. The rest of the tables backed up just fine. So I began to try and figure out how to fix my posts table.
Two hours later and not a hint of success I was forced to make the same decision I made yesterday. To revert to the Monday 4am backup. I was even scared it would not work. However I think the NAS saved us. The revert went fine and we're back online.
Fact is I think the drive we have is having hardware failure. I'm sure it's exacerbated by our extremely active website. But I was having I/O errors on just moving the raw mybb_posts.ibd file. It was a 5.3GB file but the mybb_moderator table was tiny.
How I'm going to deal with this is unknown. I'm hoping our new clean drive and NAS will keep us from losing data. But I'm not sure. If more problems continue I'll have to ask datacenter to replace this drive and rebuild this server. That's a potential nightmare for me. I just don't have the time to do that.
My next step is to create a strong backup scenario where the NAS is used twice daily.
I know all this sucks. We lose a day of posts then the next day we lose another 16 hours of posts. I'm rather pissed about it but I'm actually very tired at the moment and need some sleep.
I wish this had gone differently but it is what it is. I feel that if I didn't get that extra drive and NAS today we might have had a much bigger problem on our hands. Yes we lost some data. But some data is better than a lot of data.
So let's get past this and move on. I'm too tired to hold a grudge against the server.
EDIT: Made a video.
Most everyone read this thread:
https://hackforums.net/showthread.php?tid=1112217
We had a bad weekend with our DB server. That thread explains how our DB tables became corrupt and I had to revert to backup. In the thread I eventually explain how I'm getting a new SSD drive, backup NAS, and more ram for the server.
The NAS backup was installed Monday afternoon and I quickly threw a Monday 4am backup onto it. Worked great.
The SSD and ram were going to be added at 10pm PST. It was all scheduled and I was really happy about the new upgrades.
10pm came and the work began. I say we were down 30 minutes before server came back online. My next job was to mount new SSD drive then move the DB datadir to the new drive for the boost in performance.
That's where it began to go wrong. When I attempted a dump I had 2 tables go bad on me. One was a mod log table. The other was the posts table (uh oh). The mod logs I just truncated. I didn't need them anways. The rest of the tables backed up just fine. So I began to try and figure out how to fix my posts table.
Two hours later and not a hint of success I was forced to make the same decision I made yesterday. To revert to the Monday 4am backup. I was even scared it would not work. However I think the NAS saved us. The revert went fine and we're back online.
Fact is I think the drive we have is having hardware failure. I'm sure it's exacerbated by our extremely active website. But I was having I/O errors on just moving the raw mybb_posts.ibd file. It was a 5.3GB file but the mybb_moderator table was tiny.
How I'm going to deal with this is unknown. I'm hoping our new clean drive and NAS will keep us from losing data. But I'm not sure. If more problems continue I'll have to ask datacenter to replace this drive and rebuild this server. That's a potential nightmare for me. I just don't have the time to do that.
My next step is to create a strong backup scenario where the NAS is used twice daily.
I know all this sucks. We lose a day of posts then the next day we lose another 16 hours of posts. I'm rather pissed about it but I'm actually very tired at the moment and need some sleep.
I wish this had gone differently but it is what it is. I feel that if I didn't get that extra drive and NAS today we might have had a much bigger problem on our hands. Yes we lost some data. But some data is better than a lot of data.
So let's get past this and move on. I'm too tired to hold a grudge against the server.
EDIT: Made a video.
