Home Upgrade Search Memberlist Extras Hacker Tools Award Goals Help Wiki Contact

News 

HF Downtime Nov 2020

Well this is going to be the truth about what happened the past 72 hours even though it's going to be a bit embarrassing. 

HF gets attacks, a lot of them, all the time.  I have a lot of automated security and get alerts if I'm away from my computer and HF goes offline.

When I take vacations I do not announce it for security reasons.  Well, it just so happened a reasonable sized attack hit HF just as I was driving to California for the weekend to spend with family and friends.  It's a 5 hour drive and I sat in agony the whole time as HF loaded slowly and I tried to communicate some actions to Xerotic. I was not driving btw.

I arrived in California Thursday afternoon around 3pm but immediately I have something planned till about 6pm.  I know HF is loading slowly and the web socket isn't working and SSL Handshake errors are all over the place.  Unfortunately I didn't have availability to deal with it. 

Note that Xerotic is not a system administrator and his access to HF servers is limited.  He does not have root access nor the skillset (yet) to manage the server.  It's something I hope to guide him more with our test servers before giving him access to the HF servers.

About 6pm hits and I'm back at my hotel room and I whip open my laptop to take a look.  I'd say that I'm a DDOS mitigation specialist because that's one of my main tasks running HF and I've been doing it for 14 years. 

This is where it gets embarassing for me.  I normally don't need to use my laptop on trips.  I always bring it but rarely have needed it. My automated  DDOS protection has been fairly strong. But I always carry with me my credentials for access.  When I finally did begin working with the laptop I was having trouble logging into the server as root.  Root access is very much required to deal with a DDOS attack.  I had only about 2 hours Thursday night before our dinner plans.  I was not able to login as root. 

Then on Friday morning I woke up extra early to get it sorted out.  I got very close but again, did not gain root access in the two hour window I had before my scheduled events.  It was really frustrating for me.  I was kicking myself for not 100% confirming my access and setting up the laptop before leaving for vacation.  I'm certain given enough time I'd have gotten into the server but it was unfair to my family and friends to also ruin their plans. 

By Friday afternoon I made the decision to turn HF off for the weekend until I returned Sunday to properly deal with the issue. 

It took me about 15 minutes to normalize HF.  Took about 10 minutes more for attacks to resume and even less time for me to stop the attack.  One person is claiming responsibility and trying to blackmail me for 1 BTC.  Another person on social media claims HF has a RAT on the server.  I'm sure other rumors were spread during the downtime.

1. No RAT on the server. That's ridiculous because you don't install RAT's into a LINUX server.  You install a SHELL.

2. Not gonna ever pay any ransom.

3. Attack was shit and easily stopped once I was in the server. 

4. I don't care about the identity of the attacker nor about their motivation.  I'm only ever concerned about keeping HF online.

5. HF is secure.  Our server is not compromised. All data is safe and also backups exist off-server.

I assume the attacker will continue attacking.  I don't really care.  I'll figure it out and adjust and we'll continue business as usual.  The ONLY reason we were effected was because I was on vacation.  DDOS attacks are very normal part of my day. 

On Friday I had Xerotic extend all HF paid advertising (Banners, Stickies, Sales Tags, and Featured Forum) for 3 days.  So anyone with advertising should not be upset over lost time. 

Meh, back to business as usual. I certainly want to apologize to everyone for being an idiot and locking myself out of server.  Really noobish thing to do. It won't happen again.

Thank you.
I am glad HF is back online, this was an interesting read.

Thanks for all the work.
That vacation sounds like mental torcher, but glad to be back online.
[Image: Lw5BVVZ.gif]
Very very interesting read for sure, i know running a server and website like this can be really stressful. Take a deep breath and relax for a second. I appericate you posting and giving us this to read.
Interesting thread, thanks for the update
We'll it's good to have the website backup, lets hope these people stop attacking.
"Religion is regarded by the common people as true, by the wise as false, and by the rulers as useful"

- Seneca
It's fine no worries these noobs wont effect how we feel about the site, been on here for 10 years 2 days downtime is nothing, as frustrating as it is. A good little break for you and for us. Hope your vacation wasn't ruined and you had some fun with your family.
This will definitely put members at ease. Hope the attacker gets hit by karma and stops the DDOS and the blackmail.
[Image: 0nouhj.gif]
Discord: artisan0488
Telegram: @art154n
It's good to know that HF is safe and it's never doubtful when it's handled by you, Omni.
Great job though. I although wonder if people would go furious since they missed daily check-in.
I nominate Omni for the Dunce award for 1 month for locking himself out of root.
And then an A+ for handling it like a boss.
Quote:That vacation sounds like mental torcher, but glad to be back online.


I handled it well imho. Once HF was offline I was just accepting of it until I got back to deal with it. Not even an hour to take care of this. Had I gained root from my laptop I could have easily dealt with this within two hours (I'm slower on the laptop). Funny thing is that Saturdays are the slowest day of the week on HF. So being offline all day Saturday probably wasn't nearly a big of a deal.

Running HF all these years I've learned how to handle with the various stresses. I know that at any moment something shitty can happen. You guys have no idea the amount of crap I deal with on a regular basis. Nearly all of it is behind the scenes. This is a rare time when members were effected. One day, if I ever stop running HF, I'll be happy to tell some crazy stories.
Nice to see the site up again man
Wow, I can’t believe you had the audacity to take a vacation while HF was being attacked and then decided to continue to pursue activities you already had planned in an attempt to not ruin your friends/families time as well... I see where your priorities lie... (sarcasm)

No seriously, why do you even feel you have to explain yourself to anyone? As long as your shit is dealt with and our credentials are secure, there should be no need to explain anything.

Regardless, thanks for the clarification.
[Image: 1111.gif]
You really did good brother.Thanks for the update.
It's great to see hf online again.
Hoping it didn't reduce the quality of your California trip.


Possibly Related Threads...
Thread / Author Replies Views Last Post
by OmniscientFeb 14, 2025 11:16 AM 0 278 Feb 14, 2025 11:16 AM
Last Post: Omniscient
by OmniscientNov 4, 2024 02:24 AM 14 393 Nov 4, 2024 02:24 AM
Last Post: Wiyek
[News] by OmniscientSep 2, 2024 12:29 AM 24 634 Sep 2, 2024 12:29 AM
Last Post: Blazin