Going to have to discuss this openly to members.
Cloudflare has an exploit that's being used to bypass their security. It's related basically to CNAME DNS entries. Basically anyone can setup a cname pointing to HF then attack that cname entry and it will 100% bypass the Cloudflare protection scheme. It's fucked up. It's really fucked up.
I knew about this exploit months ago and reported it to them. At the time they basically dismissed it. I removed any cnames in my own account that I knew were being used to attack HF and it resolved the issue. Then about 2 weeks ago we had the weekend with 2 days of extended downtime. I was actually offline spending time with family.
I sent ticket after ticket to Cloudflare and they were basically clueless and the support person I initially had was a moron. Eventually they bumped up my ticket to higher tech and they stopped the attack. At least it appeared that way.
Over the course of the past 2 weeks the CNAME exploit has repeatedly been used as I believe it's now a little more well-known and actively being used as a Cloudflare bypass in DDOS attacks. This has become a real problem of course as unfortunately over time I've changed my protection scheme so much that I rely in-part with Cloudflare.
Because Cloudflare is a reverse proxy setup the traffic hitting my server requires DPI (Deep Packet Inspection) for the attacking IPs to be stopped at the server firewall level. Otherwise it goes to Apache which even if blocked by Apache still gobbles up resources and chokes the service thus making the denial of service complete. This is what has been happening.
Possible combats to this would be to install some sort of DPI alteration to my server firewall. Possible to do but annoying as fuck and very resource (CPU) intense. It's not optimal. My other option is NGINX reverse proxy which doesn't choke and would likely work to stop the types of attacks that have recently occurred. Last option was to double up the protection that Cloudflare offers by using another reverse proxy and WAF setup such as Sucuri.net.
Yesterday attacks resumed taking us offline once again. Another support ticket with Cloudflare was opened. I let them know point blank about the CNAME exploit and I was offline. Hours passed and still no results.
The problem with successful DDOS attacks against HF is that they just embolden attacks, word spreads, and even more attacks occur. So my main objective when HF is under attack is fast and efficient mitigation. Thwarting attacks is a high priority for me. I not only want the uptime but I also want to show attackers that their efforts are wasted. Our uptime has greatly improved because of that.
Back to Cloudflare...so the attack last night persisted. The support ticket was becoming frustrating. A change was required in order to deal with this. The fastest change I could immediately come up with was Sucuri. So that's what I did. At 10pm last night I paid for a new account, changed server configurations, and began the necessary changes to add their protection into our serving scheme.
So far it's been a reasonable success with a few problems. I think we're going to lose NSFW mode. At least in it's current form.
Sucuri allows me to block countries not just challenge them as Cloudflare does. That means a lot of users are 100% blocked. For now the listed allowed countries is relatively small and I'm sure adjustments will be made.
We still run through Cloudflare for now. I can't be 100% sure I will continue to use them or if I'll downgrade my Business account to just Pro. I'm pretty frustrated that for nearly two weeks they haven't fixed this problem and I'm guaranteed 100% uptime. Even worse is their inept responses to me in support tickets. I've lost faith in their service and that's an understatement.
Going forward I may or may not run both services. I may drop the Business CF account into a simple Pro account. I was already planning on a major change to our servers this summer which may include running Nginx on port 80 and reverse-proxy to Apache. Some wonder why I still run Apache and that's because it does have some very useful modules I use including a couple custom ones I wrote. I just don't have the time or patience to deal with the same type of setup with Nginx. But running it as a reverse-proxy should allow me even more control and another layer of protection.
I've had a sleepless night and hoping to do all I can today to get things working as best as possible but changes will occur and odd behaviors will happen. Feel free to report them to me either directly or through PM. I'm likely going to be agitated all day over this. I'm less than pleased at Cloudflare and so far I can already see limitations and problems with Sucuri.
On a final note: Many of you have been reporting to me incorrectly the exploit and what you deem as fixes. Most of these reports don't understand the attack (skids). Many of those don't understand that I already block the traffic at the service level but Apache SUCKS and if you have 5000 IPs coming in it will choke hardcore despite my efforts. And without DPI at the server firewall level the traffic can't be stopped. This was a complete failure of Cloudflare whose protection scheme I've paid for and failed. I put that downtime on them. I made them aware of the problem months ago. Before 99% of those using the exploit even knew it existed.
Thanks to everyone who has assisted and supported HF. Let's hope our uptime gets to 99.9%.
Spoiler (Click to View)
Cloudflare has an exploit that's being used to bypass their security. It's related basically to CNAME DNS entries. Basically anyone can setup a cname pointing to HF then attack that cname entry and it will 100% bypass the Cloudflare protection scheme. It's fucked up. It's really fucked up.
I knew about this exploit months ago and reported it to them. At the time they basically dismissed it. I removed any cnames in my own account that I knew were being used to attack HF and it resolved the issue. Then about 2 weeks ago we had the weekend with 2 days of extended downtime. I was actually offline spending time with family.
I sent ticket after ticket to Cloudflare and they were basically clueless and the support person I initially had was a moron. Eventually they bumped up my ticket to higher tech and they stopped the attack. At least it appeared that way.
Over the course of the past 2 weeks the CNAME exploit has repeatedly been used as I believe it's now a little more well-known and actively being used as a Cloudflare bypass in DDOS attacks. This has become a real problem of course as unfortunately over time I've changed my protection scheme so much that I rely in-part with Cloudflare.
Because Cloudflare is a reverse proxy setup the traffic hitting my server requires DPI (Deep Packet Inspection) for the attacking IPs to be stopped at the server firewall level. Otherwise it goes to Apache which even if blocked by Apache still gobbles up resources and chokes the service thus making the denial of service complete. This is what has been happening.
Possible combats to this would be to install some sort of DPI alteration to my server firewall. Possible to do but annoying as fuck and very resource (CPU) intense. It's not optimal. My other option is NGINX reverse proxy which doesn't choke and would likely work to stop the types of attacks that have recently occurred. Last option was to double up the protection that Cloudflare offers by using another reverse proxy and WAF setup such as Sucuri.net.
Yesterday attacks resumed taking us offline once again. Another support ticket with Cloudflare was opened. I let them know point blank about the CNAME exploit and I was offline. Hours passed and still no results.
The problem with successful DDOS attacks against HF is that they just embolden attacks, word spreads, and even more attacks occur. So my main objective when HF is under attack is fast and efficient mitigation. Thwarting attacks is a high priority for me. I not only want the uptime but I also want to show attackers that their efforts are wasted. Our uptime has greatly improved because of that.
Back to Cloudflare...so the attack last night persisted. The support ticket was becoming frustrating. A change was required in order to deal with this. The fastest change I could immediately come up with was Sucuri. So that's what I did. At 10pm last night I paid for a new account, changed server configurations, and began the necessary changes to add their protection into our serving scheme.
So far it's been a reasonable success with a few problems. I think we're going to lose NSFW mode. At least in it's current form.
Sucuri allows me to block countries not just challenge them as Cloudflare does. That means a lot of users are 100% blocked. For now the listed allowed countries is relatively small and I'm sure adjustments will be made.
We still run through Cloudflare for now. I can't be 100% sure I will continue to use them or if I'll downgrade my Business account to just Pro. I'm pretty frustrated that for nearly two weeks they haven't fixed this problem and I'm guaranteed 100% uptime. Even worse is their inept responses to me in support tickets. I've lost faith in their service and that's an understatement.
Going forward I may or may not run both services. I may drop the Business CF account into a simple Pro account. I was already planning on a major change to our servers this summer which may include running Nginx on port 80 and reverse-proxy to Apache. Some wonder why I still run Apache and that's because it does have some very useful modules I use including a couple custom ones I wrote. I just don't have the time or patience to deal with the same type of setup with Nginx. But running it as a reverse-proxy should allow me even more control and another layer of protection.
I've had a sleepless night and hoping to do all I can today to get things working as best as possible but changes will occur and odd behaviors will happen. Feel free to report them to me either directly or through PM. I'm likely going to be agitated all day over this. I'm less than pleased at Cloudflare and so far I can already see limitations and problems with Sucuri.
On a final note: Many of you have been reporting to me incorrectly the exploit and what you deem as fixes. Most of these reports don't understand the attack (skids). Many of those don't understand that I already block the traffic at the service level but Apache SUCKS and if you have 5000 IPs coming in it will choke hardcore despite my efforts. And without DPI at the server firewall level the traffic can't be stopped. This was a complete failure of Cloudflare whose protection scheme I've paid for and failed. I put that downtime on them. I made them aware of the problem months ago. Before 99% of those using the exploit even knew it existed.
Thanks to everyone who has assisted and supported HF. Let's hope our uptime gets to 99.9%.
Spoiler (Click to View)
![[Image: ebisfS8.png]](https://i.imgur.com/ebisfS8.png)
