[VodkaNik]

By creating this thread your probably going to have a wave of kids doing the exploit if not new ones.

Best of luck

[0x1027]

Is it just me that been having this problem atm. When I click the logo or home button. EVERY time I have to wait 6-7 to be re verified. Is that a bug with cloud flare?

[Omniscient]

I just went through my past trouble ticket with Cloudflare. It was in Dec of 2014 I said this to them:

Also I suspected origins were from a cname entry I have with hackforums.com (CF hosted DNS) which aliases to hackforums.net.

If that's the cause and it bypasses the Cloudflare WAF rules then I have to assume this is an exploitable bug as anyone can create a cname at any domain then attack it through that method to bypass the WAF.

Please investigate this.

And...

My concern though is that the cname, which anyone can setup, bypassed the WAF. This is alarming behavior unless because it was on my own CF account that you allowed the bypass.

One would hope that cname or not any traffic hitting the domain would go through the filter. I'm a bit puzzled by this and maybe you can forward this to some engineers for review.

To which they responded....

Understood, and thank you for clarifying. I am discussing this issue with our engineers, and I'll get back to you as soon as we've taken a look.

After that nothing. Obviously 5 months later and it's not fixed and the exploit is more widely known. I'm actually hoping by making this public that Cloudflare manages to address it now and deal with it.

For us, I'm still reviewing the setup we have and I don't like it. Changes will need to be made to fit my requirements. Well at least I got Gauth a handful of other things done first. :( I have a feeling my next couple weeks are going to suck for me with heavy workload.

[N1nj4Styl3]

Omni, have you tried calling them? This might work as it does with PayPal once chargebacks/limits happen.. there's a better communication, you know. Calling them might help I guess. (I know they do not offer support through phone, but there's one in WhoIs information.. I suppose?)

Found this: +1.6503198930

[Syracusia]

There is a fundamental problem in CloudFlare's system that allows these attacks to happen. There current setup of how they manage the domains in correlation with their protection and CDN is great for simple use. Although it needs to be revised so requests under a separate domain which may cname to a victims servers, still SHOULD go through the standard DPI associated with the legitimate CF user.

It seems that the CF service simply looks at the requested domain, and applies the relevant CF configurations and optimizations including firewall with that domain record.

Although what seems to be fundamentally wrong, as you noted in your OP, is that if the domain is simply appointing to lets say hackforums.net under cname.attackerswebsite.com then cloudflare will only apply the rules for "cname.attackerswebsite.com" which the attacker can set to no security.

I believe that is the issue, which they MUST fix otherwise their service is essentially useless.

I will tweet to the owner, he currently follows me on Twitter.

---

Omni, have you tried calling them? This might work as it does with PayPal once chargebacks/limits happen.. there's a better communication, you know. Calling them might help I guess. (I know they do not offer support through phone, but there's one in WhoIs information.. I suppose?)

Calling them, MAY be effective, although he has illustrated that the support system is futile.

Charging back is just ignorant.

The only effective option at this point would be to go public, as Omni has done on this forum with the issue.

Perhaps a blog post on a major security website would make a call to action for CF.

Negative publicity towards CF will bring their attention to the issue. As they care about the $, you must remember CF is a business.

---

I just went through my past trouble ticket with Cloudflare. It was in Dec of 2014 I said this to them:

And...

To which they responded....

After that nothing. Obviously 5 months later and it's not fixed and the exploit is more widely known. I'm actually hoping by making this public that Cloudflare manages to address it now and deal with it.

For us, I'm still reviewing the setup we have and I don't like it. Changes will need to be made to fit my requirements. Well at least I got Gauth a handful of other things done first. :( I have a feeling my next couple weeks are going to suck for me with heavy workload.

On a second thought, I have an idea for how we can stop these attacks.

I assume it is some form of XML-RPC or other Layer 7 attack.

Since all of the attacks are coming from example.attackerswebsite.com they would be easy to isolate.

You could then make a script that would grab ALL of the IPs with that criteria, as they would only be malicious connections.

You could then firewall the IPs upstream by contacting your DC, or even at the layer 4 which wouldn't use too much CPU time.

[Figm3ntation]

Sorry, I'm not a professional or anything and I don't really understand everything but why not replace Apache with NGINX? NGINX isn't that hard to learn and setup plus the ability to use it as a firewall.

Have you tried Incapsula?

[Omniscient]

Although what seems to be fundamentally wrong, as you noted in your OP, is that if the domain is simply appointing to lets say https://hackforums.net under cname.attackerswebsite.com then cloudflare will only apply the rules for "cname.attackerswebsite.com" which the attacker can set to no security.

My guess exactly. And in December when I pointed it out they could have looked into it. This week when I spelled it out for them they SHOULD have looked into it...they didn't. They should have corrected this by now. They haven't.

I'm fairly certain that eventually they will and the sooner the better. I'm just rather disappointed that I was forced to use another service for a service I pay Cloudflare for.

Since all of the attacks are coming from example.attackerswebsite.com they would be easy to isolate.

Isolating them isn't the problem. The problem is what to do with the IPs and block it before it hits the service level (Apache). I outlined in the first post that I have 2 basic options. Setup DPI (deep packet inspection) so I can firewall block at my server, which because all traffic come from Cloudlfare a requires special setup. Or to run Nginx as a reverse proxy on port 80 and send legit traffic to Apache on another port. Likely the Nginx reverse proxy will be easier to setup and I'm guessing by the amount of traffic that it will be effective enough to stop the attacks effectiveness.

This was my biggest fear when I moved to Cloudflare. That I would have to rely on their protection scheme because it's either you trust and accept their packets or your site goes offline. Up till now they've done a reasonably good job. I have no idea why they are dropping the ball now. I do notice that some of the people I speak with in Support do not seem as knowledgeable.

btw I already run Nginx on the server for our static content. Unfortunately had to disable it because Sucuri will only secure port 80. I just know I have a lot of work in front of me to get HF running as I need.

[Syracusia]

My guess exactly. And in December when I pointed it out they could have looked into it. This week when I spelled it out for them they SHOULD have looked into it...they didn't. They should have corrected this by now. They haven't.

I'm fairly certain that eventually they will and the sooner the better. I'm just rather disappointed that I was forced to use another service for a service I pay Cloudflare for.

Isolating them isn't the problem. The problem is what to do with the IPs and block it before it hits the service level (Apache). I outlined in the first post that I have 2 basic options. Setup DPI (deep packet inspection) so I can firewall block at my server, which because all traffic come from Cloudlfare a requires special setup. Or to run Nginx as a reverse proxy on port 80 and send legit traffic to Apache on another port. Likely the Nginx reverse proxy will be easier to setup and I'm guessing by the amount of traffic that it will be effective enough to stop the attacks effectiveness.

This was my biggest fear when I moved to Cloudflare. That I would have to rely on their protection scheme because it's either you trust and accept their packets or your site goes offline. Up till now they've done a reasonably good job. I have no idea why they are dropping the ball now. I do notice that some of the people I speak with in Support do not seem as knowledgeable.

Ah, I see a problem now with my potential solution of isolating the attacking IP addresses. Even though it bypasses the CF protection, they would still be coming from CF ip adresses so you can't simply block all connections from the malicious IP addresses. As they would simply be under the CF server's IPs.

What I was thinking, before realizing that was that we could contact your DC and have them just plain block the connections from those IP address before they reach your server, but I see now that is impossible.

CF, is actually HELPING with the attaack in this instance as it makes the attacks unable to be fire walled by IP address.

To firewall the attacks, every request must be analyzed which would be very costly, and something that CF SHOULD be doing.

The NGINX solution would be effective in moving the server load from your main content server, although I would hope that we wouldn't have to go to such a measure for a problem that CF should be able to resolve.

What happens when they start attacking your NGINX reverse proxy? If you set up some form of DPI software/firewall here it would also eat a ton of CPU resources on the nginx machine.

Which wouldn't be a problem, but it would just be expensive to have another server running when you are paying for a service which should be doing this already.

Also that would be another step of latency that we shouldn't have.

CF is supposed so speed things up, not allow hackers to get whitelisted IP address attacks on us and then make us add another relay.

The most effective solution at this point, is to make the issue known by some big security blogs/news.

One other option is to make the nginx server in the same DC as your content server(s) then relay the clean data over a local network for less latency increase.

[Syracusia]

Sorry, I'm not a professional or anything and I don't really understand everything but why not replace Apache with NGINX? NGINX isn't that hard to learn and setup plus the ability to use it as a firewall.

Have you tried Incapsula?

What the fuck? Simply using NGINX wouldn't solve this problem.

It's not an exploit manifesting from the apache server.

It's the fact that the server is having to process thousands of malicious requests every second.

[Vipеr]

Damn Omni works way too hard for us, thanks man.

[SIM-297877]

I am not sure i will ever understand the point in DDoS attacks. Anyways good luck, hope CF gets there shit fixed.

[Syracusia]

I am not sure i will ever understand the point in DDoS attacks. Anyways good luck, hope CF gets there shit fixed.

The point is that the website is offline, which is what the hackers want.

[Omniscient]

CF, is actually HELPING with the attaack in this instance as it makes the attacks unable to be fire walled by IP address.

Essentially true. Because I can't stop an attack with my server firewall without special alterations. The benefit of Sucuri is that they have the DPI setup already and already do an Nginx reverse proxy.

The NGINX solution would be effective in moving the server load from your main content server, although I would hope that we wouldn't have to go to such a measure for a problem that CF should be able to resolve.

I'm giving them another week to resolve this. I updated my ticket with them already. Unfortunately it's the weekend now so I may not get results for a few days.

What happens when they start attacking your NGINX reverse proxy?

Nginx handles more connections. At least it should. And I can block the IPs and certain attack patterns at the proxy level.

What the fuck? Simply using NGINX wouldn't solve this problem.

Technically it should based on the size and patterns of these attacks I'm fairly confident I could actually mitigate the attacks.

The way I have Apache setup is kind of retarded but it's just part of my legacy setup that I haven't had time to fully alter. I plan to redo the entire backend this summer (coming up fast). This whole problem is ing up my timeframe to get the work done.

[Syracusia]

Essentially true. Because I can't stop an attack with my server firewall without special alterations. The benefit of Sucuri is that they have the DPI setup already and already do an Nginx reverse proxy.

I'm giving them another week to resolve this. I updated my ticket with them already. Unfortunately it's the weekend now so I may not get results for a few days.

Nginx handles more connections. At least it should. And I can block the IPs and certain attack patterns at the proxy level.

Technically it should based on the size and patterns of these attacks I'm fairly confident I could actually mitigate the attacks.

The way I have Apache setup is kind of retarded but it's just part of my legacy setup that I haven't had time to fully alter. I plan to redo the entire backend this summer (coming up fast). This whole problem is ing up my timeframe to get the work done.

"
Essentially true. Because I can't stop an attack with my server firewall without special alterations. The benefit of Sucuri is that they have the DPI setup already and already do an Nginx reverse proxy."

Yeah I was reading an article by them the other day in which they developed a firewall solution for a customer which was facing similar problems to you.

"
I'm giving them another week to resolve this. I updated my ticket with them already. Unfortunately it's the weekend now so I may not get results for a few days.
"

That should be more than enough time for them, you are and have been very reasonable with them. If it was me I would have dropped them within 24 hours of no resolution.

"
Nginx handles more connections. At least it should. And I can block the IPs and certain attack patterns at the proxy level. "

Well I personally use NGINX for all of my websites due to its optimizations for my non PHP files.

Although technically Apache could be more effective for a website which is mainly PHP based.

As Nginx has to use php-fpm which is another appplication server for the PHP scripts, while apache has built in processing for php.

There is a lot of data which shows that Apache performs better than nginx for delivery of php scripts.

I am looking at a study right now which shows, Apache performed about 40% better than nginx in regards to PHP.

So for a forum, your apache setup isn't as bad as you think. In fact it might be a lot better.

" And I can block the IPs and certain attack patterns at the proxy level.

Technically it should based on the size and patterns of these attacks I'm fairly confident I could actually mitigate the attacks."

Although that would be one benefit of running an nginx website.

Here is a link to an article showing the benefits of nginx vs apache http://blog.a2o.si/2009/06/24/apache-mod...x-php-fpm/

---

" I plan to redo the entire backend this summer (coming up fast). This whole problem is ing up my timeframe to get the work done."

We should take into account the possible benefits of these changes to ensure the changes are actually effective.

As it will take some to develop everything so that the migration is seem less for our users, we must make sure the changes we are making are effective.

We can run some tests over summer and collect performance data.

Also lets do the rework on an alternate server so there is no downtime, once we have tested and ensured everything is working then we can migrate the solution onto our main content server.

---

I would definitely be able to custom develop a firewall system for you over summer

[AlphaDelta]

What the fuck? Simply using NGINX wouldn't solve this problem.

It's not an exploit manifesting from the apache server.

It's the fact that the server is having to process thousands of malicious requests every second.

You don't solve DDoS attacks you mitigate them, which using NGINX over Apache would actually do to some degree.

HF at this time runs on Apache 2.2.22 (Debian) which apparently has some denial of service problems.