Site issues

[Scott Greczkowski]

And caused the 502 error.

Yup it turned off the Apache server... while the Nginx server kept on going.

Hopefully the new rules I put in stop that from happening.

 

[Scott Greczkowski]

I am training KE4EST how to reset things if I am away from a computer.

Today I only knew I was down as the server emailed me and told me all he wasn’t running. And I saw it on my watch.

At my job work has increased greatly for me. The person who was over me has left and all his work has been thrown in my lap.

I may have to look into getting us onto Sucuri or Cloudflare if these attacks keep happening.

I will really look into it when I get a chance hopefully by this weekend.


Sent from my iPhone using SatelliteGuys

 

[worstman1]

I just got the 502 error and very slow loads to get to this thread. Android Chrome

 

[Scott Greczkowski]

See above... working on it now.

 

[worstman1]

I just got the 502 error and very slow loads to get to this thread. Android Chrome

But now a minute later everything is moving right along, go figure.

 

[worstman1]

See above... working on it now.

Thanks for all your hard work

 

[Scott Greczkowski]

Ok made another change... will keep an eye on it.

 

[Scott Greczkowski]

So far so good. (Keeping my fingers crossed.)

 

[Tampa8]

BIG Thank-you Scott for all the work you do to keep this site up and running. I really liked the pop-up letting us know also.

 

[charlesrshell]

BIG Thank-you Scott for all the work you do to keep this site up and running. I really liked the pop-up letting us know also.

Second that. Appreciate all you do for us.

 

[TRG]

Aha. Is that why SG slowed to a crawl.

No, it's that 32 bit Win7 computer of yours.

 

[Scott Greczkowski]

Been monitoring the system and found another issue with ElasticSearch, which is the search engine we use here. It was trying to do its monthly rotation of its logs and was running out of memory. I think I have that fixed now as well.

 

[Scott Greczkowski]

Ok... not fixed yet... an we are still getting attacked. But I think I found a temporary work around which will hopefully stop the web server from running.

Lets see how this goes. I have to get to bed soon as I have a important early meeting tomorrow at work. So fingers crossed this works.

 

[Scott Greczkowski]

Good morning,

Well my fix worked for overnight... sort of. It kept things running however the Apache server still kept locking up giving a Connection Refused message a few times overnight. Luckily with the change I made it restarted the Apache server a few seconds after the Connection Refused issues happened. The log show the ModSecurity seems to be the one causing it.

I am in the office early today, had a vendor in doing some work here and I had to let them in and meet with them, so now I am here at my desk and I plan to work on the server again and hopefully fix this mess once and for all.

Because of this the site may be unavailable for a few minutes at a time. I will be recompiling and upgrading our Apache web server, as well as ModSecurity.

I would rather not put us behind a Sucuri setup again as that caused issues itself, not to mention the expense, but if its something we need to do then I will have to do it.

 

[Scott Greczkowski]

Apache has been upgraded and recompiled.

ElasticSearch Index was reporting its health was in the red, so I have deleted the search index and am allowing it to reindex now. Some search features will be unavailable until that completes.

 

[Scott Greczkowski]

Ok one issue I kept seeing in the logs was issues with ElasticSearch, which is our search engine we use here on SatelliteGuys. We were running version 6.8.19 which I have removed and have installed the latest version which is 7.15.0.

The search index is still building as I type this almost 3 million out of the 4.5 million posts have been indexed. As you can imagine this is both CPU and Memory intensive as its doing its indexing. But it should be done shortly within a half hour.

As I am going through the logs and looking at the processes I can see other things I can adjust as well to help with performance.

Thank god for Google. Its helping me find, identify and fix some of these things.

 

[harshness]

they were trying to do this hack a few hundred times a second.

Sounds like a job for Fail2Ban.

 

[Scott Greczkowski]

Sounds like a job for Fail2Ban.

Not quite.

The issue is the attack is trying to use the Xenforo proxy.php to pull bad images from other sites. Mod security sees the proxy.php of our server being the bad guy as its the one that is trying (and failing) to grab the bad images... so it keeps trying to ban our server from the server.

I have put in new rules last night found at owasp-modsecurity-crs/REQUEST-903.9006-XENFORO-EXCLUSION-RULES.conf at v3.3/dev · SpiderLabs/owasp-modsecurity-crs and have been tweaking them over time.

I have been tweaking things all morning, updated APache, Php and ModSecurity, upgraded ElasticSearch and have just removed my cron job which was restarting the services every 45 minutes.... let see how this goes.

 

[charlesrshell]

Site is opening faster now. No longer than 3 seconds so that is much faster for me.

 

[Scott Greczkowski]

Site is opening faster now. No longer than 3 seconds so that is much faster for me.

I optimized a lot of things... so lets hope it stays running this way.