Self-referencing scripts?

[Evan]

Within the past 2 days our server has been having some issues with server load. Accessing either the front or the back-end of any of our 200+ Wordpress sites typically results in a 503. There will be a few brief moments where server load is fine and everything works dandy, but it's usually overloaded.

We ran into this issue months ago and was resolved by simply moving our more higher-traffic websites to their own server. The results were immediate. I did note however that if we continued adding websites to the server (which we have), eventually we'll likely run into issues again (can't just keep cramming food down someone's throat and not expect issues at some point). This one felt different though, as it wasn't really a gradual server load increase over time, but rather sudden. I reached out to Siteground support to see if they could spot anything odd going on (our account doesn't have root access, so I can't really dig into things on my own unfortunately, and rely on support for the gritty details). They noted:

I noticed, however, many of your websites (I checked the ones consuming the most resources) are generating a lot of self-referring requests. The requests are coming from the server's IP, which means that you have self-referring scripts on your websites.

That was the most detail I got out of it. When I asked if I could see the full list of websites consuming the most resources, I was told to simply view the Account Stats tool in cPanel. With over 200 websites this isn't really feasible (well, it is, but very time consuming). Where do I even begin to look for these 'self-referencing scripts'? Is there a common culprit? Some broken HTTPs plugin? A malfunctioning security script? It's incredibly vague, and as someone who is primarily a web designer/developer and definitely not a server admin, it's stressing me out because I'm literally the only support guy at work in charge of these clients, and I know come Monday I'll be asked to fix it (on top of the other projects I'm currently tackling, which itself is overwhelming, but I digress). The little research I've done hasn't really turned up any useful results, though I was mainly looking for a Wordpress-related solution, which limited my results.

Fixing some plugin conflict on a single site is one thing, and something I run into from time to time. Along with occasional theme issues and what-not. Those I can handle. An entire server facing this kind of issue - is not. But I have to fix it. Somehow.

So, suggestions?

🦃

 

[adam]

Great question and sort of my area! 😄 I’m going to just throw out the obvious there first.

What’s the PHP version you’re running? Should be 7.2 at least.

Are all the WordPress sites updated to their latest versions and plugins as well?

Some things I can think of that could potentially create a lot of requests from the server itself:

• Stuck updates
• Caching plugins (maybe with a very low cache expiry?)
• Any kind of automatic cron or cron style scripts fetching data or syncing with something
• Security plugins with a firewall on a site that is being heavily targeted
• Any image manipulation plugins chugging their way through a huge amount of images such as Smush
• Among many other possibilities

A few more questions to help diagnose:

• Is there anything showing in your error_log (May have to turn on the wp_debug flag in config)
• Is it cPanel hosting, if so you can check your usage graphs to see if you can identify a pattern of resource usage
• Are all these sites on one account sharing resources or are they sandboxed using WHM/CloudLinux etc. with their own provisions
• Do you know if it’s processes, memory or network that is getting throttled?

 

[Gummibeer]

I know it will look like an affiliate program but it isn't and I only like @m1guelpf service and have seen myself how a static pages solves so much.
Check out A static site for your dynamically-generated website - I bet it will be an amazing test case for Miguel. And it will remove the whole load from your server and send it to a CDN.

Without root debugging is very hard. I would try to get an access log, if not already enabled. And check which resources are requested this much. My first idea is also any malfunctioning or insecure and attacked plugin/theme. But to find the file/plugin generating the requests it's pretty hard without root and also in PHP. The worst case could be to disable plugins and pages one after another and see when the requests stop.

 

[Evan]

Great question and sort of my area! 😄 I’m going to just throw out the obvious there first.

What’s the PHP version you’re running? Should be 7.2 at least.

Are all the WordPress sites updated to their latest versions and plugins as well?

Some things I can think of that could potentially create a lot of requests from the server itself:

• Stuck updates
• Caching plugins (maybe with a very low cache expiry?)
• Any kind of automatic cron or cron style scripts fetching data or syncing with something
• Security plugins with a firewall on a site that is being heavily targeted
• Any image manipulation plugins chugging their way through a huge amount of images such as Smush
• Among many other possibilities

A few more questions to help diagnose:

• Is there anything showing in your error_log (May have to turn on the wp_debug flag in config)
• Is it cPanel hosting, if so you can check your usage graphs to see if you can identify a pattern of resource usage
• Are all these sites on one account sharing resources or are they sandboxed using WHM/CloudLinux etc. with their own provisions
• Do you know if it’s processes, memory or network that is getting throttled?

The majority of sites have SG Optimizer, which automatically sets the PHP version to the most recent stable release (according to SG). At the moment that looks to be 7.1. A quick glance at cPanel shows 7.2 isn't even an option. It goes from 7.1.33 to 7.4, so who knows what's going on with that.

All of these sites are sharing resources on one server. Again there's 210 websites, which for all I know is simply too much for one server to handle. I've told my team several times we should get another server and start moving a few websites over to start evening things out. A scalable option is preferable instead of fixed space/memory, but even a duplicate or smaller version of the current server would be nice.

Checking the error_log is a bit tricky because there isn't just one log to check to see what's causing all this, but instead I'd have to go through all 200+ to see if I can spot some issue.

I did some more work this morning, updating plugins and themes. We have WPMU dev connected to them which is supposed to update them automatically, but it isn't getting all of them for some reason. Sadly I'm still getting overload and 500 errors.

 

[adam]

Blimey, 210 sites with free reign of a servers resources is a recipe for disaster. You should look at moving to a reseller style setup where each has its own resources or use something like CloudLinux to allocate per site.

@Gummibeer had a nice suggestion of exploring generated static files from WP but that will be a long term thing for that many sites!

One thing you could do in the short term is try lowering your PHP settings memory limit and max execution time globally if it is already higher than it needs to be. This will cap scripts memory usage and potentially curb any excessive usage. Unfortunately this will have the side effect of showing errors on scripts that run out of memory but hopefully that would only effect ones that are using more and help you narrow down the problem.

Sites shouldn't be using 7.1 anymore: PHP: Supported Versions as it reached end of life very recently and could be vulnerable to security issues as it won't be patched.

 

[Evan]

So if anyone is curious, I was able to get this sorted out. In short, wp-cron was the culprit. What I told the team:

Well, I don't want to raise any false hope again, but the problem may have been resolved. I believe this may actually be related to the wp_cron functionality we recently brought back into action during our quality checks.

On A2, most of our sites had the WP_CRON function disabled, and had a manual WP_CRON setup in its place for once every hour I believe. WP_CRON fires every time someone visits the website, and controls any scheduled actions (such as backups).

When we transferred the websites from A2 to Siteground, the WP_CRON setting that was disabled was carried over as well, but we didn't create a manual WP_CRON in its place within every cPanel. Thus, the backups weren't firing.

During our recent quality checks we noticed that since WP_CRON was disabled, it wasn't allowing backups to be restored. So we simply re-enabled it and went along our way, not thinking much of it.

Because we re-enabled WP_CRON on a decent chunk of websites within a short amount of time, this may have been the reason behind the server load issues. Again, not to raise false hope, but the server hasn't overloaded since disabling WP_CRON again and setting up a manual one for every 6 hours on basically all websites between A - H, as well as any website that went live between July - now, and I also removed a few old staging sites we aren't using, as well as disabling the CRON on any active staging sites. It's been all green for about 35 minutes straight now.

Eventually this should be done for all current and future websites, as long as the manual CRON jobs actually work.

 

[adam]

Ah, nice job fixing! That makes perfect sense. Glad you got it resolved sounded like a nightmare. 😄

Any kind of automatic cron or cron style scripts fetching data or syncing with something

I'll take that as a win. 😉

 

[Gummibeer]

Good job! But God damn what a buggy kind of implementation of a "Cron" ...
This should get sorted in the WP core. A feature that DDoS your server if you get load on your site is wrong implemented.

 

[Evan]

Yeah, index.php was just constantly being called by the server itself. Guess there was a reason my predecessor disabled cron and setup a manual one after going live with each site. And of course having our uptime monitor go berserk didn't help things either. 🤔

F!ck you cron for being so damn aggressive, but f!ck me for not following past quality procedures...