This article explains how you can protect your WordPress site from attacks using my PHP Firewall WordPress plugin version 1.0.8.
This plugin is a commercial plugin, with one time fee, that it’s absolute worth it. It is not available as free plugin lite in WordPress.org. Only in my Software store. It is also available as PHP Firewall Drupal module.
Once you install the plugin, you click in Enable protection and the Firewall will start to detect attempts to find exploits in your server automatically, and will block the offending IPs.
If you Enable count of blocked requests you’ll see than in few minutes, the Firewall has blocked hundreds of attempts.
In this example, I activated the protection, and after some time it has automatically detected and blocked 118 offending IP Addresses, and has blocked 195 malevolent requests.
In 36 hours the number of blocked requests will get to thousands:
Enabling the count of blocked requests makes a small update in the database to increment the counter, it’s a very lightweight operation, but it is worth it to get an idea of how many malevolent requests the software is blocking. You can enable it for an hour, see how many bad requests you get, and disable it if you want.
Once you activate the protection you will start to see IP Addresses that have been blocked because they attempted a known exploit, and the IPs have been added to the list of Automatic offenders.
This function is known as WAF or Web Application Firewall.
Once an IP is detected as offender, it will get a 403 Forbidden answer from the Server immediately, and the precious resources from the server will be saved (CPU, Memory, Internet bandwidth).
If you want, you can delete the IP from the list to allow it access again (for example, if a colleague was doing a test).
As you see, you will get many IP from attackers blocked in a short window of time.
The reality is that bots are trying exploits against random IP Addresses all the times, and some times your server will go very slow because of all those bots trying to hack into your server, and your visitors will experience slow loading pages or even the server may crash due to all these bots activity. Also, depending on your Cloud provider, you may be paying for Internet Outgoing traffic that is consumed by bots.
So this plugin is a must have, in my opinion as Engineer. That’s why I created it. Because many people was asking for help.
But there are other types of abuse. For example, malicious bots will try to get access to your blog by doing requests with random passwords to wp-login.php every second.
The Firewall cannot block this route by default because is the route that legit users use to login to WordPress. But we have functionalities in the plugin to deal with these attacks (without having to analyse the web server logs, which is a task that System Administrators and Site Reliability Engineers do).
Activate Enable log of non blocked requests for analysis
This will record every allowed request (not the blocked ones), so we can analyse them.
Just scroll down, to Latest requests
As you can see there are IP Addresses that did a POST to /wp-login.php
That means that attempted to login in your site. So, they tried a password to hack your WordPress site.
Those IP Addresses are attempting to login every second or every two seconds.
These kind of attacks are not only bad, as they can get access to your site, they can also degrade the health of your database.
You can investigate where these IP’s come from doing a Whois search, or asking Google: whois 50.87.179.84
So, it’s a hosting from Bluehost / HostGator, a Cloud Provider. Servers from Cloud providers are typically hacked to try to hack other servers.
You can block the IP directly in the PHP Firewall using the custom rules, or as you can see in the information provider, this Cloud Provider has a NetRange / CIDR of 50.87.0.0/16.
A block /16 comprises 65,536 IP Addresses, so you’ll probably get attacks from many different IP’s in from that block. Instead of blocking just the IP that attacked your site, my approach is to block all the IP’s from that block.
So we add to Manual IP deny rules the block:
The reasoning about blocking all the 65,536 IP Addresses from the Cloud provider is that your WordPress site is probably a site for humans. You want humans to read your blog posts, or if it’s an e-Commerce site, you want to sell to humans. You don’t want bots to use the resources of your server or to try to hack your server. So blocking Cloud Provider’s IP ranges, seems like a good strategy.
If you are one of the few WordPress sites with third party integrations that need to be reached from Cloud providers, you can avoid denying those blocks of IP Addresses that your partners use.
We investigate the other IP:
And it’s the same case, so we also block the CIDR Block in the PHP Firewall.
A /22 block comprises 1,024 IP Addresses. So, with two lines in our firewall we have blocked potential malicious requests from 66,560 IP Addresses.
Obviously we cannot analize all the requests we get, but Firewall PHP plugin provides a very useful tool that will display the IP Addresses that did more request to our server in the last 24 hours.
Just click on Show IPs with most requests (last 24h)
The top 50 IP Addresses that performed more requests are displayed:
In this case, the first IP Address belongs to my own IP Address. Which makes sense because WordPress opened in the browser keeps doing requests to the server.
As I activated the log at 17:26:16 and it’s 21:44:10 in around 4 hours and 15 minutes I got an IP Addresses that did 264 requests.
The IP 134.209.183.0 has performed 264 requests that went through.
If I check the logs of the server I see that it was attempting to hack the server (trying user/password on wp-login.php)
grep "134.209.183.0" *.log
As you can see, this IP was requesting 12 requests per second trying to hack into /wp-login.php
And if we do the whois of this IP Address, it belongs to Digital Ocean:
So we block the entire block 134.209.176.0/20 in the PHP Firewall.
Now the server will block any request from the 4,096 in that block range and will not spend time and resources that are intended for your human visitors.
Most of the attacks come from Cloud providers. Normally are servers that got hacked, and they used those servers to try to hack other servers.
PHP Firewall provides great help in protecting your sites, specially if you don’t have a dedicated team of System Engineers that check the logs regularly, and block the offending IP Addresses in the firewalls.
After 12 hours we see more IP Addresses that made thousands of requests:
According to google those IP Addresses belong to Oracle Cloud:
So, we block the CIDR Block: 137.23.0.0/16 (which will block the two offending IP Addresses, as they form part of that block)
In the latest requests we can see how are new IP Addresses scanning for authors in order to try to attempt logins:
As you can see this IP Address is doing 10 requests in the same second trying passwords (POST to /wp-login.php) in order to attempt to hack the WordPress site.
We do a whois search to see who is the owner of that IP Address:
Is from DigitalOcean, the Cloud Provider. We block all the Netblock 139.59.0.0/16
We see a new IP attacking the server, it is attempting two connections per second POST to /xmlrpc.php which tries to perform actions on the site. xmlrpc.php is a legit mechanism from WordPress, for example to register pingbacks, so it is not blocked by default. But hackers use xmlrpc.php to try thousands of passwords at once without passing by the login screen.
If you do not use older mobile apps or specific remote publishing tools that rely on it you sholud block xmlrpc.php
The whois shows that many attacks come from this provider:
We can block the 256 IP Addresses of the class C, so: 78.142.18.0/24
We will also block the requests to /xmlrpc.php using Custom URL patterns:
Later, we see that there are more IPs attacking trying to abuse wp-login.php so we will block them:
We check the IP that is doing so many attempts to hack requesting wp-login.php
And all the range has been flagged for automated malicious activity and brute-force attempts
So we block all the 256 IP (class C) 93.152.221.0/24.
Two days later I check the IPs doing more requests:
I looked at the logs what requests was doing the first IP:
All were attempts to hack the site through /wp-login.php
We check what kind of IP is this, and it’s from a Cloud provider:
So we block the range 141.98.11.0/24 in IP Firewall.
If you have SSH to the server you can block it in the firewall.
For Ubuntu, with ufw that is:
sudo ufw insert 1 deny from 141.98.11.0/24 to any
A video of using WordPress PHP Firewall v. 1.1.0:
PHP Firewall WordPress plugin is a great help, as it blocks most of the attacks automatically, and offers you tools to detect the IPs that do more requests, so you can block them (or the entire range) easily, and to track the latest requests and export them.
Also, the price of the plugin is one time fee per site. Not a recurring payment.
If you want to acquire the Firewall rules that we have been collecting over the years, you can also buy the Firewall rules of Cloud providers from which we received attacks. These can be applied to any Firewall software, for instance Ubuntu ufw, or to my plugin/modules.
I tested Google Cloud when they were in beta, years ago, and I helped their Team to improve the service. And I’ve been using Google Cloud for years.
Here I explain, how an incident, this August, almost made me leave Google Cloud for good. In fact, I migrated many services to other providers. I walk through the steps I took.
The best advice to start with, is, enable Cloud Billing alerts.
Everything started with the alert of Cloud Billing alerting me that half of the budget for the month had been exhausted around the 7th of the month. I checked the panel, and at the beginning I didn’t pay much attention to it, as Billing was indicating that the extra cost was in Compute in a specific region where I had provisioned new instances and more storage and it was just few euros.
Compute Engine Costs are mixed with the bandwidth consumption in the Dashboards which makes no sense:
So, I had to check different dashboards to realise it was a data transfer issue, and honestly, at the beginning I didn’t know where the problem was, but as I wrongly inferred, that the instance generating egress traffic was the one named “ubuntu26-04” I was investigating mainly that.
The first thing to learn that Google Cloud Billing shows the pure Compute and bandwidth usage together. I dislike it, for obvious reasons.
Next day budget for the month was almost gone.
Running the reports I saw “SKU Network Internet Data Transfer Out from Americas to Americas” which was confusing, and initially made me think if it could be the new backups system, sending the backups to my centralised server using the AI compatible agent with the public IP for the API and encryption, but I discarded this very quickly due to the volume of data and I modified the system to use internal IPs instead of the public ones.
The only clear thing at this point was that Internet data transfer (egress) was being sent, and Google makes you pay for that.
I had released several projects recently, like https://read.carlesmateo.com/en Text-to-speech to generate MP3 from text files, or https://sendmeafile.carlesmateo.com/en to send and receive huge files, with resume, checksum verification, AI compatible… So my first thought was that one of the projects or its dependencies could have been compromised, or fanatic AI/crawlers were downloading the sample MP3 files again and again.
I made a mistake when checking Cloud Billing. It indicated that the expenses came from us-central1, so checked the list of instances’ regions and I thought the problem came from a new instance named “ubuntu26-04”.
But I made a mistake, because there are several zones in every region, like us-central1-a, us-central1-b, us-central1-c… My new instance for the new projects was in the zone us-central1-f but I had more than that one instance in the region, only that in different zones. But when I checked the zones for the instances I did not realise that there was more than one instance in the zone us-central1 and I assumed the new instance was the only one in the zone and should be the one generating the egress traffic.
I only had ports 80 and 443 opened in Google’s firewall, so there could be requests to the web server, or that the server got hacked. I analysed the nginx logs for the server and found no huge traffic or big files being requested.
Nowadays bots and crawlers and AI’s generate a crazy amount of traffic, so I had to discard this.
The Billing Dashboard showed that the increment in data sent was costing me around 11€ per day, so it allowed me to start moving services from that server, with relative calm. The forecast for all the August month displayed that the bill would cost me 300€, which was not nice.
In the meantime I hired a VPS with unlimited data and started to migrate the Docker services there as fast as I could with the idea of shutting down the compromise instance to cut the costs.
One of the things that really made me angry, while I was migrating to avoid those over costs, is that I checked again, the next day, the Cloud Billing Reports, the web loaded and informed me that it had changed with news and showed some floating popups pointing how it work and some improvements pushed by the Google Dev Team, and when I continued, the Cloud Billing Report was displaying an unexpected amount of money to pay in August 2026, more than 10 times what I expected, instead of the forecast I saw previously. So it looked like I had to pay that amount for August, and I started to look at what caused that increment of costs quickly, which fas frustrating. And after a while, when I was thinking about stopping the services, I found that Google had changed the page of the reports and the filters were reset to yearly, and the page was showing the already paid costs from 1st August 2025 to 31th August 2026.
Later I reported how frustrating was the experience to Google as “you should not do that” and “users dislike frequent changes”.
Companies are not aware that when they change things, they often annoy the customers a lot.
Years ago I migrated from my long term account with Amazon AWS to Google Cloud Compute Engine cause I was feed up of the problems in Amazon, how many programming errors they had, the changes they did, causing problems, like to the identifiers of the objects (I had older created objects, with shorter hash, and some API calls did not work any more), and how difficult they made simple things. User Interface was bad, and they also lost two of my instances and communicated really poorly (I also reported that to a friend in Amazon and I had some conversations with Amazon’s AWS PMs), and they were over expensive for developers, so I ended migrating to Google Cloud. And now Google is repeating some of the same mistakes.
They are not only expensive. They make things that could be easy, difficult, for the developers.
These big companies force you to stay continuously learning they changes. They offer great solutions for the Enterprise, but for developers and Startups the costs could be overkill.
Probably they are over-kill and over-expensive for developers and Startups.
With Google Cloud I have experienced some technical issues too, as my older instance froze several times and became irresponsive and I had to do a stop from the web console and I had to wait until the command timed out and Google realised there was a problem so it would consider the instance finally shutdown and I could start it again in other non faulty hardware.
If you are a company with a Load Balancer, with health checks, and several front servers, and 20 developers, you can afford to have Web servers KO, as the Load Balancer will handle it nicely, but if you have a single server with WordPress sites, if the instance stops it means that nobody can access those sites.
As an example of costs, for a Google e2-medium (2 vCPUs, 4 GB of Memory and 20 GB of disk) in us-central1 I pay around 30 € per month. And it happens that the forecast for this month in bandwidth was 300 €.
Also storage is around 1€/month per 10GB for standard. So 200GB of regular space is 20 €/month.
For SSD 1.7€/month per 10GB, even tough provides more IOPS, but not more speed. 200GB of space more is 34 € more per month.
I migrated to a provider were I pay 30 € per month for a 8 vCPU and 24 GB of memory and includes 200 GB of NVMe, and I have unlimited bandwidth. Extra storage is also much more cheap. And the static IP is included (for Google you pay around 1.1€ per month).
The key in here was the unlimited bandwidth.
I also could have rented a dedicated server with 64GB of RAM for around 64 €/month, but these comes with some down sides, like if your server experiences a hardware error, you’ll have to wait for the System Administrators from the Cloud Provider to physically replace the components. With a VPS or instance, I can just launch it in a healthy hardware immediately.
As I was migrating services and investigating the origin of the problems, I saw that 500 GB of data were transferred from us-central1 since the 1st of August.
The Billing Report didn’t allow me to filter for IP. It only helped to know the region, us-central1, which helped me to identify the possible services that could be having the problems.
To be honest, I should have located the origin of the problem quickly, but I did so well blocking bots and uneducated crawlers in the past years, that I had no problems for years, and GCE has mad many changes over time, so I did not remember that I could monitor real time network usage of my servers. Most of the servers I troubleshoot I have only SSH access, so I use network tools.
But as I told previously, I mistakenly thought that just one instance was running in us-central1, and that was not the case. They were in different zones of us-central1.
The live traffic inspection in the server with tools like ss, netstat, nethogs, iftop, iotop and ps revealed normal traffic and load.
A second revision from the nginx logs didn’t reveal any problem over time. And the only opened ports were 80 and 443. With the services I considered potentially more vulnerable (as they used python dependencies) migrated to the new VPS and stopped in the “ubuntu26-04” Google server, the usage shown and the forecast for bandwidth was not reduced and it was the same.
One thing I learned, and I suspected it from my tests with https://sendmeafile.carlesmateo.com/en is that the maximum speed that my instances were able to transfer from Google to Internet (egress) was around 50 Mbps per connection.
I didn’t see this in Google Cloud’s documentation, but in my tests transferring data from Google servers using the public interface and from different locations to this google server I consistently achieved always that top speed of around 50 Mbps per connection (around 6.25 MB/s), but when I communicated Google servers using their private interfaces, the speed for a single connection was much more higher. So I assumed that the bandwidth was capped. After seeing the data transferred in the Billing Cloud Report, it looks clear to me that an important part of the server’s bandwidth max was being used. Later, Apache logs, confirmed me many requests at the same exact second.
Google anomalies tool detected nothing despite the huge increment in data transfer and cost variation (8,960% respect July!).
Before taking the decision to migrating to another server one has to think, not only in costs, but also on how trustworthy are the storage systems, the resiliency of the systems, how many times they are down per year… there are surprises that you discover as you go. With the experience.
But honestly, I felt it was not worth the price to continue with Google Cloud and it was conditioning the kind of projects I could start. My architecture is resilient, based in containers and I can deploy it fast. I also have backups.
Google storage is not that super trustable in my opinion (neither Amazon’s). Some times my VMs froze and Google monitor dashboard stops drawing the lines of metrics (like CPU usage), but Google doesn’t catch that the VM is frozen. My guess is that the problem is the storage or compute servers dying.
Having to pay per GB transferred in a time where bots and AI index your sites continuously, and abuse the servers by launching many requests concurrently, generating load, and where bots consume much more data than humans, force you to take architecture decisions and to implement strategies.
Possible solutions like using CDN’s like Cloudflare to be protected against DoS, to cache statics and to cut bandwidth and CPU abuse (requests to origin), using other cache systems or migrating to other Cloud providers that are more developer and Start up friendly, and that have unlimited data transfer, etcetera.
When I was presented with the new Google Billing Reports page and the filter was reset to one year, and I saw the price that should be showing multiplied by 10, I contacted Google Cloud by X. They kindly replied and invited me to contact them privately. I did and we opened a ticket. Around three hours later I got an email from a Google staff, telling me that they will investigate the problem, and giving me an expected response deadline of 5 days.
It was nice, but by the time I received the email I already had found and sorted the problem.
I troubleshoot the instance “ubuntu26-04” but the CPU usage was low, and no permanent connections (a sign that could have indicated that was hacked):
That Looked normal
sudo ss -tunp state established
Then I did something that I should have done before, I checked the metrics from all the different instances in Google Cloud Compute, starting from the one I initially guessed was the one suffering the problems. That machine was clear. No high CPU usage. No high bandwidth usage. Nothing.
And then I saw it.
Through VM Instances I narrowed to the one in us-central1 that was consuming the Internet bandwidth, “instance-1”:
I was checking the wrong server. The consuming excess of egress was from instance-1 that hosted several WordPress sites and it was using Apache2, not nginx.
Some commands I ran:
# Check for the data sent by Apache. This found a lot of traffic. Note: the access.log of all the sites were being analysed
sudo awk '{s+=$10} END {print s/1024/1024/1024 " GB"}' /var/log/apache2/*access.log
# returned 36.4578 GB that's for a day, so something was clearly wrong
# Analyze requests: See what is more heavy. So basically bots trying to hack into the server and some heavy images. Note root home for at least a server is huge (50MB)
sudo awk '{a[$7]+=$10} END {for(u in a) print a[u]/1024/1024" MB\t"u}' /var/log/apache2/*access.log | sort -rn | head -20
54.3218 MB /
21.6173 MB /wp-login.php
17.1676 MB /wp-content/uploads/2022/10/20221001_200930-cut.png
16.5822 MB /wp-content/uploads/2022/10/20221001_201033.png
10.7245 MB /wp-content/uploads/2022/02/python_loop.gif
6.61174 MB /xmlrpc.php
6.60595 MB /page/2/?shopdetail%2FI37266592159
6.60595 MB /page/2/?shopdetail%2FI37256267952
6.60595 MB /page/2/?shopdetail%2FI37256137878
6.60595 MB /page/2/?shopdetail%2FI37256033568
6.60595 MB /page/2/?shopdetail%2FI37224959793
6.60595 MB /page/2/?shopdetail%2FI37214305672
6.60595 MB /page/2/?shopdetail%2FI37203620584
6.60595 MB /page/2/?shopdetail%2FE37266925367
6.60595 MB /page/2/?shopdetail%2FE37242498260
6.60595 MB /page/2/?shopdetail%2FE37207321261
6.60595 MB /page/2/?shopdetail%2FA37259909717
6.60595 MB /page/2/?shopdetail%2FA37259600135
6.60595 MB /page/2/?shopdetail%2FA37256306882
6.60595 MB /page/2/?shopdetail%2FA37256197883
# Analyze requests: See what IP's are requesting more traffic in total. Gotcha
sudo awk '{a[$1]+=$10} END {for(i in a) print a[i]/1024/1024" MB\t"i}' /var/log/apache2/*access.log | sort -rn | head -20
34549.2 MB 74.7.241.60
1491.36 MB 116.202.235.23
287.651 MB 3.41.188.33
232.362 MB 74.7.227.160
199.987 MB 18.97.9.169
183.748 MB 66.249.74.42
84.1681 MB 114.119.140.64
78.3209 MB 114.119.149.169
70.5166 MB 114.119.156.46
67.8224 MB 114.119.150.29
66.7544 MB 114.119.145.237
58.7278 MB 114.119.147.113
57.1822 MB 114.119.162.62
56.0061 MB 66.249.74.43
55.3612 MB 114.119.141.79
21.9245 MB 216.73.217.35
21.2516 MB 104.243.44.93
17.7313 MB 79.116.217.181
17.1676 MB 189.81.71.184
16.5822 MB 205.188.63.181
So we see that this IP 74.7.241.60 is requesting 34.5 GB in a day.
So, according to this IP Info service, the IP is from Microsoft. So apparently, crawlers/IA from Microsoft or running in Microsoft infrastructure, were costing me money because they are abusing, scanning my sites, with many connections concurrently, like horrible soulless machines:
I double checked the IP with another provider:
A command to see a sample of what kind of requests I have from this IP, brings more light:
According to the User-Agent is GPTBot. So apparently a bot running in Microsoft self-identifying as OpenAI were killing causing high usage to my server and to my budget by requesting many pages at the same time, for an old shopdetail page that does not exist in that server (maybe there was a shop years before I got that IP), and spends a lot of bandwidth responding with a heavy page of 6 MB.
In the logs I saw many request at the same precise second.
Asking to Claude it tells me that the range of IPs of the Microsoft attacker IP are not in the publicly published IPs of OpenAI for GPTBot, so apparently somebody using Microsoft IPs is pretending to be a GPT Bot, and it isn’t:
The domain is an old WordPress that has been hacked. Yes, this happens more often that we would like, with WordPress.
I checked the https and it returns a SSL potential warning in the SSL.
So, to summarise, there was a bot from a Microsoft IP, that was requesting many times per second pages to a WordPress site that was compromised and it was returning a large article with many photos, so heavy in terms of data usage.
With Apache2 server stopped, I moved the config symbolic link from /etc/apache2/sites-enabled/ to another folder /etc/apache2/sites-disabled/ that I created.
So basically, when I started Apache, the compromised site will not be served. It will be served by the default site.
I restarted the Apache server and checked that the URLs for that domain were catch by default site and no processing resources of bandwidth was used.
That was to avoid a main site marked as default in the Apache config to attend those requests and waste data.
And then I added to the firewall, to Google Cloud’s Firewall and to the servers in all my instances, to block all the IPs from Microsoft from that range.
I saw that the BGP Prefix is 74.7.0.0/16 so I block all the 65,536 IP Addresses:
ufw insert 1 deny from 74.7.0.0/16 to any
As you can see I inserted this rule on the first position. Don’t just add it with ufw deny from 74.7.0.0/16 to any or otherwise it will go to the last position, after the allow rules. And then it would not be enforced.
Check your rules with:
ufw status numbered
You can block the range in your provider’s Firewall (VPC Firewall in Google Cloud).
If you didn’t have the Ubuntu Firewall active, and want to activate it now, make sure that you have the rules for the SSH connection:
I recommend you to add the rules to a script in git (Infrastructure as Code), so you can replay it another day, and to any server you have.
By default, ufw sets deny by default for rules not specified. You can check it with:
sudo ufw status verbose | head
Bots, IA, crawlers abuse our servers nowadays.
Also the automated bots attempting to hack into the servers.
If you just check your logs, you’ll see many attempts to exploit vulnerabilities in your servers, every day.
I checked for the second IP consuming more bandwidth in the list:
116.202.235.23
It’s from hetzner, a Cloud provider. https://bgp.he.net/net/74.7.0.0/16
Another 65,536 addresses that get blocked:
# Hetzner
ufw insert 1 deny from 116.202.0.0/16 to any
The third IP will be blocked too, even if I didn’t find any information about who owns the block of IPs.
ufw insert 1 deny from 3.41.188.33 to any
The fourth was part of the same block I blocked first, so it’s already blocked.
The fifth IP, I ask to Google Gemini:
The block is 18.97.9.0/24 but I’ll block all the BGP Prefix, all the 16K addresses.
ufw insert 1 deny from 18.97.0.0/18 to any
See what is the traffic from the servers since I blocked the Microsoft range in the Ubuntu Firewall:
As you can see, after I blocked the first abusive IP allegedly from Microsoft with seemingly OpenAI User-Agent falsified, all the egress traffic went down.
I like how Google reacted when I contacted them. And I should have checked the Network traffic bandwidth first. So I decided to keep the WordPress instance in Google Cloud.
But the new instance “ubuntu26-04” I’m migrating it to the new server. What they offer for the price is much better, and the price of the storage really it’s important for some of my projects. The unlimited bandwidth usage in the new provider it’s also key.
The Billing report from Google Cloud doesn’t show the data up to the moment. But the 16th of August I was able to see that from the 12th and absolutely on 13th and 14th the egress Internet cost have almost disappeared.
Not all is perfect. I had to learn the web interface from that provider (I used it in the past, but companies change the user interface very often nowadays, causing headaches to users), the place were my VPS was listed was hidden and difficult to find, and by default the VPS had no firewall enabled. Something that I believe that is basic to have enabled by default.
But I think it will be great for my projects and for my budget, to have all those new resources available (8 vCPU and 24 GB of RAM and 200 GB of NVMe disk for the same cost provided in Google Cloud of 2 vCPU and 4 GB of RAM and 20 GB of standard disk).
In fact, the speed I have detected transferring from/to sendmeafile with the new provider shows a great improvement.
Take one thing in mind: They AI is hungry for contents, and the mega-rich AI providers are making you pay the bill, by scanning your sites non-stop.
The new provider says it provides 3 Gbps of bandwidth for the instance, with no data usage limits (not having to pay for GB transferred). If I had not reacted fast, the forecast for this month with Google for just a single instance was 300€. It is a risk for developers and small companies, to have the possibility to face a huge bill because more frequent every day bots/crawlers/AIs and processes attempting to breach in your servers, cause your server to use a lot of egress data. So it is worth considering using a Cloud Provider with unlimited data plan. Or at least, activate the Billing Alerts to be notified if your costs exceed a certain threshold.
If you have problems with abusive bots degrading your Drupal or WordPress performance, I created a commercial WordPress plugin and a commercial Drupal module that automatically and easily record and block offending IP Addresses.
In this very long session we went through actual errors in a ZFS pool, we check the Kernel, we remove and reinsert the drive, conduct zpool scrub… in the meantime I talked about Rack, Rack Servers, PSU, redundant components, ECC RAM…
I have read a lot of wrong recommendations about the use of Swap and Swappiness so I want to bring some light about it.
The first to say is that every project is different, so it is not possible to make a general rule. However in most of the cases we want systems to operate as fast and efficiently as possible.
So this suggestions try to covert 99% of the cases.
By default Linux will try to be as efficient as possible. So for example, it will use Free Memory to keep IO efficient by keeping in Memory cache and buffers.
That means that if you are using files often, Linux will keep that information cached in RAM.
The swappiness Kernel setting defines what tradeoff will take Linux between keeping buffers with Free Memory and using the available Swap Memory.
# sysctl vm.swappiness
vm.swappiness = 60
The default value is 60 and more or less means that when RAM memory gets to 60%, swap will start to be used.
And so we can find Servers with 256GB of RAM, that when they start to use more than 153 GB of RAM, they start to swap.
Let’s analyze the output of free -h:
carles@vbi78g:~/Desktop/Software/checkswap$ free -h
total used free shared buff/cache available
Mem: 2.9Gi 1.6Gi 148Mi 77Mi 1.2Gi 1.1Gi
Swap: 2.0Gi 27Mi 2.0Gi
So from this VM that has 2.9GB of RAM Memory, 1.6GB are used by applications.
The are 148MB that can immediately used by Applications, and there are 1.2GB in buffers/cache. Does that means that we can only use 148MB (plus swap)?. No, that mean that Linux tried to optimize io speed by keeping 1.2GB of RAM memory in buffers. But this is the best effort of Linux to have performance, for real applications will be also able to use 1.1GB that corresponds to the available field.
About swap, from 2GB, only 27MB have been used.
As vm.swappiness is set to 60, more RAM will be swapped out to swap, even if we have lots available.
As I said every case is different. If we are talking about a Desktop that has NVMe drives, the impact will be low. But if we are talking about a Server that is a hypervisor running VMs and has high usage on CPU and has the swap partition or the swap in a file, that could lead to huge problems. If there is a physical Server with a single spinning drive (or logical unit through RAID), and one partition is for Swap, and the other for mountpoints, and a process is heavily reading/writing to a partition mounted (an elastic search, or a telegraf, prometheus…), and the System tries to swap, then they will be competing for the magnetic head of disk, slowing down everything.
If you take a look on how the process of swapping memory pages from the memory to disk, you will understand that applications may need certain pages before being able to run, so in many cases we get to lock situations, that force everything to wait.
In my career I found Servers that temporarily stopped responding to ping. After a while ping came back, I was able to ssh and uptime showed that the Server did not reboot.
I troubleshooted that, and I saw a combination of high CPU usage spikes and Swap usage.
Using iostat and iotop I monitored what was speed of transference of only 1 MB/second!!.
I even did swapoff and it took one hour to free 4 GB swap partition!.
I also saw swap partition being in a spinning disk, and in another partition of the same spinning drive, having a swapfile. Magnetic spinning drives can only access one are of the drive at the same time, so that situation, using swap is very bad.
And I have seen situations were the swap or swapfile was mounted in a block device shared via network with the Server (like iSCSI or NFS), causing terrible performance when swapping.
So you have to adapt the strategy according to the project.
My preferred strategy for Compute Nodes and NoSQL Databases is to not use swap at all. In other cases, like MySQL Databases I may set swappiness to preferably to 1 or to 10.
The Linux kernel’s swappiness setting defines how aggressively the kernel will swap memory pages versus dropping pages from the page cache. A higher value increases swap aggressiveness, while a lower value tells the kernel to swap as little as possible to disk and favor RAM. The swappiness range is from 0 to 100, and most Linux distributions have swappiness set to 60 by default.
Couchbase Server is optimized with its managed cache to use RAM, and is capable of managing what should be in RAM and what shouldn’t be. Allowing the OS to have too much control over what memory pages are in RAM is likely to lower Couchbase Server’s performance. Therefore, it’s recommended that swappiness be set to the levels listed below.
Another theme, is when you log to a Server and you see all the Swap memory in use.
Linux may have moved the pages that were less used, and that may be Ok for some cases, for example a Cron Service that waits and runs every 24 hours. It is safe to swap that (as long as the swap IO is decent).
When Kernel Swaps it may generate locks.
But if we log to a Server and all the Swap is in use, how can we know that the Swap has been quiet there?.
Well, you can use iostat or iotop or you can:
cat /proc/vmstat
This file contains a lot of values related to Memory, we will focus on:
Paging refers to writing portions, termed pages, of a process’ memory to disk. Swapping, strictly speaking, refers to writing the entire process, not just part, to disk. In Linux, true swapping is exceedingly rare, but the terms paging and swapping often are used interchangeably.
page-out: The system’s free memory is less than a threshold “lotsfree” and unnused / least used pages are moved to the swap area. page-in: One process which is running requested for a page that is not in the current memory (page-fault), it’s pages are being brought back to memory. swap-out: System is thrashing and has deactivated a process and it’s memory pages are moved into the swap area. swap-in: A deactivated process is back to work and it’s pages are being brought into the memory.
Values from /proc/vmstat:
pgpgin, pgpgout – number of pages that are read from disk and written to memory, you usually don’t need to care that much about these numbers
pswpin, pswpout – you may want to track these numbers per time (via some monitoring like prometheus), if there are spikes it means system is heavily swapping and you have a problem.
In this actual example that means that since the start of the Server there has been 508992338 Page Swap In (with 4K memory pages this is 1,941 GB, so almost 2 TB transferred) and for Page Swat Out (with 4K memory pages this is 1,071 GB, so 1 TB of transferred). I’m talking about a Server that had a 4GB swap partition in a spinning disk and a 12 GB swapfile in another ext4 partition of the same spinning disk.
The 16 GB of swap were in use and iotop showed only two sources of IO, one being 2 VMs writing, another was a journaling process writing to the mountpoint where the swapfile was. That was an spinning drive (underlying hardware was raid, for simplicity I refer to one single drive. I checked that both spinning drives were healthy and fast). I saw small variations in the size of the Swap, so I decided to monitor the changes in pswpin and pswpout in /proc/vmstat to see how much was transferred from/to swap.
I saw then how many pages were being transferred!.
I wrote a small Python program to track those changes:
This little program works in Python 2 and Python 3, and will show the evolution of pswpin and pswpout in /proc/vmstat and will offer the average for last 5 minutes and keep the max value detected as well.
As those values show the page swaps since the start of the Server, my little program, makes the adjustments to show the Page Swaps per second.
A cheap way to reproduce collapse by using swap is using VirtualBox: install an Ubuntu 20.04 LTS in there, with 2 GB of less of memory, and one single core. Ping that VM from elsewhere.
Then you may run a little program like this in order to force it to swap:
#!/usr/bin/env python3
a_items = []
i_total = 0
# Add zeros if your VM has more memory
for i in range(0, 10000000):
i_total = i_total + i
a_items.append(i_total)
And checkswap will show you the spikes:
Many voices are discordant. Some say swappiness default value of 60 is good, as Linux will use the RAM memory to optimize the IO. In my experience, I’ve seen Hypervisors Servers running Virtual Machines that fit on the available physical RAM and were doing pure CPU calculations, no IO, and the Hypervisor was swapping just because it had swappiness to 60. Also having swap on spinning drives, mixing swap partition and swapfile, and that slowing down everything. In a case like that it would be much better not using Swap at all.
In most cases the price of Swapping to disk is much more higher than the advantage than a buffer for IO brings. And in the case of a swapfile, well, it’s also a file, so my suspect is that the swapfile is also buffered. Nothing I recommend, honestly.
My program https://gitlab.com/carles.mateo/checkswap may help you to demonstrate how much damage the swapping is doing in terms of IO. Combine it with iostat and iotop --only to see how much bandwidth is wasted writing and reading from/to swap.
You may run checkswap from a screen session and launch it with tee so results are logged. For example:
python3 checkswap.py | tee 2021-05-27-2107-checkswap.log
If you want to automatically add the datetime you can use:
python3 checkswap.py | tee `date +%Y-%m-%d-%H%M`-checkswap.log
Press CTRL + a and then d, in order to leave the screen session and return to regular Bash.
Type screen -r to resume your session if this was the only screen session running in background.
An interesting reflection from help Ubuntu:
The “diminishing returns” means that if you need more swap space than twice your RAM size, you’d better add more RAM as Hard Disk Drive (HDD) access is about 10³ slower then RAM access, so something that would take 1 second, suddenly takes more then 15 minutes! And still more then a minute on a fast Solid State Drive (SSD)…
Here is an easy trick that you can use for adding swap temporarily to a Server, VMs or Workstations, if you are in an emergency.
In this case I had a cluster composed from two instances running out of memory.
I got an alert for one of the Servers, reporting that only had 7% of free memory.
Immediately I checked it, but checked also any other forming part of the cluster.
Another one appeared, had just only a bit more memory than the other, but was considered in Critical condition too.
The owner of the Service was contacted and asked if we can hold it until US Business hours. Those Servers were going to be replaced next day in US Business hours, and when possible it would be nice not to wake up the Team. It was day in Europe, but night in US.
I checked the status of the Server with those commands:
# df -h
There are 13GB of free space in /. More than enough to be safe as this service doesn’t use much.
# free -h
total used free shared buff/cache available
Mem: 5.7G 4.8G 139M 298M 738M 320M
Swap: 0B 0B 0B
I checked the memory, ok, there are only 139MB free in this node, but 738MB are buff/cache. Buff/Cache is memory used by Linux to optimize I/O as long as it is not needed by application. These 738 MB in buff/cache (or most of it) will be used if needed by the System. The field available corresponds to the memory that is available for starting new applications (not counting the swap if there was any), and basically is the free memory plus a fragment of the buff/cache. I’m sure we could use more than 320MB and there is a lot if buff/cache, but to play safe we play by the book.
With that in mind it seemed that it would hold perfectly to Business hours.
I checked top. It is interesting to mention the meaning of the Column RES, which is resident memory, in other words, the real amount of memory that the process is using.
I had a Java process using 4.57GB of RAM, but a look at how much Heap Memory was reserved and actually being used showed a Heap of 4GB (Memory reserved) and 1.5GB actually being used for real, from the Heap, only.
It was unlikely that elastic search would use all those 4GB, and seemed really unlikely that the instance will suffer from memory starvation with 2.5GB of 4GB of the Heap free, ~1GB of RAM in buffers/cache plus free, so looked good.
To be 100% sure I created a temporary swap space in a file on the SSD.
(# means that I’m executing this as root, if you type literally with # in front, this will be a comment)
# fallocate -l 1G /swapfile-temp
# dd if=/dev/zero of=/swapfile-temp bs=1024 count=1048576 status=progress
1034236928 bytes (1.0 GB) copied, 4.020716 s, 257 MB/s
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB) copied, 4.26152 s, 252 MB/s
If you ask me why I had to dd, I will tell you that I needed to. I checked with command blkid and filesystem was xfs. I believe that was the reason.
The speed writing to the file is fair enough for a swap.
# chmod 600 /swapfile-temp
# mkswap /swapfile-temp
Setting up swapspace version 1, size = 1048572 KiB
no label, UUID=5fb12c0c-8079-41dc-aa20-21477808619a
# swapon /swapfile-temp
I check that memory is good:
# free -h
total used free shared buff/cache available
Mem: 5.7G 4.8G 117M 298M 770M 329M
Swap: 1.0G 0B 1.0G
And finally I check that the Kernel parameter swappiness is not too aggressive:
# sysctl vm.swappiness
vm.swappiness = 30
Cool. 30 is a fair enough value.
2022-01-05 Update for my students that need to add additional 16GB of swap to their SSD drive:
This is a shell script I made long time ago and I use it to monitor in real time what’s the total or individual bandwidth and maximum bandwidth achieved, for READ and WRITE, of Hard drives and NMVe devices.
It uses iostat to capture the metrics, and then processes the maximum values, the combined speed of all the drives… has also an interesting feature to let out the booting device. That’s very handy for Rack Servers where you boot from an SSD card or and SD, and you want to monitor the speed of the other (SAS probably) devices.
I used it to monitor the total bandwidth achieved by our 4U60 and 4U90 Servers, the All-Flash-Arrays 2U and the NVMe 1U units in Sanmina and the real throughput of IOC (Input Output Controllers).
I used also to compare what was the real data written to ZFS and mdraid RAID systems, and to disks and the combined speed with different pool configurations, as well as the efficiency of iSCSI and NFS from clients to the Servers.
You can specify how many times the information will be printed, whether you want to keep the max speed of each device per separate, and specify a drive to exclude. Normally it will be the boot drive.
If you want to test performance metrics you should make sure that other programs are not running or using the swap, to prevent bias. You should disable the boot drive if it doesn’t form part of your tests (like in the 4U60 with an SSD boot drive in a card, and 60 hard drive bays SAS or SATA).
You may find useful tools like iotop.
You can find the code here, and in my gitlab repo:
#!/usr/bin/env bash
AUTHOR="Carles Mateo"
VERSION="1.4"
# Changelog
# 1.4
# Added support for NVMe drives
# 1.3
# Fixed Decimals in KB count that were causing errors
# 1.2
# Added new parameter to output per drive stats
# Counting is performed in KB
# Leave boot device empty if you want to add its activity to the results
# Specially thinking about booting SD card or SSD devices versus SAS drives bandwidth calculation.
# Otherwise use i.e.: s_BOOT_DEVICE="sdcv"
s_BOOT_DEVICE=""
# If this value is positive the loop will be kept n times
# If is negative ie: -1 it will loop forever
i_LOOP_TIMES=-1
# Display all drives separatedly
i_ALL_SEPARATEDLY=0
# Display in KB or MB
s_DISPLAY_UNIT="M"
# Init variables
i_READ_MAX=0
i_WRITE_MAX=0
s_READ_MAX_DATE=""
s_WRITE_MAX_DATE=""
i_IOSTAT_READ_KB=0
i_IOSTAT_WRITE_KB=0
# Internal variables
i_NUMBER_OF_DRIVES=0
s_LIST_OF_DRIVES=""
i_UNKNOWN_OPTION=0
# So if you run in screen you see colors :)
export TERM=xterm
# ANSI colors
s_COLOR_RED='\033[0;31m'
s_COLOR_BLUE='\033[0;34m'
s_COLOR_NONE='\033[0m'
for i in "$@"
do
case $i in
-b=*|--boot_device=*)
s_BOOT_DEVICE="${i#*=}"
shift # past argument=value
;;
-l=*|--loop_times=*)
i_LOOP_TIMES="${i#*=}"
shift # past argument=value
;;
-a=*|--all_separatedly=*)
i_ALL_SEPARATEDLY="${i#*=}"
shift # past argument=value
;;
*)
# unknown option
i_UNKNOWN_OPTION=1
;;
esac
done
if [[ "${i_UNKNOWN_OPTION}" -eq 1 ]]; then
echo -e "${s_COLOR_RED}Unknown option${s_COLOR_NONE}"
echo "Use: [-b|--boot_device=sda -l|--loop_times=-1 -a|--all-separatedly=1]"
exit 1
fi
if [ -z "${s_BOOT_DEVICE}" ]; then
i_NUMBER_OF_DRIVES=`iostat -d -m | grep "sd\|nvm" | wc --lines`
s_LIST_OF_DRIVES=`iostat -d -m | grep "sd\|nvm" | awk '{printf $1" ";}'`
else
echo -e "${s_COLOR_BLUE}Excluding Boot Device:${s_COLOR_NONE} ${s_BOOT_DEVICE}"
# Add an space after the name of the device to prevent something like booting with sda leaving out drives like sdaa sdab sdac...
i_NUMBER_OF_DRIVES=`iostat -d -m | grep "sd\|nvm" | grep -v "${s_BOOT_DEVICE} " | wc --lines`
s_LIST_OF_DRIVES=`iostat -d -m | grep "sd\|nvm" | grep -v "${s_BOOT_DEVICE} " | awk '{printf $1" ";}'`
fi
AR_DRIVES=(${s_LIST_OF_DRIVES})
i_COUNTER_LOOP=0
for s_DRIVE in ${AR_DRIVES};
do
AR_DRIVES_VALUES_AVG[i_COUNTER_LOOP]=0
AR_DRIVES_VALUES_READ_MAX[i_COUNTER_LOOP]=0
AR_DRIVES_VALUES_WRITE_MAX[i_COUNTER_LOOP]=0
i_COUNTER_LOOP=$((i_COUNTER_LOOP+1))
done
echo -e "${s_COLOR_BLUE}Bandwidth for drives:${s_COLOR_NONE} ${i_NUMBER_OF_DRIVES}"
echo -e "${s_COLOR_BLUE}Devices:${s_COLOR_NONE} ${s_LIST_OF_DRIVES}"
echo ""
while [ "${i_LOOP_TIMES}" -lt 0 ] || [ "${i_LOOP_TIMES}" -gt 0 ] ;
do
s_READ_PRE_COLOR=""
s_READ_POS_COLOR=""
s_WRITE_PRE_COLOR=""
s_WRITE_POS_COLOR=""
# In MB
# s_IOSTAT_OUTPUT_ALL_DRIVES=`iostat -d -m -y 1 1 | grep "sd\|nvm"`
# In KB
s_IOSTAT_OUTPUT_ALL_DRIVES=`iostat -d -y 1 1 | grep "sd\|nvm"`
if [ -z "${s_BOOT_DEVICE}" ]; then
s_IOSTAT_OUTPUT=`printf "${s_IOSTAT_OUTPUT_ALL_DRIVES}" | awk '{sum_read += $3} {sum_write += $4} END {printf sum_read"|"sum_write"\n"}'`
else
# Add an space after the name of the device to prevent something like booting with sda leaving out drives like sdaa sdab sdac...
s_IOSTAT_OUTPUT=`printf "${s_IOSTAT_OUTPUT_ALL_DRIVES}" | grep -v "${s_BOOT_DEVICE} " | awk '{sum_read += $3} {sum_write += $4} END {printf sum_read"|"sum_write"\n"}'`
fi
if [ "${i_ALL_SEPARATEDLY}" -eq 1 ]; then
i_COUNTER_LOOP=0
for s_DRIVE in ${AR_DRIVES};
do
s_IOSTAT_DRIVE=`printf "${s_IOSTAT_OUTPUT_ALL_DRIVES}" | grep $s_DRIVE | head --lines=1 | awk '{sum_read += $3} {sum_write += $4} END {printf sum_read"|"sum_write"\n"}'`
i_IOSTAT_READ_KB=`printf "%s" "${s_IOSTAT_DRIVE}" | awk -F '|' '{print $1;}'`
i_IOSTAT_WRITE_KB=`printf "%s" "${s_IOSTAT_DRIVE}" | awk -F '|' '{print $2;}'`
if [ "${i_IOSTAT_READ_KB%.*}" -gt ${AR_DRIVES_VALUES_READ_MAX[i_COUNTER_LOOP]%.*} ]; then
AR_DRIVES_VALUES_READ_MAX[i_COUNTER_LOOP]=${i_IOSTAT_READ_KB}
echo -e "New Max Speed Reading for ${s_COLOR_BLUE}$s_DRIVE${s_COLOR_NONE} at ${s_COLOR_RED}${i_IOSTAT_READ_KB} KB/s${s_COLOR_NONE}"
echo
fi
if [ "${i_IOSTAT_WRITE_KB%.*}" -gt ${AR_DRIVES_VALUES_WRITE_MAX[i_COUNTER_LOOP]%.*} ]; then
AR_DRIVES_VALUES_WRITE_MAX[i_COUNTER_LOOP]=${i_IOSTAT_WRITE_KB}
echo -e "New Max Speed Writing for ${s_COLOR_BLUE}$s_DRIVE${s_COLOR_NONE} at ${s_COLOR_RED}${i_IOSTAT_WRITE_KB} KB/s${s_COLOR_NONE}"
fi
i_COUNTER_LOOP=$((i_COUNTER_LOOP+1))
done
fi
i_IOSTAT_READ_KB=`printf "%s" "${s_IOSTAT_OUTPUT}" | awk -F '|' '{print $1;}'`
i_IOSTAT_WRITE_KB=`printf "%s" "${s_IOSTAT_OUTPUT}" | awk -F '|' '{print $2;}'`
# CAST to Integer
if [ "${i_IOSTAT_READ_KB%.*}" -gt ${i_READ_MAX%.*} ]; then
i_READ_MAX=${i_IOSTAT_READ_KB%.*}
s_READ_PRE_COLOR="${s_COLOR_RED}"
s_READ_POS_COLOR="${s_COLOR_NONE}"
s_READ_MAX_DATE=`date`
i_READ_MAX_MB=$((i_READ_MAX/1024))
fi
# CAST to Integer
if [ "${i_IOSTAT_WRITE_KB%.*}" -gt ${i_WRITE_MAX%.*} ]; then
i_WRITE_MAX=${i_IOSTAT_WRITE_KB%.*}
s_WRITE_PRE_COLOR="${s_COLOR_RED}"
s_WRITE_POS_COLOR="${s_COLOR_NONE}"
s_WRITE_MAX_DATE=`date`
i_WRITE_MAX_MB=$((i_WRITE_MAX/1024))
fi
if [ "${s_DISPLAY_UNIT}" == "M" ]; then
# Get MB
i_IOSTAT_READ_UNIT=${i_IOSTAT_READ_KB%.*}
i_IOSTAT_WRITE_UNIT=${i_IOSTAT_WRITE_KB%.*}
i_IOSTAT_READ_UNIT=$((i_IOSTAT_READ_UNIT/1024))
i_IOSTAT_WRITE_UNIT=$((i_IOSTAT_WRITE_UNIT/1024))
fi
# When a MAX is detected it will be displayed in RED
echo -e "READ ${s_READ_PRE_COLOR}${i_IOSTAT_READ_UNIT} MB/s ${s_READ_POS_COLOR} (${i_IOSTAT_READ_KB} KB/s) Max: ${i_READ_MAX_MB} MB/s (${i_READ_MAX} KB/s) (${s_READ_MAX_DATE})"
echo -e "WRITE ${s_WRITE_PRE_COLOR}${i_IOSTAT_WRITE_UNIT} MB/s ${s_WRITE_POS_COLOR} (${i_IOSTAT_WRITE_KB} KB/s) Max: ${i_WRITE_MAX_MB} MB/s (${i_WRITE_MAX} KB/s) (${s_WRITE_MAX_DATE})"
if [ "$i_LOOP_TIMES" -gt 0 ]; then
i_LOOP_TIMES=$((i_LOOP_TIMES-1))
fi
done
This is a great new for scaling performance in the Data Centers. For routers, switches…
And this makes me think about all the Architects that are using Memcached and Redis in different Servers, in Networks of 1Gbps and makes me want to share with you what a nonsense, is often, that.
So the idea of having Memcache or Redis is just to cache the queries and unload the Database from those queries.
But 1Gbps is equivalent to 125MB (Megabytes) per second.
Local RAM Memory in Servers can perform at 24GB and more (24,000,000 Megabytes) per second, even more.
A PCIE NVMe drive at 3.5GB per second.
A local SSD drive without RAID 550 MB/s.
A SSD in the Cloud, varies a lot on the provider, number of drives, etc… but I’ve seen between 200 MB/s and 2.5GB/s aggregated in RAID.
In fact I have worked with Servers equipped with several IO Controllers, that were delivering 24GB/s of throughput writing or reading to HDD spinning drives.
If you’re in the Cloud. Instead of having 2 Load Balancers, 100 Front Web servers, with a cluster of 5 Redis with huge amount of RAM, and 1 MySQL Master and 1 Slave, all communicating at 1Gbps, probably you’ll get a better performance having the 2 LBs, and 11 Front Web with some more memory and having the Redis instance in the same machine and saving the money of that many small Front and from the 5 huge dedicated Redis.
The same applies if you’re using Docker or K8s.
Even if you just cache the queries to drive, speed will be better than sending everything through 1 Gbps.
This will matter for you if your site is really under heavy load. Most of the sites just query the MySQL Server using 1 Gbps lines, or 2 Gbps in bonding, and that’s enough.
First you have to understand that Python, Java and PHP are worlds completely different.
In Python you’ll probably use Flask, and listen to the port you want, inside Docker Container.
In PHP you’ll use a Frameworks like Laravel, or Symfony, or Catalonia Framework (my Framework) :) and a repo or many (as the idea is that the change in one microservice cannot break another it is recommended to have one git repo per Service) and split the requests with the API Gateway and Filters (so /billing/ goes to the right path in the right Server, is like rewriting URLs). You’ll rely in Software to split your microservices. Usually you’ll use Docker, but you have to add a Web Server and any other tools, as the source code is not packet with a Web Server and other Dependencies like it is in Java Spring Boot.
In Java you’ll use Spring Cloud and Spring Boot, and every Service will be auto-contained in its own JAR file, that includes Apache Tomcat and all other Dependencies and normally running inside a Docker. Tcp/Ip listening port will be set at start via command line, or through environment. You’ll have many git repositories, one per each Service.
Using many repos, one per Service, also allows to deploy only that repository and to have better security, with independent deployment tokens.
It is not unlikely that you’ll use one language for some of your Services and another for other, as well as a Database or another, as each Service is owner of their data.
In any case, you will be using CI/CD and your pipeline will be something like this:
Pull the latest code for the Service from the git repository
Compile the code (if needed)
Run the Unit and Integration Tests
Compile the service to an executable artifact (f.e. Java JAR with Tomcat server and other dependencies)
Generate a Machine image with your JAR deployed (for Java. Look at Spotify Docker Plugin to Docker build from Maven), or with Apache, PHP, other dependencies, and the code. Normally will be a Docker image. This image will be immutable. You will probably use Dockerhub.
Machine image will be started. Platform test are run.
If platform tests pass, the service is promoted to the next environment (for example Dev -> Test -> PreProd -> Prod), the exact same machine is started in the next environment and platform tests are repeated.
Before deploying to Production the new Service, I recommend running special Application Tests / Behavior-driven. By this I mean, to conduct tests that really test the functionality of everything, using a real browser and emulating the acts of a user (for example with BeHat, Cucumber or with JMeter). I recommend this specially because Microservices are end-points, independent of the implementation, but normally they are API that serve to a whole application. In an Application there are several components, often a change in the Front End can break the application. Imagine a change in Javascript Front End, that results in a call a bit different, for example, with an space before a name. Imagine that the Unit Tests for the Service do not test that, and that was not causing a problem in the old version of the Service and so it will crash when the new Service is deployed. Or another example, imagine that our Service for paying with Visa cards generates IDs for the Payment Gateway, and as a result of the new implementation the IDs generated are returned. With the mocked objects everything works, but when we deploy for real is when we are going to use the actual Bank Payment. This is also why is a good idea to have a PreProduction environment, with PreProduction versions of the actual Services we use (all banks or the GDS for flights/hotel reservation like Galileo or Amadeus have a Test, exactly like Production, Gateway)
If you work with Microsoft .NET, you’ll probably use Azure DevOps.
We IT Engineers, CTOs and Architects, serve the Business. We have to develop the most flexible approaches and enabling the business to release as fast as their need.
Take in count that Microservices is a tool, a pattern. We will use it to bring more flexibility and speed developing, resilience of the services, and speed and independence deploying. However this comes at a cost of complexity.
Microservices is more related to giving flexibility to the Business, and developing according to the Business Domains. Normally oriented to suite an API. If you have an API that is consumed by third party you will have things like independence of Services (if one is down the others will still function), gradual degradation, being able to scale the Services that have more load only, being able to deploy a new version of a Service which is independent of the rest of the Services, etc… the complexity in the technical solution comes from all this resilience, and flexibility.
If your Dev Team is up to 10 Developers or you are writing just a CRUD Web Application, a PoC, or you are an Startup with a critical Time to Market you probably you will not want to use Microservices approach. Is like killing flies with laser cannons. You can use typical Web services approach, do everything in one single Https request, have transactions, a single Database, etc…
But if your team is 100 Developer, like a big eCommerce, you’ll have multiple Teams between 5 and 10 Developers per Business Domain, and you need independence of each Service, having less interdependence. Each Service will own their own Data. That is normally around 5 to 7 tables. Each Service will serve a Business Domain. You’ll benefit from having different technologies for the different needs, however be careful to avoid having Teams with different knowledge that can have hardly rotation and difficult to continue projects when the only 2 or 3 Devs that know that technology leave. Typical benefit scenarios can be having MySql for the Billing Services, but having NoSQL Database for the image catalog, or to store logs of account activity. With Microservices, some services will be calling other Services, often asynchronously, using Queues or Streams, you’ll have Callbacks, Databases for reading, you’ll probably want to have gradual and gracefully failure of your applications, client load balancing, caches and read only databases/in-memory databases… This complexity is in order to protect one Service from the failure of others and to bring it the necessary speed under heavy load.
Here you can find a PDF Document of the typical resources I use for Microservice Projects.
Few months ago I encountered with a problem with RHEL installer and some of the M.2 drives.
I’ve productized my Product, to be released with M.2 booting SATA drives of 128GB.
The procedure for preparing the Servers (90 and 60 drives, Cold Storage) was based on the installation of RHEL in the M.2 128GB drive. Then the drives are cloned.
Few days before mass delivery the company request to change the booting M.2 drives for others of our own, 512 GB drives.
I’ve tested many different M.2 drives and all of them were slightly different.
Those 512 GB M.2 drives had one problem… Red Hat installer was failing with a python error.
We were running out of time, so I decided to clone directly from the 128GB M.2 working card, with everything installed, to the 512 GB card. Doing that is so easy as booting with a Rescue Linux USB disk, and then doing a dd from the 128GB drive to the 512GB drive.
Booting with a live USB system is important, as Filesystem should not be mounted to prevent corruption when cloning.
Then, the next operation would be booting the 512 GB drive and instructing Linux to claim the additional space.
Here is the procedure for doing it (note, the OS installed in the M.2 was CentOS in this case):
Determine the device that needs to be operated on (this will usually be the boot drive); in this example it is /dev/sdae
Extend the desired LVM partition (lvextend command)
# pvdisplay /dev/sdbm: open failed: No medium found /dev/sdbn: open failed: No medium found /dev/sdbj: open failed: No medium found /dev/sdbk: open failed: No medium found /dev/sdbl: open failed: No medium found --- Physical volume --- PV Name /dev/sdae2 VG Name centos_4602c PV Size 118.24 GiB / not usable 3.00 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 30269 Free PE 0 Allocated PE 30269 PV UUID yvHO6t-cYHM-CCCm-2hOO-mJWf-6NUI-zgxzwc
# pvresize /dev/sdae2 /dev/sdbm: open failed: No medium found /dev/sdbn: open failed: No medium found /dev/sdbj: open failed: No medium found /dev/sdbk: open failed: No medium found /dev/sdbl: open failed: No medium found Physical volume "/dev/sdae2" changed 1 physical volume(s) resized or updated / 0 physical volume(s) not resized
# pvdisplay /dev/sdbm: open failed: No medium found /dev/sdbn: open failed: No medium found /dev/sdbj: open failed: No medium found /dev/sdbk: open failed: No medium found /dev/sdbl: open failed: No medium found --- Physical volume --- PV Name /dev/sdae2 VG Name centos_4602c PV Size <475.84 GiB / not usable 3.25 MiB Allocatable yes PE Size 4.00 MiB Total PE 121813 Free PE 91544 Allocated PE 30269 PV UUID yvHO6t-cYHM-CCCm-2hOO-mJWf-6NUI-zgxzwc
# vgdisplay --- Volume group --- VG Name centos_4602c System ID Format lvm2 Metadata Areas 2 Metadata Sequence No 6 VG Access read/write VG Status resizable MAX LV 0 Cur LV 3 Open LV 3 Max PV 0 Cur PV 2 Act PV 2 VG Size <475.93 GiB PE Size 4.00 MiB Total PE 121838 Alloc PE / Size 30269 / <118.24 GiB Free PE / Size 91569 / 357.69 GiB VG UUID ORcp2t-ntwQ-CNSX-NeXL-Udd9-htt9-kLfvRc
# lvextend -l +91569 /dev/centos_4602c/root Size of logical volume centos_4602c/root changed from 50.00 GiB (12800 extents) to <407.69 GiB (104369 extents). Logical volume centos_4602c/root successfully resized.
Extend the xfs file system to use the extended space
The xfs file system for the root partition will need to be extended to use the extra space; this is done using the xfs_grow command as shown below.
This is the history it happen to me some time ago, and so the commands I used to troubleshot. The purpose is to share knowledge in a interactive way. There are some hidden gems that you’ll acquire if you have the patience to go over all the document and read it all…
I had qualified Intel Xeon single processor platform to run my DRAID (ZFS Declustered RAID) project for my employer.
The platforms I qualified were:
1) single processor for Cold Storage (SAS Spinning drives): 4U60, newest models 4602
2) for multiprocessor: the 4U90 (90 Spinning drives) and Flash: All-Flash-Arrays.
The amounts of RAM I was using for my tests range for 64GB to 384GB.
Somebody in the company, at executive level, assembled an experimental config that was totally new for us and wanted to try by their own. It was the 4602 with multiprocessor and 32GB of RAM.
When they were unable to make it work at the expected speed, they required me to troubleshot and to make it work.
The 4602 single processor had two IOC (Input Output Controller, LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02) ), while the 4602 double processor had four IOC, so given that each of those IOC can perform at peaks of 6GB/s, with a maximum total of 24 GB/s, the performance when reading/writing from all the drives should be better.
But this Server was returning double times for Rebuilding, respect the single processor version, which didn’t make any sense.
I had to check everything. There was the commands I ran:
Check the upgrade of the CPU:
htop
lscpu
Changing the Zoning.
Those Servers use SAS drives dual ported, which means that two different computers can be connected to the same drive and operate at the same time. Is up to you to make sure you don’t introduce corruption. Those systems are used mainly for HA (High Availability).
Those Systems allow to be configured in different zoning modes. That’s the way on how each of the two servers (Controllers) see the disk. In one zoning each Controller sees only 30 drives, in another each IOC sees all the drives (for redundancy but performance constrained to 1 IOC Speed).
The config I set is each IOC will see 15 drives, so each one of the 4 IOC will have 6GB/s for 15 drives. Given that these spinning drives perform in the outtermost part of the cylinder at 265MB/s, that means that at maximum speed one IOC will be using 3.97 GB/s, will say 4GB/s. Plenty of bandwidth.
Note: Spinning drives have different performance depending on how close you’re to the cylinder. In the innermost part it goes under 145 MB/s, and if you read all of those drive sequentially with dd it will return an average speed of 145 MB/s.
With this command you can sive live how it performs and the average read speed in real time. Use skip to jump to that position (relative to bs) in the drive, so you can test directly the speed at the innermost close to the cylinder part of t.
dd if=/dev/sda of=/dev/null bs=1M status=progress
I saw that the zoning was not right one, so I set it correctly:
The sleeps after rebooting the expanders are recommended. Rebooting the Operating System too, to avoid problems with some Software as the expanders changed live.
If you have ZFS pools or workloads stop them and export the pool before messing with the expanders.
In order to check to which drives is connected each IOC:
I do this for all the drives at the same time and with iostat:
iostat -y 1 1
I check the status of the memory with:
slabtop
free
htop
I checked the memory and htop during a Rebuild. Memory was more than enough. However CPU usage was higher than expected.
The red bars in the image correspond to kernel processes, in this case is the DRAID Rebuild. I see that the load is higher than the usual with a single processor.
I capture all the parameters from ZFS with:
zfs get all
All this information is logged into my forensics document, so later can be checked by my Team or I can share with other Architects or other members of the company. I started this methodology after I knew how Google do their SRE forensics / postmortem documents. Also for myself is useful for the future to have a log of the commands I executed and a verbose output of the results.
I install the smp_utils
yum install smp_utils
Check things:
ls -al /dev/bsg/
total 0drwxr-xr-x. 2 root root 3020 May 22 10:16 .
drwxr-xr-x. 20 root root 8680 May 22 10:16 ..
crw-------. 1 root root 248, 76 May 22 10:00 1:0:0:0
crw-------. 1 root root 248, 126 May 22 10:00 10:0:0:0
crw-------. 1 root root 248, 127 May 22 10:00 10:0:1:0
crw-------. 1 root root 248, 136 May 22 10:00 10:0:10:0
crw-------. 1 root root 248, 137 May 22 10:00 10:0:11:0
crw-------. 1 root root 248, 138 May 22 10:00 10:0:12:0
crw-------. 1 root root 248, 139 May 22 10:00 10:0:13:0
[...]
There are some errors, and I check with the Hardware Team, which pass a battery of tests on the machine and say that the machine passes. They tell me that if the errors counted were in order of millions then it would be a problem, but having few of them is usual.
My colleagues previously reported that the memory was performing well, and the CPU too. They told me that the speed was exactly double respect a platform with one single CPU of the same kind.
Even if they told me that, I ran cmips tests to make sure.
git clone https://github.com/cmips/cmips_bin
It scored 16,000. The performance was Ok in general terms but the problem is that I didn’t have a baseline for that processor in single processor, so I cannot make sure that the memory bandwidth was Ok. The performance was less that an Amazon c3.8xlarge. The system I was testing is a two processor system, but each CPU is cheap, around USD $400.
Still my gut feeling was telling me that this double processor server should score more.
lscpu
[root@DRAID-1135-14TB-2CPU ~]# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 79
Model name: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
Stepping: 1
CPU MHz: 2299.951
CPU max MHz: 3000.0000
CPU min MHz: 1200.0000
BogoMIPS: 4199.73
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 20480K
NUMA node0 CPU(s): 0-7,16-23
NUMA node1 CPU(s): 8-15,24-31
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts spec_ctrl intel_stibp
I check the memory configuration with:
dmidecode -t memory
I examined the results, I see that the processor can only operate the DDR4 ECC 2400 Memory at 2133 and… I see something!. This Controller before was a single processor with 2 Memory Sticks of 16GB each, dual rank.
I see that now I have the same number of sticks in that machine, but I have two CPU!. So 2 Memory sticks in total, for 2 CPU.
That’s no good. The memory must be in pairs and in the right slots to get the maximum performance.
1 memory module for 1 CPU doesn’t allow to have Dual Channel and probably is affecting the performance. Many Servers will not even boot if you add an odd number of memory sticks per CPU.
And many Servers can operate at full speed only if all the banks are filled.
I request to the Engineers in Silicon Valley to add 4 modules in the right slots. They did, and I repeated the tests and the performance was doubled then.
After some days I had some time with the machine, I repeated the test and I got a CMIPS Score of around 20,000.
Multiprocessor world is far more complicated than single processor. Some times things can work not as expected, and not be evident, for example cache pipeline can act diferent for a program working in multiprocessor and single processor. Or the QPI could be saturated.
After this I shared my forensics document with as many Engineers as I could, so they could learn how I did to troubleshot the problem, and what was the origin of it, and I asked them to do the same so we can track their steps and progress if something needs to be troubleshoot.
After proper intensive testing the Server was qualified. Lesson here is that changes cannot be commited quickly, need their time.
