Category Archives: Google Cloud

Forensics Postmortem Analysis and how abusive bots pushed me to migrate part of my services from Google Cloud to another Cloud provider

I tested Google Cloud when they were in beta, years ago, and I helped their Team to improve the service. And I’ve been using Google Cloud for years.

Here I explain, how an incident, this August, almost made me leave Google Cloud for good. In fact, I migrated many services to other providers. I walk through the steps I took.

The best advice to start with, is, enable Cloud Billing alerts.

Everything started with the alert of Cloud Billing alerting me that half of the budget for the month had been exhausted around the 7th of the month. I checked the panel, and at the beginning I didn’t pay much attention to it, as Billing was indicating that the extra cost was in Compute in a specific region where I had provisioned new instances and more storage and it was just few euros.

Compute Engine Costs are mixed with the bandwidth consumption in the Dashboards which makes no sense:

So, I had to check different dashboards to realise it was a data transfer issue, and honestly, at the beginning I didn’t know where the problem was, but as I wrongly inferred, that the instance generating egress traffic was the one named “ubuntu26-04” I was investigating mainly that.

The first thing to learn that Google Cloud Billing shows the pure Compute and bandwidth usage together. I dislike it, for obvious reasons.

Next day budget for the month was almost gone.

Running the reports I saw “SKU Network Internet Data Transfer Out from Americas to Americas” which was confusing, and initially made me think if it could be the new backups system, sending the backups to my centralised server using the AI compatible agent with the public IP for the API and encryption, but I discarded this very quickly due to the volume of data and I modified the system to use internal IPs instead of the public ones.

The only clear thing at this point was that Internet data transfer (egress) was being sent, and Google makes you pay for that.

I had released several projects recently, like https://read.carlesmateo.com/en Text-to-speech to generate MP3 from text files, or https://sendmeafile.carlesmateo.com/en to send and receive huge files, with resume, checksum verification, AI compatible… So my first thought was that one of the projects or its dependencies could have been compromised, or fanatic AI/crawlers were downloading the sample MP3 files again and again.

I made a mistake when checking Cloud Billing. It indicated that the expenses came from us-central1, so checked the list of instances’ regions and I thought the problem came from a new instance named “ubuntu26-04”.

But I made a mistake, because there are several zones in every region, like us-central1-a, us-central1-b, us-central1-c… My new instance for the new projects was in the zone us-central1-f but I had more than that one instance in the region, only that in different zones. But when I checked the zones for the instances I did not realise that there was more than one instance in the zone us-central1 and I assumed the new instance was the only one in the zone and should be the one generating the egress traffic.

I only had ports 80 and 443 opened in Google’s firewall, so there could be requests to the web server, or that the server got hacked. I analysed the nginx logs for the server and found no huge traffic or big files being requested.

Nowadays bots and crawlers and AI’s generate a crazy amount of traffic, so I had to discard this.

Also, recently Amazon had a problem were it presented huge bills (erroneous) of more than 1 Million dollars to some customers. I didn’t know if Google was calculating the bandwidth costs well.

The Billing Dashboard showed that the increment in data sent was costing me around 11€ per day, so it allowed me to start moving services from that server, with relative calm. The forecast for all the August month displayed that the bill would cost me 300€, which was not nice.

In the meantime I hired a VPS with unlimited data and started to migrate the Docker services there as fast as I could with the idea of shutting down the compromise instance to cut the costs.

One of the things that really made me angry, while I was migrating to avoid those over costs, is that I checked again, the next day, the Cloud Billing Reports, the web loaded and informed me that it had changed with news and showed some floating popups pointing how it work and some improvements pushed by the Google Dev Team, and when I continued, the Cloud Billing Report was displaying an unexpected amount of money to pay in August 2026, more than 10 times what I expected, instead of the forecast I saw previously. So it looked like I had to pay that amount for August, and I started to look at what caused that increment of costs quickly, which fas frustrating. And after a while, when I was thinking about stopping the services, I found that Google had changed the page of the reports and the filters were reset to yearly, and the page was showing the already paid costs from 1st August 2025 to 31th August 2026.

Later I reported how frustrating was the experience to Google as “you should not do that” and “users dislike frequent changes”.

Companies are not aware that when they change things, they often annoy the customers a lot.

Years ago I migrated from my long term account with Amazon AWS to Google Cloud Compute Engine cause I was feed up of the problems in Amazon, how many programming errors they had, the changes they did, causing problems, like to the identifiers of the objects (I had older created objects, with shorter hash, and some API calls did not work any more), and how difficult they made simple things. User Interface was bad, and they also lost two of my instances and communicated really poorly (I also reported that to a friend in Amazon and I had some conversations with Amazon’s AWS PMs), and they were over expensive for developers, so I ended migrating to Google Cloud. And now Google is repeating some of the same mistakes.

They are not only expensive. They make things that could be easy, difficult, for the developers.

These big companies force you to stay continuously learning they changes. They offer great solutions for the Enterprise, but for developers and Startups the costs could be overkill.

Probably they are over-kill and over-expensive for developers and Startups.

With Google Cloud I have experienced some technical issues too, as my older instance froze several times and became irresponsive and I had to do a stop from the web console and I had to wait until the command timed out and Google realised there was a problem so it would consider the instance finally shutdown and I could start it again in other non faulty hardware.

If you are a company with a Load Balancer, with health checks, and several front servers, and 20 developers, you can afford to have Web servers KO, as the Load Balancer will handle it nicely, but if you have a single server with WordPress sites, if the instance stops it means that nobody can access those sites.

As an example of costs, for a Google e2-medium (2 vCPUs, 4 GB of Memory and 20 GB of disk) in us-central1 I pay around 30 € per month. And it happens that the forecast for this month in bandwidth was 300 €.

Also storage is around 1€/month per 10GB for standard. So 200GB of regular space is 20 €/month.

For SSD 1.7€/month per 10GB, even tough provides more IOPS, but not more speed. 200GB of space more is 34 € more per month.

I migrated to a provider were I pay 30 € per month for a 8 vCPU and 24 GB of memory and includes 200 GB of NVMe, and I have unlimited bandwidth. Extra storage is also much more cheap. And the static IP is included (for Google you pay around 1.1€ per month).

The key in here was the unlimited bandwidth.

I also could have rented a dedicated server with 64GB of RAM for around 64 €/month, but these comes with some down sides, like if your server experiences a hardware error, you’ll have to wait for the System Administrators from the Cloud Provider to physically replace the components. With a VPS or instance, I can just launch it in a healthy hardware immediately.

As I was migrating services and investigating the origin of the problems, I saw that 500 GB of data were transferred from us-central1 since the 1st of August.

The Billing Report didn’t allow me to filter for IP. It only helped to know the region, us-central1, which helped me to identify the possible services that could be having the problems.

To be honest, I should have located the origin of the problem quickly, but I did so well blocking bots and uneducated crawlers in the past years, that I had no problems for years, and GCE has mad many changes over time, so I did not remember that I could monitor real time network usage of my servers. Most of the servers I troubleshoot I have only SSH access, so I use network tools.

But as I told previously, I mistakenly thought that just one instance was running in us-central1, and that was not the case. They were in different zones of us-central1.

The live traffic inspection in the server with tools like ss, netstat, nethogs, iftop, iotop and ps revealed normal traffic and load.

A second revision from the nginx logs didn’t reveal any problem over time. And the only opened ports were 80 and 443. With the services I considered potentially more vulnerable (as they used python dependencies) migrated to the new VPS and stopped in the “ubuntu26-04” Google server, the usage shown and the forecast for bandwidth was not reduced and it was the same.

One thing I learned, and I suspected it from my tests with https://sendmeafile.carlesmateo.com/en is that the maximum speed that my instances were able to transfer from Google to Internet (egress) was around 50 Mbps per connection.

I didn’t see this in Google Cloud’s documentation, but in my tests transferring data from Google servers using the public interface and from different locations to this google server I consistently achieved always that top speed of around 50 Mbps per connection (around 6.25 MB/s), but when I communicated Google servers using their private interfaces, the speed for a single connection was much more higher. So I assumed that the bandwidth was capped. After seeing the data transferred in the Billing Cloud Report, it looks clear to me that an important part of the server’s bandwidth max was being used. Later, Apache logs, confirmed me many requests at the same exact second.

Google anomalies tool detected nothing despite the huge increment in data transfer and cost variation (8,960% respect July!).

Before taking the decision to migrating to another server one has to think, not only in costs, but also on how trustworthy are the storage systems, the resiliency of the systems, how many times they are down per year… there are surprises that you discover as you go. With the experience.

But honestly, I felt it was not worth the price to continue with Google Cloud and it was conditioning the kind of projects I could start. My architecture is resilient, based in containers and I can deploy it fast. I also have backups.

Google storage is not that super trustable in my opinion (neither Amazon’s). Some times my VMs froze and Google monitor dashboard stops drawing the lines of metrics (like CPU usage), but Google doesn’t catch that the VM is frozen. My guess is that the problem is the storage or compute servers dying.

Having to pay per GB transferred in a time where bots and AI index your sites continuously, and abuse the servers by launching many requests concurrently, generating load, and where bots consume much more data than humans, force you to take architecture decisions and to implement strategies.

Possible solutions like using CDN’s like Cloudflare to be protected against DoS, to cache statics and to cut bandwidth and CPU abuse (requests to origin), using other cache systems or migrating to other Cloud providers that are more developer and Start up friendly, and that have unlimited data transfer, etcetera.

When I was presented with the new Google Billing Reports page and the filter was reset to one year, and I saw the price that should be showing multiplied by 10, I contacted Google Cloud by X. They kindly replied and invited me to contact them privately. I did and we opened a ticket. Around three hours later I got an email from a Google staff, telling me that they will investigate the problem, and giving me an expected response deadline of 5 days.

It was nice, but by the time I received the email I already had found and sorted the problem.

I troubleshoot the instance “ubuntu26-04” but the CPU usage was low, and no permanent connections (a sign that could have indicated that was hacked):

That Looked normal

sudo ss -tunp state established

Then I did something that I should have done before, I checked the metrics from all the different instances in Google Cloud Compute, starting from the one I initially guessed was the one suffering the problems. That machine was clear. No high CPU usage. No high bandwidth usage. Nothing.

And then I saw it.

Through VM Instances I narrowed to the one in us-central1 that was consuming the Internet bandwidth, “instance-1”:

I was checking the wrong server. The consuming excess of egress was from instance-1 that hosted several WordPress sites and it was using Apache2, not nginx.

Some commands I ran:

# Check for the data sent by Apache. This found a lot of traffic. Note: the access.log of all the sites were being analysed
sudo awk '{s+=$10} END {print s/1024/1024/1024 " GB"}' /var/log/apache2/*access.log
# returned 36.4578 GB that's for a day, so something was clearly wrong

# Analyze requests: See what is more heavy. So basically bots trying to hack into the server and some heavy images. Note root home for at least a server is huge (50MB)
sudo awk '{a[$7]+=$10} END {for(u in a) print a[u]/1024/1024" MB\t"u}' /var/log/apache2/*access.log | sort -rn | head -20
54.3218 MB      /
21.6173 MB      /wp-login.php
17.1676 MB      /wp-content/uploads/2022/10/20221001_200930-cut.png
16.5822 MB      /wp-content/uploads/2022/10/20221001_201033.png
10.7245 MB      /wp-content/uploads/2022/02/python_loop.gif
6.61174 MB      /xmlrpc.php
6.60595 MB      /page/2/?shopdetail%2FI37266592159
6.60595 MB      /page/2/?shopdetail%2FI37256267952
6.60595 MB      /page/2/?shopdetail%2FI37256137878
6.60595 MB      /page/2/?shopdetail%2FI37256033568
6.60595 MB      /page/2/?shopdetail%2FI37224959793
6.60595 MB      /page/2/?shopdetail%2FI37214305672
6.60595 MB      /page/2/?shopdetail%2FI37203620584
6.60595 MB      /page/2/?shopdetail%2FE37266925367
6.60595 MB      /page/2/?shopdetail%2FE37242498260
6.60595 MB      /page/2/?shopdetail%2FE37207321261
6.60595 MB      /page/2/?shopdetail%2FA37259909717
6.60595 MB      /page/2/?shopdetail%2FA37259600135
6.60595 MB      /page/2/?shopdetail%2FA37256306882
6.60595 MB      /page/2/?shopdetail%2FA37256197883

# Analyze requests: See what IP's are requesting more traffic in total. Gotcha
sudo awk '{a[$1]+=$10} END {for(i in a) print a[i]/1024/1024" MB\t"i}' /var/log/apache2/*access.log | sort -rn | head -20
34549.2 MB      74.7.241.60
1491.36 MB      116.202.235.23
287.651 MB      3.41.188.33
232.362 MB      74.7.227.160
199.987 MB      18.97.9.169
183.748 MB      66.249.74.42
84.1681 MB      114.119.140.64
78.3209 MB      114.119.149.169
70.5166 MB      114.119.156.46
67.8224 MB      114.119.150.29
66.7544 MB      114.119.145.237
58.7278 MB      114.119.147.113
57.1822 MB      114.119.162.62
56.0061 MB      66.249.74.43
55.3612 MB      114.119.141.79
21.9245 MB      216.73.217.35
21.2516 MB      104.243.44.93
17.7313 MB      79.116.217.181
17.1676 MB      189.81.71.184
16.5822 MB      205.188.63.181

So we see that this IP 74.7.241.60 is requesting 34.5 GB in a day.

So, according to this IP Info service, the IP is from Microsoft. So apparently, crawlers/IA from Microsoft or running in Microsoft infrastructure, were costing me money because they are abusing, scanning my sites, with many connections concurrently, like horrible soulless machines:

I double checked the IP with another provider:

A command to see a sample of what kind of requests I have from this IP, brings more light:

grep "74.7.241.60" /var/log/apache2/*access.log | head -20
/var/log/apache2/www_mysqlproxycache_com-access.log:74.7.241.60 - - [13/Aug/2026:00:00:22 +0000] "GET /page/2/?shopdetail%2FI37207088742 HTTP/1.1" 200 3463409 "http://www.mysqlproxycache.com/page/2/?shopdetail/I37207088742" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot)"
/var/log/apache2/www_mysqlproxycache_com-access.log:74.7.241.60 - - [13/Aug/2026:00:00:34 +0000] "GET /page/3/?shopdetail/I37207088742 HTTP/1.1" 301 264 "http://www.mysqlproxycache.com/page/2/?shopdetail/I37207088742" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot)"
/var/log/apache2/www_mysqlproxycache_com-access.log:74.7.241.60 - - [13/Aug/2026:00:00:35 +0000] "GET /page/3/?shopdetail%2FI37207088742 HTTP/1.1" 200 1900262 "http://www.mysqlproxycache.com/page/3/?shopdetail/I37207088742" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot)"
/var/log/apache2/www_mysqlproxycache_com-access.log:74.7.241.60 - - [13/Aug/2026:00:00:37 +0000] "GET /page/7/?shopdetail/E37249347969 HTTP/1.1" 301 264 "http://www.mysqlproxycache.com/page/6/?shopdetail/E37249347969" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot)"

According to the User-Agent is GPTBot. So apparently a bot running in Microsoft self-identifying as OpenAI were killing causing high usage to my server and to my budget by requesting many pages at the same time, for an old shopdetail page that does not exist in that server (maybe there was a shop years before I got that IP), and spends a lot of bandwidth responding with a heavy page of 6 MB.

In the logs I saw many request at the same precise second.

Asking to Claude it tells me that the range of IPs of the Microsoft attacker IP are not in the publicly published IPs of OpenAI for GPTBot, so apparently somebody using Microsoft IPs is pretending to be a GPT Bot, and it isn’t:

The domain is an old WordPress that has been hacked. Yes, this happens more often that we would like, with WordPress.

I checked the https and it returns a SSL potential warning in the SSL.

I tried with http http://www.mysqlproxycache.com/page/7/?shopdetail%2FI37224744693 (one of the requests in the logs) and I saw that the WordPress site was hacked.

So first things first: I stopped Apache.

sudo service apache2 stop

So, to summarise, there was a bot from a Microsoft IP, that was requesting many times per second pages to a WordPress site that was compromised and it was returning a large article with many photos, so heavy in terms of data usage.

With Apache2 server stopped, I moved the config symbolic link from /etc/apache2/sites-enabled/ to another folder /etc/apache2/sites-disabled/ that I created.

So basically, when I started Apache, the compromised site will not be served. It will be served by the default site.

I restarted the Apache server and checked that the URLs for that domain were catch by default site and no processing resources of bandwidth was used.

That was to avoid a main site marked as default in the Apache config to attend those requests and waste data.

And then I added to the firewall, to Google Cloud’s Firewall and to the servers in all my instances, to block all the IPs from Microsoft from that range.

I saw that the BGP Prefix is 74.7.0.0/16 so I block all the 65,536 IP Addresses:

ufw insert 1 deny from 74.7.0.0/16 to any

As you can see I inserted this rule on the first position. Don’t just add it with ufw deny from 74.7.0.0/16 to any or otherwise it will go to the last position, after the allow rules. And then it would not be enforced.

Check your rules with:

ufw status numbered

You can block the range in your provider’s Firewall (VPC Firewall in Google Cloud).

If you didn’t have the Ubuntu Firewall active, and want to activate it now, make sure that you have the rules for the SSH connection:

echo "Enabling SSH"
sudo ufw allow 22/tcp

echo "Enabling Web 443 and 80"
#sudo ufw allow "Apache Full"
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

echo "Enabling Firewall"
sudo ufw enable

echo "Showing rules"
sudo ufw status numbered

I recommend you to add the rules to a script in git (Infrastructure as Code), so you can replay it another day, and to any server you have.

By default, ufw sets deny by default for rules not specified. You can check it with:

sudo ufw status verbose | head

Bots, IA, crawlers abuse our servers nowadays.

Also the automated bots attempting to hack into the servers.

If you just check your logs, you’ll see many attempts to exploit vulnerabilities in your servers, every day.

I checked for the second IP consuming more bandwidth in the list:

116.202.235.23

It’s from hetzner, a Cloud provider. https://bgp.he.net/net/74.7.0.0/16

Another 65,536 addresses that get blocked:

# Hetzner
ufw insert 1 deny from 116.202.0.0/16  to any

The third IP will be blocked too, even if I didn’t find any information about who owns the block of IPs.

ufw insert 1 deny from 3.41.188.33 to any

The fourth was part of the same block I blocked first, so it’s already blocked.

The fifth IP, I ask to Google Gemini:

The block is 18.97.9.0/24 but I’ll block all the BGP Prefix, all the 16K addresses.

ufw insert 1 deny from 18.97.0.0/18 to any

See what is the traffic from the servers since I blocked the Microsoft range in the Ubuntu Firewall:

As you can see, after I blocked the first abusive IP allegedly from Microsoft with seemingly OpenAI User-Agent falsified, all the egress traffic went down.

I like how Google reacted when I contacted them. And I should have checked the Network traffic bandwidth first. So I decided to keep the WordPress instance in Google Cloud.

But the new instance “ubuntu26-04” I’m migrating it to the new server. What they offer for the price is much better, and the price of the storage really it’s important for some of my projects. The unlimited bandwidth usage in the new provider it’s also key.

Not all is perfect. I had to learn the web interface from that provider (I used it in the past, but companies change the user interface very often nowadays, causing headaches to users), the place were my VPS was listed was hidden and difficult to find, and by default the VPS had no firewall enabled. Something that I believe that is basic to have enabled by default.

But I think it will be great for my projects and for my budget, to have all those new resources available (8 vCPU and 24 GB of RAM and 200 GB of NVMe disk for the same cost provided in Google Cloud of 2 vCPU and 4 GB of RAM and 20 GB of standard disk).

Take one thing in mind: They AI is hungry for contents, and the mega-rich AI providers

The new provider says it provides 3 Gbps of bandwidth for the instance, with no data usage limits (not having to pay for GB transferred). If I had not reacted fast, the forecast for this month with Google for just a single instance was 300€. It is a risk for developers and small companies, to have the possibility to face a huge bill because more frequent every day bots/crawlers/AIs and processes attempting to breach in your servers, cause your server to use a lot of egress data. So it is worth considering using a Cloud Provider with unlimited data plan. Or at least, activate the Billing Alerts to be notified if your costs exceed a certain threshold.

Showing the exploting Copy Fail CVE-2026-31431 in an Ubuntu 24.04 just launched in Google Cloud, and how to fix it

So I show here how I launched a fresh Ubuntu 24.04 in Google Cloud, on 2026-05-04, and demostrate the exploit of escalation privileges Copy Fail (CVE-2026-31431) which allows you to become root from a regular user account in almost any Linux since year 2017.

It consists in the execution of a Python 3 code, which is only 732 bytes.

I show how I fixed it by upgrading the kernel and rebooting.

Here you can see the original tweet I saw: https://x.com/DarkWebInformer/status/2049579219190165658?s=20

And access the code: https://github.com/theori-io/copy-fail-CVE-2026-31431

I also tried on a fresh deployed Ubuntu 26.04 LTS and it was not affected by the exploit.

Resizing the disk of your Ubuntu Server in Google Cloud GCP without rebooting

If you are running your instances in Google Gloud Compute Engine and you want to increase the size of the Disk without having to reboot, this video explains step by step how you can do it.

Go to Disks in GCP, select the disk of the instance you want to increase, then press Edit.

After you increase the Disk in Google Cloud Dashboard, then ssh to you instance.

There type:

lsblk

in order to list the devices.

In my case is sda and I want to grow the partition 1.

So I proceed with:

sudo growpart /dev/sda 1

Which growing from 30GB to 40GB produces the output:

CHANGED: partition=1 start=2324480 old: size=60590047 end=62914526 new: size=81561567 end=83886046

Is you type lsblk again you’ll see the new size.

But if you type df -h you’ll see that Linux still doesn’t see the space.

To finalize and claim the additional space execute (in my case is sda1):

sudo resize2fs /dev/sda1

Sudo problems in Ubuntu 26.04 LTS with Google Cloud: I’m sorry user. I’m afraid I can’t do that

I show in the video, how briefly after using sudo, it stops working.

I did this proof of concept and, I got the same problem: 

sleep 300 && sudo cat /etc/lsb-release

Checking with: 

id -nG

clearly showed that my user is part of google-sudoers But then:

journalctl -u google-guest-agent -f

Displays:

Apr 26 17:54:07 ubuntu26-04 google_guest_agent[851]: Adding existing user carles_mateo to google-sudoers group.
Apr 26 17:54:07 ubuntu26-04 gpasswd[89648]: user carles_mateo added by root to group google-sudoers
Apr 26 17:54:07 ubuntu26-04 google_guest_agent[851]: Updating keys for user carles_mateo.
Apr 26 17:54:11 ubuntu26-04 google_guest_agent[851]: Updating keys for user carles_mateo.
Apr 26 17:57:11 ubuntu26-04 google_guest_agent[851]: ERROR non_windows_accounts.go:219 invalid ssh key entry - expired key: carles_mateo:...google-ssh {"userName":"carles.mateo@gmail.com","expireOn":"2026-04-26T17:57:05+0000"}
Apr 26 17:57:11 ubuntu26-04 google_guest_agent[851]: ERROR non_windows_accounts.go:219 invalid ssh key entry - expired key: carles_mateo:ssh-rsa...
Apr 26 17:57:11 ubuntu26-04 google_guest_agent[851]: Removing user carles_mateo.
Apr 26 17:57:11 ubuntu26-04 gpasswd[89736]: user carles_mateo removed by root from group google-sudoers

Installing Ubuntu 26.04 LTS in Google Cloud Compute Engine

The video shows step by step how to create an Instance in Google Cloud Compute Engine of the type e2, increase the size of the disk, and install Ubuntu 26.04 LTS Server.

Also shows how the new htop looks, with new IO options.

You know that utilites from coreutils have been rewriten in Rust, like sudo. I was wondering if it would work well. I thoguht I was encountering the first problems, after I experienced that when launched sudo, like in example, a sudo apt install package , sudo then stops working and I’ve to exit the shell and relogin.

I found that it is Google Cloud that removes my user from google-sudoers after 3 minutes.

I explain it in here.

News from the Blog 2021-11-11

New Articles

How to communicate with your Python program running inside a Docker Container, using Linux Signals

Hope you’ll have fun reading this article:

Communicating with Docker Containers via Linux Signals and Python

I migrated my last services from Amazon and the blog to Google Compute Engine (GCE / GCP)

I wrote a Postmortem analysis about the process of migrating my last services from my 11 year old Amazon account.

Updates

Updates to articles

I updated the article about Python weird things that you may not know adding the Ellipsis …

I’ve been working in some Cassandra examples. I may publish an article soon about using it from Python and Docker.

Updates to My Books

I updated my Python and Docker books.

I’m currently writing a book about using Amazon AWS Python SDK (boto3).

Updates to Open Source projects

I have updated ctop, fixed two bugs and increased Code Coverage.

I made a new tag and released the last Stable Version:

https://gitlab.com/carles.mateo/ctop/-/tags/0.8.7

On top of my local Unit Testing, I have Jenkins checking that I don’t commit anything that breaks the Tests.

Some time ago I wrote some articles about how you can setup jenkins in a Docker Container.

Miscellaneous

Charity

I’ve donated to Wikipedia.

Only 2% of the viewers donate, so I answered the call every time it was made.

This is my 5th donation to Wikimedia.

I consider that Freedom is very important.

I bought these new books

One of my secrets to be on top is that I’m always studying.

I study all the time, at work and in my free time.

I use Linux Academy and I buy books in paper. I don’t connect with reading in tablets. I think information is stored better when read in paper. I use also a marker and pointers to keep a direct access to the most interesting points on the books.

And I study all kind of themes. Obviously I know a lot of Web Scraping, but there is always room for learning more. And whatever new I learn helps me to be better with my students and more clear writing my books.

I’ve never been a Front End, but I’ve been able to fix bugs in the Front End engines from the companies I worked for, like Privalia. I was passed a bug that prevented the Internet Explorer users to buy just one hour before we launching a massive campaign. I debugged and I found a variable named “value” so the html looked like <input name="value" value="">. In less than 30 minutes I proved to the incredulous Head of Development and the CTO that a bug in Internet Explored was causing a conflict when fetching the value from the input named value. We deployed to Production the update and the campaign was a total success. So I consider knowing Javascript and Front also a need, even if I don’t work directly with it. I want to be able to understand all the requirements and possibilities, and weaknesses, so I can fix bugs and save the day. That allowed me to fix scalability problems in Nodejs and Phantomjs projects too. (They are Javascript Server Side, event driven, projects)

It seems that Amazon.co.uk works well again for Ireland. My two last orders arrived on time and I had no problems of border taxes apparently.

Nice Python article

I enjoyed a lot this article, cause explains part of what I did with my student and friend Albert, in a project that analyzes the access logs from Apache for patterns of attempts of exploits, then feeds a database, and then blocks those offender Ip Addresses in the Firewall.

The article only covers the part of Pandas, of reading the access.log file and working with it, but is a very well redacted article:

https://mmas.github.io/read-apache-access-log-pandas

Nice Virtual Volumes article from VMware

I prefer Open Source, but there are very good commercial products too.

I liked this article about Virtual Volumes from VMWare:

Understanding Virtual Volumes (vVols) in VMware vSphere 6.7/7.0 (2113013)

https://kb.vmware.com/s/article/2113013

Thanks Blizzard (again)

There is a very nice initiative where we can nominate 4 colleagues a year, that we think that deserve a recognition.

My colleagues voted for me, so I received a gift voucher that I can spend in Ireland stores like Ikea, Pc World, Argos, Adidas, App Store & iTunes…

So thanks a million buds. :)

Migrating my 11 years Amazon AWS account services (Postmortem Analysis)

I started to explain that I was migrating some services from Amazon and that some of my sites were under Maintenance and that I would provide more information.

Here is the complete history of why I migrated all the services from my 11 years old Amazon account to other CSP.

Some lessons can be learned from my adventure.

I migrated my last services from Amazon to GCP

Amazon sent me an email on October 6th, this year 2021, telling me that they will disable EC2-Classic by August 2022. I thought I would not be able to keep my Static Ip’s as in the past VPC Ip’s and EC2-Classic Ip’s were not transferable, so considering that I would loss my Static Ip’s anyway I started to migrate to some to other providers like Digital Ocean.

Is not cool losing Static Ip (Elastic Ip in AWS) Addresses as this is bad for SEO, so given that I though I would lose my Static Ips that have been with me for years, I started to migrate certain services to providers much more economic.

Amazon is terrible communicating, and I talked with some product managers in the past about that, when they lost one of my Volumes, and the email was so cold and terrible that actually that hurt more than Amazon losing my Data. I believed that it was a poorly made Scam and when I realized it was true I reached one of my friends, that is manager there, as I know they care for doing things right, and he organized a meeting with two PM so I can pass my feedback.

The Cloud providers are changing things very fast, and nobody is able to be up to date with the changes, unless their work position allows plenty of time to get updated. Even if pages of documentation are provided, you have to react to an event that they externally generated forcing you to action. Action to read all the documentation about EC2-Classic migrations, action to prepare to have migrated by August 2022.

So August 2022… I was counting that I had plenty of time but I’m writing a new book about using the Amazon SDK for Python, boto3, and I was doing some API calls and they started to fail in a very unusual way, Exceptions with timeout, but only for the only region where I had EC2-Classic. 

urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x7f0347d545e0>: Failed to establish a new connection: [Errno -2] Name or service not known

My config was:

        o_config = Config(
            region_name="us-east-1a",
            signature_version="v4",
            retries={
                'max_attempts': 10,
                'mode': 'standard'
            }
        )

But if I switched to another region name, it would work:

            region_name='us-west-2',

I made a mistake in here, the region name is “us-east-1” and not “us-east-1a“. “us-east-1a” is the availability zone. So the SDK was giving a timeout because in order to connect to the endpoint it uses the region name as part of the hostname. So it doesn’t find that endpoint because it doesn’t exist.

I never understood why a company like Amazon is unable to provide the SDK with a sample project or projects 100% working, with the source code so people has a base that works to build up.

Every API that I have created, I have provided it with documentation but also with example for several languages for how to use it.

In 2013 I was CTO of an online travel agency, and we had meta-searchers consuming our API and we were having several hundreds of thousands requests per second. Everything was perfectly documented, examples were provided for several languages, the document and the SDK had version numbers…

Everybody forgets about Developers and companies throw terrible and cold products to the poor Developers, so difficult to use. How many Developers would like to say: Listen Mr. President of the big Cloud Company XXXX, I only want to spawn a VM that works, and fast, with easy wizards. I don’t want to learn 50 hours before being able to use your overpriced platform, by doing 20 things before your Ip’s are reflexes of your infrastructure and based in Microservices. Modern JavaScript frameworks can create nice gently wizards even if you have supercold APIs.

Honestly, I didn’t realize my typo in the region and I connected to the Amazon Console to investigate and I saw this.

Honestly, when I read it I understood that they were going to end my EC2 Networking the 30th of October. It was 29th. I misunderstood.

It was my fault not reading it well to the end, I got shocked by the first part telling about shutdown and I didn’t fully understood as they were going to shutdown EC2-Classic for the zones I didn’t had anything running only.

From the long errors (3 exceptions chained) I didn’t realize that the endpoint is built with the region name. (And I was passing the availability zone)

botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://ec2.us-east-1a.amazonaws.com/"

Here is when I say that a good SDM would had thought and cared for the Developers more, and would had made the SDK to check if that region exists. How difficult is to create a SDK a bit more clever that detects a invalid region id?. It is not difficult.

It is true that it was late in the evening and I was tired of all the day, and two days of the week between work and zoom university classes I work 15 hours and 13 hours respectively, not counting the assignments, so by the end of the week I am very tired. But that’s why it is very important to follow methodology and to read well. I think Amazon has 50% of the fault by the way they do things: how the created the SDK, how they communicate, and by the errors that the console returned me when I tried to create a VPC instance of an EC2-Classic AMI (they seem related to the fact I had old VPC Network objects with shorter hash than the current they use) and the other 50% was my fault for not identifying the source of the error, and not reading the message in their website well.

But the fact that there were having those errors in the API’s and timeouts made me believe they were going to cut the EC2-Classic Networking the next day.

All the mistakes fall together in a perfect storm.

I checked for documentation and I saw it was possible to migrate my Static Ip’s to VPC Static Ip’s.

It was Friday evening, and I cancelled my plans, in order to migrate the Blog to VPC in an attempt to keep running it with Amazon.

As Cloud Architect, I like to have running instances in several CSP as it allows me to stay up to date with the changes they do.

I checked the documentation for the migration. Disassociating the Static Ip (Elastic Ip in AWS jargon) was easy. Turning into VPC as well.

As I progressed, what had to be easy turned into a nightmare, as I was getting many errors from the Amazon API, without any information, and my Instances were not created.

I figured out that their API could have problems with old VPC objects I created time ago, so I had to create new objects for several things.

I managed to spawn my instances but they were being launch and terminated instantly without information. Frustrating.

When launching a new instance from the AMI (a Snapshot of the blog), I was giving shown options to add more volumes without any sense. My Instance was using 16GB from a 20GB total Space, and I was shown different volume configs, depending on the instance, in some case an additional 20GB volume, in other small SSD, ephemeral and 10 GB for the AMI (which requires at least 16GB).

After some fight I manage to make it work after deleting the volumes that made no sense, and keeping only one of 20GB, the same size of my AMI.

But then my nightmare started to make the VPC Instance to have Internet access and to be seen from outside. I had to create a new Internet Gateway, NAT, Network, etc…

As mentioned the old objects I was trying to reusing were making the process to fail.

I was running out of time, and I thought in few time they were going to shutdown EC2-Classic network (as I did not read correctly), so I decided to download everything and to migrate to another provider. For doing that first I blocked all the traffic, except for my Ip.

I worked in parallel, creating the new config in Google Cloud, just in case I had forgot something. I had created a document for the migration and it was accurate.

I managed to do everything fast enough. The slower part was to download all the Data, as I hold entire VM’s for projects like Cassandra Universal Driver.

Then I powered off my Amazon Instance for the Blog forever.

In GCP I blocked all the traffic in the firewall, except for my Ip, so I could work calmly.

When everything was ready, I had to redirect the DNS to the new static Ip from Google.

The DNS provider I used had implemented some changes in their API so I was getting errors replacing my old entry ‘.’ (their JSON calls returned Internal Server Error). Finally I figured it out how to workaround it and I was able to confirm that the first service was up and running.

I did some tests to make sure there were not unexpected permission problems, entries in the logs, etc…

Only then I opened the Google Firewall. I have a second firewall in each instance where I block or open at Ip tables level what I want. Basically abusive bot’s IPs trying to find exploits or brute force by dictionary passwords.

I checked with my phone, without Wifi that the Firewall was all good. (It is always a good idea to use another external Ip, different from the management one, to check)

I added a post explaining that I was migrating some of my Services and were under maintenance.

I mentioned in the blog that some of my services were being migrated from Amazon to Digital Ocean.

For some reasons, in the Backup of the Database one user was lost, so I created it in the MySQL with the typical commands:

CREATE USER 'username'@'localhost' IDENTIFIED BY 'password';
FLUSH PRIVILEGES;
GRANT ALL PRIVILEGES ON mydatabase.* TO 'username'@'localhost';

Google Compute Engine Talk for Group Google Developers Cork

My talk in Google Developers Cork Group.
It’s about deploying an Instance in GCE and grows in complexity until Deploying a Load Balancer with AutoScaling for a group of LAMP Webservers.

Join the group at: https://www.meetup.com/GDG-Cork/

The videos:

Keshan Sodimana: Tensors

The Cloud is for Scaling

Last Update: 2022-08-03 I added the Enterprises, as more and more services have been provided by Amazon and the other major Cloud Providers and they offer a suitable solution for Enterprises requiring high availability in services managed by Amazon, without the need to configure and maintain their own setups manually.

dell-blades-m4110The Cloud is for Startups, and for Scaling. Nothing more.

In the future will be used by phone operators, to re-dimension their infrastructure and bandwidth in real time according to demand, but nowadays the Cloud is for Startups.

Examine the prices in my post in cmips, take a look, examine the performance also of the different CPU. You see that according to CMIPS v.1.03 a Desktop Processor Intel i7-4770S, worth USD $300, performs better than an Amazon M2 High Memory Quadruple Extra Large and than a Rackspace First gen. 30 GB RAM 8 Cores?.

Today the public cost of an Amazon M2 High Memory Quadruple Extra Large running for a month is USD $1,180.80 so USD $1.64 per hour and the Rackspace First Generation 30 GB RAM 8 Cores 1200 GB of disk costs is USD $1,425.60 so USD $1.98 per hour running.

And that’s the key, the cost per hour.

Because the greatness, the majesty of the Cloud is that you pay per hour, you pay as you need, or as you go. No attaching contracts. All on demand.

I had my company at a time where the hosting companies and the Data Centers were forcing customers to sign yearly contracts. What if a company only needs to host their Servers for three months? What if they have to close?. No options. You take it or you leave it.

Even renting a dedicated hosting was for at least a month or more, and what if the latency was not good? What if the bandwidth of the provider was not enough?.

Amazon irrupted in the market with strength. I really like that company because they grew the best eCommerce company for buying books, they did a system that really worked, and was able to recommend very useful computer books, and the delivery, logistics was so good, also post-sales service. They simply started to rent the same infrastructure they were using to attend their millions of customers and was a total success.

And for a while few people knew about Amazon deep technologies and functionalities, but later became a fashion.

Now people is using Amazon or whatever provider/Service that contains the word “Cloud” because the Cloud is in the mouth of everyone. Magazines and newspapers speak about the Cloud, so many many companies use it simply because everyone is talking about the Cloud. And those ISP that didn’t had a Cloud have invested heavily to create a Cloud, just because they didn’t want to be the ones without a Cloud, since everyone was asking for it and all the ISP companies were offering their “Clouds”.

Every company claims to have “Cloud” where the only many of them have is Vmware servers, Xen servers, Open Stack… running the tenants or instances of the customers always on the same host servers. No real Cloud, professional Cloud, abstract layered in a Professional way like Amazon, only the traditional “shared hosting” with another name, sharing CPU and RAM and Disk storage using virtual machines called instances.

So, Cloud fashion has become a confusing craziness where no one knows why they are in the Cloud but they believe they have to be in.

But do companies need the Cloud?. Cloud instances?

It depends. The best would be to ask that companies Why you choose the Cloud?.

If you compare the cost of having an instance in the Cloud, is much much more expensive than having a dedicated server. And for that high cost you don’t get more performance.

Virtualization is always slower and disk speed is always an issue in Cloud providers, where all the data travels via network from the disk cabins NAS to the Host servers running the guest instances. Data cannot be at local disks, since every time you start an instance, the resources like CPU and RAM are provisioned, and your instance run in totally different hardware. Only your data remain in the NAS (Network Attached Storage).

So unless you run your in-the-Cloud instance in a special provider that offers local disks, like DigitalOcean that offers SSD but monthly paying, (and so you pay the price by losing the hardware abstraction capability because you’re attached to the CPU that has the disk connected, and also you loss the flexibility of paying per hour of use, as you go), then you’ll face a bottleneck that is the hard disk performance (that for real takes all the data from NAS, where is stored, through the local network).

So what are the motivations to use the Cloud?. I try to put some examples, out of these it has no much sense, I think. You can send me your happy-in-Cloud scenarios if you found other good uses.

Example A) Saving initial costs, avoid contract attachment and grow easily own-made

Imagine a Developer that start its own project. May be it works, may be not, but instead of having a monthly contract for a dedicated server, he starts with an Amazon Free Tier (better not, use Small instance at least) and runs a web. If it does not work, simply stop the instance and pay no more. If the project works and has more and more users he can re-dimension the server with a click. Just stop the instance, change the type of instance, start it again with more RAM and more CPU power. Fast.

Hiring a dedicated server implies at least monthly contracts, average of USD $100 per month, and is not easy to move to a bigger server, not fast and is expensive as it requires the ISP tech guys to move the data, to migrate from a Server to another.

Also the available bandwidth is to be taken in consideration. Bandwidth is expensive and Amazon can offer 150 Mbit to smaller machines. Not all the Internet Service Providers can offer that bandwidth even with most advanced packets.

If the project still grows, with a click, in seconds, 20 instances with a lot of bandwidth can be deployed and serving traffic to your customers very quick.

You save the init costs of buying Servers, and the time to deal with hardware, bandwidth limitations and avoid contracts, but you pay an hourly rate a lot more expensive. So in the long run is much much expensive using Amazon and less powerful than having dedicated servers. That happened to Zynga, that was paying $63M annually to Amazon and decided to step back from Amazon to their own Data Centers again. (another fortune tech link)

The limited CPU power was also a deal breaker for many companies that needed really powerful CPU and gigs of RAM for their Database Servers. Now this situation is much better with the introduction of the new Servers.

This developer can benefit from doing bacups with a click, cloning, starting instances from an image, having more static ip’s with a click, deploying built-in (from the Cloud provider) load balancers, using monitoring services like CloudWatch, creating Volumes and attaching to the servers for additional space…

Example B) An Startup with fluctuating number of users and hopes of growing

Imagine an Startup with a wonderful Facebook Application.

During 80% of the day has few visits, may be only need 3 Servers, but during 20% of the hours of the day from 10:00 to 15:00 users connect like hell, so they need 20 servers to attend this traffic and workload, and may be tomorrow needs 30 servers.

With the Cloud they pay for 3 servers 24 hours per day and for the other 17 servers only pay the hours they are on, that’s 5 hours per day. Doing that they save money and they have an unlimited * amount of power. (* There are limits for real, you have to specially request authorisation to run more than default max. servers for the zone, that is normally 20 instances for Amazon. Also it can happen theoretically that when you request new instances the Zone has no instances available).

So well, for an Startup growing, avoiding hiring 20 dedicated servers and instead running into the Cloud as many as they need, for just the time they need, Auto-Scaling up and down, and can use the servers NOW and pay the next month with Visa card, all of that can make a difference for a growing Startup.

If the servers chosen are not powerful enough that is solved with a click, changing instance type. So fast. A minute.

It’s only a matter of money.

Example C) e-Learning companies and online universities

e-Learning platforms also get benefits from the Auto-Scaling for the full occupation hours.

The built-in functionalities of the Cloud to clone instances is very useful to deploy new web servers, or new environments for students doing practices, in the case of teaching Information Technology subjects, where the users need to practice against a real server (Linux or windows).

Those servers can be created and destroyed, cloned from the main -ready to go- template. And also servers can be scheduled to stop at a certain hour and to start also, so saving the money from the hours not needed.

Example D) Digital agencies, sports and other events

When there is an Special event, like motorcycle running, when a Football Team scores, when there is an spot in tv announcing a product…

At those moments the traffic to the site can multiply, so more servers and more bandwidth have to be deployed instantly. That cannot be done with physical servers, hardware, but is very easy to provision instances from the Cloud.

Mass mailing email campaigns can also benefit from creating new Servers when needed.

Example E) Proximity and SEO

Cloud providers have Data Centres everywhere. If you want to have servers in Asia, or static content to be deployed faster, or in South-America, or in Europe… the Cloud providers have plenty of Data Centers all over the world.

Example F) Game aficionado and friends sharing contents

People that loves cooperative games can find the needed hungry bandwidth and at a moderate price. If they run their private server few hours, at night, from 22:00 to 01:00 as example, they will benefit from a great bandwidth from the big Cloud provider and pay only 3 hours per day (the exceed of traffic uses to be paid in most providers, but price of additional GB uses to be really really competitive).

Friends sharing contents in an Ftp also, can benefit from this Cloud servers, but probably they will find more easy to use services like Dropbox.

Example G) Startup serving contents

An Startup serving videos, images, or books, can benefit not only from the great bandwidth of big Cloud providers (this has been covered before), but for a very cheap price for exceeding Gigabyte transferred.

Local ISP can’t offer 150 Mbit for an instance of USD $20 and USD $0.12 per additional GB transferred.

Many Cloud providers also allow unlimited incoming traffic from the Internet, and from Server to Server through private ip’s.

Example H) Enterprises that want to outsource the solutions and having better availability

Since I wrote this article in 2013, many CSP have been incorporation more and more powerful services to their catalalog.

For instance Amazon has service like DynamoDB, RDS, Load Balancers able to balance servers in different availability zones, with Auto-Scaling, SQS for Message Queues, ElastiCache, Lambda, S3…

Many Enterprise find convenient to have all their services offloaded to Amazon, Google Cloud, Microsoft Azure… so they require less Operations and System Administration Engineers.

Other cases

For other cases Dedicated Servers are much more Powerful, faster and cheaper, at the price of being “static” in the sense of attached, not layer abstracted, but all the aspects of your Project have to be taken in count before deciding stepping into or out of the Cloud.

In general terms I would say that the Cloud is for Scaling, but in 2022 it’s also to outsource Operation Services, to fully managed services by the CSPs.