Category Archives: Uncategorized

Quick Access to my selection Last Update: 2022-10-24 09:36:42 Ireland Time / 2022-10-24 01:36:42 PDT-0700 Unix epoch: 1666600602
Cloud Computing Operations Engineering/DevOps/SRE Engineering My Books
Provisioning AWS EC2 Instances with Ansible and Automating Apache deployment with or without using Ansible Dynamic Inventory from Ubuntu 20.04 LTS
Using Ansible in Ubuntu 20.04 LTS to provision to Amazon, and use the Dynamic Inventory (or my own Python 3 code, and alternative Bash code) to create different inventories per group, so you can provision Apache2 in your desired group of instances only.
Published: 2021-Dec Views: 5,609 views
Migrating my 11 years Amazon AWS account services (Postmortem Analysis)
Why I migrated my Services out of Amazon AWS, how I did, the problems I had and the mistakes I did.
Published: 2021-Nov Views: 5,421 views
Upgrading Amazon AWS EC2 Ubuntu 18.04 LTS to Ubuntu 20.04 LTS
Upgrading Amazon EC2 Instance with LAMP from Ubuntu Linux 18.04 LTS to version 20.04 LTS.
Published: 2021-June Views: 3,962 views
How to recover access to your Amazon AWS EC2 instance if you loss your Private Key for SSH
Procedure to get access to an Amazon AWS EC2 Instance and its Data, after you have lost your SSH Private Key and access to the instances using that Key pair.
Published: 2020-Sept Views: 8,274 views
Benchmarking Google Cloud Compute Engine (2015)
The analysis I did from Google Compute Engine, in my project CMIPS, when google launched their GCE in beta.
Paul Nash wrote me and I helped all the Cloud Team to fix some problems. :)
This analysis is a bit old however contains some interesting information.
Published: 2015-Jan
Comparison of Cloud Provider’s Instances performance
From my CMIPS (Cloud Million Instructions per second) a performance comparison of different Instances, from different Cloud Providers, compared to bare metal as well.
Is old, from 2015, however brings very valuable information.
Published: 2015-Jan

The Cloud is for Scaling
The Cloud is for Startups, and for Scaling and for Enterprises. Nothing more. Published: 2013-Sept Views: 15,190 views
Creating a RabbitMQ Docker Container accessed with Python and pika
Published: 2022-July Views: 6,169 views
How to deploy a DigitalOcean droplet (instance) and use userdata
Published: 2022-June Views: 4,101 views
Renewing a SSL Certificate for Apache2 in Ubuntu 20.04
Published: 2022-Mar Views: 6,926 views
Linux Command Line tools I usually install (if they are not on the system)
Published: 2016-March Views: 18,005 views

Troubleshooting upgrading and loading a ZFS module in RHEL7.4
Published: 2018-July Views: 16,928 views
A sample forensic post mortem for a iSCSI Initiator (client) that had connectivity problems to the Server (Troubleshooting)
Published: 2019-August Views: 9,971 views
Creating a content filter for Postfix in PHP
Published: 2016-June Views: 31,159 views
Stopping definitively the massive Distributed DoS attack
Published: 2015-Feb Views: 37,990 views
Stopping and investigating a DoS XMLRPC attack
Published: 2014-August Views: 155,996 views
Stopping a BitTorrent DDoS attack
Published: 2015-January Views: 96,636 views
Dropping caches in Linux, to check if memory is actually being used
Published: 2019-April Views: 5,559 views
Troubleshooting a shell prompt irresponsible/that locks intermittently
Published: 2020-April Views: 7,462 views
Post-Mortem: The mystery of the duplicated Transactions into an e-Commerce
Published: 2020-Nov Views: 7,870 views
Adding a swapfile on the fly as a temporary solution for a Server with few memory
Published: 2020-Nov Views: 4,513 views
Swap, swappiness, Servers not responding
Published: 2021-May Views: 5,775 views
Erasure Code
My project for infinite Storage scaling with no single point of failure, based on Erasure Codes.

Published: 2022-May Views: 5,539 views
Troubleshooting apps in Linux
Published: 2013-November Views: 6,475 views
Performance of Several Languages
Published: 2014-Oct Views: 198,533 views
CSort Multithread versus Quicksort (Java)
Published: 2017-March Views: 26,214 views
CSort my algorithm that heavily beats Quicksort (Java)
Published: 2015-May Views: 19,457 views
Buy my books:
Python 3 Combat Guide
Pages: 403 DIN-A4 PDF DRM-free
Accompanying Source Code: https://gitlab.com/carles.mateo/python_combat_guide
Last Update: v.1.08 2022-05-11

Automating and Provisioning to Amazon Web Services (AWS) with boto3 SDK for Python (plus some Ansible)
Pages: 128 Full Size DIN-A4 PDF DRM-free
Last Update: v.16 2022-01-16

Docker Combat Guide
Learn Docker, focused on Developers, and Docker’s Python 3 SDK
Pages: 178 DIN-A4 PDF DRM-free
Last Update: v.25 2022-07-03

Assemble and upgrade your PC and laptops
Pages: 107 DIN-A4 PDF DRM-free
Last Update: v.0.17 2021-01-11

ZFS on Ubuntu 20.04 LTS
Pages: 159 DIN-A4 PDF DRM-free
Covers ZFS 0.8.3 version. Shows tricks, fix errors, secrets and commands against real systems with LSI Controllers, SAS SLOG ZIL devices, SATA, SSD…
Last Update: v.0.25 2022-02-07

Python 3 Exercises for Beginners
Pages: 200 DIN-A4 PDF DRM-free
Exercises for people starting coding in Python, explaining the solution, tricks, etc…
Last Update: v.48 2022-03-15
Python Open Source Utilities Python PHP Java
CTOP.py SysAdmin tool to get all the System Information at a glance
Published: 2020-Jan Views: 14,758 views
Current Version: 0.8.8 Last Update: 2022-02-13

Simple sample to print colors in Terminal
Published: 2018-May Views: 14,187 views

LDAPGUI.py a simple Python GUI application that queries LDAP
Published: 2020-Jun Views: 9,432 views

cmemgzip.py compress logs (and any file) in memory and replace uncompressed files by .gz when drive has no space left. Supports compressing by blogs to use less memory
Published: 2021-Feb Views: 6,027 views

checkswap.py Monitor the impact of swap memory pages on a live system. Compatible with Python 2.x and 3.x
Published: 2021-May Views: 5,775 views

Carleslibs v. 1.0.8 (2022-06-05) Python Open Source package.
Published: 2021-July (Updated 2022-February)Views: 6,330 views

Cliptype is a utility to paste the Clipboard into the focussed Windows. Ideal for working with Web SSH Terminal clients/QEMU/KVM that don’t support paste
Published: 2021-NovemberViews: 27,554 views
Video: Object Oriented Programming in Python 3 for beginners
Published: 2022-July Views: 2,825 views
For beginners: How to start coding Python with PyCharm and Git
Published: 2022-March Views: 4,307 views
Lesson 0, learning to code in Python for non programmers
Published: 2020-March Views: 9,033 views
Video for beginners: Python for, range, lists, dicts
Published: 2021 July Views: 4,502 views
Programming class for beginners on 2021-11-11.
Published: 2021-November Views: 4,398 views

Sorting an Array of Tuples with Lambda in Python (videos)
Published: 2022-May Views: 3,262 views
Learn to do Unit Testing with pytest in Python 3.
Published: 2021-October Views: 4,281 views

Video: Parse the Tables from a Website with Python pandas
Published: 2022-July Views: 3,055 views
A handy trick command line to get the usages of our Python Methods in the code
Published: 2019-July Views: 7,546 views
A small Python + MySql + Docker program as a sample (plus LAMP PHP sample).
Published: 2021-July Views: 14,292 views

A simple Flask Application, a Star Wars game in Python and Docker.
Published: 2021-July Views: 4,736 views

A simple Python Tic Tac Toe game.
Published: 2021-September Views: 4,799 views

Some weird things from Python 3 that you may not know.
Published: 2021-September Views: 5,289 views

Generating a Word Cloud of Tags in Python.
Published: 2021-September Views: 4,401 views

Some graphics with matplotlib.
Published: 2021-October Views: 3,960 views

Web Top – Displaying top with Python 3 Web Server and carleslibs.
Published: 2021-October Views: 5,109 views

Why I think in Python is not a good idea to raise exceptions inside your methods
Published: 2022-May Views: 3,317 views
A Sudoku Solver in Python, an engineering approach solution (with Source Code)
Published: 2022-April Views: 4,337 views
MT Notation prefix variables system for Python
Last Update: 2021-07-15 Views: 9,488 views
My PHP Script to see WordPress Posts and Views ordered by Views
Published: 2021-August Views: 7,969 views
Improving performance in PHP
Published: 2014-August Views: 35,418 views
Catalonia Framework PHP Open Source

MT Notation prefix variables system for PHP
From: 2014-July Last Update: 2021-07-15 Views: 9,185 views
Java validation Classes for Keyboard
Published: 2020-Dec Views: 31,605 views
MT Notation prefix variables system for Java
From: 2017-March Views: 7,943 views
Docker Commodity Hardware ZFS Architecture
Communicating with Docker Containers via Linux Signals and Python
Published: 2021-Nov Views: 5,151 views
Refreshing settings in a Docker immutable image with Python and Flask
Published: 2020-May Views: 7,742 views
In March 2021, Why I propose you to use Python 3.8, at least, for your Internal Automation Tools in Docker Containers and Ubuntu 20.04 LTS Published: 2021-March Views: 5,662 views
Solving the problem when running a Docker Container: standard_init_linux.go:190: exec user process caused “no such file or directory” Published: 2021-March Views: 9,752 views
Adding my Server as Docker, with PHP Catalonia Framework, explained
Published: 2019-July Views: 10,097 views
A base Dockerfile for my Jenkins (home) deployments
Published: 2021-March Views: 6,334 views
Install jenkins on Docker Ubuntu 20.04 LTS with Blue Ocean pipeline plugin and persistent Volumes in 4 minutes
Published: 2022-June Views: 5,198 views
Migrating some Services from Amazon AWS EC2 to Digital Ocean, using Docker
Published: 2021-Aug Views: 4,850 views
Have a cheap Ubuntu in your Windows or Mac with Docker
Published: 2021-September Views: 4,763 views
Video: How to create a Docker Container for Linux Apache MySQL PHP Python (LAMPP) step by step
Published: 2022-July Views: 2,776 views
Dealing with Performance degradation on ZFS (DRAID) Rebuilds when migrating from a single processor to a multiprocessor platform (Troubleshooting explained)
Published: 2019-June Views: 9,178 views
Solving a persistent MDRAID and ZFS problem in RHEL7.4 (Dual Port SAS drives)
Published: 2018-Oct Views: 8,279 views
Simulating a SAS physical pull out of a drive
Published: 2019-March Views: 8,077 views
Create a small partition on the drives for tests
Published: 2019-April Views: 8,146 views
zpool_watch is an Open Source Python 3 utility that watches your ZFS Pools and open a window in your Linux if there is a problem
Published: 2022-February Views: 3,462 views
Adding a RAMDISK as SLOG ZIL to ZFS
Published: 2020-August Views: 15,066 views
Some handy tricks for working with ZFS
Published: 2019-June Views: 9,405 views
ZFS Improving iSCSI for Block Devices (trick for Volumes)
Published: 2018-Oct Views: 23,396 views
Creating a compressed filesystem with Linux and ZFS
Published: 2018-Sept Views: 11,634 views
Compiling ZFS with RHEL6.10
Published: 2019-August Views: 7,721 views
My talk at OpenZFS 2018 about DRAID (San Francisco, US) Published: 2018-Sept Views: 9,979 views
Extend existing Single ZFS disk with a mirror without losing the Data on the existing drive
Published: 2021-Jan Views: 8,874 views
Resources for Microservices and Business Domain Solutions for the Cloud Architect / Microservices Architect
Published: 2019-Oct Views: 8,161 views
Scaling PhantomJS with PHP
Published: 2015-June Views: 26,508 views
Improving Performance in PHP (Stack and tricks)
Published: 2014-August Views: 35,418 views
Begin developing in Cassandra in Java, PHP or Python
Published: 2014-July Views: 15,807 views
Windows Raspberry Pi Cassandra Relational Databases
Using Docker in Windows 10 without Windows Desktop with Docker Engine and without WSL
Published: 2021-Feb Views: 10,505 views
Solving Windows 10 PRO running Active Directory as Domain Admin
Published: 2021-Jan Views: 5,190 views
Install Windows Subsystem for Linux, WSL 2 on Windows 10 64 bit, with Ubuntu, solution to error WslRegisterDistribution failed with error: 0x80070057
Published: 2021-Jan Views: 35,770 views
Programs I use for Windows in my Workstations
Published: 2019-Sept Views: 6,300 views
Using Windows 10 Appliance in Ubuntu Virtual Box 4.3.10 and later versions
Published: 2015-August Views: 38,819 views
Reinstall PIP only in Windows 10 after it got removed
Published: 2021-March Views: 6,100 views
Install a Media Player on the Raspberry Pi 4
Published: 2020-March Views: 6,151 views
Raspberry Pi 3 and OMSC Media Player
Published: 2015-April Views: 17,149 views
Solving the problem GPIO.setup(self.number, GPIO.IN, self.GPIO_PULL_UPS[self._pull]) RuntimeError: Not running on a RPi! in Ubuntu 20.04LTS
Published: 2021-Feb Views: 9,787 views
CQLSÍ a wrapper to use Cassandra from PHP
Written in 2014, a time when there were no drivers for PHP.
Published: 2014 Views: 3,954 views
Cassandra Universal Driver
A HTTP gateway for all the languages supporting curl/sockets. Written in 2014, a time when there were no drivers for many languages.
Published: 2014 Views: 3,963 views
Solving Oracle error ORA 600 [KGL-heap-size-exceeded]
Published: 2021-Febreruary Views: 21,231 views
Bash Open Source Utilities Bash My Tech Talks Miscellaneous
count_repeated_pattern_in_logs.sh
A easy way to see errors that are repeating, e.g.: NFS/iSCSI timeouts.
Published: 2020-May Views: 8,208 views
backup_partition_in_files.sh
Compressing an unmounted partition to a image file while compressing on the fly, and breaking into 1GB gz files.
Also explains in a funny way about STDIN, STDOUT, STDERR and methodology investigating in deep.
Published: 2020-May Views: 8,783 views
iostat_bandwidth.sh
See the aggregated bandwidth used by all the drives, and the maximum speed achieved.
Published: 2020-Aug Views: 7,442 views
count_lines_of_code.sh
Count the lines in .py Python source files recursively and displays individual and total results.
Published: 2021-Jan Views: 6,769 views
One line script to log the temperature of HDDs and CPUs in Ubuntu
Published: 2021-Jan Views: 6,081 views
compress_old.sh A simple Bash script to compress files in a directory, older than n days
You can use it to delete older files, or perform other commands.
Published: 2021-March Views: 6,819 views
A simple script to upload a pypi/pip package
Is what I use to maintain packages like cmemgzip, carleslibs, etc…
Published: 2021-March Views: 5,147 views
Backup and Restore your Ubuntu Linux Workstations – with support for Wine and Docker
Published: 2022-October Views: 2,734 views
My talk at OpenZFS 2018 about DRAID (San Francisco, US)

Published: 2018-Sept Views: 9,979 views
Google Compute Engine Talk for Group Google Developers Cork, Ireland

How to do autoscaling from 0 using Google Cloud.
Published: 2019-Feb Views: 7,923 views
How is: Working in Cork for IT Engineers
Published: 2020-Feb Views: 8,770 views

A mistake that all the universities are doing
Published: 2018-Jun Views: 10,582 views

Some advice for WFH (Working from Home/Remotely)
Published: 2020-May Views: 11,604 views

My radio program at RAB 2022-06-27 [Catalan and English]
Published: 2022-June Views: 1,395 views

News from the Blog 2022-06-22
Published: 2022-June Views: 2,818 views

News from the Blog 2022-05-22
Published: 2022-May Views: 2,760 views

News from the Blog 2022-04-22
Published: 2022-April Views: 3,248 views

News from the Blog 2022-03-22
Published: 2022-March Views: 3,600 views

News from the Blog 2022-02-22
Published: 2022-February Views: 3,677 views

Old News:

Upgrading the Blog after 5 years, AWS Amazon Web Services, under DoS and Spam attacks

Few days ago I was under a heavy DoS attack.

Nothing new, zombie computers, hackers, pirates, networks of computers… trying to abuse the system and to hack into it. Why? There could be many reasons, from storing pirate movies, trying to use your Server for sending Spam, try to phishing or to host Ransomware pages…

Most of those guys doesn’t know that is almost impossible to Spam from Amazon. Few emails per hour can come out from the Server unless you explicitly requests that update and configure everything.

But I thought it was a great opportunity to force myself to update the Operating System, core tools, versions of PHP and MySql.

Forensics / Postmortem of the incident

The task was divided in two parts:

  • Understanding the origin of the attack
  • Blocking the offending Ip addresses or disabling XMLRPC
  • Making the VM boot again (problems with Amazon AWS)
    • I didn’t know why it was not booting so.
  • Upgrading the OS

I disabled the access to the site while I was working using Amazon Web Services Firewall. Basically I turned access to my ip only. Example: 8.8.8.8/32

I changed 0.0.0.0/0 so the world wide mask to my_Ip/3

That way the logs were reflecting only what I was doing from my Ip.

Dealing with Snapshots and Volumes in AWS

Well the first thing was doing an Snapshot.

After, I tried to boot the original Blog Server (so I don’t stop offering service) but no way, the Server appeared to be dead.

So then I attached the Volume to a new Server with the same base OS, in order to extract (dump) the database. Later I would attach the same Volume to a new Server with the most recent OS and base Software.

Something that is a bit annoying is that the new Instances, the new generation instances, run only in VPC, not in Amazon EC2 Classic. But my static Ip addresses are created for Amazon EC2 Classic, so I could not use them in new generation instances.

I choose the option to see all the All the generations.

Upgrading the system base Software had its own challenges too.

Upgrading the OS / Base Software

My approach was to install an Ubuntu 18.04 LTS, and install the base Software clean, and add any modification I may need.

I wanted to have all the supported packages and a recent version of PHP 7 and the latest Software pieces link Apache or MySQL.

sudo apt update

sudo apt install apache2

sudo apt install mysql-server

sudo apt install php libapache2-mod-php php-mysql

Apache2

Config files that before were working stopped working as the new Apache version requires the files or symlinks under /etc/apache2/sites-enabled/ to end with .conf extension.

Also some directives changed, so some websites will not able to work properly.

Those projects using my Catalonia Framework were affected, although I have this very well documented to make it easy to work with both versions of Apache Http Server, so it was a very straightforward change.

From the previous version I had to change my www.cataloniaframework.com.conf file and enable:

    <Directory /www/www.cataloniaframework.com>
Options Indexes FollowSymLinks MultiViews
AllowOverride All
Order allow,deny
allow from all
</Directory>

Then Open the ports for the Web Server (443 and 80).

sudo ufw allow in "Apache Full"

Then service apache restart

Catalonia Framework Web Site, which is also created with Catalonia Framework itself once restored

MySQL

The problem was to use the most updated version of the Database. I could use one of the backups I keep, from last week, but I wanted more fresh data.

I had the .db files and it should had been very straightforward to copy to /var/lib/mysql/ … if they were the same version. But they weren’t. So I launched an instance with the same base Software as the old previous machine had, installed mysql-server, stopped it, copied the .db files, started it, and then I made a dump with mysqldump –all-databases > 2019-04-29-all-databases.sql

Note, I copied the .db files using the mythical mc, which is a clone from Norton Commander.

Then I stopped that instance and I detached that volume and attached it to the new Blog Instance.

I did a Backup of my original /var/lib/mysql/ files for the purpose of faster restoring if something went wrong.

I mounted it under /mnt/blog_old and did mysql -u root -p < /mnt/blog_old/home/ubuntu/2019-04-29-all-databases.sql

That worked well I had restored the blog. But as I was watching the /var/log/mysql/error.log I noticed some columns were not where they should be. That’s because inadvertently I overwritten the MySql table as well, which in MySQL 5.7 has different structure than in MySQL 5.5. So I screwed. As I previewed this possibility I restored from the backup in seconds.

So basically then I edited my .sql files and removed all that was for the mysql database.

I started MySql, and run the mysql import procedure again. It worked, but I had to recreate the users for all the Databases and Grant them permissions.

GRANT ALL PRIVILEGES ON db_mysqlproxycache.* TO 'wp_dbuser_mysqlproxy'@'localhost' IDENTIFIED BY 'XWy$&{yS@qlC|<¡!?;:-ç';

PHP7

Some modules in my blogs where returning errors in /var/log/apache2/mysite-error.log so I checked that it was due to lack of support of latest PHP versions, and so I patched manually the code or I just disabled the offending plugin.

WordPress

As seen checking the /var/log/apache2/blog.carlesmateo.com-error.log some URLs where not located by WordPress.

For example:

The requested URL /wordpress/wp-json/ was not found on this server

I had to activate modrewrite and then restart Apache.

a2enmod rewrite; service apache2 restart

Making the site more secure

Checking at the logs of Apache, /var/log/apache2/blog.carlesmateo.com-access.log I checked for Ip’s accessing Admin areas, I looked for 404 Errors pointing to intents to exploit any unsafe WP Plugin, I checked for POST protocol as well.

I added to the Ubuntu Uncomplicated Firewall (UFW) the offending Ip’s and patched the xmlrpc.php file to exit always.