Quick Access to my selection Last Update: 2022-08-04 19:59:56 Ireland Time / 2022-08-04 11:59:56 PDT-0700 Unix epoch: 1659639596
Cloud Computing Operations Engineering/DevOps/SRE Engineering My Books
Provisioning AWS EC2 Instances with Ansible and Automating Apache deployment with or without using Ansible Dynamic Inventory from Ubuntu 20.04 LTS
Using Ansible in Ubuntu 20.04 LTS to provision to Amazon, and use the Dynamic Inventory (or my own Python 3 code, and alternative Bash code) to create different inventories per group, so you can provision Apache2 in your desired group of instances only.
Published: 2021-Dec Views: 2,042 views
Migrating my 11 years Amazon AWS account services (Postmortem Analysis)
Why I migrated my Services out of Amazon AWS, how I did, the problems I had and the mistakes I did.
Published: 2021-Nov Views: 1,508 views
Upgrading Amazon AWS EC2 Ubuntu 18.04 LTS to Ubuntu 20.04 LTS
Upgrading Amazon EC2 Instance with LAMP from Ubuntu Linux 18.04 LTS to version 20.04 LTS.
Published: 2021-June Views: 1,458 views
How to recover access to your Amazon AWS EC2 instance if you loss your Private Key for SSH
Procedure to get access to an Amazon AWS EC2 Instance and its Data, after you have lost your SSH Private Key and access to the instances using that Key pair.
Published: 2020-Sept Views: 4,484 views
Benchmarking Google Cloud Compute Engine (2015)
The analysis I did from Google Compute Engine, in my project CMIPS, when google launched their GCE in beta.
Paul Nash wrote me and I helped all the Cloud Team to fix some problems. :)
This analysis is a bit old however contains some interesting information.
Published: 2015-Jan
Comparison of Cloud Provider’s Instances performance
From my CMIPS (Cloud Million Instructions per second) a performance comparison of different Instances, from different Cloud Providers, compared to bare metal as well.
Is old, from 2015, however brings very valuable information.
Published: 2015-Jan

The Cloud is for Scaling
The Cloud is for Startups, and for Scaling and for Enterprises. Nothing more. Published: 2013-Sept Views: 11,481 views
Blocking Russian and Belarus IP Addresses in Ubuntu Firewall until they stop the war and leave Ukraine
Published: 2022-March Views: 1,661 views
Creating a RabbitMQ Docker Container accessed with Python and pika
Published: 2022-July Views: 174 views
How to deploy a DigitalOcean droplet (instance) and use userdata
Published: 2022-June Views: 277 views
Renewing a SSL Certificate for Apache2 in Ubuntu 20.04
Published: 2022-Mar Views: 1,156 views
Linux Command Line tools I usually install (if they are not on the system)
Published: 2016-March Views: 12,678 views

Troubleshooting upgrading and loading a ZFS module in RHEL7.4
Published: 2018-July Views: 11,731 views
A sample forensic post mortem for a iSCSI Initiator (client) that had connectivity problems to the Server (Troubleshooting)
Published: 2019-August Views: 6,046 views
Creating a content filter for Postfix in PHP
Published: 2016-June Views: 25,037 views
Stopping definitively the massive Distributed DoS attack
Published: 2015-Feb Views: 26,295 views
Stopping and investigating a DoS XMLRPC attack
Published: 2014-August Views: 147,763 views
Stopping a BitTorrent DDoS attack
Published: 2015-January Views: 66,121 views
Dropping caches in Linux, to check if memory is actually being used
Published: 2019-April Views: 2,460 views
Troubleshooting a shell prompt irresponsible/that locks intermittently
Published: 2020-April Views: 4,665 views
Post-Mortem: The mystery of the duplicated Transactions into an e-Commerce
Published: 2020-Nov Views: 4,838 views
Adding a swapfile on the fly as a temporary solution for a Server with few memory
Published: 2020-Nov Views: 1,581 views
Swap, swappiness, Servers not responding
Published: 2021-May Views: 2,360 views
Erasure Code
My project for infinite Storage scaling with no single point of failure, based on Erasure Codes.

Published: 2022-May Views: 778 views
Troubleshooting apps in Linux
Published: 2013-November Views: 3,563 views
Performance of Several Languages
Published: 2014-Oct Views: 180,665 views
CSort Multithread versus Quicksort (Java)
Published: 2017-March Views: 22,367 views
CSort my algorithm that heavily beats Quicksort (Java)
Published: 2015-May Views: 14,803 views
Buy my books:
Python 3 Combat Guide
Pages: 403 DIN-A4 PDF DRM-free
Accompanying Source Code: https://gitlab.com/carles.mateo/python_combat_guide
Last Update: v.1.08 2022-05-11

Automating and Provisioning to Amazon Web Services (AWS) with boto3 SDK for Python (plus some Ansible)
Pages: 128 Full Size DIN-A4 PDF DRM-free
Last Update: v.16 2022-01-16

Docker Combat Guide
Learn Docker, focused on Developers, and Docker’s Python 3 SDK
Pages: 178 DIN-A4 PDF DRM-free
Last Update: v.25 2022-07-03

Assemble and upgrade your PC and laptops
Pages: 107 DIN-A4 PDF DRM-free
Last Update: v.0.17 2021-01-11

ZFS on Ubuntu 20.04 LTS
Pages: 159 DIN-A4 PDF DRM-free
Covers ZFS 0.8.3 version. Shows tricks, fix errors, secrets and commands against real systems with LSI Controllers, SAS SLOG ZIL devices, SATA, SSD…
Last Update: v.0.25 2022-02-07

Python 3 Exercises for Beginners
Pages: 200 DIN-A4 PDF DRM-free
Exercises for people starting coding in Python, explaining the solution, tricks, etc…
Last Update: v.48 2022-03-15
Python Open Source Utilities Python PHP Java
CTOP.py SysAdmin tool to get all the System Information at a glance
Published: 2020-Jan Views: 9,616 views
Current Version: 0.8.8 Last Update: 2022-02-13

Simple sample to print colors in Terminal
Published: 2018-May Views: 9,188 views

LDAPGUI.py a simple Python GUI application that queries LDAP
Published: 2020-Jun Views: 5,147 views

cmemgzip.py compress logs (and any file) in memory and replace uncompressed files by .gz when drive has no space left. Supports compressing by blogs to use less memory
Published: 2021-Feb Views: 2,697 views

checkswap.py Monitor the impact of swap memory pages on a live system. Compatible with Python 2.x and 3.x
Published: 2021-May Views: 2,360 views

Carleslibs v. 1.0.8 (2022-06-05) Python Open Source package.
Published: 2021-July (Updated 2022-February)Views: 2,167 views

Cliptype is a utility to paste the Clipboard into the focussed Windows. Ideal for working with Web SSH Terminal clients/QEMU/KVM that don’t support paste
Published: 2021-NovemberViews: 5,859 views
Video: Object Oriented Programming in Python 3 for beginners
Published: 2022-July Views: 162 views
For beginners: How to start coding Python with PyCharm and Git
Published: 2022-March Views: 1,058 views
Lesson 0, learning to code in Python for non programmers
Published: 2020-March Views: 5,236 views
Video for beginners: Python for, range, lists, dicts
Published: 2021 July Views: 1,676 views
Programming class for beginners on 2021-11-11.
Published: 2021-November Views: 1,525 views

Sorting an Array of Tuples with Lambda in Python (videos)
Published: 2022-May Views: 525 views
Learn to do Unit Testing with pytest in Python 3.
Published: 2021-October Views: 1,231 views

Video: Parse the Tables from a Website with Python pandas
Published: 2022-July Views: 107 views
A handy trick command line to get the usages of our Python Methods in the code
Published: 2019-July Views: 4,804 views
A small Python + MySql + Docker program as a sample (plus LAMP PHP sample).
Published: 2021-July Views: 6,050 views

A simple Flask Application, a Star Wars game in Python and Docker.
Published: 2021-July Views: 1,736 views

A simple Python Tic Tac Toe game.
Published: 2021-September Views: 1,382 views

Some weird things from Python 3 that you may not know.
Published: 2021-September Views: 1,542 views

Generating a Word Cloud of Tags in Python.
Published: 2021-September Views: 1,272 views

Some graphics with matplotlib.
Published: 2021-October Views: 1,358 views

Web Top – Displaying top with Python 3 Web Server and carleslibs.
Published: 2021-October Views: 1,214 views

Why I think in Python is not a good idea to raise exceptions inside your methods
Published: 2022-May Views: 660 views
A Sudoku Solver in Python, an engineering approach solution (with Source Code)
Published: 2022-April Views: 769 views
MT Notation prefix variables system for Python
Last Update: 2021-07-15 Views: 5,099 views
My PHP Script to see WordPress Posts and Views ordered by Views
Published: 2021-August Views: 1,810 views
a href=”/2014/08/11/improving-performance-in-php/” target=”_blank” rel=”noopener noreferrer”>Improving performance in PHP
Published: 2014-August Views: 28,857 views
Catalonia Framework PHP Open Source

MT Notation prefix variables system for PHP
From: 2014-July Last Update: 2021-07-15 Views: 5,604 views
Java validation Classes for Keyboard
Published: 2020-Dec Views: 28,141 views
MT Notation prefix variables system for Java
From: 2017-March Views: 4,546 views
Docker Commodity Hardware ZFS Architecture
Communicating with Docker Containers via Linux Signals and Python
Published: 2021-Nov Views: 1,594 views
Refreshing settings in a Docker immutable image with Python and Flask
Published: 2020-May Views: 4,852 views
In March 2021, Why I propose you to use Python 3.8, at least, for your Internal Automation Tools in Docker Containers and Ubuntu 20.04 LTS Published: 2021-March Views: 2,839 views
Solving the problem when running a Docker Container: standard_init_linux.go:190: exec user process caused “no such file or directory” Published: 2021-March Views: 5,225 views
Adding my Server as Docker, with PHP Catalonia Framework, explained
Published: 2019-July Views: 6,704 views
A base Dockerfile for my Jenkins (home) deployments
Published: 2021-March Views: 3,218 views
Install jenkins on Docker Ubuntu 20.04 LTS with Blue Ocean pipeline plugin and persistent Volumes in 4 minutes
Published: 2022-June Views: 289 views
Migrating some Services from Amazon AWS EC2 to Digital Ocean, using Docker
Published: 2021-Aug Views: 2,018 views
Have a cheap Ubuntu in your Windows or Mac with Docker
Published: 2021-September Views: 1,829 views
Video: How to create a Docker Container for Linux Apache MySQL PHP Python (LAMPP) step by step
Published: 2022-July Views: 205 views
Dealing with Performance degradation on ZFS (DRAID) Rebuilds when migrating from a single processor to a multiprocessor platform (Troubleshooting explained)
Published: 2019-June Views: 5,694 views
Solving a persistent MDRAID and ZFS problem in RHEL7.4 (Dual Port SAS drives)
Published: 2018-Oct Views: 5,291 views
Simulating a SAS physical pull out of a drive
Published: 2019-March Views: 5,130 views
Create a small partition on the drives for tests
Published: 2019-April Views: 5,226 views
zpool_watch is an Open Source Python 3 utility that watches your ZFS Pools and open a window in your Linux if there is a problem
Published: 2022-February Views: 686 views
Adding a RAMDISK as SLOG ZIL to ZFS
Published: 2020-August Views: 7,960 views
Some handy tricks for working with ZFS
Published: 2019-June Views: 5,611 views
ZFS Improving iSCSI for Block Devices (trick for Volumes)
Published: 2018-Oct Views: 9,356 views
Creating a compressed filesystem with Linux and ZFS
Published: 2018-Sept Views: 7,115 views
Compiling ZFS with RHEL6.10
Published: 2019-August Views: 4,867 views
My talk at OpenZFS 2018 about DRAID (San Francisco, US) Published: 2018-Sept Views: 6,525 views
Extend existing Single ZFS disk with a mirror without losing the Data on the existing drive
Published: 2021-Jan Views: 4,073 views
Resources for Microservices and Business Domain Solutions for the Cloud Architect / Microservices Architect
Published: 2019-Oct Views: 5,352 views
Scaling PhantomJS with PHP
Published: 2015-June Views: 22,023 views
Improving Performance in PHP (Stack and tricks)
Published: 2014-August Views: 28,857 views
Begin developing in Cassandra in Java, PHP or Python
Published: 2014-July Views: 12,238 views
Windows Raspberry Pi Cassandra Relational Databases
Using Docker in Windows 10 without Windows Desktop with Docker Engine and without WSL
Published: 2021-Feb Views: 2,680 views
Solving Windows 10 PRO running Active Directory as Domain Admin
Published: 2021-Jan Views: 2,434 views
Install Windows Subsystem for Linux, WSL 2 on Windows 10 64 bit, with Ubuntu, solution to error WslRegisterDistribution failed with error: 0x80070057
Published: 2021-Jan Views: 16,882 views
Programs I use for Windows in my Workstations
Published: 2019-Sept Views: 3,441 views
Using Windows 10 Appliance in Ubuntu Virtual Box 4.3.10 and later versions
Published: 2015-August Views: 33,618 views
Reinstall PIP only in Windows 10 after it got removed
Published: 2021-March Views: 2,559 views
Install a Media Player on the Raspberry Pi 4
Published: 2020-March Views: 3,235 views
Raspberry Pi 3 and OMSC Media Player
Published: 2015-April Views: 13,481 views
Solving the problem GPIO.setup(self.number, GPIO.IN, self.GPIO_PULL_UPS[self._pull]) RuntimeError: Not running on a RPi! in Ubuntu 20.04LTS
Published: 2021-Feb Views: 4,950 views
CQLSÍ a wrapper to use Cassandra from PHP
Written in 2014, a time when there were no drivers for PHP.
Published: 2014 Views: 930 views
Cassandra Universal Driver
A HTTP gateway for all the languages supporting curl/sockets. Written in 2014, a time when there were no drivers for many languages.
Published: 2014 Views: 924 views
Solving Oracle error ORA 600 [KGL-heap-size-exceeded]
Published: 2021-Febreruary Views: 9,782 views
Bash Open Source Utilities Bash My Tech Talks Miscellaneous
count_repeated_pattern_in_logs.sh
A easy way to see errors that are repeating, e.g.: NFS/iSCSI timeouts.
Published: 2020-May Views: 4,949 views
backup_partition_in_files.sh
Compressing an unmounted partition to a image file while compressing on the fly, and breaking into 1GB gz files.
Also explains in a funny way about STDIN, STDOUT, STDERR and methodology investigating in deep.
Published: 2020-May Views: 5,285 views
iostat_bandwidth.sh
See the aggregated bandwidth used by all the drives, and the maximum speed achieved.
Published: 2020-Aug Views: 4,353 views
count_lines_of_code.sh
Count the lines in .py Python source files recursively and displays individual and total results.
Published: 2021-Jan Views: 3,329 views
One line script to log the temperature of HDDs and CPUs in Ubuntu
Published: 2021-Jan Views: 2,539 views
compress_old.sh A simple Bash script to compress files in a directory, older than n days
You can use it to delete older files, or perform other commands.
Published: 2021-March Views: 3,084 views
A simple script to upload a pypi/pip package
Is what I use to maintain packages like cmemgzip, carleslibs, etc… Published: 2021-March Views: 2,509 views
My talk at OpenZFS 2018 about DRAID (San Francisco, US)

Published: 2018-Sept Views: 6,525 views
Google Compute Engine Talk for Group Google Developers Cork, Ireland

How to do autoscaling from 0 using Google Cloud.
Published: 2019-Feb Views: 4,863 views
How is: Working in Cork for IT Engineers
Published: 2020-Feb Views: 5,352 views

How is: Working in Barcelona for IT Engineers
Published: 2020-Feb Views: 5,188 views

A mistake that all the universities are doing
Published: 2018-Jun Views: 7,251 views

Some advice for WFH (Working from Home/Remotely)
Published: 2020-May Views: 7,665 views

My radio program at RAB 2022-06-27 [Catalan and English]
Published: 2022-June Views: 439 views

News from the Blog 2022-06-22
Published: 2022-June Views: 278 views

News from the Blog 2022-05-22
Published: 2022-May Views: 515 views

News from the Blog 2022-04-22
Published: 2022-April Views: 677 views

News from the Blog 2022-03-22
Published: 2022-March Views: 1,039 views

News from the Blog 2022-02-22
Published: 2022-February Views: 1,137 views

Old News:

Docker with Ubuntu with telnet server and Python code to access via telnet

Explanations building the Container and running the python code

Here you can see this Python code to connect via Telnet and executing a command in a Server:

File: telnet_demo.py

#!/usr/bin/env python3
import telnetlib

s_host = "localhost"
s_user = "telnet"
s_password = "telnet"

o_tn = telnetlib.Telnet(s_host)

o_tn.read_until(b"login: ")
o_tn.write(s_user.encode('ascii') + b"\n")

o_tn.read_until(b"Password: ")

o_tn.write(s_password.encode('ascii') + b"\n")

o_tn.write(b"hostname\n")
o_tn.write(b"uname -a\n")
o_tn.write(b"ls -hal /\n")
o_tn.write(b"exit\n")

print(o_tn.read_all().decode('ascii'))

File: Dockerfile

FROM ubuntu:20.04

MAINTAINER Carles Mateo

ARG DEBIAN_FRONTEND=noninteractive

# This will make sure printing in the Screen when running in detached mode
ENV PYTHONUNBUFFERED=1

RUN apt-get update -y && apt install -y sudo telnetd vim systemctl  && apt-get clean

RUN adduser -gecos --disabled-password --shell /bin/bash telnet

RUN echo "telnet:telnet" | chpasswd

EXPOSE 23

CMD systemctl start inetd; while [ true ]; do sleep 60; done

You can see that I use chpasswd command to change the password for the user telnet and set it to telnet. That deals with the complexity of setting the encrypted password.

File: build_docker.sh

#!/bin/bash

s_DOCKER_IMAGE_NAME="ubuntu_telnet"

echo "We will build the Docker Image and name it: ${s_DOCKER_IMAGE_NAME}"
echo "After, we will be able to run a Docker Container based on it."

printf "Removing old image %s\n" "${s_DOCKER_IMAGE_NAME}"
sudo docker rm "${s_DOCKER_IMAGE_NAME}"

printf "Creating Docker Image %s\n" "${s_DOCKER_IMAGE_NAME}"
sudo docker build -t ${s_DOCKER_IMAGE_NAME} .
# If you don't want to use cache this is your line
# sudo docker build -t ${s_DOCKER_IMAGE_NAME} . --no-cache

i_EXIT_CODE=$?
if [ $i_EXIT_CODE -ne 0 ]; then
    printf "Error. Exit code %s\n" ${i_EXIT_CODE}
    exit
fi

echo "Ready to run ${s_DOCKER_IMAGE_NAME} Docker Container"
echo "To run in type: sudo docker run -it -p 23:23 --name ${s_DOCKER_IMAGE_NAME} ${s_DOCKER_IMAGE_NAME}"

When you run sudo ./build_docker.sh the image will be built. Then run it with:

sudo docker run -it -p 23:23 --name ubuntu_telnet ubuntu_telnet

If you get an error indicating that the port is in use, then your computer has already a process listening on the port 23, use another.

You will be able to stop the Container by pressing CTRL + C

From another terminal run the Python program:

python3 ./telnet_demo.py

Validate IP Addresses and Networks with CIDR in Python

Python has a built-in package named ipaddress

You don’t need to install anything to use it.

This simple code shows how to use it

import ipaddress


def check_ip(s_ip_or_net):
    b_valid = True
    try:
        # The IP Addresses are expected to be passed without / even if it's /32 it would fail
        # If it uses / so, the CIDR notation, check it as a Network, even if it's /32
        if "/" in s_ip_or_net:
            o_net = ipaddress.ip_network(s_ip_or_net)
        else:
            o_ip = ipaddress.ip_address(s_ip_or_net)

    except ValueError:
        b_valid = False

    return b_valid


if __name__ == "__main__":
    a_ips = ["127.0.0.2.4",
             "127.0.0.0",
             "192.168.0.0",
             "192.168.0.1",
             "192.168.0.1 ",
             "192.168.0. 1",
             "192.168.0.1/32",
             "192.168.0.1 /32",
             "192.168.0.0/32",
             "192.0.2.0/255.255.255.0",
             "0.0.0.0/31",
             "0.0.0.0/32",
             "0.0.0.0/33",
             "1.2.3.4",
             "1.2.3.4/24",
             "1.2.3.0/24"]

    for s_ip in a_ips:
        b_success = check_ip(s_ip)
        if b_success is True:
            print(f"The IP Address or Network {s_ip} is valid")
        else:
            print(f"The IP Address or Network {s_ip} is not valid")

And the output is like this:

The IP Address or Network 127.0.0.2.4 is not valid
The IP Address or Network 127.0.0.0 is valid
The IP Address or Network 192.168.0.0 is valid
The IP Address or Network 192.168.0.1 is valid
The IP Address or Network 192.168.0.1  is not valid
The IP Address or Network 192.168.0. 1 is not valid
The IP Address or Network 192.168.0.1/32 is valid
The IP Address or Network 192.168.0.1 /32 is not valid
The IP Address or Network 192.168.0.0/32 is valid
The IP Address or Network 192.0.2.0/255.255.255.0 is valid
The IP Address or Network 0.0.0.0/31 is valid
The IP Address or Network 0.0.0.0/32 is valid
The IP Address or Network 0.0.0.0/33 is not valid
The IP Address or Network 1.2.3.4 is valid
The IP Address or Network 1.2.3.4/24 is not valid
The IP Address or Network 1.2.3.0/24 is valid

As you can read in the code comments, ipaddress.ip_address() will not validate an IP Address with the CIDR notation, even if it’s /32.

You should strip the /32 or use ipaddress.ip_network() instead.

As you can see 1.2.3.4/24 is returned as not valid.

You can pass the parameter strict=False and it will be returned as valid.

ipaddress.ip_network(s_ip_or_net, strict=False)

See where is the space used in your Android phone from Ubuntu Terminal

So, you may have your Android phone full and you don’t know where the space is.

You may have tried Apps for Android but none shows the information in the detail you would like. Linux to the rescue.

First of all you need a cable able to transfer Data. It is a cable that will be connected to your computer, normally with an USB 3.0 cable and to your smartphone, normally with USB-C.

Sometimes phone’s connectors are dirty and don’t allow a stable connection. Your connections should allow a stable connection, otherwise the connection will be interrupted in the middle.

Once you connect the Android smartphone to the computer, unlock the phone and authorize the data connection.

You’ll see that your computer recognizes the phone:

Open the terminal and enter this directory:

cd /run/user/1000/gvfs/

Here you will see your device and the name is very evident.

The usual is to have just one device listed, but if you had several Android devices attached you may want to query first, in order to identify it.

The Android devices use a protocol named Media Transfer Protocol (MTP) when connecting to the USB port, and that’s different on the typical way to access the USB port.

usb-devices  | grep "Manufacturer=" -B 3

Run this command to see all the devices connected to the USB.

You may see Manufacturer=SAMSUNG or Manufacturer=OnePlus etc…

The information returned will allow you to identify your device in /run/user/1000/gvfs/

You may get different type of outputs, but if you get:

T:  Bus=02 Lev=01 Prnt=01 Port=00 Cnt=01 Dev#= 13 Spd=480 MxCh= 0

your device can be accessed inside:

cd mtp\:host\=%5Busb%3A002%2C013%5D/

There you’ll see Card and Phone.

You can enter the Phone internal storage or the SD Card storage directory:

cd Phone

To see how the space is distributed nicely I recommend you to use the program ncdu if you don’t have you can install it with:

sudo apt install ncdu

Then run ncdu:

ncdu

It will calculate the space…

… and let you know, sorted from more to less, and will allow you to browse the sub-directories with the keyboard arrow keys and enter to get major clarity.

For example, in my case I saw 8.5 GB in the folder Movies on the phone, where I didn’t download any movie, so I checked.

I entered inside by pressing Enter:

So Instagram App was keeping a copy all the videos I uploaded, in total 8.5 GB of my Phone’s space and never releasing them!.

Example for the SD card, where the usage was as expected:

News from the blog 2022-07-22

Carles in the News

For all my friends and followers, I started to translate my radio space “El nou món digital” (in Catalan) to English “The New Digital World”. I cover Science, Technology, Entertainment and Video games.

You can see the first programs I translated here:

Catalan and English programs RAB

And the script and links mentioned here:

https://blog.carlesmateo.com/2022/06/20/rab-el-nou-mon-digital-2022-06-27-ca/

Social

I’ve created an Instagram fan page for me / for the blog.

It is open for everybody.

https://www.instagram.com/blog_carlesmateo/

Videos for learning how to code

Learning How to code Python Unit Tests with pytest from the scratch, for beginners:

I added it to my series of videos:

https://blog.carlesmateo.com/learn-python-3-unit-testing/

Video for learning how to use RabbitMQ with Python in a Docker Container

https://blog.carlesmateo.com/2022/07/20/creating-a-rabbitmq-docker-container-accessed-with-python-and-pika/

Site carlesmateo.com

I use mostly this site https://blog.carlesmateo.com to centralize everything, so I’ve kept http://carlesmateo.com as a simple landing page made with my old (from 2013) ultra fast PHP Framework Catalonia Framework.

I decided to create a Jenkins Pipeline to deploy any updates to this pages and I updated it a bit at least to provide the most common information searched.

Don’t expect anything fancy at Front End level. :)

Cloud

I created a video about how to provision a Ubuntu Droplet in Digital Ocean.

It’s just for beginners, or if you used other CSP’s and you wonder how Digital Ocean user interface is.

It is really easy, to be honest. Amazon AWS should learn from them.

I also created another about how to provision using User Data Cloud Init feature:

https://blog.carlesmateo.com/2022/06/25/how-to-deploy-a-digitalocean-droplet-instance-and-use-userdata/

Books

My Books

I have updated Docker Combat Guide to show how to work with different users in the Dockerfile and accessing an interactive terminal.

I have also added how to create a Jenkins containerized.

Books I bought > CI/CD

I recommend you these books:

I’ve created a video about how to deploy jenkins in Docker, following the official documentation, in 4 minutes.

Install jenkins on Docker in ubuntu in 4 minutes

And posted an article about solving the error load key invalid format when provisioning from Jenkins with your SSH .pem Key in a variable.

https://blog.carlesmateo.com/2022/07/05/solving-linux-load-key-ssh_yourserver-invalid-format-when-provisioning-from-jenkins/

Open Source from Carles

CTOP.py

I have released a new version of CTOP, ctop.py version 0.8.9.

This version fixes few bugs, and adds better Unit Testing Code Coverage, and is integrated with jenkins (provides Jenkinsfile and a Dockerfile ready to automate the testing pipeline)

Sudo ku solver, Sudoku solver in Python, and Engineering solving problem approach

I’ve created this video explaining my experience writing a program to solve two impossible, very annoying Sudokus. :)

https://blog.carlesmateo.com/2022/04/26/working-on-a-sudoku-solver-in-python-source-code/

Commander Turtle: a small program in Python so children can learn to code by drawing

Children can learn to code and share their scripts, which are comma separated, easily.

Commander Turtle

My life at Activision Blizzard

We have released World of Warcraft Dragonflight Alpha.

In the sync meetings I lead with Wow SRE and product Team I was informed that streaming would be open. Myself I was granted to stream over twitch, but so far I didn’t want to stream video games in my engineering channels. It’s different kind of audiences IMO. Let me know if you would like to get video game streams in my streaming channels.

Lich King

Humor

If you have been in a madness of Servers of a Cluster getting irresponsible and having to cold reboot them from remote hands iDracs or similar, you know why my friends sent me this image :D

Creating a RabbitMQ Docker Container accessed with Python and pika

In this video, that I streamed on Twitch, I demonstrate the code showed here.

I launch the Docker Container and operated it a bit, so you can get to learn few tricks.

I created the RabbitMQ Docker installation based on the official RabbitMQ installation instructions for Ubuntu/Debian:

https://www.rabbitmq.com/install-debian.html#apt-cloudsmith

One interesting aspect is that I cover how the messages are delivered as byte sequence. I show this by sending Unicode characters

Files in the project

Dockerfile

FROM ubuntu:20.04

MAINTAINER Carles Mateo

ARG DEBIAN_FRONTEND=noninteractive

# This will make sure printing in the Screen when running in dettached mode
ENV PYTHONUNBUFFERED=1

ARG PATH_RABBIT_INSTALL=/tmp/rabbit_install/

ARG PATH_RABBIT_APP_PYTHON=/opt/rabbit_python/

RUN mkdir $PATH_RABBIT_INSTALL

COPY cloudsmith.sh $PATH_RABBIT_INSTALL

RUN chmod +x ${PATH_RABBIT_INSTALL}cloudsmith.sh

RUN apt-get update -y && apt install -y sudo python3 python3-pip mc htop less strace zip gzip lynx && apt-get clean

RUN ${PATH_RABBIT_INSTALL}cloudsmith.sh

RUN service rabbitmq-server start

RUN mkdir $PATH_RABBIT_APP_PYTHON

COPY requirements.txt $PATH_RABBIT_APP_PYTHON

WORKDIR $PATH_RABBIT_APP_PYTHON

RUN pwd

RUN pip install -r requirements.txt

COPY *.py $PATH_RABBIT_APP_PYTHON

COPY loop_send_get_messages.sh $PATH_RABBIT_APP_PYTHON

RUN chmod +x loop_send_get_messages.sh

CMD ./loop_send_get_messages.sh

cloudsmith.sh

#!/usr/bin/sh
# From: https://www.rabbitmq.com/install-debian.html#apt-cloudsmith

sudo apt-get update -y && apt-get install curl gnupg apt-transport-https -y

## Team RabbitMQ's main signing key
curl -1sLf "https://keys.openpgp.org/vks/v1/by-fingerprint/0A9AF2115F4687BD29803A206B73A36E6026DFCA" | sudo gpg --dearmor | sudo tee /usr/share/keyrings/com.rabbitmq.team.gpg > /dev/null
## Cloudsmith: modern Erlang repository
curl -1sLf https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-erlang/gpg.E495BB49CC4BBE5B.key | sudo gpg --dearmor | sudo tee /usr/share/keyrings/io.cloudsmith.rabbitmq.E495BB49CC4BBE5B.gpg > /dev/null
## Cloudsmith: RabbitMQ repository
curl -1sLf https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-server/gpg.9F4587F226208342.key | sudo gpg --dearmor | sudo tee /usr/share/keyrings/io.cloudsmith.rabbitmq.9F4587F226208342.gpg > /dev/null

## Add apt repositories maintained by Team RabbitMQ
sudo tee /etc/apt/sources.list.d/rabbitmq.list <<EOF
## Provides modern Erlang/OTP releases
##
deb [signed-by=/usr/share/keyrings/io.cloudsmith.rabbitmq.E495BB49CC4BBE5B.gpg] https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-erlang/deb/ubuntu bionic main
deb-src [signed-by=/usr/share/keyrings/io.cloudsmith.rabbitmq.E495BB49CC4BBE5B.gpg] https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-erlang/deb/ubuntu bionic main

## Provides RabbitMQ
##
deb [signed-by=/usr/share/keyrings/io.cloudsmith.rabbitmq.9F4587F226208342.gpg] https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-server/deb/ubuntu bionic main
deb-src [signed-by=/usr/share/keyrings/io.cloudsmith.rabbitmq.9F4587F226208342.gpg] https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-server/deb/ubuntu bionic main
EOF

## Update package indices
sudo apt-get update -y

## Install Erlang packages
sudo apt-get install -y erlang-base \
                        erlang-asn1 erlang-crypto erlang-eldap erlang-ftp erlang-inets \
                        erlang-mnesia erlang-os-mon erlang-parsetools erlang-public-key \
                        erlang-runtime-tools erlang-snmp erlang-ssl \
                        erlang-syntax-tools erlang-tftp erlang-tools erlang-xmerl

## Install rabbitmq-server and its dependencies
sudo apt-get install rabbitmq-server -y --fix-missing

build_docker.sh

#!/bin/bash

s_DOCKER_IMAGE_NAME="rabbitmq"

echo "We will build the Docker Image and name it: ${s_DOCKER_IMAGE_NAME}"
echo "After, we will be able to run a Docker Container based on it."

printf "Removing old image %s\n" "${s_DOCKER_IMAGE_NAME}"
sudo docker rm "${s_DOCKER_IMAGE_NAME}"

printf "Creating Docker Image %s\n" "${s_DOCKER_IMAGE_NAME}"
sudo docker build -t ${s_DOCKER_IMAGE_NAME} . --no-cache

i_EXIT_CODE=$?
if [ $i_EXIT_CODE -ne 0 ]; then
    printf "Error. Exit code %s\n" ${i_EXIT_CODE}
    exit
fi

echo "Ready to run ${s_DOCKER_IMAGE_NAME} Docker Container"
echo "To run in type: sudo docker run -it --name ${s_DOCKER_IMAGE_NAME} ${s_DOCKER_IMAGE_NAME}"
echo "or just use run_in_docker.sh"

requirements.txt

pika

loop_send_get_messages.sh

#!/bin/bash

echo "Starting RabbitMQ"
service rabbitmq-server start

echo "Launching consumer in background which will be listening and executing the callback function"
python3 rabbitmq_getfrom.py &

while true; do

    i_MESSAGES=$(( RANDOM % 10 ))

    echo "Sending $i_MESSAGES messages"
    for i_MESSAGE in $(seq 1 $i_MESSAGES); do
        python3 rabbitmq_sendto.py
    done

    echo "Sleeping 5 seconds"
    sleep 5

done

echo "Exiting loop"

rabbitmq_sendto.py

#!/usr/bin/env python3
import pika
import time

connection = pika.BlockingConnection(pika.ConnectionParameters(host="localhost"))

channel = connection.channel()

channel.queue_declare(queue="hello")

s_now = str(time.time())

s_message = "Hello World! " + s_now + " Testing Unicode: çÇ àá😀"
channel.basic_publish(exchange="", routing_key="hello", body=s_message)
print(" [x] Sent '" + s_message + "'")
connection.close()

rabbitmq_getfrom.py

#!/usr/bin/env python3
import pika


def callback(ch, method, properties, body):
    # print(f" [x] Received in channel: {ch} method: {method} properties: {properties} body: {body}")
    print(f" [x] Received body: {body}")


connection = pika.BlockingConnection(pika.ConnectionParameters(host="localhost"))

channel = connection.channel()

channel.queue_declare(queue="hello")

print(" [*] Waiting for messages. To exit press Ctrl+C")

# This will loop
channel.basic_consume(queue="hello", on_message_callback=callback)
channel.start_consuming()

print("Finishing consumer")

Video: Parse the Tables from a Website with Python pandas

A quick video, of 3 minutes, that shows you how it works.

If you don’t have pandas installed you’ll have to install it and lxml, otherwise you’ll get an error:

  File "/home/carles/Desktop/code/carles/blog.carlesmateo.com-source-code/venv/lib/python3.8/site-packages/pandas/io/html.py", line 872, in _parser_dispatch
    raise ImportError("lxml not found, please install it")
ImportError: lxml not found, please install it

You can install both from PyCharm or from command line with:

pip install pandas
pip install lxml

And here the source code:

import pandas as pd


if __name__ == "__main__":

    # Do not truncate the data when printing
    pd.set_option('display.max_colwidth', None)
    # Do not truncate due to length of all the columns
    pd.set_option('display.max_columns', None)
    pd.set_option('display.max_rows', None)
    pd.set_option('display.width', 2000)
    # pd.set_option('display.float_format', '{:20,.2f}'.format)

    o_pd_my_movies = pd.read_html("https://blog.carlesmateo.com/movies-i-saw/")
    print(len(o_pd_my_movies))

    print(o_pd_my_movies[0])

Video: How to create a Docker Container for LAMPP step by step

How to create a Docker Container for Linux Apache MySQL PHP and Python for beginners.

Note: Containers are not persistent. Use this for tests only. If you want to keep persistent information use Volumes.

Sources: https://gitlab.com/carles.mateo/blog.carlesmateo.com-source-code/-/tree/master/twitch/live_20220708_dockerfile_lamp

File: Dockerfile

FROM ubuntu:20.04

MAINTAINER Carles Mateo

ARG DEBIAN_FRONTEND=noninteractive

RUN apt update && \
    apt install -y vim python3-pip &&  \
    apt install -y net-tools mc vim htop less strace zip gzip lynx && \
    apt install -y apache2 mysql-server ntpdate libapache2-mod-php7.4 mysql-server php7.4-mysql php-dev libmcrypt-dev php-pear && \
    apt install -y git && apt autoremove && apt clean && \
    pip3 install pytest

RUN a2enmod rewrite

RUN echo "Europe/Ireland" | tee /etc/timezone

ENV APACHE_RUN_USER  www-data
ENV APACHE_RUN_GROUP www-data
ENV APACHE_LOG_DIR   /var/log/apache2
ENV APACHE_PID_FILE  /var/run/apache2/apache2.pid
ENV APACHE_RUN_DIR   /var/run/apache2
ENV APACHE_LOCK_DIR  /var/lock/apache2
ENV APACHE_LOG_DIR   /var/log/apache2

COPY phpinfo.php /var/www/html/

RUN service apache2 restart

EXPOSE 80

CMD ["/usr/sbin/apache2", "-D", "FOREGROUND"]

File: phpinfo.php

<html>
<?php

// Show all information, defaults to INFO_ALL
phpinfo();

// Show just the module information.
// phpinfo(8) yields identical results.
phpinfo(INFO_MODULES);
?>
</html>

File: build_docker.sh

#!/bin/bash

s_DOCKER_IMAGE_NAME="lampp"

echo "We will build the Docker Image and name it: ${s_DOCKER_IMAGE_NAME}"
echo "After, we will be able to run a Docker Container based on it."

printf "Removing old image %s\n" "${s_DOCKER_IMAGE_NAME}"
sudo docker rm "${s_DOCKER_IMAGE_NAME}"

printf "Creating Docker Image %s\n" "${s_DOCKER_IMAGE_NAME}"
# sudo docker build -t ${s_DOCKER_IMAGE_NAME} . --no-cache
sudo docker build -t ${s_DOCKER_IMAGE_NAME} .

i_EXIT_CODE=$?
if [ $i_EXIT_CODE -ne 0 ]; then
    printf "Error. Exit code %s\n" ${i_EXIT_CODE}
    exit
fi

echo "Ready to run ${s_DOCKER_IMAGE_NAME} Docker Container"
echo "To run in type: sudo docker run -p 80:80 --name ${s_DOCKER_IMAGE_NAME} ${s_DOCKER_IMAGE_NAME}"
echo "or just use run_in_docker.sh"

echo
echo "If you want to debug do:"
echo "docker exec -i -t ${s_DOCKER_IMAGE_NAME} /bin/bash"

Solving Linux Load key “ssh_yourserver”: invalid format when provisioning from Jenkins

If you are getting an error like this when you try to provision using rsync or running commands from SSH from a Docker Instance from a worker node in Jenkins, having your SSH Key as a variable in Jenkins, here is a way to solve it.

These are the kind of errors that you’ll be receiving:

Load key "ssh_yourserver": invalid format

web@myserver.carlesmateo.com: Permission denied (publickey).

rsync: connection unexpectedly closed (0 bytes received so far) [sender]

rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.3]

script returned exit code 255

So this applies if you copied your .pem file as text and pasted in a variable in Jenkins.

You’ll find yourself with the load key invalid format error.

I would suggest to use tokens and Vault or Consul instead of pasting a SSH Key, but if you need to just solve this ASAP that’s the trick that you need.

First encode your key with base64 without any wrapping. This is done with this command:

cat keys/key_azure_myserver_carlesmateo_com.pem | base64 --wrap=0

In your Jenkins steps you’ll add this code:

#!/bin/bash
echo "Creating credentials"
echo $SSH_YOURSERVER | base64 --decode > ssh_yourserver
echo "Setting permissions"
chmod 600 ssh_yourserver

Having a certificate then you can define new steps that will deploy to Production by rsyncing:

#!/bin/bash
echo "Deploying www..."
rsync -e "ssh -i ssh_carlesmateo -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" -av --progress --exclude={} --stats --human-readable -z www/ web@myserver.carlesmateo.com:/var/www/myawesomeproject/www/

Note that in this case I’m ignoring Strict Host Key Checking, which is not the preferred option for security, but you may want to use it depending on your strategy and characteristics of your Cloud Deployments.

Note also that I’m indicating as User Known Hosts File /dev/null. That is something you may want to have is you provision using Docker Containers that immediately destroyed after and Jenkins has not created the user properly and it is unable to write to ~home/.ssh/known_hosts

I mention the typical errors where engineers go crazy and spend more time fixing.

Install Jenkins on Docker with Blue Ocean and persisten Voluemes in Ubuntu 20.04 LTS in 4 minutes

Following the official documentation:

https://www.jenkins.io/doc/book/installing/docker/#setup-wizard

The steps are:

Create the network bridge named jenkins

docker network create jenkins

to execute Docker commands inside jenkins nodes we will use docker:dind

docker run \
  --name jenkins-docker \
  --rm \
  --detach \
  --privileged \
  --network jenkins \
  --network-alias docker \
  --env DOCKER_TLS_CERTDIR=/certs \
  --volume jenkins-docker-certs:/certs/client \
  --volume jenkins-data:/var/jenkins_home \
  --publish 2376:2376 \
  docker:dind \
  --storage-driver overlay2

Created a Dockerfile with these contents:

FROM jenkins/jenkins:2.346.1-jdk11
USER root
RUN apt-get update && apt-get install -y lsb-release
RUN curl -fsSLo /usr/share/keyrings/docker-archive-keyring.asc \
  https://download.docker.com/linux/debian/gpg
RUN echo "deb [arch=$(dpkg --print-architecture) \
  signed-by=/usr/share/keyrings/docker-archive-keyring.asc] \
  https://download.docker.com/linux/debian \
  $(lsb_release -cs) stable" > /etc/apt/sources.list.d/docker.list
RUN apt-get update && apt-get install -y docker-ce-cli
USER jenkins
RUN jenkins-plugin-cli --plugins "blueocean:1.25.5 docker-workflow:1.28"

Build it:

docker build -t myjenkins-blueocean:2.346.1-1 .

Run the Container:

docker run \
  --name jenkins-blueocean \
  --restart=on-failure \
  --detach \
  --network jenkins \
  --env DOCKER_HOST=tcp://docker:2376 \
  --env DOCKER_CERT_PATH=/certs/client \
  --env DOCKER_TLS_VERIFY=1 \
  --publish 8080:8080 \
  --publish 50000:50000 \
  --volume jenkins-data:/var/jenkins_home \
  --volume jenkins-docker-certs:/certs/client:ro \
  myjenkins-blueocean:2.346.1-1

See the Id of the running Containers:

docker ps

As in my case my jenkins container Id is 77b6a5a7ae8d in order to know the jenkins administrator password I check the logs for my jenkins Container with docker logs 77b6a5a7ae8d:

docker logs 77b6a5a7ae8d
Running from: /usr/share/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
2022-06-26 21:02:05.492+0000 [id=1]	INFO	org.eclipse.jetty.util.log.Log#initialized: Logging initialized @549ms to org.eclipse.jetty.util.log.JavaUtilLog
2022-06-26 21:02:05.583+0000 [id=1]	INFO	winstone.Logger#logInternal: Beginning extraction from war file
2022-06-26 21:02:05.613+0000 [id=1]	WARNING	o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath
2022-06-26 21:02:05.674+0000 [id=1]	INFO	org.eclipse.jetty.server.Server#doStart: jetty-9.4.45.v20220203; built: 2022-02-03T09:14:34.105Z; git: 4a0c91c0be53805e3fcffdcdcc9587d5301863db; jvm 11.0.15+10
2022-06-26 21:02:05.986+0000 [id=1]	INFO	o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
2022-06-26 21:02:06.020+0000 [id=1]	INFO	o.e.j.s.s.DefaultSessionIdManager#doStart: DefaultSessionIdManager workerName=node0
2022-06-26 21:02:06.020+0000 [id=1]	INFO	o.e.j.s.s.DefaultSessionIdManager#doStart: No SessionScavenger set, using defaults
2022-06-26 21:02:06.021+0000 [id=1]	INFO	o.e.j.server.session.HouseKeeper#startScavenging: node0 Scavenging every 600000ms
2022-06-26 21:02:06.463+0000 [id=1]	INFO	hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/jenkins_home found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
2022-06-26 21:02:06.647+0000 [id=1]	INFO	o.e.j.s.handler.ContextHandler#doStart: Started w.@7cf7aee{Jenkins v2.346.1,/,file:///var/jenkins_home/war/,AVAILABLE}{/var/jenkins_home/war}
2022-06-26 21:02:06.668+0000 [id=1]	INFO	o.e.j.server.AbstractConnector#doStart: Started ServerConnector@4c402120{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2022-06-26 21:02:06.669+0000 [id=1]	INFO	org.eclipse.jetty.server.Server#doStart: Started @1727ms
2022-06-26 21:02:06.669+0000 [id=25]	INFO	winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled
2022-06-26 21:02:06.925+0000 [id=32]	INFO	jenkins.InitReactorRunner$1#onAttained: Started initialization
2022-06-26 21:02:07.214+0000 [id=39]	INFO	jenkins.InitReactorRunner$1#onAttained: Listed all plugins
2022-06-26 21:02:10.781+0000 [id=47]	INFO	jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
2022-06-26 21:02:10.794+0000 [id=35]	INFO	jenkins.InitReactorRunner$1#onAttained: Started all plugins
2022-06-26 21:02:10.803+0000 [id=42]	INFO	jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/var/jenkins_home/war/WEB-INF/lib/groovy-all-2.4.21.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2022-06-26 21:02:11.634+0000 [id=30]	INFO	jenkins.InitReactorRunner$1#onAttained: System config loaded
2022-06-26 21:02:11.635+0000 [id=30]	INFO	jenkins.InitReactorRunner$1#onAttained: System config adapted
2022-06-26 21:02:11.642+0000 [id=48]	INFO	jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
2022-06-26 21:02:11.645+0000 [id=46]	INFO	jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
2022-06-26 21:02:11.668+0000 [id=67]	INFO	hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Download metadata
2022-06-26 21:02:11.675+0000 [id=67]	INFO	hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Download metadata. 4 ms
2022-06-26 21:02:11.733+0000 [id=52]	INFO	jenkins.install.SetupWizard#init: 

*************************************************************
*************************************************************
*************************************************************

Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:

3de0910b83894b9294989552e6fa9773

This may also be found at: /var/jenkins_home/secrets/initialAdminPassword

*************************************************************
*************************************************************
*************************************************************

2022-06-26 21:02:22.901+0000 [id=52]	INFO	jenkins.InitReactorRunner$1#onAttained: Completed initialization
2022-06-26 21:02:23.013+0000 [id=24]	INFO	hudson.lifecycle.Lifecycle#onReady: Jenkins is fully up and running

In my case the password is at the bottom, between the stars: 3de0910b83894b9294989552e6fa9773

Go with your browser to: http://localhost:8080