Monthly Archives: May 2020

A script to backup your partition compressed and a game to learn about dd and pipes

This article is more an exercise, like a game, so you get to know certain things about Linux, and follow my mental process to uncover this. Is nothing mysterious for the Senior Engineers but Junior Sys Admins may enjoy this reading. :)

Ok, so the first thing is I wrote an script in order to completely backup my NVMe hard drive to a gziped file and then I will use this, as a motivation to go deep into investigations to understand.

Ok, so the first script would be like this:

#!/bin/bash
SOURCE_DRIVE="/dev/nvme0n1"
TARGET_PATH="/media/carles/Seagate\ Backup\ Plus\ Drive/BCK/"
TARGET_FILE="nvme.img"
sudo bash -c "dd if=${SOURCE_DRIVE} | gzip > ${TARGET_PATH}${TARGET_FILE}.gz"

So basically, we are going to restart the computer, boot with Linux Live USB Key, mount the Seagate Hard Drive, and run the script.

We are booting with a Live Linux Cd in order to have our partition unmounted and unmodified while we do the backup. This is in order to avoid corruption or data loss as a live Filesystem is getting modifications as we read it.

The problem with this first script is that it will generate a big gzip file.

By big I mean much more bigger than 2GB. Not all physical supports support files bigger than 2GB or 4GB, but even if they do, it’s a pain to transfer this over the Network, or in USB files, so we are going to do a slight modification.

#!/bin/bash
SOURCE_DRIVE="/dev/nvme0n1"
TARGET_PATH="/media/carles/Seagate\ Backup\ Plus\ Drive/BCK/"
TARGET_FILE="nvme.img"
sudo bash -c "dd if=${SOURCE_DRIVE} | gzip | split -b 1024MiB - ${TARGET_PATH}${TARGET_FILE}-split.gz_"

Ok, so we will use pipes and split in order to generate many files as big as 1GB.

If we ls we will get:

-rwxrwxrwx 1 carles carles 1073741824 May 24 14:57 nvme.img-split.gz_aa
-rwxrwxrwx 1 carles carles 1073741824 May 24 14:58 nvme.img-split.gz_ab
-rwxrwxrwx 1 carles carles 1073741824 May 24 14:59 nvme.img-split.gz_ac

Then one may say, Ok, this is working, but how I know the progress?.

For old versions of dd you can use pv which stands for Pipe Viewer and allows you to know the transference between processes using pipes.

For more recent versions of dd you can use status=progress.

So the script updated with status=progress is:

#!/bin/bash
SOURCE_DRIVE="/dev/nvme0n1"
TARGET_PATH="/media/carles/Seagate\ Backup\ Plus\ Drive/BCK/"
TARGET_FILE="nvme.img"
sudo bash -c "dd if=${SOURCE_DRIVE} status=progress | gzip | split -b 1024MiB - ${TARGET_PATH}${TARGET_FILE}-split.gz_"

You can also download the code from:

https://gitlab.com/carles.mateo/blog.carlesmateo.com-source-code/-/blob/master/backup_partition_in_files.sh

Then one may ask himself, wait, if pipes use STDOUT and STDIN and dd is displaying into the screen, then will our gz file get corrupted?.

I like when people question things, and investigate, so let’s answer this question.

If it was a young member of my Team I would ask:

  • Ok, try,it. Check the output file to see if is corrupted.

So they can do zcat or zless to inspect the file, see if it has errors, and to make sure:

gzip -v -t nvme.img.gz
nvme.img.gz:        OK

Ok, so what happened?, because we were seeing output in the screen.

Assuming the young Engineer does not know the answer I would had told:

  • Ok, so you know that if dd would print to STDOUT, then you won’t see it, cause it would be sent to the pipe, so there is something more you’re missing. Let’s check the source code of dd to see what status=progress does

And then look for “progress”.

Soon you’ll find things like everywhere:

  if (progress_time)
    fputc ('\r', stderr);

Ok, pay attention to where is the data written: stderr. So basically the answer is: dd status=progress does not corrupt STDOUT and prints into the screen because it uses STDERR.

Other funny ways to get the progress would be to use:

watch -n10 "ls -alh /BCK/ | grep nvme | wc --lines"
So you would see in real time what was the advance and finally 512GB where compressed to around 336GB in 336 files of 1 GB each (except the last one)

Another funny way would had been sending USR1 signal to the dd process:

Hope you enjoyed this little exercise about the importance of going deep, to the end, to understand what’s going on on the system. :)

Instead of gzip you can use bzip2 or pixz. pixz is very handy if you want to just compress a file, as it uses multiple processors in parallel for the tasks.

xz or lrzip are other compressors. lrzip aims to compress very large files, specially source code.

Some advice for WFH

Those are crazy times in which is difficult to handle working from home, doing the lock down…

But very nice times in which other people help others. Doctors and sanitary personnel fight in first line, truck drivers and supermarket staff are doing extra hours to provide to the society, investigators are working hard to get a vaccine…

Is beautiful that so many people are helping and contributing to the society.

I want to provide my humble experience on working remotely, so you avoid going bananas. Is very easy get depressed, anxious… So here is my advice.

  1. Stick to a routine
    Respect the working times, like if you was going to the office.
    Dress yourself like a normal day in the office. Don’t be all day in pajamas or sport wear. Switch to sports wear when you finish your daily work at 6PM, if you want, but not before.
    If you talk via Slack, myself always keep the video turned on, as is a way to force myself into dressing and taking care of my look.
    I take a shower, dress like an Engineer at work, I have my break for breakfast and for lunch, and when I finish work I study 30 minutes and do exercise 30 minutes or more. Then I take another shower and I consider myself free.
    Note: The only exception I do to the dressing is in the shoes. I don’t wear any shoes, as my feet really enjoy walking freely over the parket.
  2. Take care of yourself
    Shave, cut your nails… be polite. The same way you would if you had to go to the office.
  3. Stick to the working clock
    As said in the stick to a routine advice, work your time, from 9AM to 6PM, and don’t get lost. The week ends are week ends, don’t work to free your mind.
    For the week ends I have my side projects, like writing books.
  4. Walk
    If you can, walk an enjoy nature as much as you can. This clears the mind and keeps your mental health wellness.
  5. Do exercise
    May be walking, but if you have a home bicycle, use it.
    Try to set a goal, like 15 minutes of bicycle daily, and grow from there, of keep it like that. But doing even 15 minutes of cardio every day will be highly beneficial for your mind and body.
  6. Stretch
    After doing exercise stretch your muscles.
  7. See the light
    As much as you can, see the sun light. Try to do walks too and see nature.
    Daylight and nature are amazingly good for your mental health and morale.
  8. Study/Learn new things
    I keep this as part of my routine. I study every day in Linux Academy or read a book at least half an hour. I’ve done this for years.
  9. Keep you hydrated
    Drink lots of water.
  10. Do your breaks.
    After 1 hour try to walk a bit in the house, to focus your view in distant points to relax your ocular muscles.
  11. Ergonomics and light
    Try to have a correct light in the working are, a comfortable chair, the right height for the keyboard and for the monitor, so your neck doesn’t hurt and your hands neither.
  12. Do like in the office: discipline
    In the office you don’t drink, you don’t smoke at your desk.
    So do the same. If you want to smoke one cigar after 2 hours of work, Ok, but don’t loss yourself in self-indulgence. Set strict rules respect alcohol if you love beer:
    No alcohol during working hours.
    Years ago I was CTO of a company with Team in half the world and I had a Team in Belarus. The Team Lead would be drunk in the sofa at business hours and start talking common words. If you have weak points, you don’t want to lose yourself. Be disciplined.
  13. Set boundaries with your family and pets
    If you want to close the door of the room you work, your cat is not gonna die.
    It is used to be alone when you’re are in the office.
    So if it’s excessively demanding and wants you to pet him, or jumps over your laptop’s keyword while you are typing commands as root, set boundaries. Close the door. He can take it.
    Also for the kids, the wife, the mother.
    Please, do not disturb me while I’m working. We will play after.
  14. During covid-19 lockdown, do videoconference
    Do videoconference with your family and friends.
    You can use Slack, Skype, Zoom, Whatsapp, Facebook Messenger…
    I schedule a daily whatsapp video conf with my family, and a weekly with my cool friend Alex :)
  15. Be very patient with your colleagues and Team members
    We are all humans and each one has its own situation.
    Some people may feel depressed, alone, others may have problems with the partners or the parents of her/him living with them.
    Other may have a cat trolling all the time stopping the work.
    Others may have hyperactive children, or just having a poor chair, poor desk, and having only the laptop and no external monitor.
    Others may have the family far, in another country, and suffer for them.
    Be patient and understanding. Be Human.
  16. Have a good Internet connection and use cable over Wifi
    I have 360 Mbit at home with Virgin and I connect the laptops with Ethernet Gigabit cables. That brings me the best and most stable connection.
    In case of emergency I can do tethering with the phone (share the connection using the phone as Wifi Hotspot)
  17. Have spare Hardware and cables
    In this time of lockdown, is good to have spare cables and adapters for everything. Just in case they die.
    If you can have a spare monitor, and spare laptops this is great too.
    I have 3 laptops, plus one tower, plus several raspberry pis, plus the tablet, plus my working laptop. If one dies, I can use the others.
    I always have all kind of Hardware and cables as spare (Gigabit switches, power adapters, international adapters, Ethernet cables, USB cables, headphones…). Even if I can buy in Amazon nothing can stop me.
    One day we had an incident with Virgin, which affected all Ireland.
    I was out as the Fiber was not working and the phone was with Virgin too, but I have two additional SIM cards and spare phones, from vodaphone and Tesco mobile.
    So I’m well protected. :)
    As per the comment of Jordi Soler, I update the list of gadgets, mentioning my Hp Laserjet Color Printer, very handy to print document that I have to sign and then scan and sending back by email, and the UPS.
    The UPS is cool, as if electricity goes down I don’t loss Fiber Internet.
    Imagine, a router connected to the UPS can last hours! however is very infrequent that in Ireland we loss electricity. And the issues I experienced in 3 years were quickly resolved (unlike Barcelona where once more than 300,000 people were 3 days without electricity).
  18. Keep doing backups
    If talking about job things, you can upload to Corporate Google Drive or Microsoft One Drive.
A practical sense of humor

Refreshing settings in a Docker immutable image with Python and Flask

This is a trick to restart a Service that is running on a immutable Docker, with some change, and you need to refresh the values very quickly without having to roll the CI/CD Jenkins Pipeline and uploading a new image.

So why would you need to do that?.

I can think about possible scenarios like:

  • Need to roll out an urgent fix in a time critical manner
  • Jenkins is broken
  • Somebody screw it on the git master branch
  • Docker Hub is down
  • GitHub is down
  • Your artifactory is down
  • The lines between your jumpbox or workstation and the secure Server are down and you have really few bandwidth
  • You have to fix something critical and you only have a phone with you and SSH only
  • Maybe the Dockerfile had latest, and the latest image has changed
FROM os:latest

The ideal is that if you work with immutable images, you roll out a new immutable image and that’s it.

But if for whatever reason you need to update this super fast, this trick may become really handy.

Let’s go for it!.

Normally you’ll start your container with a command similar to this:

docker run -d --rm -p 5000:5000 api_carlesmateo_com:v7 prod 

The first thing we have to do is to stop the container.

So:

docker ps

Locate your container across the list of running containers and stop it, and then restart without the –rm:

docker stop container_name
docker run -d -p 5000:5000 api_carlesmateo_com:v7 prod

the –rm makes the container to cleanup. By default a container’s file system persists even after the container exits. So don’t start it with –rm.

Ok, so login to the container:

docker exec -it container_name /bin/sh 

Edit the config you require to change, for example config.yml

If what you have to update is a password, and is encoded in base64, encode it:

echo -n "ThePassword" | base64
VGhlUGFzc3dvcmQ=

Stop the container. You can do it by stopping the container with docker stop or from inside the container, killing the listening process, probably a Python Flask.

If your Dockerfile ends with something like:

ENTRYPOINT ["./webservice.py"]

And webservice.py has Python Flask code similar to this:

#!/usr/bin/python3
#
# webservice.py
#
# Author: Carles Mateo
# Creation Date: 2020-05-10 20:50 GMT+1
# Description: A simple Flask Web Application
#              Part of the samples of https://leanpub.com/pythoncombatguide
#              More source code for the book at https://gitlab.com/carles.mateo/python_combat_guide
#


from flask import Flask, request
import logging

# Initialize Flask
app = Flask(__name__)


# Sample route so http://127.0.0.1/carles
@app.route('/carles', methods=['GET'])
def carles():
    logging.critical("A connection was established")
    return "200"

logging.info("Initialized...")

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=5000, debug=True)

Then you can kill the process, and so ending the container, from inside the container by doing:

ps -ax | grep webservice
 5750 root     56:31 {webservice.py} /usr/bin/python /opt/webservice/webservice.py
kill -9 5790

This will finish the container the same way as docker stop container_name.

Then start the container (not run)

docker_start container_name

You can now test from outside or from inside the container. If from insise:

/opt/webservice # wget localhost:5000/carles
Connecting to localhost:5000 (127.0.0.1:5000)
carles               100% |**************************************************************************************************************|     3  0:00:00 ETA
/opt/webservice # cat debug.log
2020-05-06 20:46:24,349 Initialized...
2020-05-06 20:46:24,359  * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
2020-05-06 20:46:24,360  * Restarting with stat
2020-05-06 20:46:24,764 Initialized...
2020-05-06 20:46:24,771  * Debugger is active!
2020-05-06 20:46:24,772  * Debugger PIN: 123-456-789
2020-05-07 13:18:43,890 127.0.0.1 - - [07/May/2020 13:18:43] "GET /carles HTTP/1.1" 200 -

if you don’t use YAML files or what you need is to change the code, all this can be avoided as when you update the Python code, Flash realizes that and reloads. See this line in the logs:

2020-05-07 13:18:40,431  * Detected change in '/opt/webservice/wwebservice.py', reloading

Python Combat Guide published

After some work reviewing it and ensuring it has the expected quality, I finally published my book Python Combat Guide.

Is an atypical creation. Is more a Master Class to my best friend, it could be a SDM, TL leading a small Software Development department, a Coder or a Scientist wanting to join IT as programmer and to learn a lot of stuff very quickly, than rather a formal Python Book for learning. Absolutely is not for beginners.

If you want to buy it, to explore the TOC, extended description…

https://leanpub.com/pythoncombatguide

Bash Script: Count repeated lines in the logs

This small script will count repeated patterns in the Logs.

Ideal for checking if there are errors that you’re missing while developing.

#!/usr/bin/env bash
# count_repeated_pattern_in_logs.sh
# By Carles Mateo
# Helps to find repeated lines in Logs
LOGFILE_MESSAGES="/var/log/messages"
LOGFILE_SYSLOG="/var/log/syslog"
if [[ -f "${LOGFILE_MESSAGES}" ]]; then
LOGFILE=${LOGFILE_MESSAGES}
else
LOGFILE=${LOGFILE_SYSLOG}
fi
echo "Using Logfile: ${LOGFILE}"
CMD_OUTPUT=`cat ${LOGFILE} | awk '{ $1=$2=$3=$4=""; print $0 }' | sort | uniq --
count | sort --ignore-case --reverse --numeric-sort`
echo -e "$CMD_OUTPUT"

Basically it takes out the non relevant fields that can prevent from detecting repetition, like the time, and prints the rest.
Then you will launch it like this:

count_repeated_pattern_in_logs.sh | head -n20

If you are checking a machine with Ubuntu UFW (Firewall) and want to skip those likes:

./repeated.sh | grep -v "UFW BLOCK" | head -n20