Bash Script: Count repeated lines in the logs

Last Update: 2022-02-19 15:08 Irish Time

This small script will count repeated patterns in the Logs.

Ideal for checking if there are errors that you’re missing while developing.

#!/usr/bin/env bash
# count_repeated_pattern_in_logs.sh
# By Carles Mateo
# Helps to find repeated lines in Logs
LOGFILE_MESSAGES="/var/log/messages"
LOGFILE_SYSLOG="/var/log/syslog"
if [[ -f "${LOGFILE_MESSAGES}" ]]; then
    LOGFILE=${LOGFILE_MESSAGES}
else
    LOGFILE=${LOGFILE_SYSLOG}
    if [[ ! -f "${LOGFILE_SYSLOG}" ]]; then
        echo "${LOGFILE_MESSAGES} and ${LOGFILE_SYSLOG} do not exist. Exitting"
        exit 1
    fi
fi
echo "Using Logfile: ${LOGFILE}"
CMD_OUTPUT=`cat ${LOGFILE} | awk '{ $1=$2=$3=$4=""; print $0 }' | sort | uniq --count | sort --ignore-case --reverse --numeric-sort`
echo -e "$CMD_OUTPUT"

Basically it takes out the non relevant fields that can prevent from detecting repetition, like the time, and prints the rest.
Then you will launch it like this:

count_repeated_pattern_in_logs.sh | head -n20

If you are checking a machine with Ubuntu UFW (Firewall) and want to skip those likes:

./count_repeated_pattern_in_logs.sh | grep -v "UFW BLOCK" | head -n20

You can also run the same against the output of dmesg -T for counting over the messages in the Kernel this year:

dmesg -T | awk '{ $1=$2=$3=$4=""; print $0 }' | sort | uniq --count | sort --ignore-case --reverse --numeric-sort

And as sample output, the top messages are Ip’s blocked by the Firewall:

     10     2022] [UFW BLOCK] IN=ens4 OUT= MAC=42:01:02:03:04:05:06:07:0a:80:00:01:08:00 SRC=3.217.247.223 DST=10.128.0.2 LEN=40 TOS=0x00 PREC=0x00 TTL=54 ID=0 DF PROTO=TCP SPT=60636 DPT=443 WINDOW=0 RES=0x00 RST URGP=0
      8     2022] [UFW BLOCK] IN=ens4 OUT= MAC=42:01:02:03:04:05:06:07:0a:80:00:01:08:00 SRC=99.41.165.200 DST=10.128.0.2 LEN=40 TOS=0x00 PREC=0x00 TTL=57 ID=0 DF PROTO=TCP SPT=53960 DPT=443 WINDOW=0 RES=0x00 RST URGP=0
      7     2022] [UFW BLOCK] IN=ens4 OUT= MAC=42:01:02:03:04:05:06:07:0a:80:00:01:08:00 SRC=183.82.177.237 DST=10.128.0.2 LEN=40 TOS=0x00 PREC=0x00 TTL=59 ID=0 DF PROTO=TCP SPT=17697 DPT=443 WINDOW=0 RES=0x00 RST URGP=0
      6     2022] [UFW BLOCK] IN=ens4 OUT= MAC=42:01:02:03:04:05:06:07:0a:80:00:01:08:00 SRC=157.90.181.146 DST=10.128.0.2 LEN=40 TOS=0x00 PREC=0x00 TTL=60 ID=0 DF PROTO=TCP SPT=53558 DPT=443 WINDOW=0 RES=0x00 RST URGP=0
      6     2022] [UFW BLOCK] IN=ens4 OUT= MAC=42:01:02:03:04:05:06:07:0a:80:00:01:08:00 SRC=137.226.113.44 DST=10.128.0.2 LEN=40 TOS=0x00 PREC=0x00 TTL=43 ID=0 DF PROTO=TCP SPT=55338 DPT=443 WINDOW=0 RES=0x00 RST URGP=0

Working abroad and the English complexes and insecurity of non natives

I write this article thinking in all my friends that feel insecure about talking in English.

They think about if they are pronouncing correctly, or if they are building the phrases in the correct grammar order. That’s school’s system fault, I think.

As Catalans we learn new languages easily. We talk Catalan native, and Spanish, and in the school we are taught French and English, and if things have not changed, we can choose between Latin and Greek. (I studied both)

But doing 1 or 2 hours per week of English doesn’t grant you a good level of the language, and in fact, few people in Catalonia and Barcelona speak fluent English with a good accent.

I learnt English by myself, by reading programming manuals when I was 5 years old. I also learnt to play chess by watching others playing and when I won the first time I played, against a guy 5 years older than me, he could not believe it was my first match.

I was 10, I think.

When I started classes in the school I realized that I already knew English.

Commands in Basic, like list, run, print, goto, had the same meaning than in the human spoken language.

I grew and I saw that the translations of technical books to Spanish (no Catalan was available) were horrible. They were actually translating commands, so since 15 y.o. I only read manuals in English.

In several jobs, for multinationals, I had to talk with colleagues from different parts of the world, so I was talking Portuguese, some times Italian or French, I could read a bit of German (Was head of Department in Volkswagen IT, gedas), and obviously English.

Still it is not the same when you talk using a subset of the language, basically referred to Hardware and Software, than fully living abroad.

Starting English is easy, you can use present and will for the future and did for the past, and you can make it work. But when you start with the phrasal verbs, the irregular verbs, different time conjugations… English is a context language and it is not a phonetic language, words that are written exactly the same way, sound different, and words that sound the same are written different. So it has a lot of exceptions.

But in this, in the exceptions, and in the fact that is widely expanded, is where we can find the strength to grow without fear.

Catalan is spoken very differently if we are in Barcelona, Lleida, Girona, Tarragona or if we are in València, or Menorca, or Alguer or the country of Andorra.

So the same happens with English. It is not only very different from England to the States, to Australia, to Scotland, to Ireland… also is very different from Dublin and Cork, or from different parts in The States, like Texas and California.

Also there are many people that talks it in Europe, in India… and all of them have different accent!.

So in my experience everybody will understand you. Specially because English is fully understood by the context. Maybe they need you to end the phrase to understand, but they will.

There are also annoying differences that can make you think that your are making mistakes.

Like:

  • Data Center (American) vs Data Center (England)
  • Color (American) vs Colour
  • Humor (American) vs Humour

Don’t be surprised if many native people find your accent exotic, and they love it.

That’s what happened to me many times.

Also I think the school is terrible teaching. They teach children all those rigid grammar expressions, when the live language is much more fluent and free.

For example, one person from Barcelona, will be nervous asking to a colleague:

  • Are you going to the cinema tonight?
  • Have you finally had gone to Disney World?

And he will be nervous thinking in real time if he is building the phrase right.

When, after 2 years will realize that people say:

  • You go to cinema tonight?
  • Did you go to Disney World finally?

The latest are very close to the grammar we use with Catalan, and so hence easy to express fluently.

I can share with you the process I follow to improve my English.

Since the 15 y.o. I was reading all the manuals in English.

I was watching some movies in English, at the beginning with subtitles in Spanish (no Catalan was available) and later with the subtitles in English.

Since 2013, when I was invited by Amazon to Dublin and by Facebook to Menlo Park (US), I started to watch all the movies and sit coms in English.

At the beginning with subtitles in English with the idea to correlate pronunciation and writing. To get my ears used to it.

I went to conferences, and I saw some people, with living years in English speaking countries, that had a much more difficult to understand accent than mine. They were people with reputation. And I understood that IT guys, we are very lucky to be valued by what we know. By our brain.

After all my life reading in English and 4 years watching all the movies in English, my accent had improved, but when I arrived to Cork I had difficulties understanding some the Irish. So I had to get used to the music, the cadence, of the way they talk, and to some words and expressions, and to the humor sense.

I asked my Irish colleagues to correct me when I pronounced wrong, and they were so nice to do it. And they did in a very polite way, for example if I would say:

Is a new Engineer coming to the Team?

And I would put the emphasis on the i of Engineer (the accent), Kevin would repeat the word in the right pronunciation. So I had the chance to learn how it rightly sound.

And I would repeat to make sure I got it.

One thing I think is that one has to be thankful for the time and interest that others dedicate to you. We have all a limited time on the planet, so when somebody invests some time in teaching you or helping you to learn, is giving you something that he will not get back. Even if you pay him/her, still that time will not go back to that person.

So I appreciate when people help me, and I don’t appreciate it less because I pay them.

Talking, listening, is the best real way to learn.

With 100% of reading in English, 100% of movies being watch in English, and nearly 100% of talking and listening in English my language skills reached to the next level. So I can talk in conferences, I can write books and technical documentation. And still I learn a lot of English every day. New words, or rich forms to express the things, reading the newspaper, for example. I really enjoy it.

But is like swimming or going in bicycle: learn by doing.

The Ethernet standards group announces a new 800 GbE specification

Here is the link to the new: https://www.pcgamer.com/amp/the-ethernet-standards-group-developed-a-new-speed-so-fast-it-had-to-change-its-name/

This is a great new for scaling performance in the Data Centers. For routers, switches…

And this makes me think about all the Architects that are using Memcached and Redis in different Servers, in Networks of 1Gbps and makes me want to share with you what a nonsense, is often, that.

So the idea of having Memcache or Redis is just to cache the queries and unload the Database from those queries.

But 1Gbps is equivalent to 125MB (Megabytes) per second.

Local RAM Memory in Servers can perform at 24GB and more (24,000,000 Megabytes) per second, even more.

A PCIE NVMe drive at 3.5GB per second.

A local SSD drive without RAID 550 MB/s.

A SSD in the Cloud, varies a lot on the provider, number of drives, etc… but I’ve seen between 200 MB/s and 2.5GB/s aggregated in RAID.

In fact I have worked with Servers equipped with several IO Controllers, that were delivering 24GB/s of throughput writing or reading to HDD spinning drives.

If you’re in the Cloud. Instead of having 2 Load Balancers, 100 Front Web servers, with a cluster of 5 Redis with huge amount of RAM, and 1 MySQL Master and 1 Slave, all communicating at 1Gbps, probably you’ll get a better performance having the 2 LBs, and 11 Front Web with some more memory and having the Redis instance in the same machine and saving the money of that many small Front and from the 5 huge dedicated Redis.

The same applies if you’re using Docker or K8s.

Even if you just cache the queries to drive, speed will be better than sending everything through 1 Gbps.

This will matter for you if your site is really under heavy load. Most of the sites just query the MySQL Server using 1 Gbps lines, or 2 Gbps in bonding, and that’s enough.

Troubleshooting a shell prompt irresponsible that locks/hangs intermittently

You do df -h or ls / and the terminal freezes and not even CTRL + C works, you have a lock.

Normally this is due to a lock of the system trying to perform an IO.

Could be a physical spinning disk failing, but the most probably nowadays is that you have a network mount point and it is timing out.

If you execute mount and you get a timeout, and when you finally see the list you see a NFS, iSCSI or another kind of Network mount (you will see an Ip Address), check for errors.

To do this in CentOS/RHEL you can do as root:

dmesg | grep -i "timed"

or depending on the System

cat /var/log/messages | grep -i "timed"

You’ll get something like this:

[root@compute01 carles]# dmesg -T | grep timed | head -n5
[Fri Mar 20 02:27:44 2020] nfs: server storage07 not responding, timed out
[Fri Mar 20 02:27:44 2020] nfs: server storage07 not responding, timed out
[Fri Mar 20 02:27:44 2020] nfs: server storage07 not responding, timed out
[Fri Mar 20 02:27:44 2020] nfs: server storage07 not responding, timed out
[Fri Mar 20 02:27:45 2020] nfs: server storage07 not responding, timed out

Please note I use dmesg -T in order to have human readable date instead of Unix Epoch.

You can count the errors today:

[root@compute01 carles]# dmesg -T | grep time | grep "Mon Apr 6" | wc --lines
3123

Lesson 0, learning to code in Python for non programmers

Please note: Even if I tried to make it easy, probably there are too many concepts for a non-programmer. Will try to deliver more basic previous knowledge and foundations, so people with zero knowledge don’t feel overwhelmed.

Start by installing Python 3.8 or 3.9 in your computer, and the IDE PyCharm. Install also Git, and create an account in GitLab so you can share code with other people and understand how Git works.

Here you can read the basic steps for setup PyCharm and GitLab.

Ok, so you can take a look at my video, and hopefully it makes spark your motivation to learn by yourself. :)

I’ve been asked why I used print(“”) instead of print().

Is a good question. The reason is, when we programmed in Python 2.x the native way was to print without parenthesis, like:

print "Hello World!"

Python 3.x was incompatible with that and requires to use parenthesis, like:

print("Hello World!")

Fortunately Python 2.x accepts also to print using parenthesis. In order to have compatibility within Python 2.x and Python 3.x or for future compatibility we were using always print(“Whatever”) in Python2.

However, there is one difference.

If you user print() or print(“”) in Python3 that will generate an empty line.

In Python 2 print(“”) will generate too an empty line, nevertheless print() in Python2 will print two parenthesis. We don’t want that.

This is illustrated in this screenshot:

So all the people that are at home, closed down for coronavirus, you have a chance now to start learning Python and from there get a live as programmer.

You can download the code for this lesson 0, from:

https://gitlab.com/carles.mateo/teach-unit-testing/-/blob/master/lesson0/tree.py

Capturing data from keyboard

In order to be able to do more samples, and then being a bit interesting an dynamic, I will introduce here how to get data inputted by the Keyboard.

print("Please enter your name:")
s_name = input()

This will add whatever we type, without the final Enter, to the String variable s_name.

Capturing numbers from Keyboard

How we do to capture a number, like how old are you, in years?.

The same way, and then we convert this to an Integer value. An Integer is a data type which is basically a number, not decimal. Like: 1, 2, 7, 1000 o -5.

print("Please enter your name:")
s_name = input()

print("Please enter your age:")
s_age = input()
# With int() we convert a String to an Integer, as long as it is possible.
# Wit str() we convert a Integer to a String, as long as it is possible.
i_age = int(s_age)

If you enter a number incorrectly and so that cannot be converted, you will get an Exception Error. That is something that happened in a way that was not expected. These error can be trapped, and we will see this later, in the future.

You know:

  • How to capture data from the keyboard with input()
  • How to convert data entered as String to Integer with int()
  • How to sum two numbers, like 2 + 3
  • How to subtract two numbers, like 2 – 3
  • How to multiply, like 2 * 3

So know, you should be able to solve a basic arithmetic exercise in Hacker Rank:

https://www.hackerrank.com/challenges/python-arithmetic-operators/problem

More sources with explanations

I’m teaching Unit Testing, Refactors, Quality Code and moving from Procedural to OOP to some colleagues, you can find source code for our classes here (please, be aware that there are some error made on purpose to show why and why not do things and hot to apply proper unit testing)

https://gitlab.com/carles.mateo/teach-unit-testing/-/tree/master

More resources

There are many free useful resources to learn Python:

I cancelled my Amazon Prime subscription

I was using a lot Amazon. Sending parcels to my previous job offices, and now to Blizzard offices, so I subscribed to Amazon Prime. With COVID-19 virus we were sent to do Remote Work, and now with the lock down basically I’m 99.99% of the time at home.

I did a test to see how it works sending to home during the pandemic.

I choose two different items, I reviews the order, they were going to be delivered separately, one day of distance.

I choose two items that will fit in my mailbox, separated or together. One USB3 3mts male female and a Blu-ray movie.

My surprise comes when I go to the mailbox one day before and I see that I have a paper from an-post telling that they pass by to deliver my parcel, and they did not leave because it doesn’t fit the mailbox and they did not want to leave it a common space. For my surprise both Amazon parcels were grouped and sent before time. Maybe in a bigger box. But the mailman did not ring my door.

The paper tells me to get my parcel in the middle of the city, during the lock down. No way! I’m not going to risk my health and specially from elders, just to grab a cable and a movie.

I had the chance to request re-delivery to an Post, so I do. I fill all the info, I inform my phone number, email, I indicate which door to ring, and two days after as promised… a paper from an Post!.

They did not even rang my bell again.

I go to Amazon to cancel the order, but the process is only created for if you got the items.

Fuck it. I’m not going to order anything else to Amazon until that COVID-19 passes.

I don’t know if the postman just avoids people for fear to contagion or the process of an Post is awful and he didn’t get any information. But I’ll not buy anything even if I cannot buy in other places cause the lock down.

I was going to maintain my Amazon Prime subscription, even if I know that I’ll not use it much with the lock down, but makes no sense. Also:

  • I use Netflix and my Raspberry Pi 4, I was not using Amazon Prime Video.
  • I use Spotify, I was not using Amazon Prime Music.
  • I like to read in paper, not in eBook, so I was not using the eReader options.

A nice way to loss a customer.

Datacenters, D&R and coronavirus

I’ve been working for years within Data centers, with D&R strategies, and then in the middle of COVID-19, with huge demands on increments of bandwidth and compute, some DCs decided to do not allow in the Engineers of their customers.

As somebody that had my own Startup and CSP and had infrastructure in DCs and servers from customers in colocation, and has replaced Hw components at 1AM, replaced drives from broken RAIDs, and fixed systems so many times inside so many Datacenters across the world, I’m shocked about that.

I understand health reasons can be argued, but I still have Servers in Datacenters because we all believed they were the most safe place, prepared for disaster and recovery, with security, 24×7… and now, one realise that cannot enter to fix or upgrade the own machines.
Please note, still you can use the remote hands from the DC, although this is not a good idea many times, I’m not sure this will still be an available option when the lock down in those countries becomes more strict.

I’m wondering if DCs current model have any future at all.

I think most of the D&R strategies from now will be in the cloud, in different regions, with different providers, so companies can resist providers or governments letting them down.

Quick Access to my selection Last Update: 2022-10-24 09:36:42 Ireland Time / 2022-10-24 01:36:42 PDT-0700 Unix epoch: 1666600602
Cloud Computing Operations Engineering/DevOps/SRE Engineering My Books
Provisioning AWS EC2 Instances with Ansible and Automating Apache deployment with or without using Ansible Dynamic Inventory from Ubuntu 20.04 LTS
Using Ansible in Ubuntu 20.04 LTS to provision to Amazon, and use the Dynamic Inventory (or my own Python 3 code, and alternative Bash code) to create different inventories per group, so you can provision Apache2 in your desired group of instances only.
Published: 2021-Dec Views: 10,992 views
Migrating my 11 years Amazon AWS account services (Postmortem Analysis)
Why I migrated my Services out of Amazon AWS, how I did, the problems I had and the mistakes I did.
Published: 2021-Nov Views: 10,419 views
Upgrading Amazon AWS EC2 Ubuntu 18.04 LTS to Ubuntu 20.04 LTS
Upgrading Amazon EC2 Instance with LAMP from Ubuntu Linux 18.04 LTS to version 20.04 LTS.
Published: 2021-June Views: 7,737 views
How to recover access to your Amazon AWS EC2 instance if you loss your Private Key for SSH
Procedure to get access to an Amazon AWS EC2 Instance and its Data, after you have lost your SSH Private Key and access to the instances using that Key pair.
Published: 2020-Sept Views: 13,699 views
Benchmarking Google Cloud Compute Engine (2015)
The analysis I did from Google Compute Engine, in my project CMIPS, when google launched their GCE in beta.
Paul Nash wrote me and I helped all the Cloud Team to fix some problems. :)
This analysis is a bit old however contains some interesting information.
Published: 2015-Jan
Comparison of Cloud Provider’s Instances performance
From my CMIPS (Cloud Million Instructions per second) a performance comparison of different Instances, from different Cloud Providers, compared to bare metal as well.
Is old, from 2015, however brings very valuable information.
Published: 2015-Jan
The Cloud is for Scaling
The Cloud is for Startups, and for Scaling and for Enterprises. Nothing more. Published: 2013-Sept Views: 19,843 views 		Creating a RabbitMQ Docker Container accessed with Python and pika
Published: 2022-July Views: 11,967 views
How to deploy a DigitalOcean droplet (instance) and use userdata
Published: 2022-June Views: 8,940 views
Renewing a SSL Certificate for Apache2 in Ubuntu 20.04
Published: 2022-Mar Views: 13,115 views
Linux Command Line tools I usually install (if they are not on the system)
Published: 2016-March Views: 23,505 views
Troubleshooting upgrading and loading a ZFS module in RHEL7.4
Published: 2018-July Views: 23,507 views
A sample forensic post mortem for a iSCSI Initiator (client) that had connectivity problems to the Server (Troubleshooting)
Published: 2019-August Views: 14,847 views
Creating a content filter for Postfix in PHP
Published: 2016-June Views: 36,545 views
Stopping definitively the massive Distributed DoS attack
Published: 2015-Feb Views: 50,823 views
Stopping and investigating a DoS XMLRPC attack
Published: 2014-August Views: 165,871 views
Stopping a BitTorrent DDoS attack
Published: 2015-January Views: 107,400 views
Dropping caches in Linux, to check if memory is actually being used
Published: 2019-April Views: 10,423 views
Troubleshooting a shell prompt irresponsible/that locks intermittently
Published: 2020-April Views: 11,696 views
Post-Mortem: The mystery of the duplicated Transactions into an e-Commerce
Published: 2020-Nov Views: 12,058 views
Adding a swapfile on the fly as a temporary solution for a Server with few memory
Published: 2020-Nov Views: 9,115 views
Swap, swappiness, Servers not responding
Published: 2021-May Views: 10,627 views
 Erasure Code
My project for infinite Storage scaling with no single point of failure, based on Erasure Codes.

Published: 2022-May Views: 11,370 views
Troubleshooting apps in Linux
Published: 2013-November Views: 10,687 views
Performance of Several Languages
Published: 2014-Oct Views: 211,427 views
CSort Multithread versus Quicksort (Java)
Published: 2017-March Views: 31,014 views
CSort my algorithm that heavily beats Quicksort (Java)
Published: 2015-May Views: 24,391 views
Buy my books:
Python 3 Combat Guide
Pages: 403 DIN-A4 PDF DRM-free
Accompanying Source Code: https://gitlab.com/carles.mateo/python_combat_guide
Last Update: v.1.08 2022-05-11
Automating and Provisioning to Amazon Web Services (AWS) with boto3 SDK for Python (plus some Ansible)
Pages: 128 Full Size DIN-A4 PDF DRM-free
Last Update: v.16 2022-01-16
Docker Combat Guide
Learn Docker, focused on Developers, and Docker’s Python 3 SDK
Pages: 178 DIN-A4 PDF DRM-free
Last Update: v.25 2022-07-03
Assemble and upgrade your PC and laptops
Pages: 107 DIN-A4 PDF DRM-free
Last Update: v.0.17 2021-01-11
ZFS on Ubuntu 20.04 LTS
Pages: 159 DIN-A4 PDF DRM-free
Covers ZFS 0.8.3 version. Shows tricks, fix errors, secrets and commands against real systems with LSI Controllers, SAS SLOG ZIL devices, SATA, SSD…
Last Update: v.0.25 2022-02-07
Python 3 Exercises for Beginners
Pages: 200 DIN-A4 PDF DRM-free
Exercises for people starting coding in Python, explaining the solution, tricks, etc…
Last Update: v.48 2022-03-15
Python Open Source Utilities Python PHP Java
CTOP.py SysAdmin tool to get all the System Information at a glance
Published: 2020-Jan Views: 20,606 views
Current Version: 0.8.8 Last Update: 2022-02-13
Simple sample to print colors in Terminal
Published: 2018-May Views: 19,213 views
LDAPGUI.py a simple Python GUI application that queries LDAP
Published: 2020-Jun Views: 14,579 views
cmemgzip.py compress logs (and any file) in memory and replace uncompressed files by .gz when drive has no space left. Supports compressing by blogs to use less memory
Published: 2021-Feb Views: 11,328 views
checkswap.py Monitor the impact of swap memory pages on a live system. Compatible with Python 2.x and 3.x
Published: 2021-May Views: 10,627 views
Carleslibs v. 1.0.8 (2022-06-05) Python Open Source package.
Published: 2021-July (Updated 2022-February)Views: 12,058 views
Cliptype is a utility to paste the Clipboard into the focussed Windows. Ideal for working with Web SSH Terminal clients/QEMU/KVM that don’t support paste
Published: 2021-NovemberViews: 42,415 views
Video: Object Oriented Programming in Python 3 for beginners
Published: 2022-July Views: 7,072 views
For beginners: How to start coding Python with PyCharm and Git
Published: 2022-March Views: 9,124 views
Lesson 0, learning to code in Python for non programmers
Published: 2020-March Views: 14,654 views
Video for beginners: Python for, range, lists, dicts
Published: 2021 July Views: 8,940 views
Programming class for beginners on 2021-11-11.
Published: 2021-November Views: 8,650 views
Sorting an Array of Tuples with Lambda in Python (videos)
Published: 2022-May Views: 7,291 views
Learn to do Unit Testing with pytest in Python 3.
Published: 2021-October Views: 9,036 views
Video: Parse the Tables from a Website with Python pandas
Published: 2022-July Views: 7,400 views
A handy trick command line to get the usages of our Python Methods in the code
Published: 2019-July Views: 11,682 views
A small Python + MySql + Docker program as a sample (plus LAMP PHP sample).
Published: 2021-July Views: 20,359 views
A simple Flask Application, a Star Wars game in Python and Docker.
Published: 2021-July Views: 9,119 views
A simple Python Tic Tac Toe game.
Published: 2021-September Views: 9,214 views
Some weird things from Python 3 that you may not know.
Published: 2021-September Views: 10,121 views
Generating a Word Cloud of Tags in Python.
Published: 2021-September Views: 8,766 views
Some graphics with matplotlib.
Published: 2021-October Views: 8,112 views
Web Top – Displaying top with Python 3 Web Server and carleslibs.
Published: 2021-October Views: 9,684 views
Why I think in Python is not a good idea to raise exceptions inside your methods
Published: 2022-May Views: 7,733 views
A Sudoku Solver in Python, an engineering approach solution (with Source Code)
Published: 2022-April Views: 9,567 views
MT Notation prefix variables system for Python
Last Update: 2021-07-15 Views: 15,606 views 		My PHP Script to see WordPress Posts and Views ordered by Views
Published: 2021-August Views: 12,285 views
Improving performance in PHP
Published: 2014-August Views: 42,567 views
Catalonia Framework PHP Open Source
MT Notation prefix variables system for PHP
From: 2014-July Last Update: 2021-07-15 Views: 14,817 views 		Java validation Classes for Keyboard
Published: 2020-Dec Views: 35,833 views
MT Notation prefix variables system for Java
From: 2017-March Views: 13,441 views
Docker Commodity Hardware ZFS Architecture
Communicating with Docker Containers via Linux Signals and Python
Published: 2021-Nov Views: 10,188 views
Refreshing settings in a Docker immutable image with Python and Flask
Published: 2020-May Views: 12,032 views
In March 2021, Why I propose you to use Python 3.8, at least, for your Internal Automation Tools in Docker Containers and Ubuntu 20.04 LTS Published: 2021-March Views: 9,867 views
Solving the problem when running a Docker Container: standard_init_linux.go:190: exec user process caused “no such file or directory” Published: 2021-March Views: 14,351 views
Adding my Server as Docker, with PHP Catalonia Framework, explained
Published: 2019-July Views: 14,483 views
A base Dockerfile for my Jenkins (home) deployments
Published: 2021-March Views: 10,720 views
Install jenkins on Docker Ubuntu 20.04 LTS with Blue Ocean pipeline plugin and persistent Volumes in 4 minutes
Published: 2022-June Views: 10,189 views
Migrating some Services from Amazon AWS EC2 to Digital Ocean, using Docker
Published: 2021-Aug Views: 9,207 views
Have a cheap Ubuntu in your Windows or Mac with Docker
Published: 2021-September Views: 9,026 views
Video: How to create a Docker Container for Linux Apache MySQL PHP Python (LAMPP) step by step
Published: 2022-July Views: 7,069 views 		Dealing with Performance degradation on ZFS (DRAID) Rebuilds when migrating from a single processor to a multiprocessor platform (Troubleshooting explained)
Published: 2019-June Views: 13,831 views
Solving a persistent MDRAID and ZFS problem in RHEL7.4 (Dual Port SAS drives)
Published: 2018-Oct Views: 12,816 views
Simulating a SAS physical pull out of a drive
Published: 2019-March Views: 12,198 views
Create a small partition on the drives for tests
Published: 2019-April Views: 12,374 views 		zpool_watch is an Open Source Python 3 utility that watches your ZFS Pools and open a window in your Linux if there is a problem
Published: 2022-February Views: 7,765 views
Adding a RAMDISK as SLOG ZIL to ZFS
Published: 2020-August Views: 22,981 views
Some handy tricks for working with ZFS
Published: 2019-June Views: 14,045 views
ZFS Improving iSCSI for Block Devices (trick for Volumes)
Published: 2018-Oct Views: 33,160 views
Creating a compressed filesystem with Linux and ZFS
Published: 2018-Sept Views: 16,945 views
Compiling ZFS with RHEL6.10
Published: 2019-August Views: 11,885 views
My talk at OpenZFS 2018 about DRAID (San Francisco, US) Published: 2018-Sept Views: 14,634 views
Extend existing Single ZFS disk with a mirror without losing the Data on the existing drive
Published: 2021-Jan Views: 14,245 views 		Resources for Microservices and Business Domain Solutions for the Cloud Architect / Microservices Architect
Published: 2019-Oct Views: 12,537 views
Scaling PhantomJS with PHP
Published: 2015-June Views: 31,589 views
Improving Performance in PHP (Stack and tricks)
Published: 2014-August Views: 42,567 views
Begin developing in Cassandra in Java, PHP or Python
Published: 2014-July Views: 20,094 views
Windows Raspberry Pi Cassandra Relational Databases
Using Docker in Windows 10 without Windows Desktop with Docker Engine and without WSL
Published: 2021-Feb Views: 18,337 views
Solving Windows 10 PRO running Active Directory as Domain Admin
Published: 2021-Jan Views: 8,961 views
Install Windows Subsystem for Linux, WSL 2 on Windows 10 64 bit, with Ubuntu, solution to error WslRegisterDistribution failed with error: 0x80070057
Published: 2021-Jan Views: 43,678 views
Programs I use for Windows in my Workstations
Published: 2019-Sept Views: 10,456 views
Using Windows 10 Appliance in Ubuntu Virtual Box 4.3.10 and later versions
Published: 2015-August Views: 44,506 views
Reinstall PIP only in Windows 10 after it got removed
Published: 2021-March Views: 11,199 views 		Install a Media Player on the Raspberry Pi 4
Published: 2020-March Views: 10,532 views
Raspberry Pi 3 and OMSC Media Player
Published: 2015-April Views: 21,878 views
Solving the problem GPIO.setup(self.number, GPIO.IN, self.GPIO_PULL_UPS[self._pull]) RuntimeError: Not running on a RPi! in Ubuntu 20.04LTS
Published: 2021-Feb Views: 14,371 views 		CQLSÍ a wrapper to use Cassandra from PHP
Written in 2014, a time when there were no drivers for PHP.
Published: 2014 Views: 8,727 views
Cassandra Universal Driver
A HTTP gateway for all the languages supporting curl/sockets. Written in 2014, a time when there were no drivers for many languages.
Published: 2014 Views: 9,056 views 		Solving Oracle error ORA 600 [KGL-heap-size-exceeded]
Published: 2021-Febreruary Views: 31,887 views
Bash Open Source Utilities Bash My Tech Talks Miscellaneous
count_repeated_pattern_in_logs.sh
A easy way to see errors that are repeating, e.g.: NFS/iSCSI timeouts.
Published: 2020-May Views: 12,746 views
backup_partition_in_files.sh
Compressing an unmounted partition to a image file while compressing on the fly, and breaking into 1GB gz files.
Also explains in a funny way about STDIN, STDOUT, STDERR and methodology investigating in deep.
Published: 2020-May Views: 13,702 views
iostat_bandwidth.sh
See the aggregated bandwidth used by all the drives, and the maximum speed achieved.
Published: 2020-Aug Views: 11,640 views
count_lines_of_code.sh
Count the lines in .py Python source files recursively and displays individual and total results.
Published: 2021-Jan Views: 12,463 views
One line script to log the temperature of HDDs and CPUs in Ubuntu
Published: 2021-Jan Views: 10,761 views 		compress_old.sh A simple Bash script to compress files in a directory, older than n days
You can use it to delete older files, or perform other commands.
Published: 2021-March Views: 10,948 views
A simple script to upload a pypi/pip package
Is what I use to maintain packages like cmemgzip, carleslibs, etc…
Published: 2021-March Views: 9,126 views
Backup and Restore your Ubuntu Linux Workstations – with support for Wine and Docker
Published: 2022-October Views: 7,513 views 		My talk at OpenZFS 2018 about DRAID (San Francisco, US)

Published: 2018-Sept Views: 14,634 views
Google Compute Engine Talk for Group Google Developers Cork, Ireland

How to do autoscaling from 0 using Google Cloud.
Published: 2019-Feb Views: 12,193 views 		How is: Working in Cork for IT Engineers
Published: 2020-Feb Views: 13,163 views
A mistake that all the universities are doing
Published: 2018-Jun Views: 15,161 views
Some advice for WFH (Working from Home/Remotely)
Published: 2020-May Views: 17,039 views
My radio program at RAB 2022-06-27 [Catalan and English]
Published: 2022-June Views: 2,407 views
News from the Blog 2022-06-22
Published: 2022-June Views: 6,928 views
News from the Blog 2022-05-22
Published: 2022-May Views: 6,578 views
News from the Blog 2022-04-22
Published: 2022-April Views: 7,189 views
News from the Blog 2022-03-22
Published: 2022-March Views: 7,656 views
News from the Blog 2022-02-22
Published: 2022-February Views: 7,929 views
Old News:

Media Player in my Raspberry Pi 4

Just installed a media player in my Raspberry Pi 4

So I mentioned it was one of my pending tasks, to do while I’m confined here, at home, to help the Irish government to stop the quick spread of the coronavirus.

I’m happy that the situation in Ireland has stabilized, unlikely in Spain, where that historical lack of discipline and selfishness and super ego to believe Madrid the capital of the world, and so deciding not to close it for quarantine, will cause a lot of pain. I hope the closing of frontiers in Catalonia works.

Well, what I do you’re probably asking yourself, so I installed LibreELEC https://libreelec.tv/.

They have a very nice SD image writer for Linux, Mac and Windows, that will install the proper image on the micro-SD for your ARM device.

This Raspberry Pi 4 comes with Wifi integrated and a Gigabit Ethernet network port.

When I was in Barcelona, I had Kodi with Raspberry pi 2 and version 3.

This model v. 4 is much more cooler. I bought the 4GB version, and has 2xHDMI 4K.

So it is great to connect to any modern TV.

In Barcelona, I have Linux tower as NFS Server sharing my files with the Pi. Work good, even for the 100Mbit NIC of the version 3, but at that time I was only playing Full HD as the Pi didn’t supported greater resolution, and I only had that resolution on my displays too.

For now, I’m going to explore how is reading from a USB 3.0. Let’s see if it’s able to play smoothly.

The cool thing also is that I have SSH access, and so I can use the Pi for many more things. :)

I have my first update, I noticed that copying to that USB was not the best for me, as I tried to copy a .MKV file of 4.9GB and I encountered the limit of 4GB of FAT32. I could format the USB as ext4, but what I did is, SSH into the box, I see that I have two partitions on the SD for booting the Pi, the second one is a ext4 called storage. So I copied to the SD, through the network, using sftp the file I wanted.

The Gigabit connection was fast, but when the buffer fulled it started to show the real speed of the SD which is 15MB/s for writing.

Ext4 has no problem in holding a file 4.9GB so I’m watching my movie now. Will think about setting a NFS for the Pi as it will be very convenient. :)

I have an external, remote, keyboard logitech, but it happens that LibreELEC recognizes my Sony command, from the television. I don’t need the keyboard/mouse. Nice.

Here you can see my Raspberry Pi 4, connected to TV, in “combat mode”, naked, as PoC, before setting in its definitive place behind the TV.

Playing from the external USB 3.0 stick was also fluid, allowing 4K perfectly.

The only problem I has was when I was pushing movies to the USB through the network, and playing at the same time from the SD. It seems like the Raspberry reached its limits doing this and playing stuck frequently.

Remote working is here

So remote working is here.

After years in which many Engineers requested to the companies to be able to Remote Work, with most of answers No, now it happens that not only is good for the company, is the only way to ensure continuity of business, of many businesses.

One of my colleagues from Denmark, which government has shutdown the country by sending all the public servants to home, in order to prevent the spread of the coronavirus, told me:

“Yes, remote working is here, but has been necessary the four horsemen of the apocalypse”

It is curious, how Remote Working has arrived, no thanks to that was obvious, but due to external emergencies. And I’m glad that my company was prepared for business continuity.

I’ll be staying home, working remotely, in order to contribute to non-spreading the virus, specially among old people. I’m perfectly healthy but that’s a use case, many people will not develop the symptoms and still be able to spread to others.

So I have some plans related to technology to do at home, including few improvements to the blog. What are your plans?.

Update: 2020-03-13 23:16 UTC I’m thinking in all those business which are forces to close, and all the employees that will not get a salary, or will be fired, or will get a salary and the business owner maybe ends in bankrupt as is paying the salaries and no income is being generated.

Update: 2020-03-19 10:58 UTC Some of my friends, even in Human Resources/Recruiting, are starting to remote work for first time. So here is some advice:

I would recommend to get an external monitor, at least 22″, so you neck is not forcing position looking low and your eyes don’t suffer, good light (don’t in dark), a nespresso can be a good friend in the morning, and to have your hands and arms aligned correctly so you don’t suffer from a bad position. Watch the position of the wrists, your arms should be comfortably at the same level than the table, similar in an L, and your eyes be aligned to the top of your monitor. Finally I would recommend to follow a routine, like if you were going to work, so dress like you would do. Don’t stay at home all day in pijamas! ;)