More funny things happened like when I was installing a VirtualBox VM live, and the ZFS pool became irresponsible due hardware errors in one SATA Spinning drive.
Things from broadcasting live…
Some of the feedback I got from talented Engineers is that even if the original matter to talk about was interesting, seeing everything falling apart live due to unexpected hardware problems, and me troubleshooting live is being the best of the show… which I found very amusing.
RAB Radio the new digital world
I keep doing my radio space for Radio America Barcelona, once per week, addressed to the Catalan Community across the world and expats.
This radio program, streamed also via Twitch, is available in Catalan language only. RAB.
Open Source
carleslibs
I’ve been working in version 1.0.8 branch, and after a session of refactor on Twitch where I found a bug in MenuUtils class, I fixed it and released v. 1.0.8. You can see the video on the link.
Now I’m working on the branch v. 1.0.9.
ctop
I’ve been working in the branch 0.8.9.
My first Twitch broadcast was about adding Unit Testing to MemUtils class.
This week I decommissioned my last physical server in a Data Center.
It has been a long journey since I created my company to launch my own projects, and I started having my own infrastructure, back at 2000.
I was offering VPS at that time, with VMWare as Hypervisor.
This last Rack Server served me well for 21 years.
Now everything is Cloud, and is not viable to host and maintain servers unless this is your main occupation. Server’s motherboards die, hard drives die and they need to be replaced. Maintaining infrastructure it’s a full time job and you require somebody to do it. Also using fixed servers only prevents you from moving fast, locks a lot of money, and from spawning more compute capacity.
If you are curious this Rack Server is a Super Micro with Intel Xeon processor and SCSI drives.
Security
Firewall
I keep blocking thousands of IP Addresses every day.
When I see a pattern of an IP trying an attacks against the Server I look at the IP and if it’s from a hosting provider I just block the entire range.
I keep blocking any IP Address coming from Russia or Belarus since they invaded Ukraine.
My Health
I visited the hospital for a programmed following on my health.
The analysis are super good, and it’s super clear that I’ve improved radically. My discipline with the diet, taking the medicines and doing exercise regularly has been crucial.
My Doctor is confident that I’ll have a full recovery, but to do so I need to loss a lot of weight in a year or two.
So, I need to focus on my health and in doing exercise, being happy and avoid any kind of negative stress.
The cost of the travels and the medicines have put some stress into my economy, but I’m fortunate that I can handle it.
Entertainment / Life / Reflections
Star Wars and racism
I’m really enjoying new Start Wars series Obi Wan, and I’ve been profoundly shocked to read that there are fans being racist against the black characters.
Recently a colleague was asking me for advice on their design of error handling in a Python application.
They were catching an error and raising an Exception, inside the except part of a method, to be catch outside the method.
And at some point a simple logic got really messy and unnecessarily complicated. Also troubleshooting and debugging an error was painful because they were only getting a Custom Exception and not context.
I explained to my colleague that I believed that the person that created that Exception chain of catch came from Java background and why I think they choose that path, and why I think in Python it’s a bad idea.
In Java, functions and methods can only return one object.
I programmed a lot in Java in my career, and it was a pain having to create value objects, and having to create all kind of objects for the return. Is it a good thing that types are strongly verified by the language? Yes. It worked? Yes. It made me invest much more time than necessary? Also yes.
Having the possibility to return only one object makes it mandatory having a way to return when there was an error. Otherwise you would need to encapsulate an error code and error description fields in each object, which is contrary to the nature of the object.
For example, a class Persona. Doesn’t make any sense having an attribute inside the class Persona to register if an operation related to this object went wrong.
For example, if we are in a class Spaceship that has a method GetPersonaInCommand() and there is a problem in that method, doesn’t make any sense to return an empty Persona object with attributes idError, errorDescription. Probably the Constructor or Persona will require at least a name or Id to build the object…. so in this case, makes sense that the method raises an Exception so the calling code catches it and knows that something went wrong or when there is no data to return.
This will force to write Custom Exceptions, but it’s a solution.
Another solution is creating a generic response object which could be an Object with these attributes:
idError
errorDescription
an Object which is the response, in our example Persona or null
I created this kind of approach for my Cassandra libraries to easily work with Cassandra from Java and from PHP, and for Cassandra Universal Driver (a http/s gateway created in year 2014).
Why this in not necessary in Python
Python allows you to return multiple values, so I encourage you tor return a boolean for indicating the success of the operation, and the object/value you’re interested.
You can see it easily if you take a look to FileUtils class from my OpenSource libraries carleslibs.
The method get_file_size_in_bytes(self, s_file) for example:
def get_file_size_in_bytes(self, s_file):
b_success = False
i_file_size = 0
try:
# This will help with Unit Testing by raisin IOError Exception
self.test_helper()
i_file_size = os.path.getsize(s_file)
b_success = True
except IOError:
b_success = False
return b_success, i_file_size
It will always return a boolean value to indicate success or failure of the operation and an integer for the size of the file.
The calling code will do something like this:
o_fileutils = FileUtils()
b_success, i_bytes = o_fileutils.get_file_size_in_bytes("profile.png")
if b_succes is False:
print("Error! The file does not exist or cannot be accessed!")
exit(1)
if i_bytes < 1024:
print("The profile picture should be at least 1KB")
exit(1)
print("Profile picture exists and is", i_bytes, " bytes in length!")
The fact that Python can return multiple variables makes super easy dealing with error handling without having to take the road of Custom Exceptions.
And it is Ok if you want to follow this path, but in my opinion, for most of the developers up to Senior levels, it only over complicates the logic of your code and the amount of try/excepts you have to have everywhere.
If you use PHP you can mix different types in an Array, so you can always return an Array with a boolean, or an i_id_error, and your object or data of whatever type it’s.
Getting back to my carleslibs Open Source package, it is super easy to Unit Test these methods.
In my opinion, this level of simplicity, brings only advantages. Including Software Development speed, which is good for the business.
I’m not advocating for not using Custom Exceptions or to not develop a Exceptions Raising strategy if you need it and you know what you’re doing. I’m just suggesting why I think most of the developments in Python do not really need this and only over complicates the development. There are situations where raising exceptions will be a perfectly useful or even the best approach, there are many scenarios, but I think that in most of cases, using raise inside except will only multiply the time of the development and slow down the speed of bringing new features to the business, over complicating Unit Test as well, and be a real pain for the Junior and Intermediate developers.
The Constructor
Obviously, as the Constructor doesn’t return any value, it is perfectly fine to raise an exception in there, or just to use try/except in the code that is instancing the objects.
So, I was working on a project and i wanted to test how long a file can be.
The resources I checked, and also the Kernel and C source I reviewed, were pointing that effectively the limit is 255 characters.
But I had one doubt… given that Python 3 String are UTF-8, and that Linux encode by default is UTF-8, will I be able to use 255 characters length, or this will be reduced as the Strings will be encoded as UTF-8 or Unicode?.
So I did a small Proof of Concept.
For the filenames I grabbed a fragment of the translation to English of the book epic book “Tirant lo Blanc” (The White Knight), one of the first books written in Catalan language and the first chivalry novel known in the world. You can download it for free it in:
from carleslibs.fileutils import FileUtils
from colorama import Fore, Back, Style , init
class ColorUtils:
def __init__(self):
# For Colorama on Windows
init()
def print_error(self, s_text, s_end="\n"):
"""
Prints errors in Red.
:param s_text:
:return:
"""
print(Fore.RED + s_text)
print(Style.RESET_ALL, end=s_end)
def print_success(self, s_text, s_end="\n"):
"""
Prints errors in Green.
:param s_text:
:return:
"""
print(Fore.GREEN + s_text)
print(Style.RESET_ALL, end=s_end)
def print_label(self, s_text, s_end="\n"):
"""
Prints a label and not the end line
:param s_text:
:return:
"""
print(Fore.BLUE + s_text, end="")
print(Style.RESET_ALL, end=s_end)
def return_text_blue(self, s_text):
"""
Restuns a Text
:param s_text:
:return: String
"""
s_text_return = Fore.BLUE + s_text + Style.RESET_ALL
return s_text_return
if __name__ == "__main__":
o_color = ColorUtils()
o_file = FileUtils()
s_text = "In the fertile, rich and lovely island of England there lived a most valiant knight, noble by his lineage and much more for his "
s_text += "courage. In his great wisdom and ingenuity he had served the profession of chivalry for many years and with a great deal of honor, "
s_text += "and his fame was widely known throughout the world. His name was Count William of Warwick. This was a very strong knight "
s_text += "who, in his virile youth, had practiced the use of arms, following wars on sea as well as land, and he had brought many "
s_text += "battles to a successful conclusion."
o_color.print_label("Erasure Code project by Carles Mateo")
print()
print("Task 237 - Proof of Concep of Long File Names, encoded is ASCii, in Linux ext3 and ext4 Filesystems")
print()
print("Using as a sample a text based on the translation of Tirant Lo Blanc")
print("Sample text used:")
print("-"*30)
print(s_text)
print("-" * 30)
print()
print("This test uses the OpenSource libraries from Carles Mateo carleslibs")
print()
print("Initiating tests")
print("================")
s_dir = "task237_tests"
if o_file.folder_exists(s_dir) is True:
o_color.print_success("Directory " + s_dir + " already existed, skipping creation")
else:
b_success = o_file.create_folder(s_dir)
if b_success is True:
o_color.print_success("Directory " + s_dir + " created successfully")
else:
o_color.print_error("Directory " + s_dir + " creation failed")
exit(1)
for i_length in range(200, 512, 1):
s_filename = s_dir + "/" + s_text[0:i_length]
b_success = o_file.write(s_file=s_filename, s_text=s_text)
s_output = "Writing file length: "
print(s_output, end="")
o_color.print_label(str(i_length).rjust(3), s_end="")
print(" file name: ", end="")
o_color.print_label(s_filename, s_end="")
print(": ", end="")
if b_success is False:
o_color.print_error("failed", s_end="\n")
exit(0)
else:
o_color.print_success("success", s_end="\n")
# Note: up to 255 work, 256 fails
I tried this in an Ubuntu Virtual Box VM.
As part of my tests I tried to add a non typical character in English, ASCii >127, like Ç.
When I use ASCii character < 128 (0 to 127) for the filenames in ext3/ext4 and in a ZFS Pool, I can use 255 positions. But when I add characters that are not typical, like the Catalan ç Ç or accents Àí the space available is reduced, suggesting the characters are being encoded:
Ç has value 128 in the ASCii table and ç 135.
I used my Open Source carleslibs and the package colorama.
zpool watch is a small Python program for Linux workstations with graphical environment and ZFS, that checks every 30 seconds if your OpenZFS pools are Ok.
If a pool is not healthy, it displays a message in a window using tk inter.
Basically allows you to skip checking from the terminal zpool status continuously or to having to customize the ZED service to send an email and having to figure out how to it can spawn a window alert to the graphical system or what to do if the session has not been initiated.
carleslibs
Since last News from the Blog I’ve released carleslibs v.1.06, v.1.0.5 and v.1.0.4.
v.1.0.6 adds a new class OsUtils to deal with mostly-Linux Os tasks, like knowing the userid, the username, if it’s root, the distribution name and kernel version.
It also adds:
DatetimeUtils.sleep(i_seconds)
In v.1.0.5 I’ve included a new method for getting the Datetime in Unix Epoc format as Integer and increased Code Coverage to 95% for ScreenUtils class.
v. 1.0.4 contains a minor update, a method in StringUtils to escape html from a string.
It uses the library html (part of Python core) so it was small work to do for me to create this method, and the Unit Test for it, but I wanted to use carleslibs in more projects and adding it as core functionality, makes the code of these projects I’m working on, much more clear.
Minor refactors and adding more Code Coverage (Unit Testing), and protection in the code for division per zero when seconds passed as int are 0. (this was not an actual error, but is worth protecting the code just in case for the future)
Currently in Master there is a stable version of 0.8.9 mainly fixing https://gitlab.com/carles.mateo/ctop/-/issues/51 which was not detecting when CTOP was running inside a Docker Container (reporting Unable to decode DMI).
My Books
Docker Combat Guide
Added 20 new pages with some tricks, like clearing the logs (1.6GB in my workstation), using some cool tools, using bind mounts and using Docker in Windows from command line without activating Docker Desktop or WSL.
One of my SATA 2TB 2.5″ 5,400 rpm drive got damaged and so was generating errors, so that was a fantastic opportunity to show how to detect and deal with the situation to replace it with a new SATA 2TB 3.5″ 7,200 rpm and fix the pool.
The company sent me the Stein, which is sent to the employees that serve for two years, with a recognition and a celebration called “The Circle of Honor”.
Books purchased
I bought this book as often I discover new ways, better, to explain the things to my students.
Sometimes I buy books for beginners, as I can get explained what I want to do super fast and some times they teach nice tricks that I didn’t know. I have huge Django books, and it took a lot to finish them.
A simpler book may only talk about how to install and work with it under a platform (Windows or Mac, as instance) but it is all that I require as the command to create projects are the same cross platform.
For example, you can get to install and to create a simple project with ORM, connected to the database, very quickly.
Software
So I just discovered that Zoom has an option to draw in the shared screen, like Slack has. It is called Annotate. It is super useful for my classes. :)
Also discovered the icons in the Chat. It seems that not all the video calls accept it.
Hardware
As Working From Home I needed an scanner, I looked in Amazon and all of them were costing more than €200.
I changed my strategy and I bought a All-In-One from HP, which costed me €68.
So I’ll have a scanner and a backup printer, which always comes handy.
The nightmare started after I tried to connect it with Ubuntu.
Ubuntu was not recognizing it. Checking the manuals they force to configure the printer from an Android/iPhone app or from their web page, my understanding is for windows only. In any case I would not install the proprietary drivers in my Linux system.
Annoyed, I installed the Android application, and it was requesting to get Location permissions to configure it. No way. There was not possible to configure the printer without giving GPS/Location permissions to the app, so I cancelled the process.
I grabbed a Windows 10 laptop and plugged the All-in-one through the USB. I ran the wizard to search for Scanners and Printers and was not unable to use my scanner, only to configure as a printer, so I was forced to install HP drivers.
Irritated I did, and they were suggesting to configure the printer so I can print from Internet or from the phone. Thanks HP, you’ll be the next SolarWinds big-security-hole. I said no way, and in order to use the Wifi I have to agree to open that security door which is that the printer would be connected to Internet permanently, sending and receiving information. I said no, I’ll use only via USB.
Even selecting that, in order to scan, the Software forces me to create an account.
Disappointing. HP is doing very big stupid mistakes. They used to be a good company.
Since they stopped doing the drivers in Barcelona years ago, their Software and solutions (not the hardware) went to hell.
I checked the reviews in the App Store and so many people gave them 1 star and have problems… what a shame the way they created this solution.
This is a great Open Source, multi-platform editor, so I wanted to support the creator.
Security
Attacks: looking for exploits
This is just a sample of a set of attacks to the blog in a 3 minutes interval.
Another one this morning:
Now all are blocked in the Firewall.
This is a non stop practice from spammers and pirates that has been going on for years.
It was almost three decades ago, when I was the Linux responsible of an ISP, and I was installing a brand new Linux system connected to a service called “infovia”, at the time when Internet was used with dial-up and modems, and in the interval of time of the installation, it got hacked. I had the Ethernet connected. So then already, this was happening.
The morning I was writing this, I blocked thousands of offending Ip Addresses.
Protection solutions
I recommend you to use CloudFlare, is a CDN/Cache/Accelerator with DoS protection and even in its Free version is really useful.
Fun/Games
So I come with a game kind of Quiz that you can play with your friends, family or work colleagues working from home (WFH).
The idea is that the master shares screen and sound in Zoom, and then the rest connect to jackbox.tv and enter the code displayed on the master’s screen on their own browser, and an interactive game is started.
It is recommended that the master has two monitors so they can also play.
The games are so fun as a phrase appearing and people having to complete with a lie. If your friends vote your phrase, believing is true, you get points. If you vote the true answer, you get points too.
Very funny and recommendable.
Stuff
<humor>Skynet sent another terminator to end me, but I terminated it. Its processor lays exhibited in my home now</humor>
I bought a laminator.
It has also a ruler and a trimmer to cut the paper.
It was only €39 and I’ve to say that I’m very happy with the results.
It takes around 5 minutes to be ready, it takes to get to the hot-enough temperature, and feeds the pages slowly, around 50 secs a DIN-A4, but the results are worth the time.
I’ve protected my medical receipts and other value documents and the work was perfect. No bubbles at all. No big deal if the plastic covers are introduced not 100% straight. Even if you pass again an already plasticized document, all is good.
Fun
Databases
One of my friends sent me this image.
It is old, but still it’s fun. So it assumes the cameras of the parking or speed cameras, will OCR the plate to build a query, and that the code is not well protected. So basically is exploiting a Sql Injection.
Anybody working on the systems side, and with databases, knows how annoying are those potential situations.
I released my Python Open Source libraries for rapid application development: carleslibs to v. 1.0.3.
I updated the page of my 2013 PHP Framework Catalonia Framework, linking to the blog. This is a very nice and lightweight PHP Framework that I created then and it worked so well that I had not the need to update it since 2014.
Donations
Blizzard offered to the employees the opportunity to donate in the name of Blizzard, up to USD $100. It has not been the first time, is a regular thing. Also in the past the company matched any donation we employees did, to help diversity will less resources to study in the university.
I chose to donate to Lions in Cork, Ireland.
When they lived, my grand parents were members and donors from Lion’s in Catalonia, so I trust them.
Security
Log4j Java vulnerability
A critical vulnerability named Log4Shell was found in Apache Log4j Java Open Source logging Library.
I published this book to help developers to understand and use Docker.
It is not targeted to SysAdmins, is aimed to Developers that want to get an operative know how by examples very quickly, and easy to read.
My other books have also updates not yet published, however an update has been published for Python 3 exercises for beginners book.
University classes are restarted, and I fixed my tower.
For the Cloud computing degree this semester VMWare is used intensively.
I have a dedicated tower with an AMD Ryzen 7 processor, a Samsung NMVe drive PCIe 4.0, which provides me a throughput of 6GB/second (six Gigabytes, so 48 Gbit/second), SAS drives and SATA too. It’s a little monster with 64 GB of RAM and 2.5 Gbps NIC.
It was not starting.
The problem was in the Video card, which made loosely contact to the motherboard.
I had to disconnect everything until I found what it was, but after moving the video card to another PCI slot, it worked.
I knew it was some sort of short circuit / bad contact as the fans were turning for a second and turning off immediately.
After this, the computer works fine but it will poweroff in about 4h and 12 hours. I’ve been testing and removing each component until I believe is the PSU. I’ve ordered a new one from a Dutch provider with web store in Ireland that my former colleague Thomas showed me one year and half ago.
Since England leaved the EU, it is impossible to buy from amazon.co.uk without experiencing problems in the border and delays.
If you want to learn how to assemble a PC, fix the problems and upgrade your laptop, I wrote this book:
If you are curious about what I use in my day to day:
A tower for developing and reading my email, with Linux, Intel i7 7800X (12 cores) and 64 GB of RAM, with Nvidia graphics card
A tower for holding Virtual Machines, with Linux, AMD Ryzen 7 3700x (16 cores) and 64 GB of RAM, with Nvidia graphics card
An upgraded HP laptop for programming in the cafe, is a Windows 10, with 16 GB of RAM
Raspberry Pi 4 and 3, from time to time
A laptop for programming, for Work, 16 GB of RAM
A tower for programming, for Work, at the office, 32 GB of RAM
I also had a Dell computer which battery inflated elevating the touchpad, an Acer 11.6 Latop very lightweight which screen died cracked apparently (it’s a mistery to me how this happened as I removed from the bag and it was cracked. That little laptop accompanied me during years, to many countries, as for a while I carried it with me 100% of the time. At that time if the companies I worked for had outages they were losing thousands of euros per hour, so as CTO I fixed broken stuff even in a restaurant. Believe when I recommend you and your teams to use Unit Testing) and a 15.6″ Acer with 16GB of RAM that was part of the payment of an Start up I was CTO for, and which screen flicks intermittently and I managed to fix it by applying a pressure point to a connector, so I managed to use as fixed computer at the beginning of being in Ireland. I was not using it much, as I had two laptops from work when working for Sanmina, a Dell with 16 GB of RAM and Core i7 with two external monitors and an Intel Xeon with 32GB of RAM, heavy weight, but very useful for my job (programming, doing demos, having VMs…).
I’ve assembled all my PC from the scratch, piece by piece, and I force myself to do it so I keep up to date of the upcoming technologies, buses, etc…
My students are doing well. Congrats to Albert for getting 8.67 from 10 in his university programming course exams!.
Diablo 2 Resurrected is published and I am in the credits :)
I’m in the credits of all our games since I joined, but I’m happy every time I see myself and my colleagues on them. :)
This release includes SubProcessUtils which is a class that allows you to execute commands to the shell (or without shell) and capture the STDOUT, STDERR, and Exit Code very easily.
I’ve used my libraries for a hackaton PoC for work, for Monitoring one aspect of one of our top games side, and I coded it super quickly. :)
They loved it and we have a meeting scheduled to create a Service from my PoC. :)
The addition I made to this version is StringUtils class which offer functionalities for handling amount conversions (to different units), number formatting, string formatting and align (left, right…). I added a 85% of Unit Testing Code Coverage.
Here you have some general information about how to install and how to use the package:
I’ve released v. 0.99 of carleslibs package This package includes utilities for:
Files and Directories handling
Date/Time retrieval
Python version detection
You can install it with:
pip install carleslibs
The minimum requirement declared is Python 3.6, although they work with Python 3.5 and Python 2.7, as I want to drop support for no longer supported versions.
If you are enrolled with Linux Academy, I recommend you this Python course: Python 3 Scripting for System Administrators https://linuxacademy.com/cp/modules/view/id/168 Does not cover OOP, but it does a bit of Unit Testing, and talks a lot about PIP and Virtual Environments.
