Only 2% of the viewers donate, so I answered the call every time it was made.
This is my 5th donation to Wikimedia.
I consider that Freedom is very important.
I bought these new books
One of my secrets to be on top is that I’m always studying.
I study all the time, at work and in my free time.
I use Linux Academy and I buy books in paper. I don’t connect with reading in tablets. I think information is stored better when read in paper. I use also a marker and pointers to keep a direct access to the most interesting points on the books.
And I study all kind of themes. Obviously I know a lot of Web Scraping, but there is always room for learning more. And whatever new I learn helps me to be better with my students and more clear writing my books.
I’ve never been a Front End, but I’ve been able to fix bugs in the Front End engines from the companies I worked for, like Privalia. I was passed a bug that prevented the Internet Explorer users to buy just one hour before we launching a massive campaign. I debugged and I found a variable named “value” so the html looked like <input name="value" value="">. In less than 30 minutes I proved to the incredulous Head of Development and the CTO that a bug in Internet Explored was causing a conflict when fetching the value from the input named value. We deployed to Production the update and the campaign was a total success. So I consider knowing Javascript and Front also a need, even if I don’t work directly with it. I want to be able to understand all the requirements and possibilities, and weaknesses, so I can fix bugs and save the day. That allowed me to fix scalability problems in Nodejs and Phantomjs projects too. (They are Javascript Server Side, event driven, projects)
It seems that Amazon.co.uk works well again for Ireland. My two last orders arrived on time and I had no problems of border taxes apparently.
Nice Python article
I enjoyed a lot this article, cause explains part of what I did with my student and friend Albert, in a project that analyzes the access logs from Apache for patterns of attempts of exploits, then feeds a database, and then blocks those offender Ip Addresses in the Firewall.
The article only covers the part of Pandas, of reading the access.log file and working with it, but is a very well redacted article:
Normally if we need to refresh a config in a Container we will spawn a new one, or we will access with sudo docker exec -it /bin/sh mycontainer for instance and force a reload, or we will have to restart the Container.
What if we want to be able to reload the config at any moment without restarting the process, or to trigger a process in our Container (like a dump or a flush) in another way than implementing an API?.
An unexplored way, for many, to communicate with your Container’s main process is to send Signals.
So basically I will show you how you can trap Signals within a Python process which is the main process for your Docker Container, and send them from your Hypervisor with the command:
sudo docker kill --signal=SIGUSR1
I choose to use SIGUSR1 as it is reserved for user defined Signals.
You can clone the project or get the source code from:
FROM ubuntu:20.04
MAINTAINER Carles Mateo
RUN apt update && apt install -y python3 python3-pip vim less && apt-get clean
# This will make sure printing in the Screen when running in dettached mode
ENV PYTHONUNBUFFERED=1
ENV DOCKERSIGNAL /var/dockersignal
RUN mkdir -p $DOCKERSIGNAL
COPY *.py $DOCKERSIGNAL
WORKDIR $DOCKERSIGNAL
# Again to enforce printing to the Screen when running dettached
CMD ["python3", "-u", "/var/dockersignal/dockersignal.py"]
The dockersignal.py file
# By Carles Mateo https://blog.carlesmateo.com
import signal
import time
def handler(signum, frame):
print('Signal handler called with signal', signum)
if signum == 10:
# 10 is the equivalent to SIGUSR1 for most x86/ARM (not for Alpha/Sparc, MIPS, PARISC)
print("Simulated action: Reload config")
if __name__ == "__main__":
print("Waiting for a Signal")
# Listed for this signal, so can listen for more
signal.signal(signal.SIGUSR1, handler)
while True:
# Do Whatever
time.sleep(1)
A shell file to build and run the Container like a pro
#!/bin/bash
DOCKER_CONTAINER_NAME="docker-signal"
DOCKER_IMAGE_NAME="docker-signal"
printf "Removing old Container %s\n" "${DOCKER_CONTAINER_NAME}"
sudo docker rm "${DOCKER_IMAGE_NAME}"
printf "Removing old Image %s\n" "${DOCKER_IMAGE_NAME}"
sudo docker image rm "${DOCKER_IMAGE_NAME}"
echo "Creating Docker Image"
sudo docker build -t ${DOCKER_IMAGE_NAME} . --no-cache
retVal=$?
if [ $retVal -ne 0 ]; then
printf "Error. Exit code %s\n" ${retVal}
exit
fi
echo "Running Docker Container ${DOCKER_CONTAINER_NAME} based in image ${DOCKER_IMAGE_NAME}"
sudo docker run --cpus="1.0" --name ${DOCKER_CONTAINER_NAME} ${DOCKER_IMAGE_NAME}
Here is the complete history of why I migrated all the services from my 11 years old Amazon account to other CSP.
Some lessons can be learned from my adventure.
I migrated my last services from Amazon to GCP
Amazon sent me an email on October 6th, this year 2021, telling me that they will disable EC2-Classic by August 2022. I thought I would not be able to keep my Static Ip’s as in the past VPC Ip’s and EC2-Classic Ip’s were not transferable, so considering that I would loss my Static Ip’s anyway I started to migrate to some to other providers like Digital Ocean.
Is not cool losing Static Ip (Elastic Ip in AWS) Addresses as this is bad for SEO, so given that I though I would lose my Static Ips that have been with me for years, I started to migrate certain services to providers much more economic.
Amazon is terrible communicating, and I talked with some product managers in the past about that, when they lost one of my Volumes, and the email was so cold and terrible that actually that hurt more than Amazon losing my Data. I believed that it was a poorly made Scam and when I realized it was true I reached one of my friends, that is manager there, as I know they care for doing things right, and he organized a meeting with two PM so I can pass my feedback.
The Cloud providers are changing things very fast, and nobody is able to be up to date with the changes, unless their work position allows plenty of time to get updated. Even if pages of documentation are provided, you have to react to an event that they externally generated forcing you to action. Action to read all the documentation about EC2-Classic migrations, action to prepare to have migrated by August 2022.
So August 2022… I was counting that I had plenty of time but I’m writing a new book about using the Amazon SDK for Python, boto3, and I was doing some API calls and they started to fail in a very unusual way, Exceptions with timeout, but only for the only region where I had EC2-Classic.
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x7f0347d545e0>: Failed to establish a new connection: [Errno -2] Name or service not known
But if I switched to another region name, it would work:
region_name='us-west-2',
I made a mistake in here, the region name is “us-east-1” and not “us-east-1a“. “us-east-1a” is the availability zone. So the SDK was giving a timeout because in order to connect to the endpoint it uses the region name as part of the hostname. So it doesn’t find that endpoint because it doesn’t exist.
I never understood why a company like Amazon is unable to provide the SDK with a sample project or projects 100% working, with the source code so people has a base that works to build up.
Every API that I have created, I have provided it with documentation but also with example for several languages for how to use it.
In 2013 I was CTO of an online travel agency, and we had meta-searchers consuming our API and we were having several hundreds of thousands requests per second. Everything was perfectly documented, examples were provided for several languages, the document and the SDK had version numbers…
Everybody forgets about Developers and companies throw terrible and cold products to the poor Developers, so difficult to use. How many Developers would like to say: Listen Mr. President of the big Cloud Company XXXX, I only want to spawn a VM that works, and fast, with easy wizards. I don’t want to learn 50 hours before being able to use your overpriced platform, by doing 20 things before your Ip’s are reflexes of your infrastructure and based in Microservices. Modern JavaScript frameworks can create nice gently wizards even if you have supercold APIs.
Honestly, I didn’t realize my typo in the region and I connected to the Amazon Console to investigate and I saw this.
Honestly, when I read it I understood that they were going to end my EC2 Networking the 30th of October. It was 29th. I misunderstood.
It was my fault not reading it well to the end, I got shocked by the first part telling about shutdown and I didn’t fully understood as they were going to shutdown EC2-Classic for the zones I didn’t had anything running only.
From the long errors (3 exceptions chained) I didn’t realize that the endpoint is built with the region name. (And I was passing the availability zone)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://ec2.us-east-1a.amazonaws.com/"
Here is when I say that a good SDM would had thought and cared for the Developers more, and would had made the SDK to check if that region exists. How difficult is to create a SDK a bit more clever that detects a invalid region id?. It is not difficult.
It is true that it was late in the evening and I was tired of all the day, and two days of the week between work and zoom university classes I work 15 hours and 13 hours respectively, not counting the assignments, so by the end of the week I am very tired. But that’s why it is very important to follow methodology and to read well. I think Amazon has 50% of the fault by the way they do things: how the created the SDK, how they communicate, and by the errors that the console returned me when I tried to create a VPC instance of an EC2-Classic AMI (they seem related to the fact I had old VPC Network objects with shorter hash than the current they use) and the other 50% was my fault for not identifying the source of the error, and not reading the message in their website well.
But the fact that there were having those errors in the API’s and timeouts made me believe they were going to cut the EC2-Classic Networking the next day.
All the mistakes fall together in a perfect storm.
I checked for documentation and I saw it was possible to migrate my Static Ip’s to VPC Static Ip’s.
It was Friday evening, and I cancelled my plans, in order to migrate the Blog to VPC in an attempt to keep running it with Amazon.
As Cloud Architect, I like to have running instances in several CSP as it allows me to stay up to date with the changes they do.
I checked the documentation for the migration. Disassociating the Static Ip (Elastic Ip in AWS jargon) was easy. Turning into VPC as well.
As I progressed, what had to be easy turned into a nightmare, as I was getting many errors from the Amazon API, without any information, and my Instances were not created.
I figured out that their API could have problems with old VPC objects I created time ago, so I had to create new objects for several things.
I managed to spawn my instances but they were being launch and terminated instantly without information. Frustrating.
When launching a new instance from the AMI (a Snapshot of the blog), I was giving shown options to add more volumes without any sense. My Instance was using 16GB from a 20GB total Space, and I was shown different volume configs, depending on the instance, in some case an additional 20GB volume, in other small SSD, ephemeral and 10 GB for the AMI (which requires at least 16GB).
After some fight I manage to make it work after deleting the volumes that made no sense, and keeping only one of 20GB, the same size of my AMI.
But then my nightmare started to make the VPC Instance to have Internet access and to be seen from outside. I had to create a new Internet Gateway, NAT, Network, etc…
As mentioned the old objects I was trying to reusing were making the process to fail.
I was running out of time, and I thought in few time they were going to shutdown EC2-Classic network (as I did not read correctly), so I decided to download everything and to migrate to another provider. For doing that first I blocked all the traffic, except for my Ip.
I worked in parallel, creating the new config in Google Cloud, just in case I had forgot something. I had created a document for the migration and it was accurate.
I managed to do everything fast enough. The slower part was to download all the Data, as I hold entire VM’s for projects like Cassandra Universal Driver.
Then I powered off my Amazon Instance for the Blog forever.
In GCP I blocked all the traffic in the firewall, except for my Ip, so I could work calmly.
When everything was ready, I had to redirect the DNS to the new static Ip from Google.
The DNS provider I used had implemented some changes in their API so I was getting errors replacing my old entry ‘.’ (their JSON calls returned Internal Server Error). Finally I figured it out how to workaround it and I was able to confirm that the first service was up and running.
I did some tests to make sure there were not unexpected permission problems, entries in the logs, etc…
Only then I opened the Google Firewall. I have a second firewall in each instance where I block or open at Ip tables level what I want. Basically abusive bot’s IPs trying to find exploits or brute force by dictionary passwords.
I checked with my phone, without Wifi that the Firewall was all good. (It is always a good idea to use another external Ip, different from the management one, to check)
You may like some of other graphs that can be easily generated with matlib package.
I’ve been always working on BackEnd and APIs and I don’t work on FrontEnd, although I programmed some videogames by myself and I’ve fixed some huge bugs in JavaScript in some of the companies I work, but they considered myself the last resource, so I would fix a FrontEnd bug when nobody else could. But even if you work 99.9% of your time in BackEnd, Scaling, Architecture… like me, it is useful being able to draw graphics, for example, when you create a tool that shows the number of players per minute, and its evolution over time, or web visitors in real time, etc…
I’m not talking about the wonderful things, like how big can the Integers be, but about the bizarre things that may ruin your day.
What sums 0.1 + 0.1 + 0.1 in Python?
0.3?
Wrong answer.
A bit of humor
Well, to be honest the computer was wrong. They way programming languages handle the Floats tend to be less than ideal.
Floats
Maybe you know JavaScript and its famous NaN (Not a number).
You are probably sure that Python is much more exact than that…
…well, until you do a big operation with Floats, like:
10.0**304 * 10.0**50
and
It returns infinite
I see your infinite and I add one :)
However If we try to define a number too big directly it will return OverflowError:
Please note Integers are handled in a much more robust cooler way:
Negative floats
Ok. What happens if we define a number with a negative power, like 10 ** -300 ?
And if we go somewhere a bit more far? Like 10 ** -329
It returns 0.0
Ups!
I mention in my books why is better to work with Integers, and in fact most of the eCommerces, banks and APIs work with Integers. For example, if the amount in USD 10.00 they send multiplied by 100, so they will send 1000. All the actor know that they have to divide by 2.
Breaking the language innocently
I mentioned always that I use the MT Notation, the prefix notation I invented, inspired by the Hungarian Notation and by an amazing C++ programmer I worked with in Volkswagen and in la caixa (now caixabank), that passed away many years ago.
Well, that system of prefixes will name a variable with a prefix for its type.
It’s very useful and also prevents the next weird thing from Python.
Imagine a Junior wants to print a String and they put in a variable. And unfortunately they call this variable print. Well…
print = "Hello World!"
print("That will hurt")
Observe the output of this and try not to scream:
Variables and Functions named equally
Well, most of languages are able to differentiate a function, with its parenthesis, from a variable.
The way Python does it hurts my coder heart:
Another good reason to use MT Notation for the variables, and for taking seriously doing Unit Testing and giving a chance to using getters and setters and class Constructor for implementing limits and sanitation.
Nested Loops
This will work in Python, it doesn’t work in other languages (but please never do it).
for i in range(3):
print("First Loop", i)
for i in range(4):
print("Second Loop", i)
The code will not crash by overwriting i used in the first loop, but the new i will mask the first variable.
And please, name variables properly.
Import… once?
Imports are imported only once. Even if different files imported do import the same file.
So don’t have code in the middle of them, outside functions/classes, unless you’re really know what you’re doing.
Define functions first, and execute code after if __name__ == “__main__”:
Take a look at this code:
def first_function():
print("Inside first function")
second_function()
first_function()
def second_function():
print("Inside second function")
Well, this will crash as Python executes the code from top to bottom, and when it gets to first_function() it will attempt to call second_function() which has not been read by Python yet. This example will throw an error.
You’ll get an error like:
Inside first function
Traceback (most recent call last):
File "/home/carles/Desktop/code/carles/python_combat_guide/src/structure_dont_do_this.py", line 14, in <module>
first_function()
File "/home/carles/Desktop/code/carles/python_combat_guide/src/structure_dont_do_this.py", line 12, in first_function
second_function()
NameError: name 'second_function' is not defined
Process finished with exit code 1
Add your code at the bottom always, under:
if __name__ == "__main__":
first_function()
The code inside this if will only be executed if you directly call this code as main file, but will not be executed if you import this file from another one.
You don’t have this problem with classes in Python, as they are defined first, completely read, and then you instantiate or use them. To avoid messing and creating bugs, have the imports always on the top of your file.
…Ellipsis
Today is Halloween and one of my colleagues asked me help to improve his Automation project.
I found something weird in his code.
He had something like that.
class Router:
def router_get_info(self):
...
def get_help_command(self):
return "help"
So I asked why you use … (dot dot dot) on that empty method?.
He told me that when he don’t want to implement code he just put that.
Well, dot dot dot is Ellipsis.
And what is Ellipsis?.
Ellipsis is an object that may appear in slice notation.
In Python all the methods, functions, if, while …. require to have an instruction at least.
So the instruction my colleague was looking for is pass.
Just a variable?
In Python you can have just a var, without anything else, like no operation with it, no call, nothing.
This makes it easy to commit an error and not detecting it.
As you see we can have just s_var variable in a line, which is a String, and this does not raises an error.
If we do from python interpreter interactively, it will print the String “I’m a pickle” (famous phrase from Rick and Morty).
Variables are case sensitive
So you can define true false none … as they are different from True False None
Variables in Unicode
Python3 accepts variables in Unicode.
I would completely discourage you to use variables with accents or other characters different from a-z 0-9 and _
Python files with these names yes, but kaboom if you import them
So you can create Python files with dash or beginning with numbers, like 20220314_programming_class.py and execute them, but you cannot import them.
RYYFTK RODRIGUEZ,LEELA,FRY, FUTURAMA, 1999
A Tuple of a String is not a Tuple, it’s a String
This can be very messy and confusing. Normally you define a tuple with parenthesis, although you can use tuple() too.
Parenthesis are the way we normally build tuples. But if we do:
print(type('this is a String'))
You get that this is a String, I mean
<class 'str'>
If you want to get a tuple of a String you can add a comma after the first String, which is weird. You can also do tuple("this is a String")
I think the definition of a tuple should be consistent and idempotent, no matter if you use one or more parameters. Probably as parenthesis are used for other tasks, like invoking functions or methods, or separating arithmetic operations, that reuse of the signs () for multiple purposes is what caused a different behavior depending on if there is one or more parameters the mayhem IMO.
See some example cases.
Python simplifies the jump of line \n platform independent and some times it’s messy
If you come from a C background you will expect text file in different platforms: Linux, Mac OS X (changes from old to new versions), Windows… to be represented different. In some cases this is an ASCii code 10 (LF), in others 13 (CR), and in other two characters: 13 and immediately after 10.
Python simplifies the Enter character by naming it \n like in C.
So, platform independent, whenever you read a text file you will get \n for any ASCii 10 [LF] or 13 [CR]. [CR] will be converted to [10] in Linux.
If you read a file in a Linux system, where enters are represented by 10, which was generated in a Windows system, so it has [CR][LF] instead of [LF] at the end of each line, you’ll get a \n too, but two times.
And if you do len(“\n”) to know the len of that String, this returns 1 in all the platform.
To read the [LF] and [CR] (represented by \r) you need to open the file as binary. By default Python opens the files as text.
You can check this by writting [LF] and [CR] in Linux and see how Python seamlessly reads the file as it was [LF].
A file generated by Windows will get \n\n:
Random code when the class is imported
In a procedural file, the code that is outside a function, will be executed when it is imported. But if this file is imported again it will not be re-executed.
Things are more messy if you import a class file. Inside the body of the class, in the space you would reserve for static variables definition, you can have random code. And this code will be only executed on the first import, not on subsequent.
Disclaimer: the pictures from Futurama are from their respective owners.
This is a very simple code but generates very cool Word Cloud result in PNG format.
from wordcloud import WordCloud
# Add your tags in here separated by commas and as many times as they appear
s_text = "Python, Software development, PHP, Cloud providers, Python, Python, Software development, Scaling"
o_word_cloud = WordCloud(height=800,
width=800,
background_color="white",
max_words=150,
min_font_size=5,
collocation_threshold=10).generate_from_text(s_text)
o_word_cloud.to_file("words.png")
That version generated the image .PNG file.
If you want to display this in mathlib or inside PyCharm embedded view, you can install matplotlib with:
pip3 install matplotlib
Then with this code you can display a matplotlib viewer:
import matplotlib.pyplot as plt
from wordcloud import WordCloud
if __name__ == "__main__":
# Add your tags in here separated by commas and as many times as they appear
s_text = "Python, Software development, PHP, Cloud providers, Python, Python, Software development, Scaling"
o_word_cloud = WordCloud(height=800,
width=800,
background_color="white",
max_words=150,
min_font_size=5,
collocation_threshold=10).generate_from_text(s_text)
plt.figure(figsize=(10,8))
plt.imshow(o_word_cloud)
plt.axis("off")
plt.tight_layout(pad=0)
plt.show()
I’ve raised back the price for my books to normal levels. I’ve been keeping the price to the minimum to help people that wanted to learn during covid-19. I consider that who wanted to learn has already done it.
I still have bundles with a somewhat reduced price, and I authorized LeanPub platform to do discounts up to 50% at their discretion.
v.0.29 and v.0.28 Python 3 Simple Exercises for Beginners Added a new section Games, with a first game “Guess my number”
Provided a solution for the recipe Exercise 3: Create a function that will ask a user for a number from Keyboard input, and return the result, only when the value is between the accepted ranges.
Added two new questions to the Quiz section.
Fixed a docstring in Recipes Exercise 4, referring to a String return value which it was an Integer.
Added an exercise for retrieving a JSON with your public Ip
Added a new exercise for converting bytes to kilobytes with two decimal positions.
Added a new exercise / recipe to SSH to a Server with Username and Password and execute a command, using the Paramiko library.
Added a new exercise / recipe to SSH to a Server with Username and Password and execute a command using Paramiko library. I added two examples executing commands uptime and df -h /
Show an alternative way to run Flask apps.
Added new interesting packages.
As long as covid is active I plan to keep the minimum price of each of my books at the minimum accepted by LeanPub which is USD $5.
I also enable bundles and enable LeanPub to make punctual discounts to make them even more affordable to humble pockets.
I’m going to teach an initiation to programming class, a live Zoom session of 1 hour, plus 15 minutes for questions, for Free. It will be a very basic starting class for absolute beginners.
It will be performed at a time around 19:00 Irish time, wich is 11:00 AM in Irivine, CA time, to maximize the opportunities for people to assist.
If you would like to join, write me an email:
There are not many spots available, but if there is no room for you this time I may contact you for the next time.
This version adds class StringUtils, and a set of methods to perform different useful tasks with Strings, like converting a number to the biggest unit with sense, like a large number of bytes to PB, a smaller one to TB, or GB, MB, KB, justifying strings to the right or to the left, and cap the number of chars to a specified one, etc…
I have many libraries that I’ve been building across the years, and I’m liberating them as Open Source, as soon as I have time to make sure that are compatible with Python 3.5 or superior (and with Python 2.6 when possible), and I have time to add a decent Unit Testing Code Coverage.
I try to release libraries that have no other dependencies. After that I’ll start releasing my libs that have dependencies, like to work with MySQL, SQLite, web scrapping, etc…
