Tag Archives: Python 3

Provisioning AWS EC2 Instances with Ansible and Automating Apache deployment with or without using Ansible Dynamic Inventory from Ubuntu 20.04 LTS

This article is being included in my book Provisioning to Amazon AWS using boto3 SDK for Python 3.

Pre-requisites

I’ll use Ubuntu 20.04 LTS.

Python 2 is required for Ansible.

Python 3 is required for our programs.

sudo apt install python2 python3 python3-pip
# Install boto for Python 2 for Ansible (alternative way if pip install boto doesn't work for you)
python2 -m pip install boto
# Install Ansible
sudo apt install ansible

If you want to use Dynamic Inventory

So you can use the Python 2 ec2.py and ec2.ini files, adding them as to the /etc/ansible with mask +x, to use the Dynamic Inventory.

You will need to have your credentials set.

I use Environment variables:

#!/bin/bash

export ANSIBLE_HOST_KEY_CHECKING=false
export AWS_ACCESS_KEY=AKIXXXXXXXXXXXXXOS
export AWS_SECRET_KEY=e4dXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXY6F

Then use the calls inside the shell script, or assuming that the previous file was named credentiasl.sh use source credentials.sh

ec2.py is written in Python 2, so probably will fail for you as it is invoked by python and your default interpreter will be Python 3.

So edit the first line of /etc/ansible/ec2.py and add:

#!/bin/env python2

Once credentials.sh is sourced, then you can just invoke ec2.py to get the list of your Instances in a JSON format dumped by ec2.py

/etc/ansible/ec2.py --list

You can get that JSON file and load it and get the information you need, filtering by group.

You can call:

/etc/ansible/ec2.py --list > instances.json

Or you can run a Python program that escapes to shell and executes ec2.py –list and loads the Output as a JSON file.

I use my carleslibs here to escape to shell using my class SubProcessUtils. You can install them, they are Open Source, or you can code manually if you prefer importing subprocess Python library and catching the stdout, stderr.

import json
from carleslibs import SubProcessUtils

if __name__ == "__main__":
    s_command = "/etc/ansible/ec2.py"

    o_subprocess = SubProcessUtils()
    i_error_code, s_output, s_error = o_subprocess.execute_command_for_output(s_command, b_shell=True, b_convert_to_ascii=True, b_convert_to_utf8=False)
    if i_error_code != 0:
        print("Error escaping to shell!", i_error_code)
        print(s_error)
        exit(1)

    json = json.loads(s_output)

    d_hosts = json["_meta"]["hostvars"]

    for s_host in d_hosts:
        # You'll get a ip/hostnamename in s_host which is the key
        # You have to check for groups and the value for the key Name, in order to get the Name of the group
        # As an exercise, print(d_hosts[s_host]) and look for:
        # @TODO: Capture the s_group_name
        # @TODO: Capture the s_addres
        if s_group_name == "yourgroup":
             # This filters only the instances with your group name, as you want to create an inventory file just for them
             # That's because you don't want to launch the playbook for all the instances, but for those in your group name in the inventory file.
             a_hostnames.append(s_address)

    # After this you can parse you list a_hostnames and generate an inventory file yourinventoryfile 
    # The [ec2hosts] in your inventory file must match the hosts section in your yaml files
    # You'll execute your playbook with:
    # ansible-playbook -i yourinventoryfile youryamlfile.yaml

So an example of a yaml to install Apache2 in Ubuntu 20.04 LTS spawned instances , let’s call it install_apache2.yaml would be:

---
- name: Update web servers
  hosts: ec2hosts
  remote_user: ubuntu

  tasks:
  - name: Ensure Apache is at the latest version
    apt:
      name: apache2
      state: latest
      update_cache: yes
    become: yes

As you can see the section hosts: in the YAML playbook matches the [ec2hosts] in your inventory file.

You can choose to have your private key certificate .pem file in /etc/ansible/ansible.cfg or if you want to have different certificates per host, add them after the ip/address in your inventory file, like in this example:

[localhost]
127.0.0.1
[ec2hosts]
63.35.186.109	ansible_ssh_private_key_file=ansible.pem

The ansible.pem certificate must have restricted permissions, for example chmod 600 ansible.pem

Then you end by running:

ansible-playbook -i yourinventoryfile install_ubuntu.yaml

If you don’t want to use Dynamic Directory

The first method is to use add_host to print in the screen the properties form the ec2 Instances provisioned.

The trick is to escape to shell, executing ansible-playbook and capturing the output, then parsing the text looking for the ‘public_ip:’

This is the Python 3 code I created:

class AwesomeAnsible:

    def extract_public_ips_from_text(self, s_text=""):
        """
        Extracts the addresses returned by Ansible
        :param s_text:
        :return: Boolean for success, Array with the Ip's
        """

        b_found = False
        a_ips = []

        i_count = 0
        while True:
            i_count += 1
            if i_count > 20:
                print("Breaking look")
                break
            s_substr = "'public_ip': '"
            i_first_pos = s_text.find(s_substr)
            if i_first_pos > -1:
                s_text_sub = s_text[i_first_pos + len(s_substr):]
                # Find the ending delimiter
                i_second_pos = s_text_sub.find("'")
                if i_second_pos > -1:
                    b_found = True
                    s_ip = s_text_sub[0:i_second_pos]
                    a_ips.append(s_ip)
                    s_text_sub = s_text_sub[i_second_pos:]
                    s_text = s_text_sub
                    continue

            # No more Ip's
            break

        return b_found, a_ips

Then you’ll use with something like:

        # Catching the Ip's from the output
        b_success, a_ips = self.extract_public_ips_from_text(s_output)
        if b_success is True:
            print("Public Ips:")
            s_ips = ""
            for s_ip in a_ips:
                print(s_ip)
                s_ips = s_ips + self.get_ip_text_line_for_inventory(s_ip)
            print("Adding Ips to group1_inventory file")
            self.o_fileutils.append_to_file("group1_inventory", s_ips)
            print() 

The get_ip_text_line_for_inventory_method() returns a line for the inventory file, with the ip and the key to use separated by a tab (\t):

    def get_ip_text_line_for_inventory(self, s_ip, s_key_path="ansible.pem"):
        """
        Returns the line to add to the inventory, with the Ip and the keypath
        """
        return s_ip + "\tansible_ssh_private_key_file=" + s_key_path + "\n"

Once you have the inventory file, like this below, you can execute the playbook for your group of hosts:

[localhost]
127.0.0.1
[ec2hosts]
63.35.186.109	ansible_ssh_private_key_file=ansible.pem
ansible-playbook -i yourinventoryfile install_ubuntu.yaml

Alternative way parsing with awk and grep

You can run this Bash Shell Script to get only the public ips when you provision to Amazon AWS EC2 the Instances from your group named group1 in this case:

./launch_aws_instances-group1.sh | grep "public_ip" | awk '{ print $13; }' | tr -d "',"
52.213.232.199

In this example 52.213.232.199 is the Ip from the Instance I provisioned.

So to put it together, from a Python file I generate this Bash file and I escape to shell to execute it:

#!/bin/bash

export ANSIBLE_HOST_KEY_CHECKING=false
export AWS_ACCESS_KEY=AKIXXXXXXXXXXXXXOS
export AWS_SECRET_KEY=e4dXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXY6F

# Generate a new Inventory File
echo "[localhost]" > group1_inventory
echo "127.0.0.1" >> group1_inventory
echo "[ec2hosts]" >> group1_inventory

ansible-playbook -i group1_inventory launch_aws_instances-group1.yaml

I set again the credentials because as this Bash Shell Script is invoked from Python, there are not sourced.

The trick in here is that the launch_aws_instances-group1.yaml file has a task to add the hosts to Ansible’s in memory inventory, and to print the information.

That output is what I scrap from Python and then I use extract_public_ips_from_text() showed before.

So my launch_aws_instances-group1.yaml (which I generate from Python customizing the parameter) looks like this:

# launch_aws_instances.yaml

- hosts: localhost
  connection: local
  gather_facts: False
  vars:
      s_keypair_name: "ansible"
      s_instance_type: "t1.micro"
      s_image: "ami-08edbb0e85d6a0a07"
      s_group: "ansible"
      s_region: "eu-west-1"
      i_exact_count: 1
      s_tag_name: "ansible_group1"
  tasks:

    - name: Provision a set of instances
      ec2:
         key_name: "{{ s_keypair_name }}"
         region: "{{ s_region }}"
         group: "{{ s_group }}"
         instance_type: "{{ s_instance_type }}"
         image: "{{ s_image }}"
         wait: true
         exact_count: "{{ i_exact_count }}"
         count_tag:
            Name: "{{ s_tag_name }}"
         instance_tags:
            Name: "{{ s_tag_name }}"
      register: ec2_ips
      
    - name: Add all instance public IPs to host group
      add_host: hostname={{ item.public_ip }} groups=ec2hosts
      loop: "{{ ec2_ips.instances }}"

In this case I use t1.micro cause I provision to EC2-Classic and not to the default VPC, otherwise I would use t2.micro.

So I have a Security Group named ansible created in Amazon AWS EC2 console as EC2-Classic, and not as VPC.

In this Security group I opened the Inbound HTTP Port and the SSH port for the Ip from I’m provisioning, so Ansible can SSH using the Key ansible.pem

The Public Key has been created and named ansible as well (section key_name under ec2).

The Image used is Ubuntu 20.04 LTS (free tier) for the region eu-west-1 which is my wonderful Ireland.

For the variables (vars) I use the MT Notation, so the prefixes show exactly what we are expecting s_ for Strings i_ for Integers and I never have collisions with reserved names.

It is very important to use the count_tag and instance_tags with the name of the group, as the actions will be using that group name. Remember the idempotency.

The task with the add_host is the one that makes the information for the instances to be displayed, like in this screenshot.

Python Programming class for beginners 2021-11-11

Suggested optional pre-requisites to have before the class

  • Install Python 3 (3.8 recommended, any version from Python 3.6 is fine)
  • Install PyCharm Community Edition or Professional Edition (is free for 30 days only)

Suggested optional reading material

Examples of the class today have been upload here

Material from some other classes

On this class, recorded on the video (1h 48m)

Why there are different types of variables

  • Memory
    • All are 0 and 1’s
  • How big can be a variable (float, integer, string)
  • Operations performed, like +
  • MT Notation I use

The order in which the instructions are performed (top to bottom).

  • Declaring a variable
  • We can call thinks that were previously defined up (like functions)
  • Loops will send the execution pointer up
  • Operations with integer variables
    • Additions
  • Operations with Strings

Language syntax and tricks (write a Notebook with your own notes)

  • Pre-created solutions, like reverse an string with for
  • Open a file, read the contents by lines to string, split the strings to arrays by a separator, like tab or space, get what you want by position.
with open(s_filename) as file:
    for s_line in file:
        print(s_line)

# Show Exceptions

# Show hexadecimal of the text:

# cd /home/carles/Desktop/code/carles/python-classes/2021-11-11

# hexdump -c text.txt

# remove enters, spaces at the end of the file with s_line.rstrip()

The blocks and functions

  • Indentation
  • Typical error missing :
  • Functions are for reusing code and reducing errors
    • As they are for reusing, they are flexible (parameters)

The while loop

  • The condition True
  • The condition in a variable
  • The condition in a if
  • Break
  • Pattern counter inside a loop

Building a Menu for user selection in text mode

  • Input
  • Validation

Questions

  • As part of the questions, a question about the mental process to Build the solution was raised.
    • TODO: Was explained
    • The importance of keeping Notes with snippets of code that you normally use one time and another
    • How to search in Google to find explanations in Python sites
  • A question was raised about how menus could be implemented using OOP
    • A parent Menu class, with a MenuAdmin class inheriting was demonstrated. The MenuAdmin inherits the menu options from the parent Menu, and adds Admin only options.

News from the Blog 2021-11-11

New Articles

How to communicate with your Python program running inside a Docker Container, using Linux Signals

Hope you’ll have fun reading this article:

Communicating with Docker Containers via Linux Signals and Python

I migrated my last services from Amazon and the blog to Google Compute Engine (GCE / GCP)

I wrote a Postmortem analysis about the process of migrating my last services from my 11 year old Amazon account.

Updates

Updates to articles

I updated the article about Python weird things that you may not know adding the Ellipsis …

I’ve been working in some Cassandra examples. I may publish an article soon about using it from Python and Docker.

Updates to My Books

I updated my Python and Docker books.

I’m currently writing a book about using Amazon AWS Python SDK (boto3).

Updates to Open Source projects

I have updated ctop, fixed two bugs and increased Code Coverage.

I made a new tag and released the last Stable Version:

https://gitlab.com/carles.mateo/ctop/-/tags/0.8.7

On top of my local Unit Testing, I have Jenkins checking that I don’t commit anything that breaks the Tests.

Some time ago I wrote some articles about how you can setup jenkins in a Docker Container.

Miscellaneous

Charity

I’ve donated to Wikipedia.

Only 2% of the viewers donate, so I answered the call every time it was made.

This is my 5th donation to Wikimedia.

I consider that Freedom is very important.

I bought these new books

One of my secrets to be on top is that I’m always studying.

I study all the time, at work and in my free time.

I use Linux Academy and I buy books in paper. I don’t connect with reading in tablets. I think information is stored better when read in paper. I use also a marker and pointers to keep a direct access to the most interesting points on the books.

And I study all kind of themes. Obviously I know a lot of Web Scraping, but there is always room for learning more. And whatever new I learn helps me to be better with my students and more clear writing my books.

I’ve never been a Front End, but I’ve been able to fix bugs in the Front End engines from the companies I worked for, like Privalia. I was passed a bug that prevented the Internet Explorer users to buy just one hour before we launching a massive campaign. I debugged and I found a variable named “value” so the html looked like <input name="value" value="">. In less than 30 minutes I proved to the incredulous Head of Development and the CTO that a bug in Internet Explored was causing a conflict when fetching the value from the input named value. We deployed to Production the update and the campaign was a total success. So I consider knowing Javascript and Front also a need, even if I don’t work directly with it. I want to be able to understand all the requirements and possibilities, and weaknesses, so I can fix bugs and save the day. That allowed me to fix scalability problems in Nodejs and Phantomjs projects too. (They are Javascript Server Side, event driven, projects)

It seems that Amazon.co.uk works well again for Ireland. My two last orders arrived on time and I had no problems of border taxes apparently.

Nice Python article

I enjoyed a lot this article, cause explains part of what I did with my student and friend Albert, in a project that analyzes the access logs from Apache for patterns of attempts of exploits, then feeds a database, and then blocks those offender Ip Addresses in the Firewall.

The article only covers the part of Pandas, of reading the access.log file and working with it, but is a very well redacted article:

https://mmas.github.io/read-apache-access-log-pandas

Nice Virtual Volumes article from VMware

I prefer Open Source, but there are very good commercial products too.

I liked this article about Virtual Volumes from VMWare:

Understanding Virtual Volumes (vVols) in VMware vSphere 6.7/7.0 (2113013)

https://kb.vmware.com/s/article/2113013

Thanks Blizzard (again)

There is a very nice initiative where we can nominate 4 colleagues a year, that we think that deserve a recognition.

My colleagues voted for me, so I received a gift voucher that I can spend in Ireland stores like Ikea, Pc World, Argos, Adidas, App Store & iTunes…

So thanks a million buds. :)

Migrating my 11 years Amazon AWS account services (Postmortem Analysis)

I started to explain that I was migrating some services from Amazon and that some of my sites were under Maintenance and that I would provide more information.

Here is the complete history of why I migrated all the services from my 11 years old Amazon account to other CSP.

Some lessons can be learned from my adventure.

I migrated my last services from Amazon to GCP

Amazon sent me an email on October 6th, this year 2021, telling me that they will disable EC2-Classic by August 2022. I thought I would not be able to keep my Static Ip’s as in the past VPC Ip’s and EC2-Classic Ip’s were not transferable, so considering that I would loss my Static Ip’s anyway I started to migrate to some to other providers like Digital Ocean.

Is not cool losing Static Ip (Elastic Ip in AWS) Addresses as this is bad for SEO, so given that I though I would lose my Static Ips that have been with me for years, I started to migrate certain services to providers much more economic.

Amazon is terrible communicating, and I talked with some product managers in the past about that, when they lost one of my Volumes, and the email was so cold and terrible that actually that hurt more than Amazon losing my Data. I believed that it was a poorly made Scam and when I realized it was true I reached one of my friends, that is manager there, as I know they care for doing things right, and he organized a meeting with two PM so I can pass my feedback.

The Cloud providers are changing things very fast, and nobody is able to be up to date with the changes, unless their work position allows plenty of time to get updated. Even if pages of documentation are provided, you have to react to an event that they externally generated forcing you to action. Action to read all the documentation about EC2-Classic migrations, action to prepare to have migrated by August 2022.

So August 2022… I was counting that I had plenty of time but I’m writing a new book about using the Amazon SDK for Python, boto3, and I was doing some API calls and they started to fail in a very unusual way, Exceptions with timeout, but only for the only region where I had EC2-Classic.

urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x7f0347d545e0>: Failed to establish a new connection: [Errno -2] Name or service not known

My config was:

        o_config = Config(
            region_name="us-east-1a",
            signature_version="v4",
            retries={
                'max_attempts': 10,
                'mode': 'standard'
            }
        )

But if I switched to another region name, it would work:

            region_name='us-west-2',

I made a mistake in here, the region name is “us-east-1” and not “us-east-1a“. “us-east-1a” is the availability zone. So the SDK was giving a timeout because in order to connect to the endpoint it uses the region name as part of the hostname. So it doesn’t find that endpoint because it doesn’t exist.

I never understood why a company like Amazon is unable to provide the SDK with a sample project or projects 100% working, with the source code so people has a base that works to build up.

Every API that I have created, I have provided it with documentation but also with example for several languages for how to use it.

In 2013 I was CTO of an online travel agency, and we had meta-searchers consuming our API and we were having several hundreds of thousands requests per second. Everything was perfectly documented, examples were provided for several languages, the document and the SDK had version numbers…

Everybody forgets about Developers and companies throw terrible and cold products to the poor Developers, so difficult to use. How many Developers would like to say: Listen Mr. President of the big Cloud Company XXXX, I only want to spawn a VM that works, and fast, with easy wizards. I don’t want to learn 50 hours before being able to use your overpriced platform, by doing 20 things before your Ip’s are reflexes of your infrastructure and based in Microservices. Modern JavaScript frameworks can create nice gently wizards even if you have supercold APIs.

Honestly, I didn’t realize my typo in the region and I connected to the Amazon Console to investigate and I saw this.

Honestly, when I read it I understood that they were going to end my EC2 Networking the 30th of October. It was 29th. I misunderstood.

It was my fault not reading it well to the end, I got shocked by the first part telling about shutdown and I didn’t fully understood as they were going to shutdown EC2-Classic for the zones I didn’t had anything running only.

From the long errors (3 exceptions chained) I didn’t realize that the endpoint is built with the region name. (And I was passing the availability zone)

botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://ec2.us-east-1a.amazonaws.com/"

Here is when I say that a good SDM would had thought and cared for the Developers more, and would had made the SDK to check if that region exists. How difficult is to create a SDK a bit more clever that detects a invalid region id?. It is not difficult.

It is true that it was late in the evening and I was tired of all the day, and two days of the week between work and zoom university classes I work 15 hours and 13 hours respectively, not counting the assignments, so by the end of the week I am very tired. But that’s why it is very important to follow methodology and to read well. I think Amazon has 50% of the fault by the way they do things: how the created the SDK, how they communicate, and by the errors that the console returned me when I tried to create a VPC instance of an EC2-Classic AMI (they seem related to the fact I had old VPC Network objects with shorter hash than the current they use) and the other 50% was my fault for not identifying the source of the error, and not reading the message in their website well.

But the fact that there were having those errors in the API’s and timeouts made me believe they were going to cut the EC2-Classic Networking the next day.

All the mistakes fall together in a perfect storm.

I checked for documentation and I saw it was possible to migrate my Static Ip’s to VPC Static Ip’s.

It was Friday evening, and I cancelled my plans, in order to migrate the Blog to VPC in an attempt to keep running it with Amazon.

As Cloud Architect, I like to have running instances in several CSP as it allows me to stay up to date with the changes they do.

I checked the documentation for the migration. Disassociating the Static Ip (Elastic Ip in AWS jargon) was easy. Turning into VPC as well.

As I progressed, what had to be easy turned into a nightmare, as I was getting many errors from the Amazon API, without any information, and my Instances were not created.

I figured out that their API could have problems with old VPC objects I created time ago, so I had to create new objects for several things.

I managed to spawn my instances but they were being launch and terminated instantly without information. Frustrating.

When launching a new instance from the AMI (a Snapshot of the blog), I was giving shown options to add more volumes without any sense. My Instance was using 16GB from a 20GB total Space, and I was shown different volume configs, depending on the instance, in some case an additional 20GB volume, in other small SSD, ephemeral and 10 GB for the AMI (which requires at least 16GB).

After some fight I manage to make it work after deleting the volumes that made no sense, and keeping only one of 20GB, the same size of my AMI.

But then my nightmare started to make the VPC Instance to have Internet access and to be seen from outside. I had to create a new Internet Gateway, NAT, Network, etc…

As mentioned the old objects I was trying to reusing were making the process to fail.

I was running out of time, and I thought in few time they were going to shutdown EC2-Classic network (as I did not read correctly), so I decided to download everything and to migrate to another provider. For doing that first I blocked all the traffic, except for my Ip.

I worked in parallel, creating the new config in Google Cloud, just in case I had forgot something. I had created a document for the migration and it was accurate.

I managed to do everything fast enough. The slower part was to download all the Data, as I hold entire VM’s for projects like Cassandra Universal Driver.

Then I powered off my Amazon Instance for the Blog forever.

In GCP I blocked all the traffic in the firewall, except for my Ip, so I could work calmly.

When everything was ready, I had to redirect the DNS to the new static Ip from Google.

The DNS provider I used had implemented some changes in their API so I was getting errors replacing my old entry ‘.’ (their JSON calls returned Internal Server Error). Finally I figured it out how to workaround it and I was able to confirm that the first service was up and running.

I did some tests to make sure there were not unexpected permission problems, entries in the logs, etc…

Only then I opened the Google Firewall. I have a second firewall in each instance where I block or open at Ip tables level what I want. Basically abusive bot’s IPs trying to find exploits or brute force by dictionary passwords.

I checked with my phone, without Wifi that the Firewall was all good. (It is always a good idea to use another external Ip, different from the management one, to check)

I added a post explaining that I was migrating some of my Services and were under maintenance.

I mentioned in the blog that some of my services were being migrated from Amazon to Digital Ocean.

For some reasons, in the Backup of the Database one user was lost, so I created it in the MySQL with the typical commands:

CREATE USER 'username'@'localhost' IDENTIFIED BY 'password';
FLUSH PRIVILEGES;
GRANT ALL PRIVILEGES ON mydatabase.* TO 'username'@'localhost';

News from the blog 2021-10-21

  • I made a Donation to The Document Foundation, which makes the OpenOffice.

I use OpenOffice suite for writing my books and other documents, so I think it’s fair to contribute with their operating costs.

  • I’ve installed a plugin to add Code Highlighting

It also allows me to add blocks of Code, like this:

if CodeHighlighting.b_is_installed == True:
    return VisualImprovement.update_to(10), "It's easy to read"
else:
    return VisualImprovement.get_still_the_same_difficult_to_read(), "The blog lives in the medieval age"

Or Inline Code like print(self.awareness) which is also great

  • I’ve improved a bit, visually, the blog

I modified a bit my template. The changes consist into adding an id attribute to the table for the Quick Selection of the articles, and modifying my template: the styles in the file css/blocks.css and modifying the version in functions.php to reflect the new timestamp.

I also made that when the mouse goes over a link it is displayed in blue, and the already visited in a slightly darker blue.

#articles_selection a:hover {color: #2222FF;}

In the images below you can see the before, the intermediate, and the final.

I’ve also added a button to hide or show the Quick Selection

If you have a WordPress and jQuery does not work for you, with error:

TypeError: $ is not a function

$(document).ready(function(){

This is because for compatibility reasons you have to do different in WordPress:

jQuery(document).ready(function($){
  • I created several videos of 5 minutes to learn Unit Testing in Python 3 with pytest

I also use my package carleslibs to execute the command from shell.

Web CTOP in this case :)

As I did this I discovered a bug (bug #47) in CTOP for setting the number of rows.

  • I fixed the bug #47 and the bug #48 in CTOP and started version 0.8.7 (available in Master).

The changelog.txt file details all the changes for each version.

Here CTOP is displayed with a fixed width and height as by launching with:
ctop.py –rows=50 –columns=170
  • The new PSU arrived and I replaced it on Saturday 16th

After 5 days working nonstop, with no problems, it seems clear that the failing item was the expensive, 850W, Corsair PSU. Sometimes it happens that a new component comes defective, but I paid overprice expecting quality, and it seems that the PSU was defective. Since the beginning the computer powered off every few hours max, so I have to finally assume that effectively it was the PSU. Disappointed with Corsair.

  • Firewall. This month I’ve blocked around 2,000 visitors that were mainly bots searching for exploits

I review the logs several times every day.

Actually I’ve blocked many more Ip’s in the firewall, as when I identify a company source of bots, I block all their range (Imagine, as I block entire class C addresses, there are 256 Ips each class C /24). This has translated into 2,000 visitors less per month to the blog, that were offenders.

  • I added some rules / guidelines to the Leave a Reply section

I moderate all the comments to keep the blog an useful and healthy place.

And I don’t publish Spam, or Marketing messages.

Abusive comments are blocked. Competent Engineers and nice human beings share their points and doubts with data, with technical arguments, with education, in a respectful and polite way. People that cannot observe a minimum decoration are not welcome.

Web Top – Displaying top with Python 3 Web Server and Carleslibs

So this is a super simple example on how quickly you can create nice solutions with my package carleslibs.

In this example I use Python 3 incorporated Web Server and carleslibs, to execute top and display in the browser.

Requisites:

Having Python3 and have installed carleslibs 1.0.1 or superior.

pip3 install carleslibs

Having this running in a Linux with top installed. All of them come with top, as long as I know.

This is the 84 lines code for WebTop:

from http.server import BaseHTTPRequestHandler, HTTPServer

from carleslibs.subprocessutils import SubProcessUtils
from carleslibs.datetimeutils import DateTimeUtils


class Top():

    def __init__(self, o_subprocess):
        self.o_subprocess = o_subprocess

    def get_top(self):

        a_domains_offline = []

        s_command = "/usr/bin/top -n 1 -b"
        i_code, s_stdout, s_stderr = self.o_subprocess.execute_command_for_output(s_command, b_shell=True, b_convert_to_ascii=True)

        return i_code, s_stdout, s_stderr


class WebServer(BaseHTTPRequestHandler):

    def do_GET(self):
        o_subprocess = SubProcessUtils()
        self.o_top = Top(o_subprocess)

        self.o_datetime = DateTimeUtils()

        self.i_max_domains_offline = 0

        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()

        WebTop.log(self.path)
        if self.path == "/favicon.ico":
            return

        s_html = "<html><body>"
        s_html = s_html + "<h1>Web Top</h1>"
        s_html = s_html + '<small>by Carles Mateo - <a href="https://blog.carlesmateo.com">blog.carlesmateo.com</a></small>'

        i_code, s_stdout, s_stderr = self.o_top.get_top()

        if i_code != 0:
            s_html = s_html + "Error Code: " + str(i_code) + "&lt;/br&gt;"
            s_html = s_html + "Message: " + s_stderr + "&lt;/br&gt;"
        else:
            s_html = s_html + "<pre>"
            s_html = s_html + s_stdout
            s_html = s_html + "</pre>"

        s_html = s_html + "</body>"
        s_html = s_html + "</html>"

        by_html = bytes(s_html, encoding="utf-8")

        self.wfile.write(by_html)


class WebTop():

    o_datetime = DateTimeUtils()

    @staticmethod
    def log(s_text):
        s_datetime = WebTop.o_datetime.get_datetime()
        print(s_datetime, s_text)


if __name__ == "__main__":

    o_webserver = HTTPServer(("localhost", 80), WebServer)
    WebTop.log("Server started")

    try:
        o_webserver.serve_forever()
    except KeyboardInterrupt:
        pass

    o_webserver.server_close()
    WebTop.log("Server stopped")

Just run the code and go to localhost with your favorite browser.

If you get an error like this it means that another process is listening on port 80. Just use another like 8080, 8181, etc…

Traceback (most recent call last):
  File "/home/carles/Desktop/code/carles/json-realm-live/web_top.py", line 74, in <module>
    o_webserver = HTTPServer(("localhost", 80), WebServer)
  File "/usr/lib/python3.8/socketserver.py", line 452, in __init__
    self.server_bind()
  File "/usr/lib/python3.8/http/server.py", line 138, in server_bind
    socketserver.TCPServer.server_bind(self)
  File "/usr/lib/python3.8/socketserver.py", line 466, in server_bind
    self.socket.bind(self.server_address)
PermissionError: [Errno 13] Permission denied

And you will see the requests coming:

2021-10-09 10:47:46 Server started
127.0.0.1 - - [13/Oct/2021 10:47:48] "GET / HTTP/1.1" 200 -
2021-10-09 10:47:48 /
127.0.0.1 - - [13/Oct/2021 11:25:24] "GET / HTTP/1.1" 200 -
2021-10-09 11:25:24 /

So instead of using top, you can use ctop.py :)

Just replace the command by:

s_command = "ctop.py -b -n=1 --rows=30 --columns=200"

You can also create a Dockerfile very easily and run this in a Container.

Some graphics with matplotlib

Recently I showed you how to generate a Cloud Tag.

You may like some of other graphs that can be easily generated with matlib package.

I’ve been always working on BackEnd and APIs and I don’t work on FrontEnd, although I programmed some videogames by myself and I’ve fixed some huge bugs in JavaScript in some of the companies I work, but they considered myself the last resource, so I would fix a FrontEnd bug when nobody else could. But even if you work 99.9% of your time in BackEnd, Scaling, Architecture… like me, it is useful being able to draw graphics, for example, when you create a tool that shows the number of players per minute, and its evolution over time, or web visitors in real time, etc…

I wrote this article with two simple examples for my book Python 3 exercises for beginners.

You can find this source code here:

https://gitlab.com/carles.mateo/python-classes/-/blob/main/2021-09-10/draw_points.py


import matplotlib.pyplot as plt

a_points1 = [7, 3, 15, 5, 10, 2, 9]
a_points2 = [2, 4, 9, 2, 7, 8, 4]

plt.plot(a_points1)
plt.plot(a_points2)

plt.show()

We can also add customized axis:

https://gitlab.com/carles.mateo/python-classes/-/blob/main/2021-09-10/draw_points2.py

import matplotlib.pyplot as plt

a_points1 = [7, 3, 15, 5, 10, 2, 9]
a_points2 = [2, 4, 9, 2, 7, 8, 4]
a_points3 = [12, 10, 1, 7, 14, 16, 1]

a_days_of_the_week = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"]

plt.plot(a_days_of_the_week, a_points1)
plt.plot(a_days_of_the_week, a_points2)
plt.plot(a_days_of_the_week, a_points3)
plt.grid(axis='y', color='black', linestyle='solid')
plt.show()

Draw a pie chart

import matplotlib.pyplot as plt

a_scores = [70, 20, 5, 5]
a_languages = ["Python", "Bash", "Java", "PHP"]
a_colors = ["Red", "Blue", "Green", "Cyan"]

plt.pie(a_scores, labels=a_languages, colors=a_colors)
plt.legend()
plt.show()

This graphic represents in which languages I use my time nowadays, or if I update it by adding HTML and jQuery:

https://gitlab.com/carles.mateo/python-classes/-/blob/main/2021-09-10/draw_circle.py

Some weird things from Python 3 that you may not know

Last Update: 2021-10-31 16:41 IST

You can find those bizarre things and more in my book Python 3 Combat Guide.

I’m not talking about the wonderful things, like how big can the Integers be, but about the bizarre things that may ruin your day.

What sums 0.1 + 0.1 + 0.1 in Python?

0.3?

Wrong answer.

A bit of humor

Well, to be honest the computer was wrong. They way programming languages handle the Floats tend to be less than ideal.

Floats

Maybe you know JavaScript and its famous NaN (Not a number).

You are probably sure that Python is much more exact than that…

…well, until you do a big operation with Floats, like:

10.0**304 * 10.0**50 

and

It returns infinite

I see your infinite and I add one :)

However If we try to define a number too big directly it will return OverflowError:

Please note Integers are handled in a much more robust cooler way:

Negative floats

Ok. What happens if we define a number with a negative power, like 10 ** -300 ?

And if we go somewhere a bit more far? Like 10 ** -329

It returns 0.0

Ups!

I mention in my books why is better to work with Integers, and in fact most of the eCommerces, banks and APIs work with Integers. For example, if the amount in USD 10.00 they send multiplied by 100, so they will send 1000. All the actor know that they have to divide by 2.

Breaking the language innocently

I mentioned always that I use the MT Notation, the prefix notation I invented, inspired by the Hungarian Notation and by an amazing C++ programmer I worked with in Volkswagen and in la caixa (now caixabank), that passed away many years ago.

Well, that system of prefixes will name a variable with a prefix for its type.

It’s very useful and also prevents the next weird thing from Python.

Imagine a Junior wants to print a String and they put in a variable. And unfortunately they call this variable print. Well…

print = "Hello World!"
print("That will hurt")

Observe the output of this and try not to scream:

Variables and Functions named equally

Well, most of languages are able to differentiate a function, with its parenthesis, from a variable.

The way Python does it hurts my coder heart:

Another good reason to use MT Notation for the variables, and for taking seriously doing Unit Testing and giving a chance to using getters and setters and class Constructor for implementing limits and sanitation.

Nested Loops

This will work in Python, it doesn’t work in other languages (but please never do it).

for i in range(3):
    print("First Loop", i)
    for i in range(4):
        print("Second Loop", i)

The code will not crash by overwriting i used in the first loop, but the new i will mask the first variable.

And please, name variables properly.

Import… once?

Imports are imported only once. Even if different files imported do import the same file.

So don’t have code in the middle of them, outside functions/classes, unless you’re really know what you’re doing.

Define functions first, and execute code after if __name__ == “__main__”:

Take a look at this code:

def first_function():
    print("Inside first function")
    second_function()

first_function()

def second_function():
    print("Inside second function")

Well, this will crash as Python executes the code from top to bottom, and when it gets to first_function() it will attempt to call second_function() which has not been read by Python yet. This example will throw an error.

You’ll get an error like:

Inside first function
Traceback (most recent call last):
  File "/home/carles/Desktop/code/carles/python_combat_guide/src/structure_dont_do_this.py", line 14, in <module>
    first_function()
  File "/home/carles/Desktop/code/carles/python_combat_guide/src/structure_dont_do_this.py", line 12, in first_function
    second_function()
NameError: name 'second_function' is not defined

Process finished with exit code 1

Add your code at the bottom always, under:

if __name__ == "__main__":
    main()

The code inside this if will only be executed if you directly call this code as main file, but will not be executed if you import this file from another one.

You don’t have this problem with classes in Python, as they are defined first, completely read, and then you instantiate or use them. To avoid messing and creating bugs, have the imports always on the top of your file.

…Ellipsis

Today is Halloween and one of my colleagues asked me help to improve his Automation project.

I found something weird in his code.

He had something like that.

class Router:

    def router_get_info(self):
        ...

    def get_help_command(self):
        return "help"

So I asked why you use … (dot dot dot) on that empty method?.

He told me that when he don’t want to implement code he just put that.

Well, dot dot dot is Ellipsis.

And what is Ellipsis?.

Ellipsis is an object that may appear in slice notation.

A good explanation of what is Ellipsis is found in this answer in StackOverflow.

In Python all the methods, functions, if, while …. require to have an instruction at least.

So the instruction my colleague was looking for is pass.

Just a variable?

In Python you can have just a var, without anything else, like no operation with it, no call, nothing.

This makes it easy to commit an error and not detecting it.

As you see we can have just s_var variable in a line, which is a String, and this does not raises an error.

If we do from python interpreter interactively, it will print the String “I’m a pickle” (famous phrase from Rick and Morty).

Generating a Word Cloud of Tags in Python

This is a very simple code but generates very cool Word Cloud result in PNG format.

from wordcloud import WordCloud

# Add your tags in here separated by commas and as many times as they appear
s_text = "Python, Software development, PHP, Cloud providers, Python, Python, Software development, Scaling"

o_word_cloud = WordCloud(height=800,
                         width=800,
                         background_color="white",
                         max_words=150,
                         min_font_size=5,
                         collocation_threshold=10).generate_from_text(s_text)

o_word_cloud.to_file("words.png")

That version generated the image .PNG file.

If you want to display this in mathlib or inside PyCharm embedded view, you can install matplotlib with:

pip3 install matplotlib

Then with this code you can display a matplotlib viewer:

import matplotlib.pyplot as plt
from wordcloud import WordCloud

if __name__ == "__main__":
    # Add your tags in here separated by commas and as many times as they appear
    s_text = "Python, Software development, PHP, Cloud providers, Python, Python, Software development, Scaling"

    o_word_cloud = WordCloud(height=800,
                             width=800,
                             background_color="white",
                             max_words=150,
                             min_font_size=5,
                             collocation_threshold=10).generate_from_text(s_text)

    plt.figure(figsize=(10,8))
    plt.imshow(o_word_cloud)
    plt.axis("off")
    plt.tight_layout(pad=0)
    plt.show()

Python Game Tic Tac Toe

I implemented this very simple game for my book Python 3 Exercises for Beginners.

Source Code available here:

https://gitlab.com/carles.mateo/python-classes/-/blob/main/2021-09-10/game_tic-tac-toe.py


class TicTacToe:

    def __init__(self):
        self.a_a_s_map = []
        self.generate_map()

    def generate_map(self):
        self.a_a_s_map = []

        for i_y in range(3):
            a_s_pos_x = [" ", " ", " "]
            self.a_a_s_map.append(a_s_pos_x)

    def get_map(self):
        s_map = ""

        s_map = s_map + "    1   2   3\n"
        s_map = s_map + "  -------------\n"
        for i_y in range(3):
            s_map = s_map + str(i_y + 1) + " |"
            for s_char in self.a_a_s_map[i_y]:
                s_map = s_map + " " + s_char + " |"
            s_map = s_map + "\n"
            s_map = s_map + "  -------------\n"

        return s_map

    def validate_move(self, s_char, i_x, i_y):
        """
        Validates the movement and updates the map
        :param s_char:
        :param i_x:
        :param i_y:
        :return: bool
        """
        i_x = i_x - 1
        i_y = i_y - 1

        if self.a_a_s_map[i_y][i_x] == " ":
            self.a_a_s_map[i_y][i_x] = s_char
            return True

        return False

    def check_win(self):
        for s_char in ["O", "X"]:

            # check horizontal
            for i_y in range(3):
                i_horizontal_match = 0
                for i_x in range(3):
                    if self.a_a_s_map[i_y][i_x] == s_char:
                        i_horizontal_match = i_horizontal_match + 1
                if i_horizontal_match == 3:
                    return True

            # Check vertical
            for i_x in range(3):
                i_vertical_match = 0
                for i_y in range(3):
                    if self.a_a_s_map[i_y][i_x] == s_char:
                        i_vertical_match = i_vertical_match + 1
                if i_vertical_match == 3:
                    return True

            # Check diagonal
            if self.a_a_s_map[1][1] == s_char:
                if self.a_a_s_map[0][0] == s_char and self.a_a_s_map[2][2] == s_char:
                    return True
                if self.a_a_s_map[0][2] == s_char and self.a_a_s_map[2][0] == s_char:
                    return True

        return False

    def check_stale(self):
        for i_y in range(3):
            for i_x in range(3):
                if self.a_a_s_map[i_y][i_x] == " ":
                    # Is not full
                    return False

        # We checked all and all were full
        return True


def get_from_keyboard(s_question, i_min, i_max):
    i_number = 0
    while True:
        s_answer = input(s_question)
        try:
            i_number = int(s_answer)
        except:
            print("Please, type a number")
            continue

        if i_number < i_min or i_number > i_max:
            print("Invalid value. Values should be between", i_min, "and", i_max)
            continue

        # Validations are Ok
        break

    return i_number


if __name__ == "__main__":
    o_tictactoe = TicTacToe()

    while True:

        s_map = o_tictactoe.get_map()
        print(s_map)

        while True:
            i_x = get_from_keyboard("Your move O for x: ", i_min=1, i_max=3)
            i_y = get_from_keyboard("Your move O for y: ", i_min=1, i_max=3)

            b_valid_move = o_tictactoe.validate_move("O", i_x, i_y)
            if b_valid_move is False:
                print("Invalid move")
                continue

            break

        s_map = o_tictactoe.get_map()
        print(s_map)
        b_check_win = o_tictactoe.check_win()
        if b_check_win is True:
            print("Player O wins!")
            exit(0)

        b_stale = o_tictactoe.check_stale()
        if b_stale is True:
            print("Nobody wins in war")
            exit(0)

        while True:
            i_x = get_from_keyboard("Your move X for x: ", i_min=1, i_max=3)
            i_y = get_from_keyboard("Your move X for y: ", i_min=1, i_max=3)

            b_valid_move = o_tictactoe.validate_move("X", i_x, i_y)
            if b_valid_move is False:
                print("Invalid move")
                continue

            break

        s_map = o_tictactoe.get_map()
        print(s_map)
        b_check_win = o_tictactoe.check_win()
        if b_check_win is True:
            print("Player X wins!")
            exit(0)