Category Archives: Software development

Web Top – Displaying top with Python 3 Web Server and Carleslibs

So this is a super simple example on how quickly you can create nice solutions with my package carleslibs.

In this example I use Python 3 incorporated Web Server and carleslibs, to execute top and display in the browser.

Requisites:

Having Python3 and have installed carleslibs 1.0.1 or superior.

pip3 install carleslibs

Having this running in a Linux with top installed. All of them come with top, as long as I know.

This is the 84 lines code for WebTop:

from http.server import BaseHTTPRequestHandler, HTTPServer

from carleslibs.subprocessutils import SubProcessUtils
from carleslibs.datetimeutils import DateTimeUtils


class Top():

    def __init__(self, o_subprocess):
        self.o_subprocess = o_subprocess

    def get_top(self):

        a_domains_offline = []

        s_command = "/usr/bin/top -n 1 -b"
        i_code, s_stdout, s_stderr = self.o_subprocess.execute_command_for_output(s_command, b_shell=True, b_convert_to_ascii=True)

        return i_code, s_stdout, s_stderr


class WebServer(BaseHTTPRequestHandler):

    def do_GET(self):
        o_subprocess = SubProcessUtils()
        self.o_top = Top(o_subprocess)

        self.o_datetime = DateTimeUtils()

        self.i_max_domains_offline = 0

        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()

        WebTop.log(self.path)
        if self.path == "/favicon.ico":
            return

        s_html = "<html><body>"
        s_html = s_html + "<h1>Web Top</h1>"
        s_html = s_html + '<small>by Carles Mateo - <a href="https://blog.carlesmateo.com">blog.carlesmateo.com</a></small>'

        i_code, s_stdout, s_stderr = self.o_top.get_top()

        if i_code != 0:
            s_html = s_html + "Error Code: " + str(i_code) + "&lt;/br&gt;"
            s_html = s_html + "Message: " + s_stderr + "&lt;/br&gt;"
        else:
            s_html = s_html + "<pre>"
            s_html = s_html + s_stdout
            s_html = s_html + "</pre>"

        s_html = s_html + "</body>"
        s_html = s_html + "</html>"

        by_html = bytes(s_html, encoding="utf-8")

        self.wfile.write(by_html)


class WebTop():

    o_datetime = DateTimeUtils()

    @staticmethod
    def log(s_text):
        s_datetime = WebTop.o_datetime.get_datetime()
        print(s_datetime, s_text)


if __name__ == "__main__":

    o_webserver = HTTPServer(("localhost", 80), WebServer)
    WebTop.log("Server started")

    try:
        o_webserver.serve_forever()
    except KeyboardInterrupt:
        pass

    o_webserver.server_close()
    WebTop.log("Server stopped")

Just run the code and go to localhost with your favorite browser.

If you get an error like this it means that another process is listening on port 80. Just use another like 8080, 8181, etc…

Traceback (most recent call last):
  File "/home/carles/Desktop/code/carles/json-realm-live/web_top.py", line 74, in <module>
    o_webserver = HTTPServer(("localhost", 80), WebServer)
  File "/usr/lib/python3.8/socketserver.py", line 452, in __init__
    self.server_bind()
  File "/usr/lib/python3.8/http/server.py", line 138, in server_bind
    socketserver.TCPServer.server_bind(self)
  File "/usr/lib/python3.8/socketserver.py", line 466, in server_bind
    self.socket.bind(self.server_address)
PermissionError: [Errno 13] Permission denied

And you will see the requests coming:

2021-10-09 10:47:46 Server started
127.0.0.1 - - [13/Oct/2021 10:47:48] "GET / HTTP/1.1" 200 -
2021-10-09 10:47:48 /
127.0.0.1 - - [13/Oct/2021 11:25:24] "GET / HTTP/1.1" 200 -
2021-10-09 11:25:24 /

So instead of using top, you can use ctop.py :)

Just replace the command by:

s_command = "ctop.py -b -n=1 --rows=30 --columns=200"

You can also create a Dockerfile very easily and run this in a Container.

Some graphics with matplotlib

Recently I showed you how to generate a Cloud Tag.

You may like some of other graphs that can be easily generated with matlib package.

I’ve been always working on BackEnd and APIs and I don’t work on FrontEnd, although I programmed some videogames by myself and I’ve fixed some huge bugs in JavaScript in some of the companies I work, but they considered myself the last resource, so I would fix a FrontEnd bug when nobody else could. But even if you work 99.9% of your time in BackEnd, Scaling, Architecture… like me, it is useful being able to draw graphics, for example, when you create a tool that shows the number of players per minute, and its evolution over time, or web visitors in real time, etc…

I wrote this article with two simple examples for my book Python 3 exercises for beginners.

You can find this source code here:

https://gitlab.com/carles.mateo/python-classes/-/blob/main/2021-09-10/draw_points.py


import matplotlib.pyplot as plt

a_points1 = [7, 3, 15, 5, 10, 2, 9]
a_points2 = [2, 4, 9, 2, 7, 8, 4]

plt.plot(a_points1)
plt.plot(a_points2)

plt.show()

We can also add customized axis:

https://gitlab.com/carles.mateo/python-classes/-/blob/main/2021-09-10/draw_points2.py

import matplotlib.pyplot as plt

a_points1 = [7, 3, 15, 5, 10, 2, 9]
a_points2 = [2, 4, 9, 2, 7, 8, 4]
a_points3 = [12, 10, 1, 7, 14, 16, 1]

a_days_of_the_week = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"]

plt.plot(a_days_of_the_week, a_points1)
plt.plot(a_days_of_the_week, a_points2)
plt.plot(a_days_of_the_week, a_points3)
plt.grid(axis='y', color='black', linestyle='solid')
plt.show()

Draw a pie chart

import matplotlib.pyplot as plt

a_scores = [70, 20, 5, 5]
a_languages = ["Python", "Bash", "Java", "PHP"]
a_colors = ["Red", "Blue", "Green", "Cyan"]

plt.pie(a_scores, labels=a_languages, colors=a_colors)
plt.legend()
plt.show()

This graphic represents in which languages I use my time nowadays, or if I update it by adding HTML and jQuery:

https://gitlab.com/carles.mateo/python-classes/-/blob/main/2021-09-10/draw_circle.py

Some weird things from Python 3 that you may not know

Last Update: 2021-09-29 18:12 IST

You can find those bizarre things and more in my book Python 3 Combat Guide.

I’m not talking about the wonderful things, like how big can the Integers be, but about the bizarre things that may ruin your day.

What sums 0.1 + 0.1 + 0.1 in Python?

0.3?

Wrong answer.

A bit of humor

Well, to be honest the computer was wrong. They way programming languages handle the Floats tend to be less than ideal.

Floats

Maybe you know JavaScript and its famous NaN (Not a number).

You are probably sure that Python is much more exact than that…

…well, until you do a big operation with Floats, like:

10.0**304 * 10.0**50 

and

It returns infinite

I see your infinite and I add one :)

However If we try to define a number too big directly it will return OverflowError:

Please note Integers are handled in a much more robust cooler way:

Negative floats

Ok. What happens if we define a number with a negative power, like 10 ** -300 ?

And if we go somewhere a bit more far? Like 10 ** -329

It returns 0.0

Ups!

I mention in my books why is better to work with Integers, and in fact most of the eCommerces, banks and APIs work with Integers. For example, if the amount in USD 10.00 they send multiplied by 100, so they will send 1000. All the actor know that they have to divide by 2.

Breaking the language innocently

I mentioned always that I use the MT Notation, the prefix notation I invented, inspired by the Hungarian Notation and by an amazing C++ programmer I worked with in Volkswagen and in la caixa (now caixabank), that passed away many years ago.

Well, that system of prefixes will name a variable with a prefix for its type.

It’s very useful and also prevents the next weird thing from Python.

Imagine a Junior wants to print a String and they put in a variable. And unfortunately they call this variable print. Well…

print = "Hello World!"
print("That will hurt")

Observe the output of this and try not to scream:

Variables and Functions named equally

Well, most of languages are able to differentiate a function, with its parenthesis, from a variable.

The way Python does it hurts my coder heart:

Another good reason to use MT Notation for the variables, and for taking seriously doing Unit Testing and giving a chance to using getters and setters and class Constructor for implementing limits and sanitation.

Nested Loops

This will work in Python, it doesn’t work in other languages (but please never do it).

for i in range(3):
    print("First Loop", i)
    for i in range(4):
        print("Second Loop", i)

The code will not crash by overwriting i used in the first loop, but the new i will mask the first variable.

And please, name variables properly.

Import… once?

Imports are imported only once. Even if different files imported do import the same file.

So don’t have code in the middle of them, outside functions/classes, unless you’re really know what you’re doing.

Define functions first, and execute code after if __name__ == “__main__”:

Take a look at this code:

def first_function():
    print("Inside first function")
    second_function()

first_function()

def second_function():
    print("Inside second function")

Well, this will crash as Python executes the code from top to bottom, and when it gets to first_function() it will attempt to call second_function() which has not been read by Python yet. This example will throw an error.

You’ll get an error like:

Inside first function
Traceback (most recent call last):
  File "/home/carles/Desktop/code/carles/python_combat_guide/src/structure_dont_do_this.py", line 14, in <module>
    first_function()
  File "/home/carles/Desktop/code/carles/python_combat_guide/src/structure_dont_do_this.py", line 12, in first_function
    second_function()
NameError: name 'second_function' is not defined

Process finished with exit code 1

Add your code at the bottom always, under:

if __name__ == "__main__":
    main()

The code inside this if will only be executed if you directly call this code as main file, but will not be executed if you import this file from another one.

You don’t have this problem with classes in Python, as they are defined first, completely read, and then you instantiate or use them. To avoid messing and creating bugs, have the imports always on the top of your file.

Generating a Word Cloud of Tags in Python

This is a very simple code but generates very cool Word Cloud result in PNG format.

from wordcloud import WordCloud

# Add your tags in here separated by commas and as many times as they appear
s_text = "Python, Software development, PHP, Cloud providers, Python, Python, Software development, Scaling"

o_word_cloud = WordCloud(height=800,
                         width=800,
                         background_color="white",
                         max_words=150,
                         min_font_size=5,
                         collocation_threshold=10).generate_from_text(s_text)

o_word_cloud.to_file("words.png")

That version generated the image .PNG file.

If you want to display this in mathlib or inside PyCharm embedded view, you can install matplotlib with:

pip3 install matplotlib

Then with this code you can display a matplotlib viewer:

import matplotlib.pyplot as plt
from wordcloud import WordCloud

if __name__ == "__main__":
    # Add your tags in here separated by commas and as many times as they appear
    s_text = "Python, Software development, PHP, Cloud providers, Python, Python, Software development, Scaling"

    o_word_cloud = WordCloud(height=800,
                             width=800,
                             background_color="white",
                             max_words=150,
                             min_font_size=5,
                             collocation_threshold=10).generate_from_text(s_text)

    plt.figure(figsize=(10,8))
    plt.imshow(o_word_cloud)
    plt.axis("off")
    plt.tight_layout(pad=0)
    plt.show()

News from the blog 2021-09-20

  • I’ve published a very simple game, Tic Tac Toe, that I created for my Python 3 Exercises for Beginners book.
  • I’ve raised back the price for my books to normal levels.
    I’ve been keeping the price to the minimum to help people that wanted to learn during covid-19. I consider that who wanted to learn has already done it.

I still have bundles with a somewhat reduced price, and I authorized LeanPub platform to do discounts up to 50% at their discretion.

Bundle of four books in https://leanpub.com/b/python3-exercises-zfs-assemble-computer

https://leanpub.com/b/python3-exercises-zfs-assemble-computer

  • I’ve been deleting AMIs, Snapshots, Volumes and backups from Amazon instances I’ll no longer use.

I’ve migrated to Docker some sites and WordPress sites and now I’m CSP (Cloud Service Provider) agnostic. I can deploy wherever I want.

We pay per GB used of storage, so my money will get a better usage.

As I said in my old article from 2013, The Cloud is for Scaling. For Startups and for Enterprises. It is too expensive for small and medium companies.

  • For those studying Python there is a Virtual Meetup about Data Analysis, in Spanish ,the 23th of September

https://www.meetup.com/tech-barcelona/events/280791310/

More meetups:

https://www.meetup.com/tech-barcelona/

Python Game Tic Tac Toe

I implemented this very simple game for my book Python 3 Exercises for Beginners.

Source Code available here:

https://gitlab.com/carles.mateo/python-classes/-/blob/main/2021-09-10/game_tic-tac-toe.py


class TicTacToe:

    def __init__(self):
        self.a_a_s_map = []
        self.generate_map()

    def generate_map(self):
        self.a_a_s_map = []

        for i_y in range(3):
            a_s_pos_x = [" ", " ", " "]
            self.a_a_s_map.append(a_s_pos_x)

    def get_map(self):
        s_map = ""

        s_map = s_map + "    1   2   3\n"
        s_map = s_map + "  -------------\n"
        for i_y in range(3):
            s_map = s_map + str(i_y + 1) + " |"
            for s_char in self.a_a_s_map[i_y]:
                s_map = s_map + " " + s_char + " |"
            s_map = s_map + "\n"
            s_map = s_map + "  -------------\n"

        return s_map

    def validate_move(self, s_char, i_x, i_y):
        """
        Validates the movement and updates the map
        :param s_char:
        :param i_x:
        :param i_y:
        :return: bool
        """
        i_x = i_x - 1
        i_y = i_y - 1

        if self.a_a_s_map[i_y][i_x] == " ":
            self.a_a_s_map[i_y][i_x] = s_char
            return True

        return False

    def check_win(self):
        for s_char in ["O", "X"]:

            # check horizontal
            for i_y in range(3):
                i_horizontal_match = 0
                for i_x in range(3):
                    if self.a_a_s_map[i_y][i_x] == s_char:
                        i_horizontal_match = i_horizontal_match + 1
                if i_horizontal_match == 3:
                    return True

            # Check vertical
            for i_x in range(3):
                i_vertical_match = 0
                for i_y in range(3):
                    if self.a_a_s_map[i_y][i_x] == s_char:
                        i_vertical_match = i_vertical_match + 1
                if i_vertical_match == 3:
                    return True

            # Check diagonal
            if self.a_a_s_map[1][1] == s_char:
                if self.a_a_s_map[0][0] == s_char and self.a_a_s_map[2][2] == s_char:
                    return True
                if self.a_a_s_map[0][2] == s_char and self.a_a_s_map[2][0] == s_char:
                    return True

        return False

    def check_stale(self):
        for i_y in range(3):
            for i_x in range(3):
                if self.a_a_s_map[i_y][i_x] == " ":
                    # Is not full
                    return False

        # We checked all and all were full
        return True


def get_from_keyboard(s_question, i_min, i_max):
    i_number = 0
    while True:
        s_answer = input(s_question)
        try:
            i_number = int(s_answer)
        except:
            print("Please, type a number")
            continue

        if i_number < i_min or i_number > i_max:
            print("Invalid value. Values should be between", i_min, "and", i_max)
            continue

        # Validations are Ok
        break

    return i_number


if __name__ == "__main__":
    o_tictactoe = TicTacToe()

    while True:

        s_map = o_tictactoe.get_map()
        print(s_map)

        while True:
            i_x = get_from_keyboard("Your move O for x: ", i_min=1, i_max=3)
            i_y = get_from_keyboard("Your move O for y: ", i_min=1, i_max=3)

            b_valid_move = o_tictactoe.validate_move("O", i_x, i_y)
            if b_valid_move is False:
                print("Invalid move")
                continue

            break

        s_map = o_tictactoe.get_map()
        print(s_map)
        b_check_win = o_tictactoe.check_win()
        if b_check_win is True:
            print("Player O wins!")
            exit(0)

        b_stale = o_tictactoe.check_stale()
        if b_stale is True:
            print("Nobody wins in war")
            exit(0)

        while True:
            i_x = get_from_keyboard("Your move X for x: ", i_min=1, i_max=3)
            i_y = get_from_keyboard("Your move X for y: ", i_min=1, i_max=3)

            b_valid_move = o_tictactoe.validate_move("X", i_x, i_y)
            if b_valid_move is False:
                print("Invalid move")
                continue

            break

        s_map = o_tictactoe.get_map()
        print(s_map)
        b_check_win = o_tictactoe.check_win()
        if b_check_win is True:
            print("Player X wins!")
            exit(0)

News from the blog 2021-09-07

  • Blocking Ip addresses in the firewall

I’ve been blocking entire ranges from Cloud Providers, as some of their Ip’s were being used to try to hack/abuse the blog.

After some time blocking individual Ip’s, I opted for being some more effective and blocking /24 (the class C for the offending Ip).

If you work for a CSP and you can’t see my blog from your range, this is the reason.

  • I have updated my two Python books.

For Python 3 simple exercises for beginners I’ve added this new content:

v.0.29 and v.0.28 Python 3 Simple Exercises for Beginners
Added a new section Games, with a first game “Guess my number”

Provided a solution for the recipe Exercise 3: Create a function that will ask a user for a number from Keyboard input, and return the result, only when the value is between the accepted ranges.

Added two new questions to the Quiz section.

Fixed a docstring in Recipes Exercise 4, referring to a String return value which it was an Integer.

Added an exercise for retrieving a JSON with your public Ip

Added a new exercise for converting bytes to kilobytes with two decimal positions.

Added a new exercise / recipe to SSH to a Server with Username and Password and execute a command, using the Paramiko library.

For Python 3 Combat Guide I have added this new content:

v.1.02 Python 3 Combat Guide

Added a new exercise / recipe to SSH to a Server with Username and Password and execute a command using Paramiko library. I added two examples executing commands uptime and df -h /

Show an alternative way to run Flask apps.

Added new interesting packages.

As long as covid is active I plan to keep the minimum price of each of my books at the minimum accepted by LeanPub which is USD $5.

I also enable bundles and enable LeanPub to make punctual discounts to make them even more affordable to humble pockets.

  • I’m going to teach an initiation to programming class, a live Zoom session of 1 hour, plus 15 minutes for questions, for Free. It will be a very basic starting class for absolute beginners.

It will be performed at a time around 19:00 Irish time, wich is 11:00 AM in Irivine, CA time, to maximize the opportunities for people to assist.

If you would like to join, write me an email:

There are not many spots available, but if there is no room for you this time I may contact you for the next time.

This version adds class StringUtils, and a set of methods to perform different useful tasks with Strings, like converting a number to the biggest unit with sense, like a large number of bytes to PB, a smaller one to TB, or GB, MB, KB, justifying strings to the right or to the left, and cap the number of chars to a specified one, etc…

I have many libraries that I’ve been building across the years, and I’m liberating them as Open Source, as soon as I have time to make sure that are compatible with Python 3.5 or superior (and with Python 2.6 when possible), and I have time to add a decent Unit Testing Code Coverage.

I try to release libraries that have no other dependencies. After that I’ll start releasing my libs that have dependencies, like to work with MySQL, SQLite, web scrapping, etc…

Have a cheap Ubuntu in your Windows or Mac with Docker

I had this idea after one my Python and Linux students with two laptops, a Mac OS X and a Windows one explained me that the Mac OS X is often taken by their daughters, and that the Windows 10 laptop has not enough memory to run PyCharm and Virtual Box fluently. She wanted to have a Linux VM to practice Linux, and do the Bash exercises.

So this article explains how to create a Ubuntu 20.04 LTS Docker Container, and execute a shell were you can practice Linux, Ubuntu, Bash, and you can use it to run Python, Apache, PHP, MySQL… as well, if you want.

You need to install Docker for Windows of for Mac:

Docker for Windows is very handy and visual

Just pay attention to your type of processor: Mac with Intel chip or Mac with apple chip.

The first thing is to create the Dockerfile.

FROM ubuntu:20.04

MAINTAINER Carles Mateo

ARG DEBIAN_FRONTEND=noninteractive

RUN apt update && \
    apt install -y vim python3-pip &&  \
    apt install -y net-tools mc htop less strace zip gzip lynx && \
    pip3 install pytest && \
    apt-get clean

RUN echo "#!/bin/bash\nwhile [ true ]; do sleep 60; done" > /root/loop.sh; chmod +x /root/loop.sh

CMD ["/root/loop.sh"]

So basically the file named Dockerfile contains all the blueprints for our Docker Container to be created.

You see that I all the installs and clean ups in one single line. That’s because Docker generates a layer of virtual disk per each line in the Dockerfile. The layers are persistent, so even if in the next line we delete the temporary files, the space used will not be recovered.

You see also that I generate a Bash file with an infinite loop that sleeps 60 seconds each loop and save it as /root/loop.sh This is the file that later is called with CMD, so basically when the Container is created will execute this infinite loop. Basically we give to the Container a non ending task to prevent it from running, and exiting.

Now that you have the Dockerfile is time to build the Container.

For Mac open a terminal and type this command inside the directory where you have the Dockerfile file:

sudo docker build -t cheap_ubuntu .

I called the image cheap_ubuntu but you can set the name that you prefer.

For Windows 10 open a Command Prompt with Administrative rights and then change directory (cd) to the one that has your Dockerfile file.

docker.exe build -t cheap_ubuntu .
Image being built… (some data has been covered in white)

Now that you have the image built, you can create a Container based on it.

For Mac:

sudo docker run -d --name cheap_ubuntu cheap_ubuntu

For Windows (you can use docker.exe or just docker):

docker.exe run -d --name cheap_ubuntu cheap_ubuntu

Now you have Container named cheap_ubuntu based on the image cheap_ubuntu.

It’s time to execute an interactive shell and be able to play:

sudo docker exec -it cheap_ubuntu /bin/bash

For Windows:

docker.exe exec -it cheap_ubuntu /bin/bash
Our Ubuntu terminal inside Windows

Now you have an interactive shell, as root, to your cheap_ubuntu Ubuntu 20.04 LTS Container.

You’ll not be able to run the graphical interface, but you have a complete Ubuntu to learn to program in Bash and to use Linux from Command Line.

You will exit the interactive Bash session in the container with:

exit

If you want to stop the Container:

sudo docker stop cheap_ubuntu

Or for Windows:

docker.exe stop cheap_ubuntu

If you want to see what Containers are running do:

sudo docker ps

Released Python CarlesLibs version 0.99.2

I’ve updated my pypi carleslibs package to version 0.99.2.

The addition I made to this version is StringUtils class which offer functionalities for handling amount conversions (to different units), number formatting, string formatting and align (left, right…). I added a 85% of Unit Testing Code Coverage.

Here you have some general information about how to install and how to use the package:

https://blog.carlesmateo.com/carleslibs/

News of the blog 2021-08-16

  • I completed my ZFS on Ubuntu 20.04 LTS book.
    I had an error in an actual hard drive so I added a Troubleshooting section explaining how I fixed it.
  • I paused for a while the advance of my book Python: basic exercises for beginners, as my colleague Michela is translating it to Italian. She is a great Engineer and I cannot be more happy of having her help.
  • I added a new article about how to create a simple web Star Wars game using Flask.
    As always, I use Docker and a Dockerfile to automate the deployment, so you can test it without messing with your local system.
    The code is very simple and easy to understand.
mysql> UPDATE wp_options set option_value='blog.carlesmateo.local' WHERE option_name='siteurl';
Query OK, 1 row affected (0.02 sec)
Rows matched: 1 Changed: 1 Warnings: 0

This way I set an entry in /etc/hosts and I can do all the tests I want.

  • I added a new section to the blog, is a link where you can see all the articles published, ordered by number of views.
    /posts_and_views.php

Is in the main page, just after the recommended articles.
Here you can see the source code.

  • I removed the Categories:
    • Storage
      • ZFS
  • In favor of:
    • Hardware
      • Storage
        • ZFS
  • So the articles with Categories in the group deleted were reassigned the Categories in the second group.
  • Visually:
    • I removed some annoying lines from the Quick Selection access.
      They came from inherited CSS properties from my WordPress, long time customized, and I created new styles for this section.
    • I adjusted the line-height to avoid separation between lines being too much.
  • I added a link in the section of Other Engineering Blogs that I like, to the great https://github.com/lesterchan site, author of many super cool WordPress plugins.