Author Archives: Carles Mateo

Generating a Word Cloud of Tags in Python

This is a very simple code but generates very cool Word Cloud result in PNG format.

from wordcloud import WordCloud

# Add your tags in here separated by commas and as many times as they appear
s_text = "Python, Software development, PHP, Cloud providers, Python, Python, Software development, Scaling"

o_word_cloud = WordCloud(height=800,
                         width=800,
                         background_color="white",
                         max_words=150,
                         min_font_size=5,
                         collocation_threshold=10).generate_from_text(s_text)

o_word_cloud.to_file("words.png")

That version generated the image .PNG file.

If you want to display this in mathlib or inside PyCharm embedded view, you can install matplotlib with:

pip3 install matplotlib

Then with this code you can display a matplotlib viewer:

import matplotlib.pyplot as plt
from wordcloud import WordCloud

if __name__ == "__main__":
    # Add your tags in here separated by commas and as many times as they appear
    s_text = "Python, Software development, PHP, Cloud providers, Python, Python, Software development, Scaling"

    o_word_cloud = WordCloud(height=800,
                             width=800,
                             background_color="white",
                             max_words=150,
                             min_font_size=5,
                             collocation_threshold=10).generate_from_text(s_text)

    plt.figure(figsize=(10,8))
    plt.imshow(o_word_cloud)
    plt.axis("off")
    plt.tight_layout(pad=0)
    plt.show()

News from the blog 2021-09-20

  • I’ve published a very simple game, Tic Tac Toe, that I created for my Python 3 Exercises for Beginners book.
  • I’ve raised back the price for my books to normal levels.
    I’ve been keeping the price to the minimum to help people that wanted to learn during covid-19. I consider that who wanted to learn has already done it.

I still have bundles with a somewhat reduced price, and I authorized LeanPub platform to do discounts up to 50% at their discretion.

Bundle of four books in https://leanpub.com/b/python3-exercises-zfs-assemble-computer

https://leanpub.com/b/python3-exercises-zfs-assemble-computer

  • I’ve been deleting AMIs, Snapshots, Volumes and backups from Amazon instances I’ll no longer use.

I’ve migrated to Docker some sites and WordPress sites and now I’m CSP (Cloud Service Provider) agnostic. I can deploy wherever I want.

We pay per GB used of storage, so my money will get a better usage.

As I said in my old article from 2013, The Cloud is for Scaling. For Startups and for Enterprises. It is too expensive for small and medium companies.

  • For those studying Python there is a Virtual Meetup about Data Analysis, in Spanish ,the 23th of September

https://www.meetup.com/tech-barcelona/events/280791310/

More meetups:

https://www.meetup.com/tech-barcelona/

Python Game Tic Tac Toe

I implemented this very simple game for my book Python 3 Exercises for Beginners.

Source Code available here:

https://gitlab.com/carles.mateo/python-classes/-/blob/main/2021-09-10/game_tic-tac-toe.py


class TicTacToe:

    def __init__(self):
        self.a_a_s_map = []
        self.generate_map()

    def generate_map(self):
        self.a_a_s_map = []

        for i_y in range(3):
            a_s_pos_x = [" ", " ", " "]
            self.a_a_s_map.append(a_s_pos_x)

    def get_map(self):
        s_map = ""

        s_map = s_map + "    1   2   3\n"
        s_map = s_map + "  -------------\n"
        for i_y in range(3):
            s_map = s_map + str(i_y + 1) + " |"
            for s_char in self.a_a_s_map[i_y]:
                s_map = s_map + " " + s_char + " |"
            s_map = s_map + "\n"
            s_map = s_map + "  -------------\n"

        return s_map

    def validate_move(self, s_char, i_x, i_y):
        """
        Validates the movement and updates the map
        :param s_char:
        :param i_x:
        :param i_y:
        :return: bool
        """
        i_x = i_x - 1
        i_y = i_y - 1

        if self.a_a_s_map[i_y][i_x] == " ":
            self.a_a_s_map[i_y][i_x] = s_char
            return True

        return False

    def check_win(self):
        for s_char in ["O", "X"]:

            # check horizontal
            for i_y in range(3):
                i_horizontal_match = 0
                for i_x in range(3):
                    if self.a_a_s_map[i_y][i_x] == s_char:
                        i_horizontal_match = i_horizontal_match + 1
                if i_horizontal_match == 3:
                    return True

            # Check vertical
            for i_x in range(3):
                i_vertical_match = 0
                for i_y in range(3):
                    if self.a_a_s_map[i_y][i_x] == s_char:
                        i_vertical_match = i_vertical_match + 1
                if i_vertical_match == 3:
                    return True

            # Check diagonal
            if self.a_a_s_map[1][1] == s_char:
                if self.a_a_s_map[0][0] == s_char and self.a_a_s_map[2][2] == s_char:
                    return True
                if self.a_a_s_map[0][2] == s_char and self.a_a_s_map[2][0] == s_char:
                    return True

        return False

    def check_stale(self):
        for i_y in range(3):
            for i_x in range(3):
                if self.a_a_s_map[i_y][i_x] == " ":
                    # Is not full
                    return False

        # We checked all and all were full
        return True


def get_from_keyboard(s_question, i_min, i_max):
    i_number = 0
    while True:
        s_answer = input(s_question)
        try:
            i_number = int(s_answer)
        except:
            print("Please, type a number")
            continue

        if i_number < i_min or i_number > i_max:
            print("Invalid value. Values should be between", i_min, "and", i_max)
            continue

        # Validations are Ok
        break

    return i_number


if __name__ == "__main__":
    o_tictactoe = TicTacToe()

    while True:

        s_map = o_tictactoe.get_map()
        print(s_map)

        while True:
            i_x = get_from_keyboard("Your move O for x: ", i_min=1, i_max=3)
            i_y = get_from_keyboard("Your move O for y: ", i_min=1, i_max=3)

            b_valid_move = o_tictactoe.validate_move("O", i_x, i_y)
            if b_valid_move is False:
                print("Invalid move")
                continue

            break

        s_map = o_tictactoe.get_map()
        print(s_map)
        b_check_win = o_tictactoe.check_win()
        if b_check_win is True:
            print("Player O wins!")
            exit(0)

        b_stale = o_tictactoe.check_stale()
        if b_stale is True:
            print("Nobody wins in war")
            exit(0)

        while True:
            i_x = get_from_keyboard("Your move X for x: ", i_min=1, i_max=3)
            i_y = get_from_keyboard("Your move X for y: ", i_min=1, i_max=3)

            b_valid_move = o_tictactoe.validate_move("X", i_x, i_y)
            if b_valid_move is False:
                print("Invalid move")
                continue

            break

        s_map = o_tictactoe.get_map()
        print(s_map)
        b_check_win = o_tictactoe.check_win()
        if b_check_win is True:
            print("Player X wins!")
            exit(0)

News from the blog 2021-09-07

  • Blocking Ip addresses in the firewall

I’ve been blocking entire ranges from Cloud Providers, as some of their Ip’s were being used to try to hack/abuse the blog.

After some time blocking individual Ip’s, I opted for being some more effective and blocking /24 (the class C for the offending Ip).

If you work for a CSP and you can’t see my blog from your range, this is the reason.

  • I have updated my two Python books.

For Python 3 simple exercises for beginners I’ve added this new content:

v.0.29 and v.0.28 Python 3 Simple Exercises for Beginners
Added a new section Games, with a first game “Guess my number”

Provided a solution for the recipe Exercise 3: Create a function that will ask a user for a number from Keyboard input, and return the result, only when the value is between the accepted ranges.

Added two new questions to the Quiz section.

Fixed a docstring in Recipes Exercise 4, referring to a String return value which it was an Integer.

Added an exercise for retrieving a JSON with your public Ip

Added a new exercise for converting bytes to kilobytes with two decimal positions.

Added a new exercise / recipe to SSH to a Server with Username and Password and execute a command, using the Paramiko library.

For Python 3 Combat Guide I have added this new content:

v.1.02 Python 3 Combat Guide

Added a new exercise / recipe to SSH to a Server with Username and Password and execute a command using Paramiko library. I added two examples executing commands uptime and df -h /

Show an alternative way to run Flask apps.

Added new interesting packages.

As long as covid is active I plan to keep the minimum price of each of my books at the minimum accepted by LeanPub which is USD $5.

I also enable bundles and enable LeanPub to make punctual discounts to make them even more affordable to humble pockets.

  • I’m going to teach an initiation to programming class, a live Zoom session of 1 hour, plus 15 minutes for questions, for Free. It will be a very basic starting class for absolute beginners.

It will be performed at a time around 19:00 Irish time, wich is 11:00 AM in Irivine, CA time, to maximize the opportunities for people to assist.

If you would like to join, write me an email:

There are not many spots available, but if there is no room for you this time I may contact you for the next time.

This version adds class StringUtils, and a set of methods to perform different useful tasks with Strings, like converting a number to the biggest unit with sense, like a large number of bytes to PB, a smaller one to TB, or GB, MB, KB, justifying strings to the right or to the left, and cap the number of chars to a specified one, etc…

I have many libraries that I’ve been building across the years, and I’m liberating them as Open Source, as soon as I have time to make sure that are compatible with Python 3.5 or superior (and with Python 2.6 when possible), and I have time to add a decent Unit Testing Code Coverage.

I try to release libraries that have no other dependencies. After that I’ll start releasing my libs that have dependencies, like to work with MySQL, SQLite, web scrapping, etc…

Have a cheap Ubuntu in your Windows or Mac with Docker

I had this idea after one my Python and Linux students with two laptops, a Mac OS X and a Windows one explained me that the Mac OS X is often taken by their daughters, and that the Windows 10 laptop has not enough memory to run PyCharm and Virtual Box fluently. She wanted to have a Linux VM to practice Linux, and do the Bash exercises.

So this article explains how to create a Ubuntu 20.04 LTS Docker Container, and execute a shell were you can practice Linux, Ubuntu, Bash, and you can use it to run Python, Apache, PHP, MySQL… as well, if you want.

You need to install Docker for Windows of for Mac:

Docker for Windows is very handy and visual

Just pay attention to your type of processor: Mac with Intel chip or Mac with apple chip.

The first thing is to create the Dockerfile.

FROM ubuntu:20.04

MAINTAINER Carles Mateo

ARG DEBIAN_FRONTEND=noninteractive

RUN apt update && \
    apt install -y vim python3-pip &&  \
    apt install -y net-tools mc htop less strace zip gzip lynx && \
    pip3 install pytest && \
    apt-get clean

RUN echo "#!/bin/bash\nwhile [ true ]; do sleep 60; done" > /root/loop.sh; chmod +x /root/loop.sh

CMD ["/root/loop.sh"]

So basically the file named Dockerfile contains all the blueprints for our Docker Container to be created.

You see that I all the installs and clean ups in one single line. That’s because Docker generates a layer of virtual disk per each line in the Dockerfile. The layers are persistent, so even if in the next line we delete the temporary files, the space used will not be recovered.

You see also that I generate a Bash file with an infinite loop that sleeps 60 seconds each loop and save it as /root/loop.sh This is the file that later is called with CMD, so basically when the Container is created will execute this infinite loop. Basically we give to the Container a non ending task to prevent it from running, and exiting.

Now that you have the Dockerfile is time to build the Container.

For Mac open a terminal and type this command inside the directory where you have the Dockerfile file:

sudo docker build -t cheap_ubuntu .

I called the image cheap_ubuntu but you can set the name that you prefer.

For Windows 10 open a Command Prompt with Administrative rights and then change directory (cd) to the one that has your Dockerfile file.

docker.exe build -t cheap_ubuntu .
Image being built… (some data has been covered in white)

Now that you have the image built, you can create a Container based on it.

For Mac:

sudo docker run -d --name cheap_ubuntu cheap_ubuntu

For Windows (you can use docker.exe or just docker):

docker.exe run -d --name cheap_ubuntu cheap_ubuntu

Now you have Container named cheap_ubuntu based on the image cheap_ubuntu.

It’s time to execute an interactive shell and be able to play:

sudo docker exec -it cheap_ubuntu /bin/bash

For Windows:

docker.exe exec -it cheap_ubuntu /bin/bash
Our Ubuntu terminal inside Windows

Now you have an interactive shell, as root, to your cheap_ubuntu Ubuntu 20.04 LTS Container.

You’ll not be able to run the graphical interface, but you have a complete Ubuntu to learn to program in Bash and to use Linux from Command Line.

You will exit the interactive Bash session in the container with:

exit

If you want to stop the Container:

sudo docker stop cheap_ubuntu

Or for Windows:

docker.exe stop cheap_ubuntu

If you want to see what Containers are running do:

sudo docker ps

Released Python CarlesLibs version 0.99.2

I’ve updated my pypi carleslibs package to version 0.99.2.

The addition I made to this version is StringUtils class which offer functionalities for handling amount conversions (to different units), number formatting, string formatting and align (left, right…). I added a 85% of Unit Testing Code Coverage.

Here you have some general information about how to install and how to use the package:

https://blog.carlesmateo.com/carleslibs/

News of the blog 2021-08-16

  • I completed my ZFS on Ubuntu 20.04 LTS book.
    I had an error in an actual hard drive so I added a Troubleshooting section explaining how I fixed it.
  • I paused for a while the advance of my book Python: basic exercises for beginners, as my colleague Michela is translating it to Italian. She is a great Engineer and I cannot be more happy of having her help.
  • I added a new article about how to create a simple web Star Wars game using Flask.
    As always, I use Docker and a Dockerfile to automate the deployment, so you can test it without messing with your local system.
    The code is very simple and easy to understand.
mysql> UPDATE wp_options set option_value='blog.carlesmateo.local' WHERE option_name='siteurl';
Query OK, 1 row affected (0.02 sec)
Rows matched: 1 Changed: 1 Warnings: 0

This way I set an entry in /etc/hosts and I can do all the tests I want.

  • I added a new section to the blog, is a link where you can see all the articles published, ordered by number of views.
    /posts_and_views.php

Is in the main page, just after the recommended articles.
Here you can see the source code.

  • I removed the Categories:
    • Storage
      • ZFS
  • In favor of:
    • Hardware
      • Storage
        • ZFS
  • So the articles with Categories in the group deleted were reassigned the Categories in the second group.
  • Visually:
    • I removed some annoying lines from the Quick Selection access.
      They came from inherited CSS properties from my WordPress, long time customized, and I created new styles for this section.
    • I adjusted the line-height to avoid separation between lines being too much.
  • I added a link in the section of Other Engineering Blogs that I like, to the great https://github.com/lesterchan site, author of many super cool WordPress plugins.

My PHP Script to see WordPress Posts and Views ordered by Views

This article shows the code for my PHP program posts_and_views.php that you can see at the bottom of in the Quick Selection bar.

Page rendered of all my Published Posts sorted by Views

So I wrote this small PHP Code to display the number of Post Views recorded in my WordPress Database, sorted by Views in descending order.

In other words, sorted from Most Viewed.

Please note: In order to track the post views I use the excellent plugin WP-PostViews by Lester ‘GaMerZ’ Chan

Here is the code for posts_and_views.php

As you can see I use wp-config.php file to get the Database Settings (username, password, database) but I don’t use the WordPress Engine, it is just a stand alone PHP. So I wrote my own function to get the slug based on the category name. I believe the slug for it is in the Database and I could have added this as a SubQuery with JOINs, which would be better, but I wanted to keep the Database workload lightweight and especially, I did not want to invest more time investigating how the get the slug.

As I tested my function I didn’t find any Category failing but I saw that I had the Category Storage repeated in two different structure tree in Categories, and I will only always link to just the first one (So I was not linkin to the slug storage-2). I fixed that Category that I found repeated but to be honest if this script was a commercial solution or an Open Source solution properly maintained rather than just a sample, I would update it to have Categories and Tags’ slugs coming from the Database.

I would probably make it work with a Cron that would generate a cached page, updated every hour or every ten minutes. I did this in other projects I worked or in my PHP Framework Catalonia Framerwork.

But honestly, the load of the page does not justify a major effort in optimizing here, in this case.

<!DOCTYPE html>
<html>
<head>
    <style>
        .first {

            background-color: blue;
            padding: 12px;
        }
        .second {
            background-color: rgba(254, 253, 252, 0.7);
            text-align: center;
            padding:20px 0;
            font-size: 20px;
        }

        .centered {
            text-align: center;
        }

        /* unvisited link */
        a:link {
            color: blue;
            text-decoration: none;
        }

        /* visited link */
        a:visited {
            color: blue;
            text-decoration: none;
        }

        /* mouse over link */
        a:hover {
            color: #00A8EF;
            text-decoration: underline;
        }

        /* selected link */
        a:active {
            color: blue;
        }
    </style>
<body>
<?php

include "wp-config.php";

$s_site_name = "Carles Mateo's blog";
$s_site_link = "https://blog.carlesmateo.com";

function get_category_slug($s_text) {
    $s_output_text = strtolower($s_text);
    $s_output_text = str_replace(" ", "-", $s_output_text);

    return $s_output_text;
}

?>
<h1><a href="<?php print $s_site_link; ?>"><?php print($s_site_name); ?></a></h1>
<?php


$s_sort = "views DESC, post_date, post_title DESC";

if (array_key_exists("sort", $_GET)) {
    if ($_GET["sort"] == "date") {
        $s_sort = "post_date, views, post_title";
    }
}

$s_servername = "localhost";
$s_database = DB_NAME;
$s_username = DB_USER;
$s_password = DB_PASSWORD;


// Create connection
$o_conn = new mysqli($s_servername, $s_username, $s_password, $s_database);

// Check connection
if ($o_conn->connect_error) {
    die("Connection failed: " . $o_conn->connect_error);
}

/*
mysql> DESCRIBE wp_posts;
+-----------------------+-----------------+------+-----+---------------------+----------------+
| Field                 | Type            | Null | Key | Default             | Extra          |
+-----------------------+-----------------+------+-----+---------------------+----------------+
| ID                    | bigint unsigned | NO   | PRI | NULL                | auto_increment |
| post_author           | bigint unsigned | NO   | MUL | 0                   |                |
| post_date             | datetime        | NO   |     | 0000-00-00 00:00:00 |                |
| post_date_gmt         | datetime        | NO   |     | 0000-00-00 00:00:00 |                |
| post_content          | longtext        | NO   |     | NULL                |                |
| post_title            | text            | NO   |     | NULL                |                |
| post_excerpt          | text            | NO   |     | NULL                |                |
| post_status           | varchar(20)     | NO   |     | publish             |                |
| comment_status        | varchar(20)     | NO   |     | open                |                |
| ping_status           | varchar(20)     | NO   |     | open                |                |
| post_password         | varchar(255)    | NO   |     |                     |                |
| post_name             | varchar(200)    | NO   | MUL |                     |                |
| to_ping               | text            | NO   |     | NULL                |                |
| pinged                | text            | NO   |     | NULL                |                |
| post_modified         | datetime        | NO   |     | 0000-00-00 00:00:00 |                |
| post_modified_gmt     | datetime        | NO   |     | 0000-00-00 00:00:00 |                |
| post_content_filtered | longtext        | NO   |     | NULL                |                |
| post_parent           | bigint unsigned | NO   | MUL | 0                   |                |
| guid                  | varchar(255)    | NO   |     |                     |                |
| menu_order            | int             | NO   |     | 0                   |                |
| post_type             | varchar(20)     | NO   | MUL | post                |                |
| post_mime_type        | varchar(100)    | NO   |     |                     |                |
| comment_count         | bigint          | NO   |     | 0                   |                |
+-----------------------+-----------------+------+-----+---------------------+----------------+
 */

/*
mysql> describe wp_postmeta;
+------------+-----------------+------+-----+---------+----------------+
| Field      | Type            | Null | Key | Default | Extra          |
+------------+-----------------+------+-----+---------+----------------+
| meta_id    | bigint unsigned | NO   | PRI | NULL    | auto_increment |
| post_id    | bigint unsigned | NO   | MUL | 0       |                |
| meta_key   | varchar(255)    | YES  | MUL | NULL    |                |
| meta_value | longtext        | YES  |     | NULL    |                |
+------------+-----------------+------+-----+---------+----------------+
 */

$s_sql = "SELECT DISTINCT post_title, post_content,
                 (SELECT CAST(meta_value AS SIGNED) FROM wp_postmeta WHERE wp_postmeta.meta_key = 'views' AND wp_postmeta.post_id = wp_posts.ID) AS 'views',
                 (SELECT group_concat(wp_terms.name separator ',') 
                         FROM wp_terms
                  INNER JOIN wp_term_taxonomy 
                          ON wp_terms.term_id = wp_term_taxonomy.term_id
                  INNER JOIN wp_term_relationships wpr 
                          ON wpr.term_taxonomy_id = wp_term_taxonomy.term_taxonomy_id
                  WHERE 
                         taxonomy= 'category' AND wp_posts.ID = wpr.object_id
                ) AS 'Categories',
                (SELECT group_concat(wp_terms.name separator ', ') 
                        FROM wp_terms
                 INNER JOIN wp_term_taxonomy 
                         ON wp_terms.term_id = wp_term_taxonomy.term_id
                 INNER JOIN wp_term_relationships wpr 
                         ON wpr.term_taxonomy_id = wp_term_taxonomy.term_taxonomy_id
                 WHERE 
                        taxonomy= 'post_tag' AND wp_posts.ID = wpr.object_id
                ) AS 'Tags',
               ID, post_name, post_date, post_modified, post_type
          FROM wp_posts
          WHERE 
                post_type = 'post' AND post_status = 'publish'
          ORDER BY
                $s_sort";

$o_result = $o_conn->query($s_sql);

if ($o_result->num_rows > 0) {
    ?><table style="border:1px solid">
    <tr class="first"><th>Title</th><th style="min-width:100px">Views</th><th style="min-width:150px">Creation Date</th><th>Categories</th><th>Tags</th></tr>

    <?php
    $i_total_views = 0;
    $i_articles = 0;

    // output data of each row
    while($row = $o_result->fetch_assoc()) {
        $s_style='style="border:1px solid"';
        $s_style='';
        $s_url = $row['post_name'];
        print('<tr>');
        print("<td $s_style>");
        print('<a href="'.$s_url.'" target="_blank">');
        print($row["post_title"]);
        print('</a>');
        print("</td>");
        print('<td class="centered" '.$s_style.'>');
        print(number_format($row["views"]));
        print("</td>");
        print("<td $s_style>");
        print("<small>");
        print($row["post_date"]);
        print("</small>");
        print("</td>");
        print("<td $s_style>");
        $s_categories = $row["Categories"];
        $a_categories = explode (",", $s_categories);
        $s_categories_content = "";
        foreach($a_categories as $s_category) {
            $s_category_slug = "/category/".get_category_slug($s_category)."/";
            $s_categories_content = $s_categories_content .'<a href="'.$s_category_slug.'" target="_blank">';
            $s_categories_content = $s_categories_content .$s_category;
            $s_categories_content = $s_categories_content ."</a>, ";
        }

        if (strlen($s_categories_content) > 0) {
            $s_categories_content = substr($s_categories_content, 0, -2);
        }

        print($s_categories_content);
        print("</td>");
        print("<td $s_style>");
        print($row["Tags"]);
        print("</td>");
        // $row["post_content"];
        $i_total_views = $i_total_views + intval($row["views"]);
        $i_articles++;
        echo "</tr>";
    } ?></table><?php
    print("<strong>Total articles:</strong> ".number_format($i_articles)." <strong>Total Views:</strong> ".number_format($i_total_views));
    print("<br>");
} else {
    echo "<p>0 results</p>";
}
$o_conn->close();

?>
</body>
</html>

Migrating some Services from Amazon to Digital Ocean

Analyzing the needs

I start with a VM, to learn about the providers and the migration project as I go.

My VM has been running in Amazon AWS for years.

It has 3.5GB of RAM and 1 Core. However is uses only 580MB of RAM. I’m paying around $85/month for this with Amazon.

I need to migrate:

  • DNS Server
  • Email
  • Web
  • Database

For the DNS Server I don’t need it anymore, each Domain provider has included DNS Service for free, so I do not longer to have my two DNS.

For the email I find myself in the same scenario, most providers offer 3 email accounts for your domain, and some alias, for free.

I’ll start the Service as Docker in the new CSP, so I will make it work in my computer first, locally, and so I can move easily in the future.

Note: exporting big images is not the idea I have to make backups.

I locate a Digital Ocean droplet with 1GB of RAM and 1 core and SSD disks for $5, for $6 I can have a NVMe version. That I choose.

Disk Space for the Statics

The first thing I do is to analyze the disk space needs of the service.

In this old AWS CentOS based image I have:

[root@ip-10-xxx-yyy-zzz ec2-user]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       79G   11G   69G  14% /
devtmpfs        1.8G   12K  1.8G   1% /dev
tmpfs           1.8G     0  1.8G   0% /dev/shm

Ok, so if I keep the same I have I need 11GB.

I have plenty of space on this server so I do a zip of all the contents of the blog:

cd /var/www/wordpress
zip -r /home/ec2-user/wp_sizeZ.zip wp_siteZ

Database dump

I need a dump of the databases I want to migrate.

I check what databases are in this Server.

mysql -u root -p

mysql> show databases;

I do a dump of the databases that I want:

sudo mysqldump --password='XXXXXXXX' --databases wp_mysiteZ > wp_mysiteZ.sql

I get an error, meaning MySQL needs repair:

mysqldump: Got error: 145: Table './wp_mysiteZ/wp_visitor_maps_wo' is marked as crashed and should be repaired when using LOCK TABLES

So I launch a repair:

sudo mysqlcheck --password='XXXXXXXX' --repair --all-databases

And after the dump works.

My dump takes 88MB, not much, but I compress it with gzip.

gzip wp_mysiteZ.sql

It takes only 15MB compressed.

Do not forget the parameter –databases even if only one database is exported, otherwise the CREATE DATABASE and USE `wp_mysiteZ`; will not be added to your dump.

I will need to take some data form the mysql database, referring to the user used for accessing the blog’s database.

I always keep the CREATE USER and the GRANT permissions, if you don’t check the wp-config.php file. Note that the SQL format to create users and grant permissions may be different from a SQL version to another.

I create a file named mysql.sql with this part and I compress with gzip.

Checking PHP version

php -v
PHP 7.3.23 (cli) (built: Oct 21 2020 20:24:49) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.23, Copyright (c) 1998-2018 Zend Technologies

WordPress is updated, and PHP is not that old.

The new Ubuntu 20.04 LTS comes with PHP 7.4. It will work:

php -v
PHP 7.4.3 (cli) (built: Jul  5 2021 15:13:35) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.3, Copyright (c), by Zend Technologies

The Dockerfile

FROM ubuntu:20.04

MAINTAINER Carles Mateo

ARG DEBIAN_FRONTEND=noninteractive

# RUN echo "nameserver 8.8.8.8" > /etc/resolv.conf

RUN echo "Europe/Ireland" | tee /etc/timezone

# Note: You should install everything in a single line concatenated with
#       && and finalizing with 
# apt autoremove && apt clean

#       In order to use the less space possible, as every command 
#       is a layer

RUN apt update && apt install -y apache2 ntpdate libapache2-mod-php7.4 mysql-server php7.4-mysql php-dev libmcrypt-dev php-pear git mysql-server less zip vim mc && apt autoremove && apt clean

RUN a2enmod rewrite

RUN mkdir -p /www

# If you want to activate Debug
# RUN sed -i "s/display_errors = Off/display_errors = On/" /etc/php/7.2/apache2/php.ini 
# RUN sed -i "s/error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT/error_reporting = E_ALL/" /etc/php/7.2/apache2/php.ini 
# RUN sed -i "s/display_startup_errors = Off/display_startup_errors = On/" /etc/php/7.2/apache2/php.ini 
# To Debug remember to change:
# config/{production.php|preproduction.php|devel.php|docker.php} 
# in order to avoid Error Reporting being set to 0.

ENV PATH_WP_MYSITEZ /var/www/wordpress/wp_mysitez/
ENV PATH_WORDPRESS_SITES /var/www/wordpress/

ENV APACHE_RUN_USER  www-data
ENV APACHE_RUN_GROUP www-data
ENV APACHE_LOG_DIR   /var/log/apache2
ENV APACHE_PID_FILE  /var/run/apache2/apache2.pid
ENV APACHE_RUN_DIR   /var/run/apache2
ENV APACHE_LOCK_DIR  /var/lock/apache2
ENV APACHE_LOG_DIR   /var/log/apache2

RUN mkdir -p $APACHE_RUN_DIR
RUN mkdir -p $APACHE_LOCK_DIR
RUN mkdir -p $APACHE_LOG_DIR
RUN mkdir -p $PATH_WP_MYSITEZ

# Remove the default Server
RUN sed -i '/<Directory \/var\/www\/>/,/<\/Directory>/{/<\/Directory>/ s/.*/# var-www commented/; t; d}' /etc/apache2/apache2.conf 

RUN rm /etc/apache2/sites-enabled/000-default.conf

COPY wp_mysitez.conf /etc/apache2/sites-available/

RUN chown --recursive $APACHE_RUN_USER.$APACHE_RUN_GROUP $PATH_WP_MYSITEZ

RUN ln -s /etc/apache2/sites-available/wp_mysitez.conf /etc/apache2/sites-enabled/

# Please note: It would be better to git clone from another location and
# gunzip and delete temporary files in the same line, 
# to save space in the layer.
COPY *.sql.gz /tmp/

RUN gunzip /tmp/*.sql.gz; echo "Starting MySQL"; service mysql start && mysql -u root < /tmp/wp_mysitez.sql && mysql -u root < /tmp/mysql.sql; rm -f /tmp/*.sql; rm -f /tmp/*.gz
# After this root will have password assigned

COPY *.zip /tmp/

COPY services_up.sh $PATH_WORDPRESS_SITES

RUN echo "Unzipping..."; cd /var/www/wordpress/; unzip /tmp/*.zip; rm /tmp/*.zip

RUN chown --recursive $APACHE_RUN_USER.$APACHE_RUN_GROUP $PATH_WP_MYSITEZ

EXPOSE 80

CMD ["/var/www/wordpress/services_up.sh"]

Services up

For starting MySQL and Apache I relay in services_up.sh script.

#!/bin/bash
echo "Starting MySql"
service mysql start

echo "Starting Apache"
service apache2 start
# /usr/sbin/apache2 -D FOREGROUND

while [ true ];
do
    ps ax | grep mysql | grep -v "grep "
    if [ $? -gt 0 ];
    then
        service mysql start
    fi
    sleep 10
done

You see that instead of launching apache2 as FOREGROUND, what keeps the loop, not exiting from my Container is a while [ true ]; that will keep looping and checking if MySQL is up, and restarting otherwise.

MySQL shutting down

Some of my sites receive DoS attacks. More than trying to shutdown my sites, are spammers trying to publish comment announcing fake glasses, or medicines for impotence, etc… also some try to hack into the Server to gain control of it with dictionary attacks or trying to explode vulnerabilities.

The downside of those attacks is that some times the Database is under pressure, and uses more and more memory until it crashes.

More memory alleviate the problem and buys time, but I decided not to invest more than $6 USD per month on this old site. I’m just keeping the contents alive and even this site still receives many visits. A restart of the MySQL if it dies is enough for me.

As you have seen in my Dockerfile I only have one Docker Container that runs both Apache and MySQL. One of the advantages of doing like that is that if MySQL dies, the container does not exit. However I could have had two containers with both scripts with the while [ true ];

When planning I decided to have just one single Container, all-in-one, as when I export the image for a Backup, I’ll be dealing only with a single image, not two.

Building and Running the Container

I created a Bash script named build_docker.sh that does the build for me, stopping and cleaning previous Containers:

#!/bin/bash

# Execute with sudo

s_DOCKER_IMAGE_NAME="wp_sitez"

printf "Stopping old image %s\n" "${s_DOCKER_IMAGE_NAME}"
sudo docker stop "${s_DOCKER_IMAGE_NAME}"

printf "Removing old image %s\n" "${s_DOCKER_IMAGE_NAME}"
sudo docker rm "${s_DOCKER_IMAGE_NAME}"

printf "Creating Docker Image %s\n" "${s_DOCKER_IMAGE_NAME}"
# sudo docker build -t ${s_DOCKER_IMAGE_NAME} . --no-cache
sudo docker build -t ${s_DOCKER_IMAGE_NAME} .

i_EXIT_CODE=$?
if [ $i_EXIT_CODE -ne 0 ]; then
    printf "Error. Exit code %s\n" ${i_EXIT_CODE}
    exit
fi

echo "Ready to run ${s_DOCKER_IMAGE_NAME} Docker Container"
echo "To run type: sudo docker run -d -p 80:80 --name ${s_DOCKER_IMAGE_NAME} ${s_DOCKER_IMAGE_NAME}"
echo "or just use run_in_docker.sh"
echo
echo "Debug running Docker:"
echo "docker exec -it ${s_DOCKER_IMAGE_NAME} /bin/bash"
echo

I assign to the image and the Running Container the same name.

Running in Production

Once it works in local, I set the Firewall rules and I deploy the Droplet (VM) with Digital Ocean, I upload the files via SFTP, and then I just run my script build_docker.sh

And assuming everything went well, I run it:

sudo docker run -d -p 80:80 --name wp_mysitez wp_mysitez

I check that the page works, and here we go.

Some improvements

This could also have been put in a private Git repository. You only have to care about not storing the passwords in it. (Like the MySQL grants)

The build from the Git repository can be validated with a Jenkins. Here you have an article about setup a Jenkins for yourself.

A sample Flask application

Today I bring you a game made with Python and Flask extracted from my book Python 3 Combat Guide.

It is a very simple game where you have to choose what Star wars robot you prefer.

Then an internal counter, kept in a static variable, is updated.

I display the time as well, to show the use of a in import and dynamic contents printed as well.

I added a Dockerfile and a bash script to build the Docker Image, so you can run the Docker Container without installing anything in your computer.

You can download the code from here:

https://gitlab.com/carles.mateo/python-flask-r2d2

Or clone the project:

git clone https://gitlab.com/carles.mateo/python-flask-r2d2.git

Then build the image with the script I provided:

sudo ./build_docker.sh 

After Docker Image flask_app is built, you can run a Docker Container based on it with:

sudo docker run -d -p 5000:5000 --name flask_app flask_app

After you’re done, in order to stop the Container type:

sudo docker stop flask_app

Here is the source code of the Python file flask_app.py:

#
# flask_app.py
#
# Author: Carles Mateo
# Creation Date: 2020-05-10 20:50 GMT+1
# Description: A simple Flask Web Application
#              Part of the samples of https://leanpub.com/pythoncombatguide
#              More source code for the book at https://gitlab.com/carles.mateo/python_combat_guide
#

from flask import Flask
import datetime


def get_datetime(b_milliseconds=False):
    """
    Return the datetime with miliseconds in format YYYY-MM-DD HH:MM:SS.xxxxx
    or without milliseconds as YYYY-MM-DD HH:MM:SS
    """
    if b_milliseconds is True:
        s_now = str(datetime.datetime.now())
    else:
        s_now = str(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"))

    return s_now


app = Flask(__name__)

# Those variables will keep their value as long as Flask is running
i_votes_r2d2 = 0
i_votes_bb8 = 0


@app.route('/')
def page_root():
    s_page = "<html>"
    s_page += "<title>My Web Page!</title>"
    s_page += "<body>"
    s_page += "<h1>Time now is: " + get_datetime() + "</h1>"
    s_page += """<h2>Who is more sexy?</h2>
<a href="r2d2"><img src="static/r2d2.png"></a> <a href="bb8"><img width="250" src="static/bb8.jpg"></a>"""
    s_page += "</body>"
    s_page += "</html>"

    return s_page


@app.route('/bb8')
def page_bb8():
    global i_votes_bb8

    i_votes_bb8 = i_votes_bb8 + 1

    s_page = "<html>"
    s_page += "<title>My Web Page!</title>"
    s_page += "<body>"
    s_page += "<h1>Time now is: " + get_datetime() + "</h1>"
    s_page += """<h2>BB8 Is more sexy!</h2>
                <img width="250" src="static/bb8.jpg">"""
    s_page += "<p>I have: " + str(i_votes_bb8) + "</p>"
    s_page += "</body>"
    s_page += "</html>"

    return s_page


@app.route('/r2d2')
def page_r2d2():
    global i_votes_r2d2

    i_votes_r2d2 = i_votes_r2d2 + 1

    s_page = "<html>"
    s_page += "<title>My Web Page!</title>"
    s_page += "<body>"
    s_page += "<h1>Time now is: " + get_datetime() + "</h1>"
    s_page += """<h2>R2D2 Is more sexy!</h2>
                <img src="static/r2d2.png">"""
    s_page += "<p>I have: " + str(i_votes_r2d2) + "</p>"
    s_page += "</body>"
    s_page += "</html>"

    return s_page


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000, debug=True)

As always, the naming of the variables is based on MT Notation.

The Dockerfile is very straightforward:

FROM ubuntu:20.04

MAINTAINER Carles Mateo

ARG DEBIAN_FRONTEND=noninteractive

RUN apt update && \
    apt install -y vim python3-pip &&  pip3 install pytest && \
    apt-get clean

ENV PYTHON_COMBAT_GUIDE /var/python_combat_guide

RUN mkdir -p $PYTHON_COMBAT_GUIDE

COPY ./ $PYTHON_COMBAT_GUIDE

ENV PYTHONPATH "${PYTHONPATH}:$PYTHON_COMBAT_GUIDE/src/:$PYTHON_COMBAT_GUIDE/src/lib"

RUN pip3 install -r $PYTHON_COMBAT_GUIDE/requirements.txt

# This is important so when executing python3 -m current directory will be added to Syspath
# Is not necessary, as we added to PYTHONPATH
#WORKDIR $PYTHON_COMBAT_GUIDE/src/lib

EXPOSE 5000

# Launch our Flask Application
CMD ["/usr/bin/python3", "/var/python_combat_guide/src/flask_app.py"]