Tag Archives: Unit Testing

Working on a Sudoku Solver in Python (Source Code)

This is a document previous to a live code review session.

It has the information to prepare for the upcoming code review session, where I plan to share the lessons learned, decision I took, mistakes I did, refactors I had to overcome, and tentatively we will refactor code in order to add some Unit Testing.

History

I used to play sudoku with my family, so from time to time I do by myself.

Once I found a sudoku that was impossible and it happened that it was a typo from the newspaper, so, when I found another impossible sudoku I wanted to know if it was me, or if there was a typo or similar, so I decided to write a Sudoku Solver that will solve the sudoku for me.

The bad guys

I had problems solving these two sudokus:

Some Screenshots

The Source Code

You can clone the project from here:

https://gitlab.com/carles.mateo/sudo-ku-solver

You will have to install colorama package, as I used it for giving colors to the output:

pip3 install colorama

The main program sudokusolver.py:

import copy
from lib.colorutils import ColorUtils


class SudokuMap():
    
    def __init__(self, i_width, i_height, o_color=ColorUtils()):
        self.i_width = i_width
        self.i_height = i_height
        self.o_color = o_color

        self.a_map = self.generate_empty_map()

    def generate_empty_map(self):
        a_map = []
        a_row = []
        a_i_possible_numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9]
        for i_x in range(self.i_width):
            a_row.append(a_i_possible_numbers.copy())

        for i_y in range(self.i_height):
            a_map.append(copy.deepcopy(a_row))

        return a_map

    def set_number(self, i_number, i_x, i_y):
        """
        Sets a well known (already defined in the original map) number for a position
        :param i_number:
        :param i_x:
        :param i_y:
        :return:
        """
        self.a_map[i_y][i_x] = [i_number]

    def detect_and_remove_a_number_from_possibles_from_a_row(self, i_y):
        """
        We will elinate this possibility from the row
        :return: Boolean
        """

        b_found = False
        self.o_color.print_label("Detecting numbers to remove from row " + str(i_y))

        for i_x in range(0, self.i_width):
            a_i_numbers_possible = self.a_map[i_y][i_x]
            if len(a_i_numbers_possible) == 1:
                b_found = True
                i_number_found = self.a_map[i_y][i_x][0]
                print("Found a number that will be removed from horizontal and vertical and in quadrant", i_number_found, "at", i_x, i_y)
                self.remove_a_number_from_possibles_in_a_row(i_number_to_remove=i_number_found, i_y=i_y)
                self.remove_a_number_from_possibles_in_a_column(i_number_to_remove=i_number_found, i_x=i_x)
                self.remove_a_number_from_possibles_in_quadrant(i_number_to_remove=i_number_found, i_x=i_x, i_y=i_y)

        return b_found

    def remove_a_number_from_possibles_in_a_row(self, i_number_to_remove, i_y):
        """
        Removes a number from the list of possibles in that row
        :param i_number_to_remove:
        :param i_y:
        :return:
        """

        self.o_color.print_label("> Scanning for removing " + str(i_number_to_remove) + " in row " + str(i_y))

        for i_x in range(0, self.i_width):
            a_i_numbers_possible = self.a_map[i_y][i_x]
            if len(a_i_numbers_possible) == 1 and a_i_numbers_possible[0] == i_number_to_remove:
                # This is the right cell, ignore it
                pass
            else:
                # Subtract the number from the sequence
                if i_number_to_remove in a_i_numbers_possible:
                    a_i_numbers_possible_old = a_i_numbers_possible.copy()
                    a_i_numbers_possible.remove(i_number_to_remove)
                    print("> Removed", i_number_to_remove, "From:", str(i_x) + "x" + str(i_y), a_i_numbers_possible_old, "Pending:", a_i_numbers_possible)
                    self.a_map[i_y][i_x] = a_i_numbers_possible
                    if len(a_i_numbers_possible) == 1:
                        # Trigger it again for the number recently discovered
                        i_new_number_to_remove = a_i_numbers_possible[0]
                        self.o_color.print_success("> Found " + str(i_new_number_to_remove) + " From: " + str(i_x) + "x" + str(i_y))
                        self.remove_a_number_from_possibles_in_a_row(i_number_to_remove=i_new_number_to_remove, i_y=i_y)
                        self.remove_a_number_from_possibles_in_a_column(i_number_to_remove=i_new_number_to_remove, i_x=i_x)
                        self.remove_a_number_from_possibles_in_quadrant(i_number_to_remove=i_new_number_to_remove, i_x=i_x, i_y=i_y)

        self.o_color.print_label("> Leaving scan for " + str(i_number_to_remove) + " in row " + str(i_y))

    def remove_a_number_from_possibles_in_a_column(self, i_number_to_remove, i_x):
        """
        Removes a number from the list of possibles in that row
        :param i_number_to_remove:
        :param i_y:
        :return:
        """

        self.o_color.print_label("V Scanning for removing " + str(i_number_to_remove) + " in col " + str(i_x))

        for i_y in range(0, self.i_height):
            a_i_numbers_possible = self.a_map[i_y][i_x]
            if len(a_i_numbers_possible) == 1 and a_i_numbers_possible[0] == i_number_to_remove:
                # This is the right cell, ignore it
                pass
            else:
                # Subtract the number from the sequence
                if i_number_to_remove in a_i_numbers_possible:
                    a_i_numbers_possible_old = a_i_numbers_possible.copy()
                    a_i_numbers_possible.remove(i_number_to_remove)
                    print("V Removed", i_number_to_remove, "From:", i_x, i_y, a_i_numbers_possible_old, "Pending:", a_i_numbers_possible)
                    # @TODO: Remove, as it's a pointer it is not needed
                    self.a_map[i_y][i_x] = a_i_numbers_possible
                    if len(a_i_numbers_possible) == 1:
                        # Trigger it again for the number recently discovered
                        i_new_number_to_remove = a_i_numbers_possible[0]
                        self.o_color.print_success("Found " + str(i_new_number_to_remove) + " From: " + str(i_x) + " " + str(i_y))
                        self.remove_a_number_from_possibles_in_a_row(i_number_to_remove=i_new_number_to_remove, i_y=i_y)
                        self.remove_a_number_from_possibles_in_a_column(i_number_to_remove=i_new_number_to_remove, i_x=i_x)
                        self.remove_a_number_from_possibles_in_quadrant(i_number_to_remove=i_new_number_to_remove, i_x=i_x, i_y=i_y)

        self.o_color.print_label("V Leaving scan for " + str(i_number_to_remove) + " in col " + str(i_x))

    def remove_a_number_from_possibles_in_quadrant(self, i_number_to_remove, i_x, i_y):
        """

        :param i_number_to_remove:
        :param i_x:
        :param i_y:
        :return:
        """

        i_x_quadrant = int(i_x / 3)
        i_y_quadrant = int(i_y / 3)

        i_x_ini = i_x_quadrant * 3
        i_x_end = i_x_ini + 2

        i_y_ini = i_y_quadrant * 3
        i_y_end = i_y_ini + 2

        for i_y_rel in range(i_y_ini, i_y_end + 1):
            for i_x_rel in range(i_x_ini, i_x_end + 1):
                a_i_numbers_possible = self.a_map[i_y_rel][i_x_rel]
                if len(a_i_numbers_possible) == 1 and a_i_numbers_possible[0] == i_number_to_remove:
                    # This is the right cell, ignore it
                    pass
                else:
                    # Subtract the number from the sequence
                    if i_number_to_remove in a_i_numbers_possible:
                        a_i_numbers_possible_old = a_i_numbers_possible.copy()
                        a_i_numbers_possible.remove(i_number_to_remove)
                        print("X Removed", i_number_to_remove, "From:", i_x_rel, i_y_rel, a_i_numbers_possible_old, "Pending:", a_i_numbers_possible)
                        # Nota: Here I had a bug and I was "liant-la parda"
                        # if len(a_i_numbers_possible) == 1:
                        #     # Trigger it again for the number recently discovered
                        #     i_new_number_to_remove = a_i_numbers_possible[0]
                        #     string_ints = [str(int) for int in ints]
                        #     self.o_color.print_success("X Found " + str(i_new_number_to_remove) + " From: " + str(i_x) + "x" + str(i_y) + "[]")
                        #     self.remove_a_number_from_possibles_in_a_row(i_number_to_remove=i_new_number_to_remove, i_y=i_y)
                        #     self.remove_a_number_from_possibles_in_a_column(i_number_to_remove=i_new_number_to_remove, i_x=i_x)

    def check_if_number_possibles_in_quadrant_is_unique(self, i_number_to_check, i_x, i_y):
        """

        :param i_number_to_remove:
        :param i_x:
        :param i_y:
        :return: b_found
        """

        i_x_quadrant = int(i_x / 3)
        i_y_quadrant = int(i_y / 3)

        i_x_ini = i_x_quadrant * 3
        i_x_end = i_x_ini + 2

        i_y_ini = i_y_quadrant * 3
        i_y_end = i_y_ini + 2

        i_number_of_occurrences_found = 0
        i_x_position_number = 0
        i_y_position_number = 0

        b_unique = False

        for i_y_rel in range(i_y_ini, i_y_end + 1):
            for i_x_rel in range(i_x_ini, i_x_end + 1):
                a_i_numbers_possible = self.a_map[i_y_rel][i_x_rel]
                for i_number_in_possibles in a_i_numbers_possible:
                    if len(a_i_numbers_possible) > 1 and i_number_in_possibles == i_number_to_check:
                        # This is the right cell, ignore it
                        i_number_of_occurrences_found += 1
                        i_x_position_number = i_x_rel
                        i_y_position_number = i_y_rel
                        if i_number_of_occurrences_found > 1:
                            # Unsuccessful
                            break

        if i_number_of_occurrences_found == 1:
            # Success!
            a_i_numbers_possible = [i_number_to_check]
            self.a_map[i_y_position_number][i_x_position_number] = a_i_numbers_possible
            b_unique = True

        return b_unique, i_x_position_number, i_y_position_number

    def check_if_number_possibles_in_row_is_unique(self, i_number_to_check, i_y):
        """

        :param i_number_to_check:
        :param i_x:
        :param i_y:
        :return:
        """

        i_number_of_occurrences_found = 0
        i_x_position_number = 0
        i_y_position_number = 0

        b_unique = False

        for i_x_rel in range(0, 9):
            a_i_numbers_possible = self.a_map[i_y][i_x_rel]
            for i_number_in_possibles in a_i_numbers_possible:
                if len(a_i_numbers_possible) > 1 and i_number_in_possibles == i_number_to_check:
                    # This is the right cell, ignore it
                    i_number_of_occurrences_found += 1
                    i_x_position_number = i_x_rel
                    i_y_position_number = i_y
                    if i_number_of_occurrences_found > 1:
                        # Unsuccessful
                        break

        if i_number_of_occurrences_found == 1:
            # Success!
            a_i_numbers_possible = [i_number_to_check]
            self.a_map[i_y_position_number][i_x_position_number] = a_i_numbers_possible
            b_unique = True

        return b_unique, i_x_position_number, i_y_position_number

    def get_map_drawing_as_string(self, a_map_alternative=None):
        s_map = ""
        i_counter_y = 0
        s_separator_rows = "="

        a_map_to_use = self.a_map
        if a_map_alternative is not None:
            a_map_to_use = a_map_alternative

        s_map = s_map + s_separator_rows * 37 + "\n"
        for a_row in a_map_to_use:
            i_counter_y += 1
            if i_counter_y == 3:
                i_counter_y = 0
                s_separator_rows = "="
            else:
                s_separator_rows = "-"

            s_map = s_map + "|"
            i_counter = 0
            for a_i_numbers_possible in a_row:
                i_counter += 1

                if len(a_i_numbers_possible) == 1:
                    s_number = str(a_i_numbers_possible[0])
                else:
                    s_number = " "

                if i_counter == 3:
                    s_separator = "|"
                    i_counter = 0
                else:
                    s_separator = "¦"
                s_map = s_map + " " + s_number + " " + s_separator
            s_map = s_map + "\n"

            s_map = s_map + s_separator_rows * 37 + "\n"

            # Replace 0 by " "
            s_map = s_map.replace("0", " ")

        s_map = s_map + "\n\n"
        i_total_numbers_found, a_s_numbers_found = self.get_total_numbers_found()
        s_map = s_map + "Total numbers found: " + str(i_total_numbers_found) + " Numbers found: " + " ".join(a_s_numbers_found) + "\n"

        return s_map

    def get_map_drawing_of_possibles_as_string(self, a_map_alternative=None):
        s_map = ""
        i_counter_y = 0
        s_separator_rows = "="

        a_map_to_use = self.a_map
        if a_map_alternative is not None:
            a_map_to_use = a_map_alternative

        s_map = s_map + self.o_color.color_blue(s_separator_rows * ((9 * ( 9 + 2 )) + 10)) + "\n"
        for a_row in a_map_to_use:
            i_counter_y += 1
            if i_counter_y == 3:
                i_counter_y = 0
                s_separator_rows = "="
            else:
                s_separator_rows = "-"

            s_map = s_map + self.o_color.color_blue("|")
            i_counter = 0
            for a_i_numbers_possible in a_row:
                i_counter += 1

                if len(a_i_numbers_possible) == 1:
                    # The right number
                    s_number = str(a_i_numbers_possible[0]).center(9)
                    s_number = self.o_color.color_success(s_number)
                else:
                    a_i_numbers_possible_string = []
                    for i_number in a_i_numbers_possible:
                        s_number = str(i_number)
                        # Replace by the color sequence
                        if i_number == 2:
                            s_number = self.o_color.color_red(s_number)
                        if i_number == 3:
                            s_number = self.o_color.color_yellow(s_number)
                        if i_number == 4:
                            self.o_color.color_magenta(s_number)
                        a_i_numbers_possible_string.append(s_number)
                    # s_number = "".join(a_i_numbers_possible_string).ljust(9)
                    s_number = "".join(a_i_numbers_possible_string) + " " * (9-len(a_i_numbers_possible))

                if i_counter == 3:
                    s_separator = self.o_color.color_blue("|")
                    i_counter = 0
                else:
                    s_separator = self.o_color.color_blue("¦")
                s_map = s_map + " " + s_number + " " + s_separator
            s_map = s_map + "\n"

            s_map = s_map + self.o_color.color_blue(s_separator_rows * ((9 * (9 + 2)) + 10)) + "\n"

            # Replace 0 by " "
            s_map = s_map.replace("0", " ")

        return s_map

    def get_total_numbers_found(self):

        i_total_numbers_found = 0
        a_s_numbers_found = []

        for i_y in range(0, self.i_height):
            for i_x in range(0, self.i_width):
                a_i_numbers_possible = self.a_map[i_y][i_x]
                if len(a_i_numbers_possible) == 1:
                    i_total_numbers_found = i_total_numbers_found + 1
                    i_number_found = self.a_map[i_y][i_x][0]
                    s_number_found = str(i_number_found)
                    if s_number_found not in a_s_numbers_found:
                        a_s_numbers_found.append(s_number_found)

        return i_total_numbers_found, a_s_numbers_found


if __name__ == "__main__":

    o_color = ColorUtils()

    o_map = SudokuMap(9, 9, o_color=o_color)
    o_map.set_number(i_number=1, i_x=1, i_y=0)
    o_map.set_number(3, 4, 0)
    o_map.set_number(8, 7, 0)

    o_map.set_number(8, 0, 1)
    o_map.set_number(7, 3, 1)
    o_map.set_number(4, 5, 1)
    o_map.set_number(6, 8, 1)

    o_map.set_number(3, 2, 2)
    o_map.set_number(9, 6, 2)

    o_map.set_number(2, 1, 3)
    o_map.set_number(4, 4, 3)
    o_map.set_number(6, 7, 3)

    o_map.set_number(5, 0, 4)
    o_map.set_number(6, 3, 4)
    o_map.set_number(2, 5, 4)
    o_map.set_number(8, 8, 4)

    o_map.set_number(3, 1, 5)
    o_map.set_number(8, 4, 5)
    o_map.set_number(7, 7, 5)

    o_map.set_number(2, 2, 6)
    o_map.set_number(6, 6, 6)

    o_map.set_number(9, 0, 7)
    o_map.set_number(4, 3, 7)
    o_map.set_number(3, 5, 7)
    o_map.set_number(2, 8, 7)

    o_map.set_number(8, 1, 8)
    o_map.set_number(6, 4, 8)
    o_map.set_number(1, 7, 8)

    # Extra
    # o_map.set_number(2, 0, 0)

    # Speculative
    o_map.set_number(7, 0, 3)


    # Another map
    o_map2 = SudokuMap(9, 9, o_color=o_color)
    o_map2.set_number(i_number=5, i_x=0, i_y=0)
    o_map2.set_number(i_number=9, i_x=5, i_y=0)

    o_map2.set_number(i_number=7, i_x=2, i_y=1)
    o_map2.set_number(i_number=2, i_x=7, i_y=1)

    o_map2.set_number(i_number=2, i_x=0, i_y=2)
    o_map2.set_number(i_number=3, i_x=4, i_y=2)
    o_map2.set_number(i_number=1, i_x=5, i_y=2)
    o_map2.set_number(i_number=9, i_x=7, i_y=2)

    o_map2.set_number(i_number=7, i_x=0, i_y=3)
    o_map2.set_number(i_number=1, i_x=2, i_y=3)
    o_map2.set_number(i_number=6, i_x=3, i_y=3)
    o_map2.set_number(i_number=9, i_x=4, i_y=3)
    o_map2.set_number(i_number=4, i_x=8, i_y=3)

    o_map2.set_number(i_number=1, i_x=4, i_y=4)

    o_map2.set_number(i_number=6, i_x=0, i_y=5)
    o_map2.set_number(i_number=7, i_x=4, i_y=5)
    o_map2.set_number(i_number=4, i_x=5, i_y=5)
    o_map2.set_number(i_number=3, i_x=6, i_y=5)
    o_map2.set_number(i_number=1, i_x=8, i_y=5)

    o_map2.set_number(i_number=5, i_x=1, i_y=6)
    o_map2.set_number(i_number=3, i_x=3, i_y=6)
    o_map2.set_number(i_number=6, i_x=4, i_y=6)
    o_map2.set_number(i_number=8, i_x=8, i_y=6)

    o_map2.set_number(i_number=6, i_x=1, i_y=7)
    o_map2.set_number(i_number=7, i_x=6, i_y=7)

    o_map2.set_number(i_number=9, i_x=3, i_y=8)
    o_map2.set_number(i_number=3, i_x=8, i_y=8)

    # Extra help while not implemented the best algorithm
    # =============================================================================================================
    # |     5     ¦ 148       ¦ 48        |     7     ¦     2     ¦     9     | 148       ¦     3     ¦     6     |
    # -------------------------------------------------------------------------------------------------------------
    # | 13489     ¦ 13489     ¦     7     | 48        ¦ 48        ¦     6     | 148       ¦     2     ¦     5     |
    # -------------------------------------------------------------------------------------------------------------
    # |     2     ¦ 48        ¦     6     |     5     ¦     3     ¦     1     | 48        ¦     9     ¦     7     |
    # =============================================================================================================
    # |     7     ¦ 38        ¦     1     |     6     ¦     9     ¦ 358       |     2     ¦ 58        ¦     4     |
    # -------------------------------------------------------------------------------------------------------------
    # | 348       ¦ 2348      ¦ 3458      | 28        ¦     1     ¦ 358       |     6     ¦     7     ¦     9     |
    # -------------------------------------------------------------------------------------------------------------
    # |     6     ¦ 289       ¦ 589       | 28        ¦     7     ¦     4     |     3     ¦ 58        ¦     1     |
    # =============================================================================================================
    # | 14        ¦     5     ¦     2     |     3     ¦     6     ¦     7     |     9     ¦ 14        ¦     8     |
    # -------------------------------------------------------------------------------------------------------------
    # | 3489      ¦     6     ¦ 3489      |     1     ¦ 458       ¦ 58        |     7     ¦ 45        ¦     2     |
    # -------------------------------------------------------------------------------------------------------------
    # | 148       ¦     7     ¦ 48        |     9     ¦ 458       ¦     2     | 145       ¦     6     ¦     3     |
    # =============================================================================================================
    # By best algorithm I mean that the last in the middle vertical quadrant from the top right horizontally,
    # only 5 can be in a column. That clarifies that 5 must go to the other column in last quadrant, first column. Coord 6x8
    # o_map2.set_number(i_number=5, i_x=6, i_y=8)
    # ERROR traces from a bug fixed to mentioned during the code review
    # Surprisingly this fails
    # > Leaving scan for 8 in row1
    # V Scanning for removing 8 in col 4
    # V Removed 8 From: 4 7 [5, 8] Pending: [5]
    # Found 5 From: 4 7
    #
    # > Scanning for removing 5 in row 7
    # > Removed 5 From: 5 7 [5, 8] Pending: [8]
    # > Found 8 From: 5 7
    #
    # > Scanning for removing 8 in row 7
    # > Removed 8 From: 0 7 [3, 8] Pending: [3]
    # > Found 3 From: 0 7

    o_map = o_map2

    print(o_map.get_map_drawing_as_string())

    b_changes_found = True
    while b_changes_found is True:
        b_changes_found = False

        for i_y in range(0, o_map.i_height):
            b_found = o_map.detect_and_remove_a_number_from_possibles_from_a_row(i_y=i_y)
            if b_found is True:
                print(o_map.get_map_drawing_as_string())

        for i_y in range(0, o_map.i_height):
            o_map.o_color.print_label("Scanning quadrants for row " + str(i_y))
            for i_number in range(1, 10):
                for i_x in range(0, o_map.i_width):
                    b_found, i_x_found, i_y_found = o_map.check_if_number_possibles_in_quadrant_is_unique(i_number_to_check=i_number, i_x=i_x, i_y=i_y)
                    if b_found is True:
                        # Search again
                        b_changes_found = True
                        o_map.remove_a_number_from_possibles_in_a_row(i_number_to_remove=i_number, i_y=i_y_found)
                        o_map.remove_a_number_from_possibles_in_a_column(i_number_to_remove=i_number, i_x=i_x_found)

                    b_found, i_x_found, i_y_found = o_map.check_if_number_possibles_in_row_is_unique(i_number_to_check=i_number, i_y=i_y)
                    if b_found is True:
                        b_changes_found = True
                        o_map.remove_a_number_from_possibles_in_a_column(i_number_to_remove=i_number, i_x=i_x_found)

                if b_changes_found is True:
                    print(o_map.get_map_drawing_as_string())

    # @TODO: Implement check if number in quadrant can only go to a column, to remove the non possible in that column from another quadrant
    # @TODO: Implement check if in a line only one number can go to a column.

    print(o_map.get_map_drawing_as_string())
    print(o_map.get_map_drawing_of_possibles_as_string())

The color library lib/colorutils.py:

from colorama import Fore, Back, Style , init


class ColorUtils:

    def __init__(self):
        # For Colorama on Windows
        init()

    def print_error(self, m_text, s_end="\n"):
        """
        Prints errors in Red.
        :param s_text:
        :return:
        """

        # If they pass numbers
        s_text = str(m_text)

        print(Fore.RED + s_text)
        print(Style.RESET_ALL, end=s_end)

    def print_success(self, m_text, s_end="\n"):
        """
        Prints errors in Green.
        :param s_text:
        :return:
        """

        # If they pass numbers
        s_text = str(m_text)
        print(Fore.GREEN + s_text)
        print(Style.RESET_ALL, end=s_end)

    def color_success(self, m_text):
        """
        Colors only this
        :param m_text:
        :return:
        """

        s_text = str(m_text)
        return Fore.GREEN + s_text + Fore.RESET

    def color_black(self, m_text):
        s_text = str(m_text)
        return Fore.BLACK + s_text + Fore.RESET

    def color_blue(self, m_text):
        s_text = str(m_text)
        return Fore.BLUE + s_text + Fore.RESET

    def color_red(self, m_text):
        s_text = str(m_text)
        return Fore.RED + s_text + Fore.RESET

    def color_yellow(self, m_text):
        s_text = str(m_text)
        return Fore.YELLOW + s_text + Fore.RESET

    def color_magenta(self, m_text):
        s_text = str(m_text)
        return Fore.MAGENTA + s_text + Fore.RESET

    def print_label(self, m_text, s_end="\n"):
        """
        Prints a label and not the end line
        :param s_text:
        :return:
        """

        # If they pass numbers
        s_text = str(m_text)

        print(Fore.BLUE + s_text, end="")
        print(Style.RESET_ALL, end=s_end)

    def return_text_blue(self, s_text):
        """
        Restuns a Text
        :param s_text:
        :return: String
        """
        s_text_return = Fore.BLUE + s_text + Style.RESET_ALL
        return s_text_return

News from the Blog 2022-02-22

My Open Source projects

zpool watch

zpool watch is a small Python program for Linux workstations with graphical environment and ZFS, that checks every 30 seconds if your OpenZFS pools are Ok.

If a pool is not healthy, it displays a message in a window using tk inter.

Basically allows you to skip checking from the terminal zpool status continuously or to having to customize the ZED service to send an email and having to figure out how to it can spawn a window alert to the graphical system or what to do if the session has not been initiated.

carleslibs

Since last News from the Blog I’ve released carleslibs v.1.06, v.1.0.5 and v.1.0.4.

v.1.0.6 adds a new class OsUtils to deal with mostly-Linux Os tasks, like knowing the userid, the username, if it’s root, the distribution name and kernel version.

It also adds:

DatetimeUtils.sleep(i_seconds)

In v.1.0.5 I’ve included a new method for getting the Datetime in Unix Epoc format as Integer and increased Code Coverage to 95% for ScreenUtils class.

v. 1.0.4 contains a minor update, a method in StringUtils to escape html from a string.

It uses the library html (part of Python core) so it was small work to do for me to create this method, and the Unit Test for it, but I wanted to use carleslibs in more projects and adding it as core functionality, makes the code of these projects I’m working on, much more clear.

I’m working in the future v.1.0.7.

CTOP.py

I released the stable version 0.8.8 and tagged it.

Minor refactors and adding more Code Coverage (Unit Testing), and protection in the code for division per zero when seconds passed as int are 0. (this was not an actual error, but is worth protecting the code just in case for the future)

Working on branch 0.8.9.

Currently in Master there is a stable version of 0.8.9 mainly fixing https://gitlab.com/carles.mateo/ctop/-/issues/51 which was not detecting when CTOP was running inside a Docker Container (reporting Unable to decode DMI).

My Books

Docker Combat Guide

Added 20 new pages with some tricks, like clearing the logs (1.6GB in my workstation), using some cool tools, using bind mounts and using Docker in Windows from command line without activating Docker Desktop or WSL.

https://leanpub.com/docker-combat-guide/

BTW if you work with Windows and you cannot use Docker Desktop due to the new license, in this article I explain how to use docker stand alone in Windows, without using WSL.

ZFS on Ubuntu

One of my SATA 2TB 2.5″ 5,400 rpm drive got damaged and so was generating errors, so that was a fantastic opportunity to show how to detect and deal with the situation to replace it with a new SATA 2TB 3.5″ 7,200 rpm and fix the pool.

So I updated my ZFS on Ubuntu 20.04 LTS book.

Python 3

I’ve updated Python 3 Exercises for Beginners and added a new example of how to parse the <title> tag from an HTML page, using Beautifulsoup package, to the repository of Python 3 Combat Guide book.

I also added three new exercises, and solved them.

My friend Michela is translating the book to Italian. Thanks! :)

If you already purchased any of my books, you can download the updates of them when I upload them to LeanPub.

Free courses

Code Challenges

One of my students sent me this platform, which is kinda hackerrank, but oriented to video games. To solve code challenges by programming video games.

He is having plenty of fun:

https://www.codingame.com/start

More Symfony, APIs

If you enjoyed the Free Videos about Symfony, there is more.

https://symfonycast.com/screencast/api-platform

It talks about a bundle for building APIs.

And this tutorial explains in detail how to work with Webpack Encore:

https://symfonycasts.com/screencast/webpack-encore

100 Days of Code: Python Bootcamp

A friend of mine, and colleague, Michela, is following this bootcamp and recommends it for people learning from ground 0.

https://udemy.com/course/100-days-of-code/

My work at Blizzard

The company sent me the Stein, which is sent to the employees that serve for two years, with a recognition and a celebration called “The Circle of Honor”.

Books purchased

I bought this book as often I discover new ways, better, to explain the things to my students.

Sometimes I buy books for beginners, as I can get explained what I want to do super fast and some times they teach nice tricks that I didn’t know. I have huge Django books, and it took a lot to finish them.

A simpler book may only talk about how to install and work with it under a platform (Windows or Mac, as instance) but it is all that I require as the command to create projects are the same cross platform.

For example, you can get to install and to create a simple project with ORM, connected to the database, very quickly.

Software

So I just discovered that Zoom has an option to draw in the shared screen, like Slack has. It is called Annotate. It is super useful for my classes. :)

Also discovered the icons in the Chat. It seems that not all the video calls accept it.

Hardware

As Working From Home I needed an scanner, I looked in Amazon and all of them were costing more than €200.

I changed my strategy and I bought a All-In-One from HP, which costed me €68.

So I’ll have a scanner and a backup printer, which always comes handy.

The nightmare started after I tried to connect it with Ubuntu.

Ubuntu was not recognizing it. Checking the manuals they force to configure the printer from an Android/iPhone app or from their web page, my understanding is for windows only. In any case I would not install the proprietary drivers in my Linux system.

Annoyed, I installed the Android application, and it was requesting to get Location permissions to configure it. No way. There was not possible to configure the printer without giving GPS/Location permissions to the app, so I cancelled the process.

I grabbed a Windows 10 laptop and plugged the All-in-one through the USB. I ran the wizard to search for Scanners and Printers and was not unable to use my scanner, only to configure as a printer, so I was forced to install HP drivers.

Irritated I did, and they were suggesting to configure the printer so I can print from Internet or from the phone. Thanks HP, you’ll be the next SolarWinds big-security-hole. I said no way, and in order to use the Wifi I have to agree to open that security door which is that the printer would be connected to Internet permanently, sending and receiving information. I said no, I’ll use only via USB.

Even selecting that, in order to scan, the Software forces me to create an account.

Disappointing. HP is doing very big stupid mistakes. They used to be a good company.

Since they stopped doing the drivers in Barcelona years ago, their Software and solutions (not the hardware) went to hell.

I checked the reviews in the App Store and so many people gave them 1 star and have problems… what a shame the way they created this solution.

Donations

I made a donation to OpenShot Video Editor.

This is a great Open Source, multi-platform editor, so I wanted to support the creator.

Security

Attacks: looking for exploits

This is just a sample of a set of attacks to the blog in a 3 minutes interval.

Another one this morning:

Now all are blocked in the Firewall.

This is a non stop practice from spammers and pirates that has been going on for years.

It was almost three decades ago, when I was the Linux responsible of an ISP, and I was installing a brand new Linux system connected to a service called “infovia”, at the time when Internet was used with dial-up and modems, and in the interval of time of the installation, it got hacked. I had the Ethernet connected. So then already, this was happening.

The morning I was writing this, I blocked thousands of offending Ip Addresses.

Protection solutions

I recommend you to use CloudFlare, is a CDN/Cache/Accelerator with DoS protection and even in its Free version is really useful.

Fun/Games

So I come with a game kind of Quiz that you can play with your friends, family or work colleagues working from home (WFH).

The idea is that the master shares screen and sound in Zoom, and then the rest connect to jackbox.tv and enter the code displayed on the master’s screen on their own browser, and an interactive game is started.

It is recommended that the master has two monitors so they can also play.

The games are so fun as a phrase appearing and people having to complete with a lie. If your friends vote your phrase, believing is true, you get points. If you vote the true answer, you get points too.

Very funny and recommendable.

Stuff

<humor>Skynet sent another terminator to end me, but I terminated it. Its processor lays exhibited in my home now</humor>

I bought a laminator.

It has also a ruler and a trimmer to cut the paper.

It was only €39 and I’ve to say that I’m very happy with the results.

It takes around 5 minutes to be ready, it takes to get to the hot-enough temperature, and feeds the pages slowly, around 50 secs a DIN-A4, but the results are worth the time.

I’ve protected my medical receipts and other value documents and the work was perfect. No bubbles at all. No big deal if the plastic covers are introduced not 100% straight. Even if you pass again an already plasticized document, all is good.

Fun

Databases

One of my friends sent me this image.

It is old, but still it’s fun. So it assumes the cameras of the parking or speed cameras, will OCR the plate to build a query, and that the code is not well protected. So basically is exploiting a Sql Injection.

Anybody working on the systems side, and with databases, knows how annoying are those potential situations.

Python and coding

One of my colleagues shared this :)

News from the Blog 2021-11-11

New Articles

How to communicate with your Python program running inside a Docker Container, using Linux Signals

Hope you’ll have fun reading this article:

Communicating with Docker Containers via Linux Signals and Python

I migrated my last services from Amazon and the blog to Google Compute Engine (GCE / GCP)

I wrote a Postmortem analysis about the process of migrating my last services from my 11 year old Amazon account.

Updates

Updates to articles

I updated the article about Python weird things that you may not know adding the Ellipsis …

I’ve been working in some Cassandra examples. I may publish an article soon about using it from Python and Docker.

Updates to My Books

I updated my Python and Docker books.

I’m currently writing a book about using Amazon AWS Python SDK (boto3).

Updates to Open Source projects

I have updated ctop, fixed two bugs and increased Code Coverage.

I made a new tag and released the last Stable Version:

https://gitlab.com/carles.mateo/ctop/-/tags/0.8.7

On top of my local Unit Testing, I have Jenkins checking that I don’t commit anything that breaks the Tests.

Some time ago I wrote some articles about how you can setup jenkins in a Docker Container.

Miscellaneous

Charity

I’ve donated to Wikipedia.

Only 2% of the viewers donate, so I answered the call every time it was made.

This is my 5th donation to Wikimedia.

I consider that Freedom is very important.

I bought these new books

One of my secrets to be on top is that I’m always studying.

I study all the time, at work and in my free time.

I use Linux Academy and I buy books in paper. I don’t connect with reading in tablets. I think information is stored better when read in paper. I use also a marker and pointers to keep a direct access to the most interesting points on the books.

And I study all kind of themes. Obviously I know a lot of Web Scraping, but there is always room for learning more. And whatever new I learn helps me to be better with my students and more clear writing my books.

I’ve never been a Front End, but I’ve been able to fix bugs in the Front End engines from the companies I worked for, like Privalia. I was passed a bug that prevented the Internet Explorer users to buy just one hour before we launching a massive campaign. I debugged and I found a variable named “value” so the html looked like <input name="value" value="">. In less than 30 minutes I proved to the incredulous Head of Development and the CTO that a bug in Internet Explored was causing a conflict when fetching the value from the input named value. We deployed to Production the update and the campaign was a total success. So I consider knowing Javascript and Front also a need, even if I don’t work directly with it. I want to be able to understand all the requirements and possibilities, and weaknesses, so I can fix bugs and save the day. That allowed me to fix scalability problems in Nodejs and Phantomjs projects too. (They are Javascript Server Side, event driven, projects)

It seems that Amazon.co.uk works well again for Ireland. My two last orders arrived on time and I had no problems of border taxes apparently.

Nice Python article

I enjoyed a lot this article, cause explains part of what I did with my student and friend Albert, in a project that analyzes the access logs from Apache for patterns of attempts of exploits, then feeds a database, and then blocks those offender Ip Addresses in the Firewall.

The article only covers the part of Pandas, of reading the access.log file and working with it, but is a very well redacted article:

https://mmas.github.io/read-apache-access-log-pandas

Nice Virtual Volumes article from VMware

I prefer Open Source, but there are very good commercial products too.

I liked this article about Virtual Volumes from VMWare:

Understanding Virtual Volumes (vVols) in VMware vSphere 6.7/7.0 (2113013)

https://kb.vmware.com/s/article/2113013

Thanks Blizzard (again)

There is a very nice initiative where we can nominate 4 colleagues a year, that we think that deserve a recognition.

My colleagues voted for me, so I received a gift voucher that I can spend in Ireland stores like Ikea, Pc World, Argos, Adidas, App Store & iTunes…

So thanks a million buds. :)

Some weird things from Python 3 that you may not know

Last Update: 2022-06-06 10:29 IST

You can find those bizarre things and more in my book Python 3 Combat Guide.

I’m not talking about the wonderful things, like how big can the Integers be, but about the bizarre things that may ruin your day.

What sums 0.1 + 0.1 + 0.1 in Python?

0.3?

Wrong answer.

A bit of humor

Well, to be honest the computer was wrong. They way programming languages handle the Floats tend to be less than ideal.

Floats

Maybe you know JavaScript and its famous NaN (Not a number).

You are probably sure that Python is much more exact than that…

…well, until you do a big operation with Floats, like:

10.0**304 * 10.0**50 

and

It returns infinite

I see your infinite and I add one :)

However If we try to define a number too big directly it will return OverflowError:

Please note Integers are handled in a much more robust cooler way:

Negative floats

Ok. What happens if we define a number with a negative power, like 10 ** -300 ?

And if we go somewhere a bit more far? Like 10 ** -329

It returns 0.0

Ups!

I mention in my books why is better to work with Integers, and in fact most of the eCommerces, banks and APIs work with Integers. For example, if the amount in USD 10.00 they send multiplied by 100, so they will send 1000. All the actor know that they have to divide by 2.

Breaking the language innocently

I mentioned always that I use the MT Notation, the prefix notation I invented, inspired by the Hungarian Notation and by an amazing C++ programmer I worked with in Volkswagen and in la caixa (now caixabank), that passed away many years ago.

Well, that system of prefixes will name a variable with a prefix for its type.

It’s very useful and also prevents the next weird thing from Python.

Imagine a Junior wants to print a String and they put in a variable. And unfortunately they call this variable print. Well…

print = "Hello World!"
print("That will hurt")

Observe the output of this and try not to scream:

Variables and Functions named equally

Well, most of languages are able to differentiate a function, with its parenthesis, from a variable.

The way Python does it hurts my coder heart:

Another good reason to use MT Notation for the variables, and for taking seriously doing Unit Testing and giving a chance to using getters and setters and class Constructor for implementing limits and sanitation.

Nested Loops

This will work in Python, it doesn’t work in other languages (but please never do it).

for i in range(3):
    print("First Loop", i)
    for i in range(4):
        print("Second Loop", i)

The code will not crash by overwriting i used in the first loop, but the new i will mask the first variable.

And please, name variables properly.

Import… once?

Imports are imported only once. Even if different files imported do import the same file.

So don’t have code in the middle of them, outside functions/classes, unless you’re really know what you’re doing.

Define functions first, and execute code after if __name__ == “__main__”:

Take a look at this code:

def first_function():
    print("Inside first function")
    second_function()

first_function()

def second_function():
    print("Inside second function")

Well, this will crash as Python executes the code from top to bottom, and when it gets to first_function() it will attempt to call second_function() which has not been read by Python yet. This example will throw an error.

You’ll get an error like:

Inside first function
Traceback (most recent call last):
  File "/home/carles/Desktop/code/carles/python_combat_guide/src/structure_dont_do_this.py", line 14, in <module>
    first_function()
  File "/home/carles/Desktop/code/carles/python_combat_guide/src/structure_dont_do_this.py", line 12, in first_function
    second_function()
NameError: name 'second_function' is not defined

Process finished with exit code 1

Add your code at the bottom always, under:

if __name__ == "__main__":
    first_function()

The code inside this if will only be executed if you directly call this code as main file, but will not be executed if you import this file from another one.

You don’t have this problem with classes in Python, as they are defined first, completely read, and then you instantiate or use them. To avoid messing and creating bugs, have the imports always on the top of your file.

…Ellipsis

Today is Halloween and one of my colleagues asked me help to improve his Automation project.

I found something weird in his code.

He had something like that.

class Router:

    def router_get_info(self):
        ...

    def get_help_command(self):
        return "help"

So I asked why you use … (dot dot dot) on that empty method?.

He told me that when he don’t want to implement code he just put that.

Well, dot dot dot is Ellipsis.

And what is Ellipsis?.

Ellipsis is an object that may appear in slice notation.

A good explanation of what is Ellipsis is found in this answer in StackOverflow.

In Python all the methods, functions, if, while …. require to have an instruction at least.

So the instruction my colleague was looking for is pass.

Just a variable?

In Python you can have just a var, without anything else, like no operation with it, no call, nothing.

This makes it easy to commit an error and not detecting it.

As you see we can have just s_var variable in a line, which is a String, and this does not raises an error.

If we do from python interpreter interactively, it will print the String “I’m a pickle” (famous phrase from Rick and Morty).

Variables are case sensitive

So you can define true false none … as they are different from True False None

Variables in Unicode

Python3 accepts variables in Unicode.

I would completely discourage you to use variables with accents or other characters different from a-z 0-9 and _

Python files with these names yes, but kaboom if you import them

So you can create Python files with dash or beginning with numbers, like 20220314_programming_class.py and execute them, but you cannot import them.

RYYFTK RODRIGUEZ,LEELA,FRY, FUTURAMA, 1999

A Tuple of a String is not a Tuple, it’s a String

This can be very messy and confusing. Normally you define a tuple with parenthesis, although you can use tuple() too.

Parenthesis are the way we normally build tuples. But if we do:

print(type('this is a String'))

You get that this is a String, I mean

<class 'str'>

If you want to get a tuple of a String you can add a comma after the first String, which is weird. You can also do tuple("this is a String")

I think the definition of a tuple should be consistent and idempotent, no matter if you use one or more parameters. Probably as parenthesis are used for other tasks, like invoking functions or methods, or separating arithmetic operations, that reuse of the signs () for multiple purposes is what caused a different behavior depending on if there is one or more parameters the mayhem IMO.

See some example cases.

Python simplifies the jump of line \n platform independent and some times it’s messy

If you come from a C background you will expect text file in different platforms: Linux, Mac OS X (changes from old to new versions), Windows… to be represented different. In some cases this is an ASCii code 10 (LF), in others 13 (CR), and in other two characters: 13 and immediately after 10.

Python simplifies the Enter character by naming it \n like in C.

So, platform independent, whenever you read a text file you will get \n for any ASCii 10 [LF] or 13 [CR]. [CR] will be converted to [10] in Linux.

If you read a file in a Linux system, where enters are represented by 10, which was generated in a Windows system, so it has [CR][LF] instead of [LF] at the end of each line, you’ll get a \n too, but two times.

And if you do len(“\n”) to know the len of that String, this returns 1 in all the platform.

To read the [LF] and [CR] (represented by \r) you need to open the file as binary. By default Python opens the files as text.

You can check this by writting [LF] and [CR] in Linux and see how Python seamlessly reads the file as it was [LF].

A file generated by Windows will get \n\n:

Random code when the class is imported

In a procedural file, the code that is outside a function, will be executed when it is imported. But if this file is imported again it will not be re-executed.

Things are more messy if you import a class file. Inside the body of the class, in the space you would reserve for static variables definition, you can have random code. And this code will be only executed on the first import, not on subsequent.

Disclaimer: the pictures from Futurama are from their respective owners.

A base Dockerfile for my Jenkins deployments

Update: I’ve created a video and article about how to install jenkins in Docker with docker CLI and Blue Ocean plugins following the official Documentation. You may prefer to follow that one.

Update: Second part of this article: Creating Jenkins configurations for your projects

So I share with you my base Jenkins Dockerfile, so you can spawn a new Jenkins for your projects.

The Dockerfile installs Ubuntu 20.04 LTS as base image and add the required packages to run jenkins but also Development and Testing tools to use inside the Container to run Unit Testing on your code, for example. So you don’t need external Servers, for instance.

You will need 3 files:

  • Dockerfile
  • docker_run_jenkins.sh
  • requirements.txt

The requirements.txt file contains your PIP3 dependencies. In my case I only have pytest version 4.6.9 which is the default installed with Ubuntu 20.04, however, this way, I enforce that this and not any posterior version will be installed.

File requirements.txt:

pytest==4.6.9

The file docker_run_jenkins.txt start Jenkins when the Container is run and it will wait until the initial Admin password is generated and then it will display it.

File docker_run_jenkins.sh:

#!/bin/bash

echo "Starting Jenkins..."

service jenkins start

echo "Configure jenkins in http://127.0.0.1:8080"

s_JENKINS_PASSWORD_FILE="/var/lib/jenkins/secrets/initialAdminPassword"

i_PASSWORD_PRINTED=0

while [ true ];
do
    sleep 1
    if [ $i_PASSWORD_PRINTED -eq 1 ];
    then
        # We are nice with multitasking
        sleep 60
        continue
    fi

    if [ ! -f "$s_JENKINS_PASSWORD_FILE" ];
    then
        echo "File $s_FILE_ORIGIN does not exist"
    else
        echo "Password for Admin is:"
        cat $s_JENKINS_PASSWORD_FILE
        i_PASSWORD_PRINTED=1
    fi
done

That file has the objective to show you the default admin password, but you don’t need to do that, you can just start a shell into the Container and check manually by yourself.

However I added it to make it easier for you.

And finally you have the Dockerfile:

FROM ubuntu:20.04

LABEL Author="Carles Mateo" \
      Email="jenkins@carlesmateo.com" \
      MAINTAINER="Carles Mateo"

# Build this file with:
# sudo docker build -f Dockerfile -t jenkins:base .
# Run detached:
# sudo docker run --name jenkins_base -d -p 8080:8080 jenkins:base
# Run seeing the password:
# sudo docker run --name jenkins_base -p 8080:8080 -i -t jenkins:base
# After you CTRL + C you will continue with:
# sudo docker start
# To debug:
# sudo docker run --name jenkins_base -p 8080:8080 -i -t jenkins:base /bin/bash

ARG DEBIAN_FRONTEND=noninteractive

ENV SERVICE jenkins

RUN set -ex

RUN echo "Creating directories and copying code" \
    && mkdir -p /opt/${SERVICE}

COPY requirements.txt \
    docker_run_jenkins.sh \
    /opt/${SERVICE}/

# Java with Ubuntu 20.04 LST is 11, which is compatible with Jenkins.
RUN apt update \
    && apt install -y default-jdk \
    && apt install -y wget curl gnupg2 \
    && apt install -y git \
    && apt install -y python3 python3.8-venv python3-pip \
    && apt install -y python3-dev libsasl2-dev libldap2-dev libssl-dev \
    && apt install -y python3-venv \
    && apt install -y python3-pytest \
    && apt install -y sshpass \
    && wget -qO - https://pkg.jenkins.io/debian-stable/jenkins.io.key | apt-key add - \
    && echo "deb http://pkg.jenkins.io/debian-stable binary/" > /etc/apt/sources.list.d/jenkins.list \
    && apt update \
    && apt -y install jenkins \
    && apt-get clean

RUN echo "Setting work directory and listening port"
WORKDIR /opt/${SERVICE}

RUN chmod +x docker_run_jenkins.sh

RUN pip3 install --upgrade pip \
    && pip3 install -r requirements.txt


EXPOSE 8080


ENTRYPOINT ["./docker_run_jenkins.sh"]

Build the Container

docker build -f Dockerfile -t jenkins:base .

Run the Container displaying the password

sudo docker run --name jenkins_base -p 8080:8080 -i -t jenkins:base

You need this password for starting the configuration process through the web.

Visit http://127.0.0.1:8080 to configure Jenkins.

Configure as usual

Resuming after CTRL + C

After you configured it, on the terminal, press CTRL + C.

And continue, detached, by running:

sudo docker start jenkins_base

The image is 1.2GB in size, and will allow you to run Python3, Virtual Environments, Unit Testing with pytest and has Java 11 (not all versions of Java are compatible with Jenkins), use sshpass to access other Servers via SSH with Username and Password…

A live session refactoring and adding Unit Testing to my Python3 project cmemgzip

I refactor and add unit testing to my actual project cmemgzip while I comment every step so you can learn the whys and the reasoning.

Open video in full screen with max 1080 quality, in order to see the code clearly.

Update 2021-03-03: I added a third part. I’m tired but still is worth watching.