Tag Archives: Python

Why I propose you to use Python 3.8, at least, for your Internal Automation Tools in Docker Containers

This article is written at 2021-03-22 so this conclusion will evolve as time passes.

Some of my articles are checked after 7 years, so be advised this choice will not be valid in a year. Although the reasoning and considerations to take in count will be the same.

I answer to the question: Why Carles, do you suggest to adopt Python 3.8, and not 3.9 or 3.7 for our Internal Automation Tools?.

Reliability and Maturity

If you look at page https://devguide.python.org/#status-of-python-branches you will see the next table:

So you can see that:

  • Python 3.6 was released on 2016-12-23 and will get EOL on 2021-12-23.
    • That’s EOL in 9 months. We don’t want to recommend that.
  • Python 3.7 was released on 2018-06-27 and will get EOL 2023-06-27.
    • That’s 2 years and 3 months from now. The Status of development is focus in Security bugfixes.
  • Python 3.9 was released 2020-10-05 that’s 5 months approx from now.
    • Honestly, I don’t recommend for Production a version of Software that has not been in the market for a year.
      • Most of the bugs and security bugs appears before the first year.
      • New features released, often are not widely fully tested , and bugs found and fixed, once a year has passed.
  • Python 3.8 was released on 2019-10-14.
    • That means that the new features have been tested for a year and five months approximately.
    • This is enough time to make appear most bugs.
    • EOL is 2024-10, that is 3 years and 7 months from now. A good balance of EOL for the effort to standardize.
    • Finally Python 3.8 is the Python mainline for Ubuntu 20.04 LTS.
      • If our deploy strategy is synchronized, we want to use Long Time Support versions, of course.

So my recommendation would be, at least for your internal tools, to use containers based in Ubuntu 20.04 LTS with Python 3.8.

We know Docker images will be bigger using Ubuntu 20.04 LTS than using other images, but that disk space is really a small difference, and we get the advantage of being able to install additional packages in the Containers if we need to debug.

A simple script to upload a PIP package

If you want to create a package and distribute it like through pip in record time, you can customize my script from cmemgzip for Ubuntu 20.04.

Here is the official documentation if you want to do everything manually:

https://packaging.python.org/tutorials/packaging-projects

Here is the script customized for Test Environment.

You’ll need to create a Test account in test.pypi.org

#!/bin/bash

PACKAGE="cmemgzip-test"
mkdir $PACKAGE
mkdir $PACKAGE/src
mkdir $PACKAGE/src/$PACKAGE
mkdir $PACKAGE/tests

cp LICENSE $PACKAGE/

echo "[build-system]" > $PACKAGE/pyproject.toml
echo "requires = [" >> $PACKAGE/pyproject.toml
echo '    "setuptools>=42",' >> $PACKAGE/pyproject.toml
echo '    "wheel"' >> $PACKAGE/pyproject.toml
echo "]" >> $PACKAGE/pyproject.toml
echo 'build-backend = "setuptools.build_meta"' >> $PACKAGE/pyproject.toml

cat <<EOF > $PACKAGE/setup.cfg
[metadata]
name = cmemgzip
version = 0.4.1
author = Carles Mateo
author_email = cmemgzip@carlesmateo.com
description = Compresses files in memory and replaces the original by a .gz file when there is no space on drive.
long_description = file: README.md
long_description_content_type = text/markdown
url = https://gitlab.com/carles.mateo/cmemgzip
project_urls =
    Bug Tracker = https://gitlab.com/carles.mateo/cmemgzip/issues
classifiers =
    Programming Language :: Python :: 3
    License :: OSI Approved :: MIT License
    Operating System :: OS Independent

[options]
package_dir =
    = src
packages = find:
python_requires = >=3.6

[options.packages.find]
where = src
EOF

cp README.md $PACKAGE/
cp manual-cmemgzip.pdf $PACKAGE/
cp cmemgzip.py $PACKAGE/src/$PACKAGE/
touch $PACKAGE/src/$PACKAGE/__init__.py
cp test_*.py $PACKAGE/tests/


python3 -m pip install --upgrade build

# Install dependencies
sudo apt-get install python3.8-venv

python3 -m pip install --user --upgrade twine


echo
echo "Entering into directory $PACKAGE"
cd $PACKAGE

echo
echo "Generating distribution binaries"
python3 -m build

# Create account in: https://test.pypi.org/manage/account/

# Create API Token
# https://test.pypi.org/manage/account/

echo
echo "Going to upload the packages. Use your username and password"
echo
python3 -m twine upload --repository testpypi dist/cmemgzip*

The changes for the script to production are just a different package name, and last line:

python3 -m twine upload --repository pypi dist/cmemgzip*

Obviously you’ll need to use credentials for Production.

Reinstall PIP in Windows 10 after it got removed

If accidentally you removed PIP from your windows machine, or you attempted a PIP upgrade which failed after removing the current version, and let you unable to install it anymore, you can address it this way.

python -m ensurepip --default-pip

After this, you can upgrade it:

python -m install --upgrade pip setuptools wheel

Raspberry Pi: Solving the problem GPIO.setup(self.number, GPIO.IN, self.GPIO_PULL_UPS[self._pull]) RuntimeError: Not running on a RPi! in Ubuntu 20.04LTS

So you are trying to program the Raspberry expansion PINS in Python, for example for this 3D LED Christmas Tree, and you’re getting the error:

GPIO.setup(self.number, GPIO.IN, self.GPIO_PULL_UPS[self._pull])
RuntimeError: Not running on a RPi!

I’m running this on Ubuntu 20.04LTS with a Raspberry 4.

The first thing:

Make sure you have an official Raspberry Pi charger.

Or at least, make sure your USB charger provides enough intensity to power the Raspberry and the LEDs.

The LED power comes from the motherboard and if Raspberry Pi has not enough energy this is not going to work.

My colleague Michela had her tree not working because of the charger was not able to provide enough energy. When she ordered a new charger, it worked like a charm.

Install the base Software

In order to communicate with General Purpose Input Output ports (GPIO) you need to install this Software:

sudo apt install python3-pip python3-gpiozero
sudo pip3 install giozero

In order to run the 3D LED Christmas Tree code samples

sudo pip3 install colorzero
sudo pip3 install rpi.gpio --upgrade

Reboot

It may be not required in some cases.

Download the Source code

https://github.com/ThePiHut/rgbxmastree#rgbxmastree

Run the samples as root

I saw many people stuck, in the forums, because of that.

To work with the LEDs you need to run the samples as root.

Where I used it?

I used it in my Open Source monitor Software CTOP.py in order to show the plugins/extensions capability of it. :)

A trick to see what causes Python error Unindent does not match any outer indentation level with PyCharm

That’s one of the problems with Python. Blocks of code are defined by their indentation position.

That’s a pain when you copy and past and the IDE reindents the code thinking that is doing great, or generate a new inner class instead of replacing all the code.

Well, this error is very annoying cause it means that you mixed spaces and Tabs as indent separators.

But you can go crazy trying to find a tab in your code, so there is a trick that I came with:

Basically go to Menu Edit > Find and then type 4 times space. PyCharm will highlight all the places were this indentation (4 spaces) is present, so you’ll find the impostor without going blind or losing to many time.

As you can see, in front of def execute_command_without_waiting we don’t have 4 spaces. And in this case the impostor was not a camouflaged tab \t but 3 spaces instead of four.

News from the blog 2020-08-19

  • I assisted to the OpenZFS leadership meeting.

I tried to continue following it since I left Sanmina. ZFS is really an amazing Software and it’s lead by an amazing Community of super cool Engineers and companies. I would like to continue contributing ASAP.

I bought some new hard drives in order to work a bit on this. You don’t need to have dedicated hardware if you want to test features. You can run in a VirtualBox or VMWare Workstation.

  • I received more books about DevOps and Python

None is perfect. I see flaws in all of them and bad architecture practices*, however from all I learn interesting things.

*I guess that’s why I wrote my own book :)

You know, I study every day. At least 30 minutes, after work. As part of my healthy routines.

But I also study and learn during the work, as we have time available for this.

I’m very fortunate that Blizzard gives me time every day to study. That’s amazing. They also send us to events paying the ticket, travel, hotel, expenses… now with covid-19 we only go to virtual events, but the company still pay for this and give free days. Is a very nice company.

I use a lot Linux Academy too:

I continue having purchases of my book, and I’m very happy about that. I’m working on improving it and providing more contents and samples going from the scratch, with step by step code samples. From spaghetti code reading CSV files, to OOP with Full Coverage.

  • My application for a Higher degree Computer Science Cloud Computing (Level 8) has been accepted. The Irish government pays me 90% of the degree, and Blizzard will pay me the other 10% after I pass the first year course.

I’m really grateful to this beautiful country, Ireland.

Having an Irish degree is something that brings me an special illusion.

  • I have updated CTOP.py with some interesting features

It allows to pass a fixed width and height for the terminal render. That’s very useful when you run CTOP in a Docker non interactive session, or from a Cron, with the –iterations=1 so the output can be captured programmatically.

  • Jetbrains has provided me with a Free License of all their products, in order to support my work in Open Source projects. That’s very nice. I’m using now mainly PyCharm and PhpStorm.
  • At the beginning of the covid-19 I wrote a simulator in Python. That’s why I was able to anticipate that the number of cases and deaths would be very much higher when nobody around me knew what was going to happen. My first simulations were simple, and the algorithms were growing in complexity until I had a full rich Object Oriented modeler. Maybe I’ll write an article about this someday.
    • I based my data in https://www.worldometers.info/coronavirus/#countries
    • I studied the evolution of several countries and I was working with simulations in Spain until their government started blocking the information and stop providing transparent and accurate metrics.
  • I’m seeing how the covid is affecting and transforming several kind of business:
    • Meetup.com I see meetups with more than 1,000 users closing, as they are no meeting anymore
    • Airlines, obviously
    • Hotels, offering less services
    • Metasearchers and OTAs (Online travel agencies)
    • I can imagine the impact on airbnb
    • Discos, nightclubs are closing doors
    • Restaurants, they will lose the Christmas season (with families and companies doing lunch and dinners)
  • At the same time, other companies are hitting records in sales
  • After doing a Masterclass to some colleagues about Refactor, Code Reliability, Quality, The non-happy path and Unit Testing, I’m preparing some contents that I’ll publish to the Community soon. So far I created this repo, where I added the source code for lesson 0: starting to program in Python videos that I created few months ago to help beginners.

https://gitlab.com/carles.mateo/teach-unit-testing/-/tree/master/lesson0

I also added some contents to lesson 1, where we refactor pure spaghetti code with no error control, to something more elaborated with unit testing and full code coverage. Still procedural, but I will jump to next class in two weeks, where we will move to OOP and Dependency Injection.

  • Here my “Harley” assembled :)

A simple trick to find your Git Submodules imports in Python by adding to Syspath

If you are using Git Submodules, is very probable that at some point you will create you own libraries. Probably those libraries will have their own structure, even with their own tests/ folder and you’re adding into a subfolder into your new project and maybe you have problems using relative imports.

This is a trick you can use to add the relevant root folder of your project to the System Path, so the libraries are found, specially when you call by command line from anywhere in the filesystem. This works for Python2 and Python3.

#!/usr/bin/env python3

import sys
import os

s_path_program = os.path.dirname(__file__)
sys.path.append(s_path_program + '../../')

from clib.src.argsutils import ArgsUtils
from clib.src.datetimeutils import DateTimeUtils
from clib.src.fileutils import FileUtils

This sample can be found in my book Pythom Combat Guide.

LDAPGUI a LDAP GUI program in Python and Tkinter

I wanted to automate certain operations that we do very often, and so I decided to do a PoC of how handy will it be to create GUI applications that can automate tasks.

As locating information in several repositories of information (ldap, databases, websites, etc…) can be tedious I decided to create a small program that queries LDAP for the information I’m interested, in this case a Location. This small program can very easily escalated to launch the VPN, to query a Database after querying LDAP if no results are found, etc…

I share with you the basic application as you may find interesting to create GUI applications in Python, compatible with Windows, Linux and Mac.

I’m super Linux fan but this is important, as many multinationals still use Windows or Mac even for Engineers and SRE positions.

With the article I provide a Dockerfile and a docker-compose.yml file that will launch an OpenLDAP Docker Container preloaded with very basic information and a PHPLDAPMIN Container.

Installation of the dependencies

Ubuntu:

sudo apt-get install python3.8 python3-tk
pip install ldap3

Windows and Mac:

Install Python3.6 or greater and from command line.

For Mac install pip if you don’t have it.

pip install ldap3

Python Code

#!/bin/env python3

import tkinter as tk
from ldap3 import Server, Connection, MODIFY_ADD, MODIFY_REPLACE, ALL_ATTRIBUTES, ObjectDef, Reader
from ldap3.utils.dn import safe_rdn
from lib.fileutils import FileUtils

# sudo apt-get install python3.8 python3-tk
# pip install ldap3


class LDAPGUI:

    s_config_file = "config.cfg"

    s_ldap_server = "ldapslave01"
    s_connection = "uid=%USERNAME%, cn=users, cn=accounts, dc=demo1, dc=carlesmateo, dc=com"
    s_username = "carlesmateo"
    s_password = "Secret123"
    s_query = "location=%LOCATION%,dc=demo1,dc=carlesmateo,dc=com"

    i_window_width = 610
    i_window_height = 650
    i_frame_width = i_window_width-5
    i_frame_height = 30
    i_frame_results_height = 400

    # Graphical objects
    o_window = None
    o_entry_ldap = None
    o_entry_connection = None
    o_entry_results = None
    o_entry_username = None
    o_entry_location = None
    o_button_search = None

    def __init__(self, o_fileutils):
        self.o_fileutils = o_fileutils

    def replace_connection_with_username(self):
        s_connection_raw = self.o_entry_username.get()
        s_connection = self.s_connection.replace("%USERNAME%", s_connection_raw)
        return s_connection

    def replace_query_with_location(self):
        s_query_raw = self.o_entry_location.get()
        s_query = self.s_query.replace("%LOCATION%", s_query_raw)
        return s_query

    def clear_results(self):
        self.o_entry_results.delete("1.0", tk.END)

    def enable_button(self):
        self.o_button_search["state"] = tk.NORMAL

    def disable_button(self):
        self.o_button_search["state"] = tk.DISABLED

    def ldap_run_query(self):

        self.disable_button()
        self.clear_results()

        s_ldap_server = self.o_entry_ldap.get()
        self.o_entry_results.insert(tk.END, "Connecting to: " + s_ldap_server + "...\n")
        try:

            o_server = Server(s_ldap_server)
            o_conn = Connection(o_server,
                                self.replace_connection_with_username(),
                                self.s_password,
                                auto_bind=False)
            o_conn.bind()

            if isinstance(o_conn.last_error, str):
                s_last_error = o_conn.last_error
                self.o_entry_results.insert(tk.END, "Last error: " + s_last_error + "\n")

            if isinstance(o_conn.result, str):
                s_conn_result = o_conn.result
                self.o_entry_results.insert(tk.END, "Connection result: " + s_conn_result + "\n")

            s_query = self.replace_query_with_location()
            self.o_entry_results.insert(tk.END, "Performing Query: " + s_query + "\n")

            obj_location = ObjectDef('organizationalUnit', o_conn)
            r = Reader(o_conn, obj_location, s_query)
            r.search()
            self.o_entry_results.insert(tk.END, "Results: \n" + str(r) + "\n")
            #print(r)

            #print(0, o_conn.extend.standard.who_am_i())
            # https://github.com/cannatag/ldap3/blob/master/tut1.py
            # https://github.com/cannatag/ldap3/blob/master/tut2.py

        except:
            self.o_entry_results.insert(tk.END, "There has been a problem" + "\n")

            if isinstance(o_conn.last_error, str):
                s_last_error = o_conn.last_error
                self.o_entry_results.insert(tk.END, "Last error: " + s_last_error + "\n")

        try:
            self.o_entry_results.insert(tk.END, "Closing connection\n")
            o_conn.unbind()
        except:
            self.o_entry_results.insert(tk.END, "Problem closing connection" + "\n")

        self.enable_button()

    def create_frame_location(self, o_window, i_width, i_height):
        o_frame = tk.Frame(master=o_window, width=i_width, height=i_height)
        o_frame.pack()

        o_lbl_location = tk.Label(master=o_frame,
                                  text="Location:",
                                  width=10,
                                  height=1)
        o_lbl_location.place(x=0, y=0)

        o_entry = tk.Entry(master=o_frame, fg="yellow", bg="blue", width=50)
        o_entry.insert(tk.END, "")
        o_entry.place(x=100, y=0)

        return o_frame, o_entry

    def create_frame_results(self, o_window, i_width, i_height):
        o_frame = tk.Frame(master=o_window, width=i_width, height=i_height)
        o_frame.pack()

        o_entry = tk.Text(master=o_frame, fg="grey", bg="white", width=75, height=20)
        o_entry.insert(tk.END, "")
        o_entry.place(x=0, y=0)

        return o_frame, o_entry

    def create_frame_username(self, o_window, i_width, i_height, s_username):
        o_frame = tk.Frame(master=o_window, width=i_width, height=i_height)
        o_frame.pack()

        o_lbl_location = tk.Label(master=o_frame,
                                  text="Username:",
                                  width=10,
                                  height=1)
        o_lbl_location.place(x=0, y=0)

        o_entry_username = tk.Entry(master=o_frame, fg="yellow", bg="blue", width=50)
        o_entry_username.insert(tk.END, s_username)
        o_entry_username.place(x=100, y=0)

        return o_frame, o_entry_username

    def create_frame_ldapserver(self, o_window, i_width, i_height, s_server):
        o_frame = tk.Frame(master=o_window, width=i_width, height=i_height)
        o_frame.pack()

        o_lbl_ldapserver = tk.Label(master=o_frame,
                                    text="LDAP Server:",
                                    width=10,
                                    height=1)
        o_lbl_ldapserver.place(x=0, y=0)
        # We don't need to pack the label, as it is inside a Frame, packet
        # o_lbl_ldapserver.pack()

        o_entry = tk.Entry(master=o_frame, fg="yellow", bg="blue", width=50)
        o_entry.insert(tk.END, s_server)
        o_entry.place(x=100, y=0)

        return o_frame, o_entry

    def create_frame_connection(self, o_window, i_width, i_height, s_connection):
        o_frame = tk.Frame(master=o_window, width=i_width, height=i_height)
        o_frame.pack()

        o_lbl_connection = tk.Label(master=o_frame,
                                    text="Connection:",
                                    width=10,
                                    height=1)
        o_lbl_connection.place(x=0, y=0)
        # We don't need to pack the label, as it is inside a Frame, packet
        # o_lbl_ldapserver.pack()

        o_entry_connection = tk.Entry(master=o_frame, fg="yellow", bg="blue", width=50)
        o_entry_connection.insert(tk.END, s_connection)
        o_entry_connection.place(x=100, y=0)

        return o_frame, o_entry_connection

    def create_frame_query(self, o_window, i_width, i_height, s_query):
        o_frame = tk.Frame(master=o_window, width=i_width, height=i_height)
        o_frame.pack()

        o_lbl_query = tk.Label(master=o_frame,
                                    text="Connection:",
                                    width=10,
                                    height=1)
        o_lbl_query.place(x=0, y=0)

        o_entry_query = tk.Entry(master=o_frame, fg="yellow", bg="blue", width=50)
        o_entry_query.insert(tk.END, s_query)
        o_entry_query.place(x=100, y=0)

        return o_frame, o_entry_query

    def create_button(self):
        o_button = tk.Button(
            text="Search",
            width=25,
            height=1,
            bg="blue",
            fg="yellow",
            command=self.ldap_run_query
        )

        o_button.pack()

        return o_button

    def render_screen(self):
        o_window = tk.Tk()
        o_window.title("LDAPGUI by Carles Mateo")
        self.o_window = o_window
        self.o_window.geometry(str(self.i_window_width) + 'x' + str(self.i_window_height))

        o_frame_ldap, o_entry_ldap = self.create_frame_ldapserver(o_window=o_window,
                                                                  i_width=self.i_frame_width,
                                                                  i_height=self.i_frame_height,
                                                                  s_server=self.s_ldap_server)
        self.o_entry_ldap = o_entry_ldap
        o_frame_connection, o_entry_connection = self.create_frame_connection(o_window=o_window,
                                                                              i_width=self.i_frame_width,
                                                                              i_height=self.i_frame_height,
                                                                              s_connection=self.s_connection)
        self.o_entry_connection = o_entry_connection

        o_frame_user, o_entry_user = self.create_frame_username(o_window=o_window,
                                                                i_width=self.i_frame_width,
                                                                i_height=self.i_frame_height,
                                                                s_username=self.s_username)
        self.o_entry_username = o_entry_user

        o_frame_query, o_entry_query = self.create_frame_query(o_window=o_window,
                                                               i_width=self.i_frame_width,
                                                               i_height=self.i_frame_height,
                                                               s_query=self.s_query)
        self.o_entry_query = o_entry_query


        o_frame_location, o_entry_location = self.create_frame_location(o_window=o_window,
                                                                        i_width=self.i_frame_width,
                                                                        i_height=self.i_frame_height)

        self.o_entry_location = o_entry_location

        o_button_search = self.create_button()
        self.o_button_search = o_button_search

        o_frame_results, o_entry_results = self.create_frame_results(o_window=o_window,
                                                                     i_width=self.i_frame_width,
                                                                     i_height=self.i_frame_results_height)
        self.o_entry_results = o_entry_results

        o_window.mainloop()

    def load_config_values(self):
        b_success, d_s_config = self.o_fileutils.read_config_file_values(self.s_config_file)
        if b_success is True:
            if 'server' in d_s_config:
                self.s_ldap_server = d_s_config['server']
            if 'connection' in d_s_config:
                self.s_connection = d_s_config['connection']
            if 'username' in d_s_config:
                self.s_username = d_s_config['username']
            if 'password' in d_s_config:
                self.s_password = d_s_config['password']
            if 'query' in d_s_config:
                self.s_query = d_s_config['query']


def main():
    o_fileutils = FileUtils()

    o_ldapgui = LDAPGUI(o_fileutils=o_fileutils)
    o_ldapgui.load_config_values()

    o_ldapgui.render_screen()


if __name__ == "__main__":
    main()

Dockerfile

FROM osixia/openldap

# Note: Docker compose will generate the Docker Image.
# Run:  sudo docker-compose up -d

LABEL maintainer="Carles Mateo"

ENV LDAP_ORGANISATION="Carles Mateo Test Org" \
    LDAP_DOMAIN="carlesmateo.com"

COPY bootstrap.ldif /container/service/slapd/assets/config/bootstrap/ldif/50-bootstrap.ldif

docker-compose.yml

version: '3.3'
services:
  ldap_server:
      build:
        context: .
        dockerfile: Dockerfile
      environment:
        LDAP_ADMIN_PASSWORD: test1234
        LDAP_BASE_DN: dc=carlesmateo,dc=com
      ports:
        - 389:389
      volumes:
        - ldap_data:/var/lib/ldap
        - ldap_config:/etc/ldap/slapd.d
  ldap_server_admin:
      image: osixia/phpldapadmin:0.7.2
      ports:
        - 8090:80
      environment:
        PHPLDAPADMIN_LDAP_HOSTS: ldap_server
        PHPLDAPADMIN_HTTPS: 'false'
volumes:
  ldap_data:
  ldap_config:

config.cfg

# Config File for LDAPGUI by Carles Mateo
# https://blog.carlesmateo.com
#

# Configuration for working with the prepared Docker
server=127.0.0.1:389
connection=cn=%USERNAME%,dc=carlesmateo,dc=com
username=admin_gh
password=admin_gh_pass
query=cn=%LOCATION%,dc=carlesmateo,dc=com

# Carles other tests
#server=ldapslave.carlesmateo.com
#connection=uid=%USERNAME%, cn=users, cn=accounts, dc=demo1, dc=carlesmateo, dc=com
#username=carlesmateo
#password=Secret123
#query=location=%LOCATION%,dc=demo1,dc=carlesmateo,dc=com

bootstrap.ldif

In order to bootstrap the Data on the LDAP Server I use a bootstrap.ldif file copied into the Server.

Credits, I learned this trick in this page: https://medium.com/better-programming/ldap-docker-image-with-populated-users-3a5b4d090aa4

dn: cn=developer,dc=carlesmateo,dc=com
changetype: add
objectclass: Engineer
cn: developer
givenname: developer
sn: Developer
displayname: Developer User
mail: developer@carlesmateo.com
userpassword: developer_pass

dn: cn=maintainer,dc=carlesmateo,dc=com
changetype: add
objectclass: Engineer
cn: maintainer
givenname: maintainer
sn: Maintainer
displayname: Maintainer User
mail: maintainer@carlesmateo.com
userpassword: maintainer_pass

dn: cn=admin,dc=carlesmateo,dc=com
changetype: add
objectclass: Engineer
cn: admin
givenname: admin
sn: Admin
displayname: Admin
mail: admin@carlesmateo.com
userpassword: admin_pass

dn: ou=Groups,dc=carlesmateo,dc=com
changetype: add
objectclass: organizationalUnit
ou: Groups

dn: ou=Users,dc=carlesmateo,dc=com
changetype: add
objectclass: organizationalUnit
ou: Users

dn: cn=Admins,ou=Groups,dc=carlesmateo,dc=com
changetype: add
cn: Admins
objectclass: groupOfUniqueNames
uniqueMember: cn=admin,dc=carlesmateo,dc=com

dn: cn=Maintaners,ou=Groups,dc=carlesmateo,dc=com
changetype: add
cn: Maintaners
objectclass: groupOfUniqueNames
uniqueMember: cn=maintainer,dc=carlesmateo,dc=com
uniqueMember: cn=developer,dc=carlesmateo,dc=com

lib/fileutils.py

You can download this file from the lib folder in my project CTOP.py

More information about programming with tkinter:

https://realpython.com/python-gui-tkinter/#getting-user-input-with-entry-widgets

https://www.tutorialspoint.com/python/python_gui_programming.htm

https://effbot.org/tkinterbook/entry.htm

About LDAP:

https://ldap3.readthedocs.io/en/latest/

Refreshing settings in a Docker immutable image with Python and Flask

This is a trick to restart a Service that is running on a immutable Docker, with some change, and you need to refresh the values very quickly without having to roll the CI/CD Jenkins Pipeline and uploading a new image.

So why would you need to do that?.

I can think about possible scenarios like:

  • Need to roll out an urgent fix in a time critical manner
  • Jenkins is broken
  • Somebody screw it on the git master branch
  • Docker Hub is down
  • GitHub is down
  • Your artifactory is down
  • The lines between your jumpbox or workstation and the secure Server are down and you have really few bandwidth
  • You have to fix something critical and you only have a phone with you and SSH only
  • Maybe the Dockerfile had latest, and the latest image has changed
FROM os:latest

The ideal is that if you work with immutable images, you roll out a new immutable image and that’s it.

But if for whatever reason you need to update this super fast, this trick may become really handy.

Let’s go for it!.

Normally you’ll start your container with a command similar to this:

docker run -d --rm -p 5000:5000 api_carlesmateo_com:v7 prod 

The first thing we have to do is to stop the container.

So:

docker ps

Locate your container across the list of running containers and stop it, and then restart without the –rm:

docker stop container_name
docker run -d -p 5000:5000 api_carlesmateo_com:v7 prod

the –rm makes the container to cleanup. By default a container’s file system persists even after the container exits. So don’t start it with –rm.

Ok, so login to the container:

docker exec -it container_name /bin/sh 

Edit the config you require to change, for example config.yml

If what you have to update is a password, and is encoded in base64, encode it:

echo -n "ThePassword" | base64
VGhlUGFzc3dvcmQ=

Stop the container. You can do it by stopping the container with docker stop or from inside the container, killing the listening process, probably a Python Flask.

If your Dockerfile ends with something like:

ENTRYPOINT ["./webservice.py"]

And webservice.py has Python Flask code similar to this:

#!/usr/bin/python3
#
# webservice.py
#
# Author: Carles Mateo
# Creation Date: 2020-05-10 20:50 GMT+1
# Description: A simple Flask Web Application
#              Part of the samples of https://leanpub.com/pythoncombatguide
#              More source code for the book at https://gitlab.com/carles.mateo/python_combat_guide
#


from flask import Flask, request
import logging

# Initialize Flask
app = Flask(__name__)


# Sample route so http://127.0.0.1/carles
@app.route('/carles', methods=['GET'])
def carles():
    logging.critical("A connection was established")
    return "200"

logging.info("Initialized...")

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=5000, debug=True)

Then you can kill the process, and so ending the container, from inside the container by doing:

ps -ax | grep webservice
 5750 root     56:31 {webservice.py} /usr/bin/python /opt/webservice/webservice.py
kill -9 5790

This will finish the container the same way as docker stop container_name.

Then start the container (not run)

docker start container_name

You can now test from outside or from inside the container. If from insise:

/opt/webservice # wget localhost:5000/carles
Connecting to localhost:5000 (127.0.0.1:5000)
carles               100% |**************************************************************************************************************|     3  0:00:00 ETA
/opt/webservice # cat debug.log
2020-05-06 20:46:24,349 Initialized...
2020-05-06 20:46:24,359  * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
2020-05-06 20:46:24,360  * Restarting with stat
2020-05-06 20:46:24,764 Initialized...
2020-05-06 20:46:24,771  * Debugger is active!
2020-05-06 20:46:24,772  * Debugger PIN: 123-456-789
2020-05-07 13:18:43,890 127.0.0.1 - - [07/May/2020 13:18:43] "GET /carles HTTP/1.1" 200 -

if you don’t use YAML files or what you need is to change the code, all this can be avoided as when you update the Python code, Flash realizes that and reloads. See this line in the logs:

2020-05-07 13:18:40,431  * Detected change in '/opt/webservice/wwebservice.py', reloading

You can also start a container with shell directly:

sudo docker run -it ctop /bin/bash

Python Combat Guide published

After some work reviewing it and ensuring it has the expected quality, I finally published my book Python Combat Guide.

Is an atypical creation. Is more a Master Class to my best friend, it could be a SDM, TL leading a small Software Development department, a Coder or a Scientist wanting to join IT as programmer and to learn a lot of stuff very quickly, than rather a formal Python Book for learning. Absolutely is not for beginners.

If you want to buy it, to explore the TOC, extended description…

https://leanpub.com/pythoncombatguide