If accidentally you removed PIP from your windows machine, or you attempted a PIP upgrade which failed after removing the current version, and let you unable to install it anymore, you can address it this way.
python -m ensurepip --default-pipAfter this, you can upgrade it:
python -m install --upgrade pip setuptools wheel
I refactor and add unit testing to my actual project cmemgzip while I comment every step so you can learn the whys and the reasoning.
Open video in full screen with max 1080 quality, in order to see the code clearly.
Update 2021-03-03: I added a third part. I’m tired but still is worth watching.
So you are trying to program the Raspberry expansion PINS in Python, for example for this 3D LED Christmas Tree, and you’re getting the error:
GPIO.setup(self.number, GPIO.IN, self.GPIO_PULL_UPS[self._pull])
RuntimeError: Not running on a RPi!
I’m running this on Ubuntu 20.04LTS with a Raspberry 4.
The first thing:
Or at least, make sure your USB charger provides enough intensity to power the Raspberry and the LEDs.
The LED power comes from the motherboard and if Raspberry Pi has not enough energy this is not going to work.
My colleague Michela had her tree not working because of the charger was not able to provide enough energy. When she ordered a new charger, it worked like a charm.
In order to communicate with General Purpose Input Output ports (GPIO) you need to install this Software:
sudo apt install python3-pip python3-gpiozero
sudo pip3 install giozerosudo pip3 install colorzero
sudo pip3 install rpi.gpio --upgradeIt may be not required in some cases.
https://github.com/ThePiHut/rgbxmastree#rgbxmastree
I saw many people stuck, in the forums, because of that.
To work with the LEDs you need to run the samples as root.
To provide a bit of “the whole package” here are some simple examples.
from tree import RGBXmasTree
from time import sleep
o_tree = RGBXmasTree()
for o_pixel in o_tree:
o_pixel.color = (1, 0, 0)
sleep(0.1)
from tree import RGBXmasTree
from time import sleep
o_tree = RGBXmasTree()
a_t_colors = [(1, 0, 0), (0, 1, 0), (0, 0, 1)]
for t_color in a_t_colors:
o_tree.color = t_color
sleep(1)
from tree import RGBXmasTree o_tree = RGBXmasTree() o_tree.color = (0, 0, 0)
I used it in my Open Source monitor Software CTOP.py in order to show the plugins/extensions capability of it. :)
Happy New Year to all.
Is something very simple, but will help my student friends to validate Input from Keyboard without losing too many hours.
The Input Validation Classes I create in PHP for Privalia or in my PHP Catalonia Framework, are much, much, more powerful, allowing the validation of complete forms, rendering errors, etc… although they were created for Web, and not for Keyboard input.
It recursively goes to all the subdirectories looking for .py files, and then it counts the lines.
So people can benefit from this during the lock down.
I cannot share it, but I guess that at some point they will publish it on their blog:
I’m teaching a friend how to code. Covid-19 has affected many people, with many jobs loss in many sectors. I encouraged him to learn to code and I helped him to access to a course offered by the Catalan government, but they let him down with a really difficult exam in the very last moment.
I encouraged him to continue and offered to help him, and after a start with several courses via Internet, that were not very useful, I started teaching and training him personally. And I did the way the companies need: Python, PyCharm, Git, Linux, VirtualBox, Web…
I told him since the beginning that I can help him, yes, but 90% has to come from his effort. I’m really happy to see that he is really doing the effort and how much he advanced.
Was not easy as I had to combine it with the classes in the university, my work, the Open Source projects, and my life… but I’m proud of him and of the effort he is doing.
Today he remembered me how shocked he was when I showed some Python builtin libraries, like datetime, and he saw in PyCharm, that the code of that library alone, has 2,476 lines.
I wanted to explain that we can get to create a program with thousands of lines of code, easily, so I decided to write a small program to count the number of lines of our programs.
I quickly wrote it in bash in one single line:
i_counter=0; for s_filename in `find . -name "*.py"`; do wc --lines $s_filename; i_num=`cat $s_filename | wc --lines`; i_counter=$((i_counter+i_num)); done; echo "Total lines: $i_counter"Execute it from inside your directory.
It can be improved and optimized very easily.
I made some improvements abovementioned and uploaded this as a several lines script:
Please note this script does not only run in Linux, it can also be executed in Windows using GitBash.
For my Open Source project CTOP.py it’s 4,652 lines of Python code. :)
That’s one of the problems with Python. Blocks of code are defined by their indentation position.
That’s a pain when you copy and past and the IDE reindents the code thinking that is doing great, or generate a new inner class instead of replacing all the code.
Well, this error is very annoying cause it means that you mixed spaces and Tabs as indent separators.
But you can go crazy trying to find a tab in your code, so there is a trick that I came with:
Basically go to Menu Edit > Find and then type 4 times space. PyCharm will highlight all the places were this indentation (4 spaces) is present, so you’ll find the impostor without going blind or losing to many time.
As you can see, in front of def execute_command_without_waiting we don’t have 4 spaces. And in this case the impostor was not a camouflaged tab \t but 3 spaces instead of four.
If you are using Git Submodules, is very probable that at some point you will create you own libraries. Probably those libraries will have their own structure, even with their own tests/ folder and you’re adding into a subfolder into your new project and maybe you have problems using relative imports.
This is a trick you can use to add the relevant root folder of your project to the System Path, so the libraries are found, specially when you call by command line from anywhere in the filesystem. This works for Python2 and Python3.
#!/usr/bin/env python3
import sys
import os
s_path_program = os.path.dirname(__file__)
sys.path.append(s_path_program + '../../')
from clib.src.argsutils import ArgsUtils
from clib.src.datetimeutils import DateTimeUtils
from clib.src.fileutils import FileUtilsThis sample can be found in my book Pythom Combat Guide.
I wanted to automate certain operations that we do very often, and so I decided to do a PoC of how handy will it be to create GUI applications that can automate tasks.
As locating information in several repositories of information (ldap, databases, websites, etc…) can be tedious I decided to create a small program that queries LDAP for the information I’m interested, in this case a Location. This small program can very easily escalated to launch the VPN, to query a Database after querying LDAP if no results are found, etc…
I share with you the basic application as you may find interesting to create GUI applications in Python, compatible with Windows, Linux and Mac.
I’m super Linux fan but this is important, as many multinationals still use Windows or Mac even for Engineers and SRE positions.
With the article I provide a Dockerfile and a docker-compose.yml file that will launch an OpenLDAP Docker Container preloaded with very basic information and a PHPLDAPMIN Container.
Ubuntu:
sudo apt-get install python3.8 python3-tk pip install ldap3
Windows and Mac:
Install Python3.6 or greater and from command line.
For Mac install pip if you don’t have it.
pip install ldap3
#!/bin/env python3
import tkinter as tk
from ldap3 import Server, Connection, MODIFY_ADD, MODIFY_REPLACE, ALL_ATTRIBUTES, ObjectDef, Reader
from ldap3.utils.dn import safe_rdn
from lib.fileutils import FileUtils
# sudo apt-get install python3.8 python3-tk
# pip install ldap3
class LDAPGUI:
s_config_file = "config.cfg"
s_ldap_server = "ldapslave01"
s_connection = "uid=%USERNAME%, cn=users, cn=accounts, dc=demo1, dc=carlesmateo, dc=com"
s_username = "carlesmateo"
s_password = "Secret123"
s_query = "location=%LOCATION%,dc=demo1,dc=carlesmateo,dc=com"
i_window_width = 610
i_window_height = 650
i_frame_width = i_window_width-5
i_frame_height = 30
i_frame_results_height = 400
# Graphical objects
o_window = None
o_entry_ldap = None
o_entry_connection = None
o_entry_results = None
o_entry_username = None
o_entry_location = None
o_button_search = None
def __init__(self, o_fileutils):
self.o_fileutils = o_fileutils
def replace_connection_with_username(self):
s_connection_raw = self.o_entry_username.get()
s_connection = self.s_connection.replace("%USERNAME%", s_connection_raw)
return s_connection
def replace_query_with_location(self):
s_query_raw = self.o_entry_location.get()
s_query = self.s_query.replace("%LOCATION%", s_query_raw)
return s_query
def clear_results(self):
self.o_entry_results.delete("1.0", tk.END)
def enable_button(self):
self.o_button_search["state"] = tk.NORMAL
def disable_button(self):
self.o_button_search["state"] = tk.DISABLED
def ldap_run_query(self):
self.disable_button()
self.clear_results()
s_ldap_server = self.o_entry_ldap.get()
self.o_entry_results.insert(tk.END, "Connecting to: " + s_ldap_server + "...\n")
try:
o_server = Server(s_ldap_server)
o_conn = Connection(o_server,
self.replace_connection_with_username(),
self.s_password,
auto_bind=False)
o_conn.bind()
if isinstance(o_conn.last_error, str):
s_last_error = o_conn.last_error
self.o_entry_results.insert(tk.END, "Last error: " + s_last_error + "\n")
if isinstance(o_conn.result, str):
s_conn_result = o_conn.result
self.o_entry_results.insert(tk.END, "Connection result: " + s_conn_result + "\n")
s_query = self.replace_query_with_location()
self.o_entry_results.insert(tk.END, "Performing Query: " + s_query + "\n")
obj_location = ObjectDef('organizationalUnit', o_conn)
r = Reader(o_conn, obj_location, s_query)
r.search()
self.o_entry_results.insert(tk.END, "Results: \n" + str(r) + "\n")
#print(r)
#print(0, o_conn.extend.standard.who_am_i())
# https://github.com/cannatag/ldap3/blob/master/tut1.py
# https://github.com/cannatag/ldap3/blob/master/tut2.py
except:
self.o_entry_results.insert(tk.END, "There has been a problem" + "\n")
if isinstance(o_conn.last_error, str):
s_last_error = o_conn.last_error
self.o_entry_results.insert(tk.END, "Last error: " + s_last_error + "\n")
try:
self.o_entry_results.insert(tk.END, "Closing connection\n")
o_conn.unbind()
except:
self.o_entry_results.insert(tk.END, "Problem closing connection" + "\n")
self.enable_button()
def create_frame_location(self, o_window, i_width, i_height):
o_frame = tk.Frame(master=o_window, width=i_width, height=i_height)
o_frame.pack()
o_lbl_location = tk.Label(master=o_frame,
text="Location:",
width=10,
height=1)
o_lbl_location.place(x=0, y=0)
o_entry = tk.Entry(master=o_frame, fg="yellow", bg="blue", width=50)
o_entry.insert(tk.END, "")
o_entry.place(x=100, y=0)
return o_frame, o_entry
def create_frame_results(self, o_window, i_width, i_height):
o_frame = tk.Frame(master=o_window, width=i_width, height=i_height)
o_frame.pack()
o_entry = tk.Text(master=o_frame, fg="grey", bg="white", width=75, height=20)
o_entry.insert(tk.END, "")
o_entry.place(x=0, y=0)
return o_frame, o_entry
def create_frame_username(self, o_window, i_width, i_height, s_username):
o_frame = tk.Frame(master=o_window, width=i_width, height=i_height)
o_frame.pack()
o_lbl_location = tk.Label(master=o_frame,
text="Username:",
width=10,
height=1)
o_lbl_location.place(x=0, y=0)
o_entry_username = tk.Entry(master=o_frame, fg="yellow", bg="blue", width=50)
o_entry_username.insert(tk.END, s_username)
o_entry_username.place(x=100, y=0)
return o_frame, o_entry_username
def create_frame_ldapserver(self, o_window, i_width, i_height, s_server):
o_frame = tk.Frame(master=o_window, width=i_width, height=i_height)
o_frame.pack()
o_lbl_ldapserver = tk.Label(master=o_frame,
text="LDAP Server:",
width=10,
height=1)
o_lbl_ldapserver.place(x=0, y=0)
# We don't need to pack the label, as it is inside a Frame, packet
# o_lbl_ldapserver.pack()
o_entry = tk.Entry(master=o_frame, fg="yellow", bg="blue", width=50)
o_entry.insert(tk.END, s_server)
o_entry.place(x=100, y=0)
return o_frame, o_entry
def create_frame_connection(self, o_window, i_width, i_height, s_connection):
o_frame = tk.Frame(master=o_window, width=i_width, height=i_height)
o_frame.pack()
o_lbl_connection = tk.Label(master=o_frame,
text="Connection:",
width=10,
height=1)
o_lbl_connection.place(x=0, y=0)
# We don't need to pack the label, as it is inside a Frame, packet
# o_lbl_ldapserver.pack()
o_entry_connection = tk.Entry(master=o_frame, fg="yellow", bg="blue", width=50)
o_entry_connection.insert(tk.END, s_connection)
o_entry_connection.place(x=100, y=0)
return o_frame, o_entry_connection
def create_frame_query(self, o_window, i_width, i_height, s_query):
o_frame = tk.Frame(master=o_window, width=i_width, height=i_height)
o_frame.pack()
o_lbl_query = tk.Label(master=o_frame,
text="Connection:",
width=10,
height=1)
o_lbl_query.place(x=0, y=0)
o_entry_query = tk.Entry(master=o_frame, fg="yellow", bg="blue", width=50)
o_entry_query.insert(tk.END, s_query)
o_entry_query.place(x=100, y=0)
return o_frame, o_entry_query
def create_button(self):
o_button = tk.Button(
text="Search",
width=25,
height=1,
bg="blue",
fg="yellow",
command=self.ldap_run_query
)
o_button.pack()
return o_button
def render_screen(self):
o_window = tk.Tk()
o_window.title("LDAPGUI by Carles Mateo")
self.o_window = o_window
self.o_window.geometry(str(self.i_window_width) + 'x' + str(self.i_window_height))
o_frame_ldap, o_entry_ldap = self.create_frame_ldapserver(o_window=o_window,
i_width=self.i_frame_width,
i_height=self.i_frame_height,
s_server=self.s_ldap_server)
self.o_entry_ldap = o_entry_ldap
o_frame_connection, o_entry_connection = self.create_frame_connection(o_window=o_window,
i_width=self.i_frame_width,
i_height=self.i_frame_height,
s_connection=self.s_connection)
self.o_entry_connection = o_entry_connection
o_frame_user, o_entry_user = self.create_frame_username(o_window=o_window,
i_width=self.i_frame_width,
i_height=self.i_frame_height,
s_username=self.s_username)
self.o_entry_username = o_entry_user
o_frame_query, o_entry_query = self.create_frame_query(o_window=o_window,
i_width=self.i_frame_width,
i_height=self.i_frame_height,
s_query=self.s_query)
self.o_entry_query = o_entry_query
o_frame_location, o_entry_location = self.create_frame_location(o_window=o_window,
i_width=self.i_frame_width,
i_height=self.i_frame_height)
self.o_entry_location = o_entry_location
o_button_search = self.create_button()
self.o_button_search = o_button_search
o_frame_results, o_entry_results = self.create_frame_results(o_window=o_window,
i_width=self.i_frame_width,
i_height=self.i_frame_results_height)
self.o_entry_results = o_entry_results
o_window.mainloop()
def load_config_values(self):
b_success, d_s_config = self.o_fileutils.read_config_file_values(self.s_config_file)
if b_success is True:
if 'server' in d_s_config:
self.s_ldap_server = d_s_config['server']
if 'connection' in d_s_config:
self.s_connection = d_s_config['connection']
if 'username' in d_s_config:
self.s_username = d_s_config['username']
if 'password' in d_s_config:
self.s_password = d_s_config['password']
if 'query' in d_s_config:
self.s_query = d_s_config['query']
def main():
o_fileutils = FileUtils()
o_ldapgui = LDAPGUI(o_fileutils=o_fileutils)
o_ldapgui.load_config_values()
o_ldapgui.render_screen()
if __name__ == "__main__":
main()
FROM osixia/openldap
# Note: Docker compose will generate the Docker Image.
# Run: sudo docker-compose up -d
LABEL maintainer="Carles Mateo"
ENV LDAP_ORGANISATION="Carles Mateo Test Org" \
LDAP_DOMAIN="carlesmateo.com"
COPY bootstrap.ldif /container/service/slapd/assets/config/bootstrap/ldif/50-bootstrap.ldif
version: '3.3'
services:
ldap_server:
build:
context: .
dockerfile: Dockerfile
environment:
LDAP_ADMIN_PASSWORD: test1234
LDAP_BASE_DN: dc=carlesmateo,dc=com
ports:
- 389:389
volumes:
- ldap_data:/var/lib/ldap
- ldap_config:/etc/ldap/slapd.d
ldap_server_admin:
image: osixia/phpldapadmin:0.7.2
ports:
- 8090:80
environment:
PHPLDAPADMIN_LDAP_HOSTS: ldap_server
PHPLDAPADMIN_HTTPS: 'false'
volumes:
ldap_data:
ldap_config:
# Config File for LDAPGUI by Carles Mateo
# https://blog.carlesmateo.com
#
# Configuration for working with the prepared Docker
server=127.0.0.1:389
connection=cn=%USERNAME%,dc=carlesmateo,dc=com
username=admin_gh
password=admin_gh_pass
query=cn=%LOCATION%,dc=carlesmateo,dc=com
# Carles other tests
#server=ldapslave.carlesmateo.com
#connection=uid=%USERNAME%, cn=users, cn=accounts, dc=demo1, dc=carlesmateo, dc=com
#username=carlesmateo
#password=Secret123
#query=location=%LOCATION%,dc=demo1,dc=carlesmateo,dc=com
In order to bootstrap the Data on the LDAP Server I use a bootstrap.ldif file copied into the Server.
Credits, I learned this trick in this page: https://medium.com/better-programming/ldap-docker-image-with-populated-users-3a5b4d090aa4
dn: cn=developer,dc=carlesmateo,dc=com
changetype: add
objectclass: Engineer
cn: developer
givenname: developer
sn: Developer
displayname: Developer User
mail: developer@carlesmateo.com
userpassword: developer_pass
dn: cn=maintainer,dc=carlesmateo,dc=com
changetype: add
objectclass: Engineer
cn: maintainer
givenname: maintainer
sn: Maintainer
displayname: Maintainer User
mail: maintainer@carlesmateo.com
userpassword: maintainer_pass
dn: cn=admin,dc=carlesmateo,dc=com
changetype: add
objectclass: Engineer
cn: admin
givenname: admin
sn: Admin
displayname: Admin
mail: admin@carlesmateo.com
userpassword: admin_pass
dn: ou=Groups,dc=carlesmateo,dc=com
changetype: add
objectclass: organizationalUnit
ou: Groups
dn: ou=Users,dc=carlesmateo,dc=com
changetype: add
objectclass: organizationalUnit
ou: Users
dn: cn=Admins,ou=Groups,dc=carlesmateo,dc=com
changetype: add
cn: Admins
objectclass: groupOfUniqueNames
uniqueMember: cn=admin,dc=carlesmateo,dc=com
dn: cn=Maintaners,ou=Groups,dc=carlesmateo,dc=com
changetype: add
cn: Maintaners
objectclass: groupOfUniqueNames
uniqueMember: cn=maintainer,dc=carlesmateo,dc=com
uniqueMember: cn=developer,dc=carlesmateo,dc=com
You can download this file from the lib folder in my project CTOP.py
More information about programming with tkinter:
https://realpython.com/python-gui-tkinter/#getting-user-input-with-entry-widgets
https://www.tutorialspoint.com/python/python_gui_programming.htm
https://effbot.org/tkinterbook/entry.htm
About LDAP:
This is a trick to restart a Service that is running on a immutable Docker, with some change, and you need to refresh the values very quickly without having to roll the CI/CD Jenkins Pipeline and uploading a new image.
So why would you need to do that?.
I can think about possible scenarios like:
FROM os:latest
The ideal is that if you work with immutable images, you roll out a new immutable image and that’s it.
But if for whatever reason you need to update this super fast, this trick may become really handy.
Let’s go for it!.
Normally you’ll start your container with a command similar to this:
docker run -d --rm -p 5000:5000 api_carlesmateo_com:v7 prod
The first thing we have to do is to stop the container.
So:
docker ps
Locate your container across the list of running containers and stop it, and then restart without the –rm:
docker stop container_name
docker run -d -p 5000:5000 api_carlesmateo_com:v7 prod
the –rm makes the container to cleanup. By default a container’s file system persists even after the container exits. So don’t start it with –rm.
Ok, so login to the container:
docker exec -it container_name /bin/sh
Edit the config you require to change, for example config.yml
If what you have to update is a password, and is encoded in base64, encode it:
echo -n "ThePassword" | base64 VGhlUGFzc3dvcmQ=
Stop the container. You can do it by stopping the container with docker stop or from inside the container, killing the listening process, probably a Python Flask.
If your Dockerfile ends with something like:
ENTRYPOINT ["./webservice.py"]
And webservice.py has Python Flask code similar to this:
#!/usr/bin/python3
#
# webservice.py
#
# Author: Carles Mateo
# Creation Date: 2020-05-10 20:50 GMT+1
# Description: A simple Flask Web Application
# Part of the samples of https://leanpub.com/pythoncombatguide
# More source code for the book at https://gitlab.com/carles.mateo/python_combat_guide
#
from flask import Flask, request
import logging
# Initialize Flask
app = Flask(__name__)
# Sample route so http://127.0.0.1/carles
@app.route('/carles', methods=['GET'])
def carles():
logging.critical("A connection was established")
return "200"
logging.info("Initialized...")
if __name__ == "__main__":
app.run(host='0.0.0.0', port=5000, debug=True)
Then you can kill the process, and so ending the container, from inside the container by doing:
ps -ax | grep webservice
5750 root 56:31 {webservice.py} /usr/bin/python /opt/webservice/webservice.py
kill -9 5790
This will finish the container the same way as docker stop container_name.
Then start the container (not run)
docker start container_name
You can now test from outside or from inside the container. If from inside:
/opt/webservice # wget localhost:5000/carles Connecting to localhost:5000 (127.0.0.1:5000) carles 100% |**************************************************************************************************************| 3 0:00:00 ETA /opt/webservice # cat debug.log 2020-05-06 20:46:24,349 Initialized... 2020-05-06 20:46:24,359 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit) 2020-05-06 20:46:24,360 * Restarting with stat 2020-05-06 20:46:24,764 Initialized... 2020-05-06 20:46:24,771 * Debugger is active! 2020-05-06 20:46:24,772 * Debugger PIN: 123-456-789 2020-05-07 13:18:43,890 127.0.0.1 - - [07/May/2020 13:18:43] "GET /carles HTTP/1.1" 200 -
if you don’t use YAML files or what you need is to change the code, all this can be avoided as when you update the Python code, Flash realizes that and reloads. See this line in the logs:
2020-05-07 13:18:40,431 * Detected change in '/opt/webservice/wwebservice.py', reloadingThe webservice.py autoreloads because we init Flask with debug set to on.
You can also start a container with shell directly:
sudo docker run -it ctop /bin/bash
After some work reviewing it and ensuring it has the expected quality, I finally published my book Python Combat Guide.
Is an atypical creation. Is more a Master Class to my best friend, it could be a SDM, TL leading a small Software Development department, a Coder or a Scientist wanting to join IT as programmer and to learn a lot of stuff very quickly, than rather a formal Python Book for learning. Absolutely is not for beginners.
If you want to buy it, to explore the TOC, extended description…