Is something very simple, but will help my student friends to validate Input from Keyboard without losing too many hours.
The Input Validation Classes I create in PHP for Privalia or in my PHP Catalonia Framework, are much, much, more powerful, allowing the validation of complete forms, rendering errors, etc… although they were created for Web, and not for Keyboard input.
It recursively goes to all the subdirectories looking for .py files, and then it counts the lines.
I updated the price of my books to be the minimum allowed by LeanPub, to $5 USD, and created a bundle of two of them for $7 USD.
So people can benefit from this during the lock down.
I’ve updated the Python Combat Guide book with a sample of using Paramiko Libraries for SSH, and increased the Object Oriented Programing and Unit Testing, sections. I also added some books to the Bibliography.
I’ve read the postmortem initial analysis from Slack’s incident. It’s really interesting.
I cannot share it, but I guess that at some point they will publish it on their blog:
I’m teaching a friend how to code. Covid-19 has affected many people, with many jobs loss in many sectors. I encouraged him to learn to code and I helped him to access to a course offered by the Catalan government, but they let him down with a really difficult exam in the very last moment.
I encouraged him to continue and offered to help him, and after a start with several courses via Internet, that were not very useful, I started teaching and training him personally. And I did the way the companies need: Python, PyCharm, Git, Linux, VirtualBox, Web…
I told him since the beginning that I can help him, yes, but 90% has to come from his effort. I’m really happy to see that he is really doing the effort and how much he advanced.
Was not easy as I had to combine it with the classes in the university, my work, the Open Source projects, and my life… but I’m proud of him and of the effort he is doing.
Today he remembered me how shocked he was when I showed some Python builtin libraries, like datetime, and he saw in PyCharm, that the code of that library alone, has 2,476 lines.
I wanted to explain that we can get to create a program with thousands of lines of code, easily, so I decided to write a small program to count the number of lines of our programs.
I quickly wrote it in bash in one single line:
i_counter=0; for s_filename in `find . -name "*.py"`; do wc --lines $s_filename; i_num=`cat $s_filename | wc --lines`; i_counter=$((i_counter+i_num)); done; echo "Total lines: $i_counter"
Execute it from inside your directory.
It can be improved and optimized very easily.
wc is executed so it prints the lines and the filename, and then is executed as a pipe from cat to capture just the number, so the first one is not really needed. The same can be achieved with echo as we have the variables $s_filename and $i_counter
You can filter out test files or others by using the find options or grep -v, for example to exclude test directories or files.
Number of files can very easily added too.
Using “” for avoiding problems with files with spaces.
The better and most modern $() way to execute, instead of “ can be used.
I made some improvements abovementioned and uploaded this as a several lines script:
That’s one of the problems with Python. Blocks of code are defined by their indentation position.
That’s a pain when you copy and past and the IDE reindents the code thinking that is doing great, or generate a new inner class instead of replacing all the code.
Well, this error is very annoying cause it means that you mixed spaces and Tabs as indent separators.
But you can go crazy trying to find a tab in your code, so there is a trick that I came with:
Basically go to Menu Edit > Find and then type 4 times space. PyCharm will highlight all the places were this indentation (4 spaces) is present, so you’ll find the impostor without going blind or losing to many time.
As you can see, in front of def execute_command_without_waiting we don’t have 4 spaces. And in this case the impostor was not a camouflaged tab \t but 3 spaces instead of four.
If you are using Git Submodules, is very probable that at some point you will create you own libraries. Probably those libraries will have their own structure, even with their own tests/ folder and you’re adding into a subfolder into your new project and maybe you have problems using relative imports.
This is a trick you can use to add the relevant root folder of your project to the System Path, so the libraries are found, specially when you call by command line from anywhere in the filesystem. This works for Python2 and Python3.
s_path_program = os.path.dirname(__file__)
sys.path.append(s_path_program + '../../')
from clib.src.argsutils import ArgsUtils
from clib.src.datetimeutils import DateTimeUtils
from clib.src.fileutils import FileUtils
I wanted to automate certain operations that we do very often, and so I decided to do a PoC of how handy will it be to create GUI applications that can automate tasks.
As locating information in several repositories of information (ldap, databases, websites, etc…) can be tedious I decided to create a small program that queries LDAP for the information I’m interested, in this case a Location. This small program can very easily escalated to launch the VPN, to query a Database after querying LDAP if no results are found, etc…
I share with you the basic application as you may find interesting to create GUI applications in Python, compatible with Windows, Linux and Mac.
I’m super Linux fan but this is important, as many multinationals still use Windows or Mac even for Engineers and SRE positions.
With the article I provide a Dockerfile and a docker-compose.yml file that will launch an OpenLDAP Docker Container preloaded with very basic information and a PHPLDAPMIN Container.
This is a trick to restart a Service that is running on a immutable Docker, with some change, and you need to refresh the values very quickly without having to roll the CI/CD Jenkins Pipeline and uploading a new image.
So why would you need to do that?.
I can think about possible scenarios like:
Need to roll out an urgent fix in a time critical manner
Jenkins is broken
Somebody screw it on the git master branch
Docker Hub is down
GitHub is down
Your artifactory is down
The lines between your jumpbox or workstation and the secure Server are down and you have really few bandwidth
You have to fix something critical and you only have a phone with you and SSH only
Maybe the Dockerfile had latest, and the latest image has changed
The ideal is that if you work with immutable images, you roll out a new immutable image and that’s it.
But if for whatever reason you need to update this super fast, this trick may become really handy.
Let’s go for it!.
Normally you’ll start your container with a command similar to this:
docker run -d --rm -p 5000:5000 api_carlesmateo_com:v7 prod
The first thing we have to do is to stop the container.
Locate your container across the list of running containers and stop it, and then restart without the –rm:
docker stop container_name
docker run -d -p 5000:5000 api_carlesmateo_com:v7 prod
the –rm makes the container to cleanup. By default a container’s file system persists even after the container exits. So don’t start it with –rm.
Ok, so login to the container:
docker exec -it container_name /bin/sh
Edit the config you require to change, for example config.yml
If what you have to update is a password, and is encoded in base64, encode it:
echo -n "ThePassword" | base64
Stop the container. You can do it by stopping the container with docker stop or from inside the container, killing the listening process, probably a Python Flask.
If your Dockerfile ends with something like:
And webservice.py has Python Flask code similar to this:
# Author: Carles Mateo
# Creation Date: 2020-05-10 20:50 GMT+1
# Description: A simple Flask Web Application
# Part of the samples of https://leanpub.com/pythoncombatguide
# More source code for the book at https://gitlab.com/carles.mateo/python_combat_guide
from flask import Flask, request
# Initialize Flask
app = Flask(__name__)
# Sample route so http://127.0.0.1/carles
logging.critical("A connection was established")
if __name__ == "__main__":
app.run(host='0.0.0.0', port=5000, debug=True)
Then you can kill the process, and so ending the container, from inside the container by doing:
After some work reviewing it and ensuring it has the expected quality, I finally published my book Python Combat Guide.
Is an atypical creation. Is more a Master Class to my best friend, it could be a SDM, TL leading a small Software Development department, a Coder or a Scientist wanting to join IT as programmer and to learn a lot of stuff very quickly, than rather a formal Python Book for learning. Absolutely is not for beginners.
If you want to buy it, to explore the TOC, extended description…
Please note: Even if I tried to make it easy, probably there are too many concepts for a non-programmer. Will try to deliver more basic previous knowledge and foundations, so people with zero knowledge don’t feel overwhelmed.
Start by installing Python 3.8 or 3.9 in your computer, and the IDE PyCharm. Install also Git, and create an account in GitLab so you can share code with other people and understand how Git works.
Ok, so you can take a look at my video, and hopefully it makes spark your motivation to learn by yourself. :)
I’ve been asked why I used print(“”) instead of print().
Is a good question. The reason is, when we programmed in Python 2.x the native way was to print without parenthesis, like:
print "Hello World!"
Python 3.x was incompatible with that and requires to use parenthesis, like:
Fortunately Python 2.x accepts also to print using parenthesis. In order to have compatibility within Python 2.x and Python 3.x or for future compatibility we were using always print(“Whatever”) in Python2.
However, there is one difference.
If you user print() or print(“”) in Python3 that will generate an empty line.
In Python 2 print(“”) will generate too an empty line, nevertheless print() in Python2 will print two parenthesis. We don’t want that.
This is illustrated in this screenshot:
So all the people that are at home, closed down for coronavirus, you have a chance now to start learning Python and from there get a live as programmer.
You can download the code for this lesson 0, from:
In order to be able to do more samples, and then being a bit interesting an dynamic, I will introduce here how to get data inputted by the Keyboard.
print("Please enter your name:")
s_name = input()
This will add whatever we type, without the final Enter, to the String variable s_name.
Capturing numbers from Keyboard
How we do to capture a number, like how old are you, in years?.
The same way, and then we convert this to an Integer value. An Integer is a data type which is basically a number, not decimal. Like: 1, 2, 7, 1000 o -5.
print("Please enter your name:")
s_name = input()
print("Please enter your age:")
s_age = input()
# With int() we convert a String to an Integer, as long as it is possible.
# Wit str() we convert a Integer to a String, as long as it is possible.
i_age = int(s_age)
If you enter a number incorrectly and so that cannot be converted, you will get an Exception Error. That is something that happened in a way that was not expected. These error can be trapped, and we will see this later, in the future.
How to capture data from the keyboard with input()
How to convert data entered as String to Integer with int()
How to sum two numbers, like 2 + 3
How to subtract two numbers, like 2 – 3
How to multiply, like 2 * 3
So know, you should be able to solve a basic arithmetic exercise in Hacker Rank:
I’m teaching Unit Testing, Refactors, Quality Code and moving from Procedural to OOP to some colleagues, you can find source code for our classes here (please, be aware that there are some error made on purpose to show why and why not do things and hot to apply proper unit testing)
First you have to understand that Python, Java and PHP are worlds completely different.
In Python you’ll probably use Flask, and listen to the port you want, inside Docker Container.
In PHP you’ll use a Frameworks like Laravel, or Symfony, or Catalonia Framework (my Framework) :) and a repo or many (as the idea is that the change in one microservice cannot break another it is recommended to have one git repo per Service) and split the requests with the API Gateway and Filters (so /billing/ goes to the right path in the right Server, is like rewriting URLs). You’ll rely in Software to split your microservices. Usually you’ll use Docker, but you have to add a Web Server and any other tools, as the source code is not packet with a Web Server and other Dependencies like it is in Java Spring Boot.
In Java you’ll use Spring Cloud and Spring Boot, and every Service will be auto-contained in its own JAR file, that includes Apache Tomcat and all other Dependencies and normally running inside a Docker. Tcp/Ip listening port will be set at start via command line, or through environment. You’ll have many git repositories, one per each Service.
Using many repos, one per Service, also allows to deploy only that repository and to have better security, with independent deployment tokens.
It is not unlikely that you’ll use one language for some of your Services and another for other, as well as a Database or another, as each Service is owner of their data.
In any case, you will be using CI/CD and your pipeline will be something like this:
Pull the latest code for the Service from the git repository
Compile the code (if needed)
Run the Unit and Integration Tests
Compile the service to an executable artifact (f.e. Java JAR with Tomcat server and other dependencies)
Generate a Machine image with your JAR deployed (for Java. Look at Spotify Docker Plugin to Docker build from Maven), or with Apache, PHP, other dependencies, and the code. Normally will be a Docker image. This image will be immutable. You will probably use Dockerhub.
Machine image will be started. Platform test are run.
If platform tests pass, the service is promoted to the next environment (for example Dev -> Test -> PreProd -> Prod), the exact same machine is started in the next environment and platform tests are repeated.
If you work with Microsoft .NET, you’ll probably use Azure DevOps.
We IT Engineers, CTOs and Architects, serve the Business. We have to develop the most flexible approaches and enabling the business to release as fast as their need.
Take in count that Microservices is a tool, a pattern. We will use it to bring more flexibility and speed developing, resilience of the services, and speed and independence deploying. However this comes at a cost of complexity.
Microservices is more related to giving flexibility to the Business, and developing according to the Business Domains. Normally oriented to suite an API. If you have an API that is consumed by third party you will have things like independence of Services (if one is down the others will still function), gradual degradation, being able to scale the Services that have more load only, being able to deploy a new version of a Service which is independent of the rest of the Services, etc… the complexity in the technical solution comes from all this resilience, and flexibility.
If your Dev Team is up to 10 Developers or you are writing just a CRUD Web Application, a PoC, or you are an Startup with a critical Time to Market you probably you will not want to use Microservices approach. Is like killing flies with laser cannons. You can use typical Web services approach, do everything in one single Https request, have transactions, a single Database, etc…
But if your team is 100 Developer, like a big eCommerce, you’ll have multiple Teams between 5 and 10 Developers per Business Domain, and you need independence of each Service, having less interdependence. Each Service will own their own Data. That is normally around 5 to 7 tables. Each Service will serve a Business Domain. You’ll benefit from having different technologies for the different needs, however be careful to avoid having Teams with different knowledge that can have hardly rotation and difficult to continue projects when the only 2 or 3 Devs that know that technology leave. Typical benefit scenarios can be having MySql for the Billing Services, but having NoSQL Database for the image catalog, or to store logs of account activity. With Microservices, some services will be calling other Services, often asynchronously, using Queues or Streams, you’ll have Callbacks, Databases for reading, you’ll probably want to have gradual and gracefully failure of your applications, client load balancing, caches and read only databases/in-memory databases… This complexity is in order to protect one Service from the failure of others and to bring it the necessary speed under heavy load.
Here you can find a PDF Document of the typical resources I use for Microservice Projects.
Please, take in count that the colors may be different depending on the Terminal used, so if you’re creating a commercial application I recommend you to try with some of them like: ubuntu terminal, ssh, putty for windows, MobaXTerm…