In this video I show a nice way to work with Data in Python, by using Tuples.
I also show how to easily and conveniently sort the Data based on your preferred criteria by using lambdas.
What happens if we have accents, ç, Ç etc…
You can download the code from:
https://gitlab.com/carles.mateo/python_combat_guide/-/blob/master/src/arrays_with_tuples.py
One of my friends wanted to use SaltStack and https://github.com/napalm-automation/napalm-salt
But he had problems installing napalm-base package.
Note that the package is no longer maintained.
He tried with the last one, and with the previous one (0.25.0), but he always got the error: ModuleNotFoundError: No module named ‘pip.req’
pip3 install napalm-base==0.25.0
Defaulting to user installation because normal site-packages is not writeable
Collecting napalm-base==0.25.0
Using cached napalm-base-0.25.0.tar.gz (35 kB)
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [6 lines of output]
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "/tmp/pip-install-gzd07xzq/napalm-base_aace1b03ac0e4045bbc85e27c788ebc1/setup.py", line 5, in <module>
from pip.req import parse_requirements
ModuleNotFoundError: No module named 'pip.req'
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
TL;TR: The problem is that pip version 10, changed the structure for req.
There are several solutions that can be done to make it work, but the easiest way is to downgrade pip, and install the package. After pip can be upgraded again.
python -m pip install pip==9.0.3 pip3 install napalm-base
Last update: 2022-05-18 10:48 Irish Time
Recently a colleague was asking me for advice on their design of error handling in a Python application.
They were catching an error and raising an Exception, inside the except part of a method, to be catch outside the method.
And at some point a simple logic got really messy and unnecessarily complicated. Also troubleshooting and debugging an error was painful because they were only getting a Custom Exception and not context.
I explained to my colleague that I believed that the person that created that Exception chain of catch came from Java background and why I think they choose that path, and why I think in Python it’s a bad idea.
In Java, functions and methods can only return one object.
I programmed a lot in Java in my career, and it was a pain having to create value objects, and having to create all kind of objects for the return. Is it a good thing that types are strongly verified by the language? Yes. It worked? Yes. It made me invest much more time than necessary? Also yes.
Having the possibility to return only one object makes it mandatory having a way to return when there was an error. Otherwise you would need to encapsulate an error code and error description fields in each object, which is contrary to the nature of the object.
For example, a class Persona. Doesn’t make any sense having an attribute inside the class Persona to register if an operation related to this object went wrong.
For example, if we are in a class Spaceship that has a method GetPersonaInCommand() and there is a problem in that method, doesn’t make any sense to return an empty Persona object with attributes idError, errorDescription. Probably the Constructor or Persona will require at least a name or Id to build the object…. so in this case, makes sense that the method raises an Exception so the calling code catches it and knows that something went wrong or when there is no data to return.
This will force to write Custom Exceptions, but it’s a solution.
Another solution is creating a generic response object which could be an Object with these attributes:
I created this kind of approach for my Cassandra libraries to easily work with Cassandra from Java and from PHP, and for Cassandra Universal Driver (a http/s gateway created in year 2014).
Python allows you to return multiple values, so I encourage you tor return a boolean for indicating the success of the operation, and the object/value you’re interested.
You can see it easily if you take a look to FileUtils class from my OpenSource libraries carleslibs.
The method get_file_size_in_bytes(self, s_file) for example:
def get_file_size_in_bytes(self, s_file):
b_success = False
i_file_size = 0
try:
# This will help with Unit Testing by raisin IOError Exception
self.test_helper()
i_file_size = os.path.getsize(s_file)
b_success = True
except IOError:
b_success = False
return b_success, i_file_size
It will always return a boolean value to indicate success or failure of the operation and an integer for the size of the file.
The calling code will do something like this:
o_fileutils = FileUtils()
b_success, i_bytes = o_fileutils.get_file_size_in_bytes("profile.png")
if b_succes is False:
print("Error! The file does not exist or cannot be accessed!")
exit(1)
if i_bytes < 1024:
print("The profile picture should be at least 1KB")
exit(1)
print("Profile picture exists and is", i_bytes, " bytes in length!")
The fact that Python can return multiple variables makes super easy dealing with error handling without having to take the road of Custom Exceptions.
And it is Ok if you want to follow this path, but in my opinion, for most of the developers up to Senior levels, it only over complicates the logic of your code and the amount of try/excepts you have to have everywhere.
If you use PHP you can mix different types in an Array, so you can always return an Array with a boolean, or an i_id_error, and your object or data of whatever type it’s.
Getting back to my carleslibs Open Source package, it is super easy to Unit Test these methods.
In my opinion, this level of simplicity, brings only advantages. Including Software Development speed, which is good for the business.
I’m not advocating for not using Custom Exceptions or to not develop a Exceptions Raising strategy if you need it and you know what you’re doing. I’m just suggesting why I think most of the developments in Python do not really need this and only over complicates the development. There are situations where raising exceptions will be a perfectly useful or even the best approach, there are many scenarios, but I think that in most of cases, using raise inside except will only multiply the time of the development and slow down the speed of bringing new features to the business, over complicating Unit Test as well, and be a real pain for the Junior and Intermediate developers.
Obviously, as the Constructor doesn’t return any value, it is perfectly fine to raise an exception in there, or just to use try/except in the code that is instancing the objects.
This is a document previous to a live code review session.
It has the information to prepare for the upcoming code review session, where I plan to share the lessons learned, decision I took, mistakes I did, refactors I had to overcome, and tentatively we will refactor code in order to add some Unit Testing.
I used to play sudoku with my family, so from time to time I do by myself.
Once I found a sudoku that was impossible and it happened that it was a typo from the newspaper, so, when I found another impossible sudoku I wanted to know if it was me, or if there was a typo or similar, so I decided to write a Sudoku Solver that will solve the sudoku for me.
I had problems solving these two sudokus:
You can clone the project from here:
https://gitlab.com/carles.mateo/sudo-ku-solver
You will have to install colorama package, as I used it for giving colors to the output:
pip3 install colorama
The main program sudokusolver.py:
import copy
from lib.colorutils import ColorUtils
class SudokuMap():
def __init__(self, i_width, i_height, o_color=ColorUtils()):
self.i_width = i_width
self.i_height = i_height
self.o_color = o_color
self.a_map = self.generate_empty_map()
def generate_empty_map(self):
a_map = []
a_row = []
a_i_possible_numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9]
for i_x in range(self.i_width):
a_row.append(a_i_possible_numbers.copy())
for i_y in range(self.i_height):
a_map.append(copy.deepcopy(a_row))
return a_map
def set_number(self, i_number, i_x, i_y):
"""
Sets a well known (already defined in the original map) number for a position
:param i_number:
:param i_x:
:param i_y:
:return:
"""
self.a_map[i_y][i_x] = [i_number]
def detect_and_remove_a_number_from_possibles_from_a_row(self, i_y):
"""
We will elinate this possibility from the row
:return: Boolean
"""
b_found = False
self.o_color.print_label("Detecting numbers to remove from row " + str(i_y))
for i_x in range(0, self.i_width):
a_i_numbers_possible = self.a_map[i_y][i_x]
if len(a_i_numbers_possible) == 1:
b_found = True
i_number_found = self.a_map[i_y][i_x][0]
print("Found a number that will be removed from horizontal and vertical and in quadrant", i_number_found, "at", i_x, i_y)
self.remove_a_number_from_possibles_in_a_row(i_number_to_remove=i_number_found, i_y=i_y)
self.remove_a_number_from_possibles_in_a_column(i_number_to_remove=i_number_found, i_x=i_x)
self.remove_a_number_from_possibles_in_quadrant(i_number_to_remove=i_number_found, i_x=i_x, i_y=i_y)
return b_found
def remove_a_number_from_possibles_in_a_row(self, i_number_to_remove, i_y):
"""
Removes a number from the list of possibles in that row
:param i_number_to_remove:
:param i_y:
:return:
"""
self.o_color.print_label("> Scanning for removing " + str(i_number_to_remove) + " in row " + str(i_y))
for i_x in range(0, self.i_width):
a_i_numbers_possible = self.a_map[i_y][i_x]
if len(a_i_numbers_possible) == 1 and a_i_numbers_possible[0] == i_number_to_remove:
# This is the right cell, ignore it
pass
else:
# Subtract the number from the sequence
if i_number_to_remove in a_i_numbers_possible:
a_i_numbers_possible_old = a_i_numbers_possible.copy()
a_i_numbers_possible.remove(i_number_to_remove)
print("> Removed", i_number_to_remove, "From:", str(i_x) + "x" + str(i_y), a_i_numbers_possible_old, "Pending:", a_i_numbers_possible)
self.a_map[i_y][i_x] = a_i_numbers_possible
if len(a_i_numbers_possible) == 1:
# Trigger it again for the number recently discovered
i_new_number_to_remove = a_i_numbers_possible[0]
self.o_color.print_success("> Found " + str(i_new_number_to_remove) + " From: " + str(i_x) + "x" + str(i_y))
self.remove_a_number_from_possibles_in_a_row(i_number_to_remove=i_new_number_to_remove, i_y=i_y)
self.remove_a_number_from_possibles_in_a_column(i_number_to_remove=i_new_number_to_remove, i_x=i_x)
self.remove_a_number_from_possibles_in_quadrant(i_number_to_remove=i_new_number_to_remove, i_x=i_x, i_y=i_y)
self.o_color.print_label("> Leaving scan for " + str(i_number_to_remove) + " in row " + str(i_y))
def remove_a_number_from_possibles_in_a_column(self, i_number_to_remove, i_x):
"""
Removes a number from the list of possibles in that row
:param i_number_to_remove:
:param i_y:
:return:
"""
self.o_color.print_label("V Scanning for removing " + str(i_number_to_remove) + " in col " + str(i_x))
for i_y in range(0, self.i_height):
a_i_numbers_possible = self.a_map[i_y][i_x]
if len(a_i_numbers_possible) == 1 and a_i_numbers_possible[0] == i_number_to_remove:
# This is the right cell, ignore it
pass
else:
# Subtract the number from the sequence
if i_number_to_remove in a_i_numbers_possible:
a_i_numbers_possible_old = a_i_numbers_possible.copy()
a_i_numbers_possible.remove(i_number_to_remove)
print("V Removed", i_number_to_remove, "From:", i_x, i_y, a_i_numbers_possible_old, "Pending:", a_i_numbers_possible)
# @TODO: Remove, as it's a pointer it is not needed
self.a_map[i_y][i_x] = a_i_numbers_possible
if len(a_i_numbers_possible) == 1:
# Trigger it again for the number recently discovered
i_new_number_to_remove = a_i_numbers_possible[0]
self.o_color.print_success("Found " + str(i_new_number_to_remove) + " From: " + str(i_x) + " " + str(i_y))
self.remove_a_number_from_possibles_in_a_row(i_number_to_remove=i_new_number_to_remove, i_y=i_y)
self.remove_a_number_from_possibles_in_a_column(i_number_to_remove=i_new_number_to_remove, i_x=i_x)
self.remove_a_number_from_possibles_in_quadrant(i_number_to_remove=i_new_number_to_remove, i_x=i_x, i_y=i_y)
self.o_color.print_label("V Leaving scan for " + str(i_number_to_remove) + " in col " + str(i_x))
def remove_a_number_from_possibles_in_quadrant(self, i_number_to_remove, i_x, i_y):
"""
:param i_number_to_remove:
:param i_x:
:param i_y:
:return:
"""
i_x_quadrant = int(i_x / 3)
i_y_quadrant = int(i_y / 3)
i_x_ini = i_x_quadrant * 3
i_x_end = i_x_ini + 2
i_y_ini = i_y_quadrant * 3
i_y_end = i_y_ini + 2
for i_y_rel in range(i_y_ini, i_y_end + 1):
for i_x_rel in range(i_x_ini, i_x_end + 1):
a_i_numbers_possible = self.a_map[i_y_rel][i_x_rel]
if len(a_i_numbers_possible) == 1 and a_i_numbers_possible[0] == i_number_to_remove:
# This is the right cell, ignore it
pass
else:
# Subtract the number from the sequence
if i_number_to_remove in a_i_numbers_possible:
a_i_numbers_possible_old = a_i_numbers_possible.copy()
a_i_numbers_possible.remove(i_number_to_remove)
print("X Removed", i_number_to_remove, "From:", i_x_rel, i_y_rel, a_i_numbers_possible_old, "Pending:", a_i_numbers_possible)
# Nota: Here I had a bug and I was "liant-la parda"
# if len(a_i_numbers_possible) == 1:
# # Trigger it again for the number recently discovered
# i_new_number_to_remove = a_i_numbers_possible[0]
# string_ints = [str(int) for int in ints]
# self.o_color.print_success("X Found " + str(i_new_number_to_remove) + " From: " + str(i_x) + "x" + str(i_y) + "[]")
# self.remove_a_number_from_possibles_in_a_row(i_number_to_remove=i_new_number_to_remove, i_y=i_y)
# self.remove_a_number_from_possibles_in_a_column(i_number_to_remove=i_new_number_to_remove, i_x=i_x)
def check_if_number_possibles_in_quadrant_is_unique(self, i_number_to_check, i_x, i_y):
"""
:param i_number_to_remove:
:param i_x:
:param i_y:
:return: b_found
"""
i_x_quadrant = int(i_x / 3)
i_y_quadrant = int(i_y / 3)
i_x_ini = i_x_quadrant * 3
i_x_end = i_x_ini + 2
i_y_ini = i_y_quadrant * 3
i_y_end = i_y_ini + 2
i_number_of_occurrences_found = 0
i_x_position_number = 0
i_y_position_number = 0
b_unique = False
for i_y_rel in range(i_y_ini, i_y_end + 1):
for i_x_rel in range(i_x_ini, i_x_end + 1):
a_i_numbers_possible = self.a_map[i_y_rel][i_x_rel]
for i_number_in_possibles in a_i_numbers_possible:
if len(a_i_numbers_possible) > 1 and i_number_in_possibles == i_number_to_check:
# This is the right cell, ignore it
i_number_of_occurrences_found += 1
i_x_position_number = i_x_rel
i_y_position_number = i_y_rel
if i_number_of_occurrences_found > 1:
# Unsuccessful
break
if i_number_of_occurrences_found == 1:
# Success!
a_i_numbers_possible = [i_number_to_check]
self.a_map[i_y_position_number][i_x_position_number] = a_i_numbers_possible
b_unique = True
return b_unique, i_x_position_number, i_y_position_number
def check_if_number_possibles_in_row_is_unique(self, i_number_to_check, i_y):
"""
:param i_number_to_check:
:param i_x:
:param i_y:
:return:
"""
i_number_of_occurrences_found = 0
i_x_position_number = 0
i_y_position_number = 0
b_unique = False
for i_x_rel in range(0, 9):
a_i_numbers_possible = self.a_map[i_y][i_x_rel]
for i_number_in_possibles in a_i_numbers_possible:
if len(a_i_numbers_possible) > 1 and i_number_in_possibles == i_number_to_check:
# This is the right cell, ignore it
i_number_of_occurrences_found += 1
i_x_position_number = i_x_rel
i_y_position_number = i_y
if i_number_of_occurrences_found > 1:
# Unsuccessful
break
if i_number_of_occurrences_found == 1:
# Success!
a_i_numbers_possible = [i_number_to_check]
self.a_map[i_y_position_number][i_x_position_number] = a_i_numbers_possible
b_unique = True
return b_unique, i_x_position_number, i_y_position_number
def get_map_drawing_as_string(self, a_map_alternative=None):
s_map = ""
i_counter_y = 0
s_separator_rows = "="
a_map_to_use = self.a_map
if a_map_alternative is not None:
a_map_to_use = a_map_alternative
s_map = s_map + s_separator_rows * 37 + "\n"
for a_row in a_map_to_use:
i_counter_y += 1
if i_counter_y == 3:
i_counter_y = 0
s_separator_rows = "="
else:
s_separator_rows = "-"
s_map = s_map + "|"
i_counter = 0
for a_i_numbers_possible in a_row:
i_counter += 1
if len(a_i_numbers_possible) == 1:
s_number = str(a_i_numbers_possible[0])
else:
s_number = " "
if i_counter == 3:
s_separator = "|"
i_counter = 0
else:
s_separator = "¦"
s_map = s_map + " " + s_number + " " + s_separator
s_map = s_map + "\n"
s_map = s_map + s_separator_rows * 37 + "\n"
# Replace 0 by " "
s_map = s_map.replace("0", " ")
s_map = s_map + "\n\n"
i_total_numbers_found, a_s_numbers_found = self.get_total_numbers_found()
s_map = s_map + "Total numbers found: " + str(i_total_numbers_found) + " Numbers found: " + " ".join(a_s_numbers_found) + "\n"
return s_map
def get_map_drawing_of_possibles_as_string(self, a_map_alternative=None):
s_map = ""
i_counter_y = 0
s_separator_rows = "="
a_map_to_use = self.a_map
if a_map_alternative is not None:
a_map_to_use = a_map_alternative
s_map = s_map + self.o_color.color_blue(s_separator_rows * ((9 * ( 9 + 2 )) + 10)) + "\n"
for a_row in a_map_to_use:
i_counter_y += 1
if i_counter_y == 3:
i_counter_y = 0
s_separator_rows = "="
else:
s_separator_rows = "-"
s_map = s_map + self.o_color.color_blue("|")
i_counter = 0
for a_i_numbers_possible in a_row:
i_counter += 1
if len(a_i_numbers_possible) == 1:
# The right number
s_number = str(a_i_numbers_possible[0]).center(9)
s_number = self.o_color.color_success(s_number)
else:
a_i_numbers_possible_string = []
for i_number in a_i_numbers_possible:
s_number = str(i_number)
# Replace by the color sequence
if i_number == 2:
s_number = self.o_color.color_red(s_number)
if i_number == 3:
s_number = self.o_color.color_yellow(s_number)
if i_number == 4:
self.o_color.color_magenta(s_number)
a_i_numbers_possible_string.append(s_number)
# s_number = "".join(a_i_numbers_possible_string).ljust(9)
s_number = "".join(a_i_numbers_possible_string) + " " * (9-len(a_i_numbers_possible))
if i_counter == 3:
s_separator = self.o_color.color_blue("|")
i_counter = 0
else:
s_separator = self.o_color.color_blue("¦")
s_map = s_map + " " + s_number + " " + s_separator
s_map = s_map + "\n"
s_map = s_map + self.o_color.color_blue(s_separator_rows * ((9 * (9 + 2)) + 10)) + "\n"
# Replace 0 by " "
s_map = s_map.replace("0", " ")
return s_map
def get_total_numbers_found(self):
i_total_numbers_found = 0
a_s_numbers_found = []
for i_y in range(0, self.i_height):
for i_x in range(0, self.i_width):
a_i_numbers_possible = self.a_map[i_y][i_x]
if len(a_i_numbers_possible) == 1:
i_total_numbers_found = i_total_numbers_found + 1
i_number_found = self.a_map[i_y][i_x][0]
s_number_found = str(i_number_found)
if s_number_found not in a_s_numbers_found:
a_s_numbers_found.append(s_number_found)
return i_total_numbers_found, a_s_numbers_found
if __name__ == "__main__":
o_color = ColorUtils()
o_map = SudokuMap(9, 9, o_color=o_color)
o_map.set_number(i_number=1, i_x=1, i_y=0)
o_map.set_number(3, 4, 0)
o_map.set_number(8, 7, 0)
o_map.set_number(8, 0, 1)
o_map.set_number(7, 3, 1)
o_map.set_number(4, 5, 1)
o_map.set_number(6, 8, 1)
o_map.set_number(3, 2, 2)
o_map.set_number(9, 6, 2)
o_map.set_number(2, 1, 3)
o_map.set_number(4, 4, 3)
o_map.set_number(6, 7, 3)
o_map.set_number(5, 0, 4)
o_map.set_number(6, 3, 4)
o_map.set_number(2, 5, 4)
o_map.set_number(8, 8, 4)
o_map.set_number(3, 1, 5)
o_map.set_number(8, 4, 5)
o_map.set_number(7, 7, 5)
o_map.set_number(2, 2, 6)
o_map.set_number(6, 6, 6)
o_map.set_number(9, 0, 7)
o_map.set_number(4, 3, 7)
o_map.set_number(3, 5, 7)
o_map.set_number(2, 8, 7)
o_map.set_number(8, 1, 8)
o_map.set_number(6, 4, 8)
o_map.set_number(1, 7, 8)
# Extra
# o_map.set_number(2, 0, 0)
# Speculative
o_map.set_number(7, 0, 3)
# Another map
o_map2 = SudokuMap(9, 9, o_color=o_color)
o_map2.set_number(i_number=5, i_x=0, i_y=0)
o_map2.set_number(i_number=9, i_x=5, i_y=0)
o_map2.set_number(i_number=7, i_x=2, i_y=1)
o_map2.set_number(i_number=2, i_x=7, i_y=1)
o_map2.set_number(i_number=2, i_x=0, i_y=2)
o_map2.set_number(i_number=3, i_x=4, i_y=2)
o_map2.set_number(i_number=1, i_x=5, i_y=2)
o_map2.set_number(i_number=9, i_x=7, i_y=2)
o_map2.set_number(i_number=7, i_x=0, i_y=3)
o_map2.set_number(i_number=1, i_x=2, i_y=3)
o_map2.set_number(i_number=6, i_x=3, i_y=3)
o_map2.set_number(i_number=9, i_x=4, i_y=3)
o_map2.set_number(i_number=4, i_x=8, i_y=3)
o_map2.set_number(i_number=1, i_x=4, i_y=4)
o_map2.set_number(i_number=6, i_x=0, i_y=5)
o_map2.set_number(i_number=7, i_x=4, i_y=5)
o_map2.set_number(i_number=4, i_x=5, i_y=5)
o_map2.set_number(i_number=3, i_x=6, i_y=5)
o_map2.set_number(i_number=1, i_x=8, i_y=5)
o_map2.set_number(i_number=5, i_x=1, i_y=6)
o_map2.set_number(i_number=3, i_x=3, i_y=6)
o_map2.set_number(i_number=6, i_x=4, i_y=6)
o_map2.set_number(i_number=8, i_x=8, i_y=6)
o_map2.set_number(i_number=6, i_x=1, i_y=7)
o_map2.set_number(i_number=7, i_x=6, i_y=7)
o_map2.set_number(i_number=9, i_x=3, i_y=8)
o_map2.set_number(i_number=3, i_x=8, i_y=8)
# Extra help while not implemented the best algorithm
# =============================================================================================================
# | 5 ¦ 148 ¦ 48 | 7 ¦ 2 ¦ 9 | 148 ¦ 3 ¦ 6 |
# -------------------------------------------------------------------------------------------------------------
# | 13489 ¦ 13489 ¦ 7 | 48 ¦ 48 ¦ 6 | 148 ¦ 2 ¦ 5 |
# -------------------------------------------------------------------------------------------------------------
# | 2 ¦ 48 ¦ 6 | 5 ¦ 3 ¦ 1 | 48 ¦ 9 ¦ 7 |
# =============================================================================================================
# | 7 ¦ 38 ¦ 1 | 6 ¦ 9 ¦ 358 | 2 ¦ 58 ¦ 4 |
# -------------------------------------------------------------------------------------------------------------
# | 348 ¦ 2348 ¦ 3458 | 28 ¦ 1 ¦ 358 | 6 ¦ 7 ¦ 9 |
# -------------------------------------------------------------------------------------------------------------
# | 6 ¦ 289 ¦ 589 | 28 ¦ 7 ¦ 4 | 3 ¦ 58 ¦ 1 |
# =============================================================================================================
# | 14 ¦ 5 ¦ 2 | 3 ¦ 6 ¦ 7 | 9 ¦ 14 ¦ 8 |
# -------------------------------------------------------------------------------------------------------------
# | 3489 ¦ 6 ¦ 3489 | 1 ¦ 458 ¦ 58 | 7 ¦ 45 ¦ 2 |
# -------------------------------------------------------------------------------------------------------------
# | 148 ¦ 7 ¦ 48 | 9 ¦ 458 ¦ 2 | 145 ¦ 6 ¦ 3 |
# =============================================================================================================
# By best algorithm I mean that the last in the middle vertical quadrant from the top right horizontally,
# only 5 can be in a column. That clarifies that 5 must go to the other column in last quadrant, first column. Coord 6x8
# o_map2.set_number(i_number=5, i_x=6, i_y=8)
# ERROR traces from a bug fixed to mentioned during the code review
# Surprisingly this fails
# > Leaving scan for 8 in row1
# V Scanning for removing 8 in col 4
# V Removed 8 From: 4 7 [5, 8] Pending: [5]
# Found 5 From: 4 7
#
# > Scanning for removing 5 in row 7
# > Removed 5 From: 5 7 [5, 8] Pending: [8]
# > Found 8 From: 5 7
#
# > Scanning for removing 8 in row 7
# > Removed 8 From: 0 7 [3, 8] Pending: [3]
# > Found 3 From: 0 7
o_map = o_map2
print(o_map.get_map_drawing_as_string())
b_changes_found = True
while b_changes_found is True:
b_changes_found = False
for i_y in range(0, o_map.i_height):
b_found = o_map.detect_and_remove_a_number_from_possibles_from_a_row(i_y=i_y)
if b_found is True:
print(o_map.get_map_drawing_as_string())
for i_y in range(0, o_map.i_height):
o_map.o_color.print_label("Scanning quadrants for row " + str(i_y))
for i_number in range(1, 10):
for i_x in range(0, o_map.i_width):
b_found, i_x_found, i_y_found = o_map.check_if_number_possibles_in_quadrant_is_unique(i_number_to_check=i_number, i_x=i_x, i_y=i_y)
if b_found is True:
# Search again
b_changes_found = True
o_map.remove_a_number_from_possibles_in_a_row(i_number_to_remove=i_number, i_y=i_y_found)
o_map.remove_a_number_from_possibles_in_a_column(i_number_to_remove=i_number, i_x=i_x_found)
b_found, i_x_found, i_y_found = o_map.check_if_number_possibles_in_row_is_unique(i_number_to_check=i_number, i_y=i_y)
if b_found is True:
b_changes_found = True
o_map.remove_a_number_from_possibles_in_a_column(i_number_to_remove=i_number, i_x=i_x_found)
if b_changes_found is True:
print(o_map.get_map_drawing_as_string())
# @TODO: Implement check if number in quadrant can only go to a column, to remove the non possible in that column from another quadrant
# @TODO: Implement check if in a line only one number can go to a column.
print(o_map.get_map_drawing_as_string())
print(o_map.get_map_drawing_of_possibles_as_string())
The color library lib/colorutils.py:
from colorama import Fore, Back, Style , init
class ColorUtils:
def __init__(self):
# For Colorama on Windows
init()
def print_error(self, m_text, s_end="\n"):
"""
Prints errors in Red.
:param s_text:
:return:
"""
# If they pass numbers
s_text = str(m_text)
print(Fore.RED + s_text)
print(Style.RESET_ALL, end=s_end)
def print_success(self, m_text, s_end="\n"):
"""
Prints errors in Green.
:param s_text:
:return:
"""
# If they pass numbers
s_text = str(m_text)
print(Fore.GREEN + s_text)
print(Style.RESET_ALL, end=s_end)
def color_success(self, m_text):
"""
Colors only this
:param m_text:
:return:
"""
s_text = str(m_text)
return Fore.GREEN + s_text + Fore.RESET
def color_black(self, m_text):
s_text = str(m_text)
return Fore.BLACK + s_text + Fore.RESET
def color_blue(self, m_text):
s_text = str(m_text)
return Fore.BLUE + s_text + Fore.RESET
def color_red(self, m_text):
s_text = str(m_text)
return Fore.RED + s_text + Fore.RESET
def color_yellow(self, m_text):
s_text = str(m_text)
return Fore.YELLOW + s_text + Fore.RESET
def color_magenta(self, m_text):
s_text = str(m_text)
return Fore.MAGENTA + s_text + Fore.RESET
def print_label(self, m_text, s_end="\n"):
"""
Prints a label and not the end line
:param s_text:
:return:
"""
# If they pass numbers
s_text = str(m_text)
print(Fore.BLUE + s_text, end="")
print(Style.RESET_ALL, end=s_end)
def return_text_blue(self, s_text):
"""
Restuns a Text
:param s_text:
:return: String
"""
s_text_return = Fore.BLUE + s_text + Style.RESET_ALL
return s_text_return
Last Update: 2022-04-16 15:22
So, I was working on a project and i wanted to test how long a file can be.
The resources I checked, and also the Kernel and C source I reviewed, were pointing that effectively the limit is 255 characters.
But I had one doubt… given that Python 3 String are UTF-8, and that Linux encode by default is UTF-8, will I be able to use 255 characters length, or this will be reduced as the Strings will be encoded as UTF-8 or Unicode?.
So I did a small Proof of Concept.
For the filenames I grabbed a fragment of the translation to English of the book epic book “Tirant lo Blanc” (The White Knight), one of the first books written in Catalan language and the first chivalry novel known in the world. You can download it for free it in:
https://onlinebooks.library.upenn.edu/webbin/gutbook/lookup?num=378
Python Code:
from carleslibs.fileutils import FileUtils
from colorama import Fore, Back, Style , init
class ColorUtils:
def __init__(self):
# For Colorama on Windows
init()
def print_error(self, s_text, s_end="\n"):
"""
Prints errors in Red.
:param s_text:
:return:
"""
print(Fore.RED + s_text)
print(Style.RESET_ALL, end=s_end)
def print_success(self, s_text, s_end="\n"):
"""
Prints errors in Green.
:param s_text:
:return:
"""
print(Fore.GREEN + s_text)
print(Style.RESET_ALL, end=s_end)
def print_label(self, s_text, s_end="\n"):
"""
Prints a label and not the end line
:param s_text:
:return:
"""
print(Fore.BLUE + s_text, end="")
print(Style.RESET_ALL, end=s_end)
def return_text_blue(self, s_text):
"""
Restuns a Text
:param s_text:
:return: String
"""
s_text_return = Fore.BLUE + s_text + Style.RESET_ALL
return s_text_return
if __name__ == "__main__":
o_color = ColorUtils()
o_file = FileUtils()
s_text = "In the fertile, rich and lovely island of England there lived a most valiant knight, noble by his lineage and much more for his "
s_text += "courage. In his great wisdom and ingenuity he had served the profession of chivalry for many years and with a great deal of honor, "
s_text += "and his fame was widely known throughout the world. His name was Count William of Warwick. This was a very strong knight "
s_text += "who, in his virile youth, had practiced the use of arms, following wars on sea as well as land, and he had brought many "
s_text += "battles to a successful conclusion."
o_color.print_label("Erasure Code project by Carles Mateo")
print()
print("Task 237 - Proof of Concep of Long File Names, encoded is ASCii, in Linux ext3 and ext4 Filesystems")
print()
print("Using as a sample a text based on the translation of Tirant Lo Blanc")
print("Sample text used:")
print("-"*30)
print(s_text)
print("-" * 30)
print()
print("This test uses the OpenSource libraries from Carles Mateo carleslibs")
print()
print("Initiating tests")
print("================")
s_dir = "task237_tests"
if o_file.folder_exists(s_dir) is True:
o_color.print_success("Directory " + s_dir + " already existed, skipping creation")
else:
b_success = o_file.create_folder(s_dir)
if b_success is True:
o_color.print_success("Directory " + s_dir + " created successfully")
else:
o_color.print_error("Directory " + s_dir + " creation failed")
exit(1)
for i_length in range(200, 512, 1):
s_filename = s_dir + "/" + s_text[0:i_length]
b_success = o_file.write(s_file=s_filename, s_text=s_text)
s_output = "Writing file length: "
print(s_output, end="")
o_color.print_label(str(i_length).rjust(3), s_end="")
print(" file name: ", end="")
o_color.print_label(s_filename, s_end="")
print(": ", end="")
if b_success is False:
o_color.print_error("failed", s_end="\n")
exit(0)
else:
o_color.print_success("success", s_end="\n")
# Note: up to 255 work, 256 fails
I tried this in an Ubuntu Virtual Box VM.
As part of my tests I tried to add a non typical character in English, ASCii >127, like Ç.
When I use ASCii character < 128 (0 to 127) for the filenames in ext3/ext4 and in a ZFS Pool, I can use 255 positions. But when I add characters that are not typical, like the Catalan ç Ç or accents Àí the space available is reduced, suggesting the characters are being encoded:
Ç has value 128 in the ASCii table and ç 135.
I used my Open Source carleslibs and the package colorama.
Install them with:
pip3 install carleslibs colorama
I’m Catalan. In 1936 the fascist military leaded by franco raised in arms against the elected government of the Spanish Republic. The Italian and nazi German fascist in power bombed the Catalan population. Hundreds of thousands of innocent citizens were assassinated and millions of Catalan and Spaniards had to exile. The sons of those that were ruling with the dictator have been insisting in naming it a “civil war”, but it was the military lead by a fascist, revolting against the legitimate Republic and ending a democracy.
The dictatorship lasted until 1975, when the dictator died in the bed. The effects of the repression never abandoned Catalonia, and nowadays in Catalonia people is still detained by the Spanish police for talking the Catalan language in front of them, and our Parliament decisions are cancelled by the Spanish courts, for example to force the exit of a President of Catalonia that they didn’t like, or to force the Catalan schools to teach 25% of the time in Spanish attacking the Catalan teaching system.
During WW2 millions of Jews were mass murdered, also people from all the nations were assassinated.
Russian population suffered a lot also fighting the nazis.
Now we have to see how Russia’s army is invading Ukraine and murdering innocent citizens.
That’s horrible.
I know Engineers from Ukraine. Those guys were doing great building wealthy based on knowledge and working well for companies across the world. Now these people are being killed or Engineers, amongst all the brave population, are arming themselves to fight the invasion. Shells destroy beautiful cities and population are starving, and young soldiers from both sides will never be seen again by their mothers.
I wrote a small article on how to identify and block in the Ubuntu firewall the Ip’s from Russia and Belarus until Russia leaves Ukraine.
Let music play in solidarity with Ukraine. First is a Catalan group. Second is a famous Irish band in this epic song dedicated to the brave International Brigades, volunteers that fought the fascism in Spain and in Catalonia trying to make a better world.
I’ve updated the SSL Certificate. The previous one I bought was issued for two years, and I renewed as it was due to expire.
I wrote a short article about how to update the SSL Certificates for Apache 2 in Ubuntu 20.04.
I published a small Python script to show the local datetime and the Unix Epoch Time.
On the 6th of January I released carleslibs v.1.0.7
https://pypi.org/project/carleslibs/1.0.7/
The new version contains these improvements:
I have updated the book, offering solution to exercises 11.1, 11.2 (simple and encapsulated in a function) and I’ve created exercise 11.3.
If you purchased the book before, you can download any update for free.
I got an offer by a super editorial to publish my book Automating and Provisioning with Amazon Python 3 SDK boto3.
Honestly, my ego was flattered. It is a lot of reputation.
Although in the past I got an offer from another monstrously big editorial to publish world wide my book Python 3 Combat Guide and I also rejected, and an offer from a digital learning platform to create an interactive course from this same book.
I’ve rejected it again this time.
If you are curious, this is what I answered to them:
Hi XXXX, I'm well, thank you. I hope you are doing well too. Thanks for taking the time to explain your conditions to me. I feel flattered by your editorial thinking about me. I respect your brand, as I mentioned, as I own several of your titles. However, I have to refuse your offer. Is not the first time an editor has offered to publish one or more of my books. For all over the world, with much higher economic expectations. I'll tell you why I love being at LeanPub: 1- I own the rights. All of them. 2- I can publish updates, and my readers get them for free. As I add new materials, the value is maximized for my readers. 3- I get 80% of the royalties. 4- If a reader is not happy, they can return the book for 60 days. 5- I can create vouchers and give a discount to certain readers, or give for free to people that are poor and are trying to get a career in Engineering. The community of readers are very honest, and I only got 2 returns. One of them I think was from an editorial that purchased the book, evaluated it, and they contacted me to publish it, and after I rejected they applied for the refund. I teach classes, and I charge 125 EUR per hour. I can make much more by my side than the one time payment you offer. The compensation for the video seems really obsolete. Also, I could be using Amazon self publishing, which also brings bigger margins than you. So many thanks for your offer. I thought about it because of the reputation, but I already have a reputation. I've thousands of visits to my tech blog, and because of the higher royalties, even if I sell less books through LeanPub it is much more rewarding. Thanks again and have a lovely day and rest of the week. Best, Carles
The provisioning in Amazon AWS through their SDK is a book I’m particularly proud, as it empowers the developers so much. And I provide source code so they can go from zero to hero, in a moment. Amazon should provide a project sample as I do, not difficult to follow documentation.
As I was requested, I’ve been offering advice and having virtual coffees with some people that recently started their journey to become Software Engineers and wanted some guidance and advice.
It has been great seeing people putting passion and studying hard to make a better future for themselves and for their families.
I’ll probably add to the blog more contents for beginners, although it will continue being a blog dedicated to extreme IT, and to super cool Engineering skills and troubleshooting.
For my regular students I have a discord space where we can talk and they can meet new friends studying or working in Engineering.
This github link provides many free books in multiple languages:
https://github.com/EbookFoundation/free-programming-books
Zoom can zoom the view. So if they are sharing their screen, and font is too small, you can give a relax to your eyes by using Zoom’s zoom feature. It is located in View.
After being in the hospital in December 2021, with risk for my life, and after my incredible recuperation, I’ve got the good news that I don’t need anymore 2 of the 3 medicines I was taking in a daily basis. It looks well through a completely recovery thanks to my discipline, doing sport every day several times, and the fantastic Catalan doctors that are supporting me so well.
Since they found what was failing in me, and after the emergency treatments I started to sleep really well. All night. That’s a privilege that I didn’t have for long long time.
Yes, this works like this.
You can try yourself:
<html>
<body>
<script>
console.log("11" + 1)
console.log("11" - 1)
</script>
</body>
</html>
As you can see if you open the Browser Developer tools (in Linux and Windows press F12 key):
The Unix Epoch is the time, in seconds, that has passed since 1970-01-01 00:00:00 UTC.
#!/usr/bin/env python3
#
# Date time methods
#
# Author: Carles Mateo
# Creation Date: 2019-11-20 17:23 IST
# Description: Class to return Date, datetime, Unix EPOCH timestamp
#
import datetime
import time
class DateTimeUtils:
def get_unix_epoch(self):
"""
Will return the EPOCH Time. For convenience is returned as String
:return:
"""
s_now_epoch = str(int(time.time()))
return s_now_epoch
def get_datetime(self, b_milliseconds=False):
"""
Return the datetime with miliseconds in format YYYY-MM-DD HH:MM:SS.xxxxx
or without milliseconds as YYYY-MM-DD HH:MM:SS"""
if b_milliseconds is True:
s_now = str(datetime.datetime.now())
else:
s_now = str(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"))
return s_now
def main():
o_datetime = DateTimeUtils()
s_now_epoch = o_datetime.get_unix_epoch()
s_now_date = o_datetime.get_datetime()
print(s_now_epoch)
print(s_now_date)
main()
You can also get the code from:
https://gitlab.com/carles.mateo/blog.carlesmateo.com-source-code/
I added this article to my Docker Combat Guide book.
The change of license of Docker Desktop for Windows has been a low punch, a dirty one.
Many big companies use Windows as for the laptops and workstations, we like it or not.
You can setup a Linux development computer or Virtual Machine, you may argue, but things are not as easier.
Big companies have Software licenses assigned to corporation machines, so you may not use your Pycharm license in a Linux VM.
You may no use Docker Desktop either, if your company did not license it.
And finally you may need to have access to internal resources, like Artifactory, or Servers where access is granted via ACL, so only you, from your Development machine can access it. So you have to be able to run Docker locally.
After Docker introduced this changed of license I was using VirtualBox with NAT attached to the VPN Virtual Ethernet, and I port forwarded to be able to SSH, deploy, test, etc… from outside to my Linux VM, and it was working for a while, until with the last VirtualBox update and some Windows updates where pushed to my Windows box and my VirtualBox VMs stopped booting most of the times and having random problems.
I configured a new Linux VM in a Development Server, and I opened Docker API so my Pycharm’s workstation was able to deploy there and I was able to test. But the Dev Ip’s do not have access to the same Test Servers I need my Python Automation projects to reach (and quickly I used 50 GB of space), so I tried WSL. I like Pycharm I didn’t want to switch to VStudio Code because of their good Docker extensions, in any case I could not run my code locally with venv cause some of the packages where not available for Windows, so I needed Linux to run the Unit Testing and see the Code Coverage, run the code, etc…
I tried Hyper-V, tried with NAT External, but it was incompatible with my VPN.
Note: WSL can be used, but I wanted to use Docker Engine, not docker in WSL.
The first thing I checked was the Docker downloads page.
I found the stand alone binary.
https://docs.docker.com/engine/install/binaries/#install-server-and-client-binaries-on-windows
In order to install it:
Status Name DisplayName
------ ---- -----------
Running docker Docker Engine
Attempt to pull an Image with: docker pull ubuntu or docker pull php
If it works, you’re done, but most probably you will get it starting and get this error:
Error response from daemon: unsupported os linux
or this other error:
no matching manifest for windows/amd64 10.0.19042 in the manifest list entries
Depending on your system you may need to do certain things:
I would make sure that are enabled:
Press OK, and restart your computer.
Try Again to docker pull ubuntu
Edit this file to enable experimental, you can run from the PowerShell:
notepad C:\ProgramData\Docker\config\daemon.json
Save the file and restart the Service:
Restart-Service docker
Check if it works
Get-Service docker
Status Name DisplayName ------ ---- ----------- Running docker Docker Engine
Try if now it works.
If it is not working, try running:
cd "C:\Program Files\Docker\Docker\" .\DockerCli.exe -SwitchDaemon
Give it two minutes and try to pull an image.
If it is still not working reboot, and try again:
cd "C:\Program Files\Docker\Docker\" .\DockerCli.exe -SwitchDaemon
I recommend you to add the new stand alone docker to the path, so you can call it from the terminal at any moment.
Edit the variable PATH of your user profile (not System wide)
I recommend you to have it on top after Python.
A really simple code I added to my Python 3 Exercises for Beginners book, to grab the title of a Web page.
from urllib import request
from bs4 import BeautifulSoup
s_url = "https://blog.carlesmateo.com/movies-i-saw/"
s_html = request.urlopen(s_url).read().decode('utf8')
o_soup = BeautifulSoup(s_html, 'html.parser')
o_title = o_soup.find('title')
print(o_title.string) # Prints the tag string content
# Another possible way
if o_soup.title is not None:
s_title = o_soup.title.string
else:
s_title = o_title.title
print(s_title)
I also included this code in the code repository for Python 3 Combat Guide book.
https://gitlab.com/carles.mateo/python_combat_guide/-/blob/master/src/html_parse_beautifulsoup4.py
This article is being included in my book Provisioning to Amazon AWS using boto3 SDK for Python 3.
I’ll use Ubuntu 20.04 LTS.
Python 2 is required for Ansible.
Python 3 is required for our programs.
sudo apt install python2 python3 python3-pip # Install boto for Python 2 for Ansible (alternative way if pip install boto doesn't work for you) python2 -m pip install boto # Install Ansible sudo apt install ansible
So you can use the Python 2 ec2.py and ec2.ini files, adding them as to the /etc/ansible with mask +x, to use the Dynamic Inventory.
You will need to have your credentials set.
I use Environment variables:
#!/bin/bash export ANSIBLE_HOST_KEY_CHECKING=false export AWS_ACCESS_KEY=AKIXXXXXXXXXXXXXOS export AWS_SECRET_KEY=e4dXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXY6F
Then use the calls inside the shell script, or assuming that the previous file was named credentiasl.sh use source credentials.sh
ec2.py is written in Python 2, so probably will fail for you as it is invoked by python and your default interpreter will be Python 3.
So edit the first line of /etc/ansible/ec2.py and add:
#!/bin/env python2
Once credentials.sh is sourced, then you can just invoke ec2.py to get the list of your Instances in a JSON format dumped by ec2.py
/etc/ansible/ec2.py --list
You can get that JSON file and load it and get the information you need, filtering by group.
You can call:
/etc/ansible/ec2.py --list > instances.json
Or you can run a Python program that escapes to shell and executes ec2.py –list and loads the Output as a JSON file.
I use my carleslibs here to escape to shell using my class SubProcessUtils. You can install them, they are Open Source, or you can code manually if you prefer importing subprocess Python library and catching the stdout, stderr.
import json
from carleslibs import SubProcessUtils
if __name__ == "__main__":
s_command = "/etc/ansible/ec2.py"
o_subprocess = SubProcessUtils()
i_error_code, s_output, s_error = o_subprocess.execute_command_for_output(s_command, b_shell=True, b_convert_to_ascii=True, b_convert_to_utf8=False)
if i_error_code != 0:
print("Error escaping to shell!", i_error_code)
print(s_error)
exit(1)
json = json.loads(s_output)
d_hosts = json["_meta"]["hostvars"]
for s_host in d_hosts:
# You'll get a ip/hostnamename in s_host which is the key
# You have to check for groups and the value for the key Name, in order to get the Name of the group
# As an exercise, print(d_hosts[s_host]) and look for:
# @TODO: Capture the s_group_name
# @TODO: Capture the s_addres
if s_group_name == "yourgroup":
# This filters only the instances with your group name, as you want to create an inventory file just for them
# That's because you don't want to launch the playbook for all the instances, but for those in your group name in the inventory file.
a_hostnames.append(s_address)
# After this you can parse you list a_hostnames and generate an inventory file yourinventoryfile
# The [ec2hosts] in your inventory file must match the hosts section in your yaml files
# You'll execute your playbook with:
# ansible-playbook -i yourinventoryfile youryamlfile.yaml
So an example of a yaml to install Apache2 in Ubuntu 20.04 LTS spawned instances , let’s call it install_apache2.yaml would be:
---
- name: Update web servers
hosts: ec2hosts
remote_user: ubuntu
tasks:
- name: Ensure Apache is at the latest version
apt:
name: apache2
state: latest
update_cache: yes
become: yes
As you can see the section hosts: in the YAML playbook matches the [ec2hosts] in your inventory file.
You can choose to have your private key certificate .pem file in /etc/ansible/ansible.cfg or if you want to have different certificates per host, add them after the ip/address in your inventory file, like in this example:
[localhost] 127.0.0.1 [ec2hosts] 63.35.186.109 ansible_ssh_private_key_file=ansible.pem
The ansible.pem certificate must have restricted permissions, for example chmod 600 ansible.pem
Then you end by running:
ansible-playbook -i yourinventoryfile install_ubuntu.yaml
The first method is to use add_host to print in the screen the properties form the ec2 Instances provisioned.
The trick is to escape to shell, executing ansible-playbook and capturing the output, then parsing the text looking for the ‘public_ip:’
This is the Python 3 code I created:
class AwesomeAnsible:
def extract_public_ips_from_text(self, s_text=""):
"""
Extracts the addresses returned by Ansible
:param s_text:
:return: Boolean for success, Array with the Ip's
"""
b_found = False
a_ips = []
i_count = 0
while True:
i_count += 1
if i_count > 20:
print("Breaking look")
break
s_substr = "'public_ip': '"
i_first_pos = s_text.find(s_substr)
if i_first_pos > -1:
s_text_sub = s_text[i_first_pos + len(s_substr):]
# Find the ending delimiter
i_second_pos = s_text_sub.find("'")
if i_second_pos > -1:
b_found = True
s_ip = s_text_sub[0:i_second_pos]
a_ips.append(s_ip)
s_text_sub = s_text_sub[i_second_pos:]
s_text = s_text_sub
continue
# No more Ip's
break
return b_found, a_ips
Then you’ll use with something like:
# Catching the Ip's from the output
b_success, a_ips = self.extract_public_ips_from_text(s_output)
if b_success is True:
print("Public Ips:")
s_ips = ""
for s_ip in a_ips:
print(s_ip)
s_ips = s_ips + self.get_ip_text_line_for_inventory(s_ip)
print("Adding Ips to group1_inventory file")
self.o_fileutils.append_to_file("group1_inventory", s_ips)
print()
The get_ip_text_line_for_inventory_method() returns a line for the inventory file, with the ip and the key to use separated by a tab (\t):
def get_ip_text_line_for_inventory(self, s_ip, s_key_path="ansible.pem"):
"""
Returns the line to add to the inventory, with the Ip and the keypath
"""
return s_ip + "\tansible_ssh_private_key_file=" + s_key_path + "\n"
Once you have the inventory file, like this below, you can execute the playbook for your group of hosts:
[localhost] 127.0.0.1 [ec2hosts] 63.35.186.109 ansible_ssh_private_key_file=ansible.pem
ansible-playbook -i yourinventoryfile install_ubuntu.yaml
You can run this Bash Shell Script to get only the public ips when you provision to Amazon AWS EC2 the Instances from your group named group1 in this case:
./launch_aws_instances-group1.sh | grep "public_ip" | awk '{ print $13; }' | tr -d "',"
52.213.232.199
In this example 52.213.232.199 is the Ip from the Instance I provisioned.
So to put it together, from a Python file I generate this Bash file and I escape to shell to execute it:
#!/bin/bash export ANSIBLE_HOST_KEY_CHECKING=false export AWS_ACCESS_KEY=AKIXXXXXXXXXXXXXOS export AWS_SECRET_KEY=e4dXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXY6F # Generate a new Inventory File echo "[localhost]" > group1_inventory echo "127.0.0.1" >> group1_inventory echo "[ec2hosts]" >> group1_inventory ansible-playbook -i group1_inventory launch_aws_instances-group1.yaml
I set again the credentials because as this Bash Shell Script is invoked from Python, there are not sourced.
The trick in here is that the launch_aws_instances-group1.yaml file has a task to add the hosts to Ansible’s in memory inventory, and to print the information.
That output is what I scrap from Python and then I use extract_public_ips_from_text() showed before.
So my launch_aws_instances-group1.yaml (which I generate from Python customizing the parameter) looks like this:
# launch_aws_instances.yaml
- hosts: localhost
connection: local
gather_facts: False
vars:
s_keypair_name: "ansible"
s_instance_type: "t1.micro"
s_image: "ami-08edbb0e85d6a0a07"
s_group: "ansible"
s_region: "eu-west-1"
i_exact_count: 1
s_tag_name: "ansible_group1"
tasks:
- name: Provision a set of instances
ec2:
key_name: "{{ s_keypair_name }}"
region: "{{ s_region }}"
group: "{{ s_group }}"
instance_type: "{{ s_instance_type }}"
image: "{{ s_image }}"
wait: true
exact_count: "{{ i_exact_count }}"
count_tag:
Name: "{{ s_tag_name }}"
instance_tags:
Name: "{{ s_tag_name }}"
register: ec2_ips
- name: Add all instance public IPs to host group
add_host: hostname={{ item.public_ip }} groups=ec2hosts
loop: "{{ ec2_ips.instances }}"
In this case I use t1.micro cause I provision to EC2-Classic and not to the default VPC, otherwise I would use t2.micro.
So I have a Security Group named ansible created in Amazon AWS EC2 console as EC2-Classic, and not as VPC.
In this Security group I opened the Inbound HTTP Port and the SSH port for the Ip from I’m provisioning, so Ansible can SSH using the Key ansible.pem
The Public Key has been created and named ansible as well (section key_name under ec2).
The Image used is Ubuntu 20.04 LTS (free tier) for the region eu-west-1 which is my wonderful Ireland.
For the variables (vars) I use the MT Notation, so the prefixes show exactly what we are expecting s_ for Strings i_ for Integers and I never have collisions with reserved names.
It is very important to use the count_tag and instance_tags with the name of the group, as the actions will be using that group name. Remember the idempotency.
The task with the add_host is the one that makes the information for the instances to be displayed, like in this screenshot.
