My talk in Google Developers Cork Group.
It’s about deploying an Instance in GCE and grows in complexity until Deploying a Load Balancer with AutoScaling for a group of LAMP Webservers.
Join the group at: https://www.meetup.com/GDG-Cork/
The videos:
Keshan Sodimana: Tensors
Another Python curiosity.
If you see the Official Python3 documentation for strip(), it says that strip without parameters will return the string without the leading and trailing white spaces.
Optionally you can pass a string with the characters you want to eliminate.
The official documentation for Python 2 says:
string.strip(s[, chars])
Return a copy of the string with leading and trailing characters removed. If chars is omitted or None, whitespace characters are removed. If given and not None, chars must be a string; the characters in the string will be stripped from the both ends of the string this method is called on.
Changed in version 2.2.3: The chars parameter was added. The chars parameter cannot be passed in earlier 2.2 versions.
https://docs.python.org/2/library/string.html
A white space is a white space. Is not an Enter.
But strip() without parameters will remove white spaces (space), and Enter \n and Tabs \t.
Probably you will not realize that unless you read from a file that has empty lines at the end for a reason, and you use strip().
You can see a demonstration following this small program, that runs the same for Python2 and Python3.
And the corresponding output for python2 and python3:
The [ ] characters where added to show that there are no hidden tabs or similar after.
Here I paste the code so you can try yourself:
import sys
def print_bar():
print("-----------------------------------------------------")
def print_between_brackets(s_test):
print("[" + s_test + "]")
s_string_with_enters = " Testing strip not only removing white spaces, but Enter and Tabs s well\n\t\n\n"
print("Testing .strip()")
print("You are running Python " + sys.version)
print("This is the original string")
print_bar()
print_between_brackets(s_string_with_enters)
print_bar()
print("Now after strip()...")
print_bar()
print_between_brackets(s_string_with_enters.strip())
print_bar()
print("As you can see the Enters and the Tabs have been removed, not just the spaced")
I think this should be disambiguate so I decided to take action. Is very easy to blame and never contribute. Not me. I went to Python to fix that and I located a bug reporting this issue:
https://bugs.python.org/issue25433
The issue was registed and made specially interesting contributions by Dimitri Papadopoulos Orfanos.
The thread is really interesting to read. I recommend it.
At a glance:
“Python heavily relies on isspace() to detect “whitespace” characters.”
* Lib/string.py near line 23: whitespace = ' \t\n\r\v\f'
So all those characters will be stripped in Python2.7 if you use just string.strip()
The ticket was opened the 2015-10-18 12:15. So it’s a shame the documentation has not been updated yet, more than 3 years later. Those are the kind of things, lack of care, that I can’t understand. Not looking for the excellence.
Please, do note that Python3 supports Unicode natively and things are always a bit different than with Python2 and AscII.
I lead a project where I decided to go with Python 2.7, for the wide compatibility across all the Servers around.
With RHEL now supporting Python 3 as well, it doesn’t make much sense any more, as all the major brands do support Python 3 directly.
I saw it coming so in my Coding Style Guide for my Team I explained that we will use print(“”) which is the required way to proceed with Python 3, as opposite to print “whatever” from Python 2. Noye Python 2 supports both methods.
But today something unexpected appeared in the Tests. One line of code was making a print of ().
The line of code was:
print()
And not
print("")
And the curiosity is that if you do print() in Python 2.7 it outputs ().
Ok, so I lend one of my Servers to two of my colleagues in The States, that required to prepare some test for a customer. I always try to be nice and to stimulate sales across the organizations I help, so if they need a Server for a PoC and demo to a customer, they know they can count on me.
It is important to remark that the Servers I was using had two motherboards, with their CPU and RAM, and Dual Port SAS drives. We had those Servers so we can implement High Availability. The Dual Port SAS allow two different computers or IO controllers to access the same drive at the same time.
I work with Declustered RAID, DRAID, and ZFS.
The Server was a 4U90, so a 4U Server with 90 SAS3 spinning drives and 4 SSD. Drives are Dual Ported, and two Controllers (motherboard + CPU + RAM) have access simultaneously to the drives for HA.
After their tests my colleagues, returned me the Server, and I needed to use it and my surprise was when I tried to provision with ZFS and I encountered problems. Not much in the logs. Please note I was using only one node (or controller), and the other was not in use but they ask me to keep the OS and the data (in 2xMD drive). I shutdown the node A after the Engineers in San Jose powered the Server off, so only my node was working.
I checked:
cat /proc/mdstat
And that was the thing 8 MD Arrays where there.
[root@4u90-B ~]# cat /proc/mdstat Personalities : md2 : inactive sdba1[9](S) sdag1[7](S) sdaf1[3](S) 11720629248 blocks super 1.2 md1 : inactive sdax1[7](S) sdad1[5](S) sdac1[1](S) sdae1[9](S) 12056071168 blocks super 1.2 md0 : inactive sdat1[1](S) sdav1[9](S) sdau1[5](S) sdab1[7](S) sdaa1[3](S) 19534382080 blocks super 1.2 md4 : inactive sdbf1[9](S) sdbe1[5](S) sdbd1[1](S) sdal1[7](S) sdak1[3](S) 19534382080 blocks super 1.2 md5 : inactive sdam1[1](S) sdan1[5](S) sdao1[9](S) 11720629248 blocks super 1.2 md8 : inactive sdcq1[7](S) sdz1[2](S) 7813752832 blocks super 1.2 md7 : inactive sdbm1[7](S) sdar1[1](S) sdy1[9](S) sdx1[5](S) 15627505664 blocks super 1.2 md3 : inactive sdaj1[9](S) sdai1[5](S) sdah1[1](S) 11720629248 blocks super 1.2 md6 : inactive sdaq1[7](S) sdap1[3](S) sdr1[8](S) sdp1[0](S) 15627505664 blocks super 1.2
Ok. So I stop the Arrays
mdadm --stop /dev/md127
And then I zero the superblock:
mdadm --zero-superblock /dev/sdb1
After doing this for all I try to provision and… surprise! does not work. /dev/md127 has respawned like in the old times from Doom video game.
I check the mdmonitor service and even disable it.
systemctl disable mdmonitor
I repeat the process.
And /dev/md127 appears again, using another device.
At this point, just in case, I check the other controller, which should be powered off.
Ok, it was on. With different Ip, so it was not answering to ping, but I still had access to BMC//IPMI. After confirming with my colleagues that I can shutdown that node (they did not turn it on apparently) I launch the poweroff command, and repeat, same!.
I see that the poweroff command on the second Controller is doing a reboot, not poweroff. Is a Firmware issue I find. So I access to the Linux from the management tool and I launch the halt command that makes it not respond to the ping anymore.
I repeat the process, and still the ghost md array appears there, and blocks me from doing my zpool create.
The /etc/mdadm.conf file did not exist (by default is not created).
I try a more aggressive approach:
DRIVES=`cat /proc/partitions | grep 3907018584 | awk '{ print $4; }'`
for DRIVE in $DRIVES; do echo "Trying /dev/${DRIVE}1"; mdadm --examine /dev/${DRIVE}1; done
Ok. And destruction time:
for DRIVE in $DRIVES; do echo "Trying /dev/${DRIVE}"; wipefs -a -f /dev/${DRIVE}; done
for DRIVE in $DRIVES; do echo "Trying /dev/${DRIVE}1"; mdadm --zero-superblock /dev/${DRIVE}1; done
Apparently the system is clean, but still I cannot provision, and /dev/md127 respawns and reappears all the time.
After googling and not finding anything about this problem, and my colleagues no having clue about what is causing this, I just proceed with a simple solution, as I need the Server for my company completing the tests in the next 24 hours.
So I create the file /etc/mdadm.conf with this content:
[root@draid-08 ~]# cat /etc/mdadm.conf AUTO -all
After that I rebooted the Server and I saw the infamous /dev/md127 is not there and I’m able to provision.
I share the solution as it may help other people.
The most straightforward procedure would had been reinstalling clean the OS, but this operation is very slow when simulating a Virtual CD remotely, so it was worth fixing that as OS level, as I save one day delaying my work.
ZFS has a performance problem with the zvol volumes.
Even using a ZIL you will experience low speed when writing to a zvol through the Network.
Even locally, if you format a zvol, for example with ext4, and mount locally, you will see that the speed is several times slower than the native ZFS filesystem.
zvol volumes are nice as they support snapshots and clone (from the snapshot), however too slow.
Using a pool with Spinning Drives and two SSD SLOG devices in mirror, with a 40Gbps Mellanox NIC accessing a zvol via iSCSI, with ext4, from the iSCSI Initiator, you can be copying Data at 70 MB/s, so not even saturating the 1Gbps.
The trick to speed up this consist into instead of using zvols, creating a file in the ZFS File System, and directly share it through iSCSI.
This will give 4 times more speed, so instead of 70MB/s you would get 280MB/s.
Many times it could be very convenient to have a compressed filesystem, so a system that compresses data in Real Time.
This not only reduces the space used, but increases the IO performance. Or better explained, if you have to write to disk 1GB log file, and it takes 5 seconds, you have a 200MB/s performance. But if you have to write 1GB file, and it takes 0.5 seconds you have 2000MB/s or 2GB/s. However the trick in here is that you really only wrote 100MB, cause the Data was compressed before being written to the disk.
This also works for reading. 100MB are Read, from Disk, and then uncompressed in the memory (using chunks, not everything is loaded at once), assuming same speed for Reading and Writing (that’s usual for sequential access on SAS drives) we have been reading from disk for 0.5 seconds instead of 5. Let’s imagine we have 0.2 seconds of CPU time, used for decompressing. That’s it: 0.7 seconds versus 5 seconds.
So assuming you have installed ZFS in your Desktop computer those instructions will allow you to create a ZFS filesystem, compressed, and mount it.
ZFS can create pools using disks, partitions or other block devices, like regular files or loop devices.
# Create the File that will hold the Filesystem, 1GB
root@xeon:/home/carles# dd if=/dev/zero of=/home/carles/compressedfile.000 bs=1M count=1024 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.621923 s, 1.7 GB/s
# Create the pool
zpool create compressedpool /home/carles/compressedfile.000
# See the result
# If you don’t have automount set, then set the mountpoint
zpool set compressedpool mountpoint=/compressedpool
# Set the compression. LZ4 is fast and well balanced
zfs set compression=lz4 compressedpool
# Push some very compressible 1GB file. Don’t use just 0s as this is optimized :)
# Myself I copied real logs
ls -al --block-size=M *.log -rw------- 1 carles carles 1329M Sep 26 14:34 messages.log root@xeon:/home/carles# cp messages.log /compressedpool/
Even if the pool only had 1GB we managed to copy 1.33 GB file.
Then we check and only 142MB are being used for real, thanks to the compression.
root@xeon:/home/carles# zfs list NAME USED AVAIL REFER MOUNTPOINT compressedpool 142M 738M 141M /compressedpool root@xeon:/home/carles# df /compressedpool Filesystem 1K-blocks Used Available Use% Mounted on compressedpool 899584 144000 755584 17% /compressedpool
By default ZFS will only import the pools that are based on drives, so in order to import your pool based on files after you reboot or did zfs export compressedpool, you must specify the directory:
zpool import -d /home/carles compressedpool
You can also create a pool using several files from different hard drives. That way you can create mirror, RAIDZ1, RAIDZ2 or RAIDZ3 and not losing any data in that pool based on drives in case you loss a physical drive.
If you use one file in several hard drive, you are aggregating the bandwidth.
You can also do this in your instances or VMs. Create one file of 1GB and creating the pool for compressed logs or compressed core dumps. If later you need more space you can add another file to he pool. You don’t need to use any redundancy, just creating a pool with mountpoint /var/log or /var/core and grow as you need.
Logs and core dumps can be greatly compressed, for example a core dump of 54MB will be around 645KB if you compress it using a tool like bzip2. Using the compression from ZFS, you can choose different algorithms of compression, so expect a massive reduction of space and huge space savings for logs and core dumps.
This September I was invited to talk in OpenZFS 2018 about DRAID and Cold Storage (Spinning drives) in San Francisco, US.
Thanks to @delphix for all their kindness.
Here you can watch mine and all the presentations.
The slides:
You can download the video of the sample Rebuild with DRAID in here:
draid-rebuild-read-write-1sec-vokoscreen-2018-09-06_11-37-48
Also in the Hackaton I presented my mini utility run_with_timeout.sh to execute a command (zdb, zpool, zfs, or any shell command like ls, “sleep 5; ping google.com”…) with a timeout, and returning a Header with the Error Level and the Error Level itself.
Hackathon presentations streaming now: https://t.co/qFIrNQ7LWX
— OpenZFS (@OpenZFS) September 11, 2018
Myself I appear at minute 53:50.
Special greetings to my Amazing Team in Cork, Ireland. :)
I illustrate this troubleshooting as it will be useful for some of you.
I requested to one of the members of my Team to compile and to install ZFS 7.9 to some of the Servers loaded with drives, that were running ZFS 7.4 older version.
Those systems were running RHEL7.4.
The compilation and install was fine, however the module was not able to load.
My Team member reported that: when trying to run “modprobe zfs“. It was giving the error:
modprobe: ERROR: could not insert 'zfs': Invalid argument
Also when trying to use a zpool command it gives the error:
Failed to initialize the libzfs library
That was only failing in one of the Servers, but not in the others.
My Engineer ran dmesg and found:
zfs: `' invalid for parameter `metaslab_debug_unload
He though it was a compilation error, but I knew that metaslab_debug_unload is an option parameter that you can set in /etc/zfs.conf
So I ran:
modprobe -v zfs
And that confirmed my suspicious, so I edited /etc/zfs.conf and commented the parameter and tried again. And it failed.
As I run modprobe -v zfs (verbose) it was returning me the verbose info, and so I saw that it was still trying to load those parameters so I knew it was reading those parameters from some file.
I could have grep all the files in the filesystem looking for the parameter failing in the verbose or find all the files in the system named zfs.conf. To me it looked inefficient as it would be slow and may not bring any result (as I didn’t know how exactly my team member had compiled the code), however I expected to get the result. But what if I found 5 or 7 zfs.conf files?. Slow.
I used strace. It was not installed but the RHEL license was active so I simple did:
yum install strace
strace stands for System Trace and so it records all the System Calls that the programs do.
That’s a pro trick that will accompany you all your career.
So I did:
strace modprobe zfs
I did not use -v in here cause all the verbose would had been logged as a System Call and made more difficult my search.
I got the output of all the System Calls and I just had to look for which files were being read.
Then I found that zfs.conf under /etc/modprobe.d/zfs.conf
That was the one being read. So I commented the line and tried modprobe zfs and it worked perfectly. :)
Article created on: 1528997557 | 2018-06-14 18:32:15 IST
Recently a mentor of the UCC university came to visit me to my office, in order to do the following of one of the members of my Team, an intern.
Conversation was well, and then at some point he asked what courses could do the university teach to their students in order to be more prepared for working with us.
The Head of Business Development, that was in the meeting with me, mentioned something interesting:
– Make the publish their best code in github, bitbucket or similar git repository, and maintain it. It is like a CV.
He pointed that some of the students sent me their repository page, and they have not committed a thing for more than a year. And usually the code that I find there is less than a tic-tac-toe exercise.
– Obviously, to have git experience.
– Having contributed to an Open Source project
I exposed some things that would be helpful to have in the interns and grads that I hire:
– git experience
– Python programming
– C programming
– Unit Testing experience
– Networking experience, in particular iSCSI exports, tcpdump
– Programming Best practices, PEP-8 at least for Python
– Usage of Professional Tools like PyCharm, JetBrains IntelliJ, PHPStorm, Code Lion, Netbeans, Eclipse
– Linux experience. Many of them use Windows at home cause they also play video games. Really few programmers in real life use windows. So at least guys install Virtual Box or VMWare and run Linux in an Virtual Machine.
– Cloud experience. Using instances, CDNs, APIs, tools…
And as the talking advanced I gave him a hint of the Epic fail that all the universities are committing.
They teach git for a semester. They teach Python for one or two semester, the first year usually one, the second year another. And that’s it. Is gone.
When they exit the university they have not programmed in Python for 2 or 3 years, they have not used git, they have not used SQL for the same amount of time, etc…
My boss pointed that the best candidates do side projects in their spare time, and have that bright in their eyes. That sparkling in the eyes is what I call the eye of the tiger, the desire to improve, to learn. That spark.
I told the mentor of my intern that the big mistake is doing things in small parcels, isolated, one block and is gone. That the best way to proceed would be to:
Make the student start a project from the very beginning, from the first semester. Then keep making it bigger and better over time.
Let them improve it over time. Screw it in all the ways possible. Make them reach the limits of their initial architecture. Allow them to face having to redo the thing from the scratch. Allow them to do screw it, to break things, and to learn from their mistakes. Over and over.
Nobody becomes a great programmer coding average things for two semesters.
But let them realize where the problems are. Let them come back to their code of two or three months ago, before holidays, and realize how important is to make comments, to give proper names to the files and to the variables. Let them run that project over so many time, that at some point they have to change computer and they realize that what worked with windows Uppercases and Lowercase mixed files, does not work with Unix (case sensitive).
Let them grow.
Let them see their mistakes over the time.
Let them run the project for so long so they switch several times from Cloud provider, and discover the pros and the cons and the not-to-do, and things like run for your life before using sharing hostings that limit your CPU quota even that kills your MySql instances when they look at the email (true history, connecting to POP3 was raising the CPU and the provider was killing the MySQL instances, and so the queries) or that limits your queries per second, and then ask them to install a drupal and they will learn the hard way why Quality is always better than price and will make the right decisions when they work for somebody else or for their own Startup.
Even many of the supposedly Senior guys never learned from their mistakes, for example the Outsourcing guys, cause they work 6 months to a year in a project and then jump to another. Nobody explains the hell in maintenance and incidental they have left there. Nobody teach them.
Programming an small project for 6 months doesn’t make a master. Doing it for 5 years, growing it, learning from your mistakes and learning the YES and DO-NOT the hard way, the real way that works, cause makes you understand why something is better than other things, is the path.
That also remembers me why I love the MT Notation and many of the guys in Barcelona that saw it criticized the method, while my colleagues at Facebook and Dropbox actually told me that they use it, specially for Python and C/C++.
Allow them to thing about how to solve sorting a list of 1000 items by themselves. Let them think. The lazy will copy, but they will not grow.
Then let them implement a Bubble sort. Let them improve it, if they can. Allow them a week to try to improve that. Then make them sort 1,000,000 items so they see that is bloody slow. How can I improve that?. May I read the data from the drive at once, reading line by line was slow… let them think. Like if they were learning Martial Arts, and so discovering their strengths, that they have fast reflexes, allow them to grow.
Universities have to create good professional, not just machines of passing the exams. Real world demands talent, problem solving abilities, passion, ability to learn, and will to do the things well and to improve, and discipline.
After 5-6 years of programming on a daily basis, with an IDE, git, deploying to the Cloud as the basic, and growing a program and seeing the downsides of the solutions chosen, observing that the caveats where for a reason, learning that the Hardware is important, that is not the same to write to memory that to disk or to network, detecting the problems, redoing things, ending in a cul-de-sac, fixing, improving, learning, growing the project, growing himself/herself as a mind, as a programmer, as a thinker, as an expert, daily, even if it’s 30 minutes per day, then that person is prepared for some serious business.
Like piano, guitar, painting, writing… and any other activity, one require continue training in order to improve.
Students have to follow a journey in order to improve.
Let them start with Command Line, i.e. in C and files. Let’s do add later database support.
Deal with buffer overflow, file descriptor, locks and conversion types. Let them migrate to another language the entire project, using Git from the beginning.
Let them migrate again when they need to add Web support. Allow them to discover that instead of reloading all the page they can use Ajax/JSON. Let them deal with click-click that many common users do on the page buttons (so they submit twice the information). To discover SQL Injections. To use a Web Framework. To add Unit Testing. Add some improvement via Javascript Frameworks like responsive for mobiles.
Allow them to use a new Database, new Webserver or technology that is fashion and everybody on Twitter talks about, so it crashes in their face. And so they discover that they will not play or discover new technologies in actual project time in the Company of their future employers, cause shit happens, and impacts the Schedule, and the Company loses money. Universities: Teach them, let the students learn this for themselves, rather than screwing it up in several companies after university.
This is a very simple code, but handy.
I love the output for the simplicity and I use to check for my programs to see what will suit best.
#!/bin/env python
# Collection of Effects
# 1m - Bold
# 2m - Normal Dark colors
# 3m - Italic
# 4m - Underline
# 7m - Background
# 9m - Strikethrough (except 38;9m)
# 40m - Bakground Dark Grey, with the colors in foreground 1 (bold),2,31-37
# 41m - Bakground Red, with the colors in foreground 1 (bold),2,30-37
# 42m - Bakground Green, with the colors in foreground 1 (bold),2,30-37
# 43m - Bakground Yellow, with the colors in foreground 1 (bold),2,30-37
# 44m - Bakground Blue, with the colors in foreground 1 (bold),2,30-37
# 45m - Bakground Violet, with the colors in foreground 1 (bold),2,30-37
# 46m - Bakground Cyan, with the colors in foreground 1 (bold),2,30-37
# 47m - Bakground Grey, with the colors in foreground 1 (bold),2,31-37
# 49m - Normal
CLEAR = "\x1B[0m"
a_s_background = ["1", "2", "3", "4", "7", "9", "40", "41", "42", "43", "44", "45", "46", "47", "49"]
a_s_color = ["1", "2", "31", "32", "33", "34", "35", "36", "37"]
for s_effect in a_s_background:
s_line = ""
for s_color in a_s_color:
s_color_text = "\x1B[" + s_color + ";" + s_effect + "m"
s_line += s_color_text + s_color + ";" + s_effect + "m" + CLEAR + " "
print(s_line)
print("--------------------------------------------")
You can find the source code here:
https://gitlab.com/carles.mateo/blog.carlesmateo.com-source-code/-/blob/master/colors_in_terminal.py
Please, take in count that the colors may be different depending on the Terminal used, so if you’re creating a commercial application I recommend you to try with some of them like: ubuntu terminal, ssh, putty for windows, MobaXTerm…
I like to write Software that does not require from external packages, but if you want to use colors and write a portable program that runs in Linux, Mac OS X and Windows, I recommend you to use colorama.