I refactor and add unit testing to my actual project cmemgzip while I comment every step so you can learn the whys and the reasoning.
Open video in full screen with max 1080 quality, in order to see the code clearly.
So you are trying to program the Raspberry expansion PINS in Python, for example for this 3D LED Christmas Tree, and you’re getting the error:
GPIO.setup(self.number, GPIO.IN, self.GPIO_PULL_UPS[self._pull])
RuntimeError: Not running on a RPi!
I’m running this on Ubuntu 20.04LTS with a Raspberry 4.
The first thing:
Or at least, make sure your USB charger provides enough intensity to power the Raspberry and the LEDs.
The LED power comes from the motherboard and if Raspberry Pi has not enough energy this is not going to work.
My colleague Michela had her tree not working because of the charger was not able to provide enough energy. When she ordered a new charger, it worked like a charm.
In order to communicate with General Purpose Input Output ports (GPIO) you need to install this Software:
sudo apt install python3-pip python3-gpiozero
sudo pip3 install giozerosudo pip3 install colorzero
sudo pip3 install rpi.gpio --upgradeIt may be not required in some cases.
https://github.com/ThePiHut/rgbxmastree#rgbxmastree
I saw many people stuck, in the forums, because of that.
To work with the LEDs you need to run the samples as root.
I used it in my Open Source monitor Software CTOP.py in order to show the plugins/extensions capability of it. :)
I’ve been helping to troubleshoot the reason one Commodity Server (with no iDrac/Ilo ipmi) is powering off randomly. One of the hypothesis is the temperature.
This is a very simple script that will print the temperature of the HDDs and the CPU and keep to a log file.
First you need to install hddtemp and lm-sensors:
sudo apt install hddtemp lm-sensorsThen this is the one line script, that you should execute as root:
while [ true ]; do date | tee -a /var/log/hddtemp.log; hddtemp /dev/sda /dev/sdb /dev/sdc /dev/sdd | tee -a /var/log/hddtemp.log; date | tee -a /var/log/cputemp.log; sensors | tee -a /var/log/cputemp.log; sleep 2; done
Feel free to change sleep 2 for the number of seconds you want to wait, like sleep 10.
Press CTRL + C to interrupt the script at any time.
You can execute this inside a screen session and leave it running in Background.
Note that I use tee command, so the output is print to the screen and to the log file.
Happy New Year to all.
Is something very simple, but will help my student friends to validate Input from Keyboard without losing too many hours.
The Input Validation Classes I create in PHP for Privalia or in my PHP Catalonia Framework, are much, much, more powerful, allowing the validation of complete forms, rendering errors, etc… although they were created for Web, and not for Keyboard input.
It recursively goes to all the subdirectories looking for .py files, and then it counts the lines.
So people can benefit from this during the lock down.
I cannot share it, but I guess that at some point they will publish it on their blog:
I’m teaching a friend how to code. Covid-19 has affected many people, with many jobs loss in many sectors. I encouraged him to learn to code and I helped him to access to a course offered by the Catalan government, but they let him down with a really difficult exam in the very last moment.
I encouraged him to continue and offered to help him, and after a start with several courses via Internet, that were not very useful, I started teaching and training him personally. And I did the way the companies need: Python, PyCharm, Git, Linux, VirtualBox, Web…
I told him since the beginning that I can help him, yes, but 90% has to come from his effort. I’m really happy to see that he is really doing the effort and how much he advanced.
Was not easy as I had to combine it with the classes in the university, my work, the Open Source projects, and my life… but I’m proud of him and of the effort he is doing.
Today he remembered me how shocked he was when I showed some Python builtin libraries, like datetime, and he saw in PyCharm, that the code of that library alone, has 2,476 lines.
I wanted to explain that we can get to create a program with thousands of lines of code, easily, so I decided to write a small program to count the number of lines of our programs.
I quickly wrote it in bash in one single line:
i_counter=0; for s_filename in `find . -name "*.py"`; do wc --lines $s_filename; i_num=`cat $s_filename | wc --lines`; i_counter=$((i_counter+i_num)); done; echo "Total lines: $i_counter"Execute it from inside your directory.
It can be improved and optimized very easily.
I made some improvements abovementioned and uploaded this as a several lines script:
Please note this script does not only run in Linux, it can also be executed in Windows using GitBash.
For my Open Source project CTOP.py it’s 4,652 lines of Python code. :)
I share with you a very lightweight, stand alone, Java validation Class for Input Keyboard, that I created for a university project.
The first thing to mention is an advice about using the Scanner with nextInt(), nextDouble(), next()…
My recommendation is to avoid using these. Work only with Strings and convert to the right Datatype. So use only nextLine(). You will save hours of frustration.
Those are the methods that I’ve implemented:
String[] a_s_acceptedValues = new String[4];
a_s_acceptedValues[0] = “Barcelona”;
a_s_acceptedValues[1] = “Cork”;
a_s_acceptedValues[2] = “Irvine”;
a_s_acceptedValues[3] = “Edinburgh”;
String s_answer = o_inputFromKeyboard.askQuestion(“What is your favorite city?”, a_s_acceptedValues);
Here is the code for the Class InputFromKeyboard.
package com.carlesmateo;
import java.util.Scanner;
public class InputFromKeyboard {
private static Scanner keyboard = new Scanner(System.in);
public static int askQuestion(String question, int minValue, int maxValue) {
// Make sure we enter in the loop
int value = minValue -1;
while (value < minValue || value > maxValue) {
try {
String valueString = askQuestion(question);
value = Integer.parseInt(valueString);
} catch (Exception e) {
System.out.println("Invalid number!");
}
}
return value;
}
public static float askQuestion(String question, float minValue, float maxValue) {
// Make sure we enter in the loop
float value = minValue -1;
while (value < minValue || value > maxValue) {
try {
String valueString = askQuestion(question);
value = Float.parseFloat(valueString);
} catch (Exception e) {
System.out.println("Invalid number!");
}
}
return value;
}
public static double askQuestion(String question, double minValue, double maxValue) {
// Make sure we enter in the loop
double value = minValue -1;
while (value < minValue || value > maxValue) {
try {
String valueString = askQuestion(question);
value = Double.parseDouble(valueString);
} catch (Exception e) {
System.out.println("Invalid number!");
}
}
return value;
}
public static String askQuestion(String question) {
// Make sure we enter in the loop
String answer = "";
while (answer.equals("")) {
try {
System.out.print(question);
answer = keyboard.nextLine();
} catch (Exception e) {
System.out.println("Invalid value!");
// System.out.println(e.getMessage());
}
}
return answer;
}
public static String askQuestionLength(String question, int minLength, int maxLength) {
// Make sure we enter in the loop
String answer = "";
while (answer.equals("")) {
try {
System.out.print(question);
answer = keyboard.nextLine();
int length = answer.length();
if (length < minLength || length > maxLength) {
answer = "";
System.out.println("Answer should be between " + minLength + " and " + maxLength + " characters.");
}
} catch (Exception e) {
System.out.println("Invalid value!");
// System.out.println(e.getMessage());
}
}
return answer;
}
public static String askQuestion(String question, String[] allowedValues) {
// Make sure we enter in the loop
String answer = "";
while (answer.equals("")) {
try {
System.out.print(question);
answer = keyboard.nextLine();
boolean found = false;
for (int counter=0; counter < allowedValues.length; counter++) {
if (answer.equals(allowedValues[counter])) {
found = true;
}
}
if (found == false) {
System.out.println("Please enter any of the allowed values:");
for (int counter=0; counter < allowedValues.length; counter++) {
System.out.println(allowedValues[counter]);
}
answer = "";
}
} catch (Exception e) {
System.out.println("Invalid value!");
// System.out.println(e.getMessage());
}
}
return answer;
}
}Is a reduced subset of functionality respect what I created in my Web PHP Framework Catalonia Framework, or the Input Data Validation that my friend Joim created in Privalia and later I extended and maintained.
All the Frameworks have similar Input Data functionalities, some much more complex, supporting RegExp, or several rules, multiselection (typical for web), accepting only a subset of characters, but this is the typical 80% functionality that everybody needs when writing a console program.
https://www.bbc.com/news/business-54482245
Long time ago I created some ZFS tools that I want to share soon as Open Source.
I equipped myself with the proper Hardware to test on SAS and SATA:
Finally I will have my empty USB keys located and protected. ;)
Remember to be always nice to robots. :)
That’s one of the problems with Python. Blocks of code are defined by their indentation position.
That’s a pain when you copy and past and the IDE reindents the code thinking that is doing great, or generate a new inner class instead of replacing all the code.
Well, this error is very annoying cause it means that you mixed spaces and Tabs as indent separators.
But you can go crazy trying to find a tab in your code, so there is a trick that I came with:
Basically go to Menu Edit > Find and then type 4 times space. PyCharm will highlight all the places were this indentation (4 spaces) is present, so you’ll find the impostor without going blind or losing to many time.
As you can see, in front of def execute_command_without_waiting we don’t have 4 spaces. And in this case the impostor was not a camouflaged tab \t but 3 spaces instead of four.
There are many robots scanning sites for vulnerabilities, to gain control or exploit the servers. Most of them come from China and Russia ip’s.
Here I explain an easy way to block them using the Ubuntu Firewall ufw.
If you use a CMS like WordPress and you know there are extensions that have had security exploits, for example, wp-file-manager then you can search directly for this request in Apache Access Logs.
For example:
cat /var/log/apache2/blog_carlesmateo_com-access.log | grep "wp-file-manager" | awk '{ print $1; }' | sort -u >> 2020-10-03-offending-ips.txt
cat /var/log/apache2/blog_carlesmateo_com-access.log.1 | grep "wp-file-manager" | awk '{ print $1; }' | sort -u >> 2020-10-03-offending-ips.txt
zcat /var/log/apache2/blog_carlesmateo_com-access.log.2.gz | grep "wp-file-manager" | awk '{ print $1; }' | sort -u >> 2020-10-03-offending-ips.txtIn the example we look for the access.log file, for the rotated access.log.1 and for the rotated and compressed access.log.2.gz. We use the tool zcat which does a cat over a compressed file.
If we don’t expect to have anybody posting to our xmlrpc Service, we can check for the offending Ip’s by doing:
cat /var/log/apache2/blog_carlesmateo_com-access.log | grep "POST /xmlrpc.php" | wc --lines
2490In my case I have 2490 request just in the last log.
cat /var/log/apache2/blog_carlesmateo_com-access.log | grep "POST /xmlrpc.php" |awk '{ print $1; }' | sort -u | wc --linesInterested in how many Ip’s are launching those requests, you can see how many different Ip’s are those:
cat /var/log/apache2/blog_carlesmateo_com-access.log | grep "POST /xmlrpc.php" |awk '{ print $1; }' | sort -u | wc --lines
145And to add those Ip’s to the offending Ip’s list:
cat /var/log/apache2/blog_carlesmateo_com-access.log | grep "POST /xmlrpc.php" | awk '{ print $1; }' | sort -u >> 2020-10-03-offending-ips.txtI can also check for repeated requests in the logs:
cat /var/log/apache2/blog_carlesmateo_com-access.log | awk '{ print $7; }' | sort | uniq -c | sort -r | lessThat shows me some requests legit and others that are not:
2532 /xmlrpc.php
209 /wp-login.php
205 /wp-admin/admin-ajax.php
84 /
83 *
48 /robots.txt
21 /favicon.ico
16 /wp-login.php?redirect_to=https%3A%2F%2Fblog.carlesmateo.com%2Fwp-admin%2F&reauth=1
15 /wp-includes/js/jquery/jquery.js?ver=1.12.4-wp
14 /wp-includes/css/dist/block-library/theme.min.css?ver=5.5.1
14 /wp-includes/css/dist/block-library/style.min.css?ver=5.5.1
14 /wp-content/themes/2012-carles/style.css?ver=5.5.1
14 /wp-content/plugins/contact-form-7/includes/js/scripts.js?ver=5.2.2
14 /wp-content/plugins/captcha/css/front_end_style.css?ver=4.4.5
13 /wp-includes/css/dashicons.min.css?ver=5.5.1
13 /wp-content/themes/2012-carles/css/blocks.css?ver=20181230
13 /wp-content/plugins/contact-form-7/includes/css/styles.css?ver=5.2.2
12 /wp-includes/js/wp-embed.min.js?ver=5.5.1
12 /wp-includes/images/w-logo-blue-white-bg.png
12 /wp-content/themes/2012-carles/js/navigation.js?ver=20140711
11 /wp-includes/js/wp-emoji-release.min.js?ver=5.5.1
11 /wp-content/plugins/captcha/css/desktop_style.css?ver=4.4.5
11 /feed/
11 /contact/
10 /wp-comments-post.php
10 /?author=1
9 /2016/06/30/creating-a-content-filter-for-postfix-in-php/
9 /2014/10/13/performance-of-several-languages/
8 /wp-includes/js/comment-reply.min.js?ver=5.5.1
8 /wp-content/plugins/captcha/js/front_end_script.js?ver=5.5.1
8 /e/admin/index.php
8 /e/admin/
7 /wp-login.php?action=register
7 /current-projects/
7 //xmlrpc.php
6 /.env
5 /2019/08/12/a-sample-forensic-post-mortem-for-a-iscsi-initiator-client-that-had-connectivity-problems-to-the-server/
5 /2017/03/26/csort-multithread-versus-quicksort-java/
4 /wp-json/wp/v2/types/wp_block?_locale=user
4 /wp-json/wp/v2/blocks?per_page=100&_locale=user
4 /wp-admin/
4 /diguo/index.php
4 /diguo/
4 /category/web-development/
4 /category/news-for-the-blog/
3 /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php
3 /mt-notation-for-python/
3 /ebk/index.php
3 /ebk/
3 /comments/feed/
3 /bf/index.php
3 /bf/
3 /beifen/index.php
3 /beifen/
3 /Ebak/index.php
3 /Ebak/
3 /Bak/index.php
3 /Bak/
3 /2020/09/21/how-to-recover-access-to-your-amazon-aws-ec2-instance-if-you-loss-your-private-key-for-ssh/
3 /2020/08/23/adding-a-ramdisk-as-slog-zil-to-zfs/
3 /2019/07/03/adding-my-server-as-docker-with-php-catalonia-framework-explained/
3 /2019/06/25/some-handy-tricks-for-working-with-zfs/
3 /2015/02/01/stopping-definitively-the-massive-distributed-dos-attack/
2 /ycadmin/login.php?gotopage=%2Fycadmin%2Findex.php
2 /ueditor/net/controller.ashx
2 /sql_beifen/index.php
2 /sql_beifen/
2 /sql/index.php
2 /sql/
2 /dgbf/index.php
2 /dgbf/
2 //xmlrpc.php?rsd
2 //.env
1 /wp-login.php?registration=disabled
1 /wp-login.php?action=lostpassword
1 /wp-json/wp/v2/users/me?_locale=user
1 /wp-json/wp/v2/users/?who=authors&per_page=100&_locale=user
1 /wp-json/wp/v2/taxonomies/post_tag?context=edit&_locale=user
1 /wp-json/wp/v2/taxonomies/category?context=edit&_locale=user
1 /wp-json/wp/v2/tags?per_page=100&orderby=count&order=desc&_fields=id%2Cname&search=ufw&_locale=user
You can identify manually what are attacks, and what are legit requests.
After you have your definitive list of offending Ip’s (and make sure you didn’t introduce yours accidentally), then you can execute the second part of the script:
echo '#!/bin/bash' > add_ufw_rules.sh
i_COUNTER_RULE=0; for s_OFFENDING_IP in $(cat 2020-10-03-offending-ips.txt); do i_COUNTER_RULE=$((i_COUNTER_RULE+1)); echo "ufw insert $i_COUNTER_RULE deny from $s_OFFENDING_IP to any" >> add_ufw_rules.sh; done
echo "ufw status numbered" >> add_ufw_rules.sh
echo "sudo ufw allow OpenSSH" >> add_ufw_rules.sh
echo "sudo ufw allow 22/tcp" >> add_ufw_rules.sh
echo 'sudo ufw allow "Apache Full"' >> add_ufw_rules.sh
echo "sudo ufw enable" >> add_ufw_rules.sh
Then you less your file add_ufw_rules.sh to see everything is Ok:
#!/bin/bash
ufw insert 1 deny from 40.79.250.88 to any
ufw insert 2 deny from 52.173.148.212 to any
ufw insert 3 deny from 94.103.85.175 to any
ufw insert 4 deny from 40.79.250.88 to any
ufw insert 5 deny from 78.85.208.240 to any
ufw insert 6 deny from 80.82.68.173 to any
ufw insert 7 deny from 188.165.230.118 to any
ufw insert 8 deny from 195.201.117.103 to any
ufw insert 9 deny from 40.79.250.88 to any
ufw insert 10 deny from 5.135.138.188 to any
ufw insert 11 deny from 51.116.189.135 to any
...
ufw insert 223 deny from 95.173.161.167 to any
ufw insert 224 deny from 95.84.228.227 to any
ufw status numbered
sudo ufw allow OpenSSH
sudo ufw allow 22/tcp
sudo ufw allow "Apache Full"
sudo ufw enable
Then you simply give permissions with chmod +x add_ufw_rules.sh and run the script to apply.
It’s up to you to turn on the Firewall logging:
sudo ufw logging onIf you are using Git Submodules, is very probable that at some point you will create you own libraries. Probably those libraries will have their own structure, even with their own tests/ folder and you’re adding into a subfolder into your new project and maybe you have problems using relative imports.
This is a trick you can use to add the relevant root folder of your project to the System Path, so the libraries are found, specially when you call by command line from anywhere in the filesystem. This works for Python2 and Python3.
#!/usr/bin/env python3
import sys
import os
s_path_program = os.path.dirname(__file__)
sys.path.append(s_path_program + '../../')
from clib.src.argsutils import ArgsUtils
from clib.src.datetimeutils import DateTimeUtils
from clib.src.fileutils import FileUtilsThis sample can be found in my book Pythom Combat Guide.