Last Update: 2022-02-19 15:08 Irish Time

This small script will count repeated patterns in the Logs.

Ideal for checking if there are errors that you’re missing while developing.

#!/usr/bin/env bash
# count_repeated_pattern_in_logs.sh
# By Carles Mateo
# Helps to find repeated lines in Logs
LOGFILE_MESSAGES="/var/log/messages"
LOGFILE_SYSLOG="/var/log/syslog"
if [[ -f "${LOGFILE_MESSAGES}" ]]; then
    LOGFILE=${LOGFILE_MESSAGES}
else
    LOGFILE=${LOGFILE_SYSLOG}
    if [[ ! -f "${LOGFILE_SYSLOG}" ]]; then
        echo "${LOGFILE_MESSAGES} and ${LOGFILE_SYSLOG} do not exist. Exitting"
        exit 1
    fi
fi
echo "Using Logfile: ${LOGFILE}"
CMD_OUTPUT=`cat ${LOGFILE} | awk '{ $1=$2=$3=$4=""; print $0 }' | sort | uniq --count | sort --ignore-case --reverse --numeric-sort`
echo -e "$CMD_OUTPUT"

Basically it takes out the non relevant fields that can prevent from detecting repetition, like the time, and prints the rest.
Then you will launch it like this:

count_repeated_pattern_in_logs.sh | head -n20

If you are checking a machine with Ubuntu UFW (Firewall) and want to skip those likes:

./count_repeated_pattern_in_logs.sh | grep -v "UFW BLOCK" | head -n20

You can also run the same against the output of dmesg -T for counting over the messages in the Kernel this year:

dmesg -T | awk '{ $1=$2=$3=$4=""; print $0 }' | sort | uniq --count | sort --ignore-case --reverse --numeric-sort

And as sample output, the top messages are Ip’s blocked by the Firewall:

     10     2022] [UFW BLOCK] IN=ens4 OUT= MAC=42:01:02:03:04:05:06:07:0a:80:00:01:08:00 SRC=3.217.247.223 DST=10.128.0.2 LEN=40 TOS=0x00 PREC=0x00 TTL=54 ID=0 DF PROTO=TCP SPT=60636 DPT=443 WINDOW=0 RES=0x00 RST URGP=0
      8     2022] [UFW BLOCK] IN=ens4 OUT= MAC=42:01:02:03:04:05:06:07:0a:80:00:01:08:00 SRC=99.41.165.200 DST=10.128.0.2 LEN=40 TOS=0x00 PREC=0x00 TTL=57 ID=0 DF PROTO=TCP SPT=53960 DPT=443 WINDOW=0 RES=0x00 RST URGP=0
      7     2022] [UFW BLOCK] IN=ens4 OUT= MAC=42:01:02:03:04:05:06:07:0a:80:00:01:08:00 SRC=183.82.177.237 DST=10.128.0.2 LEN=40 TOS=0x00 PREC=0x00 TTL=59 ID=0 DF PROTO=TCP SPT=17697 DPT=443 WINDOW=0 RES=0x00 RST URGP=0
      6     2022] [UFW BLOCK] IN=ens4 OUT= MAC=42:01:02:03:04:05:06:07:0a:80:00:01:08:00 SRC=157.90.181.146 DST=10.128.0.2 LEN=40 TOS=0x00 PREC=0x00 TTL=60 ID=0 DF PROTO=TCP SPT=53558 DPT=443 WINDOW=0 RES=0x00 RST URGP=0
      6     2022] [UFW BLOCK] IN=ens4 OUT= MAC=42:01:02:03:04:05:06:07:0a:80:00:01:08:00 SRC=137.226.113.44 DST=10.128.0.2 LEN=40 TOS=0x00 PREC=0x00 TTL=43 ID=0 DF PROTO=TCP SPT=55338 DPT=443 WINDOW=0 RES=0x00 RST URGP=0