I illustrate this troubleshooting as it will be useful for some of you.
I requested to one of the members of my Team to compile and to install ZFS 7.9 to some of the Servers loaded with drives, that were running ZFS 7.4 older version.
Those systems were running RHEL7.4.
The compilation and install was fine, however the module was not able to load.
My Team member reported that: when trying to run “modprobe zfs”. It was giving the error:
modprobe: ERROR: could not insert 'zfs': Invalid argument
Also when trying to use a zpool command it gives the error:
Failed to initialize the libzfs library
That was only failing in one of the Servers, but not in the others.
My Engineer ran dmesg and found:
zfs: `' invalid for parameter `metaslab_debug_unload
He though it was a compilation error, but I knew that metaslab_debug_unload is an option parameter that you can set in /etc/zfs.conf
So I ran:
modprobe -v zfs
And that confirmed my suspicious, so I edited /etc/zfs.conf and commented the parameter and tried again. And it failed.
As I run modprobe -v zfs (verbose) it was returning me the verbose info, and so I saw that it was still trying to load those parameters so I knew it was reading those parameters from some file.
I could have grep all the files in the filesystem looking for the parameter failing in the verbose or find all the files in the system named zfs.conf. To me it looked inefficient as it would be slow and may not bring any result (as I didn’t know how exactly my team member had compiled the code), however I expected to get the result. But what if I found 5 or 7 zfs.conf files?. Slow.
I used strace. It was not installed but the RHEL license was active so I simple did:
yum install strace
strace is for System Trace and so it records all the System Calls that the programs do.
That’s a pro trick that will accompany you all your career.
So I did strace modprobe zfs
I did not use -v in here cause all the verbose would had been logged as a System Call and made more difficult my search.
I got the output of all the System Calls and I just had to look for which files were being read.
Then I found that zfs.conf under /etc/modprobe.d/zfs.conf
That was the one being read. So I commented the line and tried modprobe zfs and it worked perfectly. :)