Here is an easy trick that you can use for adding swap temporarily to a Server, VMs or Workstations, if you are in an emergency.
In this case I had a cluster composed from two instances running out of memory.
I got an alert for one of the Servers, reporting that only had 7% of free memory.
Immediately I checked it, but checked also any other forming part of the cluster.
Another one appeared, had just only a bit more memory than the other, but was considered in Critical condition too.
The owner of the Service was contacted and asked if we can hold it until US Business hours. Those Servers were going to be replaced next day in US Business hours, and when possible it would be nice not to wake up the Team. It was day in Europe, but night in US.
I checked the status of the Server with those commands:
# df -h
There are 13GB of free space in /. More than enough to be safe as this service doesn’t use much.
# free -h total used free shared buff/cache available Mem: 5.7G 4.8G 139M 298M 738M 320M Swap: 0B 0B 0B
I checked the memory, ok, there are only 320MB free in this node, but 738MB are buff/cache. Buff/Cache is memory used by Linux to optimize I/O as long as it is not needed by application. These 738 MB in buff/cache (or most of it) will be used if needed by the System.
With 1GB free out of 5.7GB it seemed that it would hold perfectly to Business hours.
I checked top. It is interesting to mention the meaning of the Column RES, which is resident memory, in other words, the real amount of memory that the process is using.
I had a Java process using 4.57GB of RAM, but a look at how much Heap Memory was reserved and actually being used showed a Heap of 4GB (Memory reserved) and 1.5GB actually being used for real, from the Heap, only.
It was unlikely that elastic search would use all those 4GB, and seemed really unlikely that the instance will suffer from memory starvation with 2.5GB of 4GB of the Heap free, 1GB of RAM in buffers/cache and free, so looked good.
To be 100% sure I created a temporary swap space in a file on the SSD.
# fallocate -l 1G /swapfile-temp # dd if=/dev/zero of=/swapfile-temp bs=1024 count=1048576 status=progress 1034236928 bytes (1.0 GB) copied, 4.020716 s, 257 MB/s 1048576+0 records in 1048576+0 records out 1073741824 bytes (1.1 GB) copied, 4.26152 s, 252 MB/s
The speed writing to the file is fair enough for a swap.
# chmod 600 /swapfile-temp # mkswap /swapfile-temp Setting up swapspace version 1, size = 1048572 KiB no label, UUID=5fb12c0c-8079-41dc-aa20-21477808619a # swapon /swapfile-temp
I check that memory is good:
# free -h total used free shared buff/cache available Mem: 5.7G 4.8G 117M 298M 770M 329M Swap: 1.0G 0B 1.0G
And finally I check that the Kernel parameter swappiness is not too aggressive:
# sysctl vm.swappiness vm.swappiness = 30
Cool. 30 is a fair enough value.