When working with Ubuntu, Debian Linux, and ZFS, you will run into ZFS cache size problems. You see, not all Ubuntu or Debian servers need aggressive file caching. Some servers act as a web server or run Linux container workloads or KVM guest VMs where you want those guest VMs to manage their own caching. Therefore, it would be best to have tons of ECC RAM for ZFS. Unfortunately, not all projects get that kind of extravagance in real life. This page explains how to set up ZFS arc size on Ubuntu/Debian or any Linux distro of your choice. So that, Linux kernel avoid running out of memory.
|Requirements||Linux with ZFS|
|Est. reading time||7 minutes|
WARNING: Instructions presented below only works with Linux distribution. It will not work with FreeBSD, FreeNAS, or other operating systems where ZFS is supported. See the FreeBSD-specific guide here.
What is ARC on Linux for ZFS?
ZFS is an advanced file system initially created by Sun Microsystems. ARC is an acronym for Adaptive Replacement Cache. It is a modern algorithm for caching data in DRAM. In other words, ARC is nothing, but it contains cached data such as filesystem data and metadata. ZFS will try to use as much as a free ram to speed up server operations.
There is also a secondary cache called L2ARC (level II Adaptive Replacement Cach). Why use L2ARC? You know as DRAM is expensive and limited on all systems. So what we do is we use faster SSDs or PCIe NVMe storage for that purpose.
Here is how it typically look on an enterprise server:
- DRAM – ARC – 32Gbyte
- L2ARC – NVMe/SATA SSD – 512G
- ZFS Storage – Multiple mirrored disks (say 16TB)
How to tune ARC on Ubuntu/Debian or any Linux distros
There is no easy formula for everyone to get the correct ARC size. First, you need to find out your Linux server role and then set up ARC and L2ARC. That is your job as a Linux system administrator or developer. For file servers such as CIFS/NFS, we can set up a large ARC with L2ARC to speed up the operation. For MySQL/MariaDB/PostgreSQL, I set up a small ARC and tune database caching along with Redis or Memcached. In this example, my Ubuntu server runs VMs and Docker with just 16Gb RAM, and there is no space for L2ARC yet. Enough chit-chat; let’s get our hands dirty.
How to set up ZFS arc size on Ubuntu or Debian Linux
I am using an Ubuntu server, but instructions should work with any Linux distro as long as ZFS installed and configured. Let us set Max ARC size to 2GB and Min size to 1GB in bytes. My main goal for using ZFS is faster snapshots, cloning, and backups for LXD and other VMs. Hence, I am setting limits as follows, but your mileage may vary.
Limits must be set in bytes and not in GB/MB or any other unit
Step 1 – Create a new zfs.conf file
sudo vim /etc/modprobe.d/zfs.conf
Create a new file called zfs.conf as follows using a text editor such as vim command or nano command.
# Setting up ZFS ARC size on Ubuntu as per our needs # Set Max ARC size => 2GB == 2147483648 Bytes options zfs zfs_arc_max=2147483648 # Set Min ARC size => 1GB == 1073741824 options zfs zfs_arc_min=1073741824
Save and close the file by pressing Esc + :x! when using vim.
Step 2 – Updates an existing initramfs for Linux kernel
But there is one more step before you reboot the box. You need to generate an initramfs image. Here is how to do it on Debian or Ubuntu Linux:
sudo update-initramfs -u -k all
[sudo] password for vivek: update-initramfs: Generating /boot/initrd.img-5.4.0-80-generic update-initramfs: Generating /boot/initrd.img-5.4.0-77-generic update-initramfs: Generating /boot/initrd.img-5.4.0-25-generic
Step 3 – Reboot the Linux server
These settings only work after you reboot your Linux box. Hence, reboot the Linux server using the reboot command/shutdown command or systemctl command
## OR ##
sudo systemctl reboot
Step 4 – Verify that the correct ZFS ARC size set on Linux
All you have to do is type cat command as follows:
cat /sys/module/zfs/parameters/zfs_arc_min cat /sys/module/zfs/parameters/zfs_arc_max
Step 5 – Finding the arc stats on Linux
Here is what I see:
time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c 09:30:15 0 0 0 0 0 0 0 0 0 3.9G 7.8G
To get detailed ZFS subsystem report, run the following arc_summary command along with more command:
arc_summary | more
arc_summary -d | more
You learned about controlling ZFS ARC size under Ubuntu, Debian and other Linux distros as your needs. I strongly suggest that you read ZFS books that explain all of this stuff in detail.