Samba and NFS performance - ZFS on Linux vs ZFS on FreeBSD

Posted on 2020-02-01.
I am currently in the process of migrating some ZFS storage servers from GNU/Linux Debian to FreeBSD and decided to do some simple performance tests.

The tests are simple, I am downloading and uploading a ton of small files and some very big files (each about 10-60 GB in size) to and from ZFS storage servers running Samba and NFS on GNU/Linux Debian Buster and FreeBSD 12.1. I am using a GNU/Linux client for both SMB and NFS and a Windows 10 client for SMB only.

The Debian kernel is 4.19 from backports and Samba 4.9.

The FreeBSD version is 12.1 with Samba 4.10.

In both setups I am running with the out-of-the-box settings as testing has shown that performance tunning in the current setup has provided no improvements for either Samba or NFS. In both cases they utilize my gigabit network nicely.

The ZFS storage is RAIDZ on 4 x 4TB drives and 4 x 6TB drives on some i5 machines with 8 GB of memory (no ZFS is not a memory hog, that's a myth). Note that RAIDZ is slower to download from than a MIRROR.

I have used the pv tool to measure the average speed.

Samba tests

OS with ZFS Download CPU Upload CPU
Linux 98.2 MiB/s 15.9 % (smbd) 117 MiB/s 85.9 % (smbd)
FreeBSD 94.4 MiB/s 35.1 % (smbd) 120 MiB/s 22.5 % (smbd)

NFS tests

OS with ZFS Download CPU Upload CPU
Linux 93.4 MiB/s 22.7 % (nfsd) 114 MiB/s 141.2 % (nfsd)
FreeBSD 97.2 MiB/s 15.5 % (nfsd) 116 MiB/s 10.5 % (nfsd)