Using ZVOLs for bhyve VMs

One of the followup changes after converting my desktop from UFS to ZFS was to convert my bhyve VMs from raw "disk.img" files in my home directory to being backed by ZFS volumes (ZVOLs).   The conversion process was fairly simple.

First, I created a new namespace to hold them:

# zfs create zroot/bhyve

Next, for each disk image I created a new volume and dd'd the raw disk image over to it.  For example:

# zfs create -V 16G zroot/bhyve/head
# dd if=bhyve/head/disk.img of=/dev/zvol/zroot/bhyve/head bs=1m

I then booted the virtual machine passing "-d /dev/zvol/zroot/bhyve/head" to vmrun.sh.  Once that was successful I removed the old disk.img files.

By default, FreeBSD exports ZFS volumes as GEOM providers.  This means that volumes are able to be tasted on the host.  For example:

# gpart show zvol/zroot/bhyve/head
=>      34  33554365  zvol/zroot/bhyve/head  GPT  (16G)
        34       128                      1  freebsd-boot  (64K)
       162  32714523                      2  freebsd-ufs  (16G)
  32714685    839714                      3  freebsd-swap  (410M)

Once nice thing about this is that I can run fsck(8) directly against /dev/zvol/zroot/bhyve/headp2 without having to use mdconfig(8).  I could also choose to mount it on the host.

Secondly, ZFS volumes show up in gstat(8) output.  With all the volumes the output can be quite long, but "gstat -p" can give you nice output breaking down I/O by VM:

dT: 1.001s  w: 1.000s

 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name

    0      0      0      0    0.0      0      0    0.0    0.0| cd0

    6    408    271   9934   42.9    138  17534    7.7  100.6| ada0

   12    443    272   5987   43.9    172  21844    5.7  100.0| ada1

    0      0      0      0    0.0      0      0    0.0    0.0| zvol/zroot/bhyve/head

    0      0      0      0    0.0      0      0    0.0    0.0| zvol/zroot/bhyve/head-i386

    0     18      2     64  393.8     16    321  119.4   90.7| zvol/zroot/bhyve/fbsd11-i386

    1     65     24   3068   53.0     41   3320   18.7  109.9| zvol/zroot/bhyve/fbsd11

    6     59     22    786   50.8     37    578   61.2  100.1| zvol/zroot/bhyve/fbsd10

    2     61      0      0    0.0     61   6609   77.0  101.8| zvol/zroot/bhyve/fbsd9

    1     18      8     94   60.4     10     33   10.6   51.3| zvol/zroot/bhyve/fbsd10-i386

    1     14      6     50   92.3      8    108  109.3   60.7| zvol/zroot/bhyve/fbsd9-i386

    0      0      0      0    0.0      0      0    0.0    0.0| zvol/zroot/bhyve/fbsd8-i386

    0      0      0      0    0.0      0      0    0.0    0.0| zvol/zroot/bhyve/fbsd8