During my most recent server-build, I had to tackle exposing host-managed files to VMs.
Previously, I used NFS. However, that was becoming increasingly troublesome, unfortunately due to IPv6. While I have an IPv6 connection and IPv6 network, my router will stop advertising the IPv6 network if the Internet goes down, causing connectivity issues. Additionally, the uncertainty of whether any connection will come as fe80:: vs my regular routable address was a pain (especially when my IPv6 network changed). Ideally, everything would use fe80:: internally, but it didn’t, and that was likely due to me mucking about with things.
I eventually worked around the quirky issues by forcing IPv4 on the NFS clients.
Fast forward to today, and I decided to investigate using filesystem passthrough using virtio-9p. This is the “built-in” filesystem passthrough in libvirt+qemu.
On the surface, it is actually very simple. You select a path, you give it a name, and you mount it in the client.
In practice, a lot more thought and configuration is required.
There are several options for “Driver”:
Path is the option you want to use, here. I didn’t find a very good explanation of “Handle”.
Mode is how UIDs and permissions are handled between guest and host. To quote the Qemu 9p documentation wiki:
Files on the filesystem are directly created with client-user’s credentials.
This would be the simplest option if the VMs were run as root. However, Fedora runs the VMs as the
qemuuser (additionally, you can run VMs as your user). Effectively, this means that while the guest may try to set ownership permissions on a file, the host may block it based on the
qemu's user account.
I tried to use this option, going so far as making my shared directory permissions an inadvisably-open 777.
Reading files works as expected, keeping their original UIDs and permissions, as long as you keep in mind that both the guest and host are applying permissions rules. Even root on the guest can’t read a file
qemuon the host can’t access.
Writing doesn’t really work, as the Qemu host process is running as a limited user account. So if you’re user
chrisin the VM, you can write a file to the share, but be unable to read it. The file was written to the host disk by the qemu-owned VM process, failed to set ownership to chris, leaving you with a file you created, but can not read. You’ll also get errors setting ownership/permissions.
Every file created will be owned by
qemu(which may just be a non-descript UID on your guest). This is less than desirable.
This would be an option if you were running a VM as your user, and running apps in the VM as the same user.
It is equivalent to passthrough security model; the only exception is, failure of privileged operation like chown are ignored. This makes a passthrough like security model usable for people who run KVM as non root.
(Note, the wiki is a little unclear and lists this as “none”. Pretty sure this is correct)
As described, this is effectively the same as Passthrough, but it pretends ownership and permissions changes worked (even though they didn’t).
This would be a slightly better option if you were running a VM as your user, and running apps in the VM as the same user.
Files are created with
qemuuser credentials and the client-user’s credentials are saved in extended attributes.
This completely separates host and client permissions.
On the host, you would
chown -R qemu:qemuall the files you intend to share with the client. (the
qemu-owned process still needs permissions for these files)
On the guest, you can chown/chmod as much as you want. Any metadata changes made by the guest are stored in extended attributes. These are global (they’re not per-vm extended attributes), so if you share the same path with two guests, they’ll see the same UIDs.
You should confirm you’re backing up extended attributes.
Mapped. It allows permissions on the host to be handled simply (just make
qemu own everything), while permissions on the guest can also be set independently (for whatever user/service is using those files).
This is the path on the host you want to export.
This is actually a label, not a path. Use it to identify this share to the guest.
Export filesystem as readonly mount
This is handy if you want to ensure the guest can not write files back to the host. Particularly handy if you share the same directory to multiple machines, but want to ensure only one can modify the contents.
Here’s a sample configuration that exports the path
/foo/bar with the label
foobar, using the mapped security model:
And this is how the guest would mount the filesystem:
foobar /mnt/foobar 9p trans=virtio,version=9p2000.L,_netdev 0 0
Note: You may encounter problems here, and may need to skip ahead to the SELinux rules
On the guest, you can change the permissions and ownerships:
$ chown root:root /mnt/foobar $ chmod 750 /mnt/foobar $ ls -ld /mnt/foobar drwxr-x---. 1 root root 96 Mar 3 23:31 /mnt/foobar
Here’s what that looks like on the host. Note the ownership and permissions haven’t changed:
$ ls -ld /foo/bar drwxr-xr-x. 1 qemu qemu 96 Mar 3 23:31 /foo/bar
The guest values are stored as extended attributes:
$ attr -l /foo/bar Attribute "virtfs.mode" has a 4 byte value for /foo/bar Attribute "virtfs.uid" has a 4 byte value for /foo/bar Attribute "selinux" has a 35 byte value for /foo/bar Attribute "virtfs.gid" has a 4 byte value for /foo/bar
Note: I haven’t looked for an easy way to read these values. They’re binary, and dumping them doesn’t convert them to a printable character
Times are updated on the files as normal (mtime, etc).
On the guest, these filesystems actually get tagged as nfs_t, so all the standard SELinux bools to allow httpd/samba/etc to work with NFS work as expected.
$ ls -ldZ /mnt/foobar drwxr-x---. root root system_u:object_r:nfs_t:s0 /mnt/foobar
On the host, SELinux inserts itself into the permissions as well to restrict qemu to only properly tagged files.
I used svirt_image_t (the same as the libvirt images directory). There may be a better tag, but I couldn’t immediately find one.
$ semanage fcontext -a -t svirt_image_t '/foo/bar(/.*)?' $ restorecon -Rv /foo/bar $ ls -ldZ /foo/bar drwxr-xr-x. 1 qemu qemu system_u:object_r:svirt_image_t:s0 96 Mar 9 20:56 /foo/bar
RHEL and CentOS
Oh yeah, RHEL (and therefore CentOS) don’t enable the 9p filesystem driver in their kernel. You’re SOL if you want to stay on the stock kernel.
However, there are two options to add 9p:
Kernel-plus. This is the stock kernel, but “plus” a bunch of features RHEL disabled. It is available in the centosplus repository, which is available (but disabled by default) on CentOS.
kernel-ml may also include support. I haven’t actually tested 9p on this, but btrfs did work (which it doesn’t on stock).