SLES 12 SP1 QEMU/KVM RELATED SUPPORT STATEMENTS Overview -------- The QEMU packages included with SLES 12 SP1 provide a large variety of features, from the very latest customer requests to features of questionable quality or value. This document was created to assist the user in deciding which features can be relied upon to build enterprise class virtualization solutions. KVM virtualization for the System z host (s390x) is provided as a technology preview while on x86 support is offered at the L3 (full support) level. KVM is implemented as linux kernel modules, which enables the linux kernel to function as an integral part of the KVM hypervisor. The hypervisor-guest interaction is controlled by QEMU through the /dev/kvm ioctl interface. The linux host assists in the virtualization of storage, networking and display resources as well as direct hardware passthrough of PCI and USB devices. Linux memory and cpu management features are used by both KVM and QEMU to enable guests to share host resources as efficiently as possible. QEMU is a primary component of KVM based virtualization. The legacy qemu-kvm program is provided for continuity with pre SLES 12 usage, including in libvirt xml references. The QEMU emulator binaries qemu-system-x86_64 and qemu-system-i386 (x86 host) and qemu-system-s390x (System z host) are now the primary programs to use to enable KVM virtualization. When using these programs, the -machine option accel=kvm (or its alias -enable-kvm) must be specified for KVM accelation to be active. Libvirt is the preferred means of accessing QEMU/KVM functionality. This document focuses on the features and direct usage of QEMU/KVM. Refer to other documentation for libvirt and libvirt based tools usage. Major QEMU/KVM Supported Features --------------------------------- - KVM virtualization is accomplished by using the QEMU program in KVM acceleration mode. KVM acceleration requires that both guest and host have the same fundamental architecture. - Guest images created under previous QEMU versions are supported through machine types which match up with these previous versions. - For ease of use, the QEMU program has defaults which represent traditional usage patterns. - Guest virtual machine characteristics are specified by a combination of internal defaults, given on the QEMU program command-line, and optional config files. Run-time control is done through either through the Human Monitor "Protocol" (HMP), or the JSON based programmatical QEMU Monitor Protocol (QMP) interface. Since a KVM guest runs in the context of a normal linux process, some types of execution controls are managed with normal linux tools. - Various standard vCPU types are available, along with the ability to specify specific CPU features visible to the guest. - QEMU incorporates a SEABIOS based system BIOS and iPXE based PXE ROMs which allow boot options common to PC's along with other features tailored to virtualization. Various VGABIOS ROMs (also from the SEABIOS project) are also included. - Some QEMU messages have been localized to various languages. This is provided by the qemu-lang package. Keyboard mappings for various nationalities is also provided. - Virtual machine lifecycle controls include startup through the system bios or using kernel boot, ACPI or legacy based shutdown, execution pausing, and the saving and restoring of machine state. Two varieties of "monitors" are provided for controlling runtime aspects of the VM: the human monitor, and a JSON based QMP (See /usr/share/doc/packages/kvm/qmp-commands.txt). - Guest execution state may be "moved" in both time (save/restore) and space (static and live migration). Guest migration from SLES 12 GA to SLES 12 SP1 hosts is supported. - Security considerations include secccomp2 based sandboxing, vTPM, privileged helpers and security models for executing QEMU as non-root users. - QEMU is able to run a wide range of common operating systems as guests. See the online SUSE documentation for supported OS's. Windows guests can optionally be accelerated with QEMU/KVM provided Hyper-V hypercalls, or with paravirtual drivers from the SUSE Virtual Machine Driver Pack. - QEMU provides best effort reuse of existing disk images, including those with systems installed, through geometry probing. Also disk images produced by other popular virtualization technologies may be imported into QEMU supported storage formats. These QEMU formats include features which exploit the benefits of virtualization. - Memory, cpu and disk space overcommit are possible and can be beneficial when done responsibly. Additional management of these resources comes in the form of memory ballooning, Host KSM, vcpu hot-add, online disk resizing, trim, discard and hole punching. - Guest performance is enhanced through the use of virtio devices, various disk caching modes, network acceleration via the vhost-net kernel module including zero copy optimizations, multiqueue network transmit capabilities, host transparent huge pages (THP) and direct hugetlb usage. Physical PCI and USB devices may also be passed through to the guest, including SR-IOV VF's. - The guest's UI is accessable via GTK, SDL, VNC, Spice, and serial (including curses TUI) interfaces. - Guest timekeeping is supported in a variety of ways, including a paravirtual clocksource, and options for the various guest clocks for how to handle the timeslicing of the guest's execution on the host. - Guest OS's interact with virtualized hardware including a choice of either older or more recent x86 system chipsets, system devices and buses, and a variety of common storage and networking emulated devices. SMBIOS and ACPI table details can be customized. Virtio based para-virtual devices are available for more efficient I/O. - In addition to the para-virtualized devices already mentioned, other devices and infrastructure designed to avoid virtualization "problem areas" are available such as SPICE graphics, vmmouse emulation and tablet style pointer interfaces. - A built-in user-mode network (SLIRP) stack is available. - Portions of the host file system may be shared with a guest by using virtFS. - A guest "agent" is available for SLES 12 SP1 KVM guests (see qemu-guest-agent package). This allows some introspection and control of the guest OS environment from the host. - The following features were previously in a technology preview status, but are now fully supported: - virtio-blk-dataplane has now been fully integrated with the block subsystem. - RADOS Block Devices (RBD) QEMU/KVM Technology Previews ---------------------------- - KVM on System z using the qemu-system-s390x system emulator is not yet fully supported. Other support statements made throughout this document generally refer to x86 usage only. - Specifying and placing PCI devices on a PCI bridge allows for a greater number of devices. - Nested VMX and SVM virtualization is possible. - virtio-blk-data-plane allows higher block I/O rates Noteworthy QEMU/KVM Unsupported Features ---------------------------------------- - Note that some features are unsupported simply due to lack of validation. If an existing feature is desired, but not marked supported, let SUSE know about your requirements. - The TCG "acceleration" mode may be helpful for problem isolation, but otherwise presents insufficient benefit and stability. - Use of -cpu host is not supported in all host/guest configurations. - ISCSI integration is not enabled in QEMU. It is however possible for guests to access iSCSI targets available to the host via the blockio interfaces. - GlusterFS integration is not enabled. Deprecated, Superseded, Modified and Dropped Features ------------------------------------------- - When no video adapter is specified, the default used is stdvga. This differs from the default of prior releases which was cirrus. The cirrus adapter was considered too outdated to continue to use as the default. - The deprecated windows drivers (win-virtio-drivers.iso) are no longer provided. The Virtual Machine Driver Pack is the supported way to get virtio drivers for Windows guests. - The use of ",boot=on" for virtio disks is no longer needed since the bios used supports the virtio block interface directly. In fact, its usage may cause problems, and is now considered deprecated. - The use of "?" as a parameter to "-cpu", "-soundhw", "-device", "-M", "-machine", "-d", and "-clock" is now considered deprecated. Use "help" instead. - Passthrough of a host floppy device is now considered deprecated. - Use of aio=native without direct cache mode also being specified is now considered deprecated. (It has no effect in this case). - The use of image encryption in qcow and qcow2 formats is now considered deprecated. Analysis has shown it to be weak encryption, in addition to suffering from poor design. Images can easily be converted to a non-encrypted format. - Use of acpi, boot-opts, and smp-opts in a -readconfig config file are now considered deprecated. In the future those names will be standardized to acpitable, boot, and smp respectively. - When only a partial SMP topology is provided (ie only some of sockets, cores, threads) the derived guest ABI is not guaranteed to be equivalent in future versions. - These previously supported command line options are no longer recognized: -device pc-sysfw (no longer needed) - The previously supported blkdev-add QMP command has been flagged as lacking and could possibly change syntax in the future. - These previously unsupported command line options are now deprecated: -no-kvm-irqchip (use -machine kernel_irqchip=off instead) -no-kvm-pit -no-kvm-pit-reinjection -tdf - The case of specified sockets, cores, and threads not matching the vcpu count is no longer silently ignored. QEMU now requires that the topology to match up with the vcpu count. - These previously unsupported command line options are no longer recognized: -device cfi.pflash01 -device esp -device exynos4210-ehci-usb -device fusbh200-ehci-usb -device icc-bridge -device q35-pcihost -device mch -device smbus-eeprom -device SUNW,fdtwo -device sysbus-ahci -device sysbus-fdc -device sysbus-ohci -device tegra2-ehci-usb -device testdev (use -device pc-testdev instead) -device virtio-mmio -device xln,ps7-usb -enable-nesting -kvm-shadow-memory (use -machine kvm_shadow_mem= instead) -M mac -nvram -old-param -osk -pcidevice (use -device pci-assign instead) -qtest -semihosting - These previously supported monitor commands are no longer recognized: pci_add (use device_add instead) pci_del (use device_del instead) - These previously unsupported monitor commands are no longer recognized: cpu_set QEMU Command-Line and Monitor Syntax and Support ------------------------------------------------ - The QEMU program command-line syntax is as follows: qemu-system-i386 [options] [disk_image] qemu-system-s390x [options] qemu-system-x86_64 [options] [disk_image] Where 'options' are taken from the options listed below, and 'disk_image' is the file system reference to the the x86 guest's primary IDE based hard disk image. This image as well as those used with -drive or -cdrom, may be in the raw (no format), qcow2 or qed storage formats, and may be located in files within the host filesystem, logical volumes, host physical disks, or network based storage. Read only media may also be accessed via URL style protocol specifiers. Note that as a general rule, as new command line options are added which serve to replace an older option or interface, you are strongly encouraged to adapt your usage to the new option. The new option is being introduced to provide better functionality and usability going forward. In some cases existing problems or even bugs in older interfaces cannot be fixed due to functional expectations, but are resolved in the newer interface or option. This advice includes moving to the most recent machine type (eg pc-i440fx-2.3 instead of pc-i440fx-2.0) if possible. - The following command line options are supported: -alt-grab -append ... -audio-help -balloon ... -boot ... -cdrom ... -chardev .. -clock -cpu ... (all except host) -ctrl-grab -d ... -daemonize -debugcon ... -device [isa-serial|isa-parallel|isa-fdc|ide-drive|ide-hd|ide-cd| kvm-pci-assign|VGA|cirrus-vga|rtl8139|virtio-net-pci|virtio-blk-pci| virtio-balloon-pci|virtio-9p-pci|usb-hub|usb-ehci|usb-tablet| usb-storage|usb-mouse|usb-kbd|virtserialport|virtconsole| virtio-serial-pci|sga|i82559er|e1000|virtio-scsi-pci|scsi-cd|scsi-hd| scsi-generic|scsi-disk|scsi-block|pci-serial|pci-serial-2x| pci-serial-4x|ich9-ahci|piix-usb-uhci|usb-host|usb-serial| usb-wacom-tablet|usb_braille|usb-net|pci-ohci|piix4-usb-uhci| virtio-rng-pci|i6300esb|ib700|qxl|qxl-vga|pvpanic|vfio-pci|ivshmem| pci-bridge|megasas-gen2|pc-dimm] (the following are aliases of these supported devices: ahci| virtio-blk|virtio-net|virtio-serial|pci-assign|virtio-balloon) -display ... -drive ... (if specified if=[ide|floppy|virtio] and format=[raw|qcow2|qed] and snapshot=off only) -echr ... -enable-fips -enable-kvm -fda/-fdb ... -fsdev ... -full-screen -gdb ... -global ... -h -hda/-hdb/-hdc/-hdd ... -help -incoming ... -initrd ... -k ... -kernel ... -loadvm ... -m ... -machine [help|?|none|pc|pc-0.12|pc-0.14|pc-0.15|pc-i440fx-1.4|pc-i440fx-1.7| pc-q35-1.7|pc-i440fx-2.0|pc-q35-2.0|pc-i440fx-2.3|pc-q35-2.3] -mem-path ... -mem-prealloc -M [help|?|none|pc|pc-0.12|pc-0.14|pc-0.15|pc-i440fx-1.4|pc-i440fx-1.7| pc-q35-1.7|pc-i440fx-2.0|pc-q35-2.0|pc-i440fx-2.3|pc-q35-2.3] -mon ... -monitor ... -msg ... -name ... -netdev [user|tap|bridge] ... -net [nic|user|tap|bridge|none] ... (for model= only rtl8139, e1000 and virtio are supported) -no-acpi -nodefaults -nodefconfig -no-frame -nographic -no-hpet -no-quit -no-reboot -no-shutdown -no-user-config -object ... -parallel ... -pidfile ... -qmp ... -qmp-pretty ... -readconfig ... -realtime ... -rtc ... -runas ... -s -S -sandbox ... -sdl -serial ... -smbios ... -smp ... -spice -tdf -tpmdev passthrough ... -usb -usbdevice [disk|host|serial|braille|net|tablet|mouse] -uuid .. -version -vga [std|cirrus|qxl|none] -virtfs ... -vnc ... -watchdog ... -watchdog-action ... -writeconfig ... - The following monitor commands are supported: ? balloon target ... block_resize ... boot_set ... [c|cont] change device ... chardev-add ... chardev-remove ... cpu ... cpu-add ... delvm ... device_add ... device_del ... drive_add ... drive_backup ... drive_del ... dump_guest_memory ... eject ... gdbserver ... help info ... loadvm ... logfile ... logitem ... mce ... memsave ... migrate ... migrate_cancel migrate_incoming migrate_set_cache_size ... migrate_set_capability ... migrate_set_downtime ... migrate_set_speed ... mouse_button ... mouse_move ... mouse_set ... nmi ... object_add ... object_del ... pci_add ... pci_del... pmemsave ... [p|print] ... q qemu-io ... qom-list qom-set ringbuf_read ... ringbuf_write ... savevm ... sendkey ... snapshot_blkdev_internal ... snapshot_delete_blkdev_internal ... stop system_powerdown system_reset system_wakeup usb_add ... usb_del ... watchdog_action ... x ... xp ... - The following command line options are unsupported: -acpitable ... -add-fd ... -bios ... -bt ... -chroot ... -cpu host -curses -device [ipoctal232|i82562|ccid-card-passthru|nec-usb-xhci|hda-duplex| hda-output|usb-bot|lsi53c810a|ich9-usb-uhci2|ich9-usb-uhci6| ich9-usb-uhci5|ich9-usb-uhci3|isa-debug-exit|ne2k_pci|usb-uas| ich9-usb-uhci4|ioh3420|isa-ide|usb-ccid|ich9-usb-ehci2|pcnet| ich9-intel-hda|dc390|ich9-usb-ehci1|hda-micro|x3130-upstream| isa-cirrus-vga|ich9-usb-uhci1|pc-testdev|ne2k_isa|isa-vga|cs4231a|gus| vmware-svga|i82801b11-bridge|i82557a|i82557c|i82557b|i82801|AC97| am53c974|intel-hda|i82558a|i82558b|usb-audio|i82550|isa-debugcon|sb16| megasas|i82551|xio3130-downstream|vt82c686b-usb-uhci|tpci200|i82559a| i82559b|i82559c|isa-applesmc|usb-bt-dongle|adlib|ES1370|lsi53c810| nvme|pci-testdev|pvscsi|vhost-scsi|vhost-scsi-pci|virtio-9p-device| virtio-balloon-device|virtio-blk-device|virtio-net-device| virtio-rng-device|virtio-scsi-device|virtio-serial-device|vmxnet3| xen-pci-passthrough|xen-platform|xen-pvdevice|piix3-ide|piix3-ide-xen| piix3-ide|i8042|sdhci-pci|generic-sdhci|secondary-vga|edu|fw_cfg_io| fw_cfg_mem|intel_iommu|usb-mtp|e1000-82540em|e1000-82544gc| e1000-82545em] (the following are aliases of these supported devices: lsi) (note that some of these device names represent supported devices and are used internally, but not specifyable via -device) -drive ,if=[scsi|mtd|pflash], snapshot=on, format=[anything besides raw, qcow2, or qed] -dtb -g ... -icount ... -iscsi ... -L ... -machine [pc-q35-2.2|pc-q35-2.1|pc-q35-1.6|pc-q35-1.5|pc-q35-1.4|pc-i440fx-2.2| pc-i440fx-2.1|pc-i440fx-1.6|pc-i440fx-1.5|pc-1.3|pc-1.2|pc-1.1|pc-1.0| pc-0.13|pc-0.11|pc-0.10|isapc|xenpv|xenfv] -M [pc-q35-2.2|pc-q35-2.1|pc-q35-1.6|pc-q35-1.5|pc-q35-1.4|pc-i440fx-2.2| pc-i440fx-2.1|pc-i440fx-1.6|pc-i440fx-1.5|pc-1.3|pc-1.2|pc-1.1|pc-1.0| pc-0.13|pc-0.11|pc-0.10|isapc|xenpv|xenfv] -mtdblock ... -netdev [socket|dump|vde|hubport] ... -net [socket|dump|vde] ... -no-fd-bootchk -no-kvm -no-kvm-irqchip -no-kvm-pit -no-kvm-pit-reinjection -numa ... -option-rom ... -pflash ... -portrait -prom-env ... -qtest ... -qtest-log ... -rotate -sd ... -set ... -show-cursor -singlestep -snapshot -soundhw ... -tb-size ... -trace ... -vga [vmware|xenfb] -virtioconsole ... -win2k-hack -xen-attach ... -xen-create -xen-domid ... - The following monitor commands are unsupported: acl_add ... acl_policy ... acl_remove ... acl_reset ... acl_show ... block_job_cancel ... block_job_complete ... block_job_pause ... block_job_resume ... block_job_set_speed ... block_passwd ... client_migrate_info ... close_fd ... commit ... drive_mirror ... expire_password ... hostfwd_add ... hostfwd_remove ... host_net_add ... host_net_remove ... i ... nbd_server_add ... nbd server_start ... nbd_server_stop ... netdev_add netdev_del ... o ... pcie_aer_inject_error ... screendump ... set_link ... set_password ... singlestep ... snapshot_blkdev ... stopcapture ... sum ... trace_event ... wavcapture ... - The following QMP commands are supported: add_client add-fd balloon block-commit blockdev-snapshot-delete-internal-sync blockdev-snapshot-internal-sync blockdev-snapshot-sync block_passwd block_resize block_set_io_throttle block-set-write-threshold block_stream change change-vnc-password chardev-add chardev-remove client_migrate_info closefd cont cpu cpu-add device_add device_del device-list-properties dump-guest-memory eject expire_password getfd human-monitor-command inject-nmi memsave migrate migrate_cancel migrate-incoming migrate-set-cache-size migrate-set-capabilities migrate_set_downtime migrate_set_speed object-add object-del pmemsave qmp_capabilities qom-get qom-list qom-list-types qom-set query-acpi-ospm-status query-balloon query-block query-block-jobs query-blockstats query-chardev query-chardev-backends query-command-line-options query-commands query-cpu-definitions query-cpus query-dump-guest-memory-capability query-events query-fdsets query-iothreads query-kvm query-machines query-memdev query-memory-devices query-mice query-migrate query-migrate-cache-size query-migrate-capabilities query-name query-named-block-nodes query-pci query-rx-filter query-spice query-status query-target query-tpm query-tpm-models query-tpm-types query-uuid query-version query-vnc quit remove-fd ringbuf-read ringbuf-write rtc-reset-reinjection screendump send-key set_link set_password stop system_powerdown system_reset system_wakeup trace-event-get-state trace-event-set-state transaction xen-save-devices-state xen-set-global-dirty-log - The following QMP commands are unsupported: blockdev-add blockdev-backup block-job-cancel block-job-complete block-job-pause block-job-resume block-job-set-speed change-backing-file drive-backup drive-mirror netdev_add netdev_del nbd-server-add nbd-server-start nbd-server-stop x-input-send-event