README for the Xen packages =========================== This file contains SUSE-specific instructions and suggestions for using Xen. For more in-depth documentation of using Xen on SUSE, consult the virtualization chapter in the SLES or SUSE Linux manual, or read up-to-date virtualization information, at https://www.suse.com/documentation/sles11/singlehtml/book_xen/book_xen.html For more complete documentation on Xen itself, please install the xen-doc-html package and read the documentation installed into /usr/share/doc/packages/xen/. About ----- Xen allows you to run multiple virtual machines on a single physical machine. See the Xen homepage for more information: http://www.xenproject.org/ If you want to use Xen, you need to install the Xen hypervisor and a number of supporting packages. During the initial SUSE installation (or when installing from YaST) check-mark the "Xen Virtual Machine Host Server" pattern. If, instead, you wish to install Xen manually later, click on the "Install Hypervisor and Tools" icon in YaST. If you want to install and manage VMs graphically, be sure to install a graphical desktop environment like KDE or GNOME. The following optional packages are needed to manage VMs graphically. Note that "Install Hypervisor and Tools" installs all the packages below: virt-install (Optional, to install VMs) virt-manager (Optional, to manage VMs graphically) virt-viewer (Optional, to view VMs outside virt-manager) vm-install (Optional, to install VMs with xl only) You then need to reboot your machine. Instead of booting a normal Linux kernel, you will boot the Xen hypervisor and a slightly changed Linux kernel. This Linux kernel runs in the first virtual machine and will drive most of your hardware. This approach is called paravirtualization, since it is a partial virtualization (the Linux kernel needs to be changed slightly, to make the virtualization easier). It results in very good performance (consult http://www.cl.cam.ac.uk/research/srg/netos/xen/performance.html) but has the downside of unchanged operating systems not being supported. However, new hardware features (e.g., Intel VT and AMD-V) are overcoming this limitation. Terminology ----------- The Xen open-source community has a number of terms that you should be familiar with. A "domain" is Xen's term for a virtual machine. "Domain 0" is the first virtual machine. It can control all other virtual machines. It also (usually) controls the physical hardware. A kernel used in domain 0 may sometimes be referred to as a dom0 kernel. "Domain U" is any virtual machine other than domain 0. The "U" indicates it is unprivileged (that is, it cannot control other domains). A kernel used in an unprivileged domain may be referred to as a domU kernel. SUSE documentation will use the more industry-standard term "virtual machine", or "VM", rather than "domain" where possible. And to that end, domain 0 will be called the "virtual machine server", since it essentially the server on which the other VMs run. All other domains are simply "virtual machines". The acronym "HVM" refers to a hardware-assisted virtual machine. These are VMs that have not been modified (e.g., Windows) and therefore need hardware support such as Intel VT or AMD-V to run on Xen. Kernels ------- Xen supports two kinds of kernels: A privileged kernel (which boots the machine, controls other VMs, and usually controls all your physical hardware) and unprivileged kernels (which can't control other VMs, and usually don't need drivers for physical hardware). The privileged kernel boots first (as the VM server); an unprivileged kernel is used in all subsequent VMs. The VM server takes control of the boot process after Xen has initialized the CPU and the memory. This VM contains a privileged kernel and all the hardware drivers. For the other virtual machines, you usually don't need the hardware drivers. (It is possible to hide a PCI device from the VM server and re-assign it to another VM for direct access, but that is a more advanced topic.) Instead you use virtual network and block device drivers in the unprivileged VMs to access the physical network and block drivers in the VM server. SUSE ships a single PVOPS enabled Linux kernel that may be used for booting natively and also with the Xen hypervisor. Booting ------- If you installed Xen during the initial SUSE installation, or installed xen later, a "XEN" option should exist in your Grub bootloader. Select that to boot SUSE on top of Xen. If you want to add additional entries, or modify the existing ones, you may run the YaST2 Boot Loader program. Once you have booted this configuration successfully, you are running Xen with a privileged kernel on top of it. Xen Boot Parameters ------------------- Normally, xen.gz requires no parameters. However, in special cases (such as debugging or a dedicated VM server) you may wish to pass it parameters. Adding parameters to xen.gz can be done by editing the /etc/default/grub file. Add the following line to this file; GRUB_CMDLINE_XEN_DEFAULT="". The parameters may be valid options passed to xen.gz (the hypervisor). After editing this file, you must first run 'grub2-mkconfig -o /boot/grub2/grub.cfg' and then reboot for the changes to take effect. For more information on how to add options to the hypervisor, see the sections below called; "Dom0 Memory Ballooning" and "Troubleshooting". For a more complete discussion of possible parameters, see the user documentation in the xen-doc-html package. Creating a VM with virt-install ------------------------------- The virt-install program (part of the virt-install package, and accessible through YaST's Control Center) is the recommended method to create VMs. This program handles creating both the VM's libvirt XML definition and disk(s). It can help install any operating system, not just SUSE. virt-install has both a command line only mode and a graphical wizard mode that may be used to define and start VM installations. virt-install may be launched from the virt-manager VM management tool. Start virt-manager either from the YaST Control Center or from the command line. The installation icon from the main virt-manager screen may be selected to begin the virt-install installation wizard. The use of virt-install or virt-manager requires the installation of the libvirt packages and the libvirt daemon must be running on the host unless you are managing a remote host. Each VM needs to have its own root filesystem. The root filesystem can live on a block device (e.g., a hard disk partition, or an LVM2 or EVMS volume) or in a file that holds the filesystem image. VMs can share filesystems, such as /usr or /opt, that are mounted read-only from _all_ VMs. Never try to share a filesystem that is mounted read-write; filesystem corruption will result. For sharing writable data between VMs, use NFS or other networked or cluster filesystems. When defining the virtual network adapter(s), we recommend using a static MAC for the VM rather than allowing Xen to randomly select one each time the VM boots. (See "Network Troubleshooting" below.) The Xen Project has been allocated a range of MAC addresses with the OUI of 00-16-3E. By using MACs from this range you can be sure they will not conflict with any physical adapters. When the VM shuts down (because the installation -- or at least the first stage of it -- is done), the wizard finalizes the VM's configuration and restarts the VM. The creation of VMs can be automated; read the virt-install man page for more details. The installation of an OS within the VM can be automated if the OS supports it. Creating a VM with vm-install ----------------------------- The vm-install program is also provided to create VMs. Like virt-install, this optional program handles creating both the VM's libvirt XML definition and disk(s). It also creates a legacy configuration file for use with 'xl'. It can help install any operating system, not just SUSE. From the command line, run "vm-install". If the DISPLAY environment variable is set and the supporting packages (python-gtk) are installed, a graphical wizard will start. Otherwise, a text wizard will start. If vm-install is started with the '--use-xl' flag, it will not require libvirt nor attempt to communicate with libvirt when creating a VM and instead will only use the 'xl' toolstack to start VM installations. Once you have the VM configured, click "OK". The wizard will now create a configuration file for the VM, and create a disk image. The disk image will exist in /var/lib/xen/images, and a corresponding configuration file will exist in /etc/xen/vm. The operating system's installation program will then run within the VM. When the VM shuts down (because the installation -- or at least the first stage of it -- is done), the wizard finalizes the VM's configuration and restarts the VM. The creation of VMs can be automated; read the vm-install man page for more details. The installation of an OS within the VM can be automated if the OS supports it. Creating a VM Manually ---------------------- If you create a VM manually (as opposed to using virt-install, which is the recommended way), you will need to create a disk (or reuse an existing one) and a configuration file. If you are using a disk or disk image that is already installed with an operating system and you want the VM to run in paravirtual mode, you'll probably need to replace its kernel with a Xen-enabled kernel. The kernel and ramdisk used to bootstrap the VM must match any kernel modules that might be present in the VM's disk. It is possible to manually copy the kernel and ramdisk from the VM's disk (for example, after updating the kernel within that VM) to the VM server's filesystem. However, an easier (and less error-prone) method is to use /usr/lib/grub2/x86_64-xen/grub.xen as the VM kernel. When the new VM is started, it runs grub.xen to read the grub configuration from the VM disk, selecting the configured kernel and ramdisk so that it can be used to bootstrap the new VM. Next, make a copy of one of the /etc/xen/examples/* files, and modify it to suit your needs. You'll need to change (at very least) the "name" and "disk" parameters. See /etc/xen/examples/ for example configuration files. Managing Virtual Machines ------------------------- VMs can be managed from the command line using 'virsh' or from virt-manager. VMs created by virt-install or vm-install (without vm-install's --use-xl flag) will automatically be defined in libvirt. VMs defined in libvirt may be managed by virt-manager or from the command line using the 'virsh' command. However, if you copy a VM from another machine and manually create a VM XML configuration file, you will need to import it into libvirt with a command like: virsh define /my-vm.xml This imports the configuration into libvirt (and therefore virt-manager becomes aware of it, also). Now to start the VM: virsh start my-vm or start it from virt-manager's graphical menu. Have a look at running VMs with "virsh list". Attach to the VM's text console with "virsh console ". Attaching to multiple VM consoles is most conveniently done with the terminal multiplexer "screen". Have a look at the other virsh commands by typing "virsh help". Note that most virsh commands must be done as root. Changes in the Xen VM Management Toolstack ------------------------------------------ With SUSE Linux Enterprise Server 12, the way VMs are managed has changed when compared with older SLES versions. Users familiar with the 'xm' command and the xend management daemon will notice that these are absent. The xm/xend toolstack has been replaced with the xl toolstack. The xl toolstack is intended to remain backwards compatible with existing xm domain configuration files. Most 'xm' commands can simply be replaced with 'xl'. One significant difference is that xl does not support the concept of Managed Domains. The xl command can only modify running VMs. Once the VM is shutdown, there is no preserved state information other than what is saved in the configuration file used to start the VM. In order to provide Managed Domains, users are encouraged to use libvirt and it's tools to create and modify VMs. These tools include the command line tool 'virsh' and the graphical tools virt-manager and virt-install. Warning: Using xl commands to modify libvirt managed domains will result in errors when virsh or virt-manager is used. Please use only virsh or virt-manager to manage libvirt managed domains. If you are not using libvirt managed domains then using xl commands is the correct way to modify running domains. Using the Mouse via VNC in Fully Virtual Mode --------------------------------------------- In a fully virtualized VM, the mouse may be emulated as a PS/2 mouse, USB mouse, or USB tablet. The virt-install tool selects the best emulation that is known to be automatically detected and supported by the operating system. However, when accessing some fully virtualized operating systems via VNC, the mouse may be difficult to control if the VM is emulating a PS/2 mouse. PS/2 provides mouse deltas, but VNC only provides absolute coordinates. In such cases, you may want to manually switch the operating system and VM to use a USB tablet. Emulation of a SummaSketch graphics tablet is provided for this reason. To use the Summa emulation, you will need to configure your fully virtualized OS. Note that the virtual tablet is connected to the second virtual serial port (/dev/ttyS1 or COM2). Most Linux distributions ship with appropriate drivers, and only need to be configured. To configure gpm, edit /etc/sysconfig/mouse and add these lines: MOUSETYPE="summa" XMOUSETYPE="SUMMA" DEVICE=/dev/ttyS1 The format and location of your configuration file could vary depending upon your Linux distribution. The goal is to run the gpm daemon as follows: gpm -t summa -m /dev/ttyS1 X also needs to be configured to use the Summa emulation. Add the following stanza to /etc/X11/xorg.conf, or use your distribution's tools to add these settings: Section "InputDevice" Identifier "Mouse0" Driver "summa" Option "Device" "/dev/ttyS1" Option "InputFashion" "Tablet" Option "Mode" "Absolute" Option "Name" "EasyPen" Option "Compatible" "True" Option "Protocol" "Auto" Option "SendCoreEvents" "on" Option "Vendor" "GENIUS" EndSection After making these changes, restart gpm and X. HVM Console in Fully Virtual Mode --------------------------------- When running a VM in fully virtual mode, a special console is available that provides some additional ways to control the VM. Press Ctrl-Alt-2 to access the console; press Ctrl-Alt-1 to return to the VM. While at the console, type "help" for help. The two most important commands are "send-key" and "change". The "send-key" command allows you to send any key sequence to the VM, which might otherwise be intercepted by your local window manager. The "change" command allows the target of a block device to be changed; for example, use it to change from one CD ISO to another. Some versions of Xen have this command disabled for security reasons. Consult the online documentation for workarounds. Networking ---------- Your virtual machines become much more useful if you can reach them via the network. Starting with openSUSE11.1 and SLE11, networking in domain 0 is configured and managed via YaST. The yast2-networking module can be used to create and manage bridged networks. During initial installation, a bridged networking proposal will be presented if the "Xen Virtual Machine Host Server" pattern is selected. The proposal will also be presented if you install Xen after initial installation using the "Install Hypervisor and Tools" module in YaST. The default proposal creates a virtual bridge in domain 0 for each active ethernet device, enslaving the device to the bridge. Consider a machine containing two ethernet devices (eth0 and eth1), both with active carriers. YaST will create br0 and br1, enslaving the eth0 and eth1 devices repectively. VMs get a virtual network interface (e.g. eth0), which is visible in domain 0 as vifN.0 and connected to the bridge. This means that if you set up an IP address in the VMs belonging to the same subnet as br0 from your domain 0, you'll be able to communicate not only with the other slave VMs, but also with domain 0 and with the external network. If you have a DHCP server running in your network, your VMs should succeed in getting an IP address. Be aware that this may have unwanted security implications. You may want to opt for routing instead of bridging, so you can set up firewalling rules in domain 0. Please read about the network configuration in the Xen manual. You can set up bridging or routing for other interfaces also. For debugging, here's what happens on bootup of a domU: - xenstored saves the device setup in xenstore - domU is created - vifN.0 shows up in domain 0 and a hotplug event is triggered - hotplug is /sbin/udev; udev looks at /etc/udev/rules.d/40-xen.rules and calls /etc/xen/scripts/vif-bridge online - vif-bridge set the vifN.0 device up and enslaves it to the bridge - eth0 shows up in domU (hotplug event triggered) Similar things happen for block devices, except that /etc/xen/scripts/block is called. It's not recommended to use ifplugd nor NetworkManager for managing the interfaces if you use bridging mode. Use routing with nat or proxy-arp in that case. You also need to do that in case you want to send out packets on wireless; you can't bridge Xen "ethernet" packets into 802.11 packets. Network Troubleshooting ----------------------- First ensure the VM server is configured correctly and can access the network. Do not use ifplugd or NetworkManager, neither are bridge aware. Specify a static virtual MAC in the VM's configuration file. Random MACs can be problematic, since with each boot of the VM it appears that some hardware has been removed (the previous random MAC) and new hardware is present (the new random MAC). This can cause network configuration files (which were intended for the old MAC) to not be matched up with the new virtual hardware. In the VM's filesystem, ensure the ifcfg-eth* files are named appropriately. For example, if you do decide to use a randomly-selected MAC for the VM, the ifcfg-eth* file must not include the MAC in its name; name it generically ("ifcfg-eth0") instead. If you use a static virtual MAC for the VM, be sure that is reflected in the file's name. Thread-Local Storage -------------------- For some time now, the glibc thread library (NPTL) has used a shortcut to access thread-local variables at a negative segment offset from the segment selector GS instead of reading the linear address from the TDB (offset 0). Unfortunately, this optimization has been made the default by the glibc and gcc maintainers, as it saves one indirection. For Xen this is bad: The access to these variables will trap, and Xen will need to use some tricks to make the access work. It does work, but it's very slow. SUSE Linux 9.1 and SLES 9 were prior to this change, and thus are not affected. SUSE Linux 9.2 and 9.3 are affected. For SUSE Linux 10.x and SLES 10, we have disabled negative segment references in gcc and glibc, and so these are not affected. Other non-SUSE Linux distributions may be affected. For affected distributions, one way to work around the problem is to rename the /lib/tls directory, so the pre-i686 version gets used, where no such tricks are done. An example LSB-compliant init script which automates these steps is installed at /usr/share/doc/packages/xen/boot.xen. This script renames /lib/tls when running on Xen, and restores it when not running on Xen. Modify this script to work with your specific distribution. Mono has a similar problem, but this has been fixed in SUSE Linux 10.1 and SLES 10. Older or non-SUSE versions of Mono may have a performance impact. Security -------- Domain 0 has control over all domains. This means that care should be taken to keep domain 0 safe; ideally you strip it down to only do as little there as possible, preferably with no local users except for the system administrator. Most commands in domain 0 can only be performed as root, but this protection scheme only has moderate security and might be defeated. In case domain 0 is compromised, all other domains are compromised as well. To allow relocation of VMs (migration), the receiving machine listens on TCP port 8002. You might want to put firewall rules in place in domain 0 to restrict this to machines which you trust. Relocating VMs with sensitive data is not a good idea in untrusted networks, since the data is not sent encrypted. The memory protections for the domUs are effective; so far no way to break out of a virtual machine is known. A VM is an effective jail. Limitations ----------- When booting, Linux reserves data structures matching the amount of RAM found. This has the side-effect that you can't dynamically grow the memory beyond what the kernel has been booted with. But you can trick domU Linux to prepare for a larger amount of RAM by passing the mem= boot parameter. The export of virtual hard disks from files in Xen can be handled via the loopback driver (although in Xen >= 3.0.4, this is can be replaced by the "blktap" user-space driver.) If you are still using loopback, it may be possible to run out of loopback devices, as by default only 64 are supported. You can change this by inserting: options loop max_loop=128 into /etc/modprobe.conf.local in domain 0. Upgrading the Host Operating System ----------------------------------- When upgrading the host operating system from one major release to another (for example, SLES 11 to SLES 12 or openSUSE 12.3 to openSUSE 13.1) or when applying a service pack like SLES 11 SP3 to SLES 11 SP2 all running VMs must be shut down before the upgrade process is begun. On versions of SLES 11 and openSUSE 12 you are using the xm/xend toolstack. After upgrading to SLES 12 and newer openSUSE versions this toolstack will be replaced with the xl toolstack. The xl toolstack does not support Managed Domains. If you wish to continue using Managed Domains you must switch to using libvirt and its command line interface 'virsh'. You may also use virt-manager as a GUI interface to libvirt. After upgrading the host but before you can begin using libvirt on VMs that were previously managed by xm/xend, you must run a conversion tool called /usr/sbin/xen2libvirt for all VMs. For example, to convert all domains previously managed by xend: xen2libvirt -r /var/lib/xend/domains/ Now typing 'virsh list --all' will show your previously xend managed domains being managed by libvirt. Run 'xen2libvirt -h' to see additional options for using this tool. Memory Ballooning in VMs ------------------------ Setting a VMs maximum memory value greater than the initial memory value requires support for memory ballooning in the VMs operating system. Modern SLES and openSUSE guests have this capability built-in. Windows installation media does not support memory ballooning so you must first install the VM without memory ballooning (maxmem equal to initial memory). After the installation, the Virtual Machine Driver Pack (vmdp) must be installed. After this, the VMs maxmem value may be increased. A reboot of the VM is required for this action to take effect. Dom0 Memory Ballooning ---------------------- It is strongly recommended that you dedicate a fixed amount of RAM to dom0 rather than relying on dom0 auto ballooning. Doing so will ensure your dom0 has enough resources to operate well and will improve startup times for your VMs. The amount of RAM dedicated to dom0 should never be less than the recommended minimum amount for running your SUSE distribution in native mode. The actual amount of RAM needed for dom0 depends on several factors including how much physical RAM is on the host, the number of physical CPUs, and the number of VMs running simultaneously where each VM has a specific requirement for RAM. The following example shows the syntax for doing this. This would be added to your grub1 or grub2 configuration; Grub2 Example: Edit /etc/default/grub and add, GRUB_CMDLINE_XEN_DEFAULT="dom0_mem=1024M,max:1024M" and then run grub2-mkconfig -o /boot/grub2/grub.cfg Grub1 Example: Edit /boot/grub/menu.lst and edit the line containing xen.gz kernel /boot/xen.gz dom0_mem=1024M,max:1024M After modifying your grub configuration, you will need to edit /etc/xen/xl.conf and set autoballoon="off". This will prevent xl from automatically adjusting the amount of memory assigned to dom0. Reboot the host for these changes to take effect. Adjusting LIBXL_HOTPLUG_TIMEOUT at runtime ------------------------------------------ A domU with a large amount of disks may run into the hardcoded LIBXL_HOTPLUG_TIMEOUT limit, which is 40 seconds. This happens if the preparation for each disk takes an unexpected large amount of time. Then the sum of all configured disks and the individual preparation time will be larger than 40 seconds. The hotplug script which does the preparation takes a lock before doing the actual preparation. Since the hotplug scripts for each disk are spawned at nearly the same time, each one has to wait for the lock. Due to this contention, the total execution time of a script can easily exceed the timeout. In this case libxl will terminate the script because it has to assume an error condition. Example: 10 configured disks, each one takes 3 seconds within the critital section. The total execution time will be 30 seconds, which is still within the limit. With 5 additional configured disks, the total execution time will be 45 seconds, which would trigger the timeout. To handle such setup without a recompile of libxl, a special key/value has to be created in xenstore prior domain creation. This can be done either manually, or at system startup. A dedicated systemd service file exists to set the required value. To enable it, run these commands: /etc/systemd/system # systemctl enable xen-LIBXL_HOTPLUG_TIMEOUT.service /etc/systemd/system # systemctl start xen-LIBXL_HOTPLUG_TIMEOUT.service In case the value in this service file needs to be changed, a copy with the exact same name must be created in the /etc/systemd/system directory: /etc/systemd/system # cat xen-LIBXL_HOTPLUG_TIMEOUT.service [Unit] Description=set global LIBXL_HOTPLUG_TIMEOUT ConditionPathExists=/proc/xen/capabilities Requires=xenstored.service After=xenstored.service Requires=xen-init-dom0.service After=xen-init-dom0.service Before=xencommons.service [Service] Type=oneshot RemainAfterExit=true ExecStartPre=/bin/grep -q control_d /proc/xen/capabilities ExecStart=/usr/bin/xenstore-write /libxl/suse/per-device-LIBXL_HOTPLUG_TIMEOUT 10 [Install] WantedBy=multi-user.target In this example the per-device value will be set to 10 seconds. The change for libxl which handles this xenstore value will enable additional logging if the key is found. That extra logging will show how the execution time of each script. Troubleshooting --------------- First try to get Linux running on bare metal before trying with Xen. Be sure your Xen hypervisor (xen) and VM kernels are compatible. The hypervisor and domain 0 kernel are a matched set, and usually must be upgraded together. Consult the online documentation for a matrix of supported 32- and 64-bit combinations If you have trouble early in the boot, try passing pnpacpi=off to the Linux kernel. If you have trouble with interrupts or timers, passing lapic to Xen may help. Xen and Linux understand similar ACPI boot parameters. Try the options acpi=off,force,ht,noirq or acpi_skip_timer_override. Other useful debugging options to Xen may be nosmp, noreboot, mem=4096M, sync_console, noirqbalance (Dell). For a complete list of Xen boot options, consult the "Xen Hypervisor Command Line Options" documentation. If domain 0 Linux crashes on X11 startup, please try to boot into runlevel 3. 1) As a first step in debugging Xen you should add the following hypervisor options to the xen.gz line in your grub configuration file. After rebooting, the 'xl dmesg' command will produce more output to better analyze problems. Grub2 Example: Edit /etc/default/grub and add, GRUB_CMDLINE_XEN_DEFAULT="loglvl=all guest_loglvl=all" and then run, grub2-mkconfig -o /boot/grub2/grub.cfg Grub1 Example: Edit /boot/grub/menu.lst and edit the line containing xen.gz kernel /boot/xen.gz loglvl=all guest_loglvl=all 2) With the log levels specified above and the host rebooted, more useful information about domain 0 and running VMs can be obtained using the 'xl dmesg' and 'xl debug-keys' commands. For example, from the command line run: xl debug-keys h and then run: xl dmesg Note that at the end of the output from 'xl dmesg' it includes help on a series of commands that may be passed to 'xl debug-keys'. For example, by passing the letter 'q' to 'xl debug-keys' it will "dump domain (and guest debug) info". xl debug-keys q Now you can again run 'xl dmesg' to see the domain and guest debug info. 3) Sometimes it is useful to attach a serial terminal and direct Xen to send its output not only to the screen, but also to that terminal. First you need to attach a serial cable from the serial port on the server to a second machine's serial port. That second machine could be running minicom (or some other program that can be setup to read from the serial port). Do the following to prepare Xen to send its output over this serial line. Grub2 Example: Edit /etc/default/grub and add, GRUB_CMDLINE_XEN_DEFAULT="loglvl=all guest_loglvl=all console=com1 com1=115200,8n1" Also append additional serial flags to the option below such that it appears as, GRUB_CMDLINE_LINUX_DEFAULT=" console=ttyS0, 115200" where pre-existing flags are those options already present and then run, grub2-mkconfig -o /boot/grub2/grub.cfg Grub1 Example: Edit the /etc/grub/menu.lst file and add the following to the Xen entry, kernel /boot/xen.gz loglvl=all guest_loglvl=all console=com1 com1=115200,8n1 module /boot/vmlinuz-xen console=ttyS0, 115200 Once the hardware and software are configured correctly the server is rebooted and its output should appear on the other terminal as the server boots up. 4) To further debug Xen or domain 0 Linux crashes or hangs, it may be useful to use the debug-enabled hypervisor, and/or to prevent automatic rebooting. Grub2 Example: Edit /etc/default/grub and add, GRUB_CMDLINE_XEN_DEFAULT="noreboot loglvl=all guest_loglvl=all" Edit /boot/grub2/grub.cfg and look for these lines: multiboot /boot/xen-.gz ... and replace them with: multiboot /boot/xen-dbg-.gz' ... Replace with the appropriate version string contained in the filename. Note that running grub2-mkconfig -o /boot/grub2/grub.cfg will overwrite all manual changes made to grub.cfg. Grub1 Example: Edit your menu.lst configuration from something like this: kernel (hd0,5)/xen.gz To something like this: kernel (hd0,5)/xen-dbg.gz noreboot loglvl=all guest_loglvl=all All hypervisor options require a reboot to take effect. After rebooting, the Xen hypervisor will write any error messages to the log file (viewable with the "xl dmesg" command). If problems persist, check if a newer version is available. Well-tested versions will be shipped with SUSE and via YaST Online Update. Resources --------- https://www.suse.com/documentation/sles11/singlehtml/book_xen/book_xen.html http://doc.opensuse.org/products/draft/SLES/SLES-xen_sd_draft/cha.xen.basics.html Feedback -------- In case you have remarks about, problems with, ideas for, or praise for Xen, please report it back to the xen-devel list: xen-devel@lists.xen.org If you find issues with the packaging or setup done by SUSE, please report it through bugzilla: https://bugzilla.novell.com ENJOY! Your SUSE Team.