KVM: Common issues and how to troubleshoot them

This post lists KVM known issues, how to troubleshoot and resolve them.

Error: Log files

There are various trace/log files generated by KVM. The files are follows:

• /var/log/libvirt/qemu - log file for every guest VM (domain_name.log)
• $HOME/.virtinst/virt-install.log – virt-install tool log file.
• $HOME/.virt-manager/virt-manager.log - virt manager log

Error: Create KVM Guest VMs with ISO

There are many methods of creating a guest on KVM machine: the virt-install tool, the virt-manager tool, and the virsh tool. The following command code example creates a KVM guest running CentOS/RHEL 7. Example running storage pool repository (/kvm_repos/kvm_repo1/) MACVTap network bridge over public NIC btbond1 will be used.

The basic Command:

virt-install --name=RHEL7-1 \
--ram=16384 \
--vcpus=4 \
--os-type=linux \
--os-variant=rhel7 \
--accelerate \
--disk /kvm_repos/kvm_repo1/OL-1_boot.img,device=disk,size=40,sparse=yes,cache=none,format=qcow2,bus=virtio \
--network type=direct,source=btbond1,model=virtio \
--vnc \
--noautoconsole \
--cdrom=/iso/RHEL7-U3-Server-x86_64-dvd.iso

Parameters Details

--name: Name of the guest instance. –ram: Memory to allocate for guest instance in megabytes. –vcpus: Number of vcpus to configure for your guest. –os-type: The OS type being installed. –os-variant: The OS variant being installed guests. –accelerate: KVM kernel acceleration capabilities are used. –disk: you define the path, then comma delimited options, device is the type of storage, bus is the interface ide, scsi, usb, virtio – virtio is the fastest. –network: The network configuration, in this case we are connecting to a MACVTap bridge over “btbond1”, and using the virtio drivers which perform much better. –vnc: configures the graphics card to use VNC allowing you to use virt-viewer or virt-manager to see the desktop as if you were at the a monitor of a physical machine. –noautoconsole: configures the installer to NOT automatically try to open virt-viewer (if installed) to view the console to complete the installation – this is helpful if you are working on a remote system through SSH without graphical environment. –cdrom: this option specifies the iso image with which to boot off.

virt-manager, also known as Virtual Machine Manager, is a graphical tool for creating and managing guest virtual machines.

1. Lunch virt-manager. 2. Create a new virtual machine - Click the Create a new virtual machine button to open the new vm wizard. 3. Specify the installation type “Local install media (ISO image or CDROM)” 4. Locate the ISO image, Configure OS Type and Version 5. Configure CPU and memory 6. Configure the virtual storage

7. Final Configuration - Verify the settings of the virtual machine and click “Begin Installation”.

Error: (domain_definition):9: StartTag: invalid element name

The following error message appears when trying to modify Domain using virsh.

# virsh edit RHEL6.10
error: (domain_definition):9: StartTag: invalid element name
<bootmenu enable='yes'/>  <

This error message shows that the parser expects a new element name after the < symbol on line 9 of a guest’s XML file. Open the XML file, and locate the text on line 9: this snippet of a guest’s XML file contains an extra < symbol, correct this.

Error: (domain_definition):20: Unescaped ‘<’ not allowed in attributes values

The following error message appears when trying to modify Domain using virsh.

virsh edit RHEL6.10
error: (domain_definition):20: Unescaped '<' not allowed in attributes values <timer name='rtc' tickpolicy='catchup'/>
----------^

In a guest’s XML file contains an unterminated element attribute value, In this case, timer name=‘rtc’ is missing a second quotation mark. Attribute values must be opened and closed with quotation marks.

Error: (domain_definition):22: Opening and ending tag mismatch: domain line 1

The following error message appears when trying to modify Domain using virsh.

virsh edit RHEL6.10
error: (domain_definition):22: Opening and ending tag mismatch: domain line 1 and clock</clock>
----------^

Message following the last colon, clock line 22 and domain, reveals that contains a mismatched tag on line 22. To identify the problematic tag, read the error message for the context of the file, and check error, correct the XML for missing tag and save the changes.

Error: (domain_definition):1: Specification mandate value for attribute type [domain ty pe=‘kvm’]

The following error message appears when trying to modify Domain using virsh.

virsh edit RHEL7
error: (domain_definition):1: Specification mandate value for attribute ty<domain ty pe='kvm'>
-----------^

XML errors are caused by a typo error. This error message highlights the XML error — in this case, an extra white space within the word type — with a pointer, when trying to save the file and press i. To identify the problematic tag, read the error message for the context of the file, and check error, correct the XML and save the changes.

Error: failed to connect to the hypervisor using URI or Failed to connect to the hypervisor or No connection driver available

There are lots of errors that can occur while connecting to the server using URI, e.g when running a command, the following error (or similar) appears:

$ virsh -c [uri] list
error: no connection driver available for No connection for URI [uri]
error: failed to connect to the hypervisor

This can happen when libvirt is compiled from sources, verify if this is third party complied rpms.

Error: Cannot read CA certificate ‘/etc/pki/CA/cacert.pem’: No such file or directory or Failed to connect to the hypervisor

The error:

$ virsh -c [uri] list
error: Cannot read CA certificate '/etc/pki/CA/cacert.pem': No such file or directory
error: failed to connect to the hypervisor
1. specified URI is wrong (missing one '/' -- e.g. 'qemu://system')

When specifying qemu://system or qemu://session as a connection URI, virsh attempts to connect to host names’ system or session respectively. This is because virsh recognizes the text after the second forward slash as the host.

Use three forward slashes to connect to the local host. For example, specifying qemu:///system instructs virsh connect to the system instance of libvirtd on the local host. When a host name is specified, the QEMU transport defaults to TLS. This results in certificates.

The URI is correct (for example, qemu[+tls]://server/system) but the certificates are not set up properly on your machine.

$ virsh -c qemu:///system list
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied
error: failed to connect to the hypervisor

connect as non-root user, configure following options in ‘/etc/libvirt/libvirtd.conf’ accordingly:

unix_sock_group = [group]
unix_sock_ro_perms = [perms]
unix_sock_rw_perms = [perms]

unable to connect to server at ‘host:16509’: Connection refused

The error:

virsh -c qemu+tcp://HOSTNAME/system
error: failed to connect to the hypervisor
error: unable to connect to server at 'HOSTNAME:16509': Connection refused

The libvirt daemon is not listening on TCP ports even after changing configuration in /etc/libvirt/libvirtd.conf:

# cat /etc/libvirt/libvirtd.conf|grep -i listen_
#listen_tls = 0
#listen_tcp = 1
#listen_addr = "192.168.0.1"

Start the daemon with the –listen option. To do this, modify the /etc/sysconfig/libvirtd file and uncomment the following line:

#LIBVIRTD_ARGS="--listen"

Then, restart the libvirtd service with this command:

# /bin/systemctl restart libvirtd.service

Error: libvirtd failed to start

The libvirt daemon does not start automatically at boot time. Also manually Starting the libvirt daemon fails, there is no ‘more info’ about errors in /var/log/messages.

# systemctl start libvirtd.service

* Starting libvirtd ...
/usr/sbin/libvirtd: error: Unable to initialize network sockets. Check /var/log/messages or run without --daemon for more info.
* start-stop-daemon: failed to start '/usr/sbin/libvirtd' [ !! ]
* ERROR: libvirtd failed to start

# cat /etc/libvirt/libvirtd.conf | grep -i log
# Logging controls
# Logging level: 4 errors, 3 warnings, 2 information, 1 debug
# basically 1 will log everything possible
# WARNING: The "log_filters" setting is recommended instead.
# WARNING: will limit "log_level" to only allow values 3 or 4 if
#log_level = 3#
Logging outputs:
# An output is one of the places to save logging information
# level:syslog:name
# use syslog for the output and use the given name as the ident
# output to journald logging system
# e.g. to log all warnings and errors to syslog under the libvirtd ident:
#log_outputs="3:syslog:libvirtd"

Change logging in /etc/libvirt/libvirtd.conf for warningand errors by enabling the line below. To enable the setting the line, open the /etc/libvirt/libvirtd.conf file in a text editor, remove the hash (or #) symbol from the beginning of the following line, and save the change:

log_outputs="3:syslog:libvirtd"

After diagnosing the problem, comment this line again in the /etc/libvirt/libvirtd.conf file to avoid excessive logs on server. Restart libvirt to see more error and debug accordingly.

Error: Bridge networking interface does not show up

Let’s take an example, want to use a custom bridge, called br0. However, when try to setup a network and make guests use the specific interface, it does not show in the dropdown menu in the Virtual Machine Manager.

manually edit the configuration file for domain. By default, KVM stores files in two locations, either /etc/kvm/vm or /etc/libvirt/qemu, Open the relevant one and manually change the bridged adapter details under . Save and Close the file, start the virtual machine.

<interface type='bridge'>
<source bridge='br0'/>
<mac address='XXX'/>
</interface>

Error: Biosdevname & no network

Biosdevname is a utility that tries to assign BIOS-given names to devices, preserving commonality and simplifying the logic of hardware administration, especially with network devices. There are several ways you can work around the problem.

One, some versions of the utility are capable of detecting they are being invoked inside a virtual environment and will exit without machine any changes. Second one is to pass a kernel argument in the GRUB menu; biosdevname=0 will disable the utility from running.

Third option is to hack the udev rules used to assign names to network cards. e.g this one is designed for a machine with a single network adapter, hence you will need a more dynamic logic for multiple cards. Or, as the commented text says, you will need to create a single separate line for each rule.

You can make a manual change to get the classic assignment:

# vi /etc/udev/rules.d/70-persistent-net.rules

Replace NAME string from “eth_biosname” to “ethX” or anything you need:

# PCI device ...
SUBSYSTEM=="net", ACTION=="add", ATTR(type)=="1", KERNEL=="eth*", NAME="eth_biosname"
# PCI device ...
SUBSYSTEM=="net", ACTION=="add", ATTR(type)=="1",KERNEL=="eth*", NAME="eth0"

You will get different rules depending on the virtual hardware you use in your virtual machines. For example, you could opt for Realtek, e1000 or virtio virtual hardware, resulting in other strings. Pay attention and make sure you match the solution to your specific environment.

Error: Domain already exists

The error:

virsh define machine.xml
error: Failed to define domain from machine.xml
error: operation failed: domain 'machine' already exists with uuid XXX

What happens if you try to define a new domain and it already exists, except you are having trouble finding its configuration file or declaration. virsh list does not have, virt-manager does not show it anywhere.

You will have to find the configuration file and delete it. And then restart libvirtd service.

# updatedb

# locate ol6.10
/etc/libvirt/qemu/ol6.10.xml
/root/ol6.10.xml
/var/lib/libvirt/images/ol6.10.qcow2
/var/lib/libvirt/qemu/domain-1-ol6.10
/var/lib/libvirt/qemu/channel/target/domain-1-ol6.10
/var/lib/libvirt/qemu/domain-1-ol6.10/master-key.aes
/var/lib/libvirt/qemu/domain-1-ol6.10/monitor.sock
/var/log/libvirt/qemu/ol6.10.log

# rm [full path to machine.xml]
# /etc/init.d/libvirtd restart

Error: Failed to create domain from machine.xml or Internal error Unable to find cgroup for machine

This problem may manifest after restarting libvirtd, like in the example, or completely unrelated. A full error message may look like this:

virsh create machine.xml
error: Failed to create domain from machine.xml
error: internal error Unable to find cgroup for machine

The reason for this could be related to systemd, In most cases, it’s a very indelicate race between cgroups and libvirtd, caused by libvirtd service coming up before cgroups, one of the cgroups being deleted or not existing in the first place. You can resolve the problem by editing the libvirtd configuration, /etc/libvirt/qemu.conf.

inside this file, you need to edit the cgroups_controllers directive so it does not list any cgroups, in which case, libvirtd will be able to run without them.

cgroups_controllers = [ ]

After this, you will have to restart libvirtd again. Alternatively, you will have to manually create the necessary cgroups and assign the libvirtd process into the relevant subsystem.

Error: Virtual machine vanishes from VMM on halt/reboot

This could be a very simple issue, in fact. After you halt or reboot the virtual machine, the virtual machine console closes. Your configurations are in place, but this is a major inconvenience, as you have to interfere in the machine management cycle.

You need to look for the on_reboot clause in the relevant XML file for your virtual machine and make sure the action is set to restart rather than destroy. There you go, that’s all there is to it.

<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>

Error: No reboot action; function is not supported

Again, very related to the previous tip. If you specify reboot instead of restart, you will learn this is an invalid command that KVM cannot execute.

libvirtError: this function is not supported by the hypervisor: virDomainReboot

If you use restart, all will be well.

Error: Boot loader error after installation

Sometimes, you may see a GRUB error, most likely number 15, on first reboot following an installation from an external media source. This can happen if you leave the CD/DVD image attached to the virtual machine and selected as the first boot device in your XML file. The problem is similar to this VirtualBox bug, affecting some of the Linux distributions out there. For example:

root (hd0,1)
Filesystem type is ext2fs, partition type 0x83
kernel /boot/vmlinuz

Error 15: File not found

Press any key to continue...

You can resolve by unmounting the ISO image and rebooting the guest. This time, it should work well. Please note the issue will not manifest if you set your hard disk as the first bootable device, because KVM will automatically skip to the second available source, most likely PXE or CD/DVD, if it cannot find a valid partition table on the disk, which should be the case if you’re only installing now.

Error: Server hung or non-recoverable errors, how to collect troubleshooting data

How to inject an NMI to the guest virtual machine - This is used when response time is critical, such as during non-recoverable hardware errors. In addition, virsh inject-nmi is useful for triggering a crashdump in Windows guests.

The following example sends an NMI to the guest1 virtual machine:

# virsh inject-nmi RHEL7

Error: Starting Virtual Machine giving error " Failed to start domain RHEL7"  Cannot access storage file '/var/lib/libvirt/images/test_vol2.qcow2': No such file or directory

Virtual Machine attached disk was deleted from KVM Server repository without detach-disk fromVM and them VM was rebooted, verify xml file for Virtual Machine.

# virsh detach-disk --domain ORACLE_LINUX7 --persistent --live --target vdb

error: Failed to detach disk
error: Requested operation is not valid: domain is not running
[ ~]#
virsh detach-disk --domain ORACLE_LINUX7 --persistent --target vdb
Disk csuccessfully

Error: Blue screen at boot

Since Windows 10 1803 there is a problem when you are using “host-passthrough” as cpu model. The machine cannot boot and is either boot looping or you get a bluescreen. You can workaround this by:

# echo 1 > /sys/module/kvm/parameters/ignore_msrs

To make it permanently you can create a modprobe file kvm.conf:

options kvm ignore_msrs=1

To prevent logging up dmesg with “ignored rdmsr” messages you can additionally add:

options kvm report_ignored_msrs=0

Error: No disks found or Unknown media

If you are going to install Windows from a CD/DVD (instead of an ISO file), make sure that the user which you are running virt-manager as, has read access to the optical drive device on your system. Otherwise, virt-manager may not let you select your drive as an install media location.

# virsh edit RHEL 7

Domain RHEL7 XML configuration not changed.

Error: During the installation of Windows, the error “A disk read error occurred”

During the installation of Windows, the error “A disk read error occurred” shows up during boot time, not allowing you to complete the installation. The problem here is that for whatever reason, virt-manager by default creates disk images using the raw format, and the Windows installer does not like that format. The solution is to convert your disk image to qcow2 format.

To convert your existing image:

# cd /var/lib/libvirt/images/ # or whatever other location you keep your images at
# qemu-img xp.img -O qcow2 xp-qcow2.img

Have to start the installation process again, and re-format the disk, after converting the image to qcow2 format, then attach again and restart installation.