[Pkg-xen-devel] Bug#744163: xenstore problems

[Matyas Koszik]

Package: xen-utils-4.1
Version: 4.1.4-3+deb7u1
Severity: important

When starting (or migrating in) a domain, sometimes I get the following exception:

[2014-04-01 10:58:08 4821] ERROR (XendDomainInfo:2927) XendDomainInfo.initDomain: exception occurred
Traceback (most recent call last):
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line 2914, in _initDomain
    self._createDevices()
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line 2364, in _createDevices
    devid = self._createDevice(devclass, config)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line 2326, in _createDevice
    return self.getDeviceController(deviceClass).createDevice(devConfig)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/server/DevController.py", line 63, in createDevice
    (devid, back, front) = self.getDeviceDetails(config)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/server/netif.py", line 122, in getDeviceDetails
    devid = self.allocateDeviceID()
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/server/DevController.py", line 406, in allocateDeviceID
    return complete(path, self._allocateDeviceID)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/xenstore/xstransact.py", line 361, in complete
    t = xstransact(path)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/xenstore/xstransact.py", line 29, in __init__
    self.transaction = xshandle().transaction_start()
Error: (2, 'No such file or directory')
[2014-04-01 10:58:08 4821] ERROR (XendDomainInfo:488) VM start failed


After this, the system is left in an inconsistent state:

# xm list|grep x
x                                      69  1024     1     --ps--      0.0
# xm destroy x
Error: (2, 'No such file or directory')
Usage: xm destroy <Domain>

Terminate a domain immediately.


Now if I try to 'xm create' a new domain, I get the following:

[2014-04-02 18:14:20 4821] ERROR (xmlrpclib2:181) Internal error handling xend.domain.create
Traceback (most recent call last):
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/util/xmlrpclib2.py", line 134, in _marshaled_dispatch
    response = self._dispatch(method, params)
  File "/usr/lib/python2.7/SimpleXMLRPCServer.py", line 420, in _dispatch
    return func(*params)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/server/XMLRPCServer.py", line 80, in domain_create
    info = XendDomain.instance().domain_create(config)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomain.py", line 999, in domain_create
    self._refresh()
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomain.py", line 429, in _refresh
    self._refreshTxn(txn, refresh_shutdown)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomain.py", line 442, in _refreshTxn
    self.domains[domid].update(dom, refresh_shutdown, transaction)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line 3543, in update
    self.refreshShutdown(info)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line 2069, in refreshShutdown
    if self.readDom('xend/shutdown_completed'):
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line 1693, in readDom
    return xstransact.Read(self.dompath, *args)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/xenstore/xstransact.py", line 307, in Read
    return complete(path, lambda t: t.read(*args))
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/xenstore/xstransact.py", line 361, in complete
    t = xstransact(path)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/xenstore/xstransact.py", line 29, in __init__
    self.transaction = xshandle().transaction_start()
Error: (2, 'No such file or directory')



After restaring xend even a simple 'xm list' operation results in Error: (2, 'No such file or directory'):

[2014-04-02 18:17:01 28301] ERROR (xmlrpclib2:181) Internal error handling xend.domains_with_state
Traceback (most recent call last):
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/util/xmlrpclib2.py", line 134, in _marshaled_dispatch
    response = self._dispatch(method, params)
  File "/usr/lib/python2.7/SimpleXMLRPCServer.py", line 420, in _dispatch
    return func(*params)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/server/XMLRPCServer.py", line 67, in domains_with_state
    domains = XendDomain.instance().list_sorted(state)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomain.py", line 879, in list_sorted
    doms = self.list(state)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomain.py", line 848, in list
    self._refresh(refresh_shutdown = False)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomain.py", line 429, in _refresh
    self._refreshTxn(txn, refresh_shutdown)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomain.py", line 445, in _refreshTxn
    new_dom = XendDomainInfo.recreate(dom, False)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line 180, in recreate
    vmpath = xstransact.Read(dompath, "vm")
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/xenstore/xstransact.py", line 307, in Read
    return complete(path, lambda t: t.read(*args))
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/xenstore/xstransact.py", line 361, in complete
    t = xstransact(path)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/xenstore/xstransact.py", line 29, in __init__
    self.transaction = xshandle().transaction_start()
Error: (2, 'No such file or directory')



Changing the toolstack to xl gave some basic functionality back (I can at least list VMs now,
starting is not possible and other operations have not been tried yet).

The configuration of the VM that triggered this issue is the following:

kernel = '/usr/lib/xen/boot/hvmloader'
builder='hvm'
device_model = '/usr/lib/xen/bin/qemu-dm'
vfb = ['type=vnc,vnclisten=0.0.0.0:6122,vncdisplay=6122,vncpasswd=x']
memory = 1024
name = "x"
vcpus = 1
vif = [ 'vifname=x,mac=00:16:3e:00:00:22,ip=x' ]
disk = [ 'phy:x,hda,w', ]