<div dir="ltr">On 18 February 2013 12:50, Ian Campbell <span dir="ltr"><<a href="mailto:ijc@hellion.org.uk" target="_blank">ijc@hellion.org.uk</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="im">On Mon, 2013-02-18 at 12:04 +0200, Gavin wrote:<br>
<br>
<br>
> Firstly I apologise for the cross-post,<br>
<br>
</div>I've added xen-users since you also bounced this there.<br></blockquote><div><br></div><div style>Thanks. :-/ Thanks too for your quick reply.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="im"><br>
> however I don't expect to get as quick a response from the package<br>
> maintainers as I do from the Debian community, and this issue affects<br>
> a service that I've got scheduled to go live at midnight this<br>
> evening. :(<br>
><br>
><br>
> A recent update from xen-hypervisor-4.1-amd64 version 4.1.3-7, to<br>
> version 4.1.3-8 on Debian Wheezy has caused all vm's on this host to<br>
> not receive their arp replies anymore and as such they cannot reach<br>
> their gateways and are now isolated from the network.<br>
><br>
><br>
> There was a more recent update as well (4.1.4-2) which I have now<br>
> since applied however this particular issue persists.<br>
<br>
</div>Networking level stuff is all done by the dom0 (or driver domain) kernel<br>
rather than the hypervisor so it is far more likely that a kernel level<br>
change rather than a hypervisor change would be responsible. What kernel<br>
version are you running? Did it also change?<br></blockquote><div><br></div><div style>This makes sense, although when I did the apt-get upgrade, there was no kernel update, however there may have been packages/drivers that required a kernel mod.</div>
<div style><br></div><div style>Here is the apt history which details what was upgraded when this broke:-</div><div style><br></div><div>Upgrade: dnsmasq-base:amd64 (2.62-3, 2.62-3+deb7u1), tasksel-data:amd64 (3.14, 3.14+nmu1), xen-hypervisor-4.1-amd64:amd64 (4.1.3-7, 4.1.3-8), xen-utils-common:amd64 (4.1.3-7, 4.1.3-8), perl:amd64 (5.14.2-16, 5.14.2-17), firmware-linux-free:amd64 (3.1, 3.2), perl-base:amd64 (5.14.2-16, 5.14.2-17), xen-utils-4.1:amd64 (4.1.3-7, 4.1.3-8), libgnutls26:amd64 (2.12.20-2, 2.12.20-4), perl-modules:amd64 (5.14.2-16, 5.14.2-17), psmisc:amd64 (22.19-1, 22.19-1+deb7u1), python2.6:amd64 (2.6.8-0.2, 2.6.8-1.1), libxenstore3.0:amd64 (4.1.3-7, 4.1.3-8), python2.6-minimal:amd64 (2.6.8-0.2, 2.6.8-1.1), coreutils:amd64 (8.13-3.4, 8.13-3.5), libvirt0:amd64 (0.9.12-5, 0.9.12-6), libcurl3:amd64 (7.26.0-1, 7.26.0-1+wheezy1), manpages:amd64 (3.42-1, 3.44-1), tasksel:amd64 (3.14, 3.14+nmu1), libperl5.14:amd64 (5.14.2-16, 5.14.2-17), libsystemd-login0:amd64 (44-7, 44-8), libxen-4.1:amd64 (4.1.3-7, 4.1.3-8), libcurl3-gnutls:amd64 (7.26.0-1, 7.26.0-1+wheezy1), host:amd64 (9.8.4.dfsg.P1-1, 9.8.4.dfsg.P1-4), libvirt-bin:amd64 (0.9.12-5, 0.9.12-6), rinse:amd64 (2.0-1, 2.0.1-1), ca-certificates:amd64 (20120623, 20130119), xenstore-utils:amd64 (4.1.3-7, 4.1.3-8)</div>
<div><br></div><div style>The kernel I am using is: 3.2.0-2-amd64, also tried 3.2.0-4-amd64 on another host with no success.</div><div style> </div><div style>Would the upgrade above of xen-hypervisor-4.1-amd64 on this Debian system not cause the Dom0 kernel to be changed in any way ?? </div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="im">
> The arp replies are received by the host and passed all the way up to<br>
> the bridge (br200) being used by Xen, however they are not seen on the<br>
> vif (vif2.0) created for the particular vm.<br>
<br>
</div>Do you have any firewall or ebfilter entries which might have either<br>
been discarded or reintroduced by the reboot? (i.e. a manual settings<br>
modification which wasn't propagated to the startup scripts). Or perhaps<br>
sysctl tweaks?<br></blockquote><div><br></div><div style>Nope, not using iptables/ebtables, this was working 100% until the apt upgrade above.</div><div style><br></div><div style>After this broke I did try and add the following to /etc/sysctl.conf but it made no difference:-</div>
<div style><br></div><div style><div>net.bridge.bridge-nf-call-ip6tables = 0</div><div>net.bridge.bridge-nf-call-iptables = 0</div><div>net.bridge.bridge-nf-call-arptables = 0</div><div><br></div><div style>It did add iptables rules but made no difference to this issue.</div>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="im"><br>
> 1) Please let me know if I should roll-back this particular xen<br>
> update, kernel and all, and what those steps may be, or if this is a<br>
> known issue with a particular workaround that I can apply.<br>
<br>
</div>I'd certainly be tempted to try the older kernel, assuming that was also<br>
upgraded. It may even still be installed and in your grub menu already.<br></blockquote><div><br></div><div style>The problem is now we are using grub2 and it appears that on boot grub loads a Linux menu, then the Xen Menu with configs in /etc/grub.d/ so I'm battling to figure out how to do this.</div>
<div style><br></div><div style>I also do not have physical access to this host at the moment so need to set the boot order 'correctly' prior to a reboot.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="im"><br>
> 2) Would moving to openvswitch be another possible workaround?<br>
<br>
</div>Without knowing what the underlying issue is it is hard to predict<br>
whether it will also affect ovs.<br></blockquote><div><br></div><div style>Agreed. <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
> My config:-<br>
<br>
Looks correct to me.<br>
<span class=""><font color="#888888"><br>
Ian.<br></font></span></blockquote><div><br></div><div style>Thanks Ian. </div><div><br></div><div> </div></div><br></div></div>