This seems to be directly related to my incoming VPN connection (via pptpd). I'm running IPv6 and radvd over the interface so that might be additionally confusing snmpd. If I down the VPN on the client, then snmpd on the server stays up and running.