<div dir="ltr"><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif;font-size:small;color:rgb(0,0,0)">Hi Joseph,</div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif;font-size:small;color:rgb(0,0,0)"><br></div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif;font-size:small;color:rgb(0,0,0)">Thank you, and We figured out the issue with the server. The server is taking much time to respond.</div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif;font-size:small;color:rgb(0,0,0)"><br></div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif;font-size:small;color:rgb(0,0,0)">This template we for HaProxy is:</div><div class="gmail_default" style=""><div class="gmail_default" style=""><font color="#000000" face="trebuchet ms, sans-serif"><br></font></div><div class="gmail_default" style=""><font color="#000000" face="trebuchet ms, sans-serif">global</font></div><div class="gmail_default" style=""><font color="#000000" face="trebuchet ms, sans-serif"> daemon</font></div><div class="gmail_default" style=""><font color="#000000" face="trebuchet ms, sans-serif"> log 127.0.0.1 local0</font></div><div class="gmail_default" style=""><font color="#000000" face="trebuchet ms, sans-serif"> log 127.0.0.1 local1 notice</font></div><div class="gmail_default" style=""><font color="#000000" face="trebuchet ms, sans-serif"><br></font></div><div class="gmail_default" style=""><font color="#000000" face="trebuchet ms, sans-serif">defaults</font></div><div class="gmail_default" style=""><font color="#000000" face="trebuchet ms, sans-serif"> option forwardfor</font></div><div class="gmail_default" style=""><font color="#000000" face="trebuchet ms, sans-serif"> option http-server-close</font></div><div class="gmail_default" style=""><font color="#000000" face="trebuchet ms, sans-serif"> <b>mode http</b></font></div><div class="gmail_default" style=""><font color="#000000" face="trebuchet ms, sans-serif"> balance leastconn</font></div><div class="gmail_default" style=""><font color="#000000" face="trebuchet ms, sans-serif"> log global</font></div><div class="gmail_default" style=""><font color="#000000" face="trebuchet ms, sans-serif"> retries 3</font></div><div class="gmail_default" style=""><font color="#000000" face="trebuchet ms, sans-serif"> maxconn 4096</font></div><div class="gmail_default" style=""><font color="#000000" face="trebuchet ms, sans-serif"> timeout connect 10s</font></div><div class="gmail_default" style=""><font color="#000000" face="trebuchet ms, sans-serif"> timeout client 50s</font></div><div class="gmail_default" style=""><font color="#000000" face="trebuchet ms, sans-serif"> timeout server 50s</font></div><div style="color:rgb(0,0,0);font-family:'trebuchet ms',sans-serif;font-size:small"><br></div></div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif;font-size:small;color:rgb(0,0,0)">On top of haproxy we are using Nginx, we set "keepalive_timeout 15" on Nginx.</div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif;font-size:small;color:rgb(0,0,0)"><br></div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif;font-size:small;color:rgb(0,0,0)">When we do multiple reloads (when HaProxy is serving sessions), the previous process is persisting, once after the user closes his sessions, those processes are not getting killed by themselves.</div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif;font-size:small;color:rgb(0,0,0)"><br></div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif;font-size:small;color:rgb(0,0,0)"><div class="gmail_default">root 9328 0.6 0.4 56204 8608 ? Ss 08:50 0:00 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf 8163</div><div class="gmail_default">root 9918 2.3 0.4 56304 8724 ? Ss 08:11 0:57 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf 9878</div></div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif;font-size:small;color:rgb(0,0,0)"><br></div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif;font-size:small;color:rgb(0,0,0)">Meanwhile, if we update the Haproxy with new configs, sometimes, the request(new configured sites) is trying to serve by the old process, This case some user is getting 500 gateway errors.</div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif;font-size:small;color:rgb(0,0,0)"><br></div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif;font-size:small;color:rgb(0,0,0)">We are using "<b>HTTP" </b>mode.</div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif;font-size:small;color:rgb(0,0,0)"><br></div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif;font-size:small;color:rgb(0,0,0)">Thank you.</div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif;font-size:small;color:rgb(0,0,0)"><br></div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif;font-size:small;color:rgb(0,0,0)"><br></div><div class="gmail_extra"><div class="gmail_quote">On Fri, Sep 11, 2015 at 1:12 PM, Joseph Lynch <span dir="ltr"><<a href="mailto:joe.e.lynch@gmail.com" target="_blank">joe.e.lynch@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi Rajeev,<br>
<span class=""><br>
> We are using Haproxy on top of Mesos cluster, We are doing dynamic reloads for Haproxy based on marathon events (50-100 times in a day). We have nearly 300 applications that are running on Mesos (300 virtual hosts in Haproxy).<br>
<br>
</span>That should be very doable; for context we reload HAProxy thousands of<br>
times per day and have around the same number of services. We do<br>
leverage our improvements to <a href="https://github.com/airbnb/synapse" rel="noreferrer" target="_blank">https://github.com/airbnb/synapse</a> to<br>
minimize the number of reloads we have to do, but marathon is good at<br>
making us reload. Just curious, how do you have HAProxy deployed, is<br>
it running on a centralized machine somewhere or is it running on<br>
every host?<br>
<span class=""><br>
> When we do dynamic reloads, Haproxy is taking long time to reload Haproxy, we observed that for 50 applications takes 30-40secs to reload Haproxy.<br>
<br>
</span>This seems very surprising to me unless you're doing something like<br>
SSL. Can you post a portion of your config?<br>
<span class=""><br>
> We have a single config file for Haproxy, when we do reload all the applications are getting reloaded (Front-ends), this causing downtime of all applications. Is there anyway to reduce the downtime and impact on end-users.<br>
><br>
> We tried this scenario,<br>
> "<a href="http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html" rel="noreferrer" target="_blank">http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html</a>"<br>
><br>
> By this if user requests while reload, the requests are queued and serving after reload.<br>
<br>
</span>Full disclaimer, I wrote that post, and I'm not sure that it will be<br>
all that useful to you if your clients are external or your reloads<br>
take > 30s. "The largest drawback is that this works only for outgoing<br>
links and not for incoming traffic." It would theoretically not be<br>
hard to extend to incoming traffic using ifb but I haven't worked on<br>
actually proving out that solution. If the reload takes > 30s that<br>
technique simply won't work (you'll be buffering connections for 30s,<br>
and likely dropping them). If the 30s reloads are unavoidable you will<br>
likely want to consider one of the alternative strategies mentioned in<br>
the post. For example you can just drop SYNs since the 1s penalty<br>
isn't that big of a deal (will still see 30s+ of unavailability), use<br>
nginx/haproxy to route in front of haproxy (can be a bit confusing and<br>
hard to work with), or make something similar to<br>
<a href="http://inside.unbounce.com/product-dev/haproxy-reloads/" rel="noreferrer" target="_blank">http://inside.unbounce.com/product-dev/haproxy-reloads/</a> (be wary you<br>
pay conntrack with a solution like that).<br>
<span class=""><br>
> But if we do multiple reloads one after another, HaProxy old processes persist even after reloading the HaProxy service, this is causing the serious issue.<br>
><br>
> root 7816 0.1 0.0 20024 3028 ? Ss 03:52 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 6778<br>
> root 7817 0.0 0.0 20024 3148 ? Ss 03:52 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 6778<br>
><br>
> Is there any solution stop the previous process once after it serving the request.<br>
<br>
</span>That is expected behaviour <div class="gmail_default" style="font-family:'trebuchet ms',sans-serif;font-size:small;color:rgb(0,0,0);display:inline"></div>afaik. Those processes are likely still<br>
alive because there are still open connections held against them. How<br>
long is the longest timeout on your backend servers? This is common<br>
with long lived TCP mode backends, but those apps are often resilient<br>
to losing the TCP connection so you may just be able to kill the<br>
haproxy instances (it's what we do).<br>
<span class=""><br>
> Can we separate the configurations based on front-ends like in Nginx, so that only those apps will effect if there is any changes in backend.<br>
<br>
</span>I mean there is nothing that stops you from running multiple haproxy<br>
instances that bind to different ports. I think the right place to<br>
start though is figuring out why reloading takes so long, which can<br>
probably be figured out by looking at the config.<br>
<br>
Good luck,<br>
-Joey<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><font face="trebuchet ms, sans-serif" size="2" color="#0b5394">Best Regards, </font></div><div><div><font face="trebuchet ms, sans-serif" size="2" color="#0b5394">Rajeev Jaggavarapu<div class="gmail_default" style="font-family:'trebuchet ms',sans-serif;font-size:small;color:rgb(0,0,0);display:inline">,</div></font></div><div><font size="2" color="#0b5394"><font face="trebuchet ms, sans-serif"></font><font face="trebuchet ms, sans-serif"></font><font face="trebuchet ms, sans-serif">Pa</font><font face="trebuchet ms, sans-serif">x</font><font face="trebuchet ms, sans-serif">cel</font><font face="trebuchet ms, sans-serif"></font><font face="trebuchet ms, sans-serif"> Labs.</font></font></div><div><font size="2"><font face="trebuchet ms, sans-serif" color="#0b5394"><img src="https://docs.google.com/uc?export=download&id=0B47pfbqK_xmmTDlfVVFXNXktSFk&revid=0B47pfbqK_xmmeGZsSEZXSWpzczBPZkdXcUcyN2JHTmZWejhFPQ" width="200" height="79"></font></font><br></div></div></div></div></div></div>
</div></div>