HAProxy

Quick links

News

Once in a while, a user reports that some old processes remain present after a soft-restart. I could never reproduce the issue until Manuel Soto sent me a truss output of a configuration with which the problem reproduces frequently. The cause is finally that haproxy still binds listening addresses to disabled instances, but does not try to stop them and refuses to exit as long as they remain present. I took the opportunity to fix a related problem causing warnings to be emitted when haproxy tried to stop backends, and a segfault in the configuration parser if ACLs were declared in a defaults section.

That was enough to release 1.3.15.5 and 1.3.14.9. I recommend that any user of 1.3.14 or 1.3.15 upgrades, as these fixes present very minor risk and fix really annoying problems.

September 14th, 2008

Several users reported on the mailing list that they were experiencing abnormal concurrent connection counts higher than the maxconn they configured. They were very prompt to send me configurations and screenshots of the stats report showing the problem. It was indeed a bug triggered every time a connection attempt to a server failed. I've fixed it along with another minor one, and released 1.3.15.4 and 1.3.14.8. Mongrel users are particularly exposed because they run with maxconn=1 and the server cannot accept more connections, so users may experience occasional errors when a server starts to reject connections. It would also be interesting to find why some connections fail to the servers.

September 3rd, 2008

A cool video demonstration of the connection regulation mechanism (maxconn) has been posted on 37signals. It's clearly explained and explicit enough for people not much aware of the mechanism to understand it. Check it there, it's not too long and really worth seeing.

September 2nd, 2008

While working on haproxy 1.3.16, I came across a few bugs in the code, so I issued 1.3.15.3 and 1.3.14.7. The only one annoying concerns 1.3.15 for people who use the balance url_param ... check_post" construct to hash on parameters present in POST requests. There is a risk of crashing (but no server compromission though) with some invalid requests. Fortunately, this feature is very new and ver limited to niche users, but it needed a quick fix anyway. Other bugs are pretty minor and most of them concern small issues with how timeouts are handled.

July 20th, 2008 : two lines...

Two lines... That's all what is needed with the new TCP content inspection system to stop half of the spams I got home. One of my major customers who uses HAProxy a lot has sponsored the development of some preliminary content inspection which is used to decide whether to forward a connection or not. The very first usage of this feature consists in checking that only SSL is spoken on a connection. But most likely more protocols will come soon. As a nice side effect, I could now add a delay before the HELO message of my SMTP server, and reject all robots which talk first (forbidden). And since many spam bots have small timeout values, many of them abort before the timeout is reached, resulting in my incoming spam rate dropping from about 300/hour to "only" 150/hour. Those who keep up with the time out slow down due to limited resources. The small addition simply consists in adding those two lines in the frontend :

		tcp-request inspect-delay  35s
		tcp-request content reject if REQ_CONTENT
	  

June 21th, 2008

haproxy versions 1.3.15.2 and 1.3.14.6 have been released to fix a major bug in request queue handling. The problem was that due to a design problem, it was possible for new requests to be immediately served by a server before a request in queue would be served. That caused some requests to remain in queue until they reached the queue timeout, after which either they would eventually be served, or return a 503 error code to the client.

Since it was a design problem, it took a lot of time analyzing the root cause and finding a solution. However, as a positive side effect, the fix now makes the redispatch option work for requests which overflow a queue. That way, clients do not get a 503 error anymore but can be served by another server (which was the purpose of the redispatch option.

Note that it is possible that 1.2 is also affected by the issue since some parts of the faulty code have not changed since. But it is very hard to determine if it is faulty or not, and backporting the fix would take even more time. Maybe I will eventually take a look at it if people complain about the issue.

Update (2008/06/28): Alexander Staubo, who first noticed the problem, has run a new series of tests showing that the problem is definitely fixed. It also demonstrates the very nice positive effect of running with maxconn 1 with Rails servers.

May 25th, 2008

Released haproxy versions 1.3.15.1 and 1.3.14.5 with minor fixes : build fix for GCC 4.3, fix for early truncate of stats output in certain circumstances, and better handling of large amounts of highly active sockets. I indeed discovered during testing that the sepoll poller was so much efficient that when running at gigabit speed with 80000 active sockets fighting for their CPU share, almost all of them were running in speculative mode, causing starvation of the remaining ones, which in turn caused the accept() call to be very rarely called. Delays of about 40 seconds have been observed on a 3.4 GHz Pentium 4 to get the stats page under such a load. The other pollers were not better BTW. The fix consisted in ensuring that polled events are checked at much often as the speculative ones. With this fix, the stats page responds in less than one second on such a saturated machine. There is still room for improvement relying on events prioritization though. Version 1.3.15 has been promoted as the recommended one since there has been no regression report. Version 1.2.18 was also released for users of 1.2 which experienced trouble building on BSD.

April 19th, 2008

Released haproxy version 1.3.15 with many new features. The most important changes are stats updates (HTTP and UNIX), enhancements of server checks such as tracking and dynamic intervals, addition of the leastconn load-balancing algorithm, a fully transparent mode on Linux, better handling of connection failures (dead server avoidance and turn-around state), support for inter-site off-loading through redirects, updates to the build process, and large documentation updates. For more information, please check the ChangeLog. Due to the important number of changes, upgrade from earlier versions should be performed with a bit of care.

Once again, a lot of code comes from contributions. I'd like to specially thank Krzysztof Oledzki for a lot of useful contributions, including the SNMP agent, and the guys at Nokia for the good work they have done on POST parameter hashing.

March 30th, 2008

I finally assembled my new machines and installed the donated 10-Gig Myricom NICs. I ran a few benchmarks. Result: new bandwidth records set for HAPoxy: 9.897 Gbps and 35128 hits/s! It's possibly the highest bitrate achieved to date with an opensource load-balancer! BTW, even most commercial ones are commonly limited to 4 Gbps by hardware design. What's a bit frustrating for a precision-tweaker like me is that those NICs work out-of-the box on dirt cheap hardware, there's almost no joy passing beyond the first 4 Gbps :-)

March 8th, 2008

Released haproxy maintenance version 1.3.14.3 to address several minor bugs and clean up the configuration manual a little bit. One annoying issue with backup servers in round robin mode was fixed. Nothing really important was changed in this version, this makes it a good candidate for distro updates.

February 23rd, 2008

I finally decided to buy an expensive motherboard to upgrade my PC in order to begin testing with the 10Gig NICs. I selected an Asus P5E3-WS Pro because I needed PCI-X slots for my older cards. I've put a Core2 Duo E8200 (45nm) because I wanted a lot of silence. The mobo has 2 PCI-E 16x slots, which made it possible for me to run a back-to-back test between two 10Gig NICs. Since the board does not support PCI graphics cards, I had to boot on serial port (the only VGA card I've got running on this mobo was a cheap crappy GeForce 8400 GS which does not work under X). Well, my first test is quite encouraging : I can achieve 10 Gbps of HTTP traffic between the two NICs with the server and client on the same machine, which means that the hardware will be able to support haproxy under the same conditions. I tried with client + haproxy + server but the bit rate diminished to about 6-7 Gbps. I'm impatient to buy the 3 other mobos to build a full lab. I will mix 2 Athlons and one C2D so that I can experiment which one is better for which type of traffic. Stay tuned!

January 21th, 2008

Released haproxy maintenance version 1.3.14.2 to address several minor bugs as well as a major one affecting Linux 2.6 users with the sepoll poller, which can result in truncated responses if the client closes the connection before the server completes its response. Note that version 1.3.13.2 was released too with those bugs fixed. The GNU Makefile was crappy and caused trouble in some build environments. It has been rewritten in a more flexible manner, while still providing full variable compatibility with existing build systems. Distribution packagers are encouraged to migrate over to this one. The new configuration manual is almost finished. All keywords and all their options have been documented. Only the logs section remains to be completed. This version has been merged with 1.3.14.2. Some minor robustness and performance tuning parameters have been added, mostly timeouts and backlog.

January 13th, 2008

Worked all the day both in kernel and haproxy to get full transparent proxy to work on Linux. Now, with a small kernel patch, it's possible for haproxy to become completely transparent and just appear as a router, without touching either source nor destination addresses and ports. And all this without NAT, at the same performance level as in normal proxy mode. This will be great for people looking for SMTP/HTTPS/FTP relaying and load balancing. I'm even planning on installing it on my firewall ;-) Stay tuned for the updates, I will soon post the patches once cleaned up.

December 12th, 2007 : Santa Claus left a present for me at EXOSEC !

Some of you might have already got their hands on this. For those who don't know yet, this beautiful piece of art is a 10 Gbps Ethernet NIC from Myricom. For a long time, I had been tempted by their legendary high performance network cards, which were said everywhere to be able to saturate a 10 Gig wire under Linux without putting too much stress on the CPU, using a mainstream opensource driver, and without resorting to dirty tricks such as TOE. What would a performance addict like me need more ? I finally decided to mail these guys and described how I'm currently used to benchmark HAproxy with aggregated Gigabit NICs, with a minimum of 4 NICs in a setup (1 for the client, 2 for the proxy, 1 for the server). 4 hours later, when I woke up, I had a mail from Charles Seitz, Myricom's CEO. He explained to me that he was pleased to offer me 4 NICs with cables, plus one spare of each just in case, as their contribution to the project... yes, I'm talking about a donation of five 10Gig NICs! That's awsome! And if it would not be enough for some of you to find them really cool, he also provided me with french-speaking contacts, free access to their support and important advices for the choice of motherboards to get the best out of those wonderful NICs! I don't even know the polite words to say in such circumstances :-) Today I've been monitoring the shipping steps at UPS. This evening, I noticed that they arrived at EXOSEC. After leaving the customer's, I went back there to find this big parcel on my desk, with its contents very carefully packed. I must say that I was both very excited and extremely careful while opening the packaging. The first thing I noticed after extracting the first NIC from its packaging was that it had a very clean design, as can be seen on this photo. They are also very thin as shown on the picture on the right, so there will be no problem putting two of them side-by-side in the proxy. The CX4 connector looks a bit fragile, but careful manipulation is the minimal requirement to use the highest speed standard Ethernet. From what I understood, this is the same connector as used on Infiniband, except that 10GE has terminators on the board. Well, obviously, there are very nice companies out there who deserve to be talked about! Their very generous support to open source projects leaves many others far behind. People say that Santa Claus lives in the North Pole, but now I know he lives in Arcadia in California :-) Thank you very much Charles, thanks very much Myricom. Be sure to read about my first test results here.

December 6th, 2007

Released haproxy version 1.3.14. A good part of the changes comes from nice contributors of the mailing list. Most sensible changes include support for dynamic server weights offering support for slow start and graceful shutdown. The load balancer is now able to report its servers state to outer components, enabling the building of more complex multi-site architectures involving dynamic routing protocols such as BGP. People who were complaining about the rough configuration, rough statistics, or lack of logging to UNIX sockets, should really give this one a try. Rate of changes after this version should significantly drop in order to progressively switch the tree to a stable state.

October 18th, 2007

Released haproxy maintenance version 1.3.12.3 and new version 1.3.13. Thanks to quite a bunch of subscribers of the new mailing list, several nasty bugs have been ironed out and a handful of smart features have been added. The most visible add-ons are the UNIX socket to access the statistics page, and the support for Doug Lea's dlmalloc which dramatically reduces memory usage during soft-reconfigurations.

September 22th, 2007

The long awaited haproxy mailing list is finally there ! Special thanks to my friend Benoit who has spent some time setting it up on our Formilux server. For more info on how to subscribe, please check the Contacts entry at the bottom of the home page.

September 20th, 2007

Released haproxy version 1.3.12.2. It fixes several bugs affecting timeouts and retry counts when configs are split between frontends and backends. Some sanity checks on the configuration file were never executed, causing some erroneous configurations to be accepted without being fixed. Last, the license has been clarified in a few files from O'Reilly. All in all, it seems like keeping a supported version is already starting to pay off, as people are looking for something stable and report bugs very quickly. All version 1.3 users are encouraged to upgrade to 1.3.12.2.

September 5th, 2007

Released haproxy version 1.3.12.1. It fixes a few bugs discovered in 1.3.12, notably one which could lead to crashes under Linux with speculative I/O when clients disconnect before the connection has been established to the server. As a workaround, it is possible to specify "nosepoll" in the "global" section. A "stats refresh <interval>" option has also been added because some people like to have the stats page automatically refresh. It's also possible to hide all failed servers on the stats page now. This version also contains the new configuration manual which has just been started but which helps understand how to use ACL.

July 15th, 2007

Started writing the new Configuration Manual. It enumerates all configuration keywords and in what context they may be used. It also includes a few examples of ACLs. It is not finished yet, but I decided to publish it because people have really no other valuable sources of information to use content switching. It only covers version 1.3.12, and updates will only cover the latest version, making it far more readable. Please take a look at it and start from the examples in the examples/ directory from the sources. Any feedback is welcome :-)

June 17th, 2007

Released haproxy version 1.3.12. It completes integration of ACL with Content Switching, and allows you to customize your error responses. Except for the ACL and a few bugs, there have been few changes since 1.3.11.4, and I intend to support 1.3.12 during development and cleanups of the next versions which may not be as reliable. Several big content providers use 1.3 to regulate the traffic to/from their web servers, and there is a real demand for a stable version with the new features and performance of 1.3. And considering that some of them even pay for this, I understand they want something really reliable.

June 3rd, 2007

Released haproxy version 1.3.11.4. It fixes 2 long-standing bugs in timeout handling, which could sometimes cause 100% CPU usage during several seconds when a client had closed its write channel. Some small improvements to the I/O subsystem should save some CPU cycles on high bandwidth sites. It is now possible to finely tune the pollers for reduced latency.

May 14th, 2007

Released haproxy version 1.3.11.3. It fixes the (hopefully) last bug affecting Linux users with speculative I/O processing, introduced in 1.3.9. This bug was also causing random timeouts. Do not use versions 1.3.11 to 1.3.11.2 as they are all broken.

New in this release are a better timer management and a new memory manager which is able to self-manage its pools and free unused ones when memory is becoming scarce. It is also easier to code with this new one since it's not necessary anymore to declare pool sizes. Overall, yet another performance boost of 5% has been gained.

May 10th, 2007

Released haproxy version 1.3.10.2 to fix build on OpenBSD.

May 9th, 2007

Released haproxy version 1.3.10.1. It fixes a serious bug affecting Linux users with speculative I/O processing, introduced in 1.3.9. This bug was causing random timeouts on some traffic patterns, mostly noticeable in TCP mode but almost certainly in HTTP too. All Linux users of 1.3.9 and 1.3.10 should either upgrade or disable speculative I/O as a workaround, by starting haproxy with the -ds argument or by setting nosepoll in the global section.

May 9th, 2007

Released haproxy version 1.3.10. This one adds ACL, SMTP health checks (thanks to Peter van Dijk), and URI hashing (thanks to Guillaume Dallaire). Also, the rbtree was replaced with a much faster tree, leading to an overall performance boost around 5%.

The speculative I/O processing in 1.3.9 has introduced some bugs which have been fixed in this version. I feel confident that latest changes have brought their pile of bugs too. I will probably spend some time soon to do cleanup and stabilization work, eventhough both are not really compatible.

I also want to thank all the people who contribute code and testing. You are more and more at each release. I'm impatient to clean up the remains of the old code, so that even more people can contribute code. Interestingly, all contributions till now were of high quality. This is probably induced by some sort of selection caused by the technical aspect of the product, and the skills required to use the development version. Thanks again to you all !

Apr 22th, 2007

Done a quick benchmark at EXOSEC with haproxy 1.3.9 running on a nice single-core system equipped with many PCI-Express Gigabit NICs. The graph shows pretty decent results !

Apr 15th, 2007

Released haproxy version 1.3.9. This one adds modularization to the pollers, which made it possible for me to finally implement support for FreeBSD kqueue(). I'd like to thank Olivier Warin for providing me a FreeBSD account during this development.

A new concept was introduced too : speculative I/O. It is a new method consisting in reducing the number of calls to the expensive epoll_ctl() and epoll_wait() by attempting to access the file descriptors before being notified about their readiness. This provides an overall speed boost of 10%, which is quite much for just a poller.

Apr 3rd, 2007

Released haproxy version 1.3.8.2 to fix a minor and a major bug. The minor bug caused the response rewrite to fail on the status line. The major bug which was introduced in 1.3.6 could cause the process to crash in some circumstances when rewriting the request line (method and/or URI). All users of 1.3.6 and later must upgrade.

Apr 1st, 2007

Released haproxy version 1.3.8.1 to fix very minor bugs, and slightly improve performance. Request headers were not added if option httpclose was not set. Bruno Michel contributed a VIM script for syntax color highlighting.

Mar 25th, 2007

Released haproxy version 1.3.8. Several bugs which might have caused crashes on erroneous configurations have been fixed. The response processing is now completed, which means that real configurations can now be written ; HAProxy 1.3.8 now is at least equivalent to 1.2.17 in terms of features.

Just like with every release, several code optimization have led to small but noticeable performance increases, particularly on very high data bandwidth. The configuration errors are handled more gracefully now with indications about what failed and hints to resolve the issue. HAProxy now builds on MacOS 10.4 thanks to Dan Zinngrabe who provided a makefile. Also, it is now possible to send health checks to an alternate server address, thanks to a patch from Fabrice Dulaunoy.

Users of 1.3 are encouraged to upgrade to 1.3.8 as it both fixes known bugs and converges towards something less tricky than previous versions.

Mar 17th, 2007

Released haproxy version 1.2.17. I have backported Sin Yu's rbtree scheduler from version 1.3 since it proved to be stable. A few minor bugs were fixed, and two useful contributions were merged : support for user and group keywords as alternatives to numerical uid and gid from Marcus Rueckert, and the ability to prevent some source addresses from appearing in the X-Forwarded-For header, which is useful when combined with Stunnel for instance (patch from Bryan Germann). Thanks to both of them, contribs are always welcome !

The architecture manual was updated to reflect new features in branch 1.2, with examples for stunnel and for load mitigation.

Users of 1.2.16 with high loads are encouraged to upgrade to 1.2.17 as it offers them the high performance of branch 1.3 with the reliability of the stable branch 1.2.

Jan 27th, 2007

Released haproxy version 1.3.7. I found a critical bug in the new parser in development branch, causing crashes when an empty header is passed. This was caused by a missing pointer assignment in the empty header processing path. All 1.3.6 users MUST upgrade to 1.3.7.

Jan 22th, 2007

Released haproxy version 1.3.6. I spent a long time reworking the HTTP message parser. It now consists in a carefully hand-optimized 28-states FSM. The new code will look awful to goto haters, and will please FSM lovers. It's blazingly fast : parsing and indexing all of the 660 bytes of an HTTP request from Firefox on Freshmeat only takes 1.94 microsecond on my 1.7 GHz Pentium-M notebook, which means it can do it more than 500000 times a second!

The request code has been cleaned up a lot and adapted to this new FSM. Adding layer7 rules based on new criteria is now trivial. The response code will be ported next, but the code was so much cleaner and faster that it was worth releasing one version before breaking everything. Several bugs were fixed since 1.3.5. I really consider 1.3.6 as the most likely reliable 1.3 release to date.

Jan 7th, 2007

In order to support the new Linux Layer7 Switching project, I have implemented support for kernel TCP splicing using Alexandre Cassen's library. This is still experimental but already works remarkably well. On my notebook at 400 Mbps, haproxy's usage goes down from 65% to 5-10%. I have written some doc explaining how to setup up TCP splicing, with an example. Since the original code was provided for Linux kernel 2.6.19 only, I have backported the patches to kernel 2.6.16 and 2.4.33.

The second great news is that Sin Yu has provided me with a useful patch for the second time : the task scheduler is now based on an rbtree and not on the dirty old dual-linked list anymore. It means that people who had performance problems and who had to set all their timeouts to the same value as a workaround will not have to do this anymore. I have tested, and the code works like a charm ! Thanks again Sin !

Jan 2nd, 2007

After about 4500 new lines of code and some useful feedback from a bunch of brave beta-testers, I'm pleased to announce haproxy version 1.3.4 with the new Content-Switching features !!!

It is now possible to select a backend (server pool + load balancing algo) depending on any parameter in the request, such as any part of the URI, the host name, etc.... As of now, I've merged Sin Yu's patch to permit switching based on a request regex, but the framework is ready to accept many other criteria. The HTTP request parser has been completely rewritten to support unlimited header inspection, and the statistics page has been rewritten, as can be seen on the demo page. It is far from being finished right now, but it seems pretty usable. The server state machine should be adapted though.

There is still no doc, so please note that old configurations do still work, and that in order to switch from an instance to another backend, you need to use "reqisetbe <regex> <new_proxy>". Also, there's a config example here that will be worth any doc.

Dec 5th, 2006

The load balancing article has been linked to from LinuxFR. The small 128 kbps uplink is currently running at full speed but the site is still responding thanks to haproxy queuing the connections to smoothen the traffic. Next time, I should also write an article on setting up the QoS with tc, because typing remotely with SSH is still very responsive under full load :-)

Jul 4th, 2006

Opened development branch 1.3, which started with a major cleanup. Not sure yet about all features which will be merged, the first step is to clean up the code and make it modular. The API's licence has been switched to LGPL in order to later allow linking with binary external modules developped for applications covered by NDAs for example. Version 1.3.0 is exactly the same as 1.2.14+bugfixes so it is a stable starting point. It is available here.

⇐ Back to HAProxy

Contacts

• Main site : http://1wt.eu/
• e-mail :