HAProxy

The Reliable, High Performance TCP/HTTP Load Balancer

Quick links

Quick News

haproxy versions 1.3.15.2 and 1.3.14.6 have been released to fix a major bug in request queue handling. The problem was that due to a design problem, it was possible for new requests to be immediately served by a server before a request in queue would be served. That caused some requests to remain in queue until they reached the queue timeout, after which either they would eventually be served, or return a 503 error code to the client.

Since it was a design problem, it took a lot of time analyzing the root cause and finding a solution. However, as a positive side effect, the fix now makes the redispatch option work for requests which overflow a queue. That way, clients do not get a 503 error anymore but can be served by another server (which was the purpose of the redispatch option.

Note that it is possible that 1.2 is also affected by the issue since some parts of the faulty code have not changed since. But it is very hard to determine if it is faulty or not, and backporting the fix would take even more time. Maybe I will eventually take a look at it if people complain about the issue.

Update (2008/06/28): Alexander Staubo, who first noticed the problem, has run a new series of tests showing that the problem is definitely fixed. It also demonstrates the very nice positive effect of running with maxconn 1 with Rails servers.

May 25th, 2008

Released haproxy versions 1.3.15.1 and 1.3.14.5 with minor fixes : build fix for GCC 4.3, fix for early truncate of stats output in certain circumstances, and better handling of large amounts of highly active sockets. I indeed discovered during testing that the sepoll poller was so much efficient that when running at gigabit speed with 80000 active sockets fighting for their CPU share, almost all of them were running in speculative mode, causing starvation of the remaining ones, which in turn caused the accept() call to be very rarely called. Delays of about 40 seconds have been observed on a 3.4 GHz Pentium 4 to get the stats page under such a load. The other pollers were not better BTW. The fix consisted in ensuring that polled events are checked at much often as the speculative ones. With this fix, the stats page responds in less than one second on such a saturated machine. There is still room for improvement relying on events prioritization though. Version 1.3.15 has been promoted as the recommended one since there has been no regression report. Version 1.2.18 was also released for users of 1.2 which experienced trouble building on BSD.

April 19th, 2008

Released haproxy version 1.3.15 with many new features. The most important changes are stats updates (HTTP and UNIX), enhancements of server checks such as tracking and dynamic intervals, addition of the leastconn load-balancing algorithm, a fully transparent mode on Linux, better handling of connection failures (dead server avoidance and turn-around state), support for inter-site off-loading through redirects, updates to the build process, and large documentation updates. For more information, please check the ChangeLog. Due to the important number of changes, upgrade from earlier versions should be performed with a bit of care.

Once again, a lot of code comes from contributions. I'd like to specially thank Krzysztof Oledzki for a lot of useful contributions, including the SNMP agent, and the guys at Nokia for the good work they have done on POST parameter hashing.

Recent news...

Latest versions

Description

HAProxy is a free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. It is particularly suited for web sites crawling under very high loads while needing persistence or Layer7 processing. Supporting tens of thousands of connections is clearly realistic with todays hardware. Its mode of operation makes its integration into existing architectures very easy and riskless, while still offering the possibility not to expose fragile web servers to the Net, such as below :

Currently, two major versions are supported :

• version 1.1 - maintains critical sites online since 2002
  The most stable and reliable, has reached years of uptime. Receives no new feature, dedicated to mission-critical usages only.
• version 1.2 - opening the way to very high traffic sites
  The same as 1.1 with some new features such as poll/epoll support for very large number of sessions, IPv6 on the client side, application cookies, hot-reconfiguration, advanced dynamic load regulation, TCP keepalive, source hash, weighted load balancing, rbtree-based scheduler, and a nice Web status page. This code is still evolving but has significantly stabilized since 1.2.8.

Additionally, a third version 1.3 is under active development. New features include :

• Content Switching : provides ability to select a group of server based on any part of the request such as the URI, the Host field, cookies, or anything else. There is a growing request for this feature from large sites which separate dynamic and static contents.
• Full Transparent Proxy : it is possible connect to the server with the Client's IP address or even any other IP address. This is possible only on Linux 2.4/2.6 with the cttproxy patch. This feature also makes it possible to transparently handle part of the traffic for a particular server without changing any server's address.
• New faster tree-based scheduler : versions up to 1.2.16 required that all timeouts were set to the same value to support tens of hundreds of connections at full speed. With this new scheduler, it is no longer the case. I have backported it to 1.2.17.
• Kernel TCP splicing : avoiding kernel-to-user then user-to-kernel data copies improves bandwidth and lowers CPU usage. Haproxy 1.3 supports Linux L7SW in order to achieve multi-gigabit performance on commodity hardware.
• Connection Tarpitting : since the cost of maintaining a connection open is low, it is sometimes desirable to "tarpit" attack bots, which means maintain their connections open to limit their capacity. This has been developped for a site crawling under a small DDoS with easily identifiable requests from a few thousand zombies.
• Finer Header Processing : will make it easier to write header-based rules and to process parts of the URI.
• Very Fast and reliable Header Parsing : full parsing and indexing of an average request typically takes less than 2 microseconds with fully RFC2616-compliant integrity checks.
• Modular Design : allow more people to contribute to the project and make it easier to debug. The pollers have been split, already making their development a lot easier. Other subsystems will be modularized soon.
• Speculative I/O processing : try to access data on a socket before being notified about its readiness. The poller just speculates about what should be available and what should not, tries to guess, and if it wins, several expensive syscalls are saved. If it loses, those syscalls will have to be called anyway. A net overall gain of about 10% has been observed using Linux epoll().
• ACLs : use any combination of any criterion as a condition to any action.
• More load balancing algorithms : right now, Weighted Round Robin, Weighted Source Hash and Weighted URL Hash are implemented. Weighted Least Conns is pending. Other algorithms may come later such as Weighted Measured Response Time.

Unlike other free "cheap" load-balancing solutions, this product is only used by a few hundreds of people around the world, but those people run very big sites serving several millions hits and between several tens of gigabytes to several terabytes per day to hundreds of thousands of clients. They need 24x7 availability and have internal skills to risk to maintain a free software solution. Often, the solution is deployed for internal uses and I only know about it when they send me some positive feedback or when they ask for a missing feature ;-)

Design Choices and history

It began in 1996 when I wrote Webroute, a very simple HTTP proxy able to set up a modem access. But its multi-process model cloberred its performance for other usages than home access. Two years later, in 1998, I wrote the event-driven ZProx, used to compress TCP traffic to accelerate modem lines. It was when I first understood the difficulty of event-driven models. In 2000, while benchmarking a buggy application, I heavily modified ZProx to introduce a very dirty support for HTTP header rewriting. HAProxy's ancestor was born. First versions did not perform the load-balancing themselves, but it quickly proved necessary.

Now in 2007, the core engine is reliable and very robust, despite the frightening comments in the code. Event-driven programs are robust and fragile at the same time : their code needs very careful changes, but the resulting executable handles high loads and supports attacks without ever failing. It is the reason why HAProxy only supports a fine set of features.

People often ask for SSL and Keep-Alive support. Both features will complicate the code and render it fragile for several releases. By the way, both features have a negative impact on performance :

• Having SSL in the load balancer itself means that it becomes the bottleneck. When the load balancer's CPU is saturated, the overall response times will increase and the only solution will be to multiply the load balancer with another load balancer in front of them. the only scalable solution is to have an SSL/Cache layer between the clients and the load balancer.
• Keep-alive was invented to reduce CPU usage on servers when CPUs were 100 times slower. But what is not said is that persistent connections consume a lot of memory while not being usable by anybody except the client who openned them. Today in 2006, CPUs are very cheap and memory is still limited to a few gigabytes by the architecture or the price. If a site needs keep-alive, there is a real problem. Highly loaded sites often disable keep-alive to support the maximum number of simultaneous clients.

However, I'm planning on implementing both features in future versions, because it appears that there are users who mostly need availability above performance, and for them, it's understandable that having both features will not impact their performance, and will reduce the number of components.

Supported platforms

• Linux 2.4 on x86, x86_64, Alpha, SPARC, MIPS, PARISC
• Linux 2.6 on x86, ARM (ixp425), PPC64
• Solaris 8/9 on UltraSPARC 2 and 3
• Solaris 10 on Opteron
• FreeBSD 4.10 - 6.0 on x86
• OpenBSD 3.1 to -current on i386, amd64, macppc, alpha, sparc64 and VAX (check the ports)

Highest performance should be achieved with haproxy versions newer than 1.2.5 running on Linux 2.6, or epoll-patched Linux kernel 2.4. It is only because of a very OS-specific optimization : the default polling system for version 1.1 is select(), which is common among most OSes, but can become slow when dealing with thousands of file-descriptors. Version 1.2 uses poll() by default instead of select(), but on some systems it may even be slower. However, it is recommended on Solaris as its implementation is rather good. On Linux 2.6, or patched Linux 2.4, HAProxy 1.2 will default to using epoll which achieves O(1) performance at any load, thus being even faster than poll() or select() on other systems. FreeBSD is known to slightly surpass this performance with its kqueue mechanism. Support for this poller has been implemented in HAProxy 1.3, but no benchmark has been done yet.

Based on those facts, people looking for a very fast load balancer should consider the following options on x86 or x86_64 hardware, in this order :

1. HAProxy 1.2.13.1+ on Linux 2.4 + epoll
2. HAProxy 1.2.13.1+ on Linux 2.6.16 + scheduler starvation fixes
3. HAProxy 1.2.8+ on FreeBSD
4. HAProxy 1.2.8+ on Solaris 10

Note: if you're looking for a very fast web load-balancing solution, the second fastest platform I ever tested was an HP-DL145 (Dual Opteron-248 at 2.2 GHz) running an epoll-patched Linux kernel 2.4 and haproxy-1.2 at 21000 hits/s and 2 Gbps. Very impressive indeed for such a cheap system ! The fastest one was a Sun V40Z (Quad Opteron-848 at 2.2 GHz) which showed 30000 hits/s on dual gigabit links. This is only 50% more than the dual-proc, so I guess we're really getting close to system limits. On the other side, certainly that a little system tuning can improve things greatly. Anyway, the Dual-Opteron still stays the best perf/price ratio.

Performance

Well, since a user's testimony is better than a long demonstration, please take a look at Chris Knight's experience with haproxy saturating a gigabit fiber on a video download site. Also, my experiments with Myricom's 10-Gig NICs might be of interest.

HAProxy involves several techniques commonly found in Operating Systems architectures to achieve the absolute maximal performance :

• a single-process, event-driven model considerably reduces the cost of context switch and the memory usage. Processing several hundreds of tasks in a millisecond is possible, and the memory usage is in the order of a few kilobytes per session while memory consumed in Apache-like models is more in the order of megabytes per process.
• O(1) event checker on systems that allow it (currently only Linux with HAProxy 1.2), allowing instantaneous detection of any event on any connection among tens of thousands.
• Single-buffering without any data copy between reads and writes whenever possible. This saves a lot of CPU cycles and useful memory bandwidth. Often, the bottleneck will be the I/O busses between the CPU and the network interfaces.
• MRU memory allocator using fixed size memory pools for immediate memory allocation favoring hot cache regions over cold cache ones. This dramatically reduces the time needed to create a new session.
• work factoring, such as multiple accept() at once, and the ability to limit the number of accept() per iteration when running in multi-process mode, so that the load is evenly distributed among processes.
• event aggregation : timing resolution is adapted to match the system scheduler's resolution. This allows many events to be processed at once without having to sleep when we're sure that we would have woken up immediately. This also leaves a large performance margin with virtually no degradation of response time when the CPU usage approaches 100%.
• two FSMs (finite state machines) per session ensure exact knowledge of client-side and server-side connection state without having to perform a lot of controls at each call to deal with errors. Actions are decided based on the combination of both states and events. For an even improved state-based decision, future versions will use 3 FSMs with a hard-coded transition matrix like what is used in hardware processors and ASICs.
• reduced footprint for frequently and randomly accessed memory areas such as the file descriptor table which uses 4 bitmaps. This reduces the number of CPU cache misses and memory prefetching time.
• optimized HTTP header analysis : headers are parsed an interpreted on the fly, and the parsing is optimized to avoid an re-reading of any previously read memory area. Checkpointing is used when an end of buffer is reached with an incomplete header, so that the parsing does not start again from the beginning when more data is read.
• careful reduction of the number of expensive system calls. Most of the work is done in user-space by default, such as time reading, buffer aggregation, file-descriptor enabling/disabling.

All these micro-optimizations result in very low CPU usage even on moderate loads. And even at very high loads, when CPU is saturated, it is quite common to note something like 5% user and 95% system, which means that the HAProxy process consumes about 20 times less than its system counterpart. This explains why the tuning of the Operating System is very important. I personnally build my own patched Linux 2.4 kernels, and finely tune a lot of network sysctls to get the most out of a reasonable machine.

This also explains why Layer 7 processing has little impact on performance : even if user-space work is doubled, the load distribution will look more like 10% user and 90% system, which means an effective loss of only about 5% of processing power. This is why on high-end systems, HAProxy's Layer 7 performance can easily surpass hardware load balancers' in which complex processing which cannot be performed by ASICs has to be performed by slow CPUs. Here is the result of a quick benchmark performed on haproxy 1.3.9 at EXOSEC on a single core Pentium 4 with PCI-Express interfaces:

• The session rate
  This factor is very important, because it directly determines when the load balancer will not be able to distribute all the requests it receives. It is mostly dependant on the CPU. Sometimes, you will hear about requests/s or hits/s, and they are the same as sessions/s in HTTP/1.0 or HTTP/1.1 with keep-alive disabled. Requests/s with keep-alive enabled does not mean anything, and is generally useless because it is very often that keep-alive has to be disabled to offload the servers under very high loads. This factor is measured with varying object sizes, the fastest results generally coming from empty objects (eg: HTTP 302, 304 or 404 response codes). Session rates above 20000 sessions/s can be achieved on Dual Opteron systems such as HP-DL145 running a carefully patched Linux-2.4 kernel. Even the cheapest Sun's X2100-M2 achieves 25000 sessions/s in dual-core 1.8 GHz configuration.
• The session concurrency
  This factor is tied to the previous one. Generally, the session rate will drop when the number of concurrent sessions increases (except the epoll polling mechanism). The slower the servers, the higher the number of concurrent sessions for a same session rate. If a load balancer receives 10000 sessions per second and the servers respond in 100 ms, then the load balancer will have 1000 concurrent sessions. This number is limited by the amount of memory and the amount of file-descriptors the system can handle. With 8 kB buffers, HAProxy will need about 16 kB per session, which results in around 60000 sessions per GB of RAM. In practise, socket buffers in the system also need some memory and 40000 sessions per GB of RAM is more reasonable. Layer 4 load balancers generally announce millions of simultaneous sessions because they don't process any data so they don't need any buffer. Moreover, they are sometimes designed to be used in Direct Server Return mode, in which the load balancer only sees forward traffic, and which forces it to keep the sessions for a long time after their end to avoid cutting sessions before they are closed.
• The data rate
  This factor generally is at the opposite of the session rate. It is measured in Megabytes/s (MB/s), or sometimes in Megabits/s (Mbps). Highest data rates are achieved with large objects to minimise the overhead caused by session setup and teardown. Large objects generally increase session concurrency, and high session concurrency with high data rate requires large amounts of memory to support large windows. High data rates burn a lot of CPU and bus cycles on software load balancers because the data has to be copied from the input interface to memory and then back to the output device. Hardware load balancers tend to directly switch packets from input port to output port for higher data rate, but cannot process them and sometimes fail to touch a header or a cookie. For reference, the Dual Opteron systems described above can saturate 2 Gigabit Ethernet links on large objects.

A load balancer's performance related to these factors is generally announced for the best case (eg: empty objects for session rate, large objects for data rate). This is not because of lack of honnesty from the vendors, but because it is not possible to tell exactly how it will behave in every combination. So when those 3 limits are known, the customer should be aware that he will generally be below all of them. A good rule of thumb on software load balancers is to consider an average practical performance of half of maximal session and data rates for average sized objects.

You might be interested in checking the 10-Gigabit/s page.

Reliability - keeping high-traffic sites online since 2002

Security - Not even one vulnerability in 5 years

HAProxy also provides regex-based header control. Parts of the request, as well as request and response headers can be denied, allowed, removed, rewritten, or added. This is commonly used to block dangerous requests or encodings (eg: the Apache Chunk exploit), and to prevent accidental information leak from the server to the client. Other features such as Cache-control checking ensure that no sensible information gets accidentely cached by an upstream proxy consecutively to a bug in the application server for example.

Download

• Development version (1.3 - content switching) :
  • Browse directory for docs, sources and binaries
  • Daily snapshots are built once a day when the GIT repository changes

• Latest version (1.2) :
  • Release Notes for version 1.2.18
  • haproxy-1.2.18.tar.gz (MD5) : Source code under GPL
  • haproxy-1.2.18-linux-i586.gz : (MD5) Linux/i586 executable linked with Glibc 2.2
  • haproxy-1.2.18-sol8-ultrasparc-static-pcre.gz : (MD5) Solaris8/Sparc executable
  • haproxy-1.2.18-sol10-i686-static-pcre.gz : (MD5) Solaris10/x86 executable
  • Browse directory for other files or versions

• Previous branch (1.1) :
  • Release Notes for version 1.1.34
  • haproxy-1.1.34.tar.gz (MD5) : Source code under GPL
  • haproxy-1.1.34-linux-i586.gz : Linux/i586 executable linked with Glibc 2.2
  • haproxy-1.1.34-sol8-ultrasparc-static-pcre.gz : Solaris8/Sparc executable
  • Browse directory for other files or versions

• X-Forwarded-For support for Stunnel

  Stunnel currently makes a perfect complement to provide SSL client-side support to HAProxy. However, since Stunnel is a proxy an has no knowledge of HTTP, the client's IP address was lost, which is somewhat annoying. A few patches were available on the Net to add the X-Forwarded-For header, but they introduced an undesirable buffer overflow. So I took my courage and wrote a reliable and secure patch to implement this useful feature. I sent it to Stunnel's authors but got no feedback. So the patch is provided here for Stunnel-4.14 and 4.15 in the hope it will be useful to some people.

  • stunnel-4.14-xforwarded-for.diff
  • stunnel-4.15-xforwarded-for.diff
  • Browse directory for other versions

• Various Patches :
  • epoll-lt-2.4.32-0.23.diff : kernel patch to enable epoll on standard Linux 2.4 kernels.
  • linux-2.4.21-40.EL-custom.diff : kernel patch to enable epoll on Red Hat Enterprise Linux 3 U7.
  • tcp_splice-0.1.1-linux-2.4.33.diff : patch to L7SW to provide TCP splicing support to standard Linux 2.4 kernels.
  • Browse directory for other patches

• Browsable directory

Documentation

• Reference Manual for version 1.3, not always up to date (development) :
  • configuration.txt : Configuration Manual (English)
  • architecture.txt : Architecture Guide (English)
  • haproxy-en.txt : English version
  • haproxy-fr.txt : French version
• Reference Manual for version 1.2 (stable) :
  • haproxy-en.txt : English version
  • haproxy-fr.txt : French version
• Reference Manual for version 1.1 (old stable) :
  • haproxy-en.txt : English version
  • haproxy-fr.txt : French version
• architecture.txt : Architecture Guide (English)
• Article on Load Balancing (HTML version) : worth reading for people who don't know what type of load balancer they need (English)
  • PDF version (English)
  • PDF version (French)

Commercial Support (France)

If you think you don't have the time and skills to setup and maintain a free load balancer, or if you're seeking for commercial support to satisfy your customers or your boss, you should contact EXOSEC. Another solution would be to use Exceliance's ALOHA appliances (see below).

Products using HAProxy

• redWall Firewall
  From the site : "redWall is a bootable CD-ROM Firewall. Its goal is to provide a feature rich firewall solution, with the main goal, to provide a webinterface for all the logfiles generated!"
• Exceliance's ALOHA appliance
  Exceliance sells a complete solution embedding an optimized and hardened version of Formilux packaged for ease of use, reduced maintenance, and enhanced availability through the use of VRRP for box fail-over, bonding for link fail-over, etc...

Contributions

• Application Cookies

  Alexander Lazic and Klaus Wagner implemented this feature which was merged in 1.2. It allows the proxy to learn cookies sent by the server to the client, and to find it back in the URL to direct the client to the right server. The learned cookies are automatically purged after some inactive time.

• FreeBSD Port

  Clément Laforet maintains an up to date port of haproxy for FreeBSD. If you want more information about this port, please consult Clément's work here :

  http://www.freebsd.org/cgi/url.cgi?ports/net/haproxy/pkg-descr
  http://www.freebsd.org/cgi/cvsweb.cgi/ports/net/haproxy/
  http://www.freshports.org/net/haproxy

• Debian/Ubuntu Package

  Arnaud Cornet has brought HAProxy to Debian :

  http://packages.debian.org/haproxy

• OpenBSD port

  Jason Dixon has committed a port of haproxy into OpenBSD -current

• Least Connections load balancing algorithm

  This patch for haproxy-1.2.14 was submitted by Oleksandr Krailo. It implements a basic least connection algorithm. I've not merged this version into 1.3 because of scalability concerns, but I'm leaving it here for people who are tempted to include it into version 1.2, and the patch is really clean.

  haproxy-1.2.14-leastconn.diff

• Soft Server-Stop

  Alexander Lazic sent me this patch against 1.1.28 which in fact does two things. The first interesting part allows one to write a file enumerating servers which will have to be stopped, and then sending a signal to the running proxy to tell it to re-read the file and stop using these servers. This will not be merged into mainline because it has indirect implications on security since the running process will have to access a file on the file-system, while current version can run in a chrooted, empty, read-only directory. What is really needed is a way to send commands to the running process. However, I understand that some people might need this feature, so it is provided here. The second part of the patch has been merged. It allowed both an active and a backup server to share a same cookie. This may sound obvious but it was not possible earlier.

  haproxy_comafile+multi-cookie.diff

  Usage: Alex says that you just have to write the server names that you want to stop in the file, then kill -USR2 the running process. I have not tested it though.

• Server Weight

  Sébastien Brize sent me this patch against 1.1.27 which adds the 'weight' option to a server to provide smoother balancing between fast and slow servers. It is available here because there may be other people looking for this feature in version 1.1.
  
  I did not include this change because it has a side effect that with high or unequal weights, some servers might receive lots of consecutive requests. A different concept to provide a smooth and fair balancing has been implemented in 1.2.12, which also supports weighted hash load balancing.

  patch-haproxy-1.1.27-weight

  Usage: specify "weight X" on a server line.
  Note: configurations written with this patch applied will normally still work with future 1.2 versions.

• IPv6 support for 1.1.27

  I implemented IPv6 support on client side for 1.1.27, and merged it into haproxy-1.2. Anyway, the patch is still provided here for people who want to experiment with IPv6 on HAProxy-1.1.

  haproxy-1.1.27-ipv6.diff

• Other patches

  Please browse the directory for other useful contributions.

Other Solutions

• Linux Virtual Servers (LVS)
  Very fast layer 3/4 load balancing merged in Linux 2.4 and 2.6 kernels. Should be coupled with Keepalived to monitor servers. This generally is the solution embedded by default in most IP-based load balancers.
• Pure Load Balancer (PLB)
  The author adopted the same event-driven model as in HAProxy (but relying on libevent). Interestingly, he has the same conclusions about other models's limitations. However, his goal is just to achieve high performance and availability, without any particular HTTP processing nor persistence.
• Pound
  Pound can be seen as a complement to HAProxy. It supports SSL, and can direct traffic according to the requested URL. Its code is very small and will stay small for easy auditing. Its configuration file is very small too. However, it does not support persistence, and the performance associated to its multi-threaded model limits its usage to medium sites only.
• Pen
  Pen is a very simple load balancer for TCP protocols. It supports source IP-based persistence for up to 2048 clients. Supports IP-based ACLs. Uses select() and supports higher loads than Pound but will not scale very well to thousands of simultaneous connections.

Contacts

• mailing-list :
  Read the archives on Formilux site
  Subscribe to the list :
• Main site : http://1wt.eu/
• This site in IPv6 : http://haproxy.ipv6.1wt.eu/ (should be OK if you see a green square here ⇒ )
• e-mail :

An IRC channel for haproxy has been opened on FreeNode (but don't seek me there, I'm not) :

irc://irc.gnu.org/%23haproxy