In short, a hit rate above 10000/s is sustained for objects
smaller than 6 kB, and the Gigabit/s is sustained for
objects larger than 40 kB.
In production, HAProxy has been installed several times as an emergency solution
when very expensive, high-end hardware load balancers suddenly failed on Layer 7
processing. Hardware load balancers process requests at the
packet level and have a great difficulty at supporting
requests across multiple packets and high response
times because they do no buffering at all. On the
other side, software load balancers use TCP buffering
and are insensible to long requests and high response times. A
nice side effect of HTTP buffering is that it
increases the server's connection acceptance by reducing the
session duration, which leaves room for new requests. New
benchmarks will be executed soon, and results will be
published. Depending on the hardware, expected rates are in the order of a few
tens of thousands of new connections/s with tens of thousands of simultaneous
connections.
There are 3 important factors used to measure a load balancer's performance :
- The session rate
This factor is very important, because it directly determines when the load
balancer will not be able to distribute all the requests it receives. It is
mostly dependant on the CPU.
Sometimes, you will hear about requests/s or hits/s, and they are the same as
sessions/s in HTTP/1.0 or HTTP/1.1 with
keep-alive disabled. Requests/s with keep-alive enabled does not mean
anything, and is generally useless because it is very often that keep-alive has
to be disabled to offload the servers under very high loads. This factor is
measured with varying object sizes, the fastest results generally coming from
empty objects (eg: HTTP 302, 304 or 404 response codes).
Session rates above 20000 sessions/s can be achieved on
Dual Opteron systems such as HP-DL145 running a carefully
patched Linux-2.4 kernel. Even the cheapest Sun's X2100-M2 achieves 25000 sessions/s in dual-core 1.8 GHz configuration.
- The session concurrency
This factor is tied to the previous one. Generally, the session rate
will drop when the number of concurrent sessions increases (except the
epoll polling mechanism). The slower the servers, the higher
the number of concurrent sessions for a same session rate. If a load balancer
receives 10000 sessions per second and the servers respond in 100 ms, then the
load balancer will have 1000 concurrent sessions. This number is limited by the
amount of memory and the amount of file-descriptors the system can
handle. With 8 kB buffers, HAProxy will need about 16 kB per session, which
results in around 60000 sessions per GB of RAM. In practise, socket
buffers in the system also need some memory and 40000 sessions per GB of RAM is
more reasonable. Layer 4 load balancers generally announce millions of
simultaneous sessions because they don't process any data so they don't need
any buffer. Moreover, they are sometimes designed to be used in Direct Server
Return mode, in which the load balancer only sees forward traffic, and which
forces it to keep the sessions for a long time after their end to avoid cutting
sessions before they are closed.
- The data rate
This factor generally is at the opposite of the session rate. It is measured
in Megabytes/s (MB/s), or sometimes in Megabits/s (Mbps). Highest data rates
are achieved with large objects to minimise the overhead caused by session
setup and teardown. Large objects generally increase session concurrency, and
high session concurrency with high data rate requires large amounts of memory
to support large windows. High data rates burn a lot of CPU and bus cycles on
software load balancers because the data has to be copied from the input
interface to memory and then back to the output device. Hardware load balancers
tend to directly switch packets from input port to output port for higher data
rate, but cannot process them and sometimes fail to touch a header or a cookie.
For reference, the Dual Opteron systems described above can saturate 2
Gigabit Ethernet links on large objects.
A load balancer's performance related to these factors is generally announced for
the best case (eg: empty objects for session rate, large objects for data rate).
This is not because of lack of honnesty from the vendors, but because it is not
possible to tell exactly how it will behave in every combination. So when those 3
limits are known, the customer should be aware that he will generally be below
all of them. A good rule of thumb on software load balancers is to consider an
average practical performance of half of maximal session and data rates for
average sized objects.
You might be interested in checking the 10-Gigabit/s page.
Being obsessed with reliability, I tried to do my best to ensure a total
continuity of service by design. It's more difficult to design something
reliable from the ground up in the short term, but in the long term it reveals
easier to maintain than broken code which tries to hide its own bugs behind
respawning processes and tricks like this.
In single-process programs, you have no right to fail : the smallest bug
will either crash your program, make it spin like mad or freeze. There has not
been any such bug found in the code nor in production for the last 5 years.
HAProxy has been installed on Linux 2.4 systems serving millions of pages
every day,
and which have only known one reboot in 3 years for a complete OS upgrade.
Obsiouvly, they were not directly exposed to the Internet because they did not receive any
patch at all. The kernel was a heavily patched 2.4 with Robert Love's
jiffies64 patches to support time wrap-around at 497 days (which
happened twice). On such systems, the software cannot fail without being
immediately noticed !
Right now, it's being used in several Fortune 500 companies around the world to
reliably serve millions of pages per day or relay huge amounts of money. Some
people even trust it so much that they use it as the default solution to solve
simple problems (and I often tell them that they do it the dirty way). Such
people still use version 1.1 which sees very limited evolutions and which targets
mission-critical usages. It is really suited for such environments because the
indicators it returns provide a lot of valuable information about the application's
health, behaviour and defects, which are used to make it even more reliable. Version 1.2 has not yet received as much testing as 1.1,
but should be considered for high volumes of traffic or to benefit from newest
features.
As previously explained, most of the work is executed by the Operating System.
For this reason, a large part of the reliability involves the OS itself. Recent
versions of Linux 2.4 offer a high level of stability. However, it requires a
bunch of patches to achieve a high level of performance. Linux 2.6
includes the features needed to achieve this level of performance, but is not
yet stable enough for such usages. The kernel needs at least one upgrade every
month to fix a bug or vulnerability. Some people prefer to run it on Solaris (or
do not have the choice). Solaris 8 and 9 are known to be really stable right now,
offering a level of performance comparable to Linux 2.4. Solaris 10 might show
performances closer to Linux 2.6, but with the same code stability problem. I
have too few reports from FreeBSD users, but it should be close to Linux 2.4 in
terms of performance and reliability. OpenBSD sometimes shows socket allocation
failures due to sockets staying in FIN_WAIT2 state when client suddenly
disappears. Also, I've noticed that hot reconfiguration does not work under
OpenBSD.
The reliability can significantly decrease when the system is pushed to its
limits. This is why finely tuning the sysctls is important. There is no
general rule, every system and every application will be specific. However, it is
important to ensure that the system will never run out of memory and
that it will never swap. A correctly tuned system must be able to run for
years at full load without slowing down nor crashing.
Security is an important concern when deploying a software load balancer. It is
possible to harden the OS, to limit the number of open ports and accessible
services, but the load balancer itself stays exposed. For this reason, I have been
very careful about programming style. The only vulnerability found so far dates 5
years and only lasted for one week. It was introduced when logs were reworked. It
could be used to cause BUS ERRORS to crash the process, but it did not
seem possible to execute code : the overflow concerned only 3 bytes, too short to
store a pointer (and there was a variable next).
Anyway, much care is taken when writing code to manipulate headers. Impossible
state combinations are checked and returned, and errors are processed from the
creation to the death of a session. A few people around the world have reviewed
the code and suggested cleanups for better clarity to ease auditing. By the way,
I'm used to refuse patches that introduce suspect processing or in which not
enough care is taken for abnormal conditions.
I generally suggest starting HAProxy as root because it
can then jail itself in a chroot and drop all of its privileges
before starting the instances. This is not possible if it is not started as
root because only root can execute chroot().
Logs provide a lot of information to help to maintain a satisfying security
level. They can only be sent over UDP because once chrooted, the
/dev/log UNIX socket is unreachable, and it must not be possible to
write to a file. The following information are particularly useful :
- source IP and port of requestor make it possible to find their origin
in firewall logs ;
- session set up date generally matches firewall logs, while tear
down date often matches proxies dates ;
- proper request encoding ensures the requestor cannot hide
non-printable characters, nor fool a terminal.
- arbitrary request and response header and cookie capture help to
detect scan attacks, proxies and infected hosts.
- timers help to differentiate hand-typed requests from browsers's.
HAProxy also provides regex-based header control. Parts of the request, as
well as request and response headers can be denied, allowed, removed, rewritten, or
added. This is commonly used to block dangerous requests or encodings (eg: the
Apache Chunk exploit),
and to prevent accidental information leak from the server to the client.
Other features such as Cache-control checking ensure that no sensible
information gets accidentely cached by an upstream proxy consecutively to a bug in
the application server for example.
The source code is covered by GPL v2. Source code and pre-compiled binaries for
Linux/x86 and Solaris/Sparc can be downloaded right here :
- Development version (1.3 - content switching) :
- Latest version (1.2) :
- Previous branch (1.1) :
- X-Forwarded-For support for Stunnel
Stunnel currently makes a perfect
complement to provide SSL client-side support to HAProxy. However, since
Stunnel is a proxy an has no knowledge of HTTP, the client's IP address was
lost, which is somewhat annoying. A few patches were available on the Net to
add the X-Forwarded-For header, but they introduced an undesirable buffer
overflow. So I took my courage and wrote a reliable and secure patch to
implement this useful feature. I sent it to Stunnel's authors but got no
feedback. So the patch is provided here for Stunnel-4.14 and 4.15 in the hope
it will be useful to some people.
- Various Patches :
- Browsable directory
There are three types of documentation now : the Reference Manual which explains
how to configure HAProxy, the Architecture Guide which will guide you
through various typical setups, and the new Configuration Manual which will soon
replace the Reference Manual with more a explicit configuration language explanation.
- Reference Manual for version 1.3, not always up to date (development) :
- Reference Manual for version 1.2 (stable) :
- Reference Manual for version 1.1 (old stable) :
architecture.txt : Architecture Guide (English)
- Article on Load Balancing (HTML version) : worth reading for people who don't
know what type of load balancer they need (English)
If you think you don't have the time and skills to setup and maintain a free load
balancer, or if you're seeking for commercial support to satisfy your customers or
your boss, you should contact
EXOSEC. Another solution would be
to use Exceliance's ALOHA appliances (see below).
The following products or projects use HAProxy :
- redWall Firewall
From the site : "redWall is a bootable CD-ROM Firewall. Its goal is to provide
a feature rich firewall solution, with the main goal, to provide a webinterface
for all the logfiles generated!"
- Exceliance's
ALOHA appliance
Exceliance sells a complete solution embedding an optimized and hardened version of
Formilux packaged for ease of
use, reduced maintenance, and enhanced availability through the use of VRRP for
box fail-over, bonding for link fail-over, etc...
Some happy users have contributed code which may or may not be included. Others
spent a long time analysing the code, and there are some who maintain ports up to
date.
- Application Cookies
Alexander Lazic and Klaus Wagner implemented this feature which
was merged in 1.2. It allows the proxy to learn cookies sent by the server
to the client, and to find it back in the URL to direct the client to the right
server. The learned cookies are automatically purged after some inactive time.
- FreeBSD Port
Clément Laforet maintains an up to date port of haproxy for FreeBSD. If you want
more information about this port, please consult Clément's work here :
- Debian/Ubuntu Package
Arnaud Cornet has brought HAProxy to
Debian :
- Least Connections load balancing algorithm
This patch for haproxy-1.2.14 was submitted by Oleksandr Krailo. It implements
a basic least connection algorithm. I'm not really tempted to merge it into 1.2
right now, in order to limit the number of changes, but it might get merged
early into 1.3. Please note that this is not very useful now that the maxconn
can be set per server, but it might be helpful to some people, and the patch
is really clean.
- Soft Server-Stop
Alexander Lazic sent me this patch against 1.1.28 which in fact does two things.
The first interesting part allows one to write a file enumerating servers which
will have to be stopped, and then sending a signal to the running proxy to tell
it to re-read the file and stop using these servers. This will not be merged into
mainline because it has indirect implications on security since the running
process will have to access a file on the file-system, while current version can
run in a chrooted, empty, read-only directory. What is really needed is a way to
send commands to the running process. However, I understand that some people
might need this feature, so it is provided here. The second part of the patch has
been merged. It allowed both an active and a backup server to share a same
cookie. This may sound obvious but it was not possible earlier.
Usage: Alex says that you just have to write the server names that you
want to stop in the file, then kill -USR2 the running process. I have
not tested it though.
- Server Weight
Sébastien Brize sent me this patch against 1.1.27 which adds the
'weight' option to a server to provide smoother balancing between fast and slow
servers. It is available here because there may be other people looking for this
feature in version 1.1.
I did not include this change because it has a side effect that with
high or unequal weights, some servers might receive lots of consecutive
requests. A different concept to provide a smooth and fair
balancing has been implemented in 1.2.12, which also supports
weighted hash load balancing.
Usage: specify "weight X" on a server line.
Note: configurations written with this patch applied will normally still
work with future 1.2 versions.
- IPv6 support for 1.1.27
I implemented IPv6 support on client side for 1.1.27, and merged it into
haproxy-1.2. Anyway, the patch is still provided here for people who want to
experiment with IPv6 on HAProxy-1.1.
- Other patches
Please browse the directory for other useful
contributions.
If you don't need all of HAProxy's features and are looking for a simpler solution,
you may find what you need here :
-
Linux Virtual Servers (LVS)
Very fast layer 3/4 load balancing merged in Linux 2.4 and 2.6 kernels. Should
be coupled with Keepalived to monitor
servers. This generally is the solution embedded by default in most
IP-based load balancers.
-
Pure Load Balancer (PLB)
The author adopted the same event-driven model as in HAProxy (but relying on
libevent). Interestingly, he
has the same conclusions about other models's limitations. However, his goal is
just to achieve high performance and availability, without any particular HTTP
processing nor persistence.
-
Pound
Pound can be seen as a complement to HAProxy. It supports SSL, and can direct
traffic according to the requested URL. Its code is very small and will stay
small for easy auditing. Its configuration file is very small too. However, it
does not support persistence, and the performance associated to its
multi-threaded model limits its usage to medium sites only.
-
Pen
Pen is a very simple load balancer for TCP protocols. It supports source IP-based
persistence for up to 2048 clients. Supports IP-based ACLs. Uses
select() and supports higher loads than Pound but will not scale very
well to thousands of simultaneous connections.
Feel free to contact me at for any questions or comments :
An IRC channel for haproxy has been opened on FreeNode (but don't seek me there, I'm not) :
)