More SSH servers behind the same hostname
Say you have an SSH server on your premises. And say you have a huge load on that service.
Example? Sure. Git repository. Any git ‘server’ is such a use-case.
SSH acronym means
The Secure Shell. More precisely, we’ll be talking about SSH2 protocol, which is
a de-facto standard. This protocol is covered by the RFC 4253.
Barely (as simply as possible), the connection is done as follows. A client opens new TCP connection
to the server. It checks server public key(s) (using the
known_hosts file). Then server
and client do key exchange, authentication and open channels to run commands or
More precisely, SSH protocol runs on top of TCP connection. This protocol is secure. We need to load balance TCP connections, aka L4 level load balancing.
DNS Load Balancing
We need to have several SSH server running. All IP addresses are registered in the DNS for the same hostname. The DNS round-robin will make it play as load balancing.
The problem here is in the DNS update latency. DNS is cached all over the place. And we are not able to have more (or less) servers fast. In practice, it may take days to propagate the update.
Also, this approach requires as much IP addresses as we have SSH servers.
The configuration is quite simple.
This configuration is enough to make HAProxy run in the L4 mode to route traffic from the main service IP(s) to all backends.
Running several HAProxy instances behind different IP addresses is possible for redundancy and throughput.
SSH Server Server Keys
It is necessary to have same server keys on all
SERVER_N family, otherwise an SSH client
will warn a possible MIIM attach, as client likely to access any SSH server from the same IP. And the
file stores server public key info for each known IP.
TCP Balancing Issues: Request IP
There are several differences between an HTTP-level (L7) balancing and TCP-level balancing. You are not able
to send request IP address that easy as via HTTP. The protocol is only about streams. There is no notion of headers,
where one is able to pass extra data, e.g. as in HTTP with
There is no solution in general, but one can use HAProxy PROXY protocol. That is an extension to any TCP protocol that blindly sends a specific string with request/response information to the server. One has to have a server, supporting PROXY protocol on the other end.
The PROXY protocol is used in a number of places and services, e.g. AWS ELB, HAProxy, Nginx, and much mode.
Bad news. Open SSH server implementation does not support it out of the box, but one may apply patches to support that. I created and shared patches for Apache Mina SSHD library.
To enable the support, add
send-proxy each server line. You may replace the line
HAProxy supports automated health checks on TCP level. Ones a backend server is not able to reply on checks, the traffic is no longer send to an unhealthy host.
I use the following configuration for that
tcp-check expect string SSH-2.0-. The option instructs HAProxy
to run checks and to check if the remote end replies with a string that starts with
SSH-2.0-. Such answer is
the standard message required by the SSH2 protocol.
Overall we have the following configuration:
We use this approach mainly to implement Green-Blue deployments. We route traffic to the active host. We can change the configuration dynamically or even have custom weights to implement A/B tests.
An alternative is to use DNS. But with DNS you never know how much time will it take for client’s and chain of DNS servers to propagate the change. In the real world, it looks the best to combine DNS as more slow and static entry level balancing. Next to use HAProxy (or similar) balancing to implement fast configuration changes on the fly.
I showed how to easily load-balance SSH servers for fun and profit. Hope this helps you to build a highly available system.comments powered by Disqus