i Load balancing on layer 7( HTTP) – All things in moderation

Load balancing on layer 7( HTTP)

Load balancer introduction

If is the first time you heard about Load balancer you should read my previous post here. In this post, I’m going to show how to setup a simple Load balancer(http) with Nginx and HAProxy, some problems when we implement it for a system and about load balancer security.We all have already some great resouces talk about that, I just combine them and hope it helpfull for you to imagine a huge picture.

Installing Nginx, HAProxy

Installing Nginx

Ubuntu:

apt-get install nginx  

Installing HAProxy

To you are using debian or Ubuntu you can choose a package suitable for your systems in here. In my case I’m using ubuntu 14.04 and I want to install latest HAProxy(1.7 as of this writing):

apt-get install software-properties-common
add-apt-repository ppa:vbernat/haproxy-1.7  
apt-get update
apt-get install haproxy

For setup HAProxy on Centos you can install from default repositories:

yum -y install haproxy

Confirm HAProxy installed:

[email protected]:~$ haproxy -v
HA-Proxy version 1.7.5-1ppa1~trusty 2017/04/06
Copyright 2000-2017 Willy Tarreau <[email protected]>

Now we’re done install HAProxy.

How to configure Nginx,Haproxy load balancing

About HTTP load balancer we will have 2 common choices nginx or haproxy. Both of them are great and it depend your sytem you want to build. So now let’s start with configuration examples you can easy to create them to test using one of tools Virtualbox/Vargrant/Docker.

Nginx Load balancing configuration exmaple

In Nginx version > 1.10 go to edit file /etc/nginx.conf

sudo vim /etc/nginx/nginx.conf  

The simplest configuration for load balancing with nginx may look like the following:

http {
    upstream back-end {
        server srv1.example.com;
        server srv2.example.com;
        server srv3.example.com;
    }

    server {
        # load balancing server port
        listen 80;

        location / {
            # point to upstream  
            proxy_pass http://back-end;
        }
    }
}
  • To using NGINX with a group of servers, first, you need to define the group with the upstream directive. The directive is placed in the http context. In upsteam block config are servers we want to point, also add port if the sever not using port 80 default like following :
  upstream back-end {
        server srv1.example.com:9000;
    }

When the load balancing method is not specifically configured, it defaults to round-robin, we can using others method like : ip_hash, least_conn.
For exmaple:

  upstream back-end {
        ip_hash;
        server srv1.example.com:9000;
    }

To keep it fast some options(Weighted load balancing, Health checks) I’m not mention here.Read the next part to find out more.
After that, Save file nginx.conf and restart nginx

sudo service nginx restart

Haproxy load balancing configuration example

HAProxy configuration at **/etc/haproxy/haproy.cfg The defautl configuration look like:

global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    # Default ciphers to use on SSL-enabled listening sockets.
    # For more information, see ciphers(1SSL).
    ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5000
    timeout client  50000
    timeout server  50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

Within global section, we may won’t need to make any changes.
Within defauls section we have log, time out options, and load balacner mode ( tcp/http)
Load balancing configuration
* frontend – where HAProxy listens to connections
* backend – where HAProxy sends incoming connections
* starts – Optionally, HAProxy web tool for monitoring the load balaver and its nodes
For an example look like:

global  
    log /dev/log    local0  
    log /dev/log    local1 notice  
    chroot /var/lib/haproxy  
    stats socket /run/haproxy/admin.sock mode 660 level admin    
    stats timeout 30s  
    user haproxy  
    group haproxy  
    daemon  

    #Default SSL material locations  
    ca-base /etc/ssl/certs  
    crt-base /etc/ssl/private  

    #Default ciphers to use on SSL-enabled listening sockets.  
    #For more information, see ciphers(1SSL).  
    ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL  

defaults  
    log     global  
    mode    http  
    option  httplog  
    option  dontlognull  
    timeout connect 5000  
    timeout client  50000  
    timeout server  50000  
    errorfile 400 /etc/haproxy/errors/400.http  
    errorfile 403 /etc/haproxy/errors/403.http  
    errorfile 408 /etc/haproxy/errors/408.http  
    errorfile 500 /etc/haproxy/errors/500.http  
    errorfile 502 /etc/haproxy/errors/502.http  
    errorfile 503 /etc/haproxy/errors/503.http  
    errorfile 504 /etc/haproxy/errors/504.http  
frontend localnodes  
    bind *:80  
    mode http  
    default_backend nodes  

backend nodes  
    mode http  
    balance roundrobin  
    option forwardfor  
    http-request set-header X-Forwarded-Port %[dst_port]  
    http-request add-header X-Forwarded-Proto https if { ssl_fc }  
    option httpchk HEAD / HTTP/1.1\r\nHost:localhost  
    server web01 172.17.0.2 check  
    server web02 172.17.0.3 check  
    server web03 172.17.0.4 check  
listen stats *:1936  
    stats enable  
    stats uri /  
    stats hide-version  
    stats auth someuser:password    

Save and test:

    sudo service haproxy restart  

Make sure you have already servers are running and index file of each server modifed to identify which is server 01 or 02.
Nginx load balancing configuration example:

For more infomation about argument you can visit here.

Load banlancing issues

Asset management

  • Static assets(images,JS,CSS) must the same acrocss servers. The main problem here is keep statics assets are succesfully replicated and are the same on each server.
  • You might also benefit from using a CDN(content delivery network) like Coudflare, MaxCDN or Coudfront. Their cache your static assets, reducing the load on your server.
  • If you allow user upload, the uploaded file only storage on 1 server but other servers will be not found that file. A common solution is host all static assets in a sperate location, such as Amazon’s S3. We call it is Center File Storage

Sessions

Session information if often saved on a temporary location within a web server. A user may log in, and have a session on one web server. Howerver, the load balancer may route user to another web server which doesn’t have that session information.to the user, it appers that they are no longer logged in. There are three common fixes for this.
* Cookie-based sessions
The session data are not saved to the server or any other storage, but are instead within the browser’s cookie. This way is not recommend because it’s not secure when you store data in client.
* Sticky Sessions
Also called “session affinity”. This will track which server a client was routed to and always route their request to the same web server on subsequent requests.
For example below show how to configure sticky session:
HAProxy Session Affinity:

backend nodes  
    # Other options above omitted for brevity  
    cookie SRV_ID prefix  
    server web01 127.0.0.1:9000 cookie check  
    server web02 127.0.0.1:9001 cookie check  
    server web03 127.0.0.1:9002 cookie check  

Nginx Session Affinity:

upstream app_example {
    ip_hash;                 
    server 127.0.0.1:9000;
    server 127.0.0.1:9001;
    server 127.0.0.1:9002;
}
  • Central session storage
    Session storage is typically centralized withing an in-memory stores such as Redis or Memcached. Presistent stores such as a database are also commonly used.
    Using a cache(Redis,Memcached) for session-storage lets all the web servers connect to a central session store, growing your infrastructure a bit, but leting your load load be truly distributed across all web nodes.

Lost Client Information

The problem here is if the load balacner is a proxy to your web application, it might appear to your application every request is coming from the load balancer!
If logs of HTTP request are stored form your web nodes, then you may lose important information needed when auditing your logs.
Fortunately, most load balancers provide a mechanism for giving your web servers and application this information. If you inspect the headers of a request reveived from a load balancer, you might see these included:
* X-Forwarded-For
* X-Forwarded-Host
* X-Forwared-Proto / X-Forwarded-Scheme
* X-Forwarded-Port
These header can tell you the client’s IP address, the hostname used to access the application, the schema used(HTTP vs HTTPS) and which port the client made the request on.

//  JSON representation of headers sent from a load balancer
{"host":"example.com",
"cache-control":"max-age=0",
"accept":"text/html",
"accept-encoding":"gzip,deflate,sdch",
"x-forwarded-port":"80",                   // An x-forwarded-port header!
"x-forwarded-for":"172.17.42.1"}     // An x-forwarded-for header!

SSL Traffic

In a load balancer evriroment, SSL traffic is often decrypted at the load balancer. Howerver, there’s actually a few ways to handle SSL traffic when using a load balancer
* SSL Termination
When the load balancer is responible for decrypting SSL traffic before passing request on, it’s referred to as “SSL Termination”.
The downside of SSL Termination is that the traffic between the load balancers and the web servers is not encrypted. This leaves the application open to possible man-in-the-middle attacks.
Howerver, this is a risk usally mitigated by the fact that load balancers are often within the same infrastructure(data center) as the web servers.
HAProxy SSL Termination:

frontend localhost
    bind *:80
    bind *:443 ssl crt /etc/ssl/xip.io/xip.io.pem
    mode http
    default_backend nodes

backend nodes
    mode http
    balance roundrobin
    option forwardfor
    option httpchk HEAD / HTTP/1.1\r\nHost:localhost
    server web01 172.17.0.3:9000 check
    server web02 172.17.0.3:9001 check
    server web03 172.17.0.3:9002 check
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }  
  • SSL Pass-through
    In this case, the load balacner doesn’t decrypt the request, but instead passes the request though to a web server. The server than must decrypt it.
    SSL Pass-through is only supported by load balancers traffic at TCP level rather than HTTP level.
    That rules out Nginx for SSL pass-though, but HAProxy will happily accomplish this for you!
    HAProxy SSL Pass-Through
frontend localhost
    bind *:80
    bind *:443
    option tcplog
    mode tcp
    default_backend nodes

backend nodes
    mode tcp
    balance roundrobin
    option ssl-hello-chk
    server web01 172.17.0.3:443 check
    server web02 172.17.0.4:443 check

Logs

Now, we have multiple web servers, but each one generates their own log files! It’s terrrible if you need to audit logs on each servers. centralizing your logs can be very beneficial.
You may wish to just get logs form the load balancer, skipping the web server logs.
The simplest ways this is to combine Logrotate’s functionality with an uploaded to an S3 bucket. This at least puts all the log files in one place that you can look into.
However, there’s plenty of centralized logging servers that you can install in your infrastructure or purchase.
Some popular self-install loggers:
* Logstash a part of popular ELK Stack
* Graylog2
* Splunk

Fault-torelant

What happen if we have problem with load balancer server ?
Here is the problem we need consider. The solution is we need use more than one load balancer server and connect them together, whenever one of servers down because any reasons your systems still working.

The post is quite long now, and the last part about Identifying Load Balancers in Penetration Testing I’ll write it after.

References

https://serversforhackers.com/so-you-got-yourself-a-loadbalancer

https://www.nginx.com/resources/admin-guide/load-balancer/
https://www.youtube.com/watch?v=mIOw4a34LCk

Leave a Reply