Today I became aware of the proxy protocol.

The Proxy Protocol was designed to chain proxies / reverse-proxies without losing the client information.

If you are proxying an HTTP(S) server, chances are that you have used the X-Forwarded-From header to keep the real remote address of the client making the request and not receving the proxy’s address instead. But this only works for HTTP(S): if you are proxying any other kind of TCP service, you are doomed.

Take for instance the following example: we will have a simple TCP server that echo backs the client’s remote address:

package main

import (
	"fmt"
	"io/ioutil"
	"log"
	"net"
)

func main() {
	ln, err := net.Listen("tcp", ":7654")
	if err != nil {
		log.Fatal(err)
	}

	for {
		cn, err := ln.Accept()
		if err != nil {
			log.Println("ln.Accept():", err)
			continue
		}

		go handle(cn)
	}
}

func handle(cn net.Conn) {
	defer func() {
		if err := cn.Close(); err != nil {
			log.Println("cn.Close():", err)
		}
	}()

	log.Println("handling connection from", cn.RemoteAddr())

	fmt.Fprintf(cn, "Your remote address is %v\n", cn.RemoteAddr())

	data, err := ioutil.ReadAll(cn)
	if err != nil {
		log.Println("reading from client:", err)
	} else {
		log.Printf("client sent %d bytes: %q", len(data), data)
	}
}

I’m running go run server.go in a machine whose IP is 192.168.1.20, and I’ll be sending requests from another machine whose IP is 192.168.1.12. One the server machine I’m also running an https://www.haproxy.org/ server that acts as a proxy to the Go program above:

global
    debug
    maxconn 4000
    log 127.0.0.1 local0

defaults
    timeout connect 10s
    timeout client  1m
    timeout server  1m

listen wo-send-proxy
    mode tcp
    log global
    option tcplog
    bind *:17654
    server app1 192.168.1.20:7654

listen w-send-proxy
    mode tcp
    log global
    option tcplog
    bind *:27654
    server app1 192.168.1.20:7654 send-proxy

This configuration creates 2 proxies: one listening on port 17654 which just proxies the client connection to the server, and another proxy listening in port 276564 which does the same but it also enables using the proxy protocol by using the send-proxy keyword.

On the client machine, I’m running the following to send requests directly to the Go server, via the regular proxy and via the proxy with proxy protocol enabled:

$ for port in {,1,2}7654; do echo inkel | nc 192.168.1.20 ${port}; done
Your remote address is 192.168.1.12:44966
Your remote address is 192.168.1.20:57680
Your remote address is 192.168.1.20:57681

As you can see in the first case the client is informed that its remote address is 192.168.1.12, which is correct, but in both the other cases it says 192.168.1.20, which is the address of the proxy. Let’s check what the server has to say in its output:

$ go run server.go
2017/10/13 11:50:54 handling connection from 192.168.1.12:44966
2017/10/13 11:50:54 client sent 6 bytes: "inkel\n"
2017/10/13 11:50:54 handling connection from 192.168.1.20:57680
2017/10/13 11:50:54 client sent 6 bytes: "inkel\n"
2017/10/13 11:50:54 handling connection from 192.168.1.20:57681
2017/10/13 11:50:54 client sent 56 bytes: "PROXY TCP4 192.168.1.12 192.168.1.20 58472 27654\r\ninkel\n"

Here something interesting happens: the first connection, the one made directly to the Go server, properly shows the remote address as 192.168.1.12 and the contents. The second and third ones incorrectly report the remote address as 192.168.1.20 but the third one shows something interesting in what was received from the client: instead of just receiving inkel it first received PROXY TCP4 192.168.1.12 192.168.1.20 58472 27654\r\n. This is what proxy protocol does, and if you see clearly, the client’s actual IP address is there!

The proxy protocol, when enabled, will send the following initial line to the proxied server:

PROXY <inet protocol> <client IP> <proxy IP> <client port> <proxy port>\r\n

The actual specification is fairly simple, and now we can see why the only condition for proxy protocol to work is that both endpoints of the connection MUST be compatible with proxy protocol.

This explains why the Go server isn’t reporting the right remote address, even when proxy protocol is used: the net package doesn’t (currently) supports proxy protocol. But adding support to it isn’t too difficult. Here we have a custom connection type that complies with the net.Conn interface:

type myConn struct {
	cn      net.Conn
	r       *bufio.Reader
	local   net.Addr
	remote  net.Addr
	proxied bool
}

func NewProxyConn(cn net.Conn) (net.Conn, error) {
	c := &myConn{cn: cn, r: bufio.NewReader(cn)}
	if err := c.Init(); err != nil {
		return nil, err
	}
	return c, nil
}

func (c *myConn) Close() error                { return c.cn.Close() }
func (c *myConn) Write(b []byte) (int, error) { return c.cn.Write(b) }

func (c *myConn) SetDeadline(t time.Time) error      { return c.cn.SetDeadline(t) }
func (c *myConn) SetReadDeadline(t time.Time) error  { return c.cn.SetReadDeadline(t) }
func (c *myConn) SetWriteDeadline(t time.Time) error { return c.cn.SetWriteDeadline(t) }

func (c *myConn) LocalAddr() net.Addr  { return c.local }
func (c *myConn) RemoteAddr() net.Addr { return c.remote }

func (c *myConn) Read(b []byte) (int, error) { return c.r.Read(b) }

func (c *myConn) Init() error {
	buf, err := c.r.Peek(5)
	if err != io.EOF && err != nil {
		return err
	}

	if err == nil && bytes.Equal([]byte(`PROXY`), buf) {
		c.proxied = true
		proxyLine, err := c.r.ReadString('\n')
		if err != nil {
			return err
		}
		fields := strings.Fields(proxyLine)
		c.remote = &addr{net.JoinHostPort(fields[2], fields[4])}
		c.local = &addr{net.JoinHostPort(fields[3], fields[5])}
	} else {
		c.local = c.cn.LocalAddr()
		c.remote = c.cn.RemoteAddr()
	}

	return nil
}

func (c *myConn) String() string {
	if c.proxied {
		return fmt.Sprintf("proxied connection %v", c.cn)
	}
	return fmt.Sprintf("%v", c.cn)
}

type addr struct{ hp string }

func (a addr) Network() string { return "tcp" }
func (a addr) String() string  { return a.hp }

Now in our server we wrap the connection into our new type, and pass it to the handle func:

func main() {
	ln, err := net.Listen("tcp", ":7654")
	if err != nil {
		log.Fatal(err)
	}

	for {
		cn, err := ln.Accept()
		if err != nil {
			log.Println("ln.Accept():", err)
			continue
		}

		pcn, err := NewProxyConn(cn)

		if err != nil {
			log.Println("NewProxyConn():", err)
			continue
		}

		go handle(pcn)
	}
}

With this, now we see the right output in both the client:

$ for port in {,1,2}7654; do echo inkel | nc 192.168.1.20 ${port}; done
Your remote address is 192.168.1.12:45050
Your remote address is 192.168.1.20:60729
Your remote address is 192.168.1.12:58556

…and in the server:

2017/10/13 13:37:45 accepted connection from 192.168.1.12:45056
2017/10/13 13:37:45 client sent 6 bytes: "inkel\n"
2017/10/13 13:37:45 accepted connection from 192.168.1.20:60738
2017/10/13 13:37:45 client sent 6 bytes: "inkel\n"
2017/10/13 13:37:45 accepted connection from 192.168.1.12:58562
2017/10/13 13:37:45 client sent 6 bytes: "inkel\n"

This has been turned into a Go library located at github.com/inkel/go-proxy-protocol. Feel free to use it and send your feedback and error reports!