Background
I was writing code for a protocol not unlike the Simple Certificate Enrollment Protocol (SCEP; RFC 8894). It involves, like SCEP, an HTTP request in order to obtain a certificate. One of the features I wanted the protocol to have is validation beyond pre‐shared credentials. My idea was: “Oh, I know! Let’s validate that the requested DNS name in the certificate actually resolves to the origin IP address!” There are various reasons why this is typically not a good idea. It happened to be convenient for me as a second layer of authentication in my specific scenario.
Alas, as it turns out, getting the origin IP address is difficult to achieve in HTTP. At its core, HTTP is a stateless request/response protocol allowing proxies on the requesting and responding side. In my deployment model, client‐side proxies were of no interest. I could be sure clients would be connecting directly. But I did want to keep my options open to place my application behind a reverse proxy. In that case, the IP address my application server gets the request from is not the IP address of the originating client. It is instead the IP address of the reverse proxy.
The root of the problem comes down to HTTP having no way to signal trust. A client can, by definition, send any HTTP header they want. In a client‐to‐server scenario with no reverse proxy in‐between, this is no issue. The server knows not to trust any client‐side headers. Once reverse proxies enter the picture, however, a situation arises wherein the application server may wish to depend on information collected by a trusted reverse proxy. HTTP does not allow for any sensible indication thereof. So now what?
The fragmented status quo (for over a decade)
As of the time of writing, the status quo is a mishmash of several, competing solutions.
2.1 X‐Real‐IP and similar
The most straightforward of these solutions is the X‐Real‐IP header. Its entire value is the textual representation of the IP address of the original client. The first reverse proxy in a chain of HTTP servers writes the X‐Real‐IP header and discards any existing client‐ provided value for it; the other reverse proxies in the chain need to know that they need to preserve the header. Since there is no formal specification, it is also unclear what textual representations of IP addresses are permissible. This construction is particularly fragile to changes in the deployment of infrastructure. Assume the following:
client ‐> proxy ‐> applicationWhat happens if another reverse proxy in front of “proxy” is added? The value of X‐Real‐IP is replaced with the outgoing IP address of the new reverse proxy. In that case, the new reverse proxy needs to be able to handle X‐Real‐IP in the same manner as already described. If it passes a client‐provided X‐Real‐IP without verification to the proxies further back in the chain, spoofing becomes possible. This means that X‐Real‐IP is only a reasonable option if the first reverse proxy is well‐behaved while all other reverse proxies and the application server then trust at least this header is kept intact. This may not necessarily be the case.
Incidentally, Cloudflare also provides CF‐Connecting‐IP with slightly different semantics for intra‐Cloudflare infrastructure.
2.2 X‐Forwarded‐For
The X‐Forwarded‐For header takes a different approach. The first reverse proxy in the chain adds or overwrites a client‐provided X‐ Forwarded‐For header. It begins with just an IP address. Every reverse proxy thereafter strictly appends a comma, a space and the IP address of the requesting host. This makes obtaining the origin IP address straightforward: Split on comma, take the leftmost value, strip whitespace. It also makes transparent the route through which a request has passed. X‐Forwarded‐For is strictly an improvement over X‐Real‐IP. It does not, however, address the issue of spoofing; the first reverse proxy in the chain still needs to be well‐behaved. As with X‐Real‐IP, there is no formal reference documentation, leaving critical details unspecified. For example, some (but not all) reverse proxies will also add the client port, leading to bracketed IPv6 addresses with port number. There are other, related headers such as X‐Forwarded‐By and X‐Forwarded‐Proto.
2.3 Forwarded (RFC 7239)
The IETF, more specifically the ART Area General Applications Working Group, saw these informal standards and embarked upon a quest to provide the one, true specification to rule them all. Result: There are now n + 1 ways to pass information between reverse proxies.
Roadblocks in the adoption of RFC 7239
RFC 7239 suffers from one critical flaw: It is hideously difficult to parse. With how simple X‐Forwarded‐For is, it seems generally safe to assume if the parsing process fails, something went so wrong that there is likely cause for concern. With Forwarded, parsing will lead to complex code. It also straight up omits what to do if a server gets passed an invalid header. Given the difficulties in parsing it, this seems surprising. Allow me to demonstrate the examples in § 4 of RFC 7329.
Forwarded: for="_gazonk"
Starting off strong: There are double quotes. This already introduces complexity to ensure the quotes are closed. Furthermore, _gazonk is not an IP address. This is a feature specified in § 6.3 of RFC 7329. It is an “obfuscated identifier”, intended to hide the IP addresses.
Forwarded: For="[2001:db8:cafe::17]:4711"
Continuing, we now introduce case‐insensitivity for the “for=”, bracketed IPv6 addresses and port numbers. The port number may indeed be valuable information.
Forwarded: for=192.0.2.60;proto=http;by=203.0.113.43
Okay, so now every hop can have additional information with different keys, such as “proto” and “by”, delimited by semicolons. But there are no quotation marks! So now the parser cannot rely on quotation marks always being used, which now starts to strongly push implementers towards a properly implemented, non‐trivial parser.
Forwarded: for=192.0.2.43, for=198.51.100.17
Finally, we have the only example that looks actually reasonable to the untrained eye.
The document then proceeds to deliver the fatal blow in § 7.1:
Forwarded: for=192.0.2.43 Forwarded: for="[2001:db8:cafe::17]", for=unknown
being equivalent to
Forwarded: for=192.0.2.43,for="[2001:db8:cafe::17]",for=unknown
at which point, whitespace handling also comes into play and so does handling of duplicate header fields (“HTTP lists”). But watch out! Whitespace is not allowed everywhere. Did you notice the “unknown” there? This is also allowed; it’s a special value that indicates “the identity of the preceding entity is not known, but the proxy server still wants to signal that a forwarding of the request was made” (§ 6.2).
Unsurprisingly, adoption has been slow on the uptake. The Go project, for example, has been dragging their feet for seven years now, understandably so (https://github.com/golang/go/issues/30963).
Contextualizing the landscape
The officially blessed solution of the Forwarded header hasn’t taken off. While it has some nice security features (such as obfuscated hosts), the ensuing parsing nightmare with unspecified behavior on invalid values has rightfully kept it from adoption. In my opinion, the best option would be to formalize X‐Forwarded‐For with well‐ specified semantics, carefully keeping in line with existing implementations that form the living informal specification. I’ve found that this typically leads to the best results. Even if the protocols standardized in this manner may have warts, it is better for everyone to implement the same mediocre protocol in an interoperable fashion than for nobody to implement a better one.
The Forwarded header is also a valuable case study in how driving up implementation complexity can inhibit adoption. Tokenizers and parsers are typically code that few programmers are equipped to write and subsequently a common source of bugs. The lack of adoption also appears to highlight the importance of making a protocol be easy to parse and interact with. This outcome also stresses that it is important for a specification for programs that do not exist in a vacuum or are in a single entity’s complete control to specify error handling and recovery in a pragmatic, realistic manner.
Quite frankly, I would also like to relay some stern words to the people who came up with colons in the textual representation of IPv6 addresses and the corresponding hack of bracketed IPv6 to work around colons already being used for port numbers in URIs.
Finally, every reverse proxy in the chain should be aware of how many proxies ought to be in front of it. That way, the amount of hops in X‐Forwarded‐For or Forwarded can be validated. At the very least, an application server needs to know if it has a reverse proxy in front at all. How else would it know whether to trust an HTTP header or to rely on the actual IP address of the client?