So You Want a Red Team Exercise?

I originally wrote this for work, where we get a lot of requests to “Red Team” something. In a lot of these cases, a white box security review or other form of security testing is more appropriate. Because I’d heard through the grapevine that other Red Teams struggle with the same issues, I wanted to make it available publicly. Thanks to my management for their support and permission to take this public!

If you’d like to use or adapt this within your organization, feel free, but please give credit to the Google Red Team.


We frequently get requests to perform Red Team engagements on various products & services around our company. These requests often have misconceptions about the services our team provides. This document is intended to help those seeking a Red Team engagement have a better understanding of what we do, how we do it, and why we do it the way we do, and how to engage with us for optimal effectiveness.


Security 101: Learning From Home

Outside, there’s a pandemic. We’re being asked to stay indoors, shelter in place, and avoid social contact. Conferences are cancelled, live trainings are out of the question. Some businesses are closing (hopefully temporarily) and there are unfortunate layoffs and furloughs across the board. It’s a tough time.

Rather than dwell on the negative, focusing on something else can help you get through this mentally. Learning something or growing your skills can both help take away from the anxiety of the situation and also help you come out of this a better person. Whether you’re just getting started in security or looking to advance your career, or just looking to become more security-aware as an individual, there are some great options for learning from home. My lists below are by no means comprehensive – there’s more content than I can shake a stick at. However, these are intended to be good for beginners and have a diverse set of content. If you know of something I should have included, please reach out.

Some of these options are free or freemium – if you have the means, I’d strongly encourage you to pay for access and support those who are putting it together.

  • PentesterLab has a number of free exercises, and dozens available with a paid PRO subscription. ($35 for 3 months for students, or $20/month for professionals.) The exercises in this environment are very well laid out, including things like basic Unix skills all the way up to complex crypto challenges. Each challenge is well-documented and works reliably, and the challenges are arranged in “badges” helping you find the right progression for your skills.
  • Hack The Box is the premiere site for hosted hackable VMs. They provide a VPN connection to an environment with a number of challenge servers, each of which has a “User” and “Root” flag to be captured. I’ve played through 26 of their boxes as well as 22 challenges. They definitely have content for every skill level if your interest is penetration testing. If you plan to take the OSCP, these boxes are very similar in “spirit” to the OSCP lab boxes. Free tier gets access to all current boxes, pro gets you less busy servers as well as access to retired boxes.
  • Stanford University has made CS 253 - Web Security available online for free. It includes slides, video, readings, and 3 of the course assignments, accessible to anyone who would like to do this.
  • CryptoPals is a set of Cryptography challenges hosted by NCC Group. These are a progressive set of challenges to break various broken cryptography implementations, and require no pre-knowledge of crypto or advanced mathematics. (Some math is required, but it’s described as 9th grade level mathematics.)
  • CryptoHack is similar to CryptoPals, but with a CTF-style flair to it. This is another “learn crypto by breaking it” opportunity.
  • Exploit.Education is a set of challenges for vulnerability analysis/discovery, predominantly of memory corruption vulnerabilities. These are intended as a progressive set of VMs to attack and work your way through. Includes opportunities for ARM and ARM64 exploitation.
  • Why not take the opportunity to build your own home lab? I’ve been playing around in mine quite a bit in the evenings and weekends while working from home. My Windows domain skills have atrophied quite a bit, and I’m trying to reconcile that. Hopefuilly I’ll succeed. :)
  • Get in some reading time. eBooks require no shipping, so instant gratification. I can recommend almost anything from No Starch Press. At the moment, I’m looking at Black Hat Go, and just recently read Real-World Bug Hunting.
  • SANS continues to offer their online trainings. Additionally, they’ve launched a bunch of virtual CTF challenges, including many free opportunities. I played in the Virtual Mini-NetWars Mission 1, and it was a lot of fun and the content was absolutely great. It was like playing a hacking RPG.
  • Offensive Security has their online trainings as per usual. This includes both the very well-known Penetration Testing with Kali Linux (OSCP), as well as Cracking the Perimeter (OSCE), and Advanded Web Attacks and Exploitation (OSWE). I’ve done both OSCP and OSCE in the past, and can highly recommend both of them.
  • Check out a CTF on CTFTime.

No matter what you end up doing, make sure you take some time from yourself and disconnect from all the bad news. It’s so easy to become overwhelmed with everything going on. Focus on something you can control and reaching your goals.


Security 101: X-Forwarded-For vs. Forwarded vs PROXY

Over time, there have been a number of approaches to indicating the original client and the route that a request took when forwarded across multiple proxy servers. For HTTP(S), the three most common approaches you’re likely to encounter are the X-Forwarded-For and Forwarded HTTP headers, and the PROXY protocol. They’re all a little bit different, but also the same in many ways.

X-Forwarded-For

X-Forwarded-For is the oldest of the 3 solutions, and was probably introduced by the Squid caching proxy server. As the X- prefix implies, it’s not an official standard (i.e., an IETF RFC). The header is an HTTP multi-valued header, which means that it can have one or more values, each separated by a comma. Each proxy server should append the IP address of the host from which it received the request. The resulting header looks something like:

1
X-Forwarded-For: client, proxy1, proxy2

This would be a request that has passed through 3 proxy servers – the IP of the 3rd proxy (the one closest to the application server) would be the IP seen by the application server itself. (Often referred to as the “remote address” or REMOTE_ADDR in many application programming contexts.)

So, you could end up seeing something like this:

1
X-Forwarded-For: 2001:DB8::6, 192.0.2.1

Coming from a TCP connection from 127.0.0.1. This implies that the client had IPv6 address 2001:DB8::6 when connecting to the first proxy, then that proxy used IPv4 to connect from 192.0.2.1 to the final proxy, which was running on localhost. A proxy running on localhost might be nginx splitting between static and application traffic, or a proxy performing TLS termination.

Forwarded

The HTTP Forwarded header was standardized in RFC 7239 in 2014 as a way to better express the X-Forwarded-For header and related X-Forwarded-Proto and X-Forwarded-Port headers. Like X-Forwarded-For, this is a multi-valued header, so it consists of one or more comma-separated values. Each value is, itself, a set of key-value pairs, with pairs separated by semicolons (;) and the keys and values separated by equals signs (=). If the values contain any special characters, the value must be quoted.

The general syntax might look like this:

1
Forwarded: for=client, for=proxy1, for=proxy2

The key-value pairs are necessary to allow expressing not only the client, but the protocol used, the original HTTP Host header, and the interface on the proxy where the request came in. For figuring out the routing of our request (and for parity with the X-Forwarded-For header), we’re mostly interested in the field named for. While you might think it’s possible to just extract this key-value pair and look at the values, the authors of the RFC added some extra complexity here.

The RFC contains provisions for “obfuscated identifiers.” It seems this is mostly intended to prevent revealing information about internal networks when using forward proxies (e.g., to public servers), but you might even see it in operating reverse proxies. According to the RFC, these should be prefixed by an underscore (_), but I can imagine cases where this would not be respected, so you’d need to be prepared for that in parsing the identifiers.

The RFC also contains provisions for unknown upstreams, identified as unknown. This is used to indicate forwarding by a proxy in some manner that prevented identifying the upstream source (maybe it was through a TCP load balancer first).

Finally, there’s also the fact that, unlike the defacto standard of X-Forwarded-For, Forwarded allows for the option of including the port number on which it was received. Because of this, IPv6 addresses are enclosed in square brackets ([]) and quoted.

The example from the X-Forwarded-For section above written using the Forwarded header might look like:

1
Forwarded: for="[2001:DB8::6]:1337", for=192.0.2.1;proto=https

Additional examples taken from the RFC:

1
2
3
4
Forwarded: for="_gazonk"
Forwarded: For="[2001:db8:cafe::17]:4711"
Forwarded: for=192.0.2.60;proto=http;by=203.0.113.43
Forwarded: for=192.0.2.43, for=198.51.100.17

PROXY Protocol

At this point, you may have noticed that both of these headers are HTTP headers, and so can only be modified by L7/HTTP proxies or load balancers. If you use a pure TCP load balancer, you’ll lose the information about the source of the traffic coming in to you. This is particularly a problem when forwarding HTTPS connections where you don’t want to offload your TLS termination (perhaps traffic is going via an untrusted 3rd party) but you still want information about the client.

To that end, the developers of HAProxy developed the PROXY protocol. There are (currently) two versions of this protocol, but I’ll focus on the simpler (and more widely deployed) 1st version. The proxy should add a line at the very beginning of the TCP connection in the following format:

1
PROXY <protocol> <srcip> <dstip> <srcport> <dstport>\r\n

Note that, unlike the HTTP headers, this makes the PROXY protocol not backwards compatible. Sending this header to a server not expecting it will cause things to break. Consequently, this header will be used exclusively by reverse proxies.

Additionally, there’s no support for information about the hops along the way – each proxy is expected to maintain the same PROXY header along the way.

Version 2 of the PROXY protocol is a binary format with support for much more information beyond the version 1 header. I won’t go into details of the format (check the spec if you want) but the core security considerations are much the same.

Security Considerations

If you need (or want) to make use of these headers, there are some key security considerations in how you use them to use them safely. This is particularly of consideration if you use them for any sort of IP whitelisting or access control decisions.

Key to the problem is recognizing that the headers represent untrusted input to your application or system. Any of them could be forged by a client connecting, so you need to consider that.

Parsing Headers

After I spent so long telling you about the format of the headers, here’s where I tell you to disregard it all. Okay, really, you just need to be prepared to receive poorly-formatted headers. Some variation is allowed by the specifications/implementations: optional spaces, varying capitalization, etc. Some of this will be benign but still unexpected: multiple commas, multiple spaces, etc. Some of it will be erroneous: broken quoting, invalid tokens, hostnames instead of IPs, ports where they’re not expected, and so on.

None of this, however, precludes malicious input in the case of these headers. They may contain attempts at SQL Injection, Cross-Site Scripting and other malicious content, so one needs to be cautious in parsing and using the input from these headers.

Running a Proxy

As a proxy, you should consider whether you expect to be receiving these headers in your requests. You will only want that if you are expecting requests to be forwarded from another proxy, and then you should make sure the particular request came from your proxy by validating the source IP of the connection. As untrusted input, you cannot trust any headers from proxies not under your control.

If you are not expecting these headers, you should drop the headers from the request before passing it on. Blindly proxying them might cause downstream applications to trust their values when they come from your proxy, leading to false assertions about the source of the request.

Generally, you should rewrite the appropriate headers at your proxy, including adding the information on the source of the request to your proxy, before passing them on to the next stage. Most HTTP proxies have easy ways to manage this, so you don’t usually need to format the header yourself.

Running an Application

This is where it gets particularly tricky. If you’re using IP addresses for anything of significance (which you probably shouldn’t be, but it’s likely there’s some cases where people still are), you need to figure out whether you can trust these headers from incoming requests.

First off, if you’re not running the proxies: just don’t trust them. (Of course, I count a managed provider as run by you.) Also, if you’re not running the proxy, I hope we’re only talking about the PROXY protocol and you’re not exposing plaintext to untrusted 3rd parties.

If you are running proxies, you need to make sure the request actually came from one of your proxies by checking the IP of the direct TCP connection. This is the “remote address” in most web programming frameworks. If it’s not from your proxy, then you can’t trust the headers.

If it’s your proxy and you made sure not to trust incoming headers in your proxy (see above), then you can trust the full header. Otherwise, you can only trust the incoming hop to your proxy and anything before that is not trustworthy.

Man in the Middle Attacks

All of this disregards MITM attacks of course. If an attacker can inject traffic and spoof source IP addresses into your traffic, all bets on trusting headers are off. TLS will still help with header integrity, but they can still spoof the source address, convincing you to trust the headers in the request.

Bug Bounty Tip

Try inserting a few headers to see if you get different responses. Even if you don’t get a full authorization out of it, some applications will give you debug headers or other interesting behavior. Consider some of the following:

1
2
3
X-Forwarded-For: 127.0.0.1
X-Forwarded-For: 10.0.0.1
Forwarded: for="_localhost"

Security 101: Virtual Private Networks (VPNs)

I’m trying something new – a “Security 101” series. I hope to make these topics readable for those with no security background. I’m going to pick topics that are either related to my other posts (such as foundational knowledge) or just things that I think are relevant or misunderstood.

Today, I want to cover Virtual Private Networks, commonly known as VPNs. First I want to talk about what they are and how they work, then about commercial VPN providers, and finally about common misconceptions.


BSides SF 2020 CTF: Infrastructure Engineering and Lessons Learned

Last weekend, I had the pleasure of running the BSides San Francisco CTF along with friends and co-conspirators c0rg1, symmetric and iagox86. This is something like the 4th or 5th year in a row that I’ve been involved in this, and every year, we try to do a better job than the year before, but we also try to do new things and push the boundaries. I’m going to review some of the infrastructure we used, challenges we faced, and lessons we learned for next year.