PSA: Typos in mkfs commands are painful

TL;DR: I apparently typed mkfs.vfat /dev/sda1 at some point. Oops.

So I rarely reboot my machines, and last night, when I rebooted my laptop (for graphics card weirdness) Grub just came up with:

1
2
Error: unknown filesystem.
grub rescue>

WTF, I wonder how I borked my grub config? Let’s see what happens when we ls my /boot partition.

1
2
grub rescue>ls (hd0,msdos1)
unknown filesystem

Hrrm, that’s no good. An ls on my other partition isn’t going to be very useful, it’s a LUKS-encrypted LVM PV. Alright, time for a live system. I grab a Kali live USB (not because Kali is necessarily the best option here, it’s just what I happen to have handy) and put it in the system and boot from that. file tells me its an x86 boot sector, which is not at all what I’m expecting from an ext4 boot partition. It slowly dawns on me that at some point, intending to format a flash drive or SD card, I must’ve run mkfs.vfat /dev/sda1 instead of mkfs.vfat /dev/sdb1. That one letter makes all the difference. Of course, it turns out it’s not even a valid FAT filesystem… since the device was mounted, the OS had kept writing to it like an ext4 filesystem, so it was basically a mangled mess. fsck wasn’t able to restore it, even pointing to backup superblocks: it seems as though, among other things, the root inode was destroyed.

So, at this point, I basically have a completely useless /boot partition. I have approximately two options: reinstall and reconfigure the entire OS, or try to fix it manually. Since it didn’t seem I had much to lose and it would probably be faster to fix manually (if I could), I decided to give door #2 a try.

First step: recreate a valid filesystem. mkfs.ext4 -L boot /dev/sda1 takes care of that, but you better believe I checked the device name about a dozen times. Now I need to get all the partitions and filesystems mounted for a chroot and then get into it:

1
2
3
4
5
6
7
8
9
% mkdir /target
% cryptsetup luksOpen /dev/sda5 sda5_crypt
% vgchange -a y
% mount /dev/mapper/ubuntu-root /target
% mount /dev/sda1 /target/boot
% mount -o bind /proc /target/proc
% mount -o bind /sys /target/sys
% mount -o bind /dev /target/dev
% chroot /target /bin/bash

Now I’m in my system and it’s time to replace my missing files, but how to figure out what goes there? I know there are at least files for grub, kernels, initrds. I wonder if dpkg-query can be useful here?

1
2
# dpkg-query -S /boot
linux-image-3.13.0-36-generic, linux-image-3.13.0-37-generic, memtest86+, base-files: /boot

Well, there’s a handful of packages. Let’s reinstall them:

1
# apt-get install --reinstall linux-image-3.13.0-36-generic linux-image-3.13.0-37-generic memtest86+ base-files

That’s gotten our kernel and initrd replace, but no grub files. Those can be copied by grub-install /dev/sda. Just to be on the safe side, let’s also make sure our grub config and initrd images are up to date.

1
2
3
# grub-install /dev/sda
# update-grub2
# update-initramfs -k all -u

At this point, I’ve run out of things to double check, so I decide it’s time to find out if this was actually good for anything. Exit the chroot and unmount all the filesystems, then reboot from the hard drive.

It worked! Fortunately for me, /boot is such a predictable skeleton that it’s relatively easy to rebuild when destroyed. Here’s hoping you never find yourself in this situation, but if you do, maybe this will help you get back to normal without a full reinstall.


Getting Started in CTFs

My last post was about getting started in a career in information security. This post is about the sport end of information security: Capture the Flag (CTFs).

I’d played around with some wargames (Smash the Stack, Over the Wire, and Hack this Site) before, but my first real CTF (timed, competitive, etc.) was the CTF run by Mad Security at BSides SF 2013. By some bizarre twist of fate, I ended up winning the CTF, and I was hooked. I’ve probably played in about 30 CTFs since, most of them online with the team Shadow Cats. It’s been a bumpy ride, but I’ve learned a lot about a variety of topics by doing this.

If you’re in the security industry and you’ve never tried a CTF, you really should. Personally, I love CTFs because they get me to exercise skills that I never get to use at work. They also inspire some of my research and learning. The only problem is making the time. :)

Here’s some resources I’ve thought were interesting:


Getting Started in Information Security

I’ve only been an information security practitioner for about a year now, but I’ve been doing things on my own for years before that. However, many people are just getting into security, and I’ve recently stumbled on a number of resources for newcomers, so I thought I’d put together a short list.


[CVE-2014-5204] Wordpress nonce Issues

Wordpress 3.9.2, released August 6th, contained fixes for two closely related vulnerabilities (CVE-2014-5204) in the way it handles Wordpress nonces (CSRF Tokens, essentially) that I reported to the Wordpress Security Team. I’d like to say the delay in my publishing this write-up was to allow people time to patch, but the reality is I’ve just been busy and haven’t gotten around to this.

TL;DR: Wordpress < 3.9.2 generated nonces in a manner that would allow an attacker to generate valid nonces for other users for a small subset of possible actions. Additionally, nonces were compared with ==, leading to a timing attack against nonce comparison. (Although this is very difficult to execute.)

Review of CSRF Protection

A common technique for avoiding Cross Site Request Forgery (CSRF) is to have the server generate a token specific to the current user, include that in the page, and then have the client echo that token back with the request. This way the server can tell that the request was in response to a page from the server, rather than a request triggered on the user’s behalf by an attacker. OWASP calls this the Synchronizer Token Pattern and one of the requirements is that an attacker is not able to predict or determine tokens for another user.

Wordpress Nonces

Wordpress uses what they call “nonces” (but they’re not, in fact, guaranteed to be used only once) for CSRF protection. These nonces include a timestamp, a user identifier, and an action, all of which are part of best practices for CSRF tokens. These values are HMAC’d with a secret key to generate the final token. All of this is in accordance with best practices, and at first blush, the nonce generation code looks good. Here’s how nonces were generated prior to the 3.9.2 fix:

1
2
3
4
5
6
7
8
9
10
#!php
function wp_create_nonce($action = -1) {
	$user = wp_get_current_user();
	$uid = (int) $user->ID;
	# snipped

	$i = wp_nonce_tick();

	return substr(wp_hash($i . $action . $uid, 'nonce'), -12, 10);
}

wp_nonce_tick returns a monotonically increasing value that increments every 12 hours to provide a timeout on the resulting nonce. $user->ID is the auto-increment id column from the database. wp_hash performs an HMAC-MD5 using a key selected by the 2nd argument, the nonce key in this case. So, we’re esentially getting an HMAC of a string concatenation of the current time, the action value passed in, and the current user’s UID. Assuming HMAC is strong, we’ve got a user, action and time-specific token, right?

Wrong. What if we can figure out a way to collide inputs to the HMAC? Turns out this is pretty easy, actually. Let’s look at some instances where wp_create_nonce is used:

1
2
3
4
#!php
wp_create_nonce( "approve-comment_$comment->comment_ID" )
wp_create_nonce( 'set_post_thumbnail-' . $post->ID );
wp_create_nonce( 'update-post_' . $attachment->ID );

In more than one case, we see places where nonces are created that end in an ID value (an integer from the database). Note that these action values are immediately before the UID, also an integer. This means that once the concatenation is done, there is no separation between the integer values of the action and the UID, leading to collisions in the hash input, and consequently the same nonce value being generated. Take, for example, an installation where users are privileged to update their own post but not those of other users. Let’s take user 1 and post 32, and user 21 and post 3. What are the respective inputs to wp_hash? (I’m substituting 0 for the timestamp value as it’s the same for all users at the same time.)

1
2
$i . 'update-post_32' . 1 => '0update-post_321'
$i . 'update-post_3' . 21 => '0update-post_321'

Despite being two separate users and two separate actions, their nonce values will be the same. While this is fairly limited in what an attacker can do (you can’t pick arbitrary users and values, only “related” users and values), it’s also very easy to fix and completely eliminate the hole: simply add a non-integer separator between the segments of the hash input. Wordpress 3.9.2 now inserts a | between each segment, so now the hash inputs look like this:

1
2
$i . '|' . 'update-post_32' . '|' . 1 => '0|update-post_32|1'
$i . '|' . 'update-post_3' . '|' . 21 => '0|update-post_3|21'

No longer will the HMACs collide, so now two distinct nonces are generated, closing the CSRF hole. The implementation also now includes your session token, making it even harder for an attacker to generate a collision, though I can’t think of a specific hole that fixes (it does generate new nonces after a logout/login):

1
2
3
4
5
6
7
8
9
10
11
#!php
function wp_create_nonce($action = -1) {
	$user = wp_get_current_user();
	$uid = (int) $user->ID;
	# snipped

	$token = wp_get_session_token();
	$i = wp_nonce_tick();

	return substr( wp_hash( $i . '|' . $action . '|' . $uid . '|' . $token, 'nonce' ), -12, 10 );
}

Timing Attack

Though probably very difficult to exploit on modern systems, using PHP’s == to compare hashes results in a timing attack (not to mention the possibility of running afoul of PHP’s bizarre comparison behavior).

Formerly:

1
2
3
#!php
if ( substr(wp_hash($i . $action . $uid, 'nonce'), -12, 10) === $nonce ) {
  ...

Now:

1
2
3
4
#!php
$expected = substr( wp_hash( $i . '|' . $action . '|' . $uid . '|' . $token, 'nonce'), -12, 10 );
if ( hash_equals( $expected, $nonce ) ) {
  ...

hash_equals was added in PHP 5.6, but Wordpress provides their own, using a fairly common constant-time comparison pattern, if you don’t have it.

Summary

Even when you include all the right things in your CSRF implementation, it’s still possible to run into trouble if you combine them the wrong way. Much like a hash length extension attack, cryptography won’t save you if you’re putting things together without thinking about how an attacker can alter or vary it.

I’d like to thank the Wordpress security team for their responsiveness when I reported the issues here. I have nothing but positive things to say about the team and my interactions with them.


Security: Not a Binary State

I’ve been spending a fair amount of time on Security StackExchange lately, mostly looking for inspiration for research and blogging, but also answering a question every now and then. One trend I’ve noticed is asking questions of the form “Is security practice X secure?”

This is asked as a yes/no question, but security isn’t a binary state. There is no “absolutely secure.” Security is a spectrum, and it really depends on what you’re worried about, which is where threat modeling comes in. Both users and service providers need to consider their risks and decide what’s important to them.

Users

Most internet users will never be specifically targeted by an attacker. Their concerns will (should) include:

  • Run-of-the-mill malware
  • Phishing
  • Security on public hotspots
  • Password management

For these users, maintaining a patched system, being aware of phishing, using a VPN on public hotspots, and maybe using an anti-virus or anti-malware program will generally protect against the threats they’re subject to. Of course, education is still important, as if they run random programs downloaded from the internet, malware will still make its way in.

Other users might have a more determined adversary. Those with access to financial systems, valuable data, or other desirable access may end up being targeted. Spearphishing becomes an issue. Depending on what you do, you might even find yourself subject to the ire of well-funded state attackers. What is adequately secure for a “normal” user is woefully inadequate for these users.

Service Providers

I’d originally only intended to talk about providers of Internet services, but given the continuing tendency of businesses to place their infrastructure online, I think all businesses interacting with customers fall into this category.

Service providers have a responsibility to two kinds of data: their data, and their user’s data. For their data, they are essentially a user as above. The service provider should perform their own threat modeling, decide what risks are and are not acceptable to them, and then act to secure their data against the risks they are worried about.

User data, on the other hand, is sacred. While no business can protect against every possible adversary, they should consider the trust users place in them and try to protect the data as their users would want it protected. While the provider might be willing to roll the dice on, say, their corporate email, they should consider if that data can be used to compromise user data. (Nearly universally, the answer to that question is “yes.”)

We’ve had a recent series of Point-of-Sale data breaches, including Target and Home Depot. I’m disappointed that, so far, these haven’t seemed to hurt the retailers very much. Retailers will only take adequate measures to protect themselves when it becomes obvious that the consequences of a data breach will be massive. If consequences of violating PCI actually had teeth (e.g., you can’t accept credit cards anymore for some period of time until you can be re-certified, like 6 months) and customers moved elsewhere, maybe the businesses would get a bit proactive.

Conclusion

That’s enough ranting for now, but I wanted to get one main point across: It’s important to remember that you can never be “secure.” You can only be “secure enough” to defend against some set of adversaries and threats. Anything else is just wishful thinking.