System Overlord

A blog about security engineering, research, and general hacking.

The segmentation fault occurred where?!?

18 Nov 2012 in Computer (Reading time: 3 minutes)

I recently ran into a C++ problem where a segfault was occurring in code in a stable library that hadn't been changed in a while. For a while, I couldn't figure out what would have broken in that library, and the call site looked perfectly fine. Before I give away the answer, let's take a quick quiz. What does the following code output? (And yes, this is somewhat compiler dependent, so let's pretend we're talking about how g++ works.)

#include <cstdio>
 
class Foo {
  private:
    char *name;
  public:
    void whatami() {
      printf("I am a Foo.\n");
    }
    void whoami() {
      printf("I am %s.\n", name);
    }
};
 
int main(int argc, char **argv){
  Foo *f = NULL;
  f->whatami();
  f->whoami();
}

My first instinct was to say "Segmentation Fault" and nothing else, because line 17 is going to dereference a NULL pointer. It turns out, of course, it's not that simple -- it'll actually print the "I am a Foo" before segfaulting. Clearly, then, the segfault must be on line 18, right? Wrong. Line 11 is where we give up, as f is not dereferenced until then. To see why this is, let's think about the equivalent C code:

#include <stdio.h>
 
typedef struct {
  char *name;
} Foo;
 
void Foo_whatami(Foo *self) {
  printf("I am a Foo.\n");
}
 
void Foo_whoami(Foo *self) {
  printf("I am %s.\n", self->name);
}
 
int main(int argc, char **argv) {
  Foo *f = NULL;
  Foo_whatami(f);
  Foo_whoami(f);
}

As we can see here, the pointer is actually unused by the method "Foo_whatami". But wait, you say, don't we need the address of Foo to resolve the location of the method? No, as whatami and whoami are not virtual methods! Their addresses can be determined by the compiler at compile time. Virtual methods, on the other hand, need a pointer from within the data area of the object to the vtable to resolve addresses. Change whatami to a virtual method, and you'll crash much more efficiently.

So remember, even if the code looks like you're dereferencing the pointer, it may well not be dereferenced until much later!

MITM on KVM Guests

10 Nov 2012 in Security (Reading time: 1 minute)

I run a KVM virtualization system as part of my test lab.  I often want to redirect traffic to an intermediate application (such as sslsniff) on the host.  Supposing I have a guest on interface vnet7, bridged to br10, with the host running on 192.168.1.10 the following ebtables & iptables magic gets the job done:

ebtables -t broute -A BROUTING -p IPv4 -i vnet7 --ip-proto tcp --ip-dport 443 -j redirect --redirect-target DROP
iptables -t nat -A PREROUTING -i vnet7 -p tcp --dport 443 -j DNAT --to-destination 192.168.1.10:9999

Note that you can't use -j REDIRECT, as that's (roughly) equivalent to DNAT to the IP of the incoming interface, but bridged virtual network interfaces (vnet7) have no IP address.

2 Weeks at Google

11 Mar 2012 in Work (Reading time: 1 minute)

Two weeks at Google have been... amazing.  There's a lot that I can't talk about, but I can feel comfortable in confirming some of the things you hear about Google:

  • The people are insanely smart.
  • The scale blows your mind as a Noogler (new Googler).
  • The food is great.
  • It has culture.

I'm a "Site Reliability Engineer" which is a job title that may not exist anywhere else.  It's basically production-oriented operational engineering: keeping production systems running and making them run better.

Ann and I have found a new place, so we'll be moving there from the corporate housing next week.  It'll be nice to get our stuff back and get settled in.

(All opinions expressed here are mine and not my employers.  I will not comment on or discuss Google policy, unreleased products, or other proprietary information.)

The End of a Chapter

15 Feb 2012 in Personal (Reading time: 2 minute)

I'm not usually one for reflective personal blog entries, but some events require a brief mention: today was my last day at KSU, and it was an incredibly surreal day.  Though I've known this day was coming for over a month, it is still hard to believe that it got here.  In many ways, today felt like any other day: the work was similar, things needed to get done.  In other ways, there was an 800 pound gorilla in the room: everyone knew that tomorrow I wouldn't be coming to work.  When I finally cleared out my office, the finality of what was going on really hit me.

There are friends I have made at KSU that are like none other, and I will genuinely miss them.  I hope we will stay in contact and I hope to get back to the Atlanta area sometime and visit them, or perhaps they will make it out to the bay area.  Unfortunately, friendships seem to have a tendency to dissolve too easily with distance, but one can always hope that things will be different.

Today, one chapter of my life closes.  For now, a brief interlude to relocate to the Bay Area, and then a new chapter begins in 12 short days!

My Time at KSU

02 Feb 2012 in Work (Reading time: 5 minutes)

As you might have seen, I'm leaving my position at Kennesaw State University to take a position as a Site Reliability Engineer at Google.  This is something I'm very excited about, but I thought I'd take a look back at my time at KSU as I approach the end.  It's worth mentioning that I'm not leaving KSU because of KSU, but because this is an opportunity I just could not turn down.  For the most part, I like my position at KSU, and I really like most of the people that I work with.  There's a particular group that's become three of my closest friends and one treasured acquaintance.

As much for my own memories as for the readers of this blog, I'm going to take a look back at some of the projects I've enjoyed most over the three years I've been at KSU.  I've been fortunate enough to have taken on many of these projects of my own accord, and they've become successful and play varying roles in our department's operations.  I'm also fortunate that, though I may have spent (many) more than 40 hours a week working on some of these, I like what I do enough that I can't find the line where work ends and personal time begins.

So here's the look back at my time in Advanced Computing Services (formerly Online Development Group, formerly Online Learning Services).

Centralized Authentication for Departmental Resources

When I first started at KSU, every server run by the department used local users for logins, and different applications had local login systems.  Now, all shell accounts go back to an LDAP server, home directories are mounted via NFS, and many web-based applications point to the same LDAP directory.  This has reduced the number of passwords to remember and simplified system administration. 

High Availability for Production Webservices

Using Linux-HA, MySQL replication, and shared storage, our production Drupal web environment has high availability built in to reduce the risk of downtime.  Since implementation, we've achieved greater than 99.9% uptime, even through power generator replacements, network upgrades, and even the occasional configuration glitch.  (And yeah, that last one was me.)  Of course, I'm sure I've jinxed myself now, but it's been a good run of uptime.

KSU Video

In about December 2009, we were approached with something a little bit outside our normal comfort zone: develop a system to record IP-based cameras used for mock sales competitions in our College of Business.  The system needed to be web-based, precisely time recordings, make the finished recordings available on a website, control recording lights in the practice area, and also live-stream the footage to rooms full of judges.  Oh, and its inaugural run would be a competition with dozens of schools from several countries the following March.  Despite numerous mis-estimations and "surprises" along the way, we met the deadline and the competition was a success.  We're in the process of upgrading the system to be flexible for other uses on campus, including situations like nursing clinical training and professional speaking.

VMWare Implementation

I had the pleasure of deploying a VMWare cluster that now hosts a few dozen virtual machines, allowing us to quickly spin up new development environments to test applications and provide redundancy for low-utilization services.

Puppet

This is the one that got away -- I was working on deploying Puppet for configuration management on all of our servers.  Unfortunately, this may or may not ever get deployed, but I do believe that configuration management is key to any significant production environment.

Most Importantly

My biggest accomplishments have been the friends I've made and the things I've learned.  Short of Alzheimer's or a traumatic brain injury, I will never forget many of my days -- even if there are a few I wish I could forget.