System Overlord

A blog about security engineering, research, and general hacking.

A Career Plan

I've made several career plans for myself before, but I don't think I've ever done it in a formal manner.  I've never said to myself "I should make a career plan" until I was sitting in Martin Fisher's "How to Hack the Career Development Life Cycle" at B-Sides Atlanta.  It had always been more of a "I want to do this, so first I need to learn this technology" kind of mentality.  However, Martin's talk really made me think.  In some ways, it was sort of unsettling, but I think it can be unsettling anytime you start to really think about the direction your life is going.  I had a sort of "life passing me by" feeling by the end of the presentation (through no fault of his -- it was a great presentation, with some great takeaways.)  I'm hoping making myself this transparent doesn't come back to bite me later, but I'm also hoping that this transparency might get me some feedback from my more experienced readers.  (Insert "what readers?" joke here.)

Where I am now

Currently, I'm doing Devops for a small group at a university.  There are many good things about this: I'm fortunate to have a lot of autonomy and good management; I've received 2 title bumps in as many years, so I believe my efforts are being recognized and appreciated; and I've had a chance to work with a variety of things and expand my skillset and horizons quite a bit.  There's also some downsides: being in a small group, I end up doing more direct end-user support than I'd like; I don't feel that many of my coworkers and I are on the same page; and I feel like some areas of my worklife have become stagnant.  I do really enjoy doing most of the Devops tasks, but unfortunately, I'm doing more "routine Drupal" than I'd like.  I also have to admit that, given the size of the group I'm in and the title I have, I have to wonder what opportunities will come for me.

My Career Goals

The first problem I have in laying out a career plan is that I have a hard time articulating my career goals.  I have a number of interests, and I'd like to incorporate them all into what I do.  I like Devops-style System Administration (there's a lot of satisfaction in getting things running and keeping them running just right), I like writing code (interesting code, not the boring business-rules or data-shuffling kind of code that too many code monkeys are forced to write), and I really like Information Security.  Even in InfoSec, there's a lot that piques my interest.  Pentesting is quite a rush, but so is forensics/incident response.  Both of those areas are the sort of puzzle that leads to me working until 3 in the morning because I just can't bring myself to stop.  The only single title that I've seen that comes remotely close to describing my interests is "Security Operations," but that's such a vague and all-encompassing term that it doesn't narrow things down much.

What I don't want to do

So, I have certain standards that I'd like to stick to.  First off, I don't want to do things that cause me any (more) concern over my financial situation.  Secondly, I'm a FOSS guy at heart -- a Linux geek through and through.  Any job where I can't put those skills to good use would be a waste for both me and my employer.  Thirdly, I'm not a policy or management guy.  I'm happy to give input on policy, but pushing papers all day is not my forte.

So how do I get there?

This is the most important part of the career plan -- I can sit and wish and dream all day long, but if I don't do something about it, I'm not going anywhere.

  • I'm currently pursuing a M.S. in Computer Science.  I'm hoping to do a thesis project that's security-oriented to get the most out of my program.
  • I'd like to obtain either a CEH or OSCP.  While I may not necessarily put a lot of weight in certifications, I belive that either of these would be of benefit.  I've heard OSCP is an interesting one in particular.  (Unfortunately, they're fairly pricey on a higher ed salary.)
  • I need to find a security-oriented FOSS project to really get involved with.  This will help me learn, contribute back to the community, and help me focus my interests.
  • I should find more security events to attend, and do more networking.

What have I learned?

I'm okay with not wanting to be a manager.  I'm okay with not wanting to be a CSO.  People look at me like I have two heads when I say I want to "stay technical," that I don't want to "move up."  Of course I want to "move up," but I want to do so by being the best technical guy out there.  Managing people, managing budgets, managing policy -- there are others that are good at that, so let them do it.  I'm glad I went to Martin's presentation -- I'd actually begun to wonder if I was the only person who wanted to keep getting his "hands dirty" for his entire career.

Have I done something risky by posting all of this out there?  I don't think so, but it wouldn't be the first mistake I've ever made.  I have no immediate plans to leave my current job, but it's important to think about what's at the end of the tunnel.  I think it's a good thing to be perfectly clear with both present and future employers -- I've seen what happens when people aren't clear with each other, and it's not pretty.

Martian Packet Messages

Occasionally, you might see messages like the following in your Linux kernel messages:

martian source from, on dev eth1<br />
        ll header: 52:54:00:98:99:d0:52:54:00:de:d8:10:08:00 

There's a lot of discussion out there about what this means, but not a lot about how to trace down the source.  Hopefully this will provide some insight into what the messages actually mean, and how to understand them.

What are martian packets?

A martian packet is one that comes from an "impossible source".  For example, this might be from an RFC 1918 reserved netblock routed across the internet, or any packet with a "localnet" (, etc.) IP on an interface other than the loopback interface.  In other words, martian packets mean either something is misconfigured, or someone is trying to do something very sneaky.  RFC 1812 provides more detail about what is and is not a valid routable IPv4 address.

So what do those messages mean?

Let's start with the first message: it contains the destination and source IP addresses, as set in the IPv4 packet, and the network interface on which the packet arrived.  Despite how it reads, the first address is NOT the source address in the packet -- it is, in fact, the destination address!  The 2nd IP is the IP that was contained in the IPv4 packet, but this isn't of very much help in finding the sending host -- remember, the whole point of this message is an indication that the source of the packet is "suspect."

Fortunately, the 2nd line contains a lot of information for us, but it's printed in a format that's very hard to read.  This is the link-layer header of the packet that contained the erroneous source address.  Almost all machines these days use physical connectivity that looks like ethernet, and the log entry above is based on an Ethernet Frame.  (PPP connections will probably look different, but I've never seen this message on anything but ethernet.)  The ethernet header is at least 14 bytes long, and most of us will only see 14 byte headers.  The longer headers are used with 802.1q VLANs, but most people will never see those, so we're going to discuss only the 14 byte headers.

The 14 bytes above, printed in colon-separated hex, contain the destination MAC address, the source MAC address, and either the length or the ethertype code.  The MAC addresses are in standard notation, and the first 6 octets are the destination, with octets 7-12 representing the source.  These will help you track down where the packet came from (though it might be your upstream router, in which case you'll need to do more investigative work).

The last 2 bytes take more explaining.  If the value is less than 0x0600, then it represents the length of the data field for 802.3 compliant ethernet frames.  Greater values, as seen above, are actually EtherType codes.  This will mostly be 0x0800, which means an IPv4 packet is contained in the data segment.  (After all, this error does come from an IPv4 issue.)  Other values might be 0x8100 (802.1q VLAN-tagged packet) or 0x0806 (ARP packet).  There are many more, but you're unlikely to see them in this context.

Long Story Short

Look at bytes 7-12 of the link-layer header.  That's the MAC of the last machine to touch the packet.  Look there for the cause (if it's a router, look upstream from there, and then figure out why you're not filtering the martian packets on your router).  Yes, the source MAC could be spoofed, but then someone's playing games on your local network segment.  (Which probably means they have physical access, or you have a terribly designed network.  Remember, physical access=pwned.) Review

I first heard about last March at Drupalcon Chicago.  Having been a Linode customer for a couple of years, I was skeptical at first, but 7 months later, I'm very happy with the level of service provides.  When I've been working on projects for demanding clients, I've been able to scale my VPS up by adding additional nodes -- either daily or monthly.  After the project was done, I could have scaled back down -- but there's always another project on the horizon!  (One of these days, I'll have to make "sleep" a project to make sure it gets done too.)  While there has been a couple of small downtimes, has always been great about providing status updates and letting customers know where they stand.  Additionally, their service people are great and respond quickly via email or twitter. offers an innovative product -- most VPS providers do not use SANs for their storage needs, but this allows to be less susceptible to hardware failures.  In fact, I believe the only downtimes that I have been affected by are those that had an impact on the entire SAN.  Additionally, their ability to scale nodes with a quick reboot by not expanding storage immediately gives you a flexibility that is not available from all providers.  Finally, their pricing structure fits well with my needs -- as you buy more nodes, the new nodes become cheaper.

The only feature is missing that I wish they had would be slave DNS servers.  Linode offers this service, as do a couple of other VPS providers, and its something I depend heavily on with my light infrastructure.  (2 vpses.)  I know I could go somewhere like Dyn, Inc. for my DNS hosting, but for something I use for personal and side projects, the Dyn offerings are overkill.  (You can only get IPv6/DNSSEC on their more expensive offerings.)

I have never had an issue with responsiveness or latency on my VPS.  As I do a lot of work via SSH, CPU or network overselling can easily cause problems for me, but that has not been an issue for me.  I am certain the bad wireless at some of the places where I try to get work done causes orders of magnitude more delays than's infrastructure.

Overall, though I still have my main site on Linode, is a great host for my developmental projects and secondary services.  They take what they do seriously, and they do it well.

(Virtually) Setting Up A Test Lab (Part 1)

I've spent a little bit of time today doing something that was long overdue.  I've transitioned most of my day-to-day data to my laptop, so I decided it was time to put my desktop to use as a "virtual lab."

I've set up KVM on my desktop with two virtual machines (so far) in it.  The first one I call "LabManager" -- it's effectively a head node from the "Lab" network out to the real world.

Currently, LabManager has two interfaces -- one of them is bridged to eth0 on the desktop, and the other is only bridged to other VMs.  The other VMs have no direct connectivity.  At this time, LabManager is not forwarding packets, bridging, or routing.  It is running dnsmasq for DNS & DHCP and apt-cacher-ng to allow systems behind it to download packages.  Additionally, LabManager is running a tftp server to perform PXE boot installs.  These installs use a preseed file so the only manual entry is a hostname.  Everything else is automatically setup.

It's my hope that keeping the networks segregated will let me play around with "dangerous" things without posing any risk to my real home network (or the internet).  The use of RFC 1918 IPs and restrictive firewalls should help with this.  I've got the preseed installing puppet, but I haven't set up the puppetmaster yet.  That'll be the 2nd phase, which I'll hopefully get around to by next weekend.

Phase 3 will be providing distributions other than Debian stable via PXE boot.  I'd like to provide both some bleeding-edge work and some older software for pentesting practice.

Suggestions for improvement are welcome!

KSU Cyber Security Awareness Day 2011

Today was the KSU Cyber Security Awareness Day, presented by KSU's Information Technology Services (a sister department to the department I work in), and it was a resounding success!  There were several presentations that had standing-room only attendance, and for good reason.

My personal favorites:

Mike Rothman from Securosis on finding happiness in information security.  Mike's presentation was as much about being happy in your job and in your life as it was about cyber security, but he asked a number of very pointed questions.  Questions about pay/salary, job satisfaction, and life priorities.  I found the questions unsettling, not because of the actual question, but because I realized that I'd been subconsciously thinking those same things for quite a while now.  The take away from his presentation can probably be summed up as "Is what you're doing today getting you where you want to go?"

Rob Ragan of Stach & Liu came to talk about "Google hacking" and the Diggity tool he's put together.  Rob presented a number of very revealing things that can be found just by performing searches on the common search engines, and a number of people in the audience were disturbed to find out how information can be (mis-)handled.  At one point, Rob stated he had run a search on Kennesaw's web presence and found about 11,000 possible pieces of sensitive information.  It was mildly amusing to see the reaction of the members of our ISO until it was realized that he had searched on "" and not "".  I hope K-State is aware of the 11,000 possible hits.

Chris Sandy of our own ITS Information Security Office actually presented twice.  One presentation (that I rather enjoyed) was used as filler due to a last-minute cancellation.  Chris talked about the physical aspects of information security, but particularly about physical locks and lockpicking.  I had no idea KSU had a sport lockpicker on campus -- it makes me even more interested in picking up lockpicking, as a sport.

Overall, it was a great one-day event, and I hope that it helped open the eyes of the students, faculty, and staff attending the session.  I know our InfoSec guys worked really hard to put it together, and I think that did an outstanding job.  Having done conference planning before, I know its no easy task, but they pulled it off quite well.