System Overlord

A blog about security engineering, research, and general hacking.

Big Changes

Today I did one of the hardest things I have had to do: I turned in my notice that I would be leaving Kennesaw State University in February.  It was hard because I consider my management and many of my coworkers to be friends and I genuinely do enjoy my job, but I now have the opportunity to start the next chapter of my life.  At the end of February, I'll be moving to California and starting at Google as a Site Reliability Engineer!

It's a pretty exciting opportunity for me, and I'm looking forward to it very much, but I will miss the people at Kennesaw.  It's been a good time, and had this opportunity not arisen, I probably would have stayed there for quite a while longer.

Change is good, even if it is a bit hard to do.


Recruiters!

There hasn't been a lot of updates lately, and I apologize... between contract work, end of the semester, and other personal issues, writing blog posts has slipped into the cracks.  Hopefully that'll be fixed as I've finally completed my 2nd 1st semester of grad school.


I am occasionally contacted by recruiters of various sorts.  Most of them I just ignore, particularly those that provide no details on who the employer is, what the compensation package might be, etc.  As a rule, I find 3rd party recruiters who contact me out of the blue are probably contacting anyone whose website includes the word "Linux".  (Although I've received more than one recruiter looking for Windows/.NET software development...)  The most recent one was a real gem though.  No names -- I'm not trying to call him/her out individually, just to shed some light on the vagueness that is the world of recruiting.  I've trimmed the email down to the salient parts:

Hello David,

Hope you are doing good.

I am currently looking for Senior level Linux Systems Administrator, full time position with a well established startup in Bay Area, CA.

I understand that you are currently studying, but wanted to check if you would know someone who would be interested. Open to considering mid or junior level candidates as well if they have the right experience.

There are a number of things that immediately scream red flag from this post:

  1. While I am currently pursuing an MS, I think my resume makes it pretty obvious that I'm also working full-time.  If not, that's something I need to fix.
  2. If somebody has the "right experience" to be a "Senior level Linux Systems Administrator", wouldn't they now be a "Senior level Linux Systems Administrator"?  After all, the distinction between a mid- or junior-level administrator and a senior one, in my mind anyway, is the experience.
  3. This is my favorite one: it's with a "well established startup."  I don't know about those out in the valley, but in my mind, you're either "well established" or a "startup", not both.  You can be a "well funded startup" if you want to give someone confidence about their odds of getting a paycheck, but a "well established startup" is just an effort to say "we want them to think it'll feel like a cool startup, but pay like a big business!"

Don't get me wrong: I'm flattered by the contacts from recruiters.  I must be doing SOMETHING right.  There have even been a couple of interesting ones directly from major Silicon Valley players, but usually, the 3rd party ones aren't usually that exciting.  But maybe its a sign that things are turning around, at least in the IT world.


A Career Plan

I've made several career plans for myself before, but I don't think I've ever done it in a formal manner.  I've never said to myself "I should make a career plan" until I was sitting in Martin Fisher's "How to Hack the Career Development Life Cycle" at B-Sides Atlanta.  It had always been more of a "I want to do this, so first I need to learn this technology" kind of mentality.  However, Martin's talk really made me think.  In some ways, it was sort of unsettling, but I think it can be unsettling anytime you start to really think about the direction your life is going.  I had a sort of "life passing me by" feeling by the end of the presentation (through no fault of his -- it was a great presentation, with some great takeaways.)  I'm hoping making myself this transparent doesn't come back to bite me later, but I'm also hoping that this transparency might get me some feedback from my more experienced readers.  (Insert "what readers?" joke here.)

Where I am now

Currently, I'm doing Devops for a small group at a university.  There are many good things about this: I'm fortunate to have a lot of autonomy and good management; I've received 2 title bumps in as many years, so I believe my efforts are being recognized and appreciated; and I've had a chance to work with a variety of things and expand my skillset and horizons quite a bit.  There's also some downsides: being in a small group, I end up doing more direct end-user support than I'd like; I don't feel that many of my coworkers and I are on the same page; and I feel like some areas of my worklife have become stagnant.  I do really enjoy doing most of the Devops tasks, but unfortunately, I'm doing more "routine Drupal" than I'd like.  I also have to admit that, given the size of the group I'm in and the title I have, I have to wonder what opportunities will come for me.

My Career Goals

The first problem I have in laying out a career plan is that I have a hard time articulating my career goals.  I have a number of interests, and I'd like to incorporate them all into what I do.  I like Devops-style System Administration (there's a lot of satisfaction in getting things running and keeping them running just right), I like writing code (interesting code, not the boring business-rules or data-shuffling kind of code that too many code monkeys are forced to write), and I really like Information Security.  Even in InfoSec, there's a lot that piques my interest.  Pentesting is quite a rush, but so is forensics/incident response.  Both of those areas are the sort of puzzle that leads to me working until 3 in the morning because I just can't bring myself to stop.  The only single title that I've seen that comes remotely close to describing my interests is "Security Operations," but that's such a vague and all-encompassing term that it doesn't narrow things down much.

What I don't want to do

So, I have certain standards that I'd like to stick to.  First off, I don't want to do things that cause me any (more) concern over my financial situation.  Secondly, I'm a FOSS guy at heart -- a Linux geek through and through.  Any job where I can't put those skills to good use would be a waste for both me and my employer.  Thirdly, I'm not a policy or management guy.  I'm happy to give input on policy, but pushing papers all day is not my forte.

So how do I get there?

This is the most important part of the career plan -- I can sit and wish and dream all day long, but if I don't do something about it, I'm not going anywhere.

  • I'm currently pursuing a M.S. in Computer Science.  I'm hoping to do a thesis project that's security-oriented to get the most out of my program.
  • I'd like to obtain either a CEH or OSCP.  While I may not necessarily put a lot of weight in certifications, I belive that either of these would be of benefit.  I've heard OSCP is an interesting one in particular.  (Unfortunately, they're fairly pricey on a higher ed salary.)
  • I need to find a security-oriented FOSS project to really get involved with.  This will help me learn, contribute back to the community, and help me focus my interests.
  • I should find more security events to attend, and do more networking.

What have I learned?

I'm okay with not wanting to be a manager.  I'm okay with not wanting to be a CSO.  People look at me like I have two heads when I say I want to "stay technical," that I don't want to "move up."  Of course I want to "move up," but I want to do so by being the best technical guy out there.  Managing people, managing budgets, managing policy -- there are others that are good at that, so let them do it.  I'm glad I went to Martin's presentation -- I'd actually begun to wonder if I was the only person who wanted to keep getting his "hands dirty" for his entire career.

Have I done something risky by posting all of this out there?  I don't think so, but it wouldn't be the first mistake I've ever made.  I have no immediate plans to leave my current job, but it's important to think about what's at the end of the tunnel.  I think it's a good thing to be perfectly clear with both present and future employers -- I've seen what happens when people aren't clear with each other, and it's not pretty.


Martian Packet Messages

Occasionally, you might see messages like the following in your Linux kernel messages:

martian source 192.168.1.1 from 127.0.0.1, on dev eth1<br />
        ll header: 52:54:00:98:99:d0:52:54:00:de:d8:10:08:00 

There's a lot of discussion out there about what this means, but not a lot about how to trace down the source.  Hopefully this will provide some insight into what the messages actually mean, and how to understand them.

What are martian packets?

A martian packet is one that comes from an "impossible source".  For example, this might be from an RFC 1918 reserved netblock routed across the internet, or any packet with a "localnet" (127.0.0.1, etc.) IP on an interface other than the loopback interface.  In other words, martian packets mean either something is misconfigured, or someone is trying to do something very sneaky.  RFC 1812 provides more detail about what is and is not a valid routable IPv4 address.

So what do those messages mean?

Let's start with the first message: it contains the destination and source IP addresses, as set in the IPv4 packet, and the network interface on which the packet arrived.  Despite how it reads, the first address is NOT the source address in the packet -- it is, in fact, the destination address!  The 2nd IP is the IP that was contained in the IPv4 packet, but this isn't of very much help in finding the sending host -- remember, the whole point of this message is an indication that the source of the packet is "suspect."

Fortunately, the 2nd line contains a lot of information for us, but it's printed in a format that's very hard to read.  This is the link-layer header of the packet that contained the erroneous source address.  Almost all machines these days use physical connectivity that looks like ethernet, and the log entry above is based on an Ethernet Frame.  (PPP connections will probably look different, but I've never seen this message on anything but ethernet.)  The ethernet header is at least 14 bytes long, and most of us will only see 14 byte headers.  The longer headers are used with 802.1q VLANs, but most people will never see those, so we're going to discuss only the 14 byte headers.

The 14 bytes above, printed in colon-separated hex, contain the destination MAC address, the source MAC address, and either the length or the ethertype code.  The MAC addresses are in standard notation, and the first 6 octets are the destination, with octets 7-12 representing the source.  These will help you track down where the packet came from (though it might be your upstream router, in which case you'll need to do more investigative work).

The last 2 bytes take more explaining.  If the value is less than 0x0600, then it represents the length of the data field for 802.3 compliant ethernet frames.  Greater values, as seen above, are actually EtherType codes.  This will mostly be 0x0800, which means an IPv4 packet is contained in the data segment.  (After all, this error does come from an IPv4 issue.)  Other values might be 0x8100 (802.1q VLAN-tagged packet) or 0x0806 (ARP packet).  There are many more, but you're unlikely to see them in this context.

Long Story Short

Look at bytes 7-12 of the link-layer header.  That's the MAC of the last machine to touch the packet.  Look there for the cause (if it's a router, look upstream from there, and then figure out why you're not filtering the martian packets on your router).  Yes, the source MAC could be spoofed, but then someone's playing games on your local network segment.  (Which probably means they have physical access, or you have a terribly designed network.  Remember, physical access=pwned.)


VPS.net Review

I first heard about VPS.net last March at Drupalcon Chicago.  Having been a Linode customer for a couple of years, I was skeptical at first, but 7 months later, I'm very happy with the level of service vps.net provides.  When I've been working on projects for demanding clients, I've been able to scale my VPS up by adding additional nodes -- either daily or monthly.  After the project was done, I could have scaled back down -- but there's always another project on the horizon!  (One of these days, I'll have to make "sleep" a project to make sure it gets done too.)  While there has been a couple of small downtimes, VPS.net has always been great about providing status updates and letting customers know where they stand.  Additionally, their service people are great and respond quickly via email or twitter. 

VPS.net offers an innovative product -- most VPS providers do not use SANs for their storage needs, but this allows VPS.net to be less susceptible to hardware failures.  In fact, I believe the only downtimes that I have been affected by are those that had an impact on the entire SAN.  Additionally, their ability to scale nodes with a quick reboot by not expanding storage immediately gives you a flexibility that is not available from all providers.  Finally, their pricing structure fits well with my needs -- as you buy more nodes, the new nodes become cheaper.

The only feature VPS.net is missing that I wish they had would be slave DNS servers.  Linode offers this service, as do a couple of other VPS providers, and its something I depend heavily on with my light infrastructure.  (2 vpses.)  I know I could go somewhere like Dyn, Inc. for my DNS hosting, but for something I use for personal and side projects, the Dyn offerings are overkill.  (You can only get IPv6/DNSSEC on their more expensive offerings.)

I have never had an issue with responsiveness or latency on my VPS.  As I do a lot of work via SSH, CPU or network overselling can easily cause problems for me, but that has not been an issue for me.  I am certain the bad wireless at some of the places where I try to get work done causes orders of magnitude more delays than VPS.net's infrastructure.

Overall, though I still have my main site on Linode, VPS.net is a great host for my developmental projects and secondary services.  They take what they do seriously, and they do it well.