Martian Packet Messages

Occasionally, you might see messages like the following in your Linux kernel messages:

martian source 192.168.1.1 from 127.0.0.1, on dev eth1<br />
        ll header: 52:54:00:98:99:d0:52:54:00:de:d8:10:08:00 

There's a lot of discussion out there about what this means, but not a lot about how to trace down the source.  Hopefully this will provide some insight into what the messages actually mean, and how to understand them.

What are martian packets?

A martian packet is one that comes from an "impossible source".  For example, this might be from an RFC 1918 reserved netblock routed across the internet, or any packet with a "localnet" (127.0.0.1, etc.) IP on an interface other than the loopback interface.  In other words, martian packets mean either something is misconfigured, or someone is trying to do something very sneaky.  RFC 1812 provides more detail about what is and is not a valid routable IPv4 address.

So what do those messages mean?

Let's start with the first message: it contains the destination and source IP addresses, as set in the IPv4 packet, and the network interface on which the packet arrived.  Despite how it reads, the first address is NOT the source address in the packet -- it is, in fact, the destination address!  The 2nd IP is the IP that was contained in the IPv4 packet, but this isn't of very much help in finding the sending host -- remember, the whole point of this message is an indication that the source of the packet is "suspect."

Fortunately, the 2nd line contains a lot of information for us, but it's printed in a format that's very hard to read.  This is the link-layer header of the packet that contained the erroneous source address.  Almost all machines these days use physical connectivity that looks like ethernet, and the log entry above is based on an Ethernet Frame.  (PPP connections will probably look different, but I've never seen this message on anything but ethernet.)  The ethernet header is at least 14 bytes long, and most of us will only see 14 byte headers.  The longer headers are used with 802.1q VLANs, but most people will never see those, so we're going to discuss only the 14 byte headers.

The 14 bytes above, printed in colon-separated hex, contain the destination MAC address, the source MAC address, and either the length or the ethertype code.  The MAC addresses are in standard notation, and the first 6 octets are the destination, with octets 7-12 representing the source.  These will help you track down where the packet came from (though it might be your upstream router, in which case you'll need to do more investigative work).

The last 2 bytes take more explaining.  If the value is less than 0x0600, then it represents the length of the data field for 802.3 compliant ethernet frames.  Greater values, as seen above, are actually EtherType codes.  This will mostly be 0x0800, which means an IPv4 packet is contained in the data segment.  (After all, this error does come from an IPv4 issue.)  Other values might be 0x8100 (802.1q VLAN-tagged packet) or 0x0806 (ARP packet).  There are many more, but you're unlikely to see them in this context.

Long Story Short

Look at bytes 7-12 of the link-layer header.  That's the MAC of the last machine to touch the packet.  Look there for the cause (if it's a router, look upstream from there, and then figure out why you're not filtering the martian packets on your router).  Yes, the source MAC could be spoofed, but then someone's playing games on your local network segment.  (Which probably means they have physical access, or you have a terribly designed network.  Remember, physical access=pwned.)


VPS.net Review

I first heard about VPS.net last March at Drupalcon Chicago.  Having been a Linode customer for a couple of years, I was skeptical at first, but 7 months later, I'm very happy with the level of service vps.net provides.  When I've been working on projects for demanding clients, I've been able to scale my VPS up by adding additional nodes -- either daily or monthly.  After the project was done, I could have scaled back down -- but there's always another project on the horizon!  (One of these days, I'll have to make "sleep" a project to make sure it gets done too.)  While there has been a couple of small downtimes, VPS.net has always been great about providing status updates and letting customers know where they stand.  Additionally, their service people are great and respond quickly via email or twitter. 

VPS.net offers an innovative product -- most VPS providers do not use SANs for their storage needs, but this allows VPS.net to be less susceptible to hardware failures.  In fact, I believe the only downtimes that I have been affected by are those that had an impact on the entire SAN.  Additionally, their ability to scale nodes with a quick reboot by not expanding storage immediately gives you a flexibility that is not available from all providers.  Finally, their pricing structure fits well with my needs -- as you buy more nodes, the new nodes become cheaper.

The only feature VPS.net is missing that I wish they had would be slave DNS servers.  Linode offers this service, as do a couple of other VPS providers, and its something I depend heavily on with my light infrastructure.  (2 vpses.)  I know I could go somewhere like Dyn, Inc. for my DNS hosting, but for something I use for personal and side projects, the Dyn offerings are overkill.  (You can only get IPv6/DNSSEC on their more expensive offerings.)

I have never had an issue with responsiveness or latency on my VPS.  As I do a lot of work via SSH, CPU or network overselling can easily cause problems for me, but that has not been an issue for me.  I am certain the bad wireless at some of the places where I try to get work done causes orders of magnitude more delays than VPS.net's infrastructure.

Overall, though I still have my main site on Linode, VPS.net is a great host for my developmental projects and secondary services.  They take what they do seriously, and they do it well.


(Virtually) Setting Up A Test Lab (Part 1)

I've spent a little bit of time today doing something that was long overdue.  I've transitioned most of my day-to-day data to my laptop, so I decided it was time to put my desktop to use as a "virtual lab."

I've set up KVM on my desktop with two virtual machines (so far) in it.  The first one I call "LabManager" -- it's effectively a head node from the "Lab" network out to the real world.

Currently, LabManager has two interfaces -- one of them is bridged to eth0 on the desktop, and the other is only bridged to other VMs.  The other VMs have no direct connectivity.  At this time, LabManager is not forwarding packets, bridging, or routing.  It is running dnsmasq for DNS & DHCP and apt-cacher-ng to allow systems behind it to download packages.  Additionally, LabManager is running a tftp server to perform PXE boot installs.  These installs use a preseed file so the only manual entry is a hostname.  Everything else is automatically setup.

It's my hope that keeping the networks segregated will let me play around with "dangerous" things without posing any risk to my real home network (or the internet).  The use of RFC 1918 IPs and restrictive firewalls should help with this.  I've got the preseed installing puppet, but I haven't set up the puppetmaster yet.  That'll be the 2nd phase, which I'll hopefully get around to by next weekend.

Phase 3 will be providing distributions other than Debian stable via PXE boot.  I'd like to provide both some bleeding-edge work and some older software for pentesting practice.

Suggestions for improvement are welcome!


KSU Cyber Security Awareness Day 2011

Today was the KSU Cyber Security Awareness Day, presented by KSU's Information Technology Services (a sister department to the department I work in), and it was a resounding success!  There were several presentations that had standing-room only attendance, and for good reason.

My personal favorites:

Mike Rothman from Securosis on finding happiness in information security.  Mike's presentation was as much about being happy in your job and in your life as it was about cyber security, but he asked a number of very pointed questions.  Questions about pay/salary, job satisfaction, and life priorities.  I found the questions unsettling, not because of the actual question, but because I realized that I'd been subconsciously thinking those same things for quite a while now.  The take away from his presentation can probably be summed up as "Is what you're doing today getting you where you want to go?"

Rob Ragan of Stach & Liu came to talk about "Google hacking" and the Diggity tool he's put together.  Rob presented a number of very revealing things that can be found just by performing searches on the common search engines, and a number of people in the audience were disturbed to find out how information can be (mis-)handled.  At one point, Rob stated he had run a search on Kennesaw's web presence and found about 11,000 possible pieces of sensitive information.  It was mildly amusing to see the reaction of the members of our ISO until it was realized that he had searched on "ksu.edu" and not "kennesaw.edu".  I hope K-State is aware of the 11,000 possible hits.

Chris Sandy of our own ITS Information Security Office actually presented twice.  One presentation (that I rather enjoyed) was used as filler due to a last-minute cancellation.  Chris talked about the physical aspects of information security, but particularly about physical locks and lockpicking.  I had no idea KSU had a sport lockpicker on campus -- it makes me even more interested in picking up lockpicking, as a sport.

Overall, it was a great one-day event, and I hope that it helped open the eyes of the students, faculty, and staff attending the session.  I know our InfoSec guys worked really hard to put it together, and I think that did an outstanding job.  Having done conference planning before, I know its no easy task, but they pulled it off quite well.


Things I Wish Undergrad Had Taught Me

This is not an attempt to knock any particular program, professor, or course of study.  It's just some things that I think should be included in an undergrad CS program that I don't feel like I got.

  1. Serious study of data structures and algorithms.  While I know how to implement a linked list, structs, classes, vectors, and other data structures, not a whole lot was said about the best use cases for each.  That's something I've had to discover on my own.  And the most complex tree we discussed was the Binary tree.  We never talked about balanced binary trees, red-black trees, or generic n-ary trees.  Although I was taught the general idea behind Djikstra's algorithm, and can tell you the big-O runtime for about a half-dozen sorts, practice implementing them and discussion of their comparative strengths and weaknesses is not something I remember from my undergrad.  Also, there was NO discussion of time-memory tradeoffs involved in implementing some of the algorithms.  In fact, (and I'm embarassed to admit this) I only recently found out about the in-place implementation of quicksort!
  2. How to find your focus.  If there's ever been a real-world example of an NP-complete problem, it's finding your niche.  I'm still searching, and as I get into more things, I'm finding more interests than I am able to exclude.  IT/Computers/Technology is a massive field and even narrowing it a little is hard.  About the only things I've narrowed down are that I don't want to do end-user support, that I don't want to manage people, and that I want to work with/develop Open Source.  Oh, and that I like not doing the same thing every day.  (As it is, my current job is getting on the monotonous end of things.)  I hope I'll find my focus before its too late.
  3. How to develop with others.  This is a skill I've developed over the past few years of the "real world", but I'm not sure everyone I've worked with has gotten it down.  There were too few group projects in my undergrad, and those that I had were comparatively small.  We never had the big software engineering problems, and never really had to develop good APIs for others to depend on.  The division of work never seemed to be "you do the UI, I'll do the database components, and he'll do the business logic."  It was always "you do the UML diagram, I'll do all the code, and you do the final report."  That's not how it works in the real world (ok, well, sometimes it is, but it's not how its supposed to work).
  4. How to effectively use source code management.  Using SCM is critical to any serious development.  Not once in my entire undergrad career was that discussed.  No mention of any SCM.  While my experience in open source had led to me using and understanding SCM, I can say that I've seen how well prepared others are to use it -- and it's pretty scary.
  5. How to do requirements definitions and other software engineering tasks.  When I started my undergrad, there weren't really any dedicated software engineering programs -- everyone did CS or IS.  In the CS side of things, there was one software engineering class.  You can't learn to estimate time, do requirements definitions, manage deliverables, and all the other tasks that go into a software lifecycle in one class.  While I realize not every CS student will end up doing software engineering, the software engineering class should be early in the program (in my program, it was nearly at the end) and those concepts should be incorporated into every major project you do for the rest of your degree.  You've gotta do things more than once to really understand it.
  6. How to do dev/test/prod.  Much like #5, the words "unit testing" never came up in my undergrad program.  There also wasn't really any discussion of maintaining existing software, and of the different environments.  I knew about them, but not from my undergrad, and I've had to learn a lot about them "on my feet."  I'm still trying to get some of our practices at my job into this lifecycle in a sane manner, but it turns out: doing things the right way requires more work up front.  It'll save you in the long run, but it's hard to get that initial investment when it looks cheaper to "fix things later."  (It's not, by the way.  Doing it right the first time is always cheaper.)

I'm still learning a lot -- but if you're not learning, you're stagnant.  There are just some things that make you slap your forehead when you realize how nice it would have been to know those skills 5 years ago.