System Overlord

A blog about security engineering, research, and general hacking.

2 Weeks at Google

11 Mar 2012 in Work (Reading time: 1 minute)

Two weeks at Google have been... amazing.  There's a lot that I can't talk about, but I can feel comfortable in confirming some of the things you hear about Google:

  • The people are insanely smart.
  • The scale blows your mind as a Noogler (new Googler).
  • The food is great.
  • It has culture.

I'm a "Site Reliability Engineer" which is a job title that may not exist anywhere else.  It's basically production-oriented operational engineering: keeping production systems running and making them run better.

Ann and I have found a new place, so we'll be moving there from the corporate housing next week.  It'll be nice to get our stuff back and get settled in.

(All opinions expressed here are mine and not my employers.  I will not comment on or discuss Google policy, unreleased products, or other proprietary information.)

The End of a Chapter

15 Feb 2012 in Personal (Reading time: 2 minute)

I'm not usually one for reflective personal blog entries, but some events require a brief mention: today was my last day at KSU, and it was an incredibly surreal day.  Though I've known this day was coming for over a month, it is still hard to believe that it got here.  In many ways, today felt like any other day: the work was similar, things needed to get done.  In other ways, there was an 800 pound gorilla in the room: everyone knew that tomorrow I wouldn't be coming to work.  When I finally cleared out my office, the finality of what was going on really hit me.

There are friends I have made at KSU that are like none other, and I will genuinely miss them.  I hope we will stay in contact and I hope to get back to the Atlanta area sometime and visit them, or perhaps they will make it out to the bay area.  Unfortunately, friendships seem to have a tendency to dissolve too easily with distance, but one can always hope that things will be different.

Today, one chapter of my life closes.  For now, a brief interlude to relocate to the Bay Area, and then a new chapter begins in 12 short days!

My Time at KSU

02 Feb 2012 in Work (Reading time: 5 minutes)

As you might have seen, I'm leaving my position at Kennesaw State University to take a position as a Site Reliability Engineer at Google.  This is something I'm very excited about, but I thought I'd take a look back at my time at KSU as I approach the end.  It's worth mentioning that I'm not leaving KSU because of KSU, but because this is an opportunity I just could not turn down.  For the most part, I like my position at KSU, and I really like most of the people that I work with.  There's a particular group that's become three of my closest friends and one treasured acquaintance.

As much for my own memories as for the readers of this blog, I'm going to take a look back at some of the projects I've enjoyed most over the three years I've been at KSU.  I've been fortunate enough to have taken on many of these projects of my own accord, and they've become successful and play varying roles in our department's operations.  I'm also fortunate that, though I may have spent (many) more than 40 hours a week working on some of these, I like what I do enough that I can't find the line where work ends and personal time begins.

So here's the look back at my time in Advanced Computing Services (formerly Online Development Group, formerly Online Learning Services).

Centralized Authentication for Departmental Resources

When I first started at KSU, every server run by the department used local users for logins, and different applications had local login systems.  Now, all shell accounts go back to an LDAP server, home directories are mounted via NFS, and many web-based applications point to the same LDAP directory.  This has reduced the number of passwords to remember and simplified system administration. 

High Availability for Production Webservices

Using Linux-HA, MySQL replication, and shared storage, our production Drupal web environment has high availability built in to reduce the risk of downtime.  Since implementation, we've achieved greater than 99.9% uptime, even through power generator replacements, network upgrades, and even the occasional configuration glitch.  (And yeah, that last one was me.)  Of course, I'm sure I've jinxed myself now, but it's been a good run of uptime.

KSU Video

In about December 2009, we were approached with something a little bit outside our normal comfort zone: develop a system to record IP-based cameras used for mock sales competitions in our College of Business.  The system needed to be web-based, precisely time recordings, make the finished recordings available on a website, control recording lights in the practice area, and also live-stream the footage to rooms full of judges.  Oh, and its inaugural run would be a competition with dozens of schools from several countries the following March.  Despite numerous mis-estimations and "surprises" along the way, we met the deadline and the competition was a success.  We're in the process of upgrading the system to be flexible for other uses on campus, including situations like nursing clinical training and professional speaking.

VMWare Implementation

I had the pleasure of deploying a VMWare cluster that now hosts a few dozen virtual machines, allowing us to quickly spin up new development environments to test applications and provide redundancy for low-utilization services.

Puppet

This is the one that got away -- I was working on deploying Puppet for configuration management on all of our servers.  Unfortunately, this may or may not ever get deployed, but I do believe that configuration management is key to any significant production environment.

Most Importantly

My biggest accomplishments have been the friends I've made and the things I've learned.  Short of Alzheimer's or a traumatic brain injury, I will never forget many of my days -- even if there are a few I wish I could forget. 

 

Big Changes

04 Jan 2012 in Personal (Reading time: 1 minute)

Today I did one of the hardest things I have had to do: I turned in my notice that I would be leaving Kennesaw State University in February.  It was hard because I consider my management and many of my coworkers to be friends and I genuinely do enjoy my job, but I now have the opportunity to start the next chapter of my life.  At the end of February, I'll be moving to California and starting at Google as a Site Reliability Engineer!

It's a pretty exciting opportunity for me, and I'm looking forward to it very much, but I will miss the people at Kennesaw.  It's been a good time, and had this opportunity not arisen, I probably would have stayed there for quite a while longer.

Change is good, even if it is a bit hard to do.

Recruiters!

13 Dec 2011 in Work (Reading time: 4 minutes)

There hasn't been a lot of updates lately, and I apologize... between contract work, end of the semester, and other personal issues, writing blog posts has slipped into the cracks.  Hopefully that'll be fixed as I've finally completed my 2nd 1st semester of grad school.

I am occasionally contacted by recruiters of various sorts.  Most of them I just ignore, particularly those that provide no details on who the employer is, what the compensation package might be, etc.  As a rule, I find 3rd party recruiters who contact me out of the blue are probably contacting anyone whose website includes the word "Linux".  (Although I've received more than one recruiter looking for Windows/.NET software development...)  The most recent one was a real gem though.  No names -- I'm not trying to call him/her out individually, just to shed some light on the vagueness that is the world of recruiting.  I've trimmed the email down to the salient parts:

Hello David,

Hope you are doing good.

I am currently looking for Senior level Linux Systems Administrator, full time position with a well established startup in Bay Area, CA.

I understand that you are currently studying, but wanted to check if you would know someone who would be interested. Open to considering mid or junior level candidates as well if they have the right experience.

There are a number of things that immediately scream red flag from this post:

  1. While I am currently pursuing an MS, I think my resume makes it pretty obvious that I'm also working full-time.  If not, that's something I need to fix.
  2. If somebody has the "right experience" to be a "Senior level Linux Systems Administrator", wouldn't they now be a "Senior level Linux Systems Administrator"?  After all, the distinction between a mid- or junior-level administrator and a senior one, in my mind anyway, is the experience.
  3. This is my favorite one: it's with a "well established startup."  I don't know about those out in the valley, but in my mind, you're either "well established" or a "startup", not both.  You can be a "well funded startup" if you want to give someone confidence about their odds of getting a paycheck, but a "well established startup" is just an effort to say "we want them to think it'll feel like a cool startup, but pay like a big business!"

Don't get me wrong: I'm flattered by the contacts from recruiters.  I must be doing SOMETHING right.  There have even been a couple of interesting ones directly from major Silicon Valley players, but usually, the 3rd party ones aren't usually that exciting.  But maybe its a sign that things are turning around, at least in the IT world.