choose your own identity

I’ve been working with my team to come up with some visionary thoughts around where we think our services will be in the next 3-5 years.  In addition to the typical CMMI-speak, we did come up with a few ideas that I think are revolutionary from a corporate IT perspective.  The one term I came up with is “choose your own identity” which is basically the idea that one can choose to use their facebook or twitter account for access to corporate resources.  There are a number of implications to this, but here’s basically how it would work: 

  1. User is provisioned in HR system (workday/etc)
  2. Employee starts work, logs in to HR system
  3. Employee associates their externally managed credential to their Employee ID
  4. Employee is able to login to Corporate resources

This idea is not new in the world of Internet services, but it would be new to corporations which are used to controlling a user’s identity.  It would also require that certain false security controls are in place (password length, change history, complexity, etc).  I think the credential would be enhanced by a second factor – for example, we are thinking about issuing certificates to every user, and every device, in order to restrict access to the system.  In addition, data protection becomes more of a hard requirement through the use of DRM or other encryption.

relocating an HQ network

I wanted to document a few of the hurdles we had to overcome when moving our HQ from one collection of buildings to a shiny new campus.  We pretty much gutted the new campus, pulling back all Ethernet cables in order to re-use where possible.  We designed a routed access model for our new building access layer, and installed a lot of new Ethernet infrastructure.

Routed access notes:
Allows us to re-use configuration templates and know for sure what VLANs are on every switch stack, since the model generally states one VLAN per stack (per network type).  We settled on a Data VLAN, a Voice VLAN, a Building Management / Security infrastructure VLAN, and an Audio / Video VLAN (for conference room infrastructure).  The conference rooms are all fairly complex, with up to 30 network-connected devices per CR, so we chose to give it a separate VLAN to segment it from the rest of the infrastructure.  The BMS / Security VLAN allows us to also segment it from the rest of the network.

A few things we would change about the structured cabling include:
Re-use of cabling is a great idea, and having 2 ports per cube is good for expansion, but the reality is that most people only ever use a single port, and running additional cables to the cube areas is not that big of a deal.  We like to reduce the amount of patch cables in our wiring closets, so we typically deploy a single 48-port switch per patch panel and use 1-foot patch cables.  If we want to continue this model, a lot of unsed ports are present and this could be considered wasteful.  The big issue we had is that since we deployed a significant increase in structured cabling, we couldn’t justify the additional switching infrastructure that would be required to terminate every port.  The end result is a bit of a mess in the wiring closet.

Routed access allows us to remove spanning-tree from the equation (although we don’t disable STP, we could).  It also allows both uplinks to be active and equal-cost load balancing to take place, potentially reducing the need for lots of uplinks.

Routed access could be considered wasteful from an IP addressing perspective, but since we use RFC1918 space internally, we didn’t consider this much of an issue.

Server infrastructure move planning.

We procured a circuit from a major provider with the intention of bridging our server segments at L2 in order to reduce the need to re-IP anything.  This was a great success although we had some issues at first.  If you do this, be sure to procure the right kind of circuit – the one we purchased was not a true L2 circuit and had limitations on the number of MAC addresses permitted across the link.  This took a while to troubleshoot, and was extremely frustrating.  The solution we came up with was to deploy two 6509 switches we had ‘laying around’ and running l2tpv3 over the link.  This proved to be a bit frustrating as well, since we had to ensure full-size frames could pass.  Thankfully, the circuit had an MTU of 1518 so we did have headroom, but until we did a shut/no shut of the interfaces it was not working (also a bit unnerving).

Multiple l2 circuits would have been great, with the obvious caveat that we would need to use port aggregation or STP in order to break the loop.

This is all I can think of right now, I’m sure there will be more and I’ll update this post..

i know nothing

Let me just begin by saying that I don’t know anything.  There, I said it.  And I truly believe it.  The more I learn, the more I realize I don’t know shit.

Now that I’ve gotten that off my chest, let me talk a little about some things I do know, and I’m OK at (read: not GOOD at, just OK).  I’m OK at reading comprehension, and I’m OK at spelling.  I’m also good at being curious, but that seems more like an innate ability.  I suspect most people in my profession have the same innate curiosity – we all used to take things apart as kids (and much to our parents’ chagrin, not put them back together).

The DefCon 101 talks were excellent, and got me thinking about level setting.  Lostboy gave a talk about baselining knowledge in IT, and I thought I would contribute some of my thoughts.

I have the opportunity in my position to interview people for jobs in IT, including systems administration, network administration, and security.  I often am disappointed in the candidates’ lack of basic knowledge of how systems are put together.

Again, I don’t claim to know anything, but here are some basic things that I always ask candidates.  My belief (and it could be unfounded) is that if you’re going to be in IT, you should know the answers to these questions.  Especially if you’re interviewing for a sysadmin position.

1. How does DNS work?

The answer can be as simple as, well, it maps names to IP addresses, which is true.  But there is so much more to it.  I don’t claim to be a DNS master, or anything like that – I don’t know crap about the inner workings.  However, I do know, and I expect EVERYONE in IT to know what an “A” record is.  If you don’t know, look it up.  Look up what a “PTR” record is while you’re at it, and a CNAME record too.  These are SIMPLE things that you should know if you’re coming to an interview as an IT person.

2. How does DHCP work?

Again, a simple question, with a simple answer.. it assigns IP addresses dynamically.  However, the a real IT person will know HOW DHCP works – the host sends a broadcast message and the DHCP server responds.  If it’s not on the same layer 2 domain (look it up), the router will forward the DHCP request on to the DHCP server if it is configured to do so.

If you want to even think about getting into security as a profession, you should know much more about the above protocols.  Down to specifics on what the packets look like and how to manipulate the protocols to ‘trick’ hosts.

Anyone who interviews for a networking position on my team had better know much more about these protocols than the basics, and they will need to know about other things that are pervasive in today’s networks.  Things like:

1. MTU and Path MTU discovery

What is the MTU?  How does Path MTU discovery work?

2. How does traceroute work?

Not “it traces the route between points on a network”, rather, HOW does this protocol work?  For extra credit, how does a UNIX machine differ from a Windows machine in how it performs the traceroute utility?  And no, the answer isn’t “Windows uses tracert, and UNIX uses traceroute”.

Those of you who are reading this post because you googled my name after seeing that I’ll be interviewing you tomorrow, good on you for doing a little research before coming to the interview.  Just be prepared.

Giving Back

I went to DefCon 2012 this year for the first time ever and I must say that it was a great experience.  I met too many people to count, and learned a good deal about some security topics.  I also learned some information about password hashes, which is fantastic b/c it’s some timely information with all of the news recently about hashes being leaked.

I’ll post more about that as the material becomes available online.  If you live in San Diego and want to meet to discuss security topics, drop me a line, I’d love to meet up.  My twitter handle is niemesrw.

I have been listening to Exotic Liability for the past few months – the guys on there are a fantastic resource for those of you who are interested in computer security.  Their expertise in penetration testing is unparalleled, and they have some interesting guests on as well.  Plus, unlike some podcasts, they’re not trying to sell you anything.

Anyway, my re-entry to the security field has left me wondering how I can give back to the community.  I have a lot of expertise in corporate IT, networking, and systems, and I believe this gives me a good window from which to view security in a big-picture sense.  The bottom line for most of us is we’re all strapped for resources, so the ‘next big thing’ is sure to only cause us more headaches since it won’t have anyone to manage it.  It got me thinking about some recent projects I have at work – we’ve already got some web services but we will be adding more as time goes on.  Some of which will have access to not just products we’re offering customers, but some of our enterprise applications.  This got me thinking about how to best secure these applications.  The old adage of putting a server in the DMZ is over.  We have all of these technology solutions to choose from now as well – web-application firewalls, reverse proxies, etc.  Nothing, however, compares to actually implementing a real security program that your developers follow.

The guys over at OWASP are a tremendous resource for how to implement secure web applications.  They tell us that one of the most important things we can do is to perform a risk analysis.  What data will the application have access to?  And what are the risks of compromise?  Only then can you associate controls to help mitigate the risk, and some of these controls are very basic and non-technological.  You need some sort of secure development program, and a security testing program.  Technical measures, like WAFs, can be defeated, but writing secure applications are much more difficult to defeat.

Anyway, I ramble, and it’s been a long week already.  I hope this is just the first in a series of posts!  But we all know how that goes in the blogger world.


Having multiple equal-cost paths (Equal-Cost MultiPath) to a destination is nice, but you need to understand how it will work in a Cisco network.  By default, per-packet load balancing is disabled.  Anyone know why?  Leave replies in the comments section & I’ll pick the best answer.

It’s generally bad to per-packet load-balance, so IOS will hash on a few fields of the packet & ensure that subsequent packets in that ‘flow’ will take the same path.

In IOS, you can determine the exact path a packet will take through the use of the command “show ip cef exact-route“.  Of course, the Catalyst 6500 / 7600 line has a different command “show mls cef exact-route

Ivan Peplenjak has a good article on ECMP located here.

I was searching for an equivalent command on NX-OS and located the “show routing hash” command.  It works as advertised.

LISP: Locator/ID Separation Protocol available on IOS

Cisco IOS release 15.1(1)XB introduces some LISP features.  LISP is a relatively new protocol whose aim is to separate two functions contained within an IP address.  A host IP address contains both the Endpoint Identifier (EID) and the Routing Locator (RLOC).  What this basically means is that the IP address not only indicates the specific host, but also indicates how to locate the host on an IP network.

LISP is just one of the latest examples of abstracting a problem in order to more easily solve it.  Think about how the DNS solved an early problem: human inability to remember large amounts of unstructured data in the form of IP addresses.  Is it easier to remember or  In effect, the DNS decoupled the EID ( and locator (the name).

The problem is that IPv4 has no way of separating the host from the path.  If you trace the route to a host with IP address, each intermediate hop uses the host IP address in order to find out which interface to egress the packet.  This presents a few problems:

  • Mobility
  • Scalability
  • Multihoming
Mobility: host movement between branches of the topology will result in a host being unreachable (if the address doesn’t change).

Scalability: either your topology can match your addressing or your addressing can match your topology.  If not, scaling the network will be difficult since the network will contain a large amount of state.  Large amounts of state are not necessarily bad, but convergence within that network can take a large amount of time.

Multihoming: a host that resides on multiple segments will require EIDs for each segment.

Each of these problems have kludgey hacks to solve them.  For instance, a multihomed host can use a loopback address for its EID.  The kludgey part will mean that upstream devices will still need to reference the closest interface’s EID as a next-hop in order to reach the loopback.  This presentation explains it better than I can.

LISP is designed to solve these problems by decoupling the EID from the RLOC, and introduces a few new types of devices.  No end-user / end-device changes are necessary because an intermediate device simply maps & encapsulates the packet from one end of the network to the other.

You can get started with LISP using the following guides from Cisco:

LISP Lab Testing Application Note
Cisco IOS LISP Configuration Guide

And here’s a good article from the IP Journal.

Unfortunately they haven’t implemented all of the interesting parts of LISP, but you can see how a gateway router performs the map & encap function.  I just tried it in a lab and it works great.

For a deeper discussion on the challenges presented by IP, I highly recommend reading Patterns In Network Architecture by Jon Day.

The First 90 Days

The First 90 Days

I recently started a new job and have been considering what to implement during the first period of my tenure.  The following are my restrictions / requirements:

  • No budget
  • Services must fit in a VM
  • Scalable (read: easily supported / maintained)

With that in mind, I’ve decided to implement a few things:

  • syslog-ng syslog repository
  • rancid

Syslog-NG is a great syslog server replacement, and there are a number of great management / reporting tools as well.  It’s “free” and fits easily in a standard U*IX environment.

I’ll also be installing LogZilla (aka php-syslog-ng) and putting everything in a mysql database.

RANCID is a fantastic tool that will archive your network configurations & let you know if things have changed.  Some folks have integrated the CVS repository that RANCID uses with CVSWEB, so I’ll be looking into that as well.

Ok, that’s nice, but what does that have to do with the CCDE?

Nothing directly, but it does have everything to do with the care & feeding of a network.  You can’t know what’s going on with your devices without consolidating the messages they are producing, and without configuration backups / auditing you can be in trouble if a system loses its configuration or is changed.

The syslog-ng design will probably involve a hierarchy of some sort, where each site has a local repository, and all of these feed back to a central server that is being backed up.  I don’t really know yet, but syslog-ng gives you the freedom to do so.

How about a pretty picture?  I like pretty pictures:

CCDE Topic: docs I found useful

CCDE Topic: docs I found useful

Step One: the written (read this even if you already passed!)

I found the written outline to be the most useful preparation not only for the written exam, but also the practical.  I went through all of the topics outlined and made sure I understood exactly what they were trying to get at.  Honestly, there were a number of items I had never heard of, so doing the research felt like I was accomplishing something.  I’ve posted a few of the written items on this blog, like What is the issue with re-marking and OoO packets? and What is fate sharing, and what is its impact?.

If you want to see more about the security portion of the written outline, be sure to check out this post too.

In case you can’t tell, I’m saying that even if you passed the written easily, you probably want to go back & make sure you understand everything that’s covered on the outline.

Step Two: the practical

This is the difficult part, but I’ll try to give a summary of what I found most useful for me.  Of course, you may already be an expert in one of these topics and won’t need to review it (but you probably should anyway).


High Availability Campus Network Design–Routed Access Layer using EIGRP or OSPF

Enterprise QoS Solution Reference Network Design

RFCs (in no particular order):

Multicast RFC1112


If you have access to the Partner Education Connection you’re in luck.  If not, your best bet is to check out elementK.  Your company may already have a subscription.  I’m sure there are others with equivalent training, but I’m speaking specifically about the path I took.

I took the online MPLS SE training, reviewed the Multicast training, and reviewed the EIGRP training.

I’ll go into the specific courses in a later post, as well as a book listing & the Cisco Live / Networkers materials I found useful.

CCDE certified, now what?

So it’s been a few weeks since learning that I passed the practical, and other than NOT having to attend the next practical in December, not much has changed. I think the marketplace is still trying to determine the value of this certification, and my feeling is that some employers may think it’s nothing more than a written exam. They’re right, of course, but it is a hard written exam. It remains to be seen just how / when employers will start to recognize the value. I think if the CCDE team published some statistics on the pass/fail rate it might help (along with the level of experience most taking it have), but I’m not holding my breath.

The path I took in studying for this exam made me a much stronger network engineer, and IMO that’s the whole point. Now the question will be whether or not others become stronger engineers in their pursuit of the CCDE.

Some people have emailed me asking how they should prepare for the practical, and I would say the following has been invaluable preparation for me:

1. Get a job where you’re tasked with maintaining & troubleshooting a production network.
2. Never be satisfied when something magically starts working – troubleshoot & solve real-world issues and find the root cause (wireshark is your friend).
3. Visualize how you would change the network to solve systemic problems with the network (and maybe even implement some changes).
4. Work with the applications people to help them understand how the network affects their applications (and vice-versa).
5. Become the ‘buck stops here’ person, meaning that if you cannot solve the issue, nobody can. You need to take ownership of the entire system – don’t pass off issues to the applications people / server people.

In short: you cannot just have book knowledge to pass this exam, you will need real-world experience troubleshooting and designing solutions to solve systemic problems. If you gain this experience, you won’t need the CCDE to prove your level of knowledge, your CV will speak for itself.

My background is in UNIX systems administration, networking, and security. I’ve also worked in several industries including Financial Services, SP, and DoD. I worked in small companies where I was in charge of everything, and large organizations where I was in charge of just one small part of the network. In all cases, I knew not just what I was responsible for, but all (or most) of the applications and systems running within the system. Personally, I believe this experience was key preparation for the CCDE. You need to understand real-world design issues, not just the best practice picked up from reading in a book. You need to never be satisfied with the design either – always look for ways to improve it. This quote from fellow blogger Jeremy Filliben sums it up nicely:

“When I take two eggs out of the carton, I take one from each row. It isn’t more correct, but it makes me feel better.”

Isn’t that the whole point anyway? If you have a passion for this stuff it will show, and you will be rewarded with an infinite number of interesting problems to solve.


CCDE 20090006

Yesterday I received the letter from VUE confirming my pass of the CCDE practical exam: 352-011. I am now CCDE #20090010. At least according to VUE – the Cisco site is still not updated with a number. Correction: the Cisco site was updated with a number (20090006). The results were mailed on 11/13/2009 so it’s taken at least 3 days to update the Cisco system.

Needless to say, I’m thrilled.

I will begin posting some info on my path in the hopes that it will shed some light on the process for others. I will say, it certainly helped that I had the requisite 10 years of industry experience before attempting the practical. It may be possible without this, but the breadth is wide enough, and the depth just deep enough that you can’t really study for this exam.

That being said, I did study for it, and learned quite a bit over the last year and a half of reading & watching VoDs. The reading list is good (and yes, I recommend you read all of them), but there are a few books that should be highlighted:

Definitive MPLS Network Designs
Optimal Routing Design

I’m old enough to remember reading (several times) the precursor to Optimal Routing Design : Advanced IP Network Design, which is not on the reading list, but I recommend with the caveat that not all of the material is relevant anymore.

This gets into one of the most difficult parts of any exam: ignoring or realizing what is no longer relevant.

Anyway, thanks everyone for the congratulations, and keep tuned in for more.