relocating an HQ network

I wanted to document a few of the hurdles we had to overcome when moving our HQ from one collection of buildings to a shiny new campus.  We pretty much gutted the new campus, pulling back all Ethernet cables in order to re-use where possible.  We designed a routed access model for our new building access layer, and installed a lot of new Ethernet infrastructure.

Routed access notes:
Allows us to re-use configuration templates and know for sure what VLANs are on every switch stack, since the model generally states one VLAN per stack (per network type).  We settled on a Data VLAN, a Voice VLAN, a Building Management / Security infrastructure VLAN, and an Audio / Video VLAN (for conference room infrastructure).  The conference rooms are all fairly complex, with up to 30 network-connected devices per CR, so we chose to give it a separate VLAN to segment it from the rest of the infrastructure.  The BMS / Security VLAN allows us to also segment it from the rest of the network.

A few things we would change about the structured cabling include:
Re-use of cabling is a great idea, and having 2 ports per cube is good for expansion, but the reality is that most people only ever use a single port, and running additional cables to the cube areas is not that big of a deal.  We like to reduce the amount of patch cables in our wiring closets, so we typically deploy a single 48-port switch per patch panel and use 1-foot patch cables.  If we want to continue this model, a lot of unsed ports are present and this could be considered wasteful.  The big issue we had is that since we deployed a significant increase in structured cabling, we couldn’t justify the additional switching infrastructure that would be required to terminate every port.  The end result is a bit of a mess in the wiring closet.

Routed access allows us to remove spanning-tree from the equation (although we don’t disable STP, we could).  It also allows both uplinks to be active and equal-cost load balancing to take place, potentially reducing the need for lots of uplinks.

Routed access could be considered wasteful from an IP addressing perspective, but since we use RFC1918 space internally, we didn’t consider this much of an issue.

Server infrastructure move planning.

We procured a circuit from a major provider with the intention of bridging our server segments at L2 in order to reduce the need to re-IP anything.  This was a great success although we had some issues at first.  If you do this, be sure to procure the right kind of circuit – the one we purchased was not a true L2 circuit and had limitations on the number of MAC addresses permitted across the link.  This took a while to troubleshoot, and was extremely frustrating.  The solution we came up with was to deploy two 6509 switches we had ‘laying around’ and running l2tpv3 over the link.  This proved to be a bit frustrating as well, since we had to ensure full-size frames could pass.  Thankfully, the circuit had an MTU of 1518 so we did have headroom, but until we did a shut/no shut of the interfaces it was not working (also a bit unnerving).

Multiple l2 circuits would have been great, with the obvious caveat that we would need to use port aggregation or STP in order to break the loop.

This is all I can think of right now, I’m sure there will be more and I’ll update this post..

i know nothing

Let me just begin by saying that I don’t know anything.  There, I said it.  And I truly believe it.  The more I learn, the more I realize I don’t know shit.


Now that I’ve gotten that off my chest, let me talk a little about some things I do know, and I’m OK at (read: not GOOD at, just OK).  I’m OK at reading comprehension, and I’m OK at spelling.  I’m also good at being curious, but that seems more like an innate ability.  I suspect most people in my profession have the same innate curiosity – we all used to take things apart as kids (and much to our parents’ chagrin, not put them back together).


The DefCon 101 talks were excellent, and got me thinking about level setting.  Lostboy gave a talk about baselining knowledge in IT, and I thought I would contribute some of my thoughts.


I have the opportunity in my position to interview people for jobs in IT, including systems administration, network administration, and security.  I often am disappointed in the candidates’ lack of basic knowledge of how systems are put together.


Again, I don’t claim to know anything, but here are some basic things that I always ask candidates.  My belief (and it could be unfounded) is that if you’re going to be in IT, you should know the answers to these questions.  Especially if you’re interviewing for a sysadmin position.


1. How does DNS work?


The answer can be as simple as, well, it maps names to IP addresses, which is true.  But there is so much more to it.  I don’t claim to be a DNS master, or anything like that – I don’t know crap about the inner workings.  However, I do know, and I expect EVERYONE in IT to know what an “A” record is.  If you don’t know, look it up.  Look up what a “PTR” record is while you’re at it, and a CNAME record too.  These are SIMPLE things that you should know if you’re coming to an interview as an IT person.


2. How does DHCP work?


Again, a simple question, with a simple answer.. it assigns IP addresses dynamically.  However, the a real IT person will know HOW DHCP works – the host sends a broadcast message and the DHCP server responds.  If it’s not on the same layer 2 domain (look it up), the router will forward the DHCP request on to the DHCP server if it is configured to do so.


If you want to even think about getting into security as a profession, you should know much more about the above protocols.  Down to specifics on what the packets look like and how to manipulate the protocols to ‘trick’ hosts.


Anyone who interviews for a networking position on my team had better know much more about these protocols than the basics, and they will need to know about other things that are pervasive in today’s networks.  Things like:


1. MTU and Path MTU discovery


What is the MTU?  How does Path MTU discovery work?


2. How does traceroute work?


Not “it traces the route between points on a network”, rather, HOW does this protocol work?  For extra credit, how does a UNIX machine differ from a Windows machine in how it performs the traceroute utility?  And no, the answer isn’t “Windows uses tracert, and UNIX uses traceroute”.


Those of you who are reading this post because you googled my name after seeing that I’ll be interviewing you tomorrow, good on you for doing a little research before coming to the interview.  Just be prepared.

Giving Back

I went to DefCon 2012 this year for the first time ever and I must say that it was a great experience.  I met too many people to count, and learned a good deal about some security topics.  I also learned some information about password hashes, which is fantastic b/c it’s some timely information with all of the news recently about hashes being leaked.

I’ll post more about that as the material becomes available online.  If you live in San Diego and want to meet to discuss security topics, drop me a line, I’d love to meet up.  My twitter handle is niemesrw.


I have been listening to Exotic Liability for the past few months – the guys on there are a fantastic resource for those of you who are interested in computer security.  Their expertise in penetration testing is unparalleled, and they have some interesting guests on as well.  Plus, unlike some podcasts, they’re not trying to sell you anything.


Anyway, my re-entry to the security field has left me wondering how I can give back to the community.  I have a lot of expertise in corporate IT, networking, and systems, and I believe this gives me a good window from which to view security in a big-picture sense.  The bottom line for most of us is we’re all strapped for resources, so the ‘next big thing’ is sure to only cause us more headaches since it won’t have anyone to manage it.  It got me thinking about some recent projects I have at work – we’ve already got some web services but we will be adding more as time goes on.  Some of which will have access to not just products we’re offering customers, but some of our enterprise applications.  This got me thinking about how to best secure these applications.  The old adage of putting a server in the DMZ is over.  We have all of these technology solutions to choose from now as well – web-application firewalls, reverse proxies, etc.  Nothing, however, compares to actually implementing a real security program that your developers follow.


The guys over at OWASP are a tremendous resource for how to implement secure web applications.  They tell us that one of the most important things we can do is to perform a risk analysis.  What data will the application have access to?  And what are the risks of compromise?  Only then can you associate controls to help mitigate the risk, and some of these controls are very basic and non-technological.  You need some sort of secure development program, and a security testing program.  Technical measures, like WAFs, can be defeated, but writing secure applications are much more difficult to defeat.


Anyway, I ramble, and it’s been a long week already.  I hope this is just the first in a series of posts!  But we all know how that goes in the blogger world.