Recently in work Category

Now Hiring!

As mentioned in my new boss's blog ... ... if you or anyone you happen to know is looking for a new role, would love to hear from them. I have a few openings at the moment in my organization (AOL Network Engineering & Operations).

Technical Security Engineer -- -- focused on all things Network Security

Site Reliability Engineer -- -- have a couple of these openings, focused on CDN, server load balancing (both local and global), and DNS

Application Network Manager -- -- managing the 10+ person organization of Site Reliability Engineers across the world

You can apply for any of those at the links above.

Also, if you have any feedback, here is a new take on the Tech Sec Eng job description, would appreciate any feedback you might have... comments welcome!

It really annoys me how large the font and icon sizes are for Wireshark in MacOSX. I found someone else who had posted how to adjust the sizes, and for my own reference as well as anyone else who happens to find this page here is the diff:

diff ~/Wireshark-correctfontsize-pre_gtkrc /Applications/
< gtk-font-name="Lucida Grande 9"
> gtk-font-name="Lucida Grande 12"
< gtk-icon-sizes = "gtk-menu=16,16:gtk-dialog=24,24:gtk-dnd=32,32:gtk-button=20,20:gtk-large-toolbar=16,16:gtk-small-toolbar=10,10:inkscape-decoration=6,6"
> gtk-icon-sizes = "gtk-menu=16,16:gtk-dialog=48,48:gtk-dnd=32,32:gtk-button=20,20:gtk-large-toolbar=24,24:gtk-small-toolbar=16,16:inkscape-decoration=12,12"

I also hacked a quick script to just edit the file since it gets replaced every time you download a new version. You can find the script at:

Also, if you're having problem with the font characters all coming up as Rectangles on MacOSX 10.6.3 and certain development versions of Wireshark (1.3.5) for example, you may want to check out Wireshark Bug 4697. You can change line 73 of the /Applications/ script to:

sed 's|${HOME}|'"$HOME|g" "$TOP/etc/pango/pangorc" > "${HOME}/.wireshark-etc/pangorc"

Open Source System Management


This is mostly notes for myself... however, it was a useful post on NANOG that I wanted to keep track of. So I'm listing some packages to manage systems and devices via SNMP, syslog, daemons on the hosts, etc... and of course including graphing of time series data and such too.

Hyperic -
OpenNMS -
opsview -
osimius -
PandoraFMS -
Zabbix -
Groundwork -
Nagios -
Zenoss -
OpManager -
Orion -
BigBrother -
Argus -
Munin -
Spiceworks -
Cacti -

Some more updates from the ongoing email thread:

Xymon -

NMIS - - - - a (very current) fork of Nagios - another netflow tool - For those with larger campus networks - audit tool for different network devices

Wireshark Coloring Rules - Updated

I've updated my Wireshark Coloring Rules. They work on a 2.6Ghz Mac Book Pro running 10.5.4 and Wireshark 1.0.2. I had to remove some of the Analysis Flags due to TCP & CRC Offloads of the Mac Book's Ethernet NIC... well, I think. :)

NANOG43 Notes

NANOG43 is in Brooklyn, NY this year. These are my rough notes if I have any. Also, I'll not link to every topic if I'm not all that interested in the subject. You can find the agenda with links to most if not all of presentations. If you have questions about any of them, feel free to seek me out.

Security BOF
Updates from various Security groups and call for participation.

Community Meeting
Pretty quiet meeting actually. MLC wasn't nearly the hot topic that I believe everyone was expecting.

Jay Adelson, CEO of Digg
Views from the Other Side: Confessions of a Guilty Customer

Lots of discussion about being the customer side. Mostly light like the Keynotes have been at NANOG so far, but had some nuggets as all things can. Jay is a very good speaker, but being a serial entrepreneur he kind of has to be.

Coolest thing from pres was at the end when he put up:

Peering Wars: Lessons learned from the Cogent-Telia Depeering
Martin Brown, Alin Popescu, & Earl Zmijewski, Renesys Corporation

Favorite quote: "Being a tier 1 is not easy. You will be punished if you are perceived to be in a position of weakness."

Internet Traffic Trends -- A View from 67 ISPs
Craig Labovitz, Danny McPherson, Mike Hollyman & Scott Iekel-Johnson, Arbor Networks

78 ISPs now... sharing data every hour, 5 minute aggregate data
5 MSOs, 4 Tier1s, 15 Tier2s, 4 Content Providers, 1 R&E, rest did not self-classify
1300 routers

YouTube Outage, Layman Explanation

YouTube went down on Sunday the 24th of February. A good summary of the events (at least for geeks) can be found at:

There has been LOTS of comments on NANOG all weekend about it. NANOG is the North American Network Operators Group, generally a bunch of folks in the Americas that participate in some way in the operations of networks and the Internet. You can see the archives at: and see some of the mails that flew back and forth regarding the outage.

I thought I'd provide a summation for the one or two folks who read my blog but aren't geeks, or network geeks at least and maybe teach a little about networking in the process.

Basically on Sunday the Pakistan Government told Pakistan Telecom (along with other ISPs in Pakistan) to block YouTube. Pakistan Telecom decided the best way to do this was to "black hole" some YouTube routes. Black holing traffic on the Internet is basically forcing traffic to a different location and then throwing that traffic away. One of the most drastic ways you can accomplish this is by using the first decision in deciding where next to send a packet. That decision can be described as "Longest Match Wins" in routing.

Think about Longest Match this way. Say you have an address of 221 Main Street, Fairfax, Virginia. Now say you had four paths in front of you, the first path said "Virginia", the second path said "Fairfax, Virginia" and the third path said "Main Street, Fairfax, Virginia" and the fourth path said "221 Main Street, Fairfax, Virginia". You would chose the fourth path because it takes you directly to where you need to go.

So, Pakistan Telecom decided to cheat a bit and say, instead of just going to "YouTube", follow these paths to "West Coast You Tube" and "East Coast You Tube". I've greatly simplified how You Tube breaks up their IP addresses, but the concept holds for this example.

Now what SHOULD have happened is that Pakistan Telecom (PT) SHOULD NOT have advertised those more specific directions (address prefixes) to their upstream transit provider. Those more specific address prefixes should have only been used inside the PT network. However, those prefixes got "leaked". Basically someone put the road-sign up for the public telling everyone on the Internet that PT had the most specific path to get to YouTube.

YouTube responded amazingly quick (30 minutes) and basically started advertising the more specific blocks themselves thus the Longest Match rule no longer applied and instead you had two "221 Main Street, Fairfax, Virginia" road-signs posted; one just said 10 miles, and the other said 1000 miles... people are going to take the shortest path then. Determining the shortest path is another part of routing. Perhaps another day I'll take some time to explain that one.

Longest Match specifically refers to taking your address and comparing it to a routing advertisement (the prefixes) and looking to see how many bits are identical in the two. If you've worked with computers you probably know about the Subnet Mask that you have to assign along with your IP address. When dealing with Subnet Masks, this allows a machine to decide if they need to go to a router or if they can talk to another machine directly. For example: with a subnet mask of (aka can talk to any other machine whose IP address begins with 10.1.1 ... WITHOUT going through a router.

In a prefix advertisement the prefix includes something very similar to a Subnet Mask. In this case the mask basically tells other routers in the network how specific of a route any specific prefix represents.

For example: I could say that my home address is "Virginia, Fairfax, Main Street, 221" and I could say that when I advertise my address I'll advertise down to the street name. In networking there is the concept of CIDR notation to describe blocks and sizes of IP addresses. For our teaching example, we'll pretend that to advertise a direction just to street level we would add /streetname to the address. My routing advertisement for path #3 from the above example would look like "Virginia, Fairfax, Main Street/streetname" Then if like #4 I advertised, "Virginia, Fairfax, Main Street, 221/streetnumber" you realized that THAT would be the longest match if you were looking for my specific address.

What if you were looking for 223 Main Street though? In that case, the longest possible match would be path #3 for you, "Virginia, Fairfax, Main Street/streetname" and you'd take that path which would get you to my street, but not directly into my driveway. Once you get to the street you'll get further directions on how to get to #223.

So, now that you hopefully have an idea of how longest match works, what could have been done to prevent this? The simple solution and the one that NORMALLY keeps stuff like this from happening is Route Filters. In this case, PT's transit provider should NOT have accepted any route advertisements from PT for address space that PT doesn't own. Currently the best way to ask people you are providing transit for what their addresses are, then look at the various assigned numbers authorities and/or routing registries to verify the blocks of addresses really belong to them and then create a filter that only allows those addresses to be sent. It is a pretty manual process though, and of course mistakes (or mischief) can happen.

There are discussions ongoing about other ways this could be done. Routing registries could provide certificates or you could sign your routes in a public manner that are in the registries and then when one router talks to another router they could verify through the signed messages that the number/routing authority has identified you as the proper owner (by your possession of the private key/cert) and accept any of those routes. Whew! That is a pretty straightforward way to accomplish this, and hopefully this incident will remind folks that it is important to move forward with it.

Though straightforward, it isn't easy. Lots of folks have to all agree to do it the same way. Other folks have to build infrastructure to support it. Vendors have to update their routers with software that understand how to process it. And of course, then the operators of the networks have to actually understand and use it. We can dream though. :)

If nothing else though, hopefully transit providers (like UU.NET/Verizon, AT&T, Level 3, PCCW, ATDN, etc.) will pay more attention and filter any routes that don't belong to their customers and prevent this from happening at the edge. Some already do, good for them! Some don't. Bad for them!

Oh well, hopefully you found all this interesting. Didn't mean to be so wordy, just mostly wanted to pass along what happened in non-network geek terms.

Thunderbird, Kerberos, and SMTP Auth

At work our email servers are Microsoft Active Directory Exchange Servers. I use IMAP to access them. I also use Thunderbird for my IMAP mail client. I use the nightly builds and noticed that I was suddenly being prompted for a Kerberos login every time I sent email.

While I certainly could use Kerberos for the login, I prefer not to and only use standard IMAP and SMTP encrypted auth and not Kerberos. So, in order to completely disable Kerberos from Thunderbird I played around until I found the following combination. Just changing the using-native-gsslib to false will not work as I originally thought it would. I had to put the /dev/null in there to really get it to stop. All seems well for me now, MUCH happier!

1) Open Thunderbird Preferences
2) Advanced Button
3) General Tab
4) Press "Config Editor" button
5) Easiest to just Search for "gss"
6) There will be two entries that both need to be edited:
a) network.negotiate-auth.gsslib = /dev/null
b) network.negotiate-auth.using-native-gsslib = false
7) Close the Config Editor
8) Close the Prefs
9) Restart Thunderbird

Obviously be careful futzing around with the config. I hope this helps some folks!

Cisco NAG

Notes from Cisco NAG Meeting, as with NANOG notes, please don't complain, they are mostly for me.

NANOG 40 Notes

These notes are rather free form. Please don't complain, they are mostly so I know what I heard. :) Updates will be coming throughout Monday, Tuesday, and Wednesday.

Botnets! What are they good for?!?

| 1 Comment

Joe wrote in his comment from my previous entry:

I think this pertains to the issue:,1895,2004893,00.asp

As I often do, my response back got to be rather lengthy so I decided to make a new Entry for it.

Yeap, in this particular case they're using the vulnerability that I wrote about, MS06-040. BotNets are really common and are these days a primary goal of someone writing Malware. Usually the Bots (or Zombies as they are sometimes called) are used for sending spam just like the E-Week article, other times they are used to overload a website and take it down.

A few years back that is what happened to several major websites as BotNets were used to send millions of "hello" packets to major corporations to keep their websites from coming up for real visitors. These "hello" packets are a type of TCP (Transmission Control Protocol) packet called a SYN (Synchronize) packet which is the very first packet two computer "speaking" TCP send. The first three packets two TCP speakers send are called the SYN/ACK handshake, with the first machine sending a SYN packet, the next machine sending it's own SYN packet which also has an ACK (Acknowledgement) flag within it back to the initial speaker, and finally the initial speaker sending it's own ACK (no SYN this time, notice) back to the second speaker. After all of that both machines can send data via TCP back and forth.

Now what happens in the SYN Flood attack that I was mentioning is that you have a BotNet send millions of the SYN packets to a target website. The machines that make up the target website have to send SYN/ACK packets back and wait for the final ACK back. Thing is that usually the BotNets will forge their source address so no one ever answers those SYN/ACK packets back to the website machines. They can only send a certain number of those before they can't answer anyone else and so they can't do anything.

Since everyone knows this happens there has been many mechanisms created to prevent this type of attack from working. AOL and Foundry Networks have a patent on one such method for example. Unfortunately if an attacker has a large enough BotNet, then he can actually allow the machines to complete the SYN/ACK handshake and continuously request information from the website which will flood the website and still keep legitimate viewers from seeing the website.

Okay, that took a little bit of a side turn, but that's me, always wondering around with my words!

Basically, everytime the good guys figure out a way to prevent an attack, the bad guys go and find another annoying thing to do to bug people.