Firewall analysis

Another day, another anomaly to investigate.  Ah, networking.

Today we have a screenshot from our Palo Alto Networks firewall device.  This is what keeps you from running BitTorrent on campus.  This is also what protects you from a good chunk of malware out there.  And you might be surprised how much malware is out there.

And you might be surprised how much malware is out there.

You should click on that to blow it up.  What we have is a search for the number of sessions going to an external IP address in the last 24 hours.  Normally, that's not a huge deal.  DNS requests are amongst our biggest category of sessions, for example, and much larger than the numbers in this screenshot.  What's interesting is that lots of sessions are being opened by just a few computers, and they are all connecting to a single server.

Running a WHOIS at the terminal, we can see that it's owned by the defunct corpse of PSINet, an early internet privatization pioneer.  13 years ago, PSINet was a known spam host.  Today, it's a known crawler/scanner.  Now, it's worth mentioning that that is not what we're seeing here.  Here, we have multiple hosts reaching out, with unidentifiable traffic signatures, to a known crawler/scanner.

Huh.  What does that sound like to you?  It sounds to me like a botnet reaching out to spam relays and/or command and control servers.  Gotcha!!!

The next step is to identify who is associated with each machine.  This is actually rather difficult at the moment, though we're hoping to fix that next year with new processes, documentation, and the Meraki gear in hand.  For now though, we can lookup DHCP requests.

The top connector's DHCP requests.  We can tell from the consistent pattern of requests every 4 hours that this machine is practically used as a server.  We can tell from his MAC address that he has a computer made by Belkin.  Wait.  The hostname is Belkin Router.

The trail goes dead.  Technically, we could figure out what port that's associated with on our switch, track down the room number, find out who lives there (or at least contact info) from Residential Life, and reach out to have a discussion about who's using their WiFi and maybe how to secure it.  Or we can admit that there are far more pressing things to do with our time, unless it turns out to be a faculty machine (the University owns faculty laptops, so it's easier to tell the user it needs to be cleaned).

Or we can start blocking that IP in both directions on the firewall.  Thoughts?

Wireless

You wanted wireless; we're now bringing wireless.  Parts are coming in now for the network upgrade that's (finally!) been approved.

Our vendor of choice.  This thing's creepier than E.T.

After several options were considered, the Meraki option was selected and approved.  Meraki offers a cost-competitive solution that provides simplified management and new tools to our network team.  In particular, Meraki's monitoring and bandwidth shaping functionality stand out from the rest.  They also have a wonderful dashboard with per-device tracking and statistics.  This is useful is in figuring out how the network is being used and how we might optimize it.

For example, I might be able to tell that our heaviest users at peak are mostly on Netflix.  Netflix is a very intensive application with high bandwidth requirements.  This is why, currently, most students probably watch Netflix only late at night.  However, Netflix is also very tolerant of "lossy" connections, where some data gets lost on the way to the viewer.  The Meraki gear will allow us to prioritize internet traffic such that encrypted web traffic, which is often used for banking, online testing, and other secure services, will be prioritized above Netflix.  Why would we do this?  Because secure web traffic is *extremely* sensitive to lossy connections.  Thus, by prioritizing secure web traffic above Netflix traffic, our secure web services will be noticeably improved, while Netflix continues to work just fine.

And for those wondering, yes, we plan to implement wireless in every building on campus, as well as the Quad (thank you, Student Senate!).  The plan is to implement it this summer, and make substantial changes to how the wireless is structured.  Coming soon... separate wireless networks for students and staff!

Network maintenance (the boring kind)

Hello.  I'm Sean, the network administrator at Illinois Wesleyan University, and this is my unofficial blog.

This morning, between 3 and 6AM, I upgraded 3 aging switches around campus.  The "new" switches aren't much better, but at least they support SSH and password encryption!  The bigger deal is that the new switches bring the Cisco equipment up to a common platform.  Everything now runs on Cisco IOS 12.1.  This will make changes *much* simpler to implement, as we'll be able to make changes to all the Cisco equipment on campus at once, remotely, and securely.  This will also mean we have significant incentive to standardize our configurations as much as possible, so that we can make significant configuration changes to all the switches at once without worrying about special cases and configurations.  Ideally, this leads to better control of the switches.  More uniformity means issues are easier to troubleshoot and less likely to arise.  Hopefully this all leads to a better experience for the end user, with less downtime.

The "new" switch model.  This is our spare.  If any of IWU's 35 Cisco 2950 switches dies, this is the only box we have available to replace it.  I like to keep it under my desk, leaning against the file cabinet full of food.