So a week ago last Friday, I had a new Linux Box with the Beta version of VMWare Server up in about two hours flat. Debian network installs are the only way to go: quick, with only a minimal set of packages. No GUI installer and none of that X nonsense. Things were going well.

The goal was to build a guest machine that ran a Roo Honeywall inline in front of another guest machine. The honeywall “outside” interface is bound to the physical interface in VMWare bridge mode. The “inside” interface interface is a VMWare “custom” interface or “host-only” interface. Which is the same interface as the one used by the “target” guest machine attackers will go after. Frames go in the physical interface in one virtual interface out the other and in a third, then back out.

So you basically have two bridges. (Recently I’d put a similar honeywall in front of a Layer 2 OpenVPN for a total of three bridges, but I digress.)

The only problem is that it wasn’t working. ARP Requests made it in all the way to the target, and the target even replied but they didn’t go back out the honeywall bridge and on the physical network. Something similar happened when I tried to ping out from the target machine. ARP requests.

(Hint: ARP requests are broadcast, ARP replies are unicast. This should have been enough to diagnose the problem)

So Monday afternoon (after a number of Roo re-installs) I gave up and and posted to the honeypots mailing list describing the problem. No response. (Still no reponses yet.) So Tuesday morning I foolishly think (after searching the VMWare site) “this is beta software” I’ll downgrade to VMWare GSX server. No dice. Next thought, even more problematic than the first resulting in a major rat hole: there is some weird thing about the Linux kernel used by Debian Sarge (2.6.8) which I’m using both on the host and one guest machine. I also try various network combinations for the physical interface hub or Cisco 2940 switch, which are nice little SOHO switches with VLANs BTW.

So I try another Linux distribution on the Dell server. Big Mistake, because I do not have a working distribution that I can easily compile VMWare kernel modules on until Friday morning (of course the situation wasn’t helped by the recent power outages here in Austin).

To make a long story short:
- OpenSuSE 10.1 is too bleeding edge for VMWare–there are some new kernel features that led to an incompatibility
- I have always had problems with Ubuntu 5.1 building VMWare (weird errors about gcc version incompatibility) so the server version of Ubuntu (no X!) doesn’t work either
- One of the distributions complained about X libraries.

By Friday morning, I had OpenSuSE 10.0 up and running and I reinstall Roo again, but it still doesn’t work. I’m starting to get desperate.

Later, I boot into the VWMare Workstation on the XP partition on my laptop and sniff on the outside interface of the Roo honeywall.

I see spanning tree. I hadn’t ever seen spanning tree, just the ARPs!

I can still see the tcpdump output on the screen.

So this is not a honeywall issue. (Of course, early I had looked through the Honeywall iptables scripts thinking there was some weird issue with Bridge filtering that was keeping the return traffic from coming back.)

Is it a difference between Windows and Linux hosts or between the versions of VMWare? I want to avoid installing VMWare workstation because I don’t want install X and more importantly I need to be able to start up vmware guest machines from shell scripts.

It it something with the interface.

Saturday, I rewire the power (adding a new UPS) and CAT5 (thinking somehow my sloppy cabling is indirectly responsible for these problems) in my office, much to the annoyance of my family.

This morning, fueld by breakfast tacos and the first cup of coffee of the day. I remember the error message (in VMWare workstation) about sniffing within virtual machines and file premissions on the virtual interface device files, so using Google, I find this TaoSecurity blog that answers my question, after no luck on honeynet archives or the the VMWare site.

I know there is a lesson here somewhere about virtualization or complexity or troubleshooting but I’m not sure what it is–but what does this blog have to do with SCADA Security? Lots.

Stay tuned.