Tue May 27 10:09:20 2008

On IOS Rootkits

A presentation given by Sebastian 'topo' Muniz from CORE Technologies, given at EUSecWest in London and at PH-Neutral 0x7d8, currently receives some considerable attention from the media. The presentation is titled "Killing the myth of Cisco IOS rootkits: DIK (Da Ios rootKit)". I had the pleasure and privilege to read his accompanying paper and can only hope you had a chance to attend the talk if you happened to be at one of these conferences.

It is however surprising how the media covers this presentation. IOS images, modified on the binary level to allow unprivileged access, are neither new nor extremely hard to do. When I presented on BlackHat Federal 2003 in Washington DC and mentioned backdoored IOS images in the presentation (Slide 12), nobody in the quite Cisco-aware audience showed any indication of surprise. Some time before that presentation, a random hacker had sent me an email with a few bash scripts which would correct any checksum in an IOS image that was patched, just for my entertainment, and mentioned that the scripts took him about half an hour to complete. What topo excels in is the level of platform independence and automatism his rootkit tools achieve. After all, he works in a team of experts on that topic.

The early backdoors in IOS images are a natural consequence of the software being fairly expensive and priced by features. In the same way that people receive a Windows XP Home version with their new PC, but want to run Windows XP Professional or even Vista, people receive an IOS IPonly image but want to support SSH logins instead of Telnet(1) on their routers. Accordingly, people would trade IOS images on the Internet instead of spending money on legitimate copies, especially since they also have to spend the same money on memory extensions for their routers to support the image. With images traded openly, of course attackers modify the image to allow login with a special password. Traded IOS images are an invite to them, as they essentially say: "Whoever downloads this either uses it in the lab or will install it on critical routing infrastructure in their corporation." There is just no other use for an IOS image.

The fact that IOS images are nothing but one single executable without inner security boundaries makes the modification just more powerful. When backdooring a Juniper router in the same way, the attacker has to modify a number of binaries, as Juniper's operating system is based on FreeBSD and therefore has real processes implementing different functionality (login, routing, etc.). It also has a kernel. Now, on IOS, everything is in the same process / address space / memory privilege zone, so a single patch can reach integrity checking code as easily as it can reach login validation code. Oh the other hand, what's the challenge of backdooring Juniper routers? Any adjusted FreeBSD rootkit will do.

What is left is getting the privileged access to the router in the first place. Best practice security in network design, architecture and configuration should prevent that, but there are people who don't follow those and there are vulnerabilities. As noted in other occasions, security functionality seems to be an excellent place to find bugs in. Even SSH, audited to death on other systems, appears to still be a very interesting target on IOS. In this respect, I can understand some of the service provider folks being unhappy about the presentation, as the availability of an IOS rootkit will most certainly rather fuel the interest in writing code execution exploits against IOS, rather than damp it. This is probably one of the reasons the rootkid maker code is not going to be released. Additionally, the heavily increased protections on common operating systems platforms (e.g. Windows Vista), will draw more attention to 1980s style, C written, complex monolithic systems running on critical Internet infrastructure.

All this isn't exactly news. I could have named this entry "I told you so", but that wouldn't really cut it anyway. What rather happened was the ongoing concern of a few people about this whole subject, several years before it was actually presented. When working on the first (and luckily completely replaced) version of the GDB debugging agent for Zynamics BinNavi that allowed runtime debugging of Cisco IOS in a decent reverse engineering tool, I realized that the GDB debugging capabilities would also finally allow the development of an independent forensics tool for Cisco IOS. I discussed this with Gadi Evron, one of the few people that foresaw the issue as well, during the Chaos Communication Congress in 2006.

This was the beginning of the CIR project, which stands for Cisco Information Retrieval or Cisco Incident Response, whatever you like more. CIR is a framework for analyzing IOS memory core dumps, the only solid and complete evidence that can be collected remotely from a router. While the GDB link provides a better (read: less easy to circumvent) interface, network operators do not appreciate driving to every router they manage and using it remotely would be pretty slow. CIR uses the IOS image as a blueprint for the memory map, as the core dumps do not contain any memory mapping information, and also compares static elements from the IOS image to the actual running code in the core dump. A paper and presentation on the technical background of CIR can be found on the Recurity Labs website. One of the simplest features implemented is the comparison of the .text segment between the image and the core dump.

When topo announced the IOS rootkit talk, I was excited that he was willing to provide a backdoored IOS image to me for testing CIR. This is responsible disclosure at its finest hour, as the patched image means little to Cisco Systems but very much to us for testing CIR. After all, detecting your own modifications to an IOS image is kind of silly. From the beginning on, CIR was meant to be provided as a free service to anyone who suspected that their router might have been attacked, backdoored or suffered from a successful or unsuccessful Denial of Service attack using vulnerabilities in IOS. Accordingly, processing was made available online when we barely had the alpha version working and was announced at BlackHat Federal in Washington DC. Since then, we made significant progress on the functional front, but most work went into code quality, dealing with the broken-beyond-recognition ELF files produced by Cisco's build process and making the framework ready to support bigger iron platforms.

Finally, but still before his talk at EUSecWest, topo provided the backdoored image. We fed it into CIR and successfully detected the .text segment modification. However, it only told us that the image was patched and where the patch begun. A quick change of the respective plugin then produced a detailed report that pointed to all modifications in the .text segment. The resulting report can be viewed on CIR Online for those interested in the details. Getting the exact virtual addresses out of the report makes reading the code differences in IDA a piece of cake: you jump to the respective address in the core file and press C. Out comes topo's backdoor, which is a beauty to read in PPC assembly. Felicidades topo!

So, while still in BETA phase, CIR Online seems to actually work. Of course, it also makes the shortcomings of the core dump method obvious: changing the core dumping function in Cisco IOS together with any other backdoor inserted would ruin the collection of evidence and therefore give nothing to process to CIR. This, as mentioned above, could be overcome by physically connecting to the serial console of the router and use the GDB link to obtain the evidence, which again could be prevented by patching the GDB stub in IOS. You end up with the classical race between attacker and forensics person. However, it is a race worth racing if you realize that the bad guys already left the starting line and the dust cloud at the horizon is them speeding away.

Now that some people actually talk about IOS rootkits, interesting tidbits show up. One person asked me if we have tested CIR with the Russian IOS rootkit that was for sale a few years ago. No, we didn't, but good to know that these exist. Additionally, Cisco finally published an updated and much more detailed response, giving instructions on how to detect and/or prevent rootkits with the on-board tools IOS provides. However, some of the instructions are rather humorous, such as transferring images to the router using a secure protocol when your image does not support SSH. On the other hand, they also published a list of MD5 sums of all 12.x images, which is shortly going to be a test CIR performs.

So finally, we seem to have overcome the discussion on if IOS rootkits work or not and approach the age where it's an accepted fact. We also know that, so far, binary and TCL backdoors are detected by CIR. That means we can concentrate on the crash-cause analysis and exploit attempt identification features again. Stay tuned.

Posted by FX | Permanent link