The Recurity Lablog

During a non public security event, I saw a presentation by Olaf Kolkman about the new DNS server named Unbound. When he mentioned that the whole thing is written in C for performance reasons, I returned that we should simply stop developing production software in languages that produce unmanaged code. We got into quite some discussions with the whole audience after that, many people stating that there isn't an alternative and everything else is just to slow. I'm not buying this for many obvious reasons, but that's for another post.

To put our abilities where my mouth is, we ended up performing a short source code audit for the Unbound developers. After all, Unbound is an effort to produce a reliable, validating and DNSSEC ready name server, something we all want to have deployed on a larger scale. Sergio Alvarez, who by the way will be speaking at CanSecWest this year, looked at the code and found it surprisingly not riddled with remote code execution bugs. I was certainly happy about that find, because it meant the next generation DNS server deployments wouldn't look at a future comparable to ISC bind's past.

That impression, however, was largely because Sergio compiled the code with all the ASSERT statements intact. Now, people running heavy duty production DNS servers will most certainly try to make it as fast as possible, instructing the compiler to get rid of “debug” features like ASSERTs. That might not be a good idea. So here is another lesson learned: When building for production, you might want to keep those ASSERTs compiled in, since your server crashing on funny packets is probably better than to share the administrative control of the machine.

Other than that, I hope the Unbound team keeps up the good work, so people have one less excuse to not move to DNSSEC.

Some of you might have heard that I gave a talk on Cisco IOS security at the 25C3 this year. The talk was unique for me in many ways, starting with the fact that it covered content going all the way back to the beginnings of Phenoelit up to material that was developed within Recurity Labs. It could be said that the talk was a nexus of research efforts in different areas of my life.

The second unique aspect was the cheer amount of stuff to cover, which prevented more in-depth reflections on some of the issues. This begins with the question of who would actually take over Cisco routers. The short answer is of course: whoever can. But that needs to be taken apart in more detail. Let's focus on attacks that directly apply to the device in question and ignore for now that the easiest way to take over an entire network infrastructure is to attack the unpatched Sun servers running in the Network Operation Center.

Consider successful exploits a question of development cost. An exploit is in that respect not different than any other software: you find someone you think can actually pull it off, present your requirements and have them develop it for you. In almost all cases, this isn't going to be for free, so they will give you a price tag for the work, which in most cases is a linear function of the cost of a work hour for them. This implies that the better they are, the less hours they will need to develop the exploit for you, which makes it cheaper. This is what made exploits against Windows desktops so cheap that attackers mostly relied on them for gaining access to networks from what the network owners considered inside (an outdated but still widespread way of looking at it).

Our research concerning Cisco IOS security was always based on the assumption that there are entities in this world that have reached a reasonable development cost level for IOS exploits. But the publicly available knowledge on how to write IOS exploits didn't fit the bill, as they required to jump into some memory address that is specific to the IOS image running on the target. Assumed you don't know what image is running on that machine, you could still argue that the exploit writer included a list of all possible image dependent addresses into the exploit would try them out, one at a time. This would cause the router to reboot every time a wrong guess was tried.

In the presentation, I said that there are about 100.000 different IOS images in use. This is a very debatable number, as about 15.000 are supported by Cisco at any given time and good network administrators will only run one or two IOS images in their entire network, often times investing several months to figure out which exact image they want. When we however fire up the Cisco Feature Navigator and ask it for all IOS images that support IP Routing, which should be all of them, we get: “Showing 1-50 of 280715 results” at the bottom of the page. Wow, what a number. On page 5615 of the result listing (thank god they have a direct jump feature!), we see that this covers everything from IOS 12.4 to 11.2. Therefore, the number doesn't take very old networks into account, in which you do see images below 11.2 running occasionally.

It should be clear that trying them all out is not an option, especially considering reboot times of 30 seconds and more per attempt. Your exploit would constantly reboot a router for 2339 hours or 97 days. And that doesn't take into the account that you need to get and disassemble them all, estimated time with IDA: 5848 days or 16 years.

Therefore, there is either no one exploiting IOS devices or they have found a better way to do it. Our Cisco Incident Response tool was developed in the hopes of finding people attacking IOS devices, successful or not. But then again, it's hard to write a detection for the unknown, so we also had to look into finding ways to get code execution stable. The method presented at the 25C3 (and documented here, feel free to post questions in the discussion section) only reduces the number of things you have to know about your target, it doesn't eradicate the problem in general. Now, we only need to know the ROMMON version and there are a lot less ROMMONs than IOS images out there.

For smaller machines, such as 2600s, updating ROMMON did not seem to be supported and the version depends on the shipping date. However, after closer inspection, here comes an errata: Cisco does offer 6 updated ROMMONs for 2600 routers. For larger machines, e.g. 7206s, there are about 36 different versions known. That's a few magnitudes smaller than 280715. But it is also still far from the ultimate truth, as you still need to know and have that ROMMON as well as knowing a few things of the box, most importantly the hardware series. Some people like to include the hardware series in their router's DNS records or name the PTR records of the IP addresses bound to a router's interface after the interface itself, which allows to guess what type of metal it is.

Knowing the hardware platform is actually more important for the first and second stage shellcode than it is for getting stable code execution, as the same ROMMON seems to be applicable to a number of subtypes of routers, while one subtype may have memory wired into different addresses than the other. But being lucky is also a valid option, which is what happened when we selected the memory area for a direct write: I assumed the memory at 0x80000000 on a 2600 is used for global IOS variable pointers, which is incorrect. So, errata #2: I was made aware that this is of course the exception vectors, after the MMU is turned on. Accordingly, this is a very good place to store two instructions.

There is still a lot to do and research when it comes to Cisco IOS and security. But the stable, image independent code execution at least allows us finally to draw better assumptions about the attacks we should be looking for. It shows nicely that, even with CIR, we should not try to detect the exploitation while it happens, but focus on the shellcode functionality and the footprints it leaves. And the IP options vulnerability is a perfect example why critical infrastructure should always dump the core files onto its own FLASH device, as dumping core over FTP doesn't really work to well when your “IP Input” process just got popped.

This year's Chaos Communication Congress, better known as 25C3, was an exceptional event in many ways. It begins with a program committee that attracted so many interesting people over the last years that they had ample material to select from, and they did a very good job of that too. Accordingly, the quality and spectrum of the presentations was significantly above many other conferences and we all need to thank the people that put up the program. And while I'm still not done seeing all the video recordings of all presentations, there have been quite a number of highlights.

The hard working organizers and Engel of the CCC apparently are by now so well trained in running a Congress that it almost appeared as stress-less routine to the casual observer. I've never been to a Congress with less shouting and less chaos in terms of organization, and despite the event's name, I think that's a good sign. They even somehow managed to handle the insane amount of people showing up, which, as DEFCON attendees will surely know, is quite a challenge by itself.

And then of course there were the presentations, above all Alexander Sotirov and Jacob Applebaum with their successful creation of a rogue SSL CA certificate. The work shows how the combination of academia research with the practical experience and dedication of world class security professionals can achieve something that was considered a theoretical attack. It also shows how much of a pipe dream the perceived security of browser based communication over SSL/TLS actually is. If all but one trusted CAs belong to the same publicly traded commercial entity, they don't actually need to fulfill their security promise anymore, because they have a monopoly.

The purpose of a publicly traded corporation is to maximize the profit for the share holders*. And if selling certificate signatures generates enough revenue to get your stocks rated as "buy", you did your job. If you need to revoke a large part of these certificates, because you failed to react to previously published research on vulnerabilities in them (MD5), this is similar to a call back of your product and would therefore hurt your reputation on the stock markets. If you however just ignore the problem as long as you can and then trust that very few people will actually understand the problem so it doesn't impact your sales, you can even offer remedy at no additional cost and look good in the press. From a business point of view, that is a remarkable containment stunt. From a security point of view, it's devastating. Not only does it show that revocation simply does not work, but also that the one entity that must be extremely strict with revocation actually doesn't follow it at all.

Interestingly enough, this proves two points made by Dan Kaminsky. The first is about how much of a defense SSL actually is in the light of vulnerabilities like his DNS issue from summer 2008. Dan said in his presentation at BlackHat that SSL proved to be much less of a defense than we all thought it would be. The second point is actually less obvious: The much debated partial disclosure approach Dan followed had a very interesting positive side to it that nobody saw before. The big difference between Alex's and Jake's big-bang presentation and Dan's long process of informing selected people gradually over time is the learning effect they had on all the other people. I think after that summer, we security professionals will not hear that old argument of a vulnerability not being critical because the attacker would need to control DNS in order to exploit it very often anymore. On the other hand, I don't see anyone reviewing their security perception of the trust model that so-called secure web sites are build on. Everyone is just happy that the issue got "fixed" so quickly. I for one have not realized that aspect of making a big fuzz about something enhancing its long term educational value before, and I certainly thank Dan for teaching that lesson to me.

Speaking of thanks, my fellow Phenoelit members, above all Mumpi, need to be thanked too for the awesome party they put on. That also includes DJ Vela and CMOS for playing at that event and in CMOS' case for flying all the way into Berlin to do so. And last but not least, I would like to thank the audience of the 25C3, which again was one of the smartest I had the privilege to speak in front of. I apologize for the suboptimal delivery of the Cisco IOS presentation to everyone who saw it, if you found a stray "Erm" in what I said, you may keep it.

* You could argue that this is the case with any corporation, but working at Recurity Labs, I can tell you it isn't.

Unfortunately, a number of things recently broke, including the main (and only) harddrive of the machine running phenoelit.net and therefore this blog. We are recovering all the services but, as you might have noticed, switched blog engines so we no longer actually run any (potentially vulnerable) code when you hit this site.

Now we just need to port the old entries over in a consistent and permalink friendly way, which might require a few more days.

Thanks for not unsubscribing :)

Update: Things should be back to normal and the two delayed posts should be posted. Please contact me with any complains if you find something missing or broken. Thanks.

... I feel compelled to point at Derek Soeder's VMware Emulation Flaw x64 Guest Privilege Escalation (1/2) post at full disclosure. It seems that it's always the same names that come up with those beautiful exploits. Delighting.

I recently discovered that there seems to be this class of discoveries that is made independently by many people from the same community in roughly the same timeframe. What's interesting about this class of discoveries is that often they are not published. Everyone seems to assume (correctly) that others have discovered the same. One topic that certainly falls in this category is “return oriented programming”, which everyone I know invented on their own once they actually needed it. Finally, at BlackHat USA this year, academia in the names of Erik Buchanan, Shawn Moyer, Ryan Roemer and Stefan Savage took this concept and turned it into solid research. Conceptually no news, it's highly recommended reading for anyone who used this method before, as approach and execution are brilliant.

The second instance of that class this year is TCP connections that don't use the operating system's sockets. If you ever wrote networking attack tools using raw sockets or drivers, you have come across the idea and probably tried it out. The basic idea is that you don't need to keep any state as the TCP client. You can send as many SYN packets as you want and can reply with the final ACK once the SYN/ACK packet comes back. All the information you need is in the answer from the server. You can take this method one packet further and actually send the first payload packet as well, e.g. an HTTP GET request. Check out Dennis' blog post if you need a picture of this.

TCP server software is in most cases written around the accept() call, which blocks until the OS has received the final handshake packet. Once that happens, the accept() call returns and the server software starts handling the connection. If you had sent the first payload packet as well, the server software even goes into the state of processing a request. Many servers are not built to handle the case where they no longer have any communication partner (i.e. they don't time out quickly), and the rate in which they get new ones usually overwhelms the server as well as the OS. On top of it, TCP, being a reliable protocol, tries to keep the connection alive by resending the answers because it assumes lost packets. The attacker can make this worse by playing with some properties of TCP, e.g. the window size, or add other twists to it, like ignoring RST packets or sending out-of-band packets with the PSH flag. Bottom line is, it really depends on the kernel and the server software what happens after the handshake is completed, but the issue is the same all over the place. Side note: things that don't implement a proper common TCP/IP stack fail more miserably, ask people at Cisco.

After I played with this in 2002, I became aware of many earlier implementations, e.g. this one, that one and many others. I recently remembered all that when I met Jack and Robert at the Sec-T conference in Stockholm (side note: great conference, watch for their 2009 edition). We compared notes on what things reacted how when exposed to stateless TCP clients hitting them hard. While it is relatively easy to protect server implementations by limiting the number of concurrent requests served, and operating systems by limiting kernel buffers and number of connections from a single source, so-called security devices and code are entirely different stories. Back in the days, my own Linux kernel gave me problems as ip_contrack, part of the NAT code, kept killing my TCP/IP stack as it tried to keep track of the connections I created, despite the fact that neither firewall nor NAT were actually enabled. I had to remove it from my own kernel before I could perform the attack against other devices.

Today's networks are full of devices whose vendors think it's a good idea to track established TCP/IP connections. IDS and IPS systems are just the tip of the iceberg. Just about every firewall, NAT device, TCP load balancer, transparent proxies and even some network sniffers do it. That's certainly not smart. You should try to avoid tracking state of others you don't control, especially if you don't actually need to. But the ignorance of basic principles and prior work is unfortunately not that uncommon.

It should however be clear that the risk of falling victim to this attack is inherent to offering a service. When you get a phone and publish the phone number, anyone can call you. It is not possible to prevent strangers from calling you every five minutes in the middle of the night when you want to be reachable to people that you have not authorized before. Ask any call center or ask people who run a web server that got slashdoted. If you want others to be able to contact you, they can overwhelm you with contacts. If you open your shop's doors, people can come in and buy stuff, but you could also have a flash mop filling your space. You could post bouncers at your door, but that's neither friendly nor helpful.

It should also be noted that this attack is not stealthy at all. No matter if you use your regular operating system's sockets or a raw and stateless TCP/IP implementation to establish a connection; you cannot spoof your IP address unless you sit on the transit route of the legitimate owner (in which case you can do a lot more). I think the scenario of the low-bandwidth customer at home taking down important servers using full handshakes is less likely. In most cases, his DSL home router would die first. Almost all other scenarios have the same effect to the victim as if he is attacked by a regular DDoS attack today: It's always possible but there are mitigations. The easiest mitigation for stateless TCP clients is to limit the amount of connections from a single source address and get rid of useless connection tracking devices in between.

And if the attacker is located in Germany, he will end up in jail. The much contested change to §303b of the StGB, Germany's criminal law, says: Interference with a data processing system that is of importance to someone else, by committing a crime described in §303a, by entering or sending data with the intention to cause a disadvantage to the system or [...], will be punished with jail up to 3 years or a monetary fine. The jail time can be up to 5 years if the affected system belongs to a company or the government. Therefore, completing too many TCP handshakes is illegal in Germany.

I'm looking forward to Jack and Robert publishing their full findings, as I'm sure it will provide much entertainment to the security community.

Perception is an interesting thing. Since everyone apparently has their own, it is fairly hard to arrive at a common denominator. In today's world, media is the perception softening instance that decided what people see and what not. Using the media to reach a large amount of people is intentionally shaping their perception. If your goal is to make people do something specific, this is a highly effective approach. That's what happened with DNS. Every computer security blog on the planet posted statements about Kaminsky, Halvar and the Domain Name Resolution Protocol, some even unintentionally. The global perception is: this is extremely important. People talk about it, people patch their servers with a workaround and people think about the Internet's safety. Dan has accomplished his mission.

The Kaminsky DNS attack is definitively regarded as the most important vulnerability this year. This, I find highly interesting , as we have seen two other gigantic security failures already in 2008. Debian's NRNG (non-random number generator) is most certainly one of them. But honestly, raise your hands if you have even noticed SNMPv3. Anyone? I give you the quickest of run-downs:

1. SNMPv3 uses HMACs over secret keys for authentication.
2. The packet can carry a shortened HMAC for [fill in silly performance statement here] reasons.
3. Most implementations implement their HMAC match check as:
   memcmp( myHMACbuffer, packetHMACbuffer, packetHMAClength )
4. If "packetHMAClength" == 1, brute force requires 256 UDP packets.

What can an attacker do with this? SNMPv3 is used to manage routers - the routers that forward all your traffic around the world, including your DNS queries. Managing a router means being able to configure it; a.k.a. super user access. Attackers who can configure a router in your path can redirect everything, without you knowing, not just traffic that relies on name resolution.

We have been working with a customer on a security issue scoring system, to help level perceptions. We started off with CVSS, which deserves its own post some time. Let's just say we didn't stay with it very long. When you compare the three big vulnerabilities this year, here are the CVSS scores according to the National Vulnerability Database at NIST:

• Kaminsky's DNS: 9.4 (AV:N/AC:L/Au:N/C:N/I:C/A:C)
• SNMPv3: 6.8 (AV:N/AC:M/Au:N/C:P/I:P/A:P) (see below)
• Debian NRNG: 7.8  (High) (AV:N/AC:L/Au:N/C:C/I:N/A:N)

Interesting to note. So being able to own the entire infrastructure is less important than breaking the SSL certificates of banks, being less important than poisoning DNS. Is that the case, or just the perception?

What I am looking forward to is the hard factual data we will see in the penetration tests and incidents to come over the course of the next year (assumed there is not another disaster). It will tell us, what systems actually got patched to which extend. I don't expect to find many vulnerable Debian keys, I do expect to find many routers ownable by SNMPv3 and I have no idea about the DNS thing yet.

On a final side note, it was wonderful to sit down with Dan on the day he went public, drink German beer in Seattle and discuss this very topic of vulnerability perception (and by the way, he didn't tell me any details, not even after the beers). Dan tried something unprecedented with the way he handled this, a very brave thing to do. May history be a fair judge, the media won't be it.

UPDATE: The people over at NVD/NIST seem to have noticed that owning routers is potentially more dangerous than they initially thought. Accordingly, SNMPv3 got upgraded to a CVSS v2 score of 10.0 (High) (AV:N/AC:L/Au:N/C:C/I:C/A:C). Well done.