Author Archives: Robert Graham

Some notes on today’s DNS DDoS

Posted on by

Some notes on today’s DNS outages due to DDoS.

We lack details. As a techy, I want to know the composition of the traffic. Is it blindly overflowing incoming links with junk traffic? Or is it cleverly sending valid DNS requests, overloading the ability of servers to respond, and overflowing outgoing link (as responses are five times or more as big as requests). Such techy details and more make a big difference. Was Dyn the only target? Why were non-Dyn customers effected?

Nothing to do with the IANA handover. So this post blames Obama for handing control of DNS to the Russians, or some such. It’s silly, and not a shred of truth to it. For the record, I’m (or was) a Republican and opposed handing over the IANA. But the handover was a symbolic transition of a minor clerical function to a body that isn’t anything like the U.N. The handover has nothing to do with either Obama or today’s DDoS. There’s no reason to blame this on Obama, other than the general reason that he’s to blame for everything bad that happened in the last 8 years.

It’s not a practice attack. A Bruce Schneier post created the idea of hacking doing “practice” DDoS. That’s not how things work. Using a botnot for DDoS always degrades it, as owners of machines find the infections and remove them. The people getting the most practice are the defenders, who learn more from the incident than the attackers do.

It’s not practice for Nov. 8. I tweeted a possible connection to the election because I thought it’d be self-evidently a troll, but a lot of good, intelligent, well-meaning people took it seriously. A functioning Internet is not involved in counting the votes anywhere, so it’s hard to see how any Internet attack can “rig” the election. DDoSing news sources like CNN might be fun — a blackout of news might make some people go crazy and riot in the streets. Imagine if Twitter went down while people were voting. With this said, we may see DDoS anyway — lots of kids control large botnets, so it may happen on election day because they can, not because it changes anything.

Dyn stupidly uses BIND. According to “version.bind” queries, Dyn (the big DNS provider that is a major target) uses BIND. This is the most popular DNS server software, but it’s wrong. It 10x to 100x slower than alternatives, meaning that they need 100x more server hardware in order to deal with DDoS attacks. BIND is also 10x more complex — it strives to be the reference implementation that contains all DNS features, rather than a simple bit of software that just handles this one case. BIND should never be used for Internet-facing DNS, packages like KnotDNS and NSD should be used instead.

Fixing IoT. The persistent rumor is that an IoT botnet is being used. So everything is calling for regulations to secure IoT devices. This is extraordinarily bad. First of all, most of the devices are made in China and shipped to countries not in the United States, so there’s little effect our regulations can have. Except they would essentially kill the Kickstarter community coming up with innovative IoT devices. Only very large corporations can afford the regulatory burden involved. Moreover, it’s unclear what “security” means. There no real bug/vulnerability being exploited here other than default passwords — something even the US government has at times refused to recognize as a security “vulnerability”.

Fixing IoT #2. People have come up with many ways default passwords might be solved, such as having a sticker on the device with a randomly generated password. Getting the firmware to match a printed sticker during manufacturing is a hard, costly problem. I mean, they do it all the time for other reasons, but it starts to become a burden for cheaper device. But in any event, the correct solution is connecting via Bluetooth. That seems to be the most popular solution these days from Wimo to Echo. Most of the popular WiFi chips come with Bluetooth, so it’s really no burden for make devices this way.

It’s not IoT. The Mirai botnet primarily infected DVRs connected to security cameras. In other words, it didn’t infect baby monitors or other IoT devices insider your home, which are protected by your home firewall anyway. Instead, Mirai infected things that were outside in the world that needed their own IP address.

DNS failures cause email failures. When DNS goes down, legitimate email gets reclassified as spam, and dropped by spam filters

Interesting side effect of the #DynDNS #DDOS SPF checks fail, so @twitter emails were put in my spam folder. pic.twitter.com/POHpHTQEJ6

— Rik van Duijn RCX (@rikvduijn) October 21, 2016

It’s all about that TTL. You don’t contact a company’s DNS server directly. Instead, you contact your ISPs “cache”. How long something stays in that cache is determined by what’s known as the TTL or “time to live”. Long TTLs mean that if a company wants to move servers around, they’ll have to wait until for until caches have finally aged out old data. Short TTLs mean changes propagate quickly. Any company that had 24 hours as their TTL was mostly unaffected by the attack. Twitter has a TTL of 205 seconds, meaning it only takes 4 minutes of DDoS against the DNS server to take Twitter offline. One strategy, which apparently Cisco OpenDNS uses, is to retain old records in its cache if it can’t find new ones, regardless of the TTL. Using their servers, instead of your ISPs, can fix DNS DDoS for you:

OpenDNS retains cached DNS records during NS failures, no problems on Internet for me. Here’s their servers.
208.67.222.222
208.67.220.220

— SwiftOnSecurity (@SwiftOnSecurity) October 21, 2016

Why not use anycast?

The attack took down only east-coast operations, attacking only part of Dyn’s infrastructure located there. Other DNS providers, such as Google’s famed 8.8.8.8 resolver, do not have a single location. They instead us anycasting, routing packets to one of many local servers, in many locations, rather than a single server in one location. In other words, if you are in Australia and use Google’s 8.8.8.8 resolver, you’ll be sending requests to a server located in Australia, and not in Google’s headquarters.

The problem with anycasting is it technically only works for UDP. That’s because each packet finds its own way through the Internet. Two packets sent back-to-back to 8.8.8.8 may, in fact, hit different servers. This makes it impossible to establish a TCP connection, which requires all packets be sent to the same server. Indeed, when I test it here at home, I get back different responses to the same DNS query done back-to-back to 8.8.8.8, hinting that my request is being handled by different servers.

Historically, DNS has used only UDP, so that hasn’t been a problem. It still isn’t a problem for “root servers”, which server only simple responses. However, it’s becoming a problem for normal DNS servers, which give complex answers that can require multiple packets to hold a response. This is true for DNSSEC and things like DKIM (email authentication). That TCP might sometimes fail therefore means things like email authentication sometimes fail. That it will probably work 99 times out of 100 means that 1% of the time it fails — which is unacceptable.

There are ways around this. An anycast system could handle UDP directly and pass all TCP to a centralized server somewhere, for example. This allows UDP at max efficiency while still correctly with the uncommon TCP. The point is, though, that for Dyn to make anycast work requires careful thinking and engineering. It’s not a simple answer.

Continue reading Some notes on today’s DNS DDoS→

Posted in Uncategorized

Cliché: Security through obscurity (again)

Posted on by

This post keeps popping up in my timeline. It’s wrong. The phrase “security through/by security” has become such a cliché that it’s lost all meaning. When somebody brings says it, they are almost certainly saying a dumb thing, regardless if they support it or are trying to debunk it.

Let’s go back to first principles, namely Kerckhoff’s Principle from the 1800s that states cryptography should be secure even if everything is known about it except the key. In other words, there exists no double-secret military-grade encryption with secret algorithms. Today’s military crypto is public crypto.

Let’s apply this to port knocking. This is not a layer of obscurity, as proposed by the above post, but a layer of security. Applying Kerkhoff’s Principle, it should work even if everything is known about the port knocking algorithm except the sequence of ports being knocked.

Kerkhoff’s Principle is based on a few simple observations. Two relevant ones today are:

* things are not nearly as obscure as you think
* obscurity often impacts your friends more than your enemies

I know that many sites use port knocking. Therefore, if I get no response from an IP address I have reason to know exists, then I’ll assume port knocking. I know which port knocking techniques are popular. Or, sniffing at the local Starbucks, I might observe outgoing port knocking behavior, and know which sensitive systems I can attack later using the technique. Thus, though port knocking makes it look like a system doesn’t exist, the security of the system should not rest on this obscurity.

Instead of an obscurity layer, port knocking a security layer. The security it provides is that it drives up the amount of effort an attacker needs to hack the system. Some use the opposite approach, whereby the firewall in front of a subnet responds with a SYN-ACK to every SYN. This likewise increases the costs of those doing port scans (like myself, who masscans the entire Internet), by making it look that all IP addresses and port exists, not by hiding systems behind a layer of obscurity.

One plausible way of defeating a port knocking implementation is to simply scan all 64k ports many times. If you are looking for a sequence of TCP ports 1000, 5000, 2000, 4000, then you’ll see this sequence. You’ll see all sequences.

If the code for your implementation is open, then it’s easy for others to see this plausible flaw and point it out to you. You could fix this flaw by then forcing the sequence to reset every time it saw the first port, or to also listen for bad ports (ones not part of the sequence) that would likewise reset the sequence.

If your code is closed, then your friends can’t see this problem. But your enemies are still highly motivated. They might find your code, find the compiled implementation, or must just guess ways around your possible implementation. The chances that you, some random defender, is better at this than the combined effort of all your attackers is very small. Opening things up to your friends gives you a greater edge to combat your enemies.

Thus, applying Kerkoff’s Principle to this problem is that you shouldn’t rely upon the secrecy of your port knocking algorithm, or the fact that you are using port knocking in the first place.

The above post also discusses ssh on alternate ports. It points out that if an 0day is found in ssh, those who run the service on the default port of 22 will get hacked first, while those who run at odd ports, like 7837, will have time to patch their services before getting owned.

But this is just repeating the fallacy. It’s focusing only on the increase in difficulty to attackers, but ignoring the increase in difficulties to friends. Let’s say some new ssh 0day is announced. Everybody is going to rush to patch their servers. They are going to run tools like my masscan to quickly find everything listening on port 22, or a vuln scanner like Nessus. Everything on port 22 will quickly get patched. SSH servers running on port 7837, however, will not get patched. On the other other hand, Internet-wide scans like Shodan or the 2012 Internet Census may have already found that you are running ssh on port 7837. That means the attackers can quickly attack it with the latest 0day even while you, the defender, are slow to patch it.

Running ssh on alternate ports is certainly useful because, as the article points out, it dramatically cuts down on the noise that defenders have to deal with. If somebody is brute forcing passwords on port 7837, then that’s a threat worth paying more attention to than somebody doing the same at port 22. But this benefit is separate discussion from obscurity. Hiding an ssh server on an obscure port may thus be a good idea, but not because there is value to obscurity.

Thus, both port knocking and putting ssh on alternate ports are valid security strategies. However, once you mention the cliche “security by/through obscurity”, you add nothing useful to the mix.

Continue reading Cliché: Security through obscurity (again)→

Posted in SBN

Cliché: Security through obscurity (again)

Posted on by

This post keeps popping up in my timeline. It’s wrong. The phrase “security through/by security” has become such a cliché that it’s lost all meaning. When somebody says it, they are almost certainly saying a dumb thing, regardless if they support it or are trying to debunk it.

Let’s go back to first principles, namely Kerckhoff’s Principle from the 1800s that states cryptography should be secure even if everything is known about it except the key. In other words, there exists no double-secret military-grade encryption with secret algorithms. Today’s military crypto is public crypto.

Let’s apply this to port knocking. This is not a layer of obscurity, as proposed by the above post, but a layer of security. Applying Kerkhoff’s Principle, it should work even if everything is known about the port knocking algorithm except the sequence of ports being knocked.

Kerkhoff’s Principle is based on a few simple observations. Two relevant ones today are:

* things are not nearly as obscure as you think
* obscurity often impacts your friends more than your enemies

I (as an attacker) know that many sites use port knocking. Therefore, if I get no response from an IP address (which I have reason to know exists), then I’ll assume port knocking is hiding it. I know which port knocking techniques are popular. Or, sniffing at the local Starbucks, I might observe outgoing port knocking behavior, and know which sensitive systems I can attack later using the technique. Thus, though port knocking makes it look like a system doesn’t exist, this doesn’t fully hide a system from me. The security of the system should not rest on this obscurity.

Instead of an obscurity layer, port knocking a security layer. The security it provides is that it drives up the amount of effort an attacker needs to hack the system. Some use the opposite approach, whereby the firewall in front of a subnet responds with a SYN-ACK to every SYN. This likewise increases the costs of those doing port scans (like myself, who masscans the entire Internet), by making it look that all IP addresses and ports exist, not by hiding systems behind a layer of obscurity.

One plausible way of defeating a port knocking implementation is to simply scan all 64k ports many times. If you are looking for a sequence of TCP ports 1000, 5000, 2000, 4000, then you’ll see this sequence. You’ll see all sequences.

If the code for your implementation is open, then it’s easy for others to see this plausible flaw and point it out to you. You could fix this flaw by then forcing the sequence to reset every time it saw the first port, or to also listen for bad ports (ones not part of the sequence) that would likewise reset the sequence.

If your code is closed, then your friends can’t see this problem. But your enemies are still highly motivated. They might find your code, find the compiled implementation, or must just guess ways around your possible implementation. The chances that you, some random defender, is better at this than the combined effort of all your attackers is very small. Opening things up to your friends gives you a greater edge to combat your enemies.

Thus, applying Kerkoff’s Principle to this problem is that you shouldn’t rely upon the secrecy of your port knocking algorithm, or the fact that you are using port knocking in the first place.

The above post also discusses ssh on alternate ports. It points out that if an 0day is found in ssh, those who run the service on the default port of 22 will get hacked first, while those who run at odd ports, like 7837, will have time to patch their services before getting owned.

But this is just repeating the fallacy. It’s focusing only on the increase in difficulty to attackers, but ignoring the increase in difficulties to friends. Let’s say some new ssh 0day is announced. Everybody is going to rush to patch their servers. They are going to run tools like my masscan to quickly find everything listening on port 22, or a vuln scanner like Nessus. Everything on port 22 will quickly get patched. SSH servers running on port 7837, however, will not get patched. On the other other hand, Internet-wide scans like Shodan or the 2012 Internet Census may have already found that you are running ssh on port 7837. That means the attackers can quickly attack it with the latest 0day even while you, the defender, are slow to patch it.

Running ssh on alternate ports is certainly useful because, as the article points out, it dramatically cuts down on the noise that defenders have to deal with. If somebody is brute forcing passwords on port 7837, then that’s a threat worth paying more attention to than somebody doing the same at port 22. But this benefit is separate discussion from obscurity. Hiding an ssh server on an obscure port may thus be a good idea, but not because there is value to obscurity.

Thus, both port knocking and putting ssh on alternate ports are valid security strategies. However, once you mention the cliche “security by/through obscurity”, you add nothing useful to the mix.

Continue reading Cliché: Security through obscurity (again)→

Cliché: Security through obscurity (again)

Posted on by

This post keeps popping up in my timeline. It’s wrong. The phrase “security through/by security” has become such a cliché that it’s lost all meaning. When somebody says it, they are almost certainly saying a dumb thing, regardless if they support it or are trying to debunk it.

Let’s go back to first principles, namely Kerckhoff’s Principle from the 1800s that states cryptography should be secure even if everything is known about it except the key. In other words, there exists no double-secret military-grade encryption with secret algorithms. Today’s military crypto is public crypto.

Let’s apply this to port knocking. This is not a layer of obscurity, as proposed by the above post, but a layer of security. Applying Kerkhoff’s Principle, it should work even if everything is known about the port knocking algorithm except the sequence of ports being knocked.

Kerkhoff’s Principle is based on a few simple observations. Two relevant ones today are:

* things are not nearly as obscure as you think
* obscurity often impacts your friends more than your enemies

I (as an attacker) know that many sites use port knocking. Therefore, if I get no response from an IP address (which I have reason to know exists), then I’ll assume port knocking is hiding it. I know which port knocking techniques are popular. Or, sniffing at the local Starbucks, I might observe outgoing port knocking behavior, and know which sensitive systems I can attack later using the technique. Thus, though port knocking makes it look like a system doesn’t exist, this doesn’t fully hide a system from me. The security of the system should not rest on this obscurity.

Instead of an obscurity layer, port knocking a security layer. The security it provides is that it drives up the amount of effort an attacker needs to hack the system. Some use the opposite approach, whereby the firewall in front of a subnet responds with a SYN-ACK to every SYN. This likewise increases the costs of those doing port scans (like myself, who masscans the entire Internet), by making it look that all IP addresses and ports exist, not by hiding systems behind a layer of obscurity.

One plausible way of defeating a port knocking implementation is to simply scan all 64k ports many times. If you are looking for a sequence of TCP ports 1000, 5000, 2000, 4000, then you’ll see this sequence. You’ll see all sequences.

If the code for your implementation is open, then it’s easy for others to see this plausible flaw and point it out to you. You could fix this flaw by then forcing the sequence to reset every time it saw the first port, or to also listen for bad ports (ones not part of the sequence) that would likewise reset the sequence.

If your code is closed, then your friends can’t see this problem. But your enemies are still highly motivated. They might find your code, find the compiled implementation, or must just guess ways around your possible implementation. The chances that you, some random defender, is better at this than the combined effort of all your attackers is very small. Opening things up to your friends gives you a greater edge to combat your enemies.

Thus, applying Kerkoff’s Principle to this problem is that you shouldn’t rely upon the secrecy of your port knocking algorithm, or the fact that you are using port knocking in the first place.

The above post also discusses ssh on alternate ports. It points out that if an 0day is found in ssh, those who run the service on the default port of 22 will get hacked first, while those who run at odd ports, like 7837, will have time to patch their services before getting owned.

But this is just repeating the fallacy. It’s focusing only on the increase in difficulty to attackers, but ignoring the increase in difficulties to friends. Let’s say some new ssh 0day is announced. Everybody is going to rush to patch their servers. They are going to run tools like my masscan to quickly find everything listening on port 22, or a vuln scanner like Nessus. Everything on port 22 will quickly get patched. SSH servers running on port 7837, however, will not get patched. On the other other hand, Internet-wide scans like Shodan or the 2012 Internet Census may have already found that you are running ssh on port 7837. That means the attackers can quickly attack it with the latest 0day even while you, the defender, are slow to patch it.

Running ssh on alternate ports is certainly useful because, as the article points out, it dramatically cuts down on the noise that defenders have to deal with. If somebody is brute forcing passwords on port 7837, then that’s a threat worth paying more attention to than somebody doing the same at port 22. But this benefit is separate discussion from obscurity. Hiding an ssh server on an obscure port may thus be a good idea, but not because there is value to obscurity.

Thus, both port knocking and putting ssh on alternate ports are valid security strategies. However, once you mention the cliche “security by/through obscurity”, you add nothing useful to the mix.

Update: Response here. Continue reading Cliché: Security through obscurity (again)→

Trump on cybersecurity: vacuous and populist

Posted on by
Trump has published his policy on cybersecurity. It demonstrates that he and his people do not understand the first thing about cybersecurity.
Specifically, he wants “the best defense technologies” and “cyber awareness training for all government employees”. These are well known bad policies in the cybersecurity industry. They are the sort of thing the intern with a degree from Trump University would come up with.

Awareness training is the knee-jerk response to any problem. Employees already spend a lot of their time doing mandatory training for everything from environmental friendly behavior, to sexual harassment, to Sarbannes-Oxley financial compliance, to cyber-security. None of it has proven effective, but organizations continue to force it, either because they are required to, or they are covering their asses. No amount of training employees to not click on email attachments helps. Instead, the network must be secure enough that reckless clicking on attachments pose no danger.

Belief in a technological Magic Pill that will stop hackers is common among those who know nothing about cybersecurity. Such pills don’t exist. The least secure networks already have “the best defense technologies”. Things like anti-virus, firewalls, and intrusion prevention systems do not stop hackers by themselves – but area instead tools that knowledgeable teams use in order to make their jobs easier. It’s like how a chisel doesn’t make a sculpture by itself, but is instead just a tool used by the artist. The government already has all the technology it needs. It’s problems instead derive from the fact that they try to solve their problems the way Trump does – by assigning the task to some Trump University intern.

Lastly, Trump suggests that on the offensive side, we need to improve our offensive abilities, in order to create a cyber deterrence. We already do that. The United States is by far the #1 nation in offensive capabilities. In 2015, Obama forced China to the table, to sign an agreement promising they’d stop hacking us. Since then, China has kept the agreement, and has dropped out of the news as being the source of cyber attacks. Privately, many people in government tell me its because we did some major cyber attack in China that successfully deterred them.

Trump promises to be a strong leader who hires effective people. He demonstrates this nowhere. In my area of expertise, he and his people demonstrate a shocking ignorance of the issues. It’s typical populist rhetoric: when China and Russia rape our computers, he’ll blame it on some sort of rigged system, not his own incompetence.

Disclaimer: I don’t care about Trump’s locker room comments, or any of the other things that get the mass media upset. I oppose Trump because he’s a vacuous populist, as I demonstrate here.

Continue reading Trump on cybersecurity: vacuous and populist→

Posted in Uncategorized

Trump on cybersecurity: vacuous and populist

Posted on by
Trump has published his policy on cybersecurity. It demonstrates that he and his people do not understand the first thing about cybersecurity.
Specifically, he wants “the best defense technologies” and “cyber awareness training for all government employees”. These are well known bad policies in the cybersecurity industry. They are the sort of thing the intern with a degree from Trump University would come up with.

Awareness training is the knee-jerk response to any problem. Employees already spend a lot of their time doing mandatory training for everything from environmental friendly behavior, to sexual harassment, to Sarbannes-Oxley financial compliance, to cyber-security. None of it has proven effective, but organizations continue to force it, either because they are required to, or they are covering their asses. No amount of training employees to not click on email attachments helps. Instead, the network must be secure enough that reckless clicking on attachments pose no danger.

Belief in a technological Magic Pill that will stop hackers is common among those who know nothing about cybersecurity. Such pills don’t exist. The least secure networks already have “the best defense technologies”. Things like anti-virus, firewalls, and intrusion prevention systems do not stop hackers by themselves – but area instead tools that knowledgeable teams use in order to make their jobs easier. It’s like how a chisel doesn’t make a sculpture by itself, but is instead just a tool used by the artist. The government already has all the technology it needs. It’s problems instead derive from the fact that they try to solve their problems the way Trump does – by assigning the task to some Trump University intern.

Lastly, Trump suggests that on the offensive side, we need to improve our offensive abilities, in order to create a cyber deterrence. We already do that. The United States is by far the #1 nation in offensive capabilities. In 2015, Obama forced China to the table, to sign an agreement promising they’d stop hacking us. Since then, China has kept the agreement, and has dropped out of the news as being the source of cyber attacks. Privately, many people in government tell me its because we did some major cyber attack in China that successfully deterred them.

Trump promises to be a strong leader who hires effective people. He demonstrates this nowhere. In my area of expertise, he and his people demonstrate a shocking ignorance of the issues. It’s typical populist rhetoric: when China and Russia rape our computers, he’ll blame it on some sort of rigged system, not his own incompetence.

Disclaimer: I don’t care about Trump’s locker room comments, or any of the other things that get the mass media upset. I oppose Trump because he’s a vacuous populist, as I demonstrate here.

Continue reading Trump on cybersecurity: vacuous and populist→

Posted in SBN

WTF Yahoo/FISA search in kernel?

Posted on by

A surprising detail in the Yahoo/FISA email search scandal is that they do it with a kernel module. I thought I’d write up some (rambling) notes.

What the government was searching for

As described in the previoius blog post, we’ll assume the government is searching for the following string, and possibly other strings like it within emails:

### Begin ASRAR El Mojahedeen v2.0 Encrypted Message ###

I point this out because it’s simple search identifying things. It’s not natural language processing. It’s not searching for phrases like “bomb president”.

Also, it’s not AV/spam/childporn processing. Those look at different things. For example, filtering message containing childporn involves calculating a SHA2 hash of email attachments and looking up the hashes in a table of known bad content (or even more in-depth analysis). This is quite different from searching.

The Kernel vs. User Space

Operating systems have two parts, the kernel and user space. The kernel is the operating system proper (e.g. the “Linux kernel”). The software we run is in user space, such as browsers, word processors, games, web servers, databases, GNU utilities [sic], and so on.

The kernel has raw access to the machine, memory, network devices, graphics cards, and so on. User space has virtual access to these things. The user space is the original “virtual machines”, before kernels got so bloated that we needed a third layer to virtualize them too.

This separation between kernel and user has two main benefits. The first is security, controlling which bit of software has access to what. It means, for example, that one user on the machine can’t access another’s files. The second benefit is stability: if one program crashes, the others continue to run unaffected.

Downside of a Kernel Module

Writing a search program as a kernel module (instead of a user space module) defeats the benefits of user space programs, making the machine less stable and less secure.

Moreover, the sort of thing this module does (parsing emails) has a history of big gapping security flaws. Parsing stuff in the kernel makes cybersecurity experts run away screaming in terror.

On the other hand, people have been doing security stuff (SSL implementations and anti-virus scanning) in the kernel in other situations, so it’s not unprecedented. I mean, it’s still wrong, but it’s been done before.

Upside of a Kernel Module

If doing this is as a kernel module (instead of in user space) is so bad, then why does Yahoo do it? It’s probably due to the widely held, but false, belief that putting stuff in the kernel makes it faster.

Everybody knows that kernels are faster, for two reasons. First is that as a program runs, making a system call switches context, from running in user space to running in kernel space. This step is expensive/slow. Kernel modules don’t incur this expense, because code just jumps from one location in the kernel to another. The second performance issue is virtual memory, where reading memory requires an extra step in user space, to translate the virtual memory address to a physical one. Kernel modules access physical memory directly, without this extra step.

But everyone is wrong. Using features like hugepages gets rid of the cost of virtual memory translation cost. There are ways to mitigate the cost of user/kernel transitions, such as moving data in bulk instead of a little bit at a time. Also, CPUs have improved in recent years, dramatically reducing the cost of a kernel/user transition.

The problem we face, though, is inertia. Everyone knows moving modules into the kernel makes things faster. It’s hard getting them to un-learn what they’ve been taught.

Also, following this logic, Yahoo may already have many email handling functions in the kernel. If they’ve already gone down the route of bad design, then they’d have to do this email search as a kernel module as well, to avoid the user/kernel transition cost.

Another possible reason for the kernel-module is that it’s what the programmers knew how to do. That’s especially true if the contractor has experience with other kernel software, such as NSA implants. They might’ve read Phrack magazine on the topic, which might have been their sole education on the subject. [http://phrack.org/issues/61/13.html]

How it was probably done

I don’t know Yahoo’s infrastructure. Presumably they have front-end systems designed to balance the load (and accelerate SSL processing), and back-end systems that do the heavy processing, such as spam and virus checking.

The typical way to do this sort of thing (search) is simply tap into the network traffic, either as a separate computer sniffing (eavesdropping on) the network, or something within the system that taps into the network traffic, such as a netfilter module. Netfilter is the Linux firewall mechanism, and has ways to easily “hook” into specific traffic, either from user space or from a kernel module. There is also a related user space mechanism of hooking network APIs like recv() with a preload shared library.

This traditional mechanism doesn’t work as well anymore. For one thing, incoming email traffic is likely encrypted using SSL (using STARTTLS, for example). For another thing, companies are increasingly encrypting intra-data-center traffic, either with SSL or with hard-coded keys.

Therefore, instead of tapping into network traffic, the code might tap directly into the mail handling software. A good example of this is Sendmail’s milter interface, that allows the easy creation of third-party mail filtering applications, specifically for spam and anti-virus.

But it would be insane to write a milter as a kernel module, since mail handling is done in user space, thus adding unnecessary user/kernel transitions. Consequently, we make the assumption that Yahoo’s intra-data-center traffic in unencrypted, and that for FISA search thing, they wrote something like a kernel-module with netfilter hooks.

How it should’ve been done

Assuming the above guess is correct, that they used kernel netfilter hooks, there are a few alternatives.

They could do user space netfilter hooks instead, but they do have a performance impact. They require a transition from the kernel to user, then a second transition back into the kernel. If the system is designed for high performance, this might be a noticeable performance impact. I doubt it, as it’s still small compared to the rest of the computations involved, but it’s the sort of thing that engineers are prejudiced against, even before they measure the performance impact.

A better way of doing it is hooking the libraries. These days, most software uses shared libraries (.so) to make system calls like recv(). You can write your own shared library, and preload it. When the library function is called, you do your own processing, then call the original function.

Hooking the libraries then lets you tap into the network traffic, but without any additional kernel/user transition.

Yet another way is simple changes in the mail handling software that allows custom hooks to be written.

Third party contractors

We’ve been thinking in terms of technical solutions. There is also the problem of politics.

Almost certainly, the solution was developed by outsiders, by defense contractors like Booz-Allen. (I point them out because of the whole Snowden/Martin thing). This restricts your technical options.

You don’t want to give contractors access to your source code. Nor do you want to the contractors to be making custom changes to your source code, such as adding hooks. Therefore, you are looking at external changes, such as hooking the network stack.

The advantage of a netfilter hook in the kernel is that it has the least additional impact on the system. It can be developed and thoroughly tested by Booz-Allen, then delivered to Yahoo!, who can then install it with little effort.

This is my #1 guess why this was a kernel module – it allowed the most separation between Yahoo! and a defense contractor who wrote it. In other words, there is no technical reason for it — but a political reason.

Let’s talk search

There two ways to search things: using an NFA and using a DFA.

An NFA is the normal way of using regex, or grep. It allows complex patterns to be written, but it requires a potentially large amount of CPU power (i.e. it’s slow). It also requires backtracking within a message, thus meaning the entire email must be reassembled before searching can begin.

The DFA alternative instead creates a large table in memory, then does a single pass over a message to search. Because it does only a single pass, without backtracking, the message can be streamed through the search module, without needing to reassemble the message. In theory, anything searched by an NFA can be searched by a DFA, though in practice some unbounded regex expressions require too much memory, so DFAs usually require simpler patterns.

The DFA approach, by the way, is about 4-gbps per 2.x-GHz Intel x86 server CPU. Because no reassembly is required, it can tap directly into anything above the TCP stack, like netfilter. Or, it can tap below the TCP stack (like libpcap), but would require some logic to re-order/de-duplicate TCP packets, to present the same ordered stream as TCP.

DFAs would therefore require little or no memory. In contrast, the NFA approach will require more CPU and memory just to reassemble email messages, and the search itself would also be slower.

The naïve approach to searching is to use NFAs. It’s what most people start out with. The smart approach is to use DFAs. You see that in the evolution of the Snort intrusion detection engine, where they started out using complex NFAs and then over the years switched to the faster DFAs.

You also see it in the network processor market. These are specialized CPUs designed for things like firewalls. They advertise fast regex acceleration, but what they really do is just convert NFAs into something that is mostly a DFA, which you can do on any processor anyway. I have a low opinion of network processors, since what they accelerate are bad decisions. Correctly designed network applications don’t need any special acceleration, except maybe SSL public-key crypto.

So, what the government’s code needs to do is a very lightweight parse of the SMTP protocol in order to extract the from/to email addresses, then a very lightweight search of the message’s content in order to detect if any of the offending strings have been found. When the pattern is found, it then reports the addresses it found.

Conclusion

I don’t know Yahoo’s system for processing incoming emails. I don’t know the contents of the court order forcing them to do a search, and what needs to be secret. Therefore, I’m only making guesses here.

But they are educated guesses. In 9 times out of 10 in situations similar to Yahoo, I’m guessing that a “kernel module” would be the most natural solution. It’s how engineers are trained to think, and it would likely be the best fit organizationally. Sure, it really REALLY annoys cybersecurity experts, but nobody cares what we think, so that doesn’t matter.

Continue reading WTF Yahoo/FISA search in kernel?→

Posted in Uncategorized

WTF Yahoo/FISA search in kernel?

Posted on by

A surprising detail in the Yahoo/FISA email search scandal is that they do it with a kernel module. I thought I’d write up some (rambling) notes.

What the government was searching for

As described in the previoius blog post, we’ll assume the government is searching for the following string, and possibly other strings like it within emails:

### Begin ASRAR El Mojahedeen v2.0 Encrypted Message ###

I point this out because it’s simple search identifying things. It’s not natural language processing. It’s not searching for phrases like “bomb president”.

Also, it’s not AV/spam/childporn processing. Those look at different things. For example, filtering message containing childporn involves calculating a SHA2 hash of email attachments and looking up the hashes in a table of known bad content. This is quite different from searching.

The Kernel vs. User Space

Operating systems have two parts, the kernel and user space. The kernel is the operating system proper (e.g. the “Linux kernel”). The software we run is in user space, such as browsers, word processors, games, web servers, databases, GNU utilities [sic], and so on.

The kernel has raw access to the machine, memory, network devices, graphics cards, and so on. User space has virtual access to these things. The user space is the original “virtual machines”, before kernels got so bloated that we needed a third layer to virtualize them too.

This separation between kernel and user has two main benefits. The first is security, controlling which bit of software has access to what. It means, for example, that one user on the machine can’t access another’s files. The second benefit is stability: if one program crashes, the others continue to run unaffected.

Downside of a Kernel Module

Writing a search program as a kernel module (instead of a user space module) defeats the benefits of user space programs, making the machine less stable and less secure.

Moreover, the sort of thing this module does (parsing emails) has a history of big gapping security flaws. Parsing stuff in the kernel makes cybersecurity experts run away screaming in terror.

On the other hand, people have been doing security stuff (SSL implementations and anti-virus scanning) in the kernel in other situations, so it’s not unprecedented. I mean, it’s still wrong, but it’s been done before.

Upside of a Kernel Module

If doing this is as a kernel module (instead of in user space) is so bad, then why does Yahoo do it? It’s probably due to the widely held, but false, belief that putting stuff in the kernel makes it faster.

Everybody knows that kernels are faster, for two reasons. First is that as a program runs, making a system call switches context, from running in user space to running in kernel space. This step is expensive/slow. Kernel modules don’t incur this expense, because code just jumps from one location in the kernel to another. The second performance issue is virtual memory, where reading memory requires an extra step in user space, to translate the virtual memory address to a physical one. Kernel modules access physical memory directly, without this extra step.

But everyone is wrong. Using features like hugepages gets rid of the cost of virtual memory translation cost. There are ways to mitigate the cost of user/kernel transitions, such as moving data in bulk instead of a little bit at a time. Also, CPUs have improved in recent years, dramatically reducing the cost of a kernel/user transition.

The problem we face, though, is inertia. Everyone knows moving modules into the kernel makes things faster. It’s hard getting them to un-learn what they’ve been taught.

Also, following this logic, Yahoo may already have many email handling functions in the kernel. If they’ve already gone down the route of bad design, then they’d have to do this email search as a kernel module as well, to avoid the user/kernel transition cost.

Another possible reason for the kernel-module is that it’s what the programmers knew how to do. That’s especially true if the contractor has experience with other kernel software, such as NSA implants. They might’ve read Phrack magazine on the topic, which might have been their sole education on the subject. [http://phrack.org/issues/61/13.html]

How it was probably done

I don’t know Yahoo’s infrastructure. Presumably they have front-end systems designed to balance the load (and accelerate SSL processing), and back-end systems that do the heavy processing, such as spam and virus checking.

The typical way to do this sort of thing (search) is simply tap into the network traffic, either as a separate computer sniffing (eavesdropping on) the network, or something within the system that taps into the network traffic, such as a netfilter module. Netfilter is the Linux firewall mechanism, and has ways to easily “hook” into specific traffic, either from user space or from a kernel module. There is also a related user space mechanism of hooking network APIs like recv() with a preload shared library.

This traditional mechanism doesn’t work as well anymore. For one thing, incoming email traffic is likely encrypted using SSL (using STARTTLS, for example). For another thing, companies are increasingly encrypting intra-data-center traffic, either with SSL or with hard-coded keys.

Therefore, instead of tapping into network traffic, the code might tap directly into the mail handling software. A good example of this is Sendmail’s milter interface, that allows the easy creation of third-party mail filtering applications, specifically for spam and anti-virus.

But it would be insane to write a milter as a kernel module, since mail handling is done in user space, thus adding unnecessary user/kernel transitions. Consequently, we make the assumption that Yahoo’s intra-data-center traffic in unencrypted, and that for FISA search thing, they wrote something like a kernel-module with netfilter hooks.

How it should’ve been done

Assuming the above guess is correct, that they used kernel netfilter hooks, there are a few alternatives.

They could do user space netfilter hooks instead, but they do have a performance impact. They require a transition from the kernel to user, then a second transition back into the kernel. If the system is designed for high performance, this might be a noticeable performance impact. I doubt it, as it’s still small compared to the rest of the computations involved, but it’s the sort of thing that engineers are prejudiced against, even before they measure the performance impact.

A better way of doing it is hooking the libraries. These days, most software uses shared libraries (.so) to make system calls like recv(). You can write your own shared library, and preload it. When the library function is called, you do your own processing, then call the original function.

Hooking the libraries then lets you tap into the network traffic, but without any additional kernel/user transition.

Yet another way is simple changes in the mail handling software that allows custom hooks to be written.

Third party contractors

We’ve been thinking in terms of technical solutions. There is also the problem of politics.

Almost certainly, the solution was developed by outsiders, by defense contractors like Booz-Allen. (I point them out because of the whole Snowden/Martin thing). This restricts your technical options.

You don’t want to give contractors access to your source code. Nor do you want to the contractors to be making custom changes to your source code, such as adding hooks. Therefore, you are looking at external changes, such as hooking the network stack.

The advantage of a netfilter hook in the kernel is that it has the least additional impact on the system. It can be developed and thoroughly tested by Booz-Allen, then delivered to Yahoo!, who can then install it with little effort.

This is my #1 guess why this was a kernel module – it allowed the most separation between Yahoo! and a defense contractor who wrote it. In other words, there is no technical reason for it — but a political reason.

Let’s talk search

There two ways to search things: using an NFA and using a DFA.

An NFA is the normal way of using regex, or grep. It allows complex patterns to be written, but it requires a potentially large amount of CPU power (i.e. it’s slow). It also requires backtracking within a message, thus meaning the entire email must be reassembled before searching can begin.

The DFA alternative instead creates a large table in memory, then does a single pass over a message to search. Because it does only a single pass, without backtracking, the message can be streamed through the search module, without needing to reassemble the message. In theory, anything searched by an NFA can be searched by a DFA, though in practice some unbounded regex expressions require too much memory, so DFAs usually require simpler patterns.

The DFA approach, by the way, is about 4-gbps per 2.x-GHz Intel x86 server CPU. Because no reassembly is required, it can tap directly into anything above the TCP stack, like netfilter. Or, it can tap below the TCP stack (like libpcap), but would require some logic to re-order/de-duplicate TCP packets, to present the same ordered stream as TCP.

DFAs would therefore require little or no memory. In contrast, the NFA approach will require more CPU and memory just to reassemble email messages, and the search itself would also be slower.

The naïve approach to searching is to use NFAs. It’s what most people start out with. The smart approach is to use DFAs. You see that in the evolution of the Snort intrusion detection engine, where they started out using complex NFAs and then over the years switched to the faster DFAs.

You also see it in the network processor market. These are specialized CPUs designed for things like firewalls. They advertise fast regex acceleration, but what they really do is just convert NFAs into something that is mostly a DFA, which you can do on any processor anyway. I have a low opinion of network processors, since what they accelerate are bad decisions. Correctly designed network applications don’t need any special acceleration, except maybe SSL public-key crypto.

So, what the government’s code needs to do is a very lightweight parse of the SMTP protocol in order to extract the from/to email addresses, then a very lightweight search of the message’s content in order to detect if any of the offending strings have been found. When the pattern is found, it then reports the addresses it found.

Conclusion

I don’t know Yahoo’s system for processing incoming emails. I don’t know the contents of the court order forcing them to do a search, and what needs to be secret. Therefore, I’m only making guesses here.

But they are educated guesses. In 9 times out of 10 in situations similar to Yahoo, I’m guessing that a “kernel module” would be the most natural solution. It’s how engineers are trained to think, and it would likely be the best fit organizationally. Sure, it really REALLY annoys cybersecurity experts, but nobody cares what we think, so that doesn’t matter.

Continue reading WTF Yahoo/FISA search in kernel?→

Posted in SBN