Author Archives: Robert Graham

In which I have to debunk a second time

Posted on by

So Slate is doubling-down on their discredited story of a secret Trump server. Tip for journalists: if you are going to argue against an expert debunking your story, try to contact that expert first, so they don’t have to do what I’m going to do here, showing obvious flaws. Also, pay attention to the data.

The experts didn’t find anything

The story claims:

“I spoke with many DNS experts. They found the evidence strongly suggestive of a relationship between the Trump Organization and the bank”.

No, he didn’t. He gave experts limited information and asked them whether it’s consistent with a conspiracy theory. He didn’t ask if it was “suggestive” of the conspiracy theory, or that this was the best theory that fit the data.

This is why “experts” quoted in the press need to go through “media training”, to avoid getting your reputation harmed by bad journalists who try their best to put words in your mouth. You’ll be trained to recognize bad journalists like this, and how not to get sucked into their fabrications.

Jean Camp isn’t an expert

On the other hand, Jean Camp isn’t an expert. I’ve never heard of her before. She gets details wrong. Take for example in this blogpost where she discusses lookups for the domain mail.trump-email.com.moscow.alfaintra.net. She says:

This query is unusual in that is merges two hostnames into one. It makes the most sense as a human error in inserting a new hostname in some dialog window, but neglected to hit the backspace to delete the old hostname.

Uh, no. It’s normal DNS behavior with non-FQDNs. If the lookup for a name fails, computers will try again, pasting the local domain on the end. In other words, when Twitter’s DNS was taken offline by the DDoS attack a couple weeks ago, those monitoring DNS saw a zillion lookups for names like “www.twitter.com.example.com“.

I’ve reproduced this on my desktop by configuring the suffix moscow.alfaintra.net.

I then pinged “mail1.trump-email.com” and captured the packets. As you can see, after the initial lookups fail, Windows tried appending the suffix.

I don’t know what Jean Camp is an expert of, but this is sorta a basic DNS concept. It’s surprising she’d get it wrong. Of course, she may be an expert in DNS who simply had a brain fart (this happens to all of us), but looking across her posts and tweets, she doesn’t seem to be somebody who has a lot of experience with DNS. Sorry for impugning her credibility, but that’s the way the story is written. It demands that we trust the quoted “experts”. 
Call up your own IT department at Slate. Ask your IT nerds if this is how DNS operates. Note: I’m saying your average, unremarkable IT nerds can debunk an “expert” you quote in your story.
Understanding “spam” and “blacklists”

The new article has a paragraph noting that the IP address doesn’t appear on spam blocklists:

Was the server sending spam—unsolicited mail—as opposed to legitimate commercial marketing? There are databases that assiduously and comprehensively catalog spam. I entered the internet protocal address for mail1.trump-email.com to check if it ever showed up in Spamhaus and DNSBL.info. There were no traces of the IP address ever delivering spam.

This is a profound misunderstanding of how these things work.

Colloquially, we call those sending mass marketing emails, like Cendyn, “spammers”. But those running blocklists have a narrower definition. If  emails contain an option to “opt-out” of future emails, then it’s technically not “spam”.

Cendyn is constantly getting added to blocklists when people complain. They spend considerable effort contacting the many organizations maintaining blocklists, proving they do “opt-outs”, and getting “white-listed” instead of “black-listed”. Indeed, the entire spam-blacklisting industry is a bit of scam — getting white-listed often involves a bit of cash.

Those maintaining blacklists only go back a few months. The article is in error saying there’s no record ever of Cendyn sending spam. Instead, if an address comes up clean, it means there’s no record for the past few months. And, if Cendyn is in the white-lists, there would be no record of “spam” at all, anyway.

As somebody who frequently scans the entire Internet, I’m constantly getting on/off blacklists. It’s a real pain. At the moment, my scanner address “209.126.230.71” doesn’t appear to be on any blacklists. Next time a scan kicks off, it’ll probably get added — but only by a few, because most have white-listed it.

There is no IP address limitation

The story repeats the theory, which I already debunked, that the server has a weird configuration that limits who can talk to it:

The scientists theorized that the Trump and Alfa Bank servers had a secretive relationship after testing the behavior of mail1.trump-email.com using sites like Pingability. When they attempted to ping the site, they received the message “521 lvpmta14.lstrk.net does not accept mail from you.”

No, that’s how Listrake (who is the one who actually controls the server) configures all their marketing servers. Anybody can confirm this themselves by ping all the servers in this range:
In case you don’t want to do scans yourself, you can look up on Shodan and see that there’s at least 4000 servers around the Internet who give the same error message.

Again, go back to Chris Davis in your original story ask him about this. He’ll confirm that there’s nothing nefarious or weird going on here, that it’s just how Listrak has decided to configure all it’s spam-sending engines.

Either this conspiracy goes much deeper, with hundreds of servers involved, or this is a meaningless datapoint.
Where did the DNS logs come from?
Tea Leaves and Jean Camp are showing logs of private communications. Where did these logs come from? This information isn’t public. It means somebody has done something like hack into Alfa Bank. Or it means researchers who monitor DNS (for maintaing DNS, and for doing malware research) have broken their NDAs and possibly the law.
The data is incomplete and inconsistent. Those who work for other companies, like Dyn, claim it doesn’t match their own data. We have good reason to doubt these logs. There’s a good chance that the source doesn’t have as comprehensive a view as “Tea Leaves” claim. There’s also a good chance the data has been manipulated.
Specifically, I have as source who claims records for trump-email.com were changed in June, meaning either my source or Tea Leaves is lying.
Until we know more about the source of the data, it’s impossible to believe the conclusions that only Alfa Bank was doing DNS lookups.

By the way, if you are a company like Alfa Bank, and you don’t want the “research” community from seeing leaked intranet DNS requests, then you should probably reconfigure your DNS resolvers. You’ll want to look into RFC7816 “query minimization”, supported by the Unbound and Knot resolvers.

Do the graphs show interesting things?

The original “Tea Leaves” researchers are clearly acting in bad faith. They are trying to twist the data to match their conclusions. For example, in the original article, they claim that peaks in the DNS activity match campaign events. But looking at the graph, it’s clear these are unrelated. It display the common cognitive bias of seeing patterns that aren’t there.
Likewise, they claim that the timing throughout the day matches what you’d expect from humans interacting back and forth between Moscow and New York. No. This is what the activity looks like, graphing the number of queries by hour:
As you can see, there’s no pattern. When workers go home at 5pm in New York City, it’s midnight in Moscow. If humans were involved, you’d expect an eight hour lull during that time. Likewise, when workers arrive at 9am in New York City, you expect a spike in traffic for about an hour until workers in Moscow go home. You see none of that here. What you instead see is a random distribution throughout the day — the sort of distribution you’d expect if this were DNS lookups from incoming spam.
The point is that we know the original “Tea Leaves” researchers aren’t trustworthy, that they’ve convinced themselves of things that just aren’t there.
Does Trump control the server in question?

OMG, this post asks the question, after I’ve debunked the original story, and still gotten the answer wrong.
The answer is that Listrak controls the server. Not even Cendyn controls it, really, they just contract services from Listrak. In other words, not only does Trump not control it, the next level company (Cendyn) also doesn’t control it.
Does Trump control the domain in question?
OMG, this new story continues to make the claim the Trump Organization controls the domain trump-email.com, despite my debunking that Cendyn controls the domain.
Look at the WHOIS info yourself. All the contact info goes to Cendyn. It fits the pattern Cendyn chooses for their campaigns.
  • trump-email.com
  • mjh-email.com
  • denihan-email.com
  • hyatt-email.com
Cendyn even spells “Trump Orgainzation” wrong.

There’s a difference between a “server” and a “name”

The article continues to make trivial technical errors, like confusing what a server is with what a domain name is. For example:

One of the intriguing facts in my original piece was that the Trump server was shut down on Sept. 23, two days after the New York Times made inquiries to Alfa Bank

The server has never been shutdown. Instead, the name “mail1.trump-email.com” was removed from Cendyn’s DNS servers.
It’s impossible to debunk everything in these stories because they garble the technical details so much that it’s impossible to know what the heck they are claiming.
Why did Cendyn change things after Alfa Bank was notified?

It’s a curious coincidence that Cendyn changed their DNS records a couple days after the NYTimes contacted Alfa Bank.
But “coincidence” is all it is. I have years of experience with investigating data breaches. I know that such coincidences abound. There’s always weird coincidence that you are certain are meaningful, but which by the end of the investigation just aren’t.
The biggest source of coincidences is that IT is always changing things and always messing things up. It’s the nature of IT. Thus, you’ll always see a change in IT that matches some other event. Those looking for conspiracies ignore the changes that don’t match, and focus on the one that does, so it looms suspiciously.
As I’ve mentioned before, I have source that says Cendyn changed things around in June. This makes me believe that “Tea Leaves” is editing changes to highlight the one in September.
In any event, many people have noticed that the registrar email “Emily McMullin” has the same last name as Evan McMullin running against Trump in Utah. This supports my point: when you do hacking investigations, you find irrelevant connections all over the freakin’ place.
“Experts stand by their analysis”

This new article states:

I’ve checked back with eight of the nine computer scientists and engineers I consulted for my original story, and they all stood by their fundamental analysis

Well, of course, they don’t want to look like idiots. But notice the subtle rephrasing of the question: the experts stand by their analysis. It doesn’t mean the same thing as standing behind the reporters analysis. The experts made narrow judgements, which even I stand behind as mostly correct, given the data they were given at the time. None of them were asked whether the entire conspiracy theory holds up.
What you should ask is people like Chris Davis or Paul Vixie whether they stand behind my analysis in the past two posts. Or really, ask any expert. I’ve documented things in sufficient clarity. For example, go back to Chris Davis and ask him again about the “limited IP address” theory, and whether it holds up against my scan of that data center above.
Conclusion

Other major news outlets all passed on the story, because even non experts know it’s flawed. The data means nothing. The Slate journalist nonetheless went forward with the story, tricking experts, and finding some non-experts.
But as I’ve shown, given a complete technical analysis, the story falls apart. Most of what’s strange is perfectly normal. The data itself (the DNS logs) are untrustworthy. It builds upon unknown things (like how the mail server rejects IP address) as “unknowable” things that confirm the conspiracy, when they are in fact simply things unknown at the current time, which can become knowable with a little research.

What I show in my first post, and this post, is more data. This data shows context. This data explains the unknowns that Slate present. Moreover, you don’t have to trust me — anybody can replicate my work and see for themselves.


Continue reading In which I have to debunk a second time→

Posted in Uncategorized

In which I have to debunk a second time

Posted on by

So Slate is doubling-down on their discredited story of a secret Trump server. Tip for journalists: if you are going to argue against an expert debunking your story, try to contact that expert first, so they don’t have to do what I’m going to do here, showing obvious flaws.

The experts didn’t find anything

The story claims:

“I spoke with many DNS experts. They found the evidence strongly suggestive of a relationship between the Trump Organization and the bank”.

No, you didn’t. You gave experts limited information and asked them whether it’s consistent with your conspiracy theory. Of course it’s consistent with almost any conspiracy theory you want to concoct. What you didn’t ask is for experts to try to disprove the theory.

Go back and ask Chris Davis and Paul Vixie which is more credible, your version of events, or my version. I will vouch that these two are really top experts in this field, and you should trust them.

This is why people quoted in the press need to go through “media training”, to avoid getting your reputation harmed by bad journalists who try their best to put words in your mouth. You’ll be trained to recognize bad journalists like this, and how not to get sucked into their fabrications.

Jean Camp isn’t an expert

On the other hand, Jean Camp isn’t an expert. I’ve never heard of her before. She gets details wrong. Take for example in this blogpost of here’s where she discusses strange lookups. Specifically, she comments on lookups for the domain mail.trump-email.com.moscow.alfaintra.net. She says:

This query is unusual in that is merges two hostnames into one. It makes the most sense as a human error in inserting a new hostname in some dialog window, but neglected to hit the backspace to delete the old hostname.

Uh, no. It’s normal DNS behavior with non-FQDNs. If the lookup for a name fails, computers will try again, pasting the local domain on the end. In other words, when Twitter’s DNS was taken offline by the DDoS attack a couple weeks ago, those monitoring DNS saw a zillion lookups for names like “www.twitter.com.example.com“.
I don’t know what Jean Camp is an expert of, but this is sorta a basic DNS concept. It’s surprising she’d get it wrong. Of course, she may be an expert in DNS who simply had a brain fart (this happens to all of us), but looking across her posts and tweets, she doesn’t seem to be somebody who has a lot of experience with DNS.
Call up your own IT department at Slate. Ask your IT nerds if this is how DNS operates. Note: I’m saying your average, unremarkable IT nerds can debunk an “expert” you quote in your story.
There is no IP address limitation

The story repeats the theory, which I already debunked, that the server has a weird configuration that limits who can talk to it:

The scientists theorized that the Trump and Alfa Bank servers had a secretive relationship after testing the behavior of mail1.trump-email.com using sites like Pingability. When they attempted to ping the site, they received the message “521 lvpmta14.lstrk.net does not accept mail from you.”

No, that’s how Listrake (who is the one who actually controls the server) configures all their marketing servers. Anybody can confirm this themselves by ping all the servers in this range:
Again, go back to Chris Davis in your original story ask him about this. He’ll confirm that there’s nothing nefarious or weird going on here, that it’s just how Listrak has decided to configure all it’s spam-sending engines.
Either this conspiracy goes much deeper, with hundreds of servers involved, or this is a meaningless datapoint.
Where did the DNS logs come from?
Tea Leaves and Jean Camp are showing logs of private communications. Where did these logs come from? This information isn’t public. It means somebody has done something like hack into Alfa Bank. Or it means researchers who monitor DNS (for maintaing DNS, and for doing malware research) have broken their NDAs and possibly the law.
The data is incomplete and inconsistent. Those who work for other companies, like Dyn, claim it doesn’t match their own data. We have good reason to doubt these logs. There’s a good chance that the source doesn’t have as comprehensive a view as “Tea Leaves” claim. There’s also a good chance the data has been manipulated.
Specifically, I have as source who claims records for trump-email.com were changed in June, meaning either my source or Tea Leaves is lying.
Until we know more about the source of the data, it’s impossible to believe the conclusions that only Alfa Bank was doing DNS lookups.
Do the graphs show interesting things?

The original “Tea Leaves” researchers are clearly acting in bad faith. They are trying to twist the data to match their conclusions. For example, in the original article, they claim that peaks in the DNS activity match campaign events. But looking at the graph, it’s clear these are unrelated. It display the common cognitive bias of seeing patterns that aren’t there.
Likewise, they claim that the timing throughout the day matches what you’d expect from humans interacting back and forth between Moscow and New York. No. This is what the activity looks like, graphing the number of queries by hour:
As you can see, there’s no pattern. When workers go home at 5pm in New York City, it’s midnight in Moscow. If humans were involved, you’d expect an eight hour lull during that time. Likewise, when workers arrive at 9am in New York City, you expect a spike in traffic for about an hour until workers in Moscow go home. You see none of that here. What you instead see is a random distribution throughout the day — the sort of distribution you’d expect if this were DNS lookups from incoming spam.
The point is that we know the original “Tea Leaves” researchers aren’t trustworthy, that they’ve convinced themselves of things that just aren’t there.
Does Trump control the server in question?

OMG, this post asks the question, after I’ve debunked the original story, and still gotten the answer wrong.
The answer is that Listrak controls the server. Not even Cendyn controls it, really, they just contract services from Listrak. In other words, not only does Trump not control it, the next level company (Cendyn) also doesn’t control it.
Does Trump control the domain in question?
OMG, this new story continues to make the claim the Trump Organization controls the domain trump-email.com, despite my debunking that Cendyn controls the domain.
Look at the WHOIS info yourself. All the contact info goes to Cendyn. It fits the pattern Cendyn chooses for their campaigns.
  • trump-email.com
  • mjh-email.com
  • denihan-email.com
  • hyatt-email.com
Cendyn even spells “Trump Orgainzation” wrong.
There’s a difference between a “server” and a “name”

The article continues to make trivial technical errors, like confusing what a server is with what a domain name is. For example:

One of the intriguing facts in my original piece was that the Trump server was shut down on Sept. 23, two days after the New York Times made inquiries to Alfa Bank

The server has never been shutdown. Instead, the name “mail1.trump-email.com” was removed from Cendyn’s DNS servers.
It’s impossible to debunk everything in these stories because they garble the technical details so much that it’s impossible to know what the heck they are claiming.
Why did Cendyn change things after Alfa Bank was notified?

It’s a curious coincidence that Cendyn changed their DNS records a couple days after the NYTimes contacted Alfa Bank.
But “coincidence” is all it is. I have years of experience with investigating data breaches. I know that such coincidences abound. There’s always weird coincidence that you are certain are meaningful, but which by the end of the investigation just aren’t.
The biggest source of coincidences is that IT is always changing things and always messing things up. It’s the nature of IT. Thus, you’ll always see a change in IT that matches some other event. Those looking for conspiracies ignore the changes that don’t match, and focus on the one that does, so it looms suspiciously.
As I’ve mentioned before, I have source that says Cendyn changed things around in June. This makes me believe that “Tea Leaves” is editing changes to highlight the one in September.
In any event, many people have noticed that the registrar email “Emily McMullin” has the same last name as Evan McMullin running against Trump in Utah. This supports my point: when you do hacking investigations, you find irrelevant connections all over the freakin’ place.
“Experts stand by their analysis”

This new article states:

I’ve checked back with eight of the nine computer scientists and engineers I consulted for my original story, and they all stood by their fundamental analysis

Well, of course, they don’t want to look like idiots. But notice the subtle rephrasing of the question: the experts stand by their analysis. It doesn’t mean the same thing as standing behind the reporters analysis. The experts made narrow judgements, which even I stand behind as mostly correct, given the data they were given at the time. None of them were asked whether the entire conspiracy theory holds up.
What you should ask is people like Chris Davis or Paul Vixie whether they stand behind my analysis in the past two posts. Or really, ask any expert. I’ve documented things in sufficient clarity. For example, go back to Chris Davis and ask him again about the “limited IP address” theory, and whether it holds up against my scan of that data center above.
Conclusion

Other major news outlets all passed on the story, because even non experts know it’s flawed. The data means nothing. The Slate journalist nonetheless went forward with the story, tricking experts, and finding some non-experts.
But as I’ve shown, given a complete technical analysis, the story falls apart. Most of what’s strange is perfectly normal. The data itself (the DNS logs) are untrustworthy. It builds upon unknown things (like how the mail server rejects IP address) as “unknowable” things that confirm the conspiracy, when they are in fact simply things unknown at the current time, which can become knowable with a little research.

Continue reading In which I have to debunk a second time→

Posted in SBN

In which I have to debunk a second time

Posted on by

So Slate is doubling-down on their discredited story of a secret Trump server. Tip for journalists: if you are going to argue against an expert debunking your story, try to contact that expert first, so they don’t have to do what I’m going to do here, showing obvious flaws. Also, pay attention to the data.

The experts didn’t find anything

The story claims:

“I spoke with many DNS experts. They found the evidence strongly suggestive of a relationship between the Trump Organization and the bank”.

No, he didn’t. He gave experts limited information and asked them whether it’s consistent with a conspiracy theory. He didn’t ask if it was “suggestive” of the conspiracy theory, or that this was the best theory that fit the data.

This is why “experts” quoted in the press need to go through “media training”, to avoid getting your reputation harmed by bad journalists who try their best to put words in your mouth. You’ll be trained to recognize bad journalists like this, and how not to get sucked into their fabrications.

Jean Camp isn’t an expert

On the other hand, Jean Camp isn’t an expert. I’ve never heard of her before. She gets details wrong. Take for example in this blogpost where she discusses lookups for the domain mail.trump-email.com.moscow.alfaintra.net. She says:

This query is unusual in that is merges two hostnames into one. It makes the most sense as a human error in inserting a new hostname in some dialog window, but neglected to hit the backspace to delete the old hostname.

Uh, no. It’s normal DNS behavior with non-FQDNs. If the lookup for a name fails, computers will try again, pasting the local domain on the end. In other words, when Twitter’s DNS was taken offline by the DDoS attack a couple weeks ago, those monitoring DNS saw a zillion lookups for names like “www.twitter.com.example.com“.

I’ve reproduced this on my desktop by configuring the suffix moscow.alfaintra.net.

I then pinged “mail1.trump-email.com” and captured the packets. As you can see, after the initial lookups fail, Windows tried appending the suffix.

I don’t know what Jean Camp is an expert of, but this is sorta a basic DNS concept. It’s surprising she’d get it wrong. Of course, she may be an expert in DNS who simply had a brain fart (this happens to all of us), but looking across her posts and tweets, she doesn’t seem to be somebody who has a lot of experience with DNS. Sorry for impugning her credibility, but that’s the way the story is written. It demands that we trust the quoted “experts”. 
Call up your own IT department at Slate. Ask your IT nerds if this is how DNS operates. Note: I’m saying your average, unremarkable IT nerds can debunk an “expert” you quote in your story.
Understanding “spam” and “blacklists”

The new article has a paragraph noting that the IP address doesn’t appear on spam blocklists:

Was the server sending spam—unsolicited mail—as opposed to legitimate commercial marketing? There are databases that assiduously and comprehensively catalog spam. I entered the internet protocal address for mail1.trump-email.com to check if it ever showed up in Spamhaus and DNSBL.info. There were no traces of the IP address ever delivering spam.

This is a profound misunderstanding of how these things work.

Colloquially, we call those sending mass marketing emails, like Cendyn, “spammers”. But those running blocklists have a narrower definition. If  emails contain an option to “opt-out” of future emails, then it’s technically not “spam”.

Cendyn is constantly getting added to blocklists when people complain. They spend considerable effort contacting the many organizations maintaining blocklists, proving they do “opt-outs”, and getting “white-listed” instead of “black-listed”. Indeed, the entire spam-blacklisting industry is a bit of scam — getting white-listed often involves a bit of cash.

Those maintaining blacklists only go back a few months. The article is in error saying there’s no record ever of Cendyn sending spam. Instead, if an address comes up clean, it means there’s no record for the past few months. And, if Cendyn is in the white-lists, there would be no record of “spam” at all, anyway.

As somebody who frequently scans the entire Internet, I’m constantly getting on/off blacklists. It’s a real pain. At the moment, my scanner address “209.126.230.71” doesn’t appear to be on any blacklists. Next time a scan kicks off, it’ll probably get added — but only by a few, because most have white-listed it.

There is no IP address limitation

The story repeats the theory, which I already debunked, that the server has a weird configuration that limits who can talk to it:

The scientists theorized that the Trump and Alfa Bank servers had a secretive relationship after testing the behavior of mail1.trump-email.com using sites like Pingability. When they attempted to ping the site, they received the message “521 lvpmta14.lstrk.net does not accept mail from you.”

No, that’s how Listrake (who is the one who actually controls the server) configures all their marketing servers. Anybody can confirm this themselves by ping all the servers in this range:
In case you don’t want to do scans yourself, you can look up on Shodan and see that there’s at least 4000 servers around the Internet who give the same error message.

Again, go back to Chris Davis in your original story ask him about this. He’ll confirm that there’s nothing nefarious or weird going on here, that it’s just how Listrak has decided to configure all it’s spam-sending engines.

Either this conspiracy goes much deeper, with hundreds of servers involved, or this is a meaningless datapoint.
Where did the DNS logs come from?
Tea Leaves and Jean Camp are showing logs of private communications. Where did these logs come from? This information isn’t public. It means somebody has done something like hack into Alfa Bank. Or it means researchers who monitor DNS (for maintaing DNS, and for doing malware research) have broken their NDAs and possibly the law.
The data is incomplete and inconsistent. Those who work for other companies, like Dyn, claim it doesn’t match their own data. We have good reason to doubt these logs. There’s a good chance that the source doesn’t have as comprehensive a view as “Tea Leaves” claim. There’s also a good chance the data has been manipulated.
Specifically, I have as source who claims records for trump-email.com were changed in June, meaning either my source or Tea Leaves is lying.
Until we know more about the source of the data, it’s impossible to believe the conclusions that only Alfa Bank was doing DNS lookups.

By the way, if you are a company like Alfa Bank, and you don’t want the “research” community from seeing leaked intranet DNS requests, then you should probably reconfigure your DNS resolvers. You’ll want to look into RFC7816 “query minimization”, supported by the Unbound and Knot resolvers.

Do the graphs show interesting things?

The original “Tea Leaves” researchers are clearly acting in bad faith. They are trying to twist the data to match their conclusions. For example, in the original article, they claim that peaks in the DNS activity match campaign events. But looking at the graph, it’s clear these are unrelated. It display the common cognitive bias of seeing patterns that aren’t there.
Likewise, they claim that the timing throughout the day matches what you’d expect from humans interacting back and forth between Moscow and New York. No. This is what the activity looks like, graphing the number of queries by hour:
As you can see, there’s no pattern. When workers go home at 5pm in New York City, it’s midnight in Moscow. If humans were involved, you’d expect an eight hour lull during that time. Likewise, when workers arrive at 9am in New York City, you expect a spike in traffic for about an hour until workers in Moscow go home. You see none of that here. What you instead see is a random distribution throughout the day — the sort of distribution you’d expect if this were DNS lookups from incoming spam.
The point is that we know the original “Tea Leaves” researchers aren’t trustworthy, that they’ve convinced themselves of things that just aren’t there.
Does Trump control the server in question?

OMG, this post asks the question, after I’ve debunked the original story, and still gotten the answer wrong.
The answer is that Listrak controls the server. Not even Cendyn controls it, really, they just contract services from Listrak. In other words, not only does Trump not control it, the next level company (Cendyn) also doesn’t control it.
Does Trump control the domain in question?
OMG, this new story continues to make the claim the Trump Organization controls the domain trump-email.com, despite my debunking that Cendyn controls the domain.
Look at the WHOIS info yourself. All the contact info goes to Cendyn. It fits the pattern Cendyn chooses for their campaigns.
  • trump-email.com
  • mjh-email.com
  • denihan-email.com
  • hyatt-email.com
Cendyn even spells “Trump Orgainzation” wrong.

There’s a difference between a “server” and a “name”

The article continues to make trivial technical errors, like confusing what a server is with what a domain name is. For example:

One of the intriguing facts in my original piece was that the Trump server was shut down on Sept. 23, two days after the New York Times made inquiries to Alfa Bank

The server has never been shutdown. Instead, the name “mail1.trump-email.com” was removed from Cendyn’s DNS servers.
It’s impossible to debunk everything in these stories because they garble the technical details so much that it’s impossible to know what the heck they are claiming.
Why did Cendyn change things after Alfa Bank was notified?

It’s a curious coincidence that Cendyn changed their DNS records a couple days after the NYTimes contacted Alfa Bank.
But “coincidence” is all it is. I have years of experience with investigating data breaches. I know that such coincidences abound. There’s always weird coincidence that you are certain are meaningful, but which by the end of the investigation just aren’t.
The biggest source of coincidences is that IT is always changing things and always messing things up. It’s the nature of IT. Thus, you’ll always see a change in IT that matches some other event. Those looking for conspiracies ignore the changes that don’t match, and focus on the one that does, so it looms suspiciously.
As I’ve mentioned before, I have source that says Cendyn changed things around in June. This makes me believe that “Tea Leaves” is editing changes to highlight the one in September.
In any event, many people have noticed that the registrar email “Emily McMullin” has the same last name as Evan McMullin running against Trump in Utah. This supports my point: when you do hacking investigations, you find irrelevant connections all over the freakin’ place.
“Experts stand by their analysis”

This new article states:

I’ve checked back with eight of the nine computer scientists and engineers I consulted for my original story, and they all stood by their fundamental analysis

Well, of course, they don’t want to look like idiots. But notice the subtle rephrasing of the question: the experts stand by their analysis. It doesn’t mean the same thing as standing behind the reporters analysis. The experts made narrow judgements, which even I stand behind as mostly correct, given the data they were given at the time. None of them were asked whether the entire conspiracy theory holds up.
What you should ask is people like Chris Davis or Paul Vixie whether they stand behind my analysis in the past two posts. Or really, ask any expert. I’ve documented things in sufficient clarity. For example, go back to Chris Davis and ask him again about the “limited IP address” theory, and whether it holds up against my scan of that data center above.
Conclusion

Other major news outlets all passed on the story, because even non experts know it’s flawed. The data means nothing. The Slate journalist nonetheless went forward with the story, tricking experts, and finding some non-experts.
But as I’ve shown, given a complete technical analysis, the story falls apart. Most of what’s strange is perfectly normal. The data itself (the DNS logs) are untrustworthy. It builds upon unknown things (like how the mail server rejects IP address) as “unknowable” things that confirm the conspiracy, when they are in fact simply things unknown at the current time, which can become knowable with a little research.

What I show in my first post, and this post, is more data. This data shows context. This data explains the unknowns that Slate present. Moreover, you don’t have to trust me — anybody can replicate my work and see for themselves.


Continue reading In which I have to debunk a second time→

Posted in Uncategorized

Debunking Trump’s "secret server"

Posted on by

According to this Slate article, Trump has a secret server for communicating with Russia. Even Hillary has piled onto this story.

It’s time for Trump to answer serious questions about his ties to Russia. https://t.co/D8oSmyVAR4 pic.twitter.com/07dRyEmPjX

— Hillary Clinton (@HillaryClinton) October 31, 2016

This is nonsense. The evidence available on the Internet is that Trump neither (directly) controls the domain “trump-email.com“, nor has access to the server. Instead, the domain was setup and controlled by Cendyn, a company that does marketing/promotions for hotels, including many of Trump’s hotels. Cendyn outsources the email portions of its campaigns to a company called Listrak, which actually owns/operates the physical server in a data center in Philidelphia.

In other words,  Trump’s response is (minus the political bits) likely true, supported by the evidence. It’s the conclusion I came to even before seeing the response.

When you view this “secret” server in context, surrounded by the other email servers operated by Listrak on behalf of Cendyn, it becomes more obvious what’s going on. In the same Internet address range of Trump’s servers you see a bunch of similar servers, many named [client]-email.com. In other words, trump-email.com is not intended as a normal email server you and I are familiar with, but as a server used for marketing/promotional campaigns.

It’s Cendyn that registered and who controls the trump-email.com domain, as seen in the WHOIS information. That the Trump Organization is the registrant, but not the admin, demonstrates that Trump doesn’t have direct control over it.

When the domain information was changed last September 23, it was Cendyn who did the change, not the Trump Organization. This link lists a bunch of other hotel-related domains that Cendyn likewise controls, some Trump related, some related to Trump’s hotel competitors, like Hyatt and Sheraton.

Cendyn’s claim they are reusing the server for some other purpose is likely true. If you are an enterprising journalist with $399 in your budget, you can find this out. Use the website http://reversewhois.domaintools.com/ to get a complete list of the 641 other domains controlled by Cendyn, then do an MX query for each one to find out which of them is using mail1.trump-email.com as their email server.

This is why we can’t have nice things on the Internet. Investigative journalism is dead. The Internet is full of clues like this if only somebody puts a few resources into figuring things out. For example, organizations that track spam will have information on exactly which promotions this server has been used for in the recent past. Those who operate public DNS resolvers, like Google’s 8.8.8.8, OpenDNS, or Dyn, may have knowledge which domain was related to mail1.trump-email.com.

Indeed, one journalist did call one of the public resolvers, and found other people queried this domain than the two listed in the Slate story — debunking it. I’ve heard from other DNS malware researchers (names remain anonymous) who confirm they’ve seen lookups for “mail1.trump-email.com” from all over the world, especially from tools like FireEye that process lots of spam email. One person claimed that lookups started failing for them back in late June — and thus the claim of successful responses until September are false. In other words, the “change” after the NYTimes queried Alfa Bank may not be because Cendyn (or Trump) changed anything, but because that was the first they checked and noticed that lookup errors were happening.

Since I wrote this blog post at midnight, so I haven’t confirmed this with anybody yet, but there’s a good chance that the IP address 66.216.133.29 has continued to spew spam for Trump hotels during this entire time. This would, of course would generate lookups (both reverse and forward). It seems like everyone who works for IT for a large company should be able to check their incoming email logs and see if they’ve been getting emails from that address over the last few months. If you work in IT, please check your logs for the last few months and Tweet me at @erratarob with the results, either positive or negative.

And finally, somebody associated with Alfa Bank IT operations confirms that executives like to stay at Trump hotels all the time (like in Vegas and New York), and there was a company function one of Trump’s golf courses. In other words, there’s good reason for the company to get spam from, and need to communicate with, Trump hotels to coordinate events.

And so on and so forth — there’s a lot of information out there if we just start digging.

Conclusion

That this is just normal marketing business from Cendyn and Listrak is the overwhelming logical explanation for all this. People are tempted to pull nefarious explanations out of their imaginations for things they don’t understand. But for those of us with experience in this sort of thing, what we see here is a normal messed up marketing (aka. spam) system that the Trump Organization doesn’t have control over. Knowing who owns and controls these servers, it’s unreasonable to believe that Trump is using them for secret emails. Far from “secret” or “private” servers as Hillary claims, these servers are wide open and obvious.

This post provides a logic explanation, but we can’t count on this being provably debunked until those like Dyn come forward, on the record, and show us lookups that don’t come from Alfa Bank. Or, those who work in big companies can pull records from their incoming email servers, to show that they’ve been receiving spam from that IP address over the last few months. Either of these would conclusively debunk the story.

But experts say…

But the article quotes several experts confirming the story, so how does that jibe with this blog post. The answer is that none of the experts confirmed the story.

Read more carefully. None of the identified experts confirmed the story. Instead, the experts looked at pieces, and confirmed part of the story. Vixie rightly confirmed that the pattern of DNS requests came from humans, and not automated systems. Chris Davis rightly confirmed the server doesn’t look like a normal email server.

Neither of them, however, confirmed that Trump has a secret server for communicating with the Russians. Both of their statements are consistent with what I describe above — that’s it’s a Cendyn operated server for marketing campaigns independent of the Trump Organization.

Those researchers violated their principles

The big story isn’t the conspiracy theory about Trump, but that these malware researchers exploited their privileged access for some purpose other than malware research.

Malware research consists of a lot of informal relationships. Researchers get DNS information from ISPs, from root servers, from services like Google’s 8.8.8.8 public DNS. It’s a huge privacy violation — justified on the principle that it’s for the general good. Sometimes the fact that DNS information is shared is explicit, like with Google’s service. Sometimes people don’t realize how their ISP shares information, or how many of the root DNS servers are monitored.

People should be angrily calling their ISPs and ask them if they share DNS information with untrustworthy researchers. People should be angrily asking ICANN, which is no longer controlled by the US government (sic), whether it’s their policy to share DNS lookup information with those who would attempt to change US elections.

There’s not many sources for this specific DNS information. Alfa Bank’s servers do their own resolution, direction from the root on down. It’s unlikely they were monitoring Alfa Bank’s servers directly, or monitoring Cendyn’s authoritative servers. That means some sort of passive DNS on some link in between, which is unlikley. Conversely, they could be monitoring one of the root domain servers — but this monitoring wouldn’t tell them the difference between a successful or failed lookup, which they claim to have. In short, of all the sources of “DNS malware information” I’ve heard about, none of it would deliver the information these researchers claim to have (well, except the NSA with their transatlantic undersea taps, of course).

Update: this tweet points out original post mentions getting data from “ams-ix23” node, which hints at AMS-IX, Amsterdam InterXchange, where many root server nodes are located.

Continue reading Debunking Trump’s "secret server"→

Posted in Uncategorized

Debunking Trump’s "secret server"

Posted on by

According to this Slate article, Trump has a secret server for communicating with Russia. Even Hillary has piled onto this story.

It’s time for Trump to answer serious questions about his ties to Russia. https://t.co/D8oSmyVAR4 pic.twitter.com/07dRyEmPjX

— Hillary Clinton (@HillaryClinton) October 31, 2016

This is nonsense. The evidence available on the Internet is that Trump neither (directly) controls the domain “trump-email.com“, nor has access to the server. Instead, the domain was setup and controlled by Cendyn, a company that does marketing/promotions for hotels, including many of Trump’s hotels. Cendyn outsources the email portions of its campaigns to a company called Listrak, which actually owns/operates the physical server in a data center in Philidelphia.

In other words,  Trump’s response is (minus the political bits) likely true, supported by the evidence. It’s the conclusion I came to even before seeing the response.

When you view this “secret” server in context, surrounded by the other email servers operated by Listrak on behalf of Cendyn, it becomes more obvious what’s going on. In the same Internet address range of Trump’s servers you see a bunch of similar servers, many named [client]-email.com. In other words, trump-email.com is not intended as a normal email server you and I are familiar with, but as a server used for marketing/promotional campaigns.

It’s Cendyn that registered and who controls the trump-email.com domain, as seen in the WHOIS information. That the Trump Organization is the registrant, but not the admin, demonstrates that Trump doesn’t have direct control over it.

When the domain information was changed last September 23, it was Cendyn who did the change, not the Trump Organization. This link lists a bunch of other hotel-related domains that Cendyn likewise controls, some Trump related, some related to Trump’s hotel competitors, like Hyatt and Sheraton.

Cendyn’s claim they are reusing the server for some other purpose is likely true. If you are an enterprising journalist with $399 in your budget, you can find this out. Use the website http://reversewhois.domaintools.com/ to get a complete list of the 641 other domains controlled by Cendyn, then do an MX query for each one to find out which of them is using mail1.trump-email.com as their email server.

This is why we can’t have nice things on the Internet. Investigative journalism is dead. The Internet is full of clues like this if only somebody puts a few resources into figuring things out. For example, organizations that track spam will have information on exactly which promotions this server has been used for in the recent past. Those who operate public DNS resolvers, like Google’s 8.8.8.8, OpenDNS, or Dyn, may have knowledge which domain was related to mail1.trump-email.com.

Indeed, one journalist did call one of the public resolvers, and found other people queried this domain than the two listed in the Slate story — debunking it. I’ve heard from other DNS malware researchers (names remain anonymous) who confirm they’ve seen lookups for “mail1.trump-email.com” from all over the world, especially from tools like FireEye that process lots of spam email. One person claimed that lookups started failing for them back in late June — and thus the claim of successful responses until September are false. In other words, the “change” after the NYTimes queried Alfa Bank may not be because Cendyn (or Trump) changed anything, but because that was the first they checked and noticed that lookup errors were happening.

Since I wrote this blog post at midnight, so I haven’t confirmed this with anybody yet, but there’s a good chance that the IP address 66.216.133.29 has continued to spew spam for Trump hotels during this entire time. This would, of course would generate lookups (both reverse and forward). It seems like everyone who works for IT for a large company should be able to check their incoming email logs and see if they’ve been getting emails from that address over the last few months. If you work in IT, please check your logs for the last few months and Tweet me at @erratarob with the results, either positive or negative.

And finally, somebody associated with Alfa Bank IT operations confirms that executives like to stay at Trump hotels all the time (like in Vegas and New York), and there was a company function one of Trump’s golf courses. In other words, there’s good reason for the company to get spam from, and need to communicate with, Trump hotels to coordinate events.

And so on and so forth — there’s a lot of information out there if we just start digging.

Conclusion

That this is just normal marketing business from Cendyn and Listrak is the overwhelming logical explanation for all this. People are tempted to pull nefarious explanations out of their imaginations for things they don’t understand. But for those of us with experience in this sort of thing, what we see here is a normal messed up marketing (aka. spam) system that the Trump Organization doesn’t have control over. Knowing who owns and controls these servers, it’s unreasonable to believe that Trump is using them for secret emails. Far from “secret” or “private” servers as Hillary claims, these servers are wide open and obvious.

This post provides a logic explanation, but we can’t count on this being provably debunked until those like Dyn come forward, on the record, and show us lookups that don’t come from Alfa Bank. Or, those who work in big companies can pull records from their incoming email servers, to show that they’ve been receiving spam from that IP address over the last few months. Either of these would conclusively debunk the story.

But experts say…

But the article quotes several experts confirming the story, so how does that jibe with this blog post. The answer is that none of the experts confirmed the story.

Read more carefully. None of the identified experts confirmed the story. Instead, the experts looked at pieces, and confirmed part of the story. Vixie rightly confirmed that the pattern of DNS requests came from humans, and not automated systems. Chris Davis rightly confirmed the server doesn’t look like a normal email server.

Neither of them, however, confirmed that Trump has a secret server for communicating with the Russians. Both of their statements are consistent with what I describe above — that’s it’s a Cendyn operated server for marketing campaigns independent of the Trump Organization.

Those researchers violated their principles

The big story isn’t the conspiracy theory about Trump, but that these malware researchers exploited their privileged access for some purpose other than malware research.

Malware research consists of a lot of informal relationships. Researchers get DNS information from ISPs, from root servers, from services like Google’s 8.8.8.8 public DNS. It’s a huge privacy violation — justified on the principle that it’s for the general good. Sometimes the fact that DNS information is shared is explicit, like with Google’s service. Sometimes people don’t realize how their ISP shares information, or how many of the root DNS servers are monitored.

People should be angrily calling their ISPs and ask them if they share DNS information with untrustworthy researchers. People should be angrily asking ICANN, which is no longer controlled by the US government (sic), whether it’s their policy to share DNS lookup information with those who would attempt to change US elections.

There’s not many sources for this specific DNS information. Alfa Bank’s servers do their own resolution, direction from the root on down. It’s unlikely they were monitoring Alfa Bank’s servers directly, or monitoring Cendyn’s authoritative servers. That means some sort of passive DNS on some link in between, which is unlikley. Conversely, they could be monitoring one of the root domain servers — but this monitoring wouldn’t tell them the difference between a successful or failed lookup, which they claim to have. In short, of all the sources of “DNS malware information” I’ve heard about, none of it would deliver the information these researchers claim to have (well, except the NSA with their transatlantic undersea taps, of course).

Update: this tweet points out original post mentions getting data from “ams-ix23” node, which hints at AMS-IX, Amsterdam InterXchange, where many root server nodes are located.

Continue reading Debunking Trump’s "secret server"→

Posted in Uncategorized

Debunking Trump’s “secret server”

Posted on by

According to this Slate article, Trump has a secret server for communicating with Russia. Even Hillary has piled onto this story.

It’s time for Trump to answer serious questions about his ties to Russia. https://t.co/D8oSmyVAR4 pic.twitter.com/07dRyEmPjX

— Hillary Clinton (@HillaryClinton) October 31, 2016

This is nonsense. Trump’s response is (minus the political bits) likely true, supported by the evidence. It’s the conclusion I came to even before seeing the response.

The server is neither owned or controlled by the Trump organization. The server itself is controlled by Listrak, a company that specializes in email promotions. In the same Internet address range of Trump’s servers you see a bunch of similar servers, many named [client]-email.com, like trump-email.com.

While Listrak provides the raw servers, they are just providing them on behalf of another company, Cendyn. Cendyn’s business is promotional campaigns for hotels, which include mass email campaigns (which they outsource to Listrak), and a bunch of other activities.

It’s Cendyn that registered and who controls the trump-email.com domain, as seen in the WHOIS information:

When the domain information was changed last September 23, it was Cendyn who did the change, not the Trump Organization. This link lists a bunch of other hotel-related domains that Cendyn likewise controls, some Trump related, some of Trump’s competitors.

It’s how the Internet works. Companies outsource marketing campaigns, who then outsource physical infrastructure. It would be shocking if anybody within the Trump Organization had access to any of it.

Cendyn’s claim they are reusing the server for some other purpose is likely true. If you are an enterprising journalist with $399 in your budget, you can find this out. Use the website http://reversewhois.domaintools.com/ to get a complete list of the 641 other domains controlled by Cendyn, then do an MX query for each one to find out which of them is using mail1.trump-email.com as their email server.

This is why we can’t have nice things on the Internet. Investigative journalism is dead. The Internet is full of clues like this if only somebody puts a few resources into figuring things out.

Conclusion

The response from the Trump campaign is overwhelmingly the most logical explanation. Trump hotel busineses outsourced marketing campaigns, who created the domain and setup (through Listrak) the servers. It’s Cendyn who controls the servers, and not the Trump campaign. It’s unbelievable that the Trump campaign would even have access to those servers, much less be using them. Far from being “secret” or “private”, this servers are wide open and obvious.

Continue reading Debunking Trump’s “secret server”→

Posted in SBN

Lamers: the problem with bounties

Posted on by
In my last two posts, I pointed out that the anti-spam technique known as “DKIM” cryptographically verifies emails. This can be used to verify that some of the newsworthy emails are, indeed, correct and haven’t been doctored. I offer a 1 btc (one bitcoin, around ~$600 at current exchange rates) bounty if anybody can challenge this assertion.
Unfortunately, bounties attract lamers who think they deserve the bounty. 
This faked email show _undetectable_ addition of cc: field (& other fields) and whitespace in email body; no tricks #PayUpRob @ErrataRob https://t.co/X8oUplx2UL

— ((( Matt Beebe ))) (@VoteBeebe) October 25, 2016

This guy insists he wins the bounty because he can add spaces to the email, and add fields like “Cc:” that DKIM doesn’t check. Since DKIM ignores extra spaces and only checks important fields, these changes pass. The guy claims it’s “doctored” because technically, he has changed things, even though he hasn’t actually changed any of the important things (From, Date, Subject, and body content).
No. This doesn’t qualify for the bounty. It doesn’t call into question whether the Wikileaks emails say what they appear to say. It’s so obvious that people have already contacted me and passed on it, knowing it wouldn’t win the bounty. If I’d pay out this bounty for this lameness, one of the 10 people who came up with the idea before this lamer would get this bounty, not him. It’d probably go to this guy:
@ErrataRob super lame i know, but this does pass DKIM sig check in thunderbird. base64 here https://t.co/14EyaBKfNL pic.twitter.com/dG94f5lH8o

— Philip (@_miw) October 22, 2016

Let me get ahead of the lamers and point to more sophisticated stuff that also doesn’t count. The following DKIM verified email appears to say that Hillary admitting she eats kittens. This would be newsworthy if true, and a winner of this bounty if indeed it could trick people.
This is in fact also very lame. I mean, it’s damn convincing, but only to lamers. You can see my trick by looking at the email on pastebin (http://pastebin.com/wRsnz0Y6) and comparing it to the original (https://wikileaks.org/podesta-emails/emailid/2986).
The trick is that I’ve added extra From/Subject fields before the DKIM header, so DKIM doesn’t see them. DKIM only sees the fields after. It tricks other validation tools, such as this online validator. However, email readers (Thunderbird, Outlook, Apple Mail) see the first headers, and display something that DKIM hasn’t checked.

I’ve taken a screenshot of the raw email to show both From fields:

Since I don’t rely upon the “magic” of tools to verify DKIM, but look at the whole package, I’ll see the added From/Subject fields. Far from fooling anybody, such modifications will be smoking gun that somebody has attempted illicit modifications. Not just me, mostly anybody viewing the “raw source” that Wikileaks provides would instantly see shenanigans.
The Wikileaks emails can be verified with crypto, using DKIM. Anybody who can doctor an email in such a way that calls this into question, such that they could pass something incriminating through (“I eat kittens”), they win the full bounty. Any good attempts, with something interesting and innovative, wins partial bounty.
Lamers, though, are unwelcome.
BTW, the same is true for bug bounties. Bounties are becoming standard in the infosec industry, whereby companies give people money if they can show ways hackers might hack products. Paying bounties allows companies to fix products before they actually get hacked. Even the U.S. military offers bounties if you can show ways to hack their computers. Lamers are a pox on bug bounty systems — administrators must spend more time telling lamers to go away than they spend on dealing with real issues. No bounties rules are so tight that lamers can’t find a way to subvert the rules without finding anything that matches the intent.

Continue reading Lamers: the problem with bounties→

Posted in SBN