Collaborate Disseminate
I’ve just been reading Kingpin by Kevin Poulsen which sheds some really interesting light on criminal credit card fraud in the mid 2000’s. Towards the end of the book, there’s a reference to a 1997 case in which the government persuades the sentencing judge to permanently seal the court transcripts
Continue reading Breach concealment is not a security strategy
I certainly didn’t expect it would go this far when I built Have I been pwned (HIBP) a few years ago, but I’ve just loaded the 100th data breach into the system. This brings it to a grand total of 336,724,945 breached accounts that have been loaded in
Continue reading 100 data breaches later, Have I been pwned gets its first self-submission
Update (just a couple of hours later): We’re fixed! By all means, read the background it you’re having similar problems or jump to the bottom for the solution.
I’m writing this post because I’ve pretty much exhausted all other avenues. I’m out of ideas. I’m stumped. I’m also missing a
Continue reading Disqus and the disappearing comments (which still exist)
It’s been 434 blog posts over six and a half years. It’s gone from being excited about a hundred visitors in a week to hundreds of thousands on a big day. It’s taken me from a hobby to a career. In so many ways, this blog has defined who I
Remember when OPM got breached last year? There was a lot of excitement in various parts of the world (namely the US) because here we had a government department (Office of Personnel Management), and they’d just lost 21.5 million records! These records included such sensitive data as names, dates of birth and addresses and by any reasonable measure, it was serious – that’s almost 7% of the country’s population!
Yet somehow, last week’s news that 55 million Filipino voters’ data was now out in the wild went largely unnoticed. Let’s put it down to a very western-centric tech media but move past that and look at this incident for what it is – a ginormous data breach with extremely sensitive information and at 55M individuals, that’s also more than half the country’s population.
Whilst there’s been limited press coverage on the issue, a public statement from the Filipino government has suggested that nothing sensitive was disclosed. As I discovered when I reached out to some of the people involved, this is blatantly wrong. Here’s how it all unfolded.
A couple of weeks back, the COMELEC website (Commission of Elections) was defaced, allegedly by “Anonymous Philippines”:
This is the usual hacktivist ramblings, nothing out of the ordinary for a defacement but inevitably it led to them grabbing quite a bit of data. Just for context, here’s one of the search facilities on the COMELEC website (provided to me by a Filipino who assisted with data verification):
It’s a search facility requesting name and sensitive personal information (namely birthdate) over an unencrypted connection. You could try and load the page securely, except that won’t work very well:
Often when I’m verifying a breach, I’ll look at the site it allegedly came from and try to get a feel for how likely it is to be legitimate. Oversights like this combined with the W3C compliance GIF and hit counter at the bottom of the page just above the “Copyright © 2001” statement start to paint a pretty clear picture….
Firstly, unlike the OPM data, the Filipino breach has been very broadly distributed. Not only has it been readily available for download from multiple locations on the clear web, it’s been quite extensively torrented too. The genie is well and truly out of the bottle and it won’t be going back in.
The data consists of 76GB worth of (usually) compressed files, most notably a MySQL backup that expands out to 338GB. There’s a raft of other .sql files in the breach as well ranging from a few KB up to hundreds of MB. The breadth of data in these is quite significant; Trend Micro did a write-up on what they found so I won’t repeat it all here, I was more interested in verifying the legitimacy of the breach and conclusively reporting on the accuracy.
Amongst the huge volume of data is a total of 228,605 email addresses. This may sound like a small number out of the 55M records, but according to reports, a lot of the sensitive data such as passport numbers belongs to a “mere” 1.3M overseas voters. It’s entirely conceivable that records are not complete across all these individuals, but at least the email addresses gave me a verification avenue.
At the time of writing, I have 367k verified subscribers in Have I been pwned (HIBP), that is they’ve all gone to the notifications page, left their email address then received a confirmation and acknowledged it. I’ve used subscribers in the past to verify breaches where I’m not confident of the authenticity. It’s always worked out very well as I’ve got a large number of people interested in their exposure in data breaches and able to confirm whether information is accurate or not. I give them a small slice of what is allegedly their data and they then confirm the legitimacy.
Part of the reason why I particularly wanted to do that with this breach is because of this statement by COMELEC officials (emphasis mine):
Again, I want to emphasize that the database in our website is accessible to the public. There is no sensitive information there.
Now this could just be misreporting or comments taken out of context, but I’ve seen numerous articles downplaying the severity of the data in a manner that’s just not consistent with what I’ve observed in the data breach. Let’s see what those in there have to say about their data.
Yesterday I emailed a number of HIBP subscribers and got back some pretty quick responses with everyone willing to assist. I found them spread out across two tables in the data breach, the first being a table called “irdoctable2014” which has the following fields:
# FORM_ID, APP_TYPE, REGISTRATION, LASTNAME, FIRSTNAME, MATERNALNAME, SEX, CIVILSTATUS, SPOUSENAME, RESSTREET, RESPRECINCT, RESPRECINCTCODE, RESREGION, RESBARANGAY, RESCITY, RESPROVINCE, MAILSTREET, MAILEMBASSY, MAILCOUNTRY, REGCOUNTRY, REGEMBASSY, REPSTREET, REPBARANGAY, REPCITY, REPPROVINCE, EMAIL, ABROADSTATUS, ABROADSTATUSSPECIF, FLASTNAME, FFIRSTNAME, FMATERNALNAME, MLASTNAME, MFIRSTNAME, MMATERNALNAME, REPLASTNAME, REPFIRSTNAME, REPMATERNALNAME, DOBYEAR, DOBMONTH, DOBDAY, BIRTHCITY, BIRTHPROVINCE, CITIZENSHIP, NATURALIZATIONDATE, CERTIFICATENB, COUNTRYRES, CITYRESYEAR, CITYRESMONTH, PROFESSION, SECTOR, HEIGHT, WEIGHT, MARKS, DISABLED, ASSISTEDBY, TIN, PASSPORTLOST, PASSPORTNB, PASSPORTPLACE, PASSYEAR, PASSMONTH, PASSDAY, REGBARANGAY, REGREGION, REGCITY, REGPROVINCE, REG_DATE, STATIONID, LOCAL_ID, CREATE_TIME, UPDATE_TIME, IS_EXTRACTED, IS_EXPIRED, IS_CANCELLED, CONTACTNUMBER, EXPIRATION_DATE, APPOINTMENT_DATE, APPOINTMENT_TIME, SCHED_TIME, COUNTER_CHANGES, REFERENCENUMBER, ERBDATE, USER_ID, EMAIL_ID, EXTRACTED_DATETIME, IS_DELETE, UPDATED_DATETIME, IS_FRONTPAGE, IS_REPRINT, IS_OV, IS_COUNTED
This is a very large amount of data and reading through those column names, clearly many of them would be considered sensitive personally identifiable data. However, some of the data is encrypted, namely the person’s name and their data of birth. Part of the irony here though is that the email addresses appear in the clear and often contains both the first and last name anyway! Not all the fields are populated but plenty of them are and they contain very personal info. Let me demonstrate by sharing one of the responses I got to my questions in full (responses in bold, data obfuscated by XXX):
Would it be feasible that you have a record in this database? Yes, as I am a registered voter for the coming elections
Is this likely to be legitimate data? Yes
There’s some very specific information about your height (XXX) and weight (XXX) which I assume is metres and kilograms – does that sound right? Yes, that’s what I declared when they took my vital statistics and biometrics when I registered last year
Do you recognise the names “’XXX’, ‘XXX’, ‘XXX’, ‘XXX’, ‘XXX’, ‘XXX’” – I suspect they may refer to your mother and father, I’m just trying to confirm? Yes, those are my parents
Along with email address (which in this case included the person’s full name), is their “vital statistics and biometrics” as well as their parents’ names which all appear in the clear. There’s also a physical address, gender, marital status, where they were born, where they’re now living, their profession and their phone number. This is very personal information!
Another subscriber provided this confirmation:
XXX and XXX are my parents’ middle names. In the Philippine setting, your middle name is usually your mother’s surname (prior to her marriage).
I then had a further two other people provide the same emphatic confirmation about their data in the same table. A fifth person who offered support was found in a table called “doctablepost” which was in the 338GB file and contained these fields:
# ID, APPLICATION_ID, FORM_ID, APP_TYPE, ABSENTEE, REGISTRATION, LASTNAME, FIRSTNAME, MATERNALNAME, SEX, CIVILSTATUS, SPOUSENAME, RESSTREET, RESPRECINCT, RESPRECINCTCODE, RESBARANGAY, RESCITY, RESPROVINCE, ABROADSTREET, ABROADZIP, ABSENTIA, ABROADCITY, ABROADCOUNTRY, ABROADPERIOD, ABROADRESCONT, REGCOUNTRY, REGEMBASSY, MAILSTREET, MAILZIP, MAILCITY, MAILCOUNTRY, MAILEMBASSY, REPSTREET, REPBARANGAY, REPCITY, REPPROVINCE, EMAIL, ABROADSTATUS, ABROADSTATUSSPECIF, LASTENTRYDATE, ABSREGISTERED, OLDPRECINCT, OLDREGBARANGAY, OLDREGCITY, OLDREGPROVINCE, OLDREGDATE, FLASTNAME, FFIRSTNAME, FMATERNALNAME, MLASTNAME, MFIRSTNAME, MMATERNALNAME, REPLASTNAME, REPFIRSTNAME, REPMATERNALNAME, DOBYEAR, DOBMONTH, DOBDAY, BIRTHCITY, BIRTHPROVINCE, CITIZENSHIP, NATURALIZATIONDATE, CERTIFICATENB, COUNTRYRES, CITYRESYEAR, CITYRESMONTH, PROFESSION, SECTOR, HEIGHT, WEIGHT, MARKS, DISABLED, ASSISTEDBY, OLD_VIN, VINP1, VINP2, VINP3, VINCONTROLCODE, TIN, PASSPORTLOST, PASSPORTNB, PASSPORTPLACE, PASSYEAR, PASSMONTH, PASSDAY, REGBARANGAY, REGCITY, REGPROVINCE, REG_DATE, INTERNAME, OFFICERNAME, OPERNAME, STATIONID, CDID, SETID, PRINT_FLAG, FINGER_INFO, FINGER_TOPO_COORD, QUALITY, MATCHING_FINGER, TRANSFER_STATUS, TRANSFER_UPDATE_TIME, PAGES_DESCR, LOCAL_ID, CREATE_TIME, UPDATE_TIME, LOCK_USER, LOCK_TIME, PROCESSING, IS_CURRENT, DOC_VERSION, CD_STAT_ENTY, DISAPPROVED, VOTING_HIST1, VOTING_HIST2, OP_CODE, OP_DATE
This is where we now start to get into passports too and indeed this individual’s was in there, both the number and the place and date of issue. His name was also encrypted but the passport data wasn’t, nor was his birthday as it was for others in the previous table. I asked him for confirmation of his data:
Would it be feasible that you have a record in this database? Yes
Is this likely to be legitimate data? Yes
The db record appears to be my overseas absentee voter record which I have registered into back in 2012. That also contains my current address, mothers maiden name and the Comelec officer who processed my registration.
Does your passport number end in “XXX” and was it issued in the month of XXX? Yes. Starting with XXX. It was issued in XXX and will expire XXX XXX
Were you born in [month] XXX? Yes, I am born in XXX
That’s a very emphatic set of responses and he was able to not only confirm passport fragments that I gave him, but provide me with other fragments that lined up with the data in the breach. With five independent confirmations of the data, there’s no doubt in my mind that this is the real deal.
As serious as the info above is, it’s only scratching the surface. Per the reports linked to earlier, there’s also biometric data relating to fingerprints in the system. This contains columns names such as these:
PRINT_FLAG, FINGER_INFO, FINGER_TOPO_COORD, QUALITY, MATCHING_FINGER
The values within there can be quite detailed and I’ve no reason to think that this isn’t indeed legitimate print data uniquely and biologically identifying the owner. You don’t get to reset that stuff once it’s been released into the wild!
There’s what appears to be a CMS back end for the website. There’s voting history against names (it appears to just be dates rather than the candidate voted for). There’s information about embassies and polling locations and data on what appears to be electoral candidates as well. Clearly some of this should be public, but here you have a whole heap of very sensitive, poorly protected data somehow grouped in with public domain info. Given that most of this all sits in the one database as well, it’s highly liking it was all running behind the public website which may explain how such a broad set of data was obtained.
It’s just an absolute mess of huge volumes of data, tables with suffixes which appear to indicate copies or duplication, draft or temporary data and inconsistent (and frequently insufficient) cryptographic storage of sensitive data. This feels like so many other large, legacy corporate databases I’ve seen which have had numerous developers applying various practices to it over a long period of time. Only difference is, it’s got a heap of highly sensitive information in it and it’s now all public.
The 228,605 email addresses in the breach are now searchable in HIBP. I actually had to create five new data classes when loading this breach, that is I’d never seen this information in a breach before: Marital statuses, Biometric data, Physical attributes, Family members’ names
The Philippines is not exactly high on the news radar for Western media but this is a breach we should be paying attention to. There’s the potential to do serious damage to those involved and we need to remember that the same classes of data are held by all our governments in our respective corners of the world. It’s too late for those in this breach – a huge amount of personal data is now perpetually out there in the public domain – but let’s see if we can avoid those same mistakes in other parts of the world.
Continue reading When a nation is hacked: Understanding the ginormous Philippines data breach
Remember when OPM got breached last year? There was a lot of excitement in various parts of the world (namely the US) because here we had a government department (Office of Personnel Management), and they’d just lost 21.5 million records! These records included such sensitive data as
Continue reading When a nation is hacked: Understanding the ginormous Philippines data breach
A little while back, I wrote about how Lenovo were sending me some things as part of their Insiders program which meant getting to use a number of machines I probably wouldn’t have thought twice about otherwise. The Yoga 900 in that blog post, for example, is not something I would have normally considered for myself as I like a physically larger, gruntier machine yet it’s turned out to be one of the best laptops I’ve ever owned. You’d normally find me with a machine like a W540 which is the last machine I bought myself about 18 months back and it’s been sensational… until this arrived:
There we go – my new new workhorse – a P50! pic.twitter.com/oCzuQELpYE
— Troy Hunt (@troyhunt) February 26, 2016
In total honesty, had it been my own dollars I wouldn’t have gone out and bought the P50 because the W540 is still such a good machine. I want to be transparent about that because the last thing I want is a perception of my views being skewed by virtue of getting free stuff from Lenovo.
In terms of why they feel compelled to send me things, well, it’s because of stuff like this:
I just got a Lenovo P50 because @troyhunt made it sound sooooo cool
— MomMeetCoffee (@mommeetcoffee) March 18, 2016
Obviously influence is important to them particularly when it’s viewed as being objective, independent thoughts. So with that in mind, let me share how I’ve been finding the machine and I’m going to give you a bunch of stats followed by what I didn’t initially like and then why I decided I did like it!
Let’s be clear about who this class of machine is for. Actually, I’ll turn it around – here’s who it’s not for:
@mommeetcoffee @troyhunt looks nice and…heavy. Do you guys dead lift with it? 😛
— Steven Luu (@sluu99) April 6, 2016
The machine weighs in at somewhere between 2.5 and 3kg depending on how you configure it then you can add another 0.7kg for the seriously chunky power brick. My Yoga 900, by comparison, is only 1.3kg and the power supply probably just a couple of hundred grams. If the weight worries you, this is not the machine for you. For me personally, I do a lot of travel with a large machine and the Yoga all in my backpack and it doesn’t bother me, but maybe that’s just me.
The P50 isn’t intended to be a lightweight machine. It’s got a 15.6” screen so to begin with it’s a bigger chassis than many of today’s lightweight machines, then there’s the fact that it has a Xeon processor. Yep, Xeon. This machine is the world’s first laptop to use the Skylake microarchitecture and I can only imagine that it means a fair bit of cooling is also required hence the sizable fan ports on the thing. Ports is the other thing – 4 USB 3.0, a USB Type C, Ethernet, HDMI, mini display port, express card, smart card reader – it all adds up to more weight.
Look, I really like carrying the Yoga to a coffee shop or using it on the plane, but I really like working with a larger keyboard with more travel on the keys and using a large screen. But more than all that, I like power. Lots of power. Let’s see how much.
Let’s just get straight to the numbers; here’s my PassMark PerformanceTest results for the old W540 and the new P50:
Other than the very slight dip on 2D graphics, the P50 is quicker across the board but it’s the disk that really smashes it. When Lenovo shipped the machine to me, it came with a 256GB NVMe SSD which isn’t quite big enough for a primary drive these days. I went and bought a 512GB Samsung 950 Pro NVMe which appears to be widely accepted as about the faster drive you can get and oh boy – look at the rating on that sucker! That’s compared to the Crucial SSD in the old W540, but I saw similar improvements comparing it against other machines I’d had with older SSDs. (Incidentally, you’re reading this whilst struggling to get a Lenovo P50 or P70 booting from this drive, read this.)
In terms of the other specs, you can read the whole machine configuration in this system overview from PerformanceTest. Lenovo sent it to me with 16GB of DDR4 ECC RAM which I probably would have specced at 32 had I the choice (I’ve already run short whilst running a VM). The machine will actually go up to 64 which is pretty nuts for a laptop but if you’re doing anything memory intensive then that would be neat. I’ll likely just order some more shortly.
What’s also interesting is Greg Pakes recently picked up a Dell XPS 9550 and asked to compare baselines. We did a swap and here’s the end result:
The Lenovo was faster in every area, even with the disk which is an identical drive in each so I can only assume it’s related to bus architecture on the machine or something else specific to the P50’s design. That said, do be conscious that all of these benchmarks can fluctuate between runs, particularly depending on what’s running on the machine at the time and which drivers are installed.
For the curious, if you’d like a copy of my benchmark you can grab it here. Note that the “fully configured” suffix indicates that this was after I configured everything optimally. Also note that it was before configuring Bitlocker so I expect there’d be a bit of a perf hit there. In short, this is the best result I could get out of the machine.
I’m notorious for wanting the latest shiny thing. There’s something in my head that just automatically associates “newer” with “better” and that carries with it this innate desire to upgrade and reap the benefits, whether they be perceived or real.
One such example is higher DPI devices and my love of lots more pixels in ever smaller spaces started with the iPhone 4. It was so… dense! So of course I later on had to have the iPad with lots of pixels and they are indeed beautiful devices. More recently, that’s extended to PCs and the Lenovo W540 I purchased about 18 months ago. Lovely machine, heaps of grunt and it’ll happily do 2,880×1620. More recently, I received a Yoga 900 at 3200×1800 and that’s in a smaller screen that the W540 too so it’s dense pixels FTW.
When the P50 arrived, I was initially disappointed when I saw it only did 1080p. That’s less than half the pixels than the old 540 therefore it is worse, right? No, and here’s why:
I spend a huge amount of time working in Camtasia where I edit my Pluralsight courses. Here’s what it looks like on the new P50:
And now on the W540 with the higher DPI:
The canvas no longer stretches to the edges when told to “Shrink to fit”, the buttons all become smaller than they should and various other dialogues have clearly odd spacing, particularly for someone like me who gets pedantic about these things. For an app I spend huge amounts of time in, this is enormously frustrating.
Let’s try LiveWriter where I write my blogs. Here’s the P50 at 1080p:
And now here’s the W540 with the higher DPI:
There’s no concept of scaling the pixels on the higher DPI machine to align with the size the browser will render it at once it’s published. If you’ve ever seen me post a blog with oddly sized images it’s usually because I wrote it on the W540 and what I was seeing at design time is not what I ended up getting at publish time.
There are many, many other apps that struggle too. SQL Server Management Studio has various quirks, as do Red Gate’s tools that I run within them. It gets worse as the DPI goes up too – it’s much more obvious when I use the Yoga 900 with 3,200 horizontal pixels versus the W540 with 2,880 compared with the P50 at “only” 1,920.
Let us be clear – this is the software’s fault not the machine’s – but it’s software I’m heavily dependent on. It’s not just the apps I install on the machine either, Windows 10 itself seems to struggle with the DPI namely when changing resolutions. Often I’ll see some assets appear to be targeting one pixel density and others targeting another; desktop icons suddenly become too small or the taskbar too big or any number of things that remind you how the high DPI model is still a second class citizen.
It pains me to say it because I want that pixel density so that I have super sharp visuals, but if I’m honest, I haven’t missed it at all with the P50. Like I said, a 1080p screen would not have been my first choice but having these apps that haven’t got with the times yet behave as they should and not having dramas with Windows flicking between pixel densities has been rather nice! Hopefully by the next time I come to need a new machine things will be more stable but for now, less is actually more.
So keeping in mind that this is a combination of how I find it compared to the W540 and general observations about the machine itself, there’s a few things that stick out.
Firstly, it has proper buttons next to the trackpad:
The W540 had one of those pushable trackpads which initially drove me nuts then I got kinda used to. Now I’m readjusting again but I think this is a better design, if for no other reason than it’s more tactile and there’s a clearer delineation of where each button is. I normally use a wireless external mouse anyway though so it’s infrequent I touch the pad.
Fingerprint reader – get one. The last machine had one too although the P50’s implementation just involves placing the finger on it versus swiping it like on the old machine. The best bit about a fingerprint reader is not the ease of logging back in (although that’s nice), it’s being able to do it in front of other people without that awkward thing where they try to look away and you try to hide what you’re doing. I love Touch ID on my iThings for the same reason.
The fan – why does it keep coming on?! I don’t know why, but even on normal power setting and sitting there doing nothing it wants to spin it up every now and then. The old machine didn’t do this, perhaps it’s something I need to probe into a bit further.
Malware – no, I haven’t found any! I say this simply as I pretty much have a new metric when I tweet anything Lenovo related these days which is TTFS, otherwise known as Time To First Superfish. Lenovo screwed up badly on this front last year and they know it. I usually repave the machine from scratch anyway so anything on the disk is clean (plus it’s a new disk too), so short of funky firmware or BIOS software (and yes, I’m aware of the history there too), it’s all good.
Keyboard – I mentioned it earlier but it has a much more tactile feel than something like the Yoga. I know Apple rant on about how much effort goes into the way the keys on the Macbook feel when they’re pressed but they’re actually right – a good keyboard feels nice to use. I like the concave on the keys, the smooth but not glossy surface and the spacing between each one. It’s just a good place to spend lots of time and that’s exactly what a workhorse laptop should feel like IMHO.
This is really just a casual “here’s how I find it” review, if you want to get really into depth, check out this review on the ThinkScopes blog.
I love the saying that computers are cheap and humans are expensive. Comparatively little costs in hardware can translate to big savings in productivity and as the value of your time increases, the statement becomes truer. It’s a hard thing to quantify though; if you spend $1k more on a top-spec machine, what’s the ROI? I’ve got no idea but I do know that very small gains will easily justify a dollar a day over the (likely) life of the machine and I also know that in terms of buying productivity, $1 is nothing.
It always infuriated me back in my corporate life at Pfizer how cost was viewed when it came to people and productivity. The thing that really stuck in my mind is that during an organisation-wide refresh of machines there was a push to keep the primary HDD at only 128GB which made things very tight on space. It was something like $100 to increase the capacity to a level where it would be plenty for the vast majority of people but the argument was always “Do you know how much it would cost to do that across X number of people?!” because that was the big scary figure. My retorts of “Do you know how much we’re paying those X number of people?!” generally didn’t go down too well even though it meant valuable resources wasting time purging otherwise useful files because hey, that upfront cost looks scary! Use your common sense on these things and invest in equipment that will help you do your job and above all, that you actually enjoy using.
But I digress, the point is that whenever it’s me paying the money, I always err on the side of going fast and being productive and this machine is pretty much the epitome of that.
The capability of the machine is kinda the point in terms of wearing the additional weight; this machine will do pretty much everything my usual desktop machine will do. I travel a lot and I want to be just as capable on the other side of the world or at 30,000 feet as I am when I’m sitting at home (lack of multiple monitors aside). I travel with both this machine and my Yoga in part because I always have a backup machine when I’m speaking and in part because when I’m running workshops I tend to present from one machine then use another to prepare things whilst the class is doing their exercises. For me, capability is more important than weight and if your priorities are inversed then this is not the laptop you’re looking for.
But if going fast in a fully-featured machine is your priority then this is absolutely the machine I would buy with my own cash, albeit with 32GB of RAM… and a higher DPI screen which I would then curse at!
Continue reading Lenovo P50 and my dislike of high DPI displays (for now)
A little while back, I wrote about how Lenovo were sending me some things as part of their Insiders program which meant getting to use a number of machines I probably wouldn’t have thought twice about otherwise. The Yoga 900 in that blog post, for example, is not
Continue reading Lenovo P50 and my dislike of high DPI displays (for now)
It’s now going on two and a half years since I launched Have I been pwned (HIBP) and I’m continually amazed by how much has happened in that time. It started out with a “mere” 152M breached records and has now more than doubled in volume, I’ve added an API, notifications, domain searches, pastes and a heap of other things both visible to the public and behind the scenes. It’s also gone from a hobby project which I thought only a few curious technology people would visit to a site that’s seen over a million visitors in a single day in the wake of the Ashley Madison breach and has been covered by some of the world’s largest media outlets.
The breaches themselves have gotten me involved with a side of security I previously had limited insight into, a side that many of us probably don’t think about that much. For example, the outpouring of deeply sensitive information from Ashley Madison members. Another side of data breaches I didn’t fully appreciate earlier on is the prevalence with which they’re traded online. In fact to my delight, HIBP appears to have proven detrimental to this trade which is an entirely unexpected yet pleasing outcome. As you can imagine, running this service has also exposed me to some very interesting characters; sometimes highly intelligent, other times (mostly) innocent juveniles and other times, a downright nasty side of society.
Dealing with data of this nature means that privacy is always going to be something I need to be especially mindful of. From the outset, I made the call to never store anything other than email addresses in the system so no other data breach attributes such as passwords or even names touch HIBP. Last year I introduced “sensitive” breaches as well, that is I made certain incidents such as Ashley Madison, Adult Friend Finder and YouPorn unavailable for public search due to the potential for it to have a serious adverse impact on those exposed. Yes, they’re all readily obtainable data breaches circulating around in public anyway, but I don’t want to be the one that facilitates a simple online search that then causes someone serious harm as a result of them being found in one of these sites.
The ethics around how I handle data on HIBP is something I put a huge amount of thought into. I hope that by virtue of the transparency I’ve always had with how I run this site, people see it as a resource for good and that’s almost unanimously been the feedback from people – 99.9% positive, I’d say. To that effect, I want to write about two more things I’ve changed on the site just today in order to continue focusing on privacy in an ethical way: the ability to opt out of being publicly discoverable in any data breaches in HIBP and the removal of the VTech data. Let me fully explain the rationale behind both of these changes.
I’m not sure how many people have visited HIBP since I started it back in 2013. Google Analytics reckons about 15M page views and there’s a heap more that wouldn’t have been counted thanks to ad blockers and VPNs that keep the trackers away. All I know is that it’s 8 figures which at least in my books, is a hell of a lot.
To date, I’ve had 5 people ask me to remove their data. Five. That is all. A few of them have come across tersely to begin with but I’ve always been conciliatory and immediately obliged after which they usually turn out to be pretty cool. Often they’re unaware that sites are disclosing their presence on them anyway via enumeration risks; try entering an email address into Adult Friend Finder’s password reset page and you’ll see what I mean.
But regardless of the ease with which you can discover someone’s presence on almost every one of the sites already loaded into HIBP, I understand the sentiment. Last year I had someone create a UserVoice suggestion requesting I provide a way for people to remove themselves and that’s now precisely what I’ve done. Here’s how it works: There’s now an “Opt out” link under the “About” menu item:
That brings you to the opt out page:
The reCAPTCHA is necessary to ensure there’s no abuse that causes emails to be sent to unsuspecting people en mass. Sending an email is necessary because that’s the verification channel to ensure someone does indeed want to opt out:
The email and what happens next is pretty self-explanatory, let me explain how the mechanism works on the back end.
Firstly, in order to know who wants to opt out, I need to store the person’s email address. I could always just nuke their entire record now, but that doesn’t keep them opted out in the future if a new breach appears that include them. I also wanted to give people the flexibility to still use the service if they wanted to; they should be able to privately search for their exposure and the notification system should still work. To make all of this possible and still ensure both data imports and searches remain efficient, I now flag someone’s record in both the breaches table and the pastes table when they want to opt out. These are Azure Table Storage entities which are super fast to retrieve by key but slow to run ad hoc queries on. What this model means is that when a search is done for someone and their record is flagged as “opt out”, the result is simply 404 (not found), which is identical to if they were never in there in the first place.
That last point is important too: there’s no public indication that someone has opted out. If they’re taking the view that they don’t want to be publicly located then I think it’s fair to assume that applies not disclosing that fact as well. That means that both the search facility and the opt out process itself will not publicly disclose their opted out state – there are no known enumeration risks either explicitly or via timing attacks or anything like that. Having said that, if someone was to trawl through the Adobe breach (which is still easily located on the web) and find that an address in there is not in HIBP then clearly that would imply the opt out. There’s not really anything I can do about that, but I thought it was worth a mention.
To the earlier point about still being able to search for your own exposure, at any time anyone can use the notification service to check breaches and pastes they’ve been involved in. It will send them an email with a unique link which will take them to a page with the search results regardless of whether they’ve opted out or not and regardless of whether the breach was sensitive or not (i.e. Ashley Madison).
So that’s the opt out facility, let’s get onto VTech.
In November last year, Motherboard reporter Lorenzo Franceschi Bicchierai contacted me about a serious data breach. He’d been handed a large number of records directly by an individual claiming to have broken into VTech’s systems and thus began a saga that resulted in the details of millions of adults, their children and even their kids’ photos being exfiltrated by the individual. I helped Lorenzo verify that the breach was indeed legitimate and he communicated backwards and forwards with VTech and the attacker. I wrote about it at the time in When children are breached – inside the massive VTech hack.
The VTech data breach was unique for a number of reasons. Firstly, there were only ever three individuals who held the data, those being the guy who obtained it in the first place, Lorenzo and myself. To this day, it remains the only breach I’ve ever loaded into HIBP that other parties didn’t already have their hands on. Secondly, it’s kids and there’s really not much more to say about that; when children are involved, the incident takes on a whole new level of emotion for those caught up in the incident (put yourself in the parents’ shoes for a moment).
An incident of this nature has broad-reaching and long-lasting ramifications. One of those is that the alleged perpetrator was arrested in December. I know nothing more of his fate since that time nor do I know his identity and it may well remain permanently suppressed. Whilst he clearly stepped way over the line of ethical disclosure (he originally claimed he simply wanted to make VTech aware of risks in their assets), a saving grace is that to the best of my knowledge, he never redistributed the data beyond Lorenzo. In fact I’ve frequently had people request it from me but as I’ve said in the past, I don’t give anybody any of this data under any circumstances.
I’ve had chats with many parties about the VTech incident and one thing I’ll make crystal clear up front is that they’ve all been casual, productive and friendly chats. It’s included lawyers involved in class actions on behalf of those in the breach, three letter law enforcement acronyms in multiple countries and legal representatives for VTech themselves. Obviously the nature of those discussions has differed but there’s been one constant: nobody wants to see the data spread in the way that so many other data breaches have in the past. Because of this, I’ve agreed to ensure that can never happen due to anything on my end going wrong and I’ve now permanently deleted both the original data breach Lorenzo sent to me and the VTech data in HIBP.
For the sake of full transparency, this was not as a result of a demand or a threat or anything of that nature, rather it’s to give families impacted by the incident that little bit of extra certainty that they can put this incident behind them. This was a concern that was raised by all of the parties I mentioned earlier and regardless of how capable I may think I am of protecting data, it gives these families further peace of mind because I simply cannot lose what I do not have. I’ve been assured that every single person impacted by that incident has been contacted directly by VTech so there should be no interested parties left who aren’t aware of their exposure. These factors together – me being one of the only parties with the data, the concern families have over their kids and everyone in there having been notified already – are what’s driven me to remove that data because it’s the right thing to do.
In terms of the mechanics of how this works in HIBP, I actually had to build a new process to remove the data. As I’ve written before, it’s in Azure Table Storage rather than a relational database so it’s not a simple matter of DELETE FROM, rather I had to enumerate over every single record and remove entries one by one, all 4.8 million of them. You’ll still see VTech listed as a site that was breached, but I’ve now introduced the concept of a “retired breach” and you’ll see a corresponding logo next to it:
I elected to continue showing it here and not erase all memory of it in part because it has historical significance as a major industry data breach and also due to the referential integrity dependency of HIBP subscribers. I notified 1.7k of them who were found in the VTech breach and I have a historical record of having sent them an email. Whilst removing the 4.8M addresses themselves from the searchable index in HIBP was the right thing to do, it’s a different story for that small number who willingly provided their info to HIBP and asked the service to watch out for them.
This is a natural evolution for HIBP that has negligible real world impact. Continuing to run this service responsibly is absolutely paramount and measures such as the opt out feature and removing the VTech data are consistent with the ethical approach I believe this class of information deserves.
The only reason this service has been able to continue running unhindered is because I’ve always erred on the side of caution when it comes to how I handle data. A month before the Ashley Madison data even hit the torrents, I’d concluded that it shouldn’t be publicly searchable which I’m confident is the reason HIBP never received a DMCA takedown or even a hint of a legal threat. Regardless of any concerns about that, it was the right thing to do as is removing peoples’ data from public searching and retiring the VTech breach.
The things I’ve outlined in this blog post might be viewed as taking the moral high-ground and that may well be right – I hope it’s right – but more than anything, it’s to try and help this project achieve what I ultimately created it for and that’s to help people better protect themselves online.
Continue reading Have I been pwned, opting out, VTech and general privacy things
It’s now going on two and a half years since I launched Have I been pwned (HIBP) and I’m continually amazed by how much has happened in that time. It started out with a “mere” 152M breached records and has now more than doubled
Continue reading Have I been pwned, opting out, VTech and general privacy things
