Security is Not a Checklist

In the security profession, we have a maxim that security is not a product. It’s a reminder that security doesn’t result from plugging in devices, but through continuous integration of security into design, development, management, and operations. I’d add another maxim: security is not a checklist.

When I was in QSA training a few years back, our trainer claimed that no one who was PCI DSS compliant had ever suffered a data breach. He hedged this bold statement by suggesting that anyone who had been certified as PCI DSS compliant and later suffered a breach must have fallen out of compliance by the time the breach happened. It was an entertaining exercise in circular logic: PCI DSS prevents security breaches, so obviously anyone who suffered a security breach couldn’t have been PCI DSS compliant.

Well, Heartland Payment Systems, who may have suffered the largest breach in history (giving executives at TJX something to celebrate), was certified as PCI DSS compliant. That suggests at least three possibilities:

  1. Heartland was PCI DSS compliant when they were audited, but fell out of compliance by the time of they were breached;
  2. Heartland wasn’t PCI DSS compliant, but their QSA said they were; or
  3. PCI DSS doesn’t actually prevent compliant organizations from suffering a breach.

Each of the first two conclusions would be reasonable. A PCI DSS assessment is a snapshot in time, and business are constantly changing. And because they are paid by the companies they assess, it’s fair to wonder whether QSAs are truly independent. The third conclusion is more than reasonable, it’s certain: PCI DSS compliance doesn’t guarantee security. That should be obvious. But maybe it’s not.

PCI’s strength and weakness is that it’s a checklist of detailed requirements. Its specificity is an improvement over laws like HIPAA, which calls for protecting against “reasonably anticipated threats” while considering the size of the organization and the costs of the security measures. It’s a flexible approach, but it doesn’t provide many answers. Are firewalls required? Does internal traffic have to be encrypted? It depends.

As a checklist, PCI DSS is more to the point. Companies know exactly what’s expected. They have to have firewalls between untrusted networks and any cardholder data environment (PCI DSS Requirement 1.2), install personal firewall software on laptops (Requirement 1.4), use anti-virus software (Requirement 5.1), and so on. There’s very little “it depends” in the PCI DSS requirements.

But companies sometimes think the checklist is all they need—that once they’ve checked “compliant” next to all the requirements, they’re done (until the next audit rolls around). They fall into the trap of thinking that a checklist item intended to mandate a minimum level of adequate security is also the most they need to do. They forget that being able to answer “yes, we have a process” to a checklist item is not as important as whether that process works. Then, when data is lost, they point to the checklist and ask what more they were supposed to do. That’s when a reasonableness standard starts looking awfully good.

The checklist is necessary, because there’s too much wiggle room and too much ambiguity without one. But just as security is not a product, it is also not a checklist. It is, as always, a process—one that a checklist can inform, and sometimes measure, but never complete.

Published in:  on February 16, 2009 at 5:10 pm Comments (1)

Security Breaches, Identity Fraud, and Unknowns

There’s a minor hubbub at Wired and CyberCrime & Doing Time over the most recent Javelin Identity Theft Survey (consumer version here—the full report will set you back a cool $3,000). The report surveyed 4,784 people to find out if they had experienced identity fraud, and, if so, if they knew how the perpetrator accessed their data. Javelin’s big claim is this:

Despite the hefty blame . . . placed on the Internet and cyber-crime, online identity theft methods (phishing, hacking and malware) only accounted for 11% of fraud cases in 2008. The truth is, most known cases of fraud occur through traditional methods, when a criminal has direct, physical access to the victim’s information.

The report has a chart purporting to show the sources of data used to commit identity fraud. For example, here’s a partial list of categories:

Stolen while making an online purchase: 1%
“Hackers, viruses, or spyware” on a home or work computer: 9%
Phishing: 1%
Stolen from a company in a data breach: 11%
“Primarily business controlled” data stolen while making a purchase: 19%
From a lost wallet, purse, etc.: 43%

Critics level two main charges against the report. First, they note that it was sponsored by a bank and an online identity protection company, creating a potential source of bias. But their main complaint is that the report—or at least the summary chart and Javelin’s 11% number—ignored the cases where identity fraud had an unknown source.

They have a point. Of the roughly ten percent of phone survey respondents who said they had experienced identity fraud (482 out of 4,784 people called), 65% had no idea how their data was obtained. Javelin threw away the unknowns, and calculated its percentages based on the 169 people who said they knew how their information was obtained. That 9% who said their data was accessed by hackers, viruses, or spyware? That’s 15 out of the 169. It’s also 15 out of the 482 who experienced identity fraud.

A more accurate survey result would look like this:

Stolen while making an online purchase: <1%
“Hackers, viruses, or spyware” on a home or work computer: 3%
Phishing: <1%
Stolen from a company in a data breach: 4%
“Primarily business controlled” data stolen while making a purchase: 7%
From a lost wallet, purse, etc.: 15%
Unknown: 65%

The problem boils down to this: did the 313 people who said they didn’t know how their information was obtained have their data stolen in proportionately the same ways as the 169 people who could identify a source? Javelin, by tossing out those 313 unknowns, seems to think so. But there are good reasons why the knowns may not adequately represent the unknowns. For example, not all methods of data theft have the same visibility to the victim—most people know when their wallets have been stolen; the same is not always true of data stolen from a business. And do phishing and social engineering victims usually know they’ve been had?

Unfortunately, the critics take this point and then leap too far, claiming that all of the unknown cases must have come from data breaches, malware, phishing, and other online sources. That’s a reasonable conjecture, but that’s all it is. It replaces a poor assumption—that the sub-sample accurately represents the full sample—with unabashed speculation. Both are interesting, but neither are reliable data.

The most honest approach would be to put that 65% in the margin of error for each category. But “online identity theft methods only accounted for somewhere between 5% and 70% of all identity thefts” doesn’t make nearly as catchy a headline.

Published in:  on February 14, 2009 at 2:04 am Leave a Comment