The big cybersecurity challenge: Time-to-detection

July 29, 2015 Leave a comment

Do you sunbathe? You shouldn’t in this day of hypersensitivity about skin cancer. But if you do, the sunlight falling on your liver-spotted, lizard-like skin has been traveling through space for about nine minutes. When you gaze at the night sky and see Alpha Centauri, you probably remember from grade school that light from that nearby planet takes about 4.3 years to get to earth.

If something like a Burning Man festival were held on Alpha Centauri, you wouldn’t know about it until 4.3 years after it was over. Too late to load up your Airstream and get there in time for the fun. Most stars are so far away, they probably collapsed into black holes a billion years ago, yet all we see is merry twinkling millennium after millennium.

Not to over-dramatize, but this is how things are in cybersecurity — specifically intrusion detection. When the Office of Personnel Management was patching its systems, it discovered its great breach, months after the break had occurred. It might have been still more months before anyone noticed the anomaly. It reminds me of a corny roadside display in Pennsylvania when i was a kid. A sign on a little barn said, “World’s Biggest Steer Inside.” When you pulled over and peered in the window, you saw a big jagged hole in the back of the barn, a chain lying in the dirt, and another sign, “Too bad, guess he got away!” There must’ve been a gift shop or goat’s milk fudge stand nearby.

This is one of the big problems with modern-day cyber attacks. Too often, IT and security staffs only find out about them long after the damage has been done and the hackers moved on to other soft targets. If it takes seconds or minutes to exfiltrate data, what good does discovering it do next year?

I recently spoke with John Stewart, one of the top security guys at Cisco. The topic was Cisco’s Midyear Security Report. Here’s my summary: Federal IT and security people, like everyone else, have plenty to worry about. Like the fact that a thousand new security product vendors have started up in the last five years, yet most of them sell non-interoperable software. Or that the white-hat, good-guys side of the cybersecurity equation is literally about a million qualified people short.

Yet among the most seemingly intractable problems lies time-to-detection, or how long on average it takes for organizations to find out they’ve been hacked. This makes it likely that many more successful attacks have occurred than systems administrators are aware of. Stewart says most of the data show that IT staffs routinely take months to detect breaches. A major goal of the products industry and practitioners’ skill sets must therefore be getting time-to-detection down to seconds. At this point, I’ll bet many federal agencies would be happy with days or hours.

Malicious hackers aren’t standing still, the Cisco report points out. They’re switch vectors and modalities at lightning speed. They’re using wealth transfer techniques that stretch law enforcement’s ability to detect. Stewart says, systems like Bitcoin and the murky avenues of the dark web don’t include or even require the typical middlemen of the surface financial transaction world — such as banks, transfer networks, mules. He describes the bad-hacker industry using a term the government likes to use for itself: innovative. 

Embedded IP domains and fungible URLs, jacking up data write-rewrite cycles to dizzying speeds, or quietly turning trusted systems into automated spies in the time it takes someone to go for coffee — that kind of thing. You might call it agility. They’re dancing circles around systems owners. The hacking community has become wickedly innovative at evading detection, Stewart says, exploiting the common systems and software everyone uses routinely.

He adds that the motivations of bad hackers have blossomed into a veritable bouquet. They go after systems for espionage, theft of money or intellectual property, terrorism, political activism, service disruption and even outright destruction. That’s a good case for the so-called risk-based approach to cybersecurity planning. If you’re a utility, disruption or destruction is more likely to be the hackers’ goal. If you’re a database of people with clearance, espionage and theft are good bets.

Answers? As cybersecurity people like to say, there is no silver bullet. Stewart says nations will have to cooperate more, tools will have to improve, people will have to get smarter. Cisco hopes to build some sort of architecture framework into which the polyglot of cyber tools can plug, reducing what he calls the friction of integration.

For now, a good strategy for everyone connected to cybersecurity is to bore in on the essential question: How soon can we know what’s going on?

Thoughts on bloated web sites, complex software

July 21, 2015 Leave a comment

With my wife at the wheel, we swing off Route 21 in New Jersey onto E 46. The GPS in the dash of our new Subaru is guiding us to Saratoga Springs, NY for the weekend. Kitty-corner from the exit is a big bilboard that reads, “WHO IS JESUS? CALL 855-FOR-TRUTH. Nice and succinct. I admired the certitude, but didn’t try the number.

The car is filled with slightly more mystifying tech. Somewhere I read the average modern car has 200 microprocessors. How many lines of code do they run, I wonder? No matter, the car does what it’s supposed to. Anyone who ever dealt with distributor caps, points and engine timing lights appreciates the way today’s cars work.

The GPS-bluetooth-navigation complex in the dash is another matter. It’s a mishmash of hard-to-follow menus. No matter what we do, every time we turn on the car, the podcasts on my wife’s phone starts up. As for navigation, no two systems I’ve ever seen work quite the same way, at least their user interfaces don’t. Voice commands can be ambiguous, and if occasionally directs you off the highway only to direct you right back on again.

This same overload is ruining many web sites, as it has many once-simple applications. No wonder people love apps, in the sense of applications designed or adapted to work easily and quickly on the small touch screens of mobile devices. Standards like Word, Outlook, iTunes and many other have become so choked with features and choices, I’ve practically given up on them. I can figure out what they do, but it’s all too much, too fussy and time-consuming to manage.

The major media sites are so choked with links — most of them for ads, sponsor content, and unrelated junk such as 24 celebrity face-lifts gone horribly wrong — that you can barely navigate them with out constant, unwanted and frustrating detours.

The drive to make software more and more functional may be behind what seems to be a disturbing trend towards failures in critical systems. They’ve happened a lot lately. In fact, it happened first rather close to home. Literally a minute before going on the air one recent morning, the system that delivers scripts and audio segments failed. A Federal News Radio, we’d gone paperless for a year, reading scripts online and saving a package of printing paper every day. Talking, trying to sound calm, ad-libbing while gesticulating wildly to my producer — that’s what a software crash causes. Controlled panic. Panic, anyhow. It took the engineers an hour to fix. It turned out, a buffer overflow crashed the Active Directory on which the broadcast content environment depends for user privileges. So down it went with the ship.

It was the same day United Airlines passenger boarding system failed, apparently the result of lingering incompatibility from the merger with Continental. And the same day that the New York Stock Exchange famously experienced an hours-long crash, reportedly because of network connectivity issue. Earlier in the month, a hardware-software interaction interrupted for two weeks the State Department’s globally-distributed system for issuing visas.

Successive program managers for the F-35 fighters have all complained they can’t get the software development for this fussy and delicate airplane in any sort of predictable schedule. Yet the plane is unflyable and unmaintainable without its software.

In short, two problems linger with software controlled systems. They can be difficult to interact with. And in their complexity they produce effects even expert operators can’t foresee. I believe this is the basis for the spreading appeal of agile development. It forces people to develop in pieces small enough that people can keep track of what is going on. And in ways that the users can assimilate easily.

Complexity, or the desire to avoid it, is why people like apps on mobile devices. I confess to checking Buzzfeed on my phone when I’m bored. The content is inane, but it’s such a fast, simple app, like eating gumdrops. I recently checked out the regular Web site of Buzzfeed, and sure enough, it’s a confusing kaleidoscope. Although, an ice cream cone swaddled in Cocoa Krispies does sound good.

Archuleta departs. Now what? Some ideas

July 10, 2015 1 comment

OPM director Katherine Archuleta, as I predicted three weeks ago, has resigned. Problem solved, let’s move on.

Fat chance.

In reality, Archuleta’s departure solves nothing fundamental. But she had to go, as I’m sure she understood probably from the moment she peered over the edge and realized — long before most everyone else — the size of the abyss caused by The Data Breach. Talk about big data. As primarily a politician, Archuleta must have realized that she would eventually take the fall for the administration, which of course is ultimately responsible. That’s the way of Washington; always has been. Katherine Archuleta isn’t a horrible person, nor do we have any reason to think she didn’t have the best interests of federal employees at heart. But as President Obama’s campaign manager who secured a visible, plum job, she would get it: This goes with the territory.

And the more the White House spokesman, a sort of latter day Ron Ziegler, pushed culpability away from the administration in the aftermath of Thursday afternoon’s revelation, the more it’s clear the White House itself knows it is somehow responsible for potentially messing up 22 million lives, compromising national security, and making the government look totally incompetent.

“There are significant challenges that are faced not just by the federal government, but by private-sector entities as well. This is a priority of the president,” the spokesman said. Yeah, well, the vulnerabilities of OPM’s systems and the Interior Department facility that houses them existed seven years ago and before that. The incoming just happened to land and explode now. Now we can presume they really, really are a priority.

So now what? It will fall to Beth Colbert, the deputy director for management at the White House, to salve the wounds.

And Obama himself ought to voice his personal concern over this. Some things that occur externally do get to presidents personally. Johnson and Nixon waded into crowds of Vietnam protesters. But more than that, some concrete things should happen:

  • The White House should convene a meeting of the CIO Council to make it clear the 30-day cyber sprint ordered by Federal CIO Tony Scott is now a year-long effort.
  • Pressure test every important system in the government. Hire the top corporate cybersecurity experts — a group populated in part by some famous formerly malevolent hackers — and have them bang away until they find all the weaknesses. Then give agency heads one working week to prove their vulnerabilities are plugged. Two-factor authentication, encryption of data at rest — for heaven’s sake do it already.
  • Hire a tiger team to install Einstein 3A in every agency by July 31st, never mind December 31st. Require the internet service providers to do whatever it takes to make their inbound traffic compatible with this system. If Einstein 3A is so good, how come it’s taken so long?

I know what you’re saying. Yes, it does sound naive. I wasn’t born last night either. This is one of those times, though, that requires an all-out effort. For years we’ve heard warnings of a cyber 9/11. Well, we just had one.

This data loss was no third-rate burglary. Mr. President, America is under attack.

Have you visited the deep, dark Web recently?

July 2, 2015 Leave a comment

At the end of a long cul-de-sac at the bottom of a steep hill, our house sat near a storm sewer opening that in my memory is a couple of yards or so wide. If you poked your head down the opening and looked in the right direction, you could see daylight where the culvert emptied into a sort of open catch basin. None of us ever had the nerve to slip into the drain and walk through the dark pipe to come out in the catch basin, maybe 300 yards away. But that is where, at the age of maybe 5 or 6 years old, I realized a vast network of drainage pipes existed under our street, beneath our houses. That culvert fascinated me and my friends endlessly. We’d try to peer at one other from each end, or shout to see if our voices would carry. Or, after a rain, we’d drop a paper boat down the opening and see how long it would take to flow into the catch basin.

That sewer is like the Internet. Underneath the manifest “streets” that are thoroughly used and mapped lies a vast subterranean zone with its own stores of data. Some experts say the surface or easily accessed Internet holds only 4 percent of what’s out there. Much of the out-of-view, deep Internet consists of intellectual property that people — like academics or scientists — want to keep to themselves or share only with people they choose. But other areas lie within the deep Internet where criminal and terrorist elements  gather and communicate. That’s called the dark Internet. It’s also where dissidents who might be targeted by their own country communicate with one another. To people using regular browsers and search engines, this vast online zone is like a broadcast occurring at a frequency you need a special antenna to detect.

At the recent GEOINT conference, held for the first time in Washington, I heard a theme from several companies: Agencies will need to exploit the deep web and its subset dark web to keep up with these unsavory elements. The trend in geographical intelligence is mashing up multiple, non-geographic data sources with geographic data. In this and a subsequent post I’ll describe some of the work going on. In this post, I’ll describe work at two companies, one large and one small. They have in common some serious chops in GEOINT.

Mashup is the idea behind a Lockheed-Martin service called Halogen. Clients are intelligence community and Defense agencies, but it’s easy to see how many civilian agencies could benefit from it. Matt Nieland, a former Marine Corps intelligence officer and the program manager for the product, says the Halogen team, from its operations center somewhere in Lockheed, responds to requests from clients for unconventional intel. This requires data from the deep internet. It may be inaccessible to ordinary tools, but it still falls into publicly available data. Neiland draws a crude sketch in my notebook like a potato standing on end. The upper thin slice is the ordinary Internet. A tiny slice on the bottom is the dark element. The bulk of the potato represents the deep.

Halogen uses the surface, searchable Internet in the unclassified realm. Analysts ingest material like news feeds, social media, Twitter. They mix in material that is inaccessible to standard browsers and search engines, but are neither secret nor requiring hacking. It does take skill with the anonymizing Tor browser and knowledge of how to find the lookup tables giving URLs that otherwise look like gibberish. Beyond that, Nieland says Lockheed has contacts with people around the world who can verify what it finds online. Halogen’s secret sauce is the proprietary algorithms and trade craft its analysts use to create intel products.

At the opposite end of the size spectrum from Lockheed Martin, OGSystems assembles teams of non-traditional, mostly West Coast companies to help federal agencies solve unusual problems in cybersecurity and intel, or problems they can’t find solutions for in the standard federal contractors. CEO Omar Balkissoon says the company specializes in getting non-traditional people to think about traditional questions. A typical project is the Jivango community where agencies can source answers to GEOINT questions.

OGSystems calls its R&D section VIPER Labs, crafts services, techniques and data products for national security. At GEOINT I walked to a big monitor by Jessica Thomas, a data analyst and team leader at VIPER Labs. She’s working on a OSINT (open source intelligence) product for finding and stopping human traffickers and people who exploit minors. It’s a good example of mashing up non-GEO data with GEO. The product uses an ontology used by law enforcement and national security types of words found on shady Web sites and postings to them that may be markers for this type of activity. Thomas pulled two weeks worth of posting traffic and used a geo-coding algorithm to map it to the rough locations of the IP addresses. Posters tend to be careless about how easy it is to reverse-lookup IP addresses to get a general area from where it originated. In many cases, posts included phone numbers. It wasn’t long before clusters of locations emerged indicating a possible network of human trafficking.

An enthusiastic Tor user, Thomas wants to add dark Internet material to her trafficking data mashup. She also hopes to incorporate photo recognition, and sentiment analysis that can detect emotion within language found on a web site. She says OGSystems has applied for a grant to develop its trafficking detection technology into a tool useful for wildlife trafficking — a major source of funding for terror-criminal groups like El Shabab.

Next week, some amazing things text documents can add to GEOINT.

Numbers tell the tale of the #OPMbreach

June 25, 2015 Leave a comment

Some big stories become defined by the people and the emotions connected to them. The earliest news memory burned into my gray cells occurred on November 22, 1963. Emerging from my third grade classroom, I recall the emotions of the clustering Walking home, I remember my mother in tears before our black-and-white RCA console TV, declaring to me and, I guess, heaven, “Someday there will be another Kennedy in the White House!” I’m not certain I have  direct memory of Walter Cronkite pulling off his heavy glasses, having seen the kinescopes of that moment replayed periodically through the years.

We remember other big stories for their remarkable numbers and statistics. Not that it lacks emotional content, but the Bernard Madoff crime is notable for its sheer audacity of size, expressible in lurid statistics: 11 federal felonies, $65 billion in fraudulently stated gains, 150 years in the slammer. Who knows, the word “Madoff” could well replace the word “Ponzi” in the vernacular reference to really big frauds against innocent individuals.

OPM’s data breach, which has spawned its own hashtag — the new form of vernacular among what might kindly be called the vulgate — is a numbers story, leavened by the justifiable frustration and anger of the employees involved. Mike Causey likens this story to The Blob (one of my all-time favorite movies). Here the horror is underscored by numbers:

  • 4.2 million feds and retirees affected
  • 14 million more in another breach that claimed data from SF-86 forms
  • 18 million Social Security Numbers may have been purloined, according to testimony from OPM Administrator Katherine Archuleta.
  • Two hour waits to reach someone on the telephone at contractor CSID
  • 36 hours between OPM hanging out the notice of the credit monitoring services requirement and awarding a contract.
  • $21 million for the credit monitoring services so far.
  • 15 points in the plan OPM came up with for fixing its cybersecurity vulnerabilities
  • 1 direct report cybersecurity advisor working for Archuleta

This is one of those drip-drip stories in which details come out serially, although not all that clearly. From one of the congressional hearings we got a sense of how much more money OPM thinks it will need. In a kind of symmetry, Archuleta says OPM may need still another $21 million in 2016 to button up its systems. The agency has asked for a total of $32 million more for 2016, but is also saying the total cost of the breach could be as hight as $80 million. That figure won’t buy a wing assembly for an F-35, but it’s a significant figure against OPM’s roughly $400 million spending authority.

In business, bad results tend to bring on one of two outcomes. Either your budget gets cut. Or the money becomes available to fix the problem, but you don’t get to spend it because you’re gone.

So where are all the numbers and this story headed? It’s not over yet. We still don’t know the full extent of how many names, Social Security numbers and SF-86 forms were taken. When that many people are affected, it’s hard to make it disappear. We still have yet to learn the motivation of the data thieves, which means millions will be holding their breath for a long time.

As an unseen benefit, every other agency is scrambling to make sure it’s not the next OPM. One departmental CIO told me as much just the other day. Software vendors will have Christmas in the summer as agencies get serious about two-factor authentication, continuous diagnostics and mitigation and the tools that go with them. Homeland Security is scrambling to get Einstein 3A into place. So, some silver linings.

No TV news anchor pulled off his glasses and put them on again to mask emotional turmoil over the OPM breach. But at least the lessons learned, as the government likes to say, will stick this time.

OPM left a sizzling burger on the counter. The dog ate it. Who do you blame?

June 16, 2015 1 comment

Dog trainers like to say there are no bad dogs, only bad owners. I know. We have a now-elderly greyhound. She rules the roost, mostly. But because of her mild personality, she’s never out of control, never pulls on the leash, and has never so much as made a growl at anyone. Mostly she saunters into the middle of the room and lays on her back, her tummy available for anyone who cares to rub it.

But leave a hamburger on the counter, a cold drink on a side table, or an unattended dinner plate of food, and oh boy. Don’t turn your back. She’ll pretty much have it devoured before you can turn around and say, “No!” One time the extended family retired to the living room and family room after Thanksgiving dinner. After putting away some dishes I went into the dining room to pull the tablecloth. There was Lizzie, atop of the dining room table, licking up crumbs and tidbits.

Unlike China, which denies everything when it is caught stealing data, a dog caught stealing food looks at you and says through her eyes, “What did I do? You left it there.”

Dog on table

A young Lizzie cleaning up after Thanksgiving.

This is what I thought of when reading comments former CIA Director Michael Hayden made to a Wall Street Journal conference regarding the awful database breach. The U.S. personnel records were “a legitimate foreign intelligence target,” Hayden said. He added that our intelligence apparatus would do the same thing if it had half a chance. Hayden said he wouldn’t have thought twice about grabbing any Chinese government database the CIA could.

“This is not ‘shame on China.’ This is ‘shame on us’ for not protecting that kind of information,” Hayden said.

OPM left a juicy, sizzling hamburger on the counter. The dog snatched it.

Perhaps the U.S. government does do the same thing to rival nations. We don’t know for sure. Let’s hope so, because at the least it would leave things in a rough state of Spy vs Spy equilibrium. Because it is justifiably embarrassed, and because it can’t really do anything about Chinese cyber behavior, the accusations from the administration have been mild and sporadic.

Unfortunately, I see no other recourse other than for OPM Director Katherine Archuleta to resign. I don’t say this with any satisfaction. Not that she was personally responsible for the breach. Not that she’s a bad person. But the warnings were there, she had the knowledge that the hacked systems were behind on their FISMA certifications, and of the string of attacks going back a year. It all happened on her watch and it potentially harmed enough people to fill New York City, Chicago, Baltimore and Dallas. It’s not that she was personally malfeasant, it’s just goes with the territory. Had a rocket landed on the OPM building, that would have been one thing. But an egregious organizational performance lapse of this scale claims the person ultimately responsible.

Recall what happened back in 2012 at the General Services Administration. A conference 18 months earlier on which regional officials spent indiscreetly and contracted criminally came to light. Administrator Martha Johnson resigned before the reason why became known. Veterans Affairs Secretary Eric Shinseki toughed it out for a while, but ultimately had to step down after the drip-drip-drip of bad news from the patient scheduling scandal of last year.

OPM, as Francis Rose points out, has lost its credibility. Now it needs new leadership to restore it.

Fails happen. It’s how agencies react that matters

June 9, 2015 1 comment

An old, familiar shibboleth came up again this week. “Washington is a city of second chances.” That’s what a Washington Post article said about a popular millennial writer who was fired from a popular web site for plagiarism. He popped up at another web site a year later, where he’s boosting its traffic. Dennis Hastert, the former House speaker now enmeshed in a really bad scandal, probably is too old to have a second chance.

Organizations can have second chances, often because they have the wherewithal to buy their way back. I remember the Ford Pinto gas tank scandal (1977), the time Lockheed nearly went bankrupt (1971) save for a federally-backed loan, and the Tylenol poisoning scare (1982), which was a problem not of the company’s making. Today, Ford, Lockheed-Martin and Johnson and Johnson prosper quite nicely.

Can federal agencies have a second chance, I’ve been wondering? Technically no, since they can’t go out of business unless Congress decrees it, which it never does. So when they goof up, there might be temporary hell to pay, but not the threat of going out of business. In fact, serious failures are often rewarded with big budget increases, as in the case of the Veterans Affairs Department. Congress can readily replace money. Reputation and perceived legitimacy — harder to recover.

Yet agencies are obligated to react when things go wrong. Recently two examples occurred I point out as case studies of the right way to react and retain the confidence of the public.

A whistleblower, still anonymous, complained to the FDA about poor practices and fungus contamination at the National Institutes of Health. Specifically, in the Pharmaceutical Development Section of NIH’s Clinical Center. This is where doctors and technicians whip up experimental drug for small groups of patients. Two vials of albumen, a medium for injecting drugs into patients, were found to have the fungus. Patients had been given injections from different vials in the same batch. The FDA investigated the lab, and the NIH suspended sterile production. It won’t resume until at least June 19th.

The NIH went public with the episode, including a mea culpa from the director, Dr. Francis Collins. When I spotted the release, I asked for an interview the next morning with Collins. NIH public affairs people — they are among the best in the government — got me the principal deputy director, Dr. Lawrence Tabak. He said the NIH welcomed the highly irregular incursion by another federal agency. We don’t know what personnel changes will happen with the troubled section, but the speed and forthrightness of the NIH response seemed refreshing and, well, grown up.

Another agency, the relatively small National Highway Transportation Safety Administration published last week the results of a study of how it can function more effectively. The agency launched the review in response to how sluggishly it responded to the General Motors fiasco of the malignant ignition switches and non-deploying airbags. The defects caused at least 100 deaths when people’s cars turned off at highway speeds. This last year’s incident is still in the news, overshadowed though it may be by the explosive Takata airbag situation that’s affected millions and millions of cars by many makers.

Somehow the GM ignition switch-airbag issue went on for a dozen years before the 2014 recall, and the NHTSA blames itself in part. It says it was pushed around by GM, and it lacked the technical understanding staff needed to stay on top of these issues. The NHSTA report says the agency “failed to identify and follow up on trends in its own data sources and investigations.” The upshot: The agency has produced a detailed internal improvement plan, and appointed three outside experts to guide the improvement effort, including a former astronaut.

And what of the Office of Personnel Management, from which vast amounts of personal data on current and former federal employees were stolen? The lag between discovery and disclosure is troubling. More disturbing is the frequency of similar attacks and the seeming ease with which whomever — China, some lunatic insider, maybe a combination of both — is getting into federal data bases. As Jason Miller reported this week, the government has experienced nine incidents in less than a year in which hackers attempted or succeeded in stealing personal information on government and contractor employees.

How did the agency react? OPM did the obligatory offers of credit monitoring. It worked with US-CERT and the FBI, but the US-CERT report is incomplete, and in any case isn’t available at its web site.. The agencies still don’t know how much data was taken, or else they haven’t said. The stain is still spreading. As pointed out in my interview with cyber expert Rodney Joffe of Neustar, the loss of SP-86 data exposes not only employees, but friends, neighbors, and any foreigner they’ve ever done any sort of business with. Plus travel records and passport information. That lost data could pester people for the rest of their lives.

OPM says it’s techies secured remote server access and installed new cyber tools. The White House ordered the acceleration of Einstein 3A monitoring tools, not that the current version worked so well. Lots of sturm und drang, but no clear sense that the government is doing much more than improvising against something it only dimly understands and only feebly deal with.

My hope is that when the scope of the OPM breach is known, the same unflinching, critical and public self analysis exhibited by NIH and NHTSA will occur in the federal cybersecurity apparatus.

Follow

Get every new post delivered to your Inbox.

Join 31 other followers