Tag Archive for: SolarWinds

The leading cause of death in the workplace is falls. 36.5% of all fatalities are due to falls, followed by 10.1% caused by being struck with an object. Recognizing the problem, OSHA created requirements to protect workers from falls, including:

• guardrail systems
• safety net systems
• personal fall arrest systems
• covers
• positioning device systems
• fences
• barricades
• controlled access zones

All these controls, when used properly, save lives.

Hypothetical Scenario

A successful construction company is working on a 30-story office building. Timelines were already tight, but a series of material delivery delays has put them way behind schedule. In a rush to complete the project, it’s easy overlook certain things. In this case, a properly configured personal fall arrest system was overlooked. They bought the system, the system was onsite, but the system wasn’t installed correctly. Nobody noticed until one day a worker, twenty stories up, slipped and fell to his death.

As you can imagine, there was a serious investigation. In the end, the company admitted their oversight, received a fine, settled a lawsuit with the worker’s family, and continued operations.

A few weeks later, same thing happens. Another investigation, another slap on the wrist, another settled lawsuit, and back to business as usual.

A few months go by, and there’s another incident! The investigation cited the same cause as the others, a poorly configured/installed personal fall arrest system. This time, OSHA wants a public hearing and invites company representatives to answer questions before their panel. At the hearing, company representatives were asked the following question:

“If a properly deployed personal fall arrest system had been used, would these lives have been saved?“

A company representative responds:

“It depends. In theory, it’s a sound thing, but it’s academic. In practice it is operationally cumbersome.“

Seems reasonable, right?. We certainly don’t want to get in the way of company production!

Or, wait a second. This doesn’t seem right. Poor safety because good safety is “operationally cumbersome” doesn’t sit well with you. Good, it shouldn’t!

Sadly, a similar analogy plays out all over the information security industry every day.

Hearing on the Hack of U.S. Networks by a Foreign Adversary

The construction analogy hit home while watching recent testimony in front of the U.S. Select Committee on Intelligence.

On February 23rd, 2021, Kevin Mandia (FireEye CEO), Sudhakar Ramakrishna (SolarWinds CEO), Brad Smith (Microsoft President), and George Kurtz (CrowdStrike President and CEO) were invited to give their testimony about the attacks on SolarWinds Orion last year (and ongoing). These are four very powerful men in our industry, and I appreciate what they’ve accomplished. In general, I have a great amount of respect for these men, but I’m not comfortable in their representation of our industry without also considering (many) others. Some of the reasons I’m not comfortable, include these facts:

• They run billion and multi-billion dollar companies that sell products and services to protect things.
  • If people were already protected, they’d have nothing to sell. There is incentive to keep people insecure.
  • Companies must continue to produce new products (See: product life cycle diagram below). Without new products, sales decline. As long as people keep buying (regardless of need), they’ll keep making.

• They have significant personal financial interests in the performance (sales, profit, etc.) of their companies.
• They represent shareholders who have significant financial interests in the performance of their companies.
• They may lack clear perspective of what most Americans and American companies are struggling with due to where they sit.

A hearing such as this is a fantastic opportunity for people to tout their accomplishments (which they do), tout their companies accomplishments (which they do),  and sell more stuff as a result. I DO NOT fault the witnesses for doing these things. It’s their job!

Let’s just hope our Senators take the hearing and witnesses in proper context and seek many more perspectives before attempting to draft new policy.

IMPORTANT NOTE: It may appear in this article that I’m critical of the people in this Senate hearing, but this is NOT the point. The people participating in the hearing have done tremendous things for our industry and our country. For all we know, if we were in one of their seats, we would respond in much the same way they did. If anything, I’m critical of us, our industry. We have tools sitting right under our noses that we don’t use correctly. Instead of learning to use our tools correctly, and actually using our tools correctly, we go looking for more tools. This is ILLOGICAL, and might should be negligent.

The point.

At one point during the hearing (1:22:08, if you’re watching the video), Senator Wyden (D-OR) begins a logical and enlightening line of questioning.

Senator Wyden:

“The impression that the American people might get from this hearing is that the hackers are such formidable adversaries that there was nothing that the American government or our biggest tech companies could have done to protect themselves. My view is that message leads to privacy violating laws and billions of more taxpayer funds for cybersecurity. Now it might be embarrassing, but the first order of business has to be identifying where well-know cybersecurity measures could have mitigated the damage caused by the breach. For example, there are concrete ways for the government to improve its ability to identify hackers without resorting to warrantless monitoring of the domestic internet. So, my first question is about properly configured firewalls. Now the initial malware in SolarWinds Orion software was basically harmless. It was only after that malware called home that the hackers took control, and this is consistent with what the Internal Revenue Service told me. Which is while the IRS installed Orion, their server was not connected to the Internet, and so the malware couldn’t communicate with the hackers. So, this raises the question of why other agencies didn’t take steps to stop the malware from calling home. So, my question will be for Mr. Ramakrishna, and I indicated to your folks I was going to ask this. You stated that the back door only worked if Orion had access to the internet, which was not required for Orion to operate. In your view, shouldn’t government agencies using Orion have installed it on servers that were either completely disconnected from the internet, or were behind firewalls that blocked access to the outside world?”

To which Mr. Ramakrishna (SolarWinds) responds:

“Thanks for the question Senator Wyden. It is true that the Orion platform software does not need connectivity to the internet for it to perform its regular duties, which could be network monitoring, system monitoring, application monitoring on premises of our customers.”

Key points:

1. SolarWinds Orion did not require Internet connectivity to function.
2. The IRS had Orion.
3. The IRS did not permit Orion to communicate with the Internet.
4. Attackers were not able to control the IRS Orion server (because it couldn’t communicate home).
5. The attack against the IRS was mitigated.

Senator Wyden continues:

“Yeah, it just seems to me what I’m asking about is network security 101, and any responsible organization wouldn’t allow software with this level of access to internal systems to connect to the outside world, and you basically said almost the same thing. My question then, for all of you is, the idea that organizations should use firewalls to control what parts of their networks are connected to the outside world is not exactly brand new. NSA recommends that organizations only allow traffic that is required for operational tasks, all other traffic ought to be denied. And NIST, the standards and technology group recommends that firewall policies should be based on blocking all inbound and outbound traffic with exceptions made for desired traffic. So, I would like to go down the row and ask each one of you for a “yes” or “no” answer whether you agree with the firewall advice that would really offer a measure of protection from the NSA and NIST. Just yes or no, and ah, if I don’t have my glasses on maybe I can’t see all the name tags, but let’s just go down the row.”

Points made by Senator Wyden:

1. Network security 101 includes blocking high-risk applications from connecting to the Internet when it’s not specifically required for functionality.
2. Firewalls are designed to block unwanted and unnecessary network traffic.
3. There is good authoritative guidance for using firewalls properly, including from the NSA and NIST.
4. None of this is new.
5. Organizations that don’t follow “network security 101” are irresponsible.

Kevin Mandia responds first:

“And I’m gonna give you the “it depends”. The bottom line is this, we do over 6oo red teams a year, firewalls have never stopped one of them. A firewall is like having a gate guard outside a New York City apartment building, and they can recognize if you live there or not, and some attackers are perfectly disguised as someone who lives in the building and walks right by the gate guard. It’s ah, in theory, it’s a sound thing, but it’s academic. In practice it is operationally cumbersome.“

OK, here the logic falls apart. The answer “it depends”, followed by “firewalls never stopped” a FireEye red team exercise, did NOT answer Senator Wyden’s question. Logically, this (non) answer would only be valid if (at a minimum):

• The FireEye red team exercises were run against a “network security 101” firewall configuration.
• The FireEye red team exercises were a variant or emulation of the SolarWinds attack.

The question was whether a “network security 101” (or a properly configured) firewall would have mitigated the SolarWinds attack (meaning a firewall configured to only permit necessary traffic, as per NSA and NIST guidance). The non-answer justification continues by mentioning “in theory, it’s a sound thing, but it’s academic”. Since it’s been brought up, this IS NOT theoretical, it’s factual. If an attacker cannot communicate with a system (either directly or by proxy), the attacker cannot attack or control the system.

The last part of this statement brings us (finally) to our original point. Using a firewall, the way it’s supposed to be used (“network security 101”) is “operationally cumbersome”.

Responses from the others:

• Mr. Ramakrishna: So my answer Senator is “yes”. Do standards such as NIST 800-53 and others that define specific guidelines and rules. (THE BEST ANSWER)
• Mr. Smith: I’m squarely in the “it depends” camp. (Um, OK. So, a non-answer.)
• Mr. Kurtz: Yes, and I would say firewalls help, but are insufficient, and as Kevin said, and I would agree with him. There isn’t a breach that we’ve investigated that the company didn’t have a firewall or even legacy antivirus. So, when you look at the capabilities of a firewall, they’re needed, but certainly they’re not be all end goal, and generally they’re a speed bump on the information super highway for the bad guys. (Basically the same statement as the first. DID NOT answer the question.).

So the score is 3 to 1, “it depends” (without answering the question) versus “yes” (the correct answer).

Operationally Cumbersome

If a firewall (or any tool) is effective in preventing harm when it’s used correctly, why aren’t we using it correctly? The reason “because it’s operationally cumbersome” is NOT a valid argument.

It’s like saying “I don’t do things correctly because it’s hard” or “I don’t have time to do things right, so I don’t” or (as in our construction example) “We don’t have time to use a personal fall arrest system correctly, so people die”? Truth is, our infrastructures are so interconnected today, a failure to configure a firewall properly could/will eventually result in someone’s death.

So what do we do today? We do the illogical:

• Since we don’t have time (or skill or operational bandwidth or whatever) to use an effective tool effectively, we purchase another tool.
• We won’t have the time (or skill or operational bandwidth or whatever) to use this new tool effectively either, so we purchase another tool.
• We won’t have time (or skill or operational bandwidth or whatever) to use the new tool and this newer tool effectively, so we purchase yet another tool.
• The insanity continues…

What we must do (sooner or later):

• inventory the tools we already have
• learn how to use the tools we already have properly (knowledge/skill)
• use the tools we already have properly (in practice)
• then (and ONLY then) seek additional (or different) tools to address the remaining gaps

As an industry, we must (sooner or later):

• make this “network security 101” (it’s not new, so we can’t call it the “new network security 101”)
• hold organizations responsible for “network security 101” (the opposite being, the “new irresponsible” or negligent)

Other facts

Firewalls are NOT the end all, but they are an important part of security strategy. Here we are, many years down the road and we’re still fighting the same fight: the basics.

• Firewalls have been around for more than 35 years.
• Firewalls block unwanted and unnecessary network traffic (inbound/ingress and outbound/egress).
• A properly configured, “network security 101”, “responsible”, “best practice” implementation of a firewall would have mitigated the SolarWinds (or similar) attack.
• Many (maybe most) U.S. organizations have a firewall that is capable to mitigating the SolarWinds (or similar) attack.
• There are still ways to bypass a firewall, but if you don’t have your firewall configured properly, what are the chances you’d stop a bypass anyway?
  • application vulnerabilities
  • SQL injection
  • social engineering
  • physical access
  • man-in-the-middle

Operationally cumbersome is not a valid excuse for our failures to understand and follow the basics.