By ProcessBolt | July 25, 2019
According to Gartner, vendor risk management is:
“…the process of ensuring that the use of service providers and IT suppliers does not create an unacceptable potential for business disruption or a negative impact on business performance.”
The inherent issue with most vendor risk management platforms on the market today is that they’re failing at the “ensuring” part in the statement above. Most have taken the path of least resistance by quickly applying vendor scoring or labels, which is intended to convey a level of security and give you peace of mind to do business with them. The inaccuracies and business risks in these types of methods are too numerous to name.
We recently interviewed measurement expert Douglas W. Hubbard, CEO at Hubbard Decision Research and author of How to Measure Anything in Cybersecurity Risk (Wiley, July 25, 2016). Hubbard sheds some light on the issues surrounding vendor labels and scores and offers three questions to ask before you select a vendor risk management platform.
The Wild, Wild West
Evaluating vendor risk management platforms on the market is an arduous task. Each platform assesses vendors differently, each weighs questionnaire answers differently, and each calculates risk based on those answers differently. So how can a business decide which platform truly and accurately calculates third-party vendor risk?
“It’s wild, wild west for vendor risk management right now; anybody can do VRM,” says Hubbard. “You can’t start an insurance company without an actuary, it’s against the law. Accountants have to be certified; lawyers have to pass the bar. This is not the case for cybersecurity. There are some certification methods, such as CISSP, but it’s not a requirement for building a vendor risk management platform. Some people are just making up their own scoring methods for evaluating vendor risk.”
Those scoring methods are the root of the problem, according to Hubbard, because they don’t give you the information you need to make a calculated decision on whether or not you should do business with that vendor. “What you want to know is the probability of losing X amount of dollars or more in a given period of time,” he says. “If I get some data on the risk of a particular vendor, does that help me better forecast the probability of losing say $10 million or more due to some event in the next 12 months? That’s what I should be able to do.”
The Problem with Labels
Many vendor risk management platforms on the market today place labels on vendors to signify a level of risk. Low, medium, high and critical risk are typical labels you’ll see used abundantly when classifying vendors. The problem with this method is twofold.
First, a label is only a label and doesn’t help you make an educated decision on what to do next. “Not only do labels introduce ambiguity and make the estimates worse, they don’t even support the decisions,” says Hubbard. “People have to look at these labels and then make another level of subjective judgement about whether or not they should invest in a certain vendor.”
Second, verbal labels add errors to the estimation and decision process, which creates a placebo effect and gives people a false sense of security. “These risk analysis methods can make you believe that your estimates are better when they’re actually not,” says Hubbard. “There’s nothing statistical about it. There’s no reason to believe it improved decisions at all. There’s lots of reasons to believe it made it worse though.”
The Problem with Scores
Other vendor risk management platforms rely on a scoring system to rank vendors. Each vendor is given a score upon completion of the questionnaire. This score is designed to convey a level of risk based on a certain ranking methodology, similar to a FICO score.
“I see how these ideas have spread because they’re being inspired by an already inaccurate, broken system,” says Hubbard. “But the difference between these systems and the FICO score is that FICO does have predictive power. A credit score is actually inferred from a whole history of behaviors that paint a picture of the person, which gets reduced to a single number and that number is used to try to compute the risk of non-repayment.”
This process is entirely different from evaluating one vendor based on a set of questions. “It is possible to develop a scoring method when you look at a whole population of people and correlate it to something,” says Hubbard. “What would be difficult is looking at only individual responses and trying to weight them and add them up, without any knowledge of the thing that I’m trying to forecast. If we are not even sure what specific event is being forecast, the weights don’t even make sense.”
Three Questions to Ask First
Hubbard recommends asking these three questions when evaluating vendor risk management platforms:
A Better Way
At ProcessBolt, we don’t rely on labels or scores to convey vendor risk. Instead, we simply assist in making the vendor risk analysis process easier through automation. We’ll help you streamline your vendor risk assessments and easily manage your entire vendor risk landscape. Our platform replaces spreadsheets and emails with an automated and customizable workflow that includes interactive questionnaires, data-driven dashboards, and much more. Sign up for a 1-on-1 demo today.