[podcast] IAM Metrics
For this episode Jim and Jeff talk about why IAM metrics are important and some ideas for the basic reporting of an IAM program. This was sparked by a recent Forbes article titled "Nine Cybersecurity Metrics Every CEO Should Track" which is worth checking out.
We hope you enjoy this episode and please subscribe to our podcast for updates on new episodes!
LISTEN HERE or read the full transcript below.
*Disclaimer from Identropy: These transcripts are produced using automated tools, so may not be an exact word-for-word transcription. (i.e. - if you read something that sounds wrong, it's the tool's fault!) As always, for a better experience, please listen to the actual podcast.
Podcast #28 Full Transcript:
Identity At The Center #28 - IAM Metrics
Jeff: Welcome to the Identity at the Center podcast episode #28! This is the premiere podcast Jim for IAM Talk. I'm Jeff. Hey, Jim. Hey, How you doing?
Jim: good Jeff. This is going to be as exciting as a MMA fights. I'm sure.
Jeff: Hopefully a little bit longer than the last one.
Jim: Yeah, UFC two sixty seven or whatever this is, is where it all happens.
Jeff: I watch it every once in a while. Just kind of the clips and stuff like that and, some of Gregor won and like what a minute.
Jim: I was watching some of the under-card matches and I don't know those sport's just too bloody for me.
I just still enjoy watching people beat each other to a bloody ball. But I do like watching the like you said, the highlights and everything you know about it enough for me.
Jeff: The especially bloody of hard spots. It's you know, it's the new Roman Coliseum. It's the new gladiator. I mean, most sports are like that now. And this is definitely MMA is a more primal, obviously.
Jim: Yeah, sure.
Jeff: Football's like that, too. Speaking of football, my forty Niners are in the Super Bowl.
Jim: And where are you going to be watching the game Jeff?
Jeff: This is killing me because I'm going to be on an airplane basically for the entire Super Bowl next week. I will be somewhere over the skies of Ohio or Pennsylvania or New York. I would imagine I'm hopeful that I'll land in time to be able to catch the end of it.
But yeah, I wasn't thinking when I came up with this schedule and now I'm paying for it.
Jim: Yeah, I think it would change my flight Florida Saturday and got to my hotel room and made a good deal like Chili's or something, I don't know.
Jeff: Yeah. Have a nice little tailgate party in my hotel room. And just watch it by myself. I guess that would be better than nothing, but...
Jim: It's better than missing it.
Jeff: Yeah, that's true.
So I had to figure it out. I don't know if United is gonna have Wi-Fi available on the flight that I'll have, but, maybe I can figure out some way to streaming or find some kind of work around.
Jim: IS That on FOX. Do you know what they broadcasting you?
Jeff: I think it's on Fox because I think Joe Buck is calling it.
Jim: Your favorite.
Jeff: Never mind. We don't want to go to hell. What do we want to go in today, though? I think I want to talk metrics. Right. How do we measure in IAM program.
Jeff: I know you sent me an article recently. There was a Forbes article that talked about some cybersecurity metrics that every CEO should know and track. Why don't we start there?
Jim: I really looked at the metrics and they had a lot to do with dealing with cyber security events, triaging them, how long it would take to free eyes, how long it would take to solve and things like that.
And I thought, well, isn't really specifically relevant to my IAM program managers.
It's higher up a cybersecurity ladder looking at other from defending the perimeter perspective. But there were a few cool things I pulled out of the article. I thought that were cool. And then I started saying, oh, if I'm IAM program manager which I have been. What are the key metrics that I feel like should be a part of what we're reporting on a periodic basis?
And I mean that right there is a discussion that is periodic basis. I mean, how often should you be disseminating metrics and in what form?
So a couple of things that I pulled from the article. So one of them was develop and agree on an initial set of metrics that span defined operational capabilities, ensure that there's transparency and consistency in each metric will be measured.
Jeff: First of all, how do you translate that? That's a mouthful.
Jim: Yeah, it was kind of a mouthful.
So I translated as getting agreement on the metrics. So, as I'm an IAM program manager is going to be seeing these metrics to somebody. Are they going to get them and look at them and say,I don't care, this is meaningless to me. And so that's what we want to avoid.
So for me, what this was all about was getting with those folks and saying, well, these are the metrics I think are important. What do you think? Is it matter to you? The answer is yes. You're in a great spot. Then you find out what else would they like to see if the answer is no. OK. You know, why not? And what would you like to see and say what would be more important to you? And regardless, I mean, if you're going to be selling them information, it's gotta be something that they care about. Otherwise, you're wasting your time.
Jeff: Yeah. You don't waste your own time to building up a whole bunch of metrics that nobody's ever going to see or care about. Right.
Jim: Right. Exactly. And normally you're sending these to folks who are influential on your career, maybe their higher ups, maybe they're your peers. But regardless, you want to make sure that it's information they care about. But one thing I've pulled from the article that I thought was really good was endorsing the ability to capture and record the initial metrics on a quarterly basis. So in other words, there shouldn't be some investment. It could be as simple as somebody spending time pulling data into a spreadsheet and crunching the numbers. But for some level investments, sometimes it's going to take a technology investment. But, if the metrics are important, people actually care about them. And you did that first step. And I think that, you should be able to use that to justify or help justify whatever investments required to get those metrics.
Jeff: And I think that some of that, too, is. Are you able to accurately report some of those metrics too. Right when you're making that agreement and figure out what you want to measure? Is it something that's possible? It might not be possible now. But as you move along the program and maybe there's investment in technology or reporting later, maybe you may be able to develop new metrics down the road that you can't today or that if you did today would be very time consuming and or maybe not as accurate as you'd like it to be.
Jim: Right. And, obviously, some of the metrics you would want to report may be as simple as just running a database query or running a report from your service management system or wherever you're tracking certain things like patch or resets.
But that's usually going to be pretty basic information. How many pastors resets so we'd deal with this month? Things like that. And there's not value, but I think you need to put that into context. So you're saying are this false or rolled out, new complexity expected the upper reinforced fruitfulness words we roll out know different technology. And I really well, I love using like time graphs to kind of show like, this month we had eight hundred calls the following month we had eight hundred and ten calls a month actualities eight hundred five CC, OK. And then we update password complexity to ten characters and we force everyone to change your password, 1600 calls.
And that was, you know, go back out to 8 hundred and then we rolled out password self-service and it dropped the number.
And then you can kind of like she could tag that month with a little, flag or something. And so this is what we did that. And that's why. So that takes an investment of time. But it really communicates like, we're making these investments. We're coming to you and saying we're going to do these investments and it's going to give us some benefits. It's going to give the company benefits. And then it's coming back in and showing what that benefit is.
Jeff: I like the time-based ones, too. But like you said, it's investment time because how far back are you able to go back and get information to show that history? Right. So I kind of think I've always like a history metric in order to have that. You have to have the history. So if you want to show improvement in any standpoint, you have to be able to pull that information. Now, compare that to other metrics maybe that are less historical and more compliance based. What you think about that?
Jim: Yeah, I think that's right on. I mean, one of the things that I think as an organization, you need to comply with your policies. Right. I mean people are rolling their eyes and saying, well, my company, you know, policies don't get us anywhere.
But OK, then, where are the drivers?
And, your policy should reflect what you're driving toward.
If you have obviously external compliance standards that you need to comply with regulatory standards, then you should have policies that enforce being in compliance with those standers. So I think it's ultimately the organization policies that need to be complied with. And if you can provide metrics in terms of, how well you are complying with your own policies, and you can even highlight some of the key policies that hit home with the people you're disseminating these metrics to. But if you can say, we're only 50 percent compliant with our authentication policies and, maybe this business unit is 100 percent compliant. This business is 100 percent compliant. This business over here 10 percent compliant. And then the chief I.T. person from the business, you should say, why are we only 10 percent compliant, a real 100 percent compliant? I go back to their lieutenants, if you will, and say, what's going on, guys and girls? What’s the problem here?
And, it's kind of a process where you are shaming people into complying with the policies in a way, but it's creating visibility around it. I find is one of the ways to, get steam behind information security and issues. There's a lot of times information security doesn't wield a stick. It's kind of a carrot approach and its influence to get people to comply. And as far as I'm concerned, visibility is a very key way to influence.
Jeff: Yeah. The nice thing about having metrics available to show that it's difficult to argue against numbers. I think it can't be done. But, you know, it is more factual base and gets away from more of a subjective analysis that people might have around compliance, sometimes it's not as simple as a yes or no. But numbers help tell that story. What are some examples of a compliance, the authentication compliance metric that you can think of.
Jim: So, ultimately, if you have any kind of central authentication tool like an ITP system, ADFS, ARKS and paying something like that where you can invest corporate research and you may want your goal may be to get all the applications in your organization to leverage that system. And this is where I think the subjective story line could help link up with those numbers. So, you have certain business units, 100 percent of their, in scope applications are leveraging that system. And so you're managing policy centrally. And those policies lead to contextual authentication, strong authentication, U.S. multi factor, password policy is being enforced, things like that. Then you have another business where they basically said, we're not doing that. And so we're going to do our own thing or that's just not in our priority. Were too busy to integrate into that. And there is a 10 percent and obviously vision came over the black and white example. It could be that they're 70 percent compliant. I think it's important, but they don't think it's that important. But, the company invested money because they believe in division. They want you to get to that. And then you've instituted a policy that systems need to comply by using this system or granting an exception. And by the way, exceptions should be excluded from the calculation of compliance with the policy. If you grant an exception, you should not say that they're not in compliance. I think that could be debated. But ultimately, that would be an example.
Jeff: That makes sense. Yeah, I was going to ask about the how you handle exceptions and I feel the same way as they've been granted that. So move on. Right. Maybe there is some sort of review process that goes along with exceptions. Maybe it's a temporary exception.
Jim: Right. I think exceptions should have a time to live one year, maybe two years, even if there are exceptions, such as a very common exception is technologically infeasible to integrate a mainframe application that doesn't have a web interface with an IEP that's based on, SAML or OpenIDC integration. We all understand, right. You're not going to integrate a green-screen with with Okta. It just doesn't happen. So you're going to approve that assumption. But I think that they shouldn't come back every year or two years to renew the exception. Has anything changed?
Jeff: No. We're still on this mainframe.
Jim: From the mainframe, and I think that's been the story for a long time.
But then, the question should become, OK, well, when you're getting off the mainframe?
Jeff: Right. Now, we've had several clients over the years that have looked to get off the mainframe, but they're typically really embedded in the business and the technology and the underlying processes that it's running. And it's very difficult to get out from underneath that without a significant investment. And I find it very interesting to kind of get sidetracked here is that, for a long time it was a mainframe paper are going to look for work because everything's moving to a database and so forth. And then now mainframe people are in demand again because no one knows how to run this stuff. Right. Companies need data help with it.
Jim: There are people coming out of retirement with the Y2K thing, know really dating myself here. But there were people coming out of retirement to help switch things over, get Y2K compliant. And, you can apply it in mainframes. Won't be around long. Well, I know I've done client engagements within the past year or they've had some of the biggest mainframe environments I've ever seen. So it shows, the cost benefit of replacing all that technology that's been built over the years and the processing power. And it's hard to make the case that it should be taken all.
Jeff: Yeah. So I know that kind of. Go back to that article. There was a couple of different types of metrics. One was right, the first that was kind of based around visibility. At least being aware of things. And I know that's one of the things that we always kind of think about is how do you know who has access to what? And then the other one was based on the big section and then more workflow based. So speed and an ailment. Things like that. What are some workflow type metrics that you think would be interesting to report on or helpful for an organization?
Jim: Well, one, the ones that some mind is an easy one to usually an IAM program is responsible for is access review. So you have a identity governance system in place and you're collecting and correlating entitlements and accounts from all of your applications.
Be able to have owners, whether they're managers of people or applications, review a list of who has access to these applications.
I think from a metric standpoint, you know What is the total scope that you'd like to be doing access reviews on how many are actually doing them so it's something that's valuable early in your program. So you've gotten to 90 percent plus adoption. You can start getting in two. And I've done this in the past where you report on what percentage of access reviews have actually been performed because you have some people, individuals within the business. You just think I'm just too busy. I'm not going to do this. You sent me five reminders. I'm deleting them every time they come in. And, most people aren't going to do that. Some people might. And if you can report out that, we've got 100 percent complete over here, 100 percent that a half percent complete is almost always the goal, 100 percent and 98 percent every year or it's happening. Well, there's two or three people in this business unit who have not done their access reviews.
But the CIO or the director or Rivers kind of a responsible person over the business who's going to want to know why aren't they doing them? This is part of the job, you want to work here. You want to have people report you. You need to do your office's reviews. But I've seen too much. So that's a good way to gain visibility and to get people to do their job.
Jeff: What about things like boarding, off boarding password resets? And I catapulted for like ticketing. But what about something along those lines? What would those types of things show?
Jim: The first couple that I talked about were really like, we'll see the help you're giving from the business. But I think another thing you can do with metrics is like show the value that your IAM programs bring to your organization.
What is the amount of work that you do? And showing no growth over time or, if you're, balancing manual work versus automate your work and how much you're getting done.
And I think is just good, because I think we're always at some level wanting to make sure that people understand that we're bringing value to the table. What do you think?
Jeff: No, I think it makes sense. I think a lot of it revolves, too, around the story you want to tell, too. So you're gonna build out a metrics dashboard. What is that dashboard meant to communicate? Who is going to go to? such a good type of on boarding, it goes back to what you're talking about earlier. Why do I care? Who cares? The number of on boarding is being done by IAM because the business doesn't really maybe have an involvement with that. Maybe that metric isn't really for the business. So I think that's part of trying to figure out is where do you want these metrics to be communicated and what's the story you want to tell? I'm a fan of having to different versus the metrics, one that's kind of like private. That meant for the internal team only, and another one that's maybe publicly posted that it gets promoted out to the organization to show the value that's being brought to the organization, you do something like a steering committee. Right. What's the what's the metric that you want to bring to a specific steering committee to drive a conversation and you talk about compliance? Right. Hey, if we're just starting up an IAM program and in the process let's say, pulling in applications and to centralize access management or a government solution, tracking the number of solutions that are connected to that, probably make sense. But at some point it's going to become, well, who cares, right? So how to item thinking about down the road? Well, what's replace it metric that could be in place or that might be. Yeah, we're doing this number of on boarding on a specific application. If you're able to pull that kind of data, which most modern identity governance solutions should be able to help with being able to adapt the message over time, I think it's important.
Jim: I was thinking of that. There's almost like a maturity model, like you're starting off with very basics, so you're moving to things like roles. And that may be a very catchy something in saying that we need roles. I mean, you talk to clients all the time.
We are back and you know, we're on this journey. How much of your provisioning is happening, the roles? I mean, if somebody comes on board.
How much of their access gets provisioned on day one, for example, how much on nations are around roles?
And you know what? Their requestable roles, how much of the coverage is there be a requestable roles versus individual entitlements?
Jeff: Yeah, roles are hard. I mean, I think every company wants to get there. It thinks they want to get there. But over the last four years of me being in the consulting space, I think I've only seen maybe one or two organizations that have actually gone to that level of access.
Jim: Yeah, a lot of times there's like isolated to the business areas that really adopt it and others say where we've got our other priorities.
Jeff: Right. What about different investments, I know that, my friend Jody was looking for some help with trying to talk about how to communicate the value of IAM as apartment investment. And when the idea is I came up with was plotting the investment along with maturity rankings and it got me thinking maturity rankings are good for a certain amount of time, but the bar is consistently moving.
So, what is a 100 percent score today might only be a 50 percent score in the future. How do you address something like that where the target might be a moving target as you move along the program?
Jim: Yeah, I mean, for my money, there're two polls when it comes to this one poll is how many people are using the central solution. So if you make an investment in an IDP system, how many people are using it? But then to your point, what is a 100 percent from an IDP standpoint today? There's only been 50 percent for a year for now because the technology is moving and the space is growing and the solutions are getting better.
So the two poles are how many people are using the central solution and the other is essential solution keeping up? Is it keeping up with all the advancements? And so I kind of feel like that's the two areas, you have to kind of keep an eye on and figure out a way to report on those two areas, certainly when people are you.
But my first point about compliance. The second one is a little more subjective. But I think as an IAM program manager, you've got to know your space.
You've got a new watch out there and you've got to be able to make those subjective calls.
Jeff: Yeah. I'm thinking about like situations where, MFA wasn't really as prevalent as it is back in the day. And then all of a sudden it becomes the recommended way go forward by experts and everyone else that's, part of the space. And let's say your organization isn't doing that to introduce that then as a new metric and say, OK, here's how number a percentage of our applications are using multi factor authentication and then how would that affect, if you've done some sort of assessment the past where maybe, kind of what we do, where we say, OK, well, we think here, a three, four authentication and that's, where we think you're at right now, two years from two years after that MFA becomes sort of the de facto gold standard for authentication. And all of a sudden your maturity drops to a one, having that communication with your stakeholders and your steering committee, etc... I think it's important to recognize that some metrics are moving target and that as those as the industry and as the environment changes that may affect. Maturity or other things, other related metrics going forward.
Jim: the other thing I was thinking you're talking about the, the Gartner this year and one and the presentation of those and they said MFA everywhere is the new standard.
So, the first thing came to mind was every time I use a new application, it's going to want me to use my authenticator up or get and is ridiculous. And then I thought about it more on my MFA everywhere doesn't mean by any means really contextual to know.
You've already given your MFA, by the way, MFA can mean a lot of different things. It could be I'm on the right network.
I'm using a recognized device. It could be that we accept that device has been authenticated within the past 30 days, so we're not going to ask for it again. So it doesn't necessarily mean that this is ultimate disruptive process. But if your brand new device showing up for the first time, it doesn't matter what research you're hitting, you should hit MFA. So what does that mean? Well, we're using the central IDP and we've got a common set of policies. And it doesn't matter which of the systems that are integrated, you're going to go through that same sort of policies. And those policies should be strong enough to say, to give you that MFA everywhere.
In other words, it doesn't matter what research you're winning if you haven't played it before, it's MFA.
That's at least on the enterprise side and on the customer side, maybe it's less so.
But I really feel like, we were getting to a point where 100 percent of people agree username and password and passwords are not good enough. Right. I mean, we all save me today, but I think we're finally getting to a point where nobody tries to argue that password target.
Jeff: Yeah, I'm looking at my web browser right now. I'm thinking of all the different links that I would click on that do have MFA associated with a service. And I think that's I think that's especially challenging because people view MFA is maybe a roadblock. But I think it's really it's a standard. So if you have a service that's out there, put MFA on it.
Jim: But every time I go to a Web site and I want to buy something and they want me to create a profile and they asked me to come up, create an account and given them a password, I think to myself, great. And when you're base gets hacked, my password, it's going to be out there.
And that's why I have to kind of over the unique password. And next time I come back to your site, I'm not going to know what those I'm sure these last pass or something like that. But, then doesn't work on all my devices from current log in from a new device. No, it's just a pain in the butt. And if you have an MFA, I'd be a lot happier.
Jeff: Yeah, I think we could have a whole separate discussion on ways to introduce MFA that don't have less of an impact on the user experience because that's important.
Right. You need to make sure people can use your product, but it needs to be done in secure manner.
Jim: And the idea of using last pass or something, like I must feel embarrassed by the use user being that I'm in the end shoot. But if you talk to most people on the street, you would know the last pass is. You got the concept of it.
So those are you know, that's the majority of human beings are used and being alone. I think I've no clue.
Jeff: Well, that's the whole concept around passwords is people kind of use them. Now, I'm more of a tech forward person, in case you weren't aware, Jim. Yeah. So I've been using the last pass for seven years, eight years and a long time. And, I've put my past in there, but I still reuse passwords every once in a while. It's something that I as I come across them, I'm starting to change it into random. And it does provide a little of a hassle trying to log it into a new application where I have to dig the password out. And I out of my last past, whether it's the browser or if I'm using it, I'll assets, you know, integrate with the key chain that way. So it's definitely not the easiest thing. I think that goes back to the user experience of having some sort of password vault. Right now, you could do something within Chrome itself as a browser or Safari has a built in and Microsoft with their new edge browser and they're all the browsers have always had some sort of password. Remember, not the most secure thing, but if you get to the habit of just creating a random password and then storing it somewhere, you can get into that mindset. I think people become a lot more secure. The challenges. The ease of use and being able to do that quickly when you want to log into an app.
Jim: Makes sense to me, man. I think we hit this topic pretty good. What do you think?
Jeff: Yeah. So from a metric standpoint. Right. Being able to track visibility is one thing and then being able to track workflow and being able to understand what the effectiveness is of the difference processes or technologies that you can put in place. Important. And then for me at least, being able to differentiate between what is an internal metric meaning for your team only for your eyes only and what's something that you're willing to publish and put out there from a measurement standpoint, because sometimes those introduce questions that you may interpret and answer. Some of the think about,.
Jim: Yeah, my things are I think it's you use metrics to help justify your existence and what value your IAM team is bringing to the organization is probably a better way to put it. And the other is to Gartner help from leaders in your organization. These people are just not doing their part to help keep us secure. They're in your group. we're going to turn over a list. You go get them.
Jeff: What is it you would say you do here?
Jim: Yeah. What would you say? I don't think anything is going to get that reference Jeff.
Jeff: All right. Well, I think we will leave it there for this week. Appreciate everyone listening. And for more episodes, don't forget to visit identityatthecenter.com and keep sharing the show out with friends. And if you'll like the show and if you don't like the show, share it with your enemies. That's fine, too. We're happy to have as many likes as we can and look to talk with folks down the road.