Name: Where do humans belong in today’s SOC?
Uploaded: 2026-09-21T19:16:06.682Z
Duration: 1 h 8 s
Description: Where do humans belong in today’s SOC?

Transcript for "Where do humans belong in today’s SOC?": Hey, everybody. I'm Dave Merkel, CEO and cofounder here at XPEL. We're gonna be hosting a topic today that I think, is on everybody's mind, at least in the cyberspace if you are an operator, which is amidst all the AI hype, which I guess yeah. I think that's a fair word to use. Where do humans belong in today's security operations center? So that's what we're gonna be talking about. I know we're all cybersecurity people, which means we're skeptics and negative by default. I think it's a character trait required to be in this business. I've only been doing it for thirty years. Get off my lawn. Regardless, though, this is not an anti AI panel. And I think being anti AI is probably, career limiting and also being, perhaps not realistic about the application of the technology. So we're gonna have a few panelists today, three. And so I'll I'll go ahead and let you know who they're gonna be. So first, Merlin Clemens. He is the, security engineering manager at Riot Games. Kinda cool. And, he runs incident response detection engineering and threat intelligence. He's been in cybersecurity for eight years, and, he likes long walks in video games and telling stories in tabletop role playing games, and he is now my new best friend. Lewis McIntyre. So, Lewis is the senior director of cyber fusion at Markel, not Merkel, Markel, where he leads, teams responsible security operations, incident response, cyber investigations, and threat detection. He focuses on integrating people, process, and technology to improve cyber resilience while trying to reduce risk across the global organization whose entire business is, since it's insurance, risk. So a topic he's gonna know a lot about. And then lastly, Sean Thomas, who is the senior director of threat detection and response and fraud at ZoomInfo. He has twenty years of experience in security operations, and he's built and ran threat detection and response across MSSPs and internal companies. So he, like me, also has vendor experience, which is always an interesting lens, to add when you've done cyber, operations inside of an organization. So before the conversation, we asked them to do a bit of prep work, and we have this, trust versus impact framework in terms of thinking about AI and how you apply it to cybersecurity operations. So if it's, you know, a high trust thing, meaning you really have to trust the outcome, and a high impact thing, meaning if you mess it up, the consequences are dire. That kinda sits in the top right of the, of the two by two. And if it's low risk and low impact, that's kinda that kinda bottom left. And we ask them to plot out certain tasks and things that they think about in terms of their DNR operations and where they rate those things because we found that's a good proxy for acceptance of, you know, not just, automation, but also application of new computer science, like large language models to completing those tasks. And so, that exercise informs some of today's questions and a bit of our conversation to try and tease out where do you use this new computer science, you know, large language models. You know, where is that applicable? How is it applicable? And where are you like, hell no. Don't give me any of that. And so, that's the conversation we're gonna get into today. We do have, you know, the chat is live, so feel free to interact in there. And we do have folks monitoring it. So if you have questions, we'll get back to you in the chat itself. And with that, let's go ahead and get into it with our panelists. Roland, Louis, Sean, welcome. Let's just dive right in. So I've got a handful of questions, and then we're just gonna chase the rabbits wherever they run. Alright? So, so let's start with, let's start with this. Alright. So when you hear autonomous SOC, right, when somebody says or AI SOC, pick your short, you know, buzzword of the day. Right. Like, what's your honest reaction? Like, we're at a bar. We're hanging out. I'm like, oh, you're in cyber? Autonomous SOC, I hear that's the thing. Or AI SOC. Isn't that amazing? Like, what do you do? What do you Okay. So here's here's the thing that I find fascinating about this whole concept. Right? I have spent twenty years working in or running SOCs, and I've spent twenty years talking about, like, doing conference talks on, like, why don't we trust security operations people? Mhmm. Why don't we let security operations people explore and learn and grow and, like, give them trust to take actions on endpoints? Why do we structure them down to a strict playbook that gives them one option and no thought? Yet, suddenly, we're just gonna trust a computer that's known to hallucinate all of that over a human. Mhmm. Like, it's such a fascinating transition that we're moving into Okay. Where we've spent so long not trusting a human to do a sensitive job. Yeah. But we're gonna trust the computer to do it fine, and we know the computer is wrong a reasonable amount of time. Uh-huh. That's fascinating to me. Like, that's I'll I'll start it there. Okay. Alright. Louis, what do you think? Oh, I mean, well, for me, to his point, I feel like when most people hear autonomous, like, they think of just taking out the human altogether and just letting the computer run with no human decision there. So I recently just heard a new term in regards to this, like, human over the loop versus in the loop. And, essentially, I think that's where it needs to be, where a human is just watching the whole process happening and not just letting the computer run it from end to end. So that's that's where I'm at with it. Let the AI help the humans out versus just taking little process. So a scaling technology, but not a decision elimination technology. Right. Okay. Okay. Well, how about you? Yeah. So, personally, I don't even like the word autonomous in the mix because it genuinely means without oversight. Doing, like, your own thing in a bubble where it is just allowed to do. So whether that is humans or AI, I just like it. And so for me, like, with a SOC, what I've noticed is I don't want autonomy in the ability, like, by itself. I want autonomy to do things, but I also want a SOC to be integrated with other teams. And so when I start hearing things like autonomous SOC, AI driven SOC, it's breaking apart, like, to your point. We've been doing this for so long, and we still struggle to get humans to make the right decision. And now we're making these humans that we think struggle to make the right decision, make AI make the right decision. Doesn't work in in my my logic flow with that. Okay. And so I think that, like, we've had augmented SOCs forever. We augment with tools. We never called it an EDR SOC. We we never called it a low code automation SOC. Oh, interesting. Like, this is just another tool in the toolkit that, like, yes, we're throwing at the wall to see, like, where it sticks. But, like, I don't think that's going to be the way we look at it in a year or two years. It'll just be a tool in our toolkit, same with everything else. Well, it's the same buzzword as anything else. Yeah. Like, when EDR came out, RSA had, what, a thousand EDR vendors when that was kicking off. Mhmm. When SOAR was a big thing, RSA launched a billion SOAR vendors. Does anybody remember the year of PKI? Oh my god. Like and that's that's that's old I am. Okay. When a new technology comes, everyone jumps on, tries to market it, tries to say this is gonna change the whole industry. And I think you're really right. Like, it's about how we integrate it more than it is about full on replacement. Okay. Not only, like, full on replacement's It's incredibly expensive depending on how you look at it. Right. Okay. Companies are finding that out today. Paying for token. Like, when the bill comes due, it's anthroptic. It like, our bill comes due soon, by the way. So pretty soon, like, enterprise plan? No. You know? Professional? No. It's tokens, tokens, tokens. Yeah. So many companies are finding out, like, hey. We're gonna drop people, bring an AI. Then they're looking at it, and they're going, oh, my. Actually, the AI is kinda more expensive than the people. Yeah. Like, that token cost creeps up on you fast. It it does. So we've got that trust impact sort of two by two. Sure. And, I know you guys sort of went through and looked at tasks and kinda where you mapped them, but let's maybe talk about extremes. So, you know, trust is sort of, okay, where do you need extreme trust? And then where is the impact high when the trust is not there? So sort of, you know, high trust needed, high impact if wrong means that's probably a human. Give me maybe your top humans are always chef and doing that. Yep. And then maybe your top, dear god, please AI this thing away from me Mhmm. Because it wastes my time. Yeah. No. So starting with the high impact, high trust, definitely detection engineering. I always wanna engineer to be they're either supervising that loop or to be inside of that loop. So so detection engineering in your in this context, you mean building new detections, validating they work, and then deploying them to whatever mess of infrastructure that they need to have. Yes, sir. And when you say human in the loop, or do is is the human a hard gate? Meaning, until the human says, okeydoke, it doesn't go? Right. Or does it go and the human is QA ing with a finite time frame? Mhmm. Like, which how do you think about that? Yeah. For our our company, it's more about the human giving the actual final approval. Okay. So the button the button is pushed by a human for deployment. Exactly. So they've done some process. Right. They've done the testing and then testing the environment and all the policies in our EDR. And when they finally see, like, it's not gonna produce or affect any type of revenue generating process Okay. Or make the CEO mad because it knocked him offline because he logged in in Madrid instead of logging in on the East Coast. Footnote, my team, you can knock me offline anytime. This is fucking funny. Yeah. No Slack? No email? Oh, that's a glorious one. Yes. Yeah. Yeah. So I definitely want that human, at that final point in the Fargo Okay. Protection engineering. And then I think, the biggest thing for us was remediation. We are part of the finance industry, so we have to report to regulatory entities. So Yeah. We wanna make sure, like, any parts that we're taking remediation, a human can account for, be able to speak to and defend Okay. If that makes sense. So there's E and R and and actual remediation. How about on let's go the other way. So, like, make this make this work disappear. I never want one of my people to see it again. You know? AI this magically away from me. Yeah. That's a good one. Merlin and I, we have this conversation this morning. I think repetitive task. So anything works in known infrastructure, CIDRs, IP addresses, anything like that, like alert triage, I'm okay with the Yeah. I have Low fidelity. Low fidelity. The repetitive tasks. Because going back to our earlier point mentorship, I am a big component of having our principals, our leads, teach all of the younger individuals so they understand the fundamentals. So if the principals are not having to do that triage part, they can spend their time mentoring our other analysts. So Okay. Okay. Then, how about you? So sort of never ever ever Mhmm. Always my people and dear God, please take this away from me. Totally. So I think we're in a very similar spot on this is, like, I am, and for my team, like, detection engineering and, like, remediation containment are those two that, like, I want human driven, I want human approvals, I want human testing, I want limited AI. When it comes to, like, I think you would push it, like, to deployment, that's I don't want a human to decide that. I don't want an AI to decide if something gets deployed. Okay. I don't want an AI to determine whether or not the testing was successful or not because hallucinations on data. So if we're looking at, like, high impact, a bad rule I think you had mentioned it as, like, a bad rule can generate a lot of noise. It can affect a lot of people. It can affect revenue streams. Okay. I, at this point, do not trust AI to go full loop on its own to determine that something is a gap, create a rule, and determine what kind of, you know, playbooks need to happen off of that, and then implement and fire. I would not, at any stage, do that for detection engineering and remediation. Would I say augmentation of those things, where, hey, maybe you're utilizing AI to come up with the queries that you need for the rule or things like that? Sure. Oh, okay. So so if I can restate what I think you just said is use AI to produce detection content Mhmm. But not the detection itself. Okay. And then you you're gonna QA what whatever it is Exactly. It's gonna be QA'd. But, like, I want a human to have a hypothesis. I want them to go and test the hypothesis, come up with what they needed. But if they wanna use AI to help determine what's the best query in this? You know? Maybe I don't use this tool that often. Maybe it's a new one that we just onboarded. I don't understand the query language. Hey. Spit out something for me. Make that Or I know it in this, but not in that. So Exactly. I know what I want. I just don't know what language or You know, maybe your pipeline for your data isn't, you know, exact same as another index. Yeah. And so you're trying to figure out, like, okay. Let's say IP address over here, but it's something else. Hey. Go figure out what these are. That's fine with me. But, like, I want the human to have hypothesis to do the testing to go and run all of the test to determine that it works. And then, you know, at the end, when it is shipping and creating a detection, depending like, we'll get into the the nuance of that because I agree with you on, like, low fidelity things. Cool. But if I'm taking containment action or remediation action, you're like, 95% of the time I'll leave a little bit of nuance. 95% of the time, I want a human making the determination Okay. On whether or not the thing that you downloaded is gonna get, you know it's kicking off in isolation. Okay. Okay. Sean Sean, how about you? So I I'm I'm mostly aligned, but I I wanna I wanna kinda come at this from a slightly different angle as we talked about it. Because I think context is king. Mhmm. So, like, it's not I want a human to do detection engineering. It's that I want a human to do the critical components of detection engineering. Or, like, they're they're in every case, like, if we look at the the framework that you guys sent over. Right? The stuff that is high impact, high trust isn't, like, a specific task. It could be a part of a task. It could be a task with the right content. Okay. Okay. Like, if Like, if we said containment, that's actually a multi step about containment. Let's let's let's go let's go there. Yeah. So and you all said, by the way, humans on the containment, humans on basically Somewhat. Immediate remediation. But let's let's break it down. Go another level. So, like, the context matters. So, like, if I get a suspicious login at 03:00 in the morning and I don't have staff, Why not automatically lock out that Okta account and reset sessions? Mhmm. What does that cost me as long as it's not an executive or somebody who would freak out or whatever else? So you have some finite set of exceptions. Yeah. But broadly, it's FOB in accounting is kinda fucked on Monday morning. FOB in accounting is absolutely right. Like and that's the thing. So we love you, Bob. Yeah. A 100%, Bob. It's great. But context is what's like, is that remediation action so critical to the company? So, like, where I get into the Nuance, am I gonna make changes in a production environment with AI? Hell. Am I gonna make changes to a user's, employee's box? Yeah. Maybe. Why not? Like, that's much lower. You'll laugh. Mhmm. Like, I'm looking at each decision that we would have a human or AI make rather than a category. I see. Saying, hey. What is the impact of that decision if it is wrong? So so it sounds like context becomes more important for you to more aggressively use the technology versus kind of use case. Yeah. It's all context like that is always Like, this is oh, no. That's what you're saying. Are basically in the same spot. Yeah. Yeah. I don't think we're in disagreement. I'm just No. What's the it less in a I don't wanna talk about, like, detection engineering or remediation. I said the context of The context of what you're doing, yeah, the nuance of it. Yeah. For sure. Especially when you're trying to talk about AI being intentional versus that first. Right? We work in incredible nuance every day. Sure. Like, the to do security operations and IR, any of that stuff well, it's adaptability, incredible nuance, and, frankly, a lot of human ingenuity. Like Mhmm. I will always trust and love a person's gut feeling. And from managing MSSPs to moving into, like, internal like, if somebody has a gut feeling that, like, I feel like this is off, dude, follow that. Mhmm. Like, seriously. Like, pull that thread. Do what you need to do. And I think we have to look at like, when you look at AI, like, it's not gonna have a gut feeling for you, but we do need to look at the context of each decision we might let it make individually instead of Broadlink. Okay. That's gonna allow us to more fully realize what it can do. And so when when you guys think so I heard a lot of agreement Yeah. In in terms of that breakdown. Maybe, Bruce, maybe we'll start with you. When you think about context, like, what we could list 47,000 things. Give me five. Like, when you think about context for decision making in in in this scenario, like, alright. If if I have these pieces of context that gives me more flexibility in determining where I can more aggressively push the AI or automation or whatever, nonhuman throttle Mhmm. Forward. Right. What are the most common and sort of what I'll call standard DNR ops? I realize this is a highly contrived notion. It works. But, yeah, let let's just see what what happens when I ask that question. Yeah. No. I think for me, it would be especially coming from the finance portion, it would be anything revenue generating. Okay. We wanna make sure we to triple check. So it's sort of system missions. System. Basically. Right. Okay. System administration, anything dealing with identities when it comes to the executive So who? Who it is. But, by the way, are you more or less sensitive? Like, are you less aggressive on the automation with executives or more aggressive? Less. I mean, for us, just due to Wouldn't we all want to be more, but we all end up But we have we all have to be the less. Okay. Right. Okay. And I'm speaking from I might come back to that. Experience. Yes. Throw a bomb on that one. But let's finish let's finish this, and then I'm coming back to that. Yeah. No. Less when it comes to the executive. Because at the end of the day, there are they are always traveling, always doing things. And when you're trying to explain to them, like, why they're taking an action in the grand scheme of things, they might not care. They might just wanna be back online. So Okay. Yeah. In in that case, I'll just go with list. Okay. So the so the mission of asset, the person Yep. What what else? We'll we'll take it three. Come back three. Yeah. Okay. Alright. Let's make it three. We're the one you so top three context. Sure. Yeah. So for me, I think the the number one, and this is a I don't know one of those, like, contrived notions because it was done at New York. So this is, like, critical assets. Whether that is infrastructure Yeah. It's mission critical stuff. It is, which, to be fair, there's the nuance of, like, depending on the situation, there are gonna be certain things that, like, you know, I had to have this conversation recently with, like, we we're gonna provide you a list of critical assets so that you don't action on them. Like, oh, no. Those are gonna be the things that I knew first. If I see some weird stuff happening on something that is, like, mission critical, I'm not going to, like, ask for permission. I'm going to tell you what has happened, and we'll deal with the, like, the repercussions analysis. The the difference is I'm protecting video games. Alright? Like, the nuance here is I'm a we are a video game company, so, like, I am not taking down medical equipment. I am not taking down a finance company. Like, I'm taking down services, potential services, for a little bit while we get things figured out. But like critical assets for like, hey, what are things that can go down? What are things that can't go down? What users, you know, what service accounts have admin and like are in these different areas? What AWS accounts are super important. Because you start to apply the nuance, which I think is my somebody had made a great comment when we were reading over the, like, the the page that Vin put together, which was like, hey, all of these are nuance because my the way that I'm gonna handle remediation on a low priority is different than a critical priority. So, like, low priority stuff, cool. Like, to your point, it's a Saturday night, nobody's working, and, like, Bob from finance, Bob from finance gonna have a bad weekend. Yeah. Like, he shouldn't have been working to begin with. But, like, if it is a, you know, a server that is running video games, stuff like that, I wanna know that so that we can create the appropriate response. Like, I'm not necessarily going to automate the takedown of a server for people that are, you know, millions of players in a video game. Sure. I'm gonna have an automation that alerts a human, and the human goes, oh, damn. We gotta do this. But, like, I want ownership for a human to take those into, like, critical asset. And so I I think that's all three of mine Yep. Are like, hey. The nuance of critical assets into that, because that falls into, like, identity. It falls into actually Yeah. Servers. Just saying an asset is critical is not a sufficient class. Exactly. We're breaking those classifications down to know, like, hey. What are the things that, like, we can and can't take action on? Okay. Because if we've all agreed, hey. We don't really care if we take down user accounts. Then we've all agreed on this. Newcomb. Right. Yeah. Okay. Okay. And by the way, Bob touched grass, apparently. Yeah. You want me to Sean, anything to to add to that in terms of context? Business impact, I think we hit a lot. Right? Like, that's where we talk about critical assets, revenue generating, blah blah blah. Yeah. The other two that I would add in this, and this is very much how I think about, like, detection engineering as a whole and how I think about AI and any automation that we do. Time sensitivity. Mhmm. So, like, how time sensitive is this action that we take it? Like, is my team going to be backed up on something else and it won't get done? So if we know the impact of what it doing can cause from an outage revenue generating perspective or whatever, We know the time sensitivity of it. And the last one, and the one that I think is really important to consider, especially when you think about, like, a detection is how bad is bad. Right? Like, a phishing email coming in is whatever. Somebody clicking on a link, whatever. Somebody putting in their password, whatever. Somebody getting into a system through Okta or whatever your identity provider. That's a different level of impact. Okay. And I think how I I really like to think about, like, possible impact and how I make decisions. Because the higher the risk, the more likely I might be to try to find a way to automate that in some fashion. I see. At least make sure it's raised appropriately. I'd say those are not Okay. That that's really interesting. The so I have a bunch of takeaways, but I'll just try to keep a couple here. One is, about the commentary on the the nuance around context and classification. Just because an asset is critical is insufficient for you to make determinations about how much risk from an automated response. However, it's done, deterministic computer science or LLM, whatever, that's there's more nuance required to really take advantage of the technology. And I think, you've added more to that, which is how bad is bad, which is the same sort of nomenclature. Like, oh, it's a critical incident. Not enough. Like, you need more to be able to enable your ability to use technology more aggressively. So I have I have very strong feelings on how most of our industry classifies incidents Okay. Alone, let alone critical. But Okay. Alright. I'm I'm I'm gonna come back to the bomb I threatened to throw. Executive user accounts. So in a world in a world where I think we would all agree that time to respond and or and remediate has got to get shorter and shorter, like, we've always talked about, like, the stuff we do is, you know, alert to fix in less than twenty minutes. Sure. The mission I gave the team is too slow. Twenty seconds. Okay. Cool. Why the sensitivity with executive executive accounts? Ostensibly, accounts that have access to the most sensitive information, the most context about your business, the most ability to fuck shit up if somebody impersonates them, and yet you all, I think, mentioned a little bit, you have set by the way, this may all get totally cut out so nobody gets fired, so it's fine. So and so But let's be real. A good executive shouldn't have access to most engineering systems. They're like I lost my ready to commit shit long time. That's Yeah. There you go. That's good on the server. Yeah. It gets my email and calendar. Yeah. As soon as as soon as I went into leadership, I went to somebody in an engineering org. I was like, hey. Can you spin up? Can I, like, get can I get those VMs spun up? I wanna build something. Like, you don't get to build shit anymore. No. Leadership. Mhmm. No. I was like, oh, alright. No. I don't. My job sucks. So, like, is that executives really depend on whether or not they actually do have access. What they do have access to is, like, sensitive financial records, company secrets. Sure. There's absolutely access that exists. Mhmm. But, also, like, let's be real. Like, there's a sensitivity to it because they get bad, and they sign our paychecks. Right? Like, it all goes back to the culture that you have built at the company. I would not have Which is a top down. That's a top down thought. But I would not hesitate to to lock out an executive. I would hesitate to let AI make that decision and make it wrong. There we go. That that that that I see. Yeah. So the cost of a mistake is much higher when you have that level of you know, it's fucking ownership, though, like I've done earlier as you want a human to be able to own that mistake. Because, like, I I'm a big fan of it. Right? Yeah. Speak to it. You know? Early on in my career, I, you know, I came up in an MSSP, MDR, as an analyst, and I have this vivid memory of one of the first isolations that I did, was the CEO. And I remember isolating And again, please expel. Isolate me now. Yeah. And I isolated the device. What a good day that would be. We got a phone call about five minutes later. And my VP walks over, he goes, hey, You just isolated the CEO's device. And I went, yeah? Yeah. Mhmm. Sure. He's like, you need to undo that. I went, no. And he was like, well, he wants to pick up those nobody's above the wall. And, like, that's how I've kind of treated it, and it's one of those things I got to shoot out for. Yeah. I'll be too young. If you're gonna fire me for protecting the company and I can just find my decision, cool. Go for it. Right. But I'm still going to make the right decision, and I just guess I don't have a good for you and I own it. Well, it's it's well, the three of us are here. To take the blame. Yeah. Yeah. Yeah. Job is to do that. Yeah. But we gotta put ourselves in a good position to defend it. Right? And that goes back to the AI not having the context and nuance of a particular situation. So So the why has to be Has to be come from the human. We want that human to because they can speak to the nuance of it and even give the white glove service, right, where AI is just like, no. You you don't find an executive as a precaution. You don't lock an executive without some form of notification Interesting. To say, like, hey. Like, we're doing this. Like Good. A notification, I get. Brings a certain amount of respect. Okay. We tend to give notification to anyone at risk workers. Yeah. Yeah. There's a certain amount of respect that everyone is deserving of in that situation. Mhmm. Time depends. It might be a little late depending on how, like, how serious something is. Okay. But, like, you just there there's always a different approach with different people. Okay. This this actually this this plays in well. And, like, Luis, maybe I'll start with you. Automation gone wrong. So at AI, not AI, don't care. Don't you just they're they're a human was not in a loop. Mhmm. So software of some kind, pick your version of software, and it did a thing in a detection and response operations context. You're like, fuck. They're like, give me give me a, and you can anonymize. It doesn't have to be turn them forward. It could be anywhere. So but but whatever you're willing to share, shit, that didn't work out well. Yeah. So there's a particular vendor that has a quite a bit of ship market share Okay. That, uses automation in that regard, and it has started deleting accounts in our cloud environment. Now we sent out the ticket to the vendor, let them know that it was doing this, and it was like, oh, yeah. This is part of our automation that does this. But can you explain it? A cyber vendor. That is cyber vendor. Okay. Right. Yeah. And they again, Nameless is Yeah. Yeah. Nameless. Yeah. But to that point, they had a very hard time explaining why the automation was doing that. And I had to go back to the access management team and explain that to them, like, this is what the vendor says. But at the end of the day, I'm the one holding that bag because You brought that vendor. Yeah. Yeah. It's your it was your yeah. You're a vendor. So that was one where we really would have liked for them to explain in the deployment of that automation, what it does, why it's doing it, and how we can explain it to our customers. Yeah. When it does make a mistake, this is the reason why it's doing it. So that's the one example I have. Sean, how about you? Does anybody remember FireEye HS? I know. I might need to. It. I I don't know anything about that product. So But do carry on. One of the fast one of the most fascinating things about early EDR for any of the kids who might watch this later, is it it ran on the system, and it was worried live on the system. Yes. So without getting into any specific about it, I will just say I have seen queries go awry and basically shut down three quarters of huge billion dollar companies. Based based on what the infrastructure was asked to do, what that product was asked to do. Yeah. And then Yep. Okay. And when you start to get into automation tools that then query for you, Right. Like, you know, we link in with HX or any of the other tools that did live query like that. Right. Run this query. Well, if that query is not good, you just dust half of the company. I got it. So so if, to to take it a step further, so in a nondeterministic agentic workflow, doing triage or investigation, for example, it could potentially make bad decisions around impact. I mean, I'd also be just genuinely be terrified about, like, you know, what is an AI gonna think is the right answer nine times out of 10 if they query on a box direct? Mhmm. Be this HX or or any new EDR that Anything. You can just you can open a command line to. They're probably gonna remove that file. Like, that's gonna be the first thought. That's awesome. I I want that file. I need that file. Because if that file is there, I need to analyze that file later to see what it is. Right. So if you remove that file, that's bad for me. Right. Back to deterministic. There are things that need to be okay. Okay. Well, then how about you? Bad bad automation story, AI or otherwise Uh-huh. In a cyber context. I got two. Oh, okay. One of them is nightmare situation. Mhmm. Both of them. If the person that was impacted by this is watching, I'm so sorry. Alright. Yep. So This Feel like we should be pouring some tea right now. This this is not a tea conversation. This is, like, we're the beers. Oh, alright. Alright. So get a call from on call. Doesn't normally happen. The person that on called is one of those guys that, like, if they're calling you, you know it's fucked. Yeah. Okay. Yeah. And I just get the call, and it's like, hey, this is above my pay grade. I need you to deal with this, which has never happened before. And turns out what had happened was, there was an automation where you could take a hash and ban a hash in an EDR tool. Right? Sure. Fairly simple stuff. Mhmm. The automation allowed us to go from the platform we were working in to the tool via API. So we have to pull the host name and, like, apply that band. Yeah. In theory, this is where we get to, like, you know, you talked earlier about, like, we still don't trust humans as much. So this tool goes and gets caught in a loop where the value from the tool, like, the the value that was sent via API changed in the automation and was empty. And so it read empty as anything, all, and went through a loop, banning this cache. Explainer that this was going. And I just across the entire 7,000 device deployment. Yeah. The hash in question, service host. Yeah. Which the EDR tool did not have a catch in. Oh, no. And so as computers started to restart, they didn't turn back on. Right? And so I got a call from that, you know, that night brought into that. It was probably about 3AM before we finally got things to a point where, like, the screaming had stopped, because it was it also happened on the the individual who was our contact, their first Disney vacation with their children Oh. Oh. As he's sitting in the hotel room. And the entire time, I'm like, yeah. I know. I get it. Don't worry. We're we're we're on the phone with the EDR vendor. We're like, what how what do we nothing. Nothing. Nothing to do but sit and suffer. Oh, no. And so that's where, like Oh. If you apply that to, like, humans made this system, it was a hard coded automation That there were changes that happened outside of that automation that you didn't take into consideration when writing it that maybe you should or shouldn't. I'm not a developer. I don't I don't know all of the things to look for, but, like, then you apply AI to that mix and say, hey. What are all of the variables that an AI now has to keep track of to not do the same thing in a containment sense? When does it hallucinate that host name? And or because this comes to the second one I had, which was we had a threat intel feed that accidentally pushed, the hash for an empty file. And our automation took that, banned it. And that just blew everything up that had the hash of an empty file across the environment, which is apparently a lot of stuff got caught by that. Mhmm. Let's be real, though. Like, in today's third party world, we we we barely know our own chains to know when changes happen. Uh-huh. 90% of what I feel like I've been working in, like, the last year has all been supply chains. Oh, absolutely. It's all been downstream vendor reach. I see. That have an effect. Like Interesting. Because it's so easy now to get into a code repository or one of these small companies that deliver code to multiple people, pop something there, and boom. Now you got 300, 400, 500 companies. Yeah. You've you've inherited a a a vulnerability. Yeah. Yeah. Yeah. Yeah. Like, we understand so little. Like, this is so spiderweb. We're not on internal networks that are gapped. We're we're barely even on in, like, a computer anymore, like, between phones, between SAS. Like, it's a whole different world. Right? And and to be and and now if I can be doom and gloom, I'm Irish. I'm allowed to. The, software potato thing. The, if you think about the world of sort of there's so much stuff that that relies on open code, which, by the way, they they kind of open source. But a lot of AI related software changes or AI authored software is incredibly hard for a human being to Mhmm. Yeah. Read and understand. Query doesn't necessarily follow conventions that are easily parsable. It's interesting to think about how that actually complicates the supply chain vulnerability problem and the ability to find those issues. Although, you could argue there's a Blue AI solution to that to try and find those vulnerabilities faster. It'd be interesting to see what the Datant kinda kinda pans out. The AI will create the vulnerabilities, then we'll pay the AI. Yeah. I do find them. Yeah. AI program and other AI will check it and say, you know, it's wrong. That one's wrong. And then just give me an alert. So Awesome. I just maybe we just need to get a popcorn machine. We just, like, move to universal basic income and all stuff working. That you know what? You know, as soon as as soon as my, security team cuts me off, you know, it's gonna get some popcorn and go do that. But, yeah, I'm gonna stay at home. Alright. So let's let's code a little bit. So we've we've already talked quite a bit about detection engineering and how you guys think about the application AI in that context. So I won't belabor that, but let's let's move up the stack, to the human interface that your organizations have to have. Right? So you produce results like bad thing happened, impact, blah blah blah blah blah blah. There's the technical layer of that. Sure. And there's the communicate to the business part of that. Right? How do you think about the application of, large language models in the context of report production? Because analysts this is a generalization, not universally true. But more often than not, not in love with that layer of communication. So okay. And Sean's like, good. Man. I gotta shut up. I gotta talk to it. This is this is I'm really excited about AI. Right? Okay. Because I I have a really freaking cool team right now, and one of the first things that they did was building off the back of our automation solution. They built a fully functional, like, AI enabled chatbot. Mhmm. So I'm talking, like, anything that we can do or that we do on a regular basis from investigation, pulling information, enrichment, and or remediation. We can all do from a single interface. Like, AI really is bringing us some ability to single interface in a way that we never got to before. Okay. Because, like, we get an alert that comes in. We can pull that alert directly into this bot. We can say, hey. Like, we need some enrichment on this IP address. Enter. It'll be like, hey. You wanna do that? Yes. Boom. Okay. Cool. Oh, woah. That looks bad. Block that Okta account. Here here it is. Are you sure? Yep. Boom. Enter. Done. Like, we wanna, like, go then create the report on the back end of it. Hey. Go open the ticket over here. Enter. Boom. Done. Okay. Like, it's so freaking vast, and it's such a good use case application. Now, I mean, I don't know if anybody is, like I don't know if there's, like, a a normal vendor solution that exists to do this. Right? Like, we're creating our own stuff on top of the currently established SOAR or SOAR platform, basically. Sure. Yeah. But, like, that's where you think, like, I I trust AI to do exactly the thing that I told it to do when we have preprogrammed in the guardrails that only allow it to do the thing that I told it to do. Oh, okay. And and and that would extend to things like report writing, but you're actually making a larger statement, which is yes and actually is a, you know, sort so so whatever. Pick your pick your harness of choice. We have a multi agent hierarchy where we have 10 specialist agents that are each specially coded. Again, I did not invent this. This was not my idea. This was the brilliance of the team that I work with. Okay. And I'm very lucky for this, and I they deserve credit, and I wanna give them credit. But, like, each agent has specialties. So we have, like, a remediation agent that has all the remediation context of what it can do and how. Yeah. We have a threat intel agent that is connected to all of our threat intel external sources. We have an enrichment agent that is connected to our enrichment resources, our reporting agent. That has our support. Voice and guardrails right now. Log in to the chatbot. Yeah. Yeah. Yeah. You tell her what you wanna do. It finds the right agent to help you do that stuff, and then that agent is built so that it knows exactly what it is allowed to do, and it is bound to only the actions that it's allowed to do. Okay. It is a freaking cool piece of tech. You basically have your own harness Yeah. That is wrapped around that. And it sounds like you're saying the more the tools and infrastructure you use exposes its capabilities to that harness, the more your analysts can just drive from a single place. So the day, that it I'll say the bad your single pane of glass is actually your harness, and vendors are more valuable when they let themselves be exposed to that harness. So you don't okay. Okay. Okay. It helps. And it's a, like and I I don't genuinely think you're ever gonna get to a single pane of glass for all things all the time. Sure. Mhmm. Yeah. But I think having, like we have one place that we take in alerts, and we try to make sure every alert goes to one place regardless of system. Mhmm. Because I like, today's time, you can't be, like you can't just have all your alerts to that stuff. Like, you're gonna get alerts from, like, other pieces of tech, from data warehouses that the company owns. So you have that all coming to one store, however you wanna run it. Right? So we have one place that has alerts. We have one place where we can do some of the work, and then we can pivot to any of the tools individually if we need to. Right. But we try to limit and let people have as much power in one or two places as you In a single context. Okay. So I started on report writing that Sean expanded it, made it Right. On the social too. It's all good. What what's your reaction to that? Yeah. For me, I think it's good for AI to be able to help with the templates. Again, like, we have to do a lot of reporting out to regulatory entities. So anything you can do to speed up that process. But, again, I still want my analyst to be able to explain it to the CSO, the finance, the legal team. Well, how did it come to this result? They need to be able to speak to that. So I felt like, anytime that it can be used to build templates, SOPs, playbooks, it take that work away from the analyst and the managers so that way they can focus more on making sure that they are reading all the outputs right and able to explain that piece. So I don't agree with Sean. Anything we can do to make that process more efficient and taking it out of the hand so that way we can work on burnout, alert fatigue, things like that. Out of my own morbid curiosity, like, do do any of you as you answer, I'd love to know. Do any of you worry, like, like, I will absolutely use AI to write, and my people will too. But do any of you worry that that's gonna make people, like, so much worse of explaining what's going on? That's my point exactly, actually, it used to be. Okay. Yeah. No. I don't I don't think it will. I mean, because, again, we're talking about the end result and how we got to that piece. Now if it's done without that nuance or context, like, if it's just built in a vacuum and then if you do yeah. Investigation, scratch, and outside. Exactly. Yeah. Then that piece, that that that's the part that's where we And now that now you you you apparently have a strong opinion on that, Sean, positive. So, same when it comes like, we've got a very similar tool that my team built. Not many. They're incredible. That sounds like it might be using the same framework. But that's the area where, like, I'm full send, like, have it put together, all the enrichment notes, all the action notes, the timeline notes, put that in the ticket, close it out, we're cool. When it comes to, like, if I have to, if there's any kind of after action report, if I have to sit in front of another team and explain what happened, an incident report, anything like that, I, to your question, feel that any time that an AI generates those reports and the human has to explain things, it loses the nuance of what a human would add, and then you also, and this is my personal opinion, and this might be just me becoming old and and crotchety about it, When you're presenting, you need to have something that doesn't feel like a robot wrote it. Okay. So To to make to it. Yeah. To, like, have other humans resonate with it. Because one of the goals of, like, an incident report is, like, I want you to understand what happened, and I want you to care about the takeaway so that you feel like you can personally assist and help or do better. Okay. And when a robot does that, it's, like because you'll you'll read from that in, I think, two parts of, like, when you you can have it assist, cool, that's fine, make some fancy words, maybe you don't have it to throw this around, that's cool. But, like, when you write a thing, it's easier for you to present a thing. If you have a thing written on or even if you're the one that did the investigation I see. Nerves get to everybody at some point. That's fascinating, though, because I'll bet your incident reports are very different because mine aren't too. Like, if my shit's under privilege, Mhmm. It's dry. It's very and that's where I was going with the If I go to present, like, I create stuff to present Mhmm. Specifically to bring the emotion into what people care about. But that's only ever spoken out loud, never written down. Exactly. Like, depending on the situation, it takes all the panel and you're presenting When to when you got those regulatory concerns and you got legal and lawyers and underprivileged We're just talking about a regular phishing email. Sure. Yeah. Yeah. That sounds like a like, cool. Like, if that when if I'm having to sit in front of a team and explain, hey. Here's why we took down your services. Here's what happened and what your team did with these breakdown and processes and failures that led to this incident, I want you to care. Right? I don't want to come in as the bad guys. Mhmm. I want to come in as, like, hey. We're just trying to help. And I have found that and this might also just be the theater kit coming out in me is, like, I I try to drum into You have a mic on. On oh, I do. Yeah. I try to drum into the emotion. I try to and not everybody asked to. This is why it's, like, it's my personal opinion on that. I think if you wanna be successful, you almost do that. Like, at this point, like, it doesn't have to be in your report, but, like, we're a long way. Like, it's so fascinating. I I used to work at Yahoo. And, like, Yahoo is in literally in textbooks for the biggest reason. Bob Lord? Yes. Big friend of mine. I love it. Hey. Alright. Bob. Shout out, Bob. Hi, Bob. But, That's that's not Bob in accounting. It's not No. No. No. Bob in accounting, we like you too, but Bob Lord is he's the Bob Lord. But, like, biggest breach in history in textbooks of all that. Now there's 15 breaches a day. Mhmm. Like, the reality of the matter is companies care less. Mhmm. The risk model has changed. Like, they have insurance, and they accept that it's going to be a reality to a certain extent. Yeah. So It's not the same headline event. To prey on emotions to make them care about security. So so just be in a room and rely on regular social media. So so this is a really interesting point. You're talking about that that so so we talked about report writing. We're talking about the human connection and communication. Are you seeing, whether it's with your team or or anywhere else in the business, you know, when something is purely AI written, like, you can you can see it. Like, you Oh. And so I really can't. Do you do is the business even if the content is accurate Mhmm. If it has an AI cadence, are you seeing a similar business reaction that we're seeing societally, at least here in The US right now, where there's very much a, you know, fuck that noise sort of reaction, even if the content's accurate. Yeah. I'm finding that, like and I y'all can let me know if y'all are the same, but, like, the second I see a report that comes across my desk that's AI written, unchecked out. Mhmm. Like, the second it's just like firsthand firsthand. Yeah. You you get a report that's written by AI, and I'm just like, what do we just give me the the, like, the one sentence that matters out of this entire thing. Dang. Because I don't wanna read the Exactly. Because that's the that's the hit, though, man. Mhmm. Depending on who you're talking to. Right? Like, yeah, you have to have the 70 page regulatory report. Sure. But, like, you don't go to the CEO with that shit. Right. I go to the CEO with a one paragraph summary and let him ask a question if he has it. Exactly. Like So but but but in your world, that's gonna be human written? Or are you worried could it be AI written? It'll be the same combination of both. Like, I have like, I personally have a very casual prose when I write and speak, if nobody knows. You don't say. I know. Right? So when I write, I have a very casual prose as well. So I will often, like, write it myself because I know what I want to say. Mhmm. And then I will, like, pass it into a model. You'll dress it up. Hey. Like, make this sound, like, a little less casual. Not, like, too much. Like, not flash. So you're not giving it free form. God. No. Okay. So I still want it to sound like me. Yeah. Right. At least I like heavier tone. Yeah. Right. Yeah. Do you do you get that reaction, Lewis? It's like purely AI generated prose and sort of a taken aback. Yeah. Like, to his point, this all depends on the actual context of what I'm having sent to me and where it needs to go afterwards. If it's just between me and the team, yeah, I don't want something that's AI written. Right? So just give me what I need to go, we'll go on from there. But if it's going to the CISO, I'll have the AI take a look at it, make sure I don't I haven't even spelling words or using slang, things of that nature. Yeah. Right? And then if it's going outside of the CISO to the exec ELT, then there'll be some other components in it that are are hybrid. But between me and the team, we're just talking. Alright. Alright. Cool. So so we had, you know, really good sort of kinda AI in the context of cyber, your jobs, your environment, whatever. So let's let's zoom out and pretend we're qualified to have this conversation. Think about AI broadly. Are you AI skeptical? Are you an AI doomer? Are you AI positive? You know, do you think it's gonna kill all the jobs? Do you think it will create more jobs? Do you think it is actually, you know, AGI in the context of the where that term's fuck. I can't believe it. I said, I have to now I have to do a shot. I said, AGI. Confessation. But let let's let's define it for purposes of this conversation. AGI, like, 80% of the time, it can do a better job in knowledge work than a human being. Let's use that as sort of the definition that that's coming, that that's probably not coming. Like, how do you where in the world societally do you land when I say, hey. It's the AIs. Like, how much is there an eye roll, and how much is there looking forward? Let's we'll just we'll go this way this time. So and then I'll take your call. Yeah. So, I mean, with most AI conversations, I I tend to start off with just a very broad, like, if AI doesn't have a hater, I'm dead. Like, I will hate on AI to the end of the day. Okay. But pragmatically, it's here. You have to learn how to use it. We have to accept that it's here. And I think that we are just in that bubble of what is it good for? Nobody really knows. Throw it against the wall everywhere to see what sticks. Okay. And I think that, like, in a couple years, it's just gonna be another tool. Like, all of the things that it's doing, people were doing with low code automation, you're just now doing a little bit faster. Like, you've got people that have a little bit more ability to do things. Like, I don't think it's going to be everything that everybody's saying it's gonna be. I think that it's not healthy for the public to have access to. Oh, interesting. I feel like it is removing people's ability to critically think, and I think that people are becoming too dependent on it as a whole as, again, everybody's trying to see where it fits in. I mean, is something that constantly tells you you're right and you're a genius all the time and actually good for your mental health? Yeah. Scatter of bias. Yeah. I don't I don't think it's good for anybody. Yeah. I must be using it wrong because it doesn't tell me that at all. It's like, you're a a dumbass, Mark. Like, that's usually I I need to know what models you're running. I could actually call. I'm guessing, my IT or security team swapped out my clogged throat harness for something else. But, okay. Alright. So some healthy skepticism Yeah. Some pragmatic acceptance. Yeah. Like, it's not going anywhere, but I don't think it's gonna be as prevalent in five years as it is today because, like, people are going to realize, like, what it's good at what it's good at and then, like, the cost. Like, right now, everything is still backed by VC funding. Once that's kinda gone and tokenization costs shoot up a thousand, you know, times, it's gonna become less available. People are gonna use it less. Like, to your point, like, budgets come in. Cool. We don't have budgets for tokens. Right. You built all these things with AI. Get rid of it. Mhmm. But, like, you still gotta pragmatically look at it. It's gonna be like any other tool in any other toolbox. And it's you gotta figure out where in the toolbox it sits, and I think that the public having access to it isn't necessarily healthy. Okay. Interesting. Louis, how about you? Yeah. I don't really consider myself a AI skeptic. I'm more, you know, result aware in that regard. So I just wanna make sure, like, when it's being used, it's used as a force multiplier for good positive results, not trash your results. Meaning, if you put the AI on top of bad telemetry, bad data, you're gonna get bad Outcomes. Outcomes. Right? So to his point, I feel like that's how most people are using it right now because it's the coolest thing. It's all in the news. Everybody is either scared of it or they might know how to use it. So know about everybody just running the middle of the line right now, and depending on the conversation, you can either lean one way. But just me personally, I just keep my sanity, and I know the business wants us to use it. I have to learn how to work with it, and I have to learn how to teach my team how to use it the right way. I want them to be prompt engineers into supervising AI versus just being consumers of it or dumbing down other processes that make sense. So got got it. So understand why a prompt works Yep. And understand how it does produce results Mhmm. And understand where it doesn't, meaning then stop using it. Exactly. Okay. Yep. Okay. Pragmatic. Okay. Sean, how about you? I have a lot of societal and environmental considerations that I'm gonna put on a table over here. You can not engage in there for a second. Maybe we like that. Come on, man. Just got it. I will I will start with ultimately, it is if we really cut down to what AI is, and I do feel like this needs to be said to to look at it objectively. Okay. It is the largest data theft that has probably ever happened in history. Okay. Like, every model is trained off the theft of all art, all work, all everything that it could get its hands on. And that needs to have a reckoning at some point in my personal opinion. Okay. That being said, it it's a really interesting place because it's here, and it's here now. But I do think, like, we're gonna see use taper off a lot. Like, I think the Internet is largely gonna be searched with AI in the future. That's not gonna go away. That's actually super fucking useful too, to be completely honest. Other than the fact that it still can't help me source niche stuff, it still works just like Google and, like, buries the good stuff that you're looking for. But I don't I think what you're trying to find, and they're still finding, is that, like, cost effectiveness isn't there. Like, there's not an AI company today that's possible. You're right. They're all backed by VCs, so shit's cheaper right now. But, like, they don't make money. They don't have a pathway to making real money today. And companies are already starting to roll back on the lower cost of tokens that they're paying paying now. Everybody's electricity is going up. Mhmm. Like, I think I don't know that, like, what we have envisioned for the future, what I would say, like, the AI futurists have envisioned is actually in any way, shape, or form, possibly. So I expect that, like, in the next five years, the technology will drastically change. I expect that they're gonna be working on making models that are much more efficient Mhmm. So that their token cost is cheaper and they take less resources. And that would likely The Chinese approach. Yeah. And I expect that that will likely come with trade offs as you make models more efficient and have less context. So, like, it's here, but it's so early, and it's gonna change so much. And I don't think any of us know what it's gonna look like when it comes out. I'll use it because I have to use it, but then I go back to, like, I have a ton of like, OpenAI said something like it cost them millions of dollars just people saying hi to chat CPT. Right. What does that cost on, like, electricity costs? I mean, just saying hi to ChatGPT. Right? I mean, you talked about, like, the access to it. Mhmm. Wait until this shit gets like, I think ChatGPT so far has started to add in, like, ads, like, in my hands. The data that you give is, like, at my company, our internal chatbot, not my security one, but, like, the company chatbot, is the most sensitive data source in the entire company in my opinion. Because people talk to chatbots like they talk to people. You you give personal details, medical details. They bare their deepest fucking soul Yeah. To a computer that can then shift off their shit and monetize it. Yeah. And I think that is a fascinating record, like, thing that, like, is also going to have to come to a head at some point. Yeah. I think that, like, to that that point with any of these internal chatbots that are connected to all knowledge bases, to your point, like, third party compromise with this Oh, yeah. We've seen it to where the knowledge base AI gets compromised. Does not matter what kind of RBAC you have. Mhmm. Doesn't matter what kind of controls. I can still talk to it and ask you questions about PII even if it doesn't have access to that PII. It now has that data. And if somebody is able to get in and query all of that, like, you you start to lose the ability to control data flow, And then you've got a single point of compromise where, like, whatever they want, they can have. How good of controlling your data do you have? Like, how much RBAC do you have on all of your Google Drive or any of your your data storage that, like, is going to make an attacker that gets into that that LOM, you know, not be able to get whatever they want. Yeah. Just to to close the thought, by the way, because I feel like it needs to be said, I'm not looking any way even an AI hater. Like, I I think the technology is cool as hell. Like, I think where it's gonna go is really cool. Like, I I think it has a lot of potential. Mhmm. But I also think, like, if we don't admit, same with the the red team thing I was talking about earlier, we have to be able to admit the bad sides of something. We can't just love shit blinders. Right. Right. Right. There has to be bad sides too. So Sure. Sure. Yeah. Yeah. Well, you know, I'm optimistic because we've done such a good job regulating bad behaviors out of other industries. I'm sure I'm sure this will be fine. Just gonna be a 100%. This is all good. With you. Yeah. Alright. Awesome. I, like, pick up my mic and throw it on the ground. You can. Yeah. We're gonna rage quit the the conversation. Alright. Well well well, guys, firstly, thank you all for taking the time to join me here this morning. A fun conversation, and I think we have time later for beers, actually, that are in the discussion. Say we should just go get those beers. Yeah. I was gonna say, like, I got that. I mean, my patio is a mile up the street. Like, if you guys wanna go, like, I'm totally there. Like like I said, I'm Irish. I'm allowed. Fantastic. So, thank you all for sharing those those viewpoints. Really interesting dialogue. That's gonna wrap it for, for today. We could do this for hours and hours on end. So, Sean, Louis, thank you again for joining me today. Really enjoyed the conversation. Thank you, out there, for joining us. Hope you found it useful and interesting, and we will catch you again next time. Cheers.