Do you have something cool to share? Some questions? Let us know:
Timothy: Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the product manager for threat detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and esteemed member of the cloud security team here at Google. You can find and subscribe to this podcast wherever you get your podcasts as well as at our website cloud.withgoogle.com/cloud security/podcast. If you like our content and want it delivered to you piping hot every Monday afternoon Pacific Time, please do hit that subscribe button. You can follow the show and argue with your hosts on Twitter as well. Twitter.com/cloudsecpodcast. Anton, we're talking about SOCs today
Anton: Yes, this episode was very dear to my heart.
Timothy: Why is that?
Anton: Because it touches on the work we've been doing together with the Deloitte team. The team we'll interview on the call today for actually more than a year.
Timothy: Wow.
Anton: And we are writing a series of well white papers or reports or sort of research papers that talk about how SOC is changing and hopefully giving some advice on people who are trying to build a SOC. You know, circa 2022 SOC, not circa 2002 SOC in 2022. We're trying to figure out what's the up-to-date SOC wisdom on the process* side, on the skill side, on the technology side as well. So this is fun and this is very--it's a very tricky area of security because unlike say encryption or, you know, endpoint security SOC security operations is very tied to humans.
Timothy: Yes.
Anton: Pesky humans. They're always the problem, right?
Timothy: Pesky humans. Yes. Yeah, it's a really interesting area. You're right. I hadn't thought about that particular aspect, but we get into that in the show don't we today?
Anton: We do indeed and even when the organize the papers we organize them as kind of like a process paper, people paper, and it turns out the author team, including myself kind of had passions about some things. For example, people really said, "Oh, my God process papers will be boring." And I thought, "No, come on. This is like the--where the magic happens." Is--is kind of between the organizing the processes so you're predictable, consistent yet allow for creativity. This is really tough and it's fun.
Timothy: Yeah, that's an interesting challenge. So with that listeners, I am so delighted to introduce today's guests. We are joined today by Alexi Wiemer, a senior manager at Deloitte Cyber Detection and Response Practice. And Dan Lauritzen, a senior manager also at Deloitte, but differently at the cloud security practice. So we have both cloud and cyber on the cloud security podcast today. So we wanna talk about something which is both cloud and cyber. We wanna talk about where's the SOC? What is going on? And so Dan, Alexi, you know, what's the state of SOC today? What trends in SOC can we talk about? What's going on?
Dan: Yeah, thanks for inviting us and having us on first and foremost and I feel like I'm gonna be struggling to get on your level of professional voice talent here, but I will do my--I'll do my best. Yeah, so broadening it out beyond just cloud-specific topics but rather for SOCs today. We can talk about a lot of the common themes you'll hear from SOC practitioners about overload of resources, you know the job is growing, the attack surface is growing, and more things to look at with fewer people and fewer--you know, fewer resources to do the job. But I think one of the common themes also that Alexi and I will kind of hit on is driving efficiency through tool use, through process, through automation, and really kind of, like, wading through those waters of being overwhelmed to better direct those finite resources.
Timothy: Mm-hmm. We love the theme of tool use and automation on this show. So that'll be interesting to get into. Alexi, you look like you're gonna give an answer there as well.
Alexi: Yeah, absolutely. I think just to expand on Dan's points, right? That is the name of the game today. And so technology is able to ingest this at such a large scale, but what we're noticing is even though you have the capacity to use this data it's actually properly using it with the appropriate processes is where we start to see a large disconnect, right? So yes technology is an enabler of these meaningful connections. However, you're dependent on a human analyst to make those important events and trends and connections with that context that may not be readily available in the data. And so I think that's a theme that we're seeing is emphasizing that type of information sharing across your skillset in a cohesive manner.
Anton: And by the way, Alexi, when you say data you clearly are quoting from our common Deloitte Google Cloud paper where we highlight that what matters for SOC is kind of the fidelity, the quality, the scalability of data. It may not be the brand of a tool that matters, but how you handle the quality of data. Is it conclusive? Is it clear? That type of stuff, right? Am I wrong about this or not?
Alexi: Yeah, absolutely. And I think, you know, we're seeing the need to ingest this more data because of this expanding attack surface. Organizations are noticing that threat actors are more sophisticated. The borderless enterprise is expanding, right? So obviously cloud monitoring has become a huge point of emphasis and how do we readily ingest that information and then bring it back to specific needs of the business units, right? 'Cause at the end of the day that's what the SOC needs to protect is the business delivering value to the different stakeholders.
Anton: I mean, we all know that yet when we talk to clients, when you talk to clients, and we talk to clients we do see SOCs security operation centers that are kind of permanently and woefully understaffed. During my analyst days, I've encountered people who were thinking of a 24/7 SOC when they had two analysts. And I said, "What are you talking about? Like, it's not even close to remotely conceivable." Like, you're talking 9,10 people however you count to do a 24/7 SOC. So what is our advice? What is your best advice for SOCs that are kind of permanently and woefully understaffed? Of course, you can say I outsource to Deloitte and we wouldn't hold it against you. We kind of would but okay.
Alexi: That's number one.
Anton: So that's number one. Okay, easy.
Timothy: Okay, what's number two?
Dan: I would say that my own personal take on that question is that there's an immediate knee-jerk reaction when staff is overwhelmed to reach for the staffing lever and, you know, make that claim that I need more people. And maybe part of the guidance would be, you know, take a breath, take a step back, pause for a second and understand the inputs of that being overwhelmed, right? So it's--it's highly likely and, you know, entirely possible that in today's security operation center kind of [inaudible] that there's going to be way too many projects for your team to cover, way too much attack surface, way too many alerts. It's totally likely. But within that whole crush of things to do there might also be improvements to the process you can make. Maybe there's unnecessary things your team is chasing down. Maybe you have alert fatigue from some things that are misconfigured. So before you just immediately reach and say I need more people. There's probably things you can improve or that you can make more efficient and I realize that that's not an easy thing to do, right? Everybody's got 900 tasks so kind of looking at the process is just one more of those tasks. But it might be something that's worth investing some time in because whereas you thought you may be needed 12 people maybe you really truly only need five once you kind of get more efficient in what you're doing.
Alexi: Yeah, and just to expand on Dan's point a little bit too I think one of the ways that we see clients handle these issues, right? Is you need to be realistic about your budget and the size that your team will have, right? And from there being really strategic about what capabilities you wanna focus on, right? If I know I'm only gonna ever have a three-person team, I'm never gonna be able to do 24/7/365 monitoring and I need to outsource that. What I can do though is develop my own threat intel team. My own threat hunting team. My own incident response, right? And based on what is important to your organization and the skillsets that you have, you can start to develop a plan of this is what we bring in-house. This is where we need to maybe lean on other third-party providers to do that. So--'cause outsourcing some of that is more cost-efficient I would say from also a just to address the permanently understaffed, right? We've also noticed developing a career model. Giving people a trajectory. Training opportunities. And also as we've seen with this shift in the last two years of more flexibility, right? People don't wanna have to come onsite five days a week every week, right? Can some of this be achieved remotely? And I think a lot of these trends are also helping retain analyst* when they add a position that typically experiences a lot of turn.
Anton: Being strategic is obviously good advice always, but hopefully we'll address it in slightly more practical terms because if the team has three people prioritization decisions become really tough frankly. And I'm gonna argue with the next a little bit. I probably won't invest in hunting if I have a team of three.
Timothy: Me neither.
Anton: But if I have a reliable outsourcing partner who handles the job, then it's a maybe. So to me this is kind of like if I trust my MSSMDR maybe I will hunt.
Dan: Yes, and to--to build on that point, right? Like, hunting is a very sophisticated, advanced capability. So if I have a more junior staff that is not the capability I wanna bring in-house and depend on, right? They may be start as tier twos or dabble in incident response, right? So I think it's just as important knowing the skillsets of your staff and the talent you can attract in relation to that strategic vision 'cause I think that's [inaudible], Anton.
Alexi: Well, and if there's one other thing that I can add into that point to strengthen it is that prioritization is key because even if you found a partner that performs something like threat hunting to the extent that you desire and, you know, has solid capabilities without exception I can say there's always going to be a component of collaboration and of insider knowledge needed. So even if you outsource certain components of your SOC operations there's still work needed on your team. So if it truly is not something that you can support a couple of hours a week at the bare minimum collaboration with your MSSP or MXDR provider, then it may not be something you wanna prioritize.
Timothy: That makes a ton of sense. I think the advice boils down to a bit of if you haven't got somebody who can run the ball don't try running the ball. Which is, you know, this is good advice. I wanna--I wanna scroll back up the conversation a little bit and touch on the automation piece because this comes up a lot. It's an interesting thing. We---we have simplify now here. We hear a lot of people say well you need to automate if you're drowning in manual work. What does that actually mean? How do people get started with that? What does that actually look like? I assume there's not welding machines in the SOC.
Dan: Well, maybe not in your SOC there's no welding.
Timothy: Maybe my SOC is doing it wrong.
Dan: [inaudible] I think one thing that gets lost in my opinion and Alexi I'd love to hear your thoughts. But one thing that gets lost in a lot of the client conversations that I have is the jumping to black-box magic conclusions about what automation can do. You need to start with an understanding of what it is you're trying to automate. You need to understand the process. You need to be able to execute the process well manually before you automate it. You know, automation isn't gonna be an end unto itself. So I think that lack of starting from a good solid place can lead you to undesired outcomes. I'll just put it that way.
Alexi: Yeah, I think what you wanna do, right? Is focus on as Dan mentioned the outcomes you're trying to achieve and work your way back from that, right? 'Cause if you have the automation driving your outcomes this is where you start to have rabbit holes and it doesn't impact the organization the way that you want, right? So if my goal is I'm trying to decrease mean time to the text that's gonna be a very different automation than mean time to respond, right? Or things of that nature and it depends on where along the process that is. And so understanding your processes well enough and then also what you're trying to achieve is really key because there's also things that are simpler to initially automate and then there are things that are much more consequential. Automatically pulling in threat intelligence feeds is really helpful and that's a very low-hanging fruit compared to automatically quarantining a machine of what could be a very informed user. And you need to do that crawl-walk-run model to make sure that you understand what you're automating and what risk you are accepting by automating certain things.
Dan: Yeah, and one other thing to add on to that which I-I wholeheartedly agree with everything Alexi said. But you also must understand that when you're approaching automation it can be a skillset unto itself. So you're likely going to partner with somebody, some technology that's gonna be the enabler of that automation process, but you will also likely run into some cases where you're stretching the tool to its fullest extent or need to tweak it just slightly. So do you have Python skills in-house to build automation playbook? And if you don't you kind of need to think about that upskilling or partnership to get those skills to get everything you want out of it.
Anton: And this one is not gonna be negotiable in this--in the foreseeable future. We can talk detection as code. We can talk playbooks. We can talk--even if we can talk no code low code platforms that ultimately I'm sorry this is my sincere belief will push you to code at some point. The--at least in this area. If it's not the playbook code, it's gonna be integration code.
Dan: But it's in the name. It says no code. What are you talking about?
Anton: No way, right? Okay, fine. Switching subjects to something more exciting. Well, actually that was exciting. So the next question is a trick question and I know that it's just designed as a trick question. So lots of technologies and again in our papers we covered the use of--use of SIEM. We talked about threat intel. We talked about EDR. Of course, we are talking SOAR. So if somebody pushes you in a corner and says, "Dan, Alexi, name one technology that's the most critical for your SOC." Can you do that? Would you name SIEM? Would your name SOAR? Would you name EDR? Will you chicken out and say I'm not gonna name. Five seconds. I don't know. Maybe not five seconds.
Dan: Tururu. Alexi, why don't you jump in those waters?
Alexi: Yeah, I can take first pass at that. So I think for me Anton's held me captive and said hey you better answer this question or we're no longer coauthors together. What I would say is for me it's that data pipeline that everything is upstream of your tools, right?
So obviously I'm partial to [inaudible] technology and then I think that Google Cloud and Chronicle does this especially well. But having the mechanism to ingest all of this raw data at scale, normalize it, parse it, be able to replay it back in an investigation, and have that be in a common model that is agnostic of your tool stack downstream of view, I think is really instrumental. Because then it gives you the flexibility as vendors come in and out of your environment your data is still your own. It's still in a way that's meaningful and actionable. And I think that's where we're starting to see organizations realize of wanting to break out of this vendor lock-in and having some of that flexibility. So it's a little bit of an awkward answer 'cause it's upstream of the examples that you mentioned, but I think it's a really important foundational piece because of the flexibility that it offers.
Anton: But you're right because I didn't ask the question name one tool that's sufficient for a SOC. That would be dumb. There's no such thing. I said, what's the most critical tool and you answered it's the pipeline leading into other tools. That's fair. I think we have a winner. I may or may not agree with that, but I see the merit behind the answer.
Dan: Yeah, and I-I think I would essentially say the same thing. Just move it further downstream.
Anton: Upstream or downstream?
Dan: Downstream.
Anton: Which way?
Dan: You, me--you, me, and Alexi have had this conversation relative to the paper that we're writing right now, but, you know, my--my interpretation is that you definitely need to have common--you need to have a common resting place for security, telemetry, and security alerting things in such a way that it's accessible for interpretation analysis. You know, hunting. Anything downstream past security tools telemetry so that you can then drive further observations and/or develop, you know, more machine learning-based or, you know, more broad-based observations off of the whole corpus of data, right? So a data lake essentially that's in a--in a common format. So it's not probably one technology. It's probably a couple of different technologies that serve that purpose. I think that's probably the most important technical function in my opinion of the SOC today is taking all this disparate technology and all of the logs that come from it, gathering, collecting, and storing in an accessible and--and common format.
Timothy: Well, we've got data pipeline and data lake as answers to that. And Alexi, you should go home feeling happy about this episode because Anton said you were convincing. Maybe didn't say you were right, but he said it was a good answer and that's you know--that was high praise out of him. I wanna--I wanna shift gears a little bit and give you one of my favorite hypotheticals which is imagine you are a SOC and you have been handed the keys to a cloud. What do you do? You know, imagine you've never seen this cloud before. And for bonus points, tell us what would be different in your answer if it was say an Azure Cloud or an AWS Cloud versus a Google Cloud, and extra bonus points if your answer is more different than just the names of the menus you click on.
Anton: But before--before you answer just a very quick interruption I'm sorry. Like, we do see clients and usually, it's a mainstream pre-cloud client. I don't know, like a proverbial--in [inaudible] a bank in Kansas.
Timothy: Hmm.
Anton: Somebody who didn't do cloud before and suddenly they build a SOC, SOC is working out okay, it's on the resource but ultimately it's okay. And suddenly it's like call immigration. So they're real people in this situation. So Tim's not making this up.
Timothy: No.
Anton: There are companies--there are teams and kind of pre-cloud companies who SOCs now have to do GCP, have to do Azure, and maybe they have to do Amazon. I don't know about this one. But how would you approach it?
Timothy: And I talk to these teams every week. Like, these are real people and I-I feel for them.
Dan: Yeah, I--you know, you asked what would you do first? I think that question naturally leads to maybe a--a one or two finite list of tasks. I think it's probably a combined list of multiple tasks for a common purpose and I think that that's—
Timothy: Mm-hmm.
Dan: Grasping the basics of that cloud you've been--you've been handed, right? So configuration standardization. You know, checking the traps of that cloud deployment which are the things that commonly get us--get us on in this day and age, right? So misconfigured buckets of storage, misconfigured permissions, understanding what you have in the environment that is, you know, that's SAAS that you have very limited control over versus things that you have more control over. Just kind of good [inaudible]. Like, really documenting it, understanding it, and it's a little bit of a challenge for SOC operators because you likely don't own any of this, right? Like, you'll--you'll almost guarantee to have to work with other members of the organization that--that own these cloud environments and they can really answer some of these questions. But you can't possibly hope to secure it if you don't even understand what you've--you know, what you've been given. What you've been handed. So that's definitely where I would--you know, if I was CSO for a day and I was handed a cloud that's definitely what I would recommend we do with our time.
Alexi: Yeah, and I think I'm just a very quickly expand on that for Dan, right? Like that governance model of what is my team responsible for what is being outsourced to the provider is absolutely critical 'cause every other group thinks the other one's holding the ball then you have no luck, right? And I think my first also question back to you is what is this hot environment supporting, right? Is it a business-critical application? Is there sensitive data 'cause not all cloud environments are the same. Now, obviously, we've seen a lot of examples of people exploiting cloud environments and moving laterally to higher-value assets, but if I'm starting from scratch day one it goes back to my crown jewels. It goes back to the risks that I'm willing to tolerate as we get that type of visibility into our SOC.
Dan: I think just as a closing point on that. Our answers might have sounded a little bit generic or a little mother had an apple pie, but, you know, I think a lot of SOC operators might naturally their knee-jerk reaction might be what telemetry can I get? What use cases can I hold?
Timothy: Mm-hmm.
Dan: You know, let me just research some threats to this particular type of cloud deployment. And that's definitely a necessary step, but if you don't even understand what are the native permissions that are available in that cloud deployment, right? Are they more primitive or are they more nuanced? Are they applied appropriately? Where's the data actually flowing? You know, where are my VPC boundaries? Like, that's the kind of stuff that, you know, if you don't have a good solid grasp of that, then anything you build as far as the detection will--will slightly fall--fall flat. So it really is kind of critical first step.
Anton: And we do see people who do exactly what they just said. They jump at collection and they say I need my car* trail in my SIEM and that's all they have in a year. But like how is this helping their SOC? It doesn't. Okay, fine. I rant too much. So let me--let me switch gears.
Timothy: No, I think that's a really good point though, Anton before you switch gears. I think that's critical for people to understand. It does you no good just to gobble up the logs if you don't know what meal you wanna cook with them or you don't know who you're trying to feed or what their dietary restrictions are. You gotta think things through before you dive into it and that's I think actually a really, really good answer.
Dan: Yeah, throughout my career I've always said that in the SOC there are generally two classifications of people when it comes to detection systems. There's kitchen sink people and their scalpel people, right? And you wanna be a scalpel person. You wanna know what you're doing with that data you're collecting.
Timothy: Yup.
Anton: Okay, so now we're kind of getting to the end of the--of the recording. I wanted to ask a possibly a slightly dire question.
Timothy: Oh, no.
Anton: Occasionally, you read about predictions that SOC is dead. And typically in my mind when I read the paper that says SOC is dead or a blog, they kind of mean that the physical room full of analysts watching the screens and like talking over, you know, the cubicle walls is dead. Like, nobody or very few people ran SOCs like this during COVID. That's pretty clear. So what's your reaction to SOC is dead? Like, how do you respond to that or what's your response to predictions of we don't need to solve because something else would help?
Alexi: Yeah, I mean--I think, you know, when people as you mentioned, right? Imagine the SOC of yesteryear, right? It's a bunch of people staring at these screens, maybe picking up the phone, and I think we're seeing that go away, right? It's much more dynamic and collaborative. There's not a phone pickup to call, right? We have these automated alerts that come enriched with this data and we are breaking these silos where we're able to work with these different groups and it's not just hey close the ticket as quickly as you can, right? We're trying to get as much contextual information as possible and being--this is a bit of a cliche, right? But being much more proactive as opposed to reactive. And I think that is where we are seeing technology enable folks to do that and organizations really setting up those processes whether it is through a common ticketing system or stand-ups or just getting to that point where we're getting the right people in the room to make the appropriate decisions.
Dan: Yeah, I would agree that if--as long as we're defining it as a capability and a set of people with--with common goals then no I don't think the SOC is dead as far as an aggregated and distilled group of people in front of a bunch of like NASA screens. Yeah, I don't know if that's necessarily [inaudible] anymore. But yeah as long as it's those--as long as it's a core set of capabilities you're chasing because there's always gonna be no matter how much you push dev sec ops and no matter how much you push, you know, blurring of boundaries of driving security into other portions of the organization. There's always gonna be someone who has to care about edge cases and--and really advanced threat actor behaviors and that just will not be the developers and will not be application developers. It never will be, right? So there's always gonna be a group of people that are more security-minded and then have to kind of drive the edge cases no matter where you push the boundaries.
Timothy: I-I firmly believe that. I think you're absolutely right about that. So, Dan, Alexi, we're--we're reaching the end and I get to ask our traditional closing questions. And you know, the first one I think is a bit of a giveaway because now we've mentioned twice this paper. At least twice. I don't know if I was keeping accurate count. What's your piece of recommended reading and feel free to plug the paper here. That's a great idea. And then the second piece of our closing question is what's your one bit of advice for doing SOC.better?
Dan: Yeah, definitely. I mean, first and foremost like you already mentioned I would go back and review the--the three previously released white papers that we've developed with--with Anton in bated breath waiting in anticipation for number four to be released. And in the not too distant future, that's great reading. Also, I would make a general plug as opposed to any particular book. I've been reading several lately about social media hacking and manipulation related to anything that's elections or just certain sentiment. Public sentiment. I think those are very interesting books to read. Yeah, so then in terms of advice for how the SOC can do things differently. It's an--it's an old one but it's a good one and it's don't chase shiny cars. Like, try to make the most use out of what you already have. Tools have gotten--so we--we've talked about this before. I'm gonna sound like a broken record to Alexi and Anton. But tools have gotten very strong and very capable and most people that are operating at an enterprise level have a lot of excellent tooling at their disposal. You know, really try to wring every bit of value that you can out of it 'cause number one, you know, you likely gonna make your life easier. And number two, you're gonna make your leadership very happy if you're not asking for additional budget if you're stretching what you can do with your existing tooling.
Timothy: Got it. So good advice on making leadership happy by not asking for more budget.
Alexi, what have you got for us? Recommended reading and one weird tip.
Anton: Let me guess. The papers?
Alexi: Yeah, so one I-I really liked our last process paper on the balance between having defined processes but also enabling creativity and I think that also ties into my advice that really resonates with me, right? Is--so my one advice for the SOC would be empower your people and really take the time to invest in them. As Dan mentioned we sometimes see organizations chase the shiny car so to speak and say hey it doesn't matter who's in these seats as long as we have these tools and I think it can never be further from the truth, right? Organizations spend a lot of time trying to just retain and develop talent and I think by placing incentives, getting people bought into the organizational mission makes a huge difference once we're trying to frankly counter these really sophisticated threat actors. They don't discriminate. They don't care that this group typically doesn't talk to that group, right? They're just gonna attack your organization wholesale. And so you need smart people doing smart things that are empowered to do so within obviously the context of your organization and I think that would be the [inaudible].
Timothy: Got it. I think that's great advice. Alexi, thank you so much for joining us today. Listeners, the papers mentioned will of course be available in the show notes, we will tweet them, and they will also be available at the website cloud.withgoogle.com/cloudsecurity/podcast.
Anton: And now we are at time. Thank you very much for listening and of course for subscribing. You can find this podcast at Google Podcast, Apple Podcast, Spotify, or wherever else you get your podcasts. Also, you can find us at our website cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter. Twitter.com/cloudsecpodcast. Your hosts are also on Twitter @anton_chuvakin and _timpeacock. Tweet at us, email us, argue with us. And if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.
