Nicholas King’s Post

Now everyone's an expert on deployments, believing that what happened to CrowdStrike will never happen to them. How many of you program in C/C++? I programmed a long time ago, and if I saw low-level code now, I probably wouldn't understand it well. Moreover, in the low-level world, there are too many variables; it's not something that can be solved just with testing or canary releases, etc. Sometimes it's simply the alignment of various factors that create the perfect storm. So let's avoid judging and try more to learn or understand that no one is immune.

The aftermath of the CrowdStrike incident is still being felt. Enormous amount of time, energy, and money have been wasted. Yet, the world still remains the same, but with many learnings. As the Software Engineering Community, this is our opportunity to take away those lessons and raise the bar to build better quality Software for a world that deserves nothing but the best from us. #CrowdStrike #ContinuosLearning #SoftwareEngineering #SoftwareQuality

This weekend's outage was crazy because of how preventable it was. Feature flags can stop things like this from happening in a couple of really simple ways. 1. Kill switches. Put the feature behind a feature flag that lets your product team quickly shut off a feature when you notice an issue. 2. Canary releases. This is what we’d recommend in situations like the CrowdStrike one. Instead of rolling out the feature to all Windows machines, do a progressive delivery that targets a percentage of users. Connect the feature flag data with observability tools to see how it’s performing in production before deciding if you increase the percentage or not. If something does go wrong, tools like Dynatrace and Grafana Labs can show which machines are impacted (Dynatrace have written about how to do this) With feature flags, stuff like this becomes really simple. There’s lots of talk about this right now. If you have questions DM me or anyone at Flagsmith #CrowdStrike #featureflags #observability #canaryrelease

💥 The CrowdStrike disaster was more than just a glitch—it was a failure of software engineering! In this week's update on the Continuous Delivery channel, sponsored by Equal Experts, Dave Farley digs deep into what went wrong with CrowdStrike's system and why it was an easily predictable failure. He breaks down the development process that led to this disaster and what could have been done to prevent it. Dave says: "This isn't just a "bad things happen" moment—it's a wake-up call for the industry!" Watch now 🎬 https://hubs.li/Q02KM_q30 #CrowdStrike #SoftwareEngineering #ContinuousDelivery

How do we know for sure that CrowdStrike wasn't the work of a malicious actor? I mean, a compromised GitHub/Gitlab/whatever access token could allow you to push a bug to the code and run the YOLO update... It SHOULDN'T allow you to do that. YOLO updates shouldn't be possible, like Victor Moreno said. And it shouldn't be possible to make them possible. And of course pushing a bug this obvious shouldn't be possible. Even if it survives code reviews (which is not that hard), there should be some internal testing before doing a release. Just a smoke test would catch this! But what sounds more plaussible? - A company this big doing every single thing wrong - A company this big doing things right by default, but there are ways to bypass that (which is wrong)

What are actions software engineers could take, off the back of the CrowdStrike outage? A simple, yet powerful one: quantify what would happen if your company/product crashed or went offline. This exercise is helpful because it can give a sense of how expensive an outage could be. Knowing the “blast radius” can help get buy-in to make systems more resilient, and make it quicker to detect and mitigate incidents. More actions and learnings you could take in today's free issue analyzing the CrowdStrike outage, from a software engineer's perspective. Read it here: https://lnkd.in/eYkfhFRb

The huge outage today shows us the importance of technology in our daily lives and the significant impact it could cause by deploying a bug in production. As software engineers, we sometimes don’t realize the responsibility we have and the impact our work can make on the company we work for and, more importantly, on our customers. It’s common to see some developers joking with the CrowdStrike team, but mistakes like these are quite common in our applications. We need to be more cautious when deciding to deploy code to production and adopt strategies like tests, gradual rollouts, a structured rollback plan, and a postmortem process to learn from incidents. Note: It seems that today’s outage happened because the code attempted to access a null pointer without validating if the pointer was null.”

Yes, it's unfortunate with what took place with Crowdstrike and the global impact it has created. But, a lot of folks are quick to shift entire blame on them and/or Microsoft for not properly doing testing, QA, etc. What ever happened to organizations doing proper patch(es) / updates testing in test / dev / qa environments, validating it all works and then pushing it out to production environments? Are there no more protocols in place from compliance, process, etc.? There has to be accountability and ownership there as well. Just like when there is a major organizational DB, capability, app change, etc., everything is done in stages and over time to make sure that everything works accordingly and mitigate business outages and overall financial impact that comes with it, not too mention the additional burden, stress, sleepless nights within the IT staff, dev/app/QA teams and any other stakeholders that are impacted and need to retest it all to make sure its properly up and running. Wishing the best to all the hardworking sys admins / engineers and thank you for all your dedication and support!! Hope you get some rest and time with family this weekend, and going into next week.

The CrowdStrike incident; a video explaining the course of events. Who CrowdStrike are, what happened, why this has happened and why it is so difficult to fix for those who need an easy to understand TL;DR. https://lnkd.in/e44ZdX72

Looks like CrowdStrike recently ran into a major problem due to a null pointer dereference in C++. This caused an access violation, crashing their system driver. Basically, someone missed checking if an object was valid before accessing its member variables, leading to the use of an invalid memory address (0x9c). This incident is a stark reminder of the importance of code sanitization and thorough testing before releasing software. These are fundamental practices that, if overlooked, can have serious What are your thoughts? How do you handle such issues in your development process? #SoftwareDevelopment #CrowdStrike #BestPractices