…according to a Twitter post by the Chief Informational Security Officer of Grand Canyon Education.

So, does anyone else find it odd that the file that caused everything CrowdStrike to freak out, C-00000291-
00000000-00000032.sys was 42KB of blank/null values, while the replacement file C-00000291-00000000-
00000.033.sys was 35KB and looked like a normal, if not obfuscated sys/.conf file?

Also, apparently CrowdStrike had at least 5 hours to work on the problem between the time it was discovered and the time it was fixed.

  • Pasta Dental@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    397
    arrow-down
    1
    ·
    edit-2
    5 months ago

    Every affected company should be extremely thankful that this was an accidental bug, because if crowdstrike gets hacked, it means the bad actors could basically ransom I don’t know how many millions of computers overnight

    Not to mention that crowdstrike will now be a massive target from hackers trying to do exactly this

    • qprimed@lemmy.ml
      link
      fedilink
      English
      arrow-up
      84
      arrow-down
      3
      ·
      5 months ago

      security as a service is about to cost the world a pretty penny.

      • Telorand@reddthat.com
        link
        fedilink
        English
        arrow-up
        90
        arrow-down
        3
        ·
        5 months ago

        You mean it’s going to cost corporations a pretty penny. Which means they’ll pass those “costs of operation” on to the rest of us. Fuck.

        • qprimed@lemmy.ml
          link
          fedilink
          English
          arrow-up
          32
          ·
          edit-2
          5 months ago

          well, the world does include the rest of us.

          and its not just opeerational costs. what happens when an outage lasts 3+ days and affects all communication and travel? thats another massive shock to the system.

          they come faster and faster.

        • zbyte64@awful.systems
          link
          fedilink
          English
          arrow-up
          7
          ·
          5 months ago

          You did not just fall out of a coconut tree. You exist in a context of all that came before you.

      • littlewonder@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        5 months ago

        All the more reason for companies to ignore security until they’re affected personally. The companies I’ve worked for barely ever invested in future cost-savings.

    • Echo Dot@feddit.uk
      link
      fedilink
      English
      arrow-up
      38
      arrow-down
      2
      ·
      5 months ago

      On Monday I will once again be raising the point of not automatically updating software. Just because it’s being updated does not mean it’s better and does not mean we should be running it on production servers.

      Of course they won’t listen to me but at least it’s been brought up.

      • expr@programming.dev
        link
        fedilink
        English
        arrow-up
        15
        arrow-down
        1
        ·
        5 months ago

        Thank God someone else said it. I was constantly in an existential battle with IT at my last job when they were constantly forcing updates, many of which did actually break systems we rely on because Apple loves introducing breaking changes in OS updates (like completely fucking up how dynamic libraries work).

        Updates should be vetted. It’s a pain in the ass to do because companies never provide an easy way to rollback, but this really should be standard practice.

        • Echo Dot@feddit.uk
          link
          fedilink
          English
          arrow-up
          1
          ·
          5 months ago

          You can use AirWatch to deal with Apple devices. Although it is a clunky program it does at least give you the ability to roll things back.

      • ShieldGengar@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        13
        ·
        edit-2
        5 months ago

        I thought it was a security definition download; as in, there’s nothing short of not connecting to the Internet that you can do about it.

        • Echo Dot@feddit.uk
          link
          fedilink
          English
          arrow-up
          10
          arrow-down
          4
          ·
          5 months ago

          Well I haven’t looked into it for this piece of software but essentially you can prevent automatic updates from applying to the network. Usually because the network is behind a firewall that you can use to block the update until you decide that you like it.

          Also a lot of companies recognize that businesses like to check updates and so have more streamlined ways of doing it. For instance Apple have a whole dedicated update system for iOS devices that only businesses have access to where you can decide you don’t want the latest iOS and it’s easy you just don’t enable it and it doesn’t happen.

          Regardless of the method, what should happen is you should download the update to a few testing computers (preferably also physically isolated from the main network) and run some basic checks to see if it works. In this case the testing computers would have blue screened instantly, and you would have known that this is not an update that you want on your system. Although usually requires a little bit more investigation to determine problems.

          • Angry_Autist (he/him)@lemmy.world
            link
            fedilink
            English
            arrow-up
            15
            arrow-down
            2
            ·
            5 months ago

            It makes me so fuckdamn angry that people make this assumption.

            This Crowdstrike update was NOT pausable. You cannot disable updates without disabling the service as they get fingerprint files nearly every day.

            • lando55@lemmy.world
              link
              fedilink
              English
              arrow-up
              6
              arrow-down
              1
              ·
              5 months ago

              I hear you, but there’s no reason to be angry.

              When I first learned of the issue, my first thought was, “Hey our update policy doesn’t pull the latest sensor to production servers.” After a little more research I came to the same conclusion you did, aside from disconnecting from the internet there’s nothing we really could have done.

              There will always be armchair quarterbacks, use this as an opportunity to teach, life’s too short to be upset about such things.

              • Angry_Autist (he/him)@lemmy.world
                link
                fedilink
                English
                arrow-up
                10
                arrow-down
                3
                ·
                5 months ago

                It doesn’t help that I’m medically angry 80% of the time for mostly no reason, but even without that this would incense me because I’ve had 40+ users shouting similar uneducated BS at me yesterday thinking that it was personally my fault that 40% of the world bluescreened. No I am not exaggerating.

                I have written and spoken phrases ‘No we could not prevent this update’ so many times in the last 24 hours that they have become meaningless to me through semantic satiation.

                • Gadg8eer@preserve.games
                  link
                  fedilink
                  English
                  arrow-up
                  1
                  ·
                  5 months ago

                  Take it from me, reality is a prison. If your issues are as bad as mine, escapism is the only solution and social media is the polar opposite of escapism. I’m not saying “do drugs”, I’m saying “threaten to quit, and if they call your bluff, make an untraceable alteration to fuck the company over and quietly hand in your resignation” before taking a break to indulge hobbies while searching for another job.

                  And if you can’t afford to lose your job at all for any length, you now have a morally-acceptable reason to kill everyone in your workplace because better death than slavery to a system this corrupt.

    • helpImTrappedOnline@lemmy.world
      link
      fedilink
      English
      arrow-up
      31
      arrow-down
      1
      ·
      edit-2
      5 months ago

      I’ve got a feeling crowdstrike won’t be as grand of target anymore. They’re sure to lose a lot of clients…at least until they spin up a new name and erease all traces of “crowdstrike”.

      • reddit_sux@lemmy.world
        link
        fedilink
        English
        arrow-up
        8
        ·
        5 months ago

        I don’t think they will lose any big clients. I am sure they will have insurance to take care of compensations.

      • Echo Dot@feddit.uk
        link
        fedilink
        English
        arrow-up
        7
        ·
        5 months ago

        That trick doesn’t work for B2B as organizations tend to do their research before buying. Consumers tend not to.

    • billwashere@lemmy.world
      link
      fedilink
      English
      arrow-up
      20
      ·
      5 months ago

      Third parties being able to push updates to production machines without being tested first is giant red flag for me. We’re human … we fuck up. I understand that. But that’s why you test things first.

      I don’t trust myself without double checking, so why would we completely trust a third party so completely.

    • Miaou@jlai.lu
      link
      fedilink
      English
      arrow-up
      16
      ·
      5 months ago

      I’d assume state (or other serious) actors already know about these companies.

    • Angry_Autist (he/him)@lemmy.world
      link
      fedilink
      English
      arrow-up
      16
      arrow-down
      4
      ·
      5 months ago

      This is why I openly advocate for a diverse ecosystems of services, so not everyone is affected if the biggest gets targeted.

      But unfortunately, capitalism favors only the frontrunner and everyone else can go spin, and we aren’t getting rid of capitalism anytime soon.

      So basically, it is inevitable that crowdstrike WILL be hacked, and the next time will be much much worse.

      • Cryophilia@lemmy.world
        link
        fedilink
        English
        arrow-up
        6
        arrow-down
        1
        ·
        5 months ago

        Properly regulated capitalism breaks up monopolies so new players can enter the market. What you’re seeing is dysfunctional capitalism - an economy of monopolies.

        • Angry_Autist (he/him)@lemmy.world
          link
          fedilink
          English
          arrow-up
          5
          arrow-down
          7
          ·
          5 months ago

          Sorry no, capitalism is working exactly as intended. Concentration of wealth breaks regulation with unlimited political donations.

          You call it unregulated, but that is the natural trend for when the only acceptable goal is the greater accumulation of wealth. There comes a time when that wealth is financially best spent buying politicians.

          Until there are inherent mechanisms within capitalism to prevent special interest money from pushing policy and direct regulatory capture, capitalism will ALWAYS trend to deregulation.

          • Cryophilia@lemmy.world
            link
            fedilink
            English
            arrow-up
            5
            arrow-down
            1
            ·
            5 months ago

            You call it unregulated, but that is the natural trend for when the only acceptable goal is the greater accumulation of wealth.

            Yes…obviously.

            And that IS dysfunctional capitalism.

            Until there are inherent mechanisms within capitalism to prevent special interest money from pushing policy and direct regulatory capture

            That’s exactly what I’m saying, dude.

            This is NOT capitalism working as intended. This is broken capitalism. Runaway capitalism. Corrupt capitalism.

            • hglman@lemmy.ml
              link
              fedilink
              English
              arrow-up
              1
              ·
              5 months ago

              Its like saying we just need good kings, no ids a bad system. Any capitalist system will devolve in corruption and monopoly. No regulations can survive the unavailable regulatory capture and corruption.

          • barsoap@lemm.ee
            link
            fedilink
            English
            arrow-up
            4
            arrow-down
            1
            ·
            edit-2
            5 months ago

            You call it unregulated, but that is the natural trend for when the only acceptable goal is the greater accumulation of wealth.

            Nah unregulated is the exact right word and that isn’t the kind of neolib you’re out for. Those would use “free” instead of unregulated, deliberately confusing unregulated markets with the theoretical model of the free market which allocates resources perfectly – if everyone is perfectly rational and acts on perfect information. Which obviously is not the case in the real world because real-world.

            There’s a strain of liberalism which is pretty much the cornerstone of Europe’s economical model, also, generally compatible with socdem approaches, and it says precisely that regulation should be used to bring the real-world market closer to that theoretical ideal – they’re of course not going all-out, you’d need to do stuff like outlaw trade secrets to actually do that, have all advertisement done by an equitable and accountable committee and shit. But by and large regulation does take the edge off capitalism. If you want to see actually unregulated capitalism, have a look at Mexican cartels. Rule of thumb: If you see some market failure, regulate it away. Like make producers of cereal pay for the disposal costs of the packaging they use and suddenly they have an interest in making that packaging more sensible, can’t externalise the cost any more.

            Defeating capitalism ultimately is another fight altogether, it’s nothing less than defeating greed – as in not the acquisition of things, but getting addicted to the process of acquisition: The trouble isn’t that people want shit the problem is that they aren’t satisfied once they’ve got what they wanted. Humanity is going to take some more time to learn to not do that, culturally, (and before tankies come along nah look at how corrupt all those ML states were and are same problem different coat of paint), in the meantime regulation, rule of law, democracy, even representative democracy, checks and balances, all that stuff, is indeed a good idea.

          • gremllin@lemmy.world
            link
            fedilink
            English
            arrow-up
            1
            arrow-down
            4
            ·
            5 months ago

            If you start regulating capitalism, thats called something else. That would be saying that the markets can not regulate by themselves, and proving as a myth one of the basics of capitalism.

            So I, as well, think capitalism is working as intended. and sure is based on greed.

            • Aatube@kbin.melroy.orgOP
              link
              fedilink
              arrow-up
              4
              ·
              5 months ago

              Something else, as in what? As long as the means of production is privately owned for profit, it’s capitalism.

      • driving_crooner@lemmy.eco.br
        link
        fedilink
        English
        arrow-up
        2
        ·
        edit-2
        5 months ago

        Years ago I read an study about insurance companies and diversification of assets in Brazil. By regulation, an individual insurance company need to have a diversified investment portfolio, but the insurance market as a whole not. the diversification of every individual company sum, as a whole of all the insurance market, as an was exposed market, and the researchers found, iirc, like 3 banks that if they fail they can cause a chain reaction that would take out the entire insurance market.

        Don’t know why, but your comment made me remind of that.

        • Angry_Autist (he/him)@lemmy.world
          link
          fedilink
          English
          arrow-up
          4
          arrow-down
          2
          ·
          5 months ago

          That’s kind of fascinating, never considered what the results of that kind of regulation can bring without anyone even noticing it at the time. Thanks for a good reading topic for lunch!

    • Pika@sh.itjust.works
      link
      fedilink
      English
      arrow-up
      2
      ·
      5 months ago

      Yeah the fact that this company calls it feature that they can push an update anytime without site level intervention is scary to me. If they ever did get compromised boom every device running their program suddenly has a kernel level malware essentially overnight.

  • diffusive@lemmy.world
    link
    fedilink
    English
    arrow-up
    161
    arrow-down
    9
    ·
    5 months ago

    If I had to bet my money, a bad machine with corrupted memory pushed the file at a very final stage of the release.

    The astonishing fact is that for a security software I would expect all files being verified against a signature (that would have prevented this issue and some kinds of attacks

    • BossDj@lemm.ee
      link
      fedilink
      English
      arrow-up
      51
      ·
      5 months ago

      So here’s my uneducated question: Don’t huge software companies like this usually do updates in “rollouts” to a small portion of users (companies) at a time?

      • Dashi@lemmy.world
        link
        fedilink
        English
        arrow-up
        36
        arrow-down
        1
        ·
        5 months ago

        I mean yes, but one of the issuess with “state of the art av” is they are trying to roll out updates faster than bad actors can push out code to exploit discovered vulnerabilities.

        The code/config/software push may have worked on some test systems but MS is always changing things too.

        • madcaesar@lemmy.world
          link
          fedilink
          English
          arrow-up
          9
          arrow-down
          1
          ·
          5 months ago

          Somone else said this wasn’t a case of this breaks on windows system version XXX with update YYY on a Tuesday at 12:24 pm when clock is set to eastern standard time. It literally breaks on ANY windows machine, instantly, on boot. There is no excuse for this.

      • Echo Dot@feddit.uk
        link
        fedilink
        English
        arrow-up
        13
        arrow-down
        1
        ·
        5 months ago

        Companies don’t like to be beta testers. Apparently the solution is to just not test anything and call it production ready.

      • deegeese@sopuli.xyz
        link
        fedilink
        English
        arrow-up
        2
        ·
        5 months ago

        When I worked at a different enterprise IT company, we published updates like this to our customers and strongly recommended they all have a dedicated pool of canary machines to test the update in their own environment first.

        I wonder if CRWD advised their customers to do the same, or soft-pedaled the practice because it’s an admission there could be bugs in the updates.

        I know the suggestion of keeping a stage environment was off putting to smaller customers.

    • Justin@lemmy.jlh.name
      link
      fedilink
      English
      arrow-up
      24
      arrow-down
      8
      ·
      5 months ago

      Windows kernel drivers are signed by Microsoft. They must have rubber stamped this for this to go through, though.

    • Angry_Autist (he/him)@lemmy.world
      link
      fedilink
      English
      arrow-up
      15
      arrow-down
      1
      ·
      5 months ago

      From my experience it was more likely to be an accidental overwrite from human error with recent policy changes that removed vetting steps.

        • Angry_Autist (he/him)@lemmy.world
          link
          fedilink
          English
          arrow-up
          6
          arrow-down
          2
          ·
          5 months ago

          Quick development will probably spell the end of the internet once AI code creation hits its stride. It’ll be like the most topheavy SCRUM you’ve ever seen with the devs literally incapable of disagreeing.

          I was thinking about his stint at McAfee, and I think you’re right. My real question is: will the next company he golden parachutes off to learn the lesson?

          I’m going to bet not.

    • jj4211@lemmy.world
      link
      fedilink
      English
      arrow-up
      57
      ·
      5 months ago

      Just needed to add 42k of ones to balance the data. Everyone knows that, like tires, you need to balance your data.

      • rmuk@feddit.uk
        link
        fedilink
        English
        arrow-up
        9
        ·
        5 months ago

        I mean, joking aside, isn’t that how parity calculations used to work? “Got more uppy bits than downy bits - that’s a paddlin’” or something.

        • Echo Dot@feddit.uk
          link
          fedilink
          English
          arrow-up
          7
          ·
          edit-2
          5 months ago

          Assuming they were all calculations, which they won’t have been.

          We will probably never know for sure, because the company will never actually release a postmortem, but I suspect that the file was essentially just treated as unreadable, and didn’t actually do anything. The problem will have been that important bits of code, that should have been in there, now no longer existed.

          You would have thought they’d do some testing before releasing an update wouldn’t you. I’m sure their software developers have a bright future at Boeing ahead of them. Although in fairness to them, this will almost certainly have been a management decision.

  • EleventhHour@lemmy.world
    link
    fedilink
    English
    arrow-up
    72
    arrow-down
    3
    ·
    edit-2
    5 months ago
    d'00000000 00000000 00000000 00000000 00000000
    00000000 00000000 00000000 00000000 00000000
    00000000 00000000 00000000 00000000 00000000
    00000000 00000000 00000000 00000000 00000000
    00000000 00000000 00000000 00000000 00000000
    00000000 00000000 00000000 00000000 00000000
    00000000 00000000 00000000 00000000 00000000
    00000000 00000000 00000000 00000000 00000000!
    
  • filister@lemmy.world
    link
    fedilink
    English
    arrow-up
    44
    ·
    5 months ago

    Imagine the world if those companies were using Atomic distribution and the only thing you would need to do is to boot the previous good image.

  • Socsa@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    60
    arrow-down
    23
    ·
    5 months ago

    The fact that a single bad file can cause a kernel panic like this tells you everything you need to know about using this kind of integrated security product. Crowdstrike is apparently a rootkit, and windows apparently has zero execution integrity.

    • OutsizedWalrus@lemmy.world
      link
      fedilink
      English
      arrow-up
      52
      arrow-down
      2
      ·
      5 months ago

      I’m not sure why you think this statement is so profound.

      CrowdStrike is expected to have kernel level access to operate correctly. Kernel level exceptions cause these types of errors.

      Windows handles exceptions just fine when code is run in user space.

      This is how nearly all computers operate.

    • Dark Arc@social.packetloss.gg
      link
      fedilink
      English
      arrow-up
      37
      arrow-down
      2
      ·
      5 months ago

      This is a pretty hot take. A single bad file can topple pretty much any operating system depending on what the file is. That’s part of why it’s important to be able to detect file corruption in a mission critical system.

      • Dave.@aussie.zone
        link
        fedilink
        English
        arrow-up
        3
        ·
        edit-2
        5 months ago

        This was a binary configuration file of some sort though?

        Something along the lines of:

        IF (config.parameter.read == garbage) {
             Dont_panic;
        }
        

        Would have helped greatly here.

        Edit: oh it’s more like an unsigned binary blob that gets downloaded and directly executed. What could possibly go wrong with that approach?

    • phx@lemmy.ca
      link
      fedilink
      English
      arrow-up
      24
      ·
      5 months ago

      Security products of this nature need to be tight with the kernel in order to actually be effective (and prevent actual rootkits).

      That said, the old mantra of “with great power” comes to mind…

      • arin@lemmy.world
        link
        fedilink
        English
        arrow-up
        7
        arrow-down
        1
        ·
        edit-2
        5 months ago

        with great power, don’t lay off the testing team (force return to office or get fired ultimatums)

        • tzrlk@lemmy.world
          link
          fedilink
          English
          arrow-up
          2
          ·
          5 months ago

          It’s fine, they’ve just switched to a crowd-sourced testing strategy.

    • areyouevenreal@lemm.ee
      link
      fedilink
      English
      arrow-up
      25
      arrow-down
      2
      ·
      5 months ago

      Yeah pretty much all security products need kernel level access unfortunately. The Linux ones including crowdstrike and also the Open Source tools SELinux and AppArmor all need some kind of kernel module in order to work.

      • reddithalation@sopuli.xyz
        link
        fedilink
        English
        arrow-up
        8
        ·
        5 months ago

        crowdstrike has caused issues like this with linux systems in the past, but sounds like they have now moved to eBPF user mode by default (I don’t know enough about low level linux to understand that though haha), and it now can’t crash the whole computer. source

        • areyouevenreal@lemm.ee
          link
          fedilink
          English
          arrow-up
          6
          ·
          5 months ago

          As explained in that source eBPF code is still running in kernel space. The difference is it’s not turing complete and has protections in place to make sure it can’t do anything too nasty. That being said I am sure you could still break something like networking or critical services on the system by applying the wrong eBPF code. It’s on the authors of the software to make sure they thoroughly test and review their software prior to release if it’s designed to work with the kernel especially in enterprise environments. I am glad this is something they are doing though.

      • uis@lemm.ee
        link
        fedilink
        English
        arrow-up
        4
        arrow-down
        2
        ·
        5 months ago

        At least SELinux doesn’t crash on bad config file

        • areyouevenreal@lemm.ee
          link
          fedilink
          English
          arrow-up
          15
          ·
          edit-2
          5 months ago

          I am not praising crowdstrike here. They fucked up big time. I am saying that the concept of security software needing kernel access isn’t that unheard of, and is unfortunately necessary for a reason. There is only so much a security thing can do without that kernel level access.

    • hglman@lemmy.ml
      link
      fedilink
      English
      arrow-up
      8
      arrow-down
      5
      ·
      5 months ago

      Tell me you don’t understand what it is by telling me you don’t understand what Crowdstrike does.

    • GroundedGator@lemmy.world
      link
      fedilink
      English
      arrow-up
      3
      arrow-down
      3
      ·
      5 months ago

      I don’t remember much about my OS courses from 20 years back, but I do recall something about walls between user space and kernel space. The fact that an update from the Internet could enter kernel space is insane to me.

  • Gork@lemm.ee
    link
    fedilink
    English
    arrow-up
    25
    arrow-down
    3
    ·
    5 months ago

    How can all of those zeroes cause a major OS crash?

    • tiramichu@lemm.ee
      link
      fedilink
      English
      arrow-up
      214
      arrow-down
      2
      ·
      5 months ago

      If I send you on stage at the Olympic Games opening ceremony with a sealed envelope

      And I say “This contains your script, just open it and read it”

      And then when you open it, the script is blank

      You’re gonna freak out

      • Gork@lemm.ee
        link
        fedilink
        English
        arrow-up
        43
        arrow-down
        1
        ·
        5 months ago

        Ah, makes sense. I guess a driver would completely freak out if that file gave no instructions and was just like “…”

          • planish@sh.itjust.works
            link
            fedilink
            English
            arrow-up
            38
            arrow-down
            1
            ·
            5 months ago

            That’s what the BSOD is. It tries to bring the system back to a nice safe freshly-booted state where e.g. the fans are running and the GPU is not happily drawing several kilowatts and trying to catch fire.

              • Aatube@kbin.melroy.orgOP
                link
                fedilink
                arrow-up
                22
                ·
                5 months ago

                what do you propose, run faulty code that could maybe actually nuke your system, not just memory but storage as well?

              • Windows assumes that you installed that AV for a reason. If it suddenly faults, who’s to say it’s a bug and not some virus going ham on the AV? A BSOD is the most graceful exit you could do, ignoring and booting a potentially compromised system is a fairly big no-no (especially in systems that feel the need to install AV like this in the first place).

              • Morphit @feddit.uk
                link
                fedilink
                English
                arrow-up
                9
                ·
                5 months ago

                A page fault can be what triggers a catch, but you can’t unwind what a loaded module (the Crowdstrike driver) did before it crashed. It could have messed with Windows kernel internals and left them in a state that is not safe to continue. Rather than potentially damage the system, Windows stops with a BSOD. The only solution would be to not allow code to be loaded into the kernel at all, but that would make hardware drivers basically impossible.

              • reddit_sux@lemmy.world
                link
                fedilink
                English
                arrow-up
                7
                ·
                5 months ago

                BSOD is the ultimate catch statement of the OS. It will gracefully close all open data streams and exit. Of course it is not the usual exit so it gives a graphic representation of what not have gone wrong.

                If it would have been nuking it wouldn’t show anything.

          • Kaboom@reddthat.com
            link
            fedilink
            English
            arrow-up
            7
            arrow-down
            1
            ·
            5 months ago

            For most things, yes. But if someone were to compromise the file, stopping when they see it invalid is probably a good idea for security

      • deadbeef79000@lemmy.nz
        link
        fedilink
        English
        arrow-up
        18
        arrow-down
        1
        ·
        5 months ago

        Except “freak out” could have various manifestations.

        In this case it was “burn down the venue”.

        It should have been “I’m sorry, there’s been an issue, let’s move on to the next speaker”

        • Strykker@programming.dev
          link
          fedilink
          English
          arrow-up
          17
          ·
          5 months ago

          Except since it was an antivirus software the system is basically told “I must be running for you to finish booting”, which does make sense as it means the antivirus can watch the system before any malicious code can get it’s hooks into things.

          • Morphit @feddit.uk
            link
            fedilink
            English
            arrow-up
            9
            ·
            5 months ago

            I don’t think the kernel could continue like that. The driver runs in kernel mode and took a null pointer exception. The kernel can’t know how badly it’s been screwed by that, the only feasible option is to BSOD.

            The driver itself is where the error handling should take place. First off it ought to have static checks to prove it can’t have trivial memory errors like this. Secondly, if a configuration file fails to load, it should make a determination about whether it’s safe to continue or halt the system to prevent a potential exploit. You know, instead of shitting its pants and letting Windows handle it.

        • the_crotch@sh.itjust.works
          link
          fedilink
          English
          arrow-up
          12
          ·
          5 months ago

          In this case it was “burn down the venue”.

          It was more like “barricade the doors until a swat team sniper gets a clear shot at you”.

          • deadbeef79000@lemmy.nz
            link
            fedilink
            English
            arrow-up
            12
            arrow-down
            1
            ·
            5 months ago

            Hmmmm.

            More like standing there and loudly shitting your pants and spreading it around the stage.

        • tiramichu@lemm.ee
          link
          fedilink
          English
          arrow-up
          6
          arrow-down
          4
          ·
          5 months ago

          You’re right of course and that should be on Microsoft to better implement their driver loading. But yes.

          • Morphit @feddit.uk
            link
            fedilink
            English
            arrow-up
            8
            ·
            5 months ago

            The driver is in kernel mode. If it crashes, the kernel has no idea if any internal structures have been left in an inconsistent state. If it doesn’t halt then it has the potential to cause all sorts of damage.

      • Imgonnatrythis@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        11
        arrow-down
        1
        ·
        5 months ago

        Maybe. But I’d like to think I’d just say something clever like, “says here that this year the pummel horse will be replaced by yours truly!”

        • Takios@discuss.tchncs.de
          link
          fedilink
          English
          arrow-up
          17
          ·
          5 months ago

          Problem is that software cannot deal with unexpected situations like a human brain can. Computers do exactly what a programmer tells it to do, nothing more nothing less. So if a situation arises that the programmer hasn’t written code for, then there will be a crash.

          • deadbeef79000@lemmy.nz
            link
            fedilink
            English
            arrow-up
            10
            arrow-down
            8
            ·
            5 months ago

            Poorly written code can’t.

            In this case:

            1. Load config data
            2. If data is valid:
              1. Use config data
            3. If data is invalid:
              1. Crash entire OS

            Is just poor code.

            • 5C5C5C@programming.dev
              link
              fedilink
              English
              arrow-up
              14
              ·
              5 months ago

              When talking about the driver level, you can’t always just proceed to the next thing when an error happens.

              Imagine if you went in for open heart surgery but the doctor forgot to put in the new valve while he was in there. He can’t just stitch you up and tell you to get on with it, you’ll be bleeding away inside.

              In this specific case we’re talking about security for business devices and critical infrastructure. If a security driver is compromised, in a lot of cases it may legitimately be better for the computer to not run at all, because a security compromise could mean it’s open season for hackers on your sensitive device. We’ve seen hospitals held random, we’ve seen customer data swiped from major businesses. A day of downtime is arguably better than those outcomes.

              The real answer here is crowdstrike needs a more reliable CI/CD pipeline. A failure of this magnitude is inexcusable and represents a major systemic failure in their development process. But the OS crashing as a result of that systemic failure may actually be the most reasonable desirable outcome compared to any other possible outcome.

              • Morphit @feddit.uk
                link
                fedilink
                English
                arrow-up
                5
                ·
                5 months ago

                This error isn’t intentionally crashing because of a security risk, though that could happen. It’s a null pointer exception, so there are no static or runtime checks that could have prevented or handled this more gracefully. This was presumably a bug in the driver for a long time, then a faulty config file came and triggered the crashes. Better static analysis and testing of the kernel driver is one aspect, how these live config updates are deployed and monitored is another.

              • deadbeef79000@lemmy.nz
                link
                fedilink
                English
                arrow-up
                4
                arrow-down
                2
                ·
                5 months ago

                But the OS crashing as a result of that systemic failure may actually be the most reasonable desirable outcome compared to any other possible outcome.

                In which case this should’ve been documented behaviour and probably configurable.

              • CeeBee_Eh@lemmy.world
                link
                fedilink
                English
                arrow-up
                1
                ·
                edit-2
                5 months ago

                That’s a bad analogy. CrowdStrike’s driver encountering an error isn’t the same as not having disk IO or a memory corruption. If CrowdStrike’s driver didn’t load at all wasn’t installed the system could still boot.

                It should absolutely be expected that if the CrowdStrike driver itself encounters an error, there should be a process that allows the system to gracefully recover. The issue is that CrowdStrike likely thought of their code as not being able to crash as they likely only ever tested with good configs, and thus never considered a graceful failure of their driver.

                • 5C5C5C@programming.dev
                  link
                  fedilink
                  English
                  arrow-up
                  1
                  ·
                  5 months ago

                  I don’t doubt that in this case it’s both silly and unacceptable that their driver was having this catastrophic failure, and it was probably caused by systemic failure at the company, likely driven by hubris and/or cost-cutting measures.

                  Although I wouldn’t take it as a given that the system should be allowed to continue if the anti-virus doesn’t load properly more generally.

                  For an enterprise business system, it’s entirely plausible that if a crucial anti-virus driver can’t load properly then the system itself may be compromised by malware, or at the very least the system may be unacceptably vulnerable to malware if it’s allowed to finish booting. At that point the risk of harm that may come from allowing the system to continue booting could outweigh the cost of demanding manual intervention.

                  In this specific case, given the scale and fallout of the failure, it probably would’ve been preferable to let the system continue booting to a point where it could receive a new update, but all I’m saying is that I’m not surprised more generally that an OS just goes ahead and treats an anti-virus driver failure at BSOD worthy.

            • Takios@discuss.tchncs.de
              link
              fedilink
              English
              arrow-up
              10
              ·
              5 months ago

              I agree that the code is probably poor but I doubt it was a conscious decision to crash the OS.

              The code is probably just:

              1. Load config data
              2. Do something with data

              And 2 fails unexpectedly because the data is garbage and wasn’t checked if it’s valid.

              • Morphit @feddit.uk
                link
                fedilink
                English
                arrow-up
                3
                ·
                5 months ago

                You can still catch the error at runtime and do something appropriate. That might be to say this update might have been tampered with and refuse to boot, but more likely it’d be to just send an error report back to the developers that an unexpected condition is being hit and just continuing without loading that one faulty definition file.

                • Gadg8eer@preserve.games
                  link
                  fedilink
                  English
                  arrow-up
                  2
                  ·
                  5 months ago

                  Unfortunately, an OS that covers such cases is a lost monetization opportunity, fuck the system, use a Linux distro, you get the idea. Microsoft makes money off of tech support for people too unversed in computers to fix it themselves.

        • Hazzia@infosec.pub
          link
          fedilink
          English
          arrow-up
          9
          arrow-down
          2
          ·
          5 months ago

          I’m gonna take from this that we should have AI doing disaster recovery on all deployments. Tech CEO’s have been hyping AI up so much, what could possibly go wrong?

          • Couldbealeotard@lemmy.world
            link
            fedilink
            English
            arrow-up
            7
            ·
            5 months ago

            What are the chances that Crowdstrike started using ai to do their update deployments, and they just won’t admit it?

      • DigitalDilemma@lemmy.ml
        link
        fedilink
        English
        arrow-up
        12
        arrow-down
        3
        ·
        5 months ago

        Nice analogy, except you’d check the script before you tried to use it. Computers are really good at crc/hash checking files to verify their integrity, and that’s exactly what a privileged process like antivirus should do with every source of information.

      • JasonDJ@lemmy.zip
        link
        fedilink
        English
        arrow-up
        8
        ·
        edit-2
        5 months ago

        The funny bit is, I’m sure more than a few people at Crowdstrike are preparing 3 envelopes right now.

      • CeeBee_Eh@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        arrow-down
        1
        ·
        5 months ago

        Ah yes. So Windows is the screaming in terror version and other systems are the “oh, sorry everyone, looks like there’s an error. Let’s just move on to the next bit” version.

    • driving_crooner@lemmy.eco.br
      link
      fedilink
      English
      arrow-up
      14
      arrow-down
      3
      ·
      edit-2
      5 months ago

      The file is used to store values to use as denominators on some divisions down the process. Being all zeros is caused a division by zero erro. Pretty rookie mistake, you should do IFERROR(;0) when using divisions to avoid that.

      • sugar_in_your_tea@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        19
        arrow-down
        2
        ·
        5 months ago

        I disagree. I’d rather things crash than silently succeed or change the computation. They should have done better input and output validation, and gracefully fail into a recoverable state that sends a message to an admin to correct. A divide by zero doesn’t crash a system, it’s a recoverable error they should 100% detect and handle, hot sweep under the rug.

      • Morphit @feddit.uk
        link
        fedilink
        English
        arrow-up
        5
        ·
        5 months ago

        IFERROR(;0)

        Maybe they should use a more appropriate development tool for their critical security platform than Excel.

  • yokonzo@lemmy.world
    link
    fedilink
    English
    arrow-up
    16
    ·
    5 months ago

    I’m not a dev, but don’t they have like a/b updates or at least test their updates in a sandbox before releasing them?

    • kalleboo@lemmy.world
      link
      fedilink
      English
      arrow-up
      18
      ·
      5 months ago

      It could have been the release process itself that was bugged. The actual update that was supposed to go out was tested and worked, then the upload was corrupted/failed. They need to add tests on the actual released version instead of a local copy.

      • FiniteBanjo@lemmy.today
        link
        fedilink
        English
        arrow-up
        2
        arrow-down
        2
        ·
        5 months ago

        Could also be that the Windows versions they tested on weren’t as problematic as the updated drivers around the time they released.

  • orcrist@lemm.ee
    link
    fedilink
    arrow-up
    6
    ·
    5 months ago

    I wonder how many governments and companies will take this as a lesson on why brittle systems suck. My guess is most of them won’t… It’s popular to rely on very large third party services, which makes this type of incident inevitable.

    • Pika@sh.itjust.works
      link
      fedilink
      English
      arrow-up
      2
      ·
      5 months ago

      oh they’ll take it as a lesson all right, up until they get the quote to fix it suddenly the downtime becomes non-issue as long as it “doesn’t happen again”