Recently, Microsoft had quite frankly a kicking from the US Department of Homeland Security over their security practices in a Cyber Safety Review Board report. I’ve tried to keep as quiet as possible about this one for various reasons (and I was not involved in the CSRB report, even anonymously) — although long time followers will know I’ve been often critical of Microsoft’s security posture. The CSRB report is well worth a read — they did a great job.
In particular, I aired some pretty critical (but vague) thoughts after leaving Microsoft — which although there was a period where I gather people were told it was because I had an axe to grind, it wasn’t — it was because I had been concerned about what I had seen. I couldn’t do anything about it at the time due to other issues going on there. Truthfully I’d always been critical of Microsoft over various things, e.g. prior to joining Microsoft I talked to BBC News about docs.com being the source of Microsoft’s customers accidentally leaking their documents, which ultimately led to the quiet mothballing of the product.
To give Microsoft its credit, it has a unique (from what I’ve seen) and good corporate culture that tolerates dissent. The reason I aired concerns is.. well.. what happens at Microsoft impacts society. And, selfishly, also me, and the ability for me to defend the orgs I work for.
They’re uniquely placed as a software and services vendor. Microsoft’s security woes are, in my opinion, a safety issue for society because the dependency on them for civil society is so vast. Those woes are not the fault of one person, or team — in fact, I think they’re due to security debt building up over time.
Microsoft have now announced to customers their response — after a slightly weird few days where they announced what was happening via anonymous press briefings, to shareholders and then attached to pitching Security Copilot in news reports. Lesson for Microsoft here — tell staff and customers first, and keep AI sales separate.
After watching the dust settle, I thought it was important to dig into the announcements and what I think they mean in practice.
The Microsoft response
The Verge reports that Microsoft are re-prioritising around cybersecurity as their top priority, with six prioritized security pillars (pro-tip: if you’re ever talking to Satya, talk in pillars) —largely talking about Microsoft’s internal corporate systems.
These pillars are:
Protect identities and secrets
Protect tenants and isolate production systems
Protect networks
Protect engineering systems
Monitor and detect threats
Accelerate response and remediation
The announcement to customers comes with a blog from Charlie Bell, their Executive Vice President of Microsoft Security:
And an all company email from Satya Nadella, their CEO. To quote the end of it:
Security is a team sport, and accelerating SFI isn’t just job number one for our security teams — it’s everyone’s top priority and our customers’ greatest need.
If you’re faced with the tradeoff between security and another priority, your answer is clear: Do security. In some cases, this will mean prioritizing security above other things we do, such as releasing new features or providing ongoing support for legacy systems. This is key to advancing both our platform quality and capability such that we can protect the digital estates of our customers and build a safer world for all.
I’ll say up front, it’s a great email. I think Satya and Charlie are killing it with messaging. Frankly, everybody likes to talk about cybersecurity and pretend it’s super interesting.. until they actually have to do it.
Because cybersecurity is, frankly, major boring (and far too expensive) shit — and also really hard to do right and super easy to do wrong. So it’s the kind of email you have to send from the top as you need people internally to go ‘oh, this is a real thing’.
What I think it means
It’s a really good response, which high level grapples with a whole bunch of problems I saw within Microsoft. Truthfully, the problems I saw at Microsoft extended way beyond what is known, and I don’t want to document them publicly as — frankly — they appear to know parts of the barn are, in fact, on fire.
The other thing I’m cognizant of is I was just an idiot analyst there, a pleb — I did not have a full picture of what was happening, I was sat looking at things going ‘oooooooooohkay, apparently it’s 1997 here in cybersecurity’.
One of the things with the Snowden disclosures is that guy was, essentially, a SharePoint administrator as I understand it, who somehow promoted himself to Chief Expert Spy after fleeing the country. I was just a boring threat intelligence person for Microsoft in Redmond. And I also offer no documents to read. I want to be clear I’m not comparing myself to Snowden here, more making the opposite point.
I think when you had Brad Smith talking about “being on the frontlines” of the Russia/Ukraine war, on the side supporting Ukraine, it was the right thing to do — but also, you’re relying on having the best cybersecurity in the world. Microsoft was already a target, but essentially it’s like screaming ‘come at me, I’m hard’ in a Call of Duty match. I don’t think the execs at Microsoft had a full view of the security challenges they are facing.
That isn’t to say everything Microsoft has been doing in cybersecurity is wrong, by the way — it isn’t, they employ some of the smartest people in security and deal with an incredible amount of incidents nobody knows about. What I mean is, some risky things there are normalised — things straight up in CISA’s Bad Practice list, that Microsoft helped write — and were deemed okay because.. uh… they’re Microsoft.
do as i say not as i do — the customer experience
Watermelon Green
In a PowerPoint slide, it looks like every unpatched system is automatically booted from the network after 2 weeks, 100% green! In reality, if you looked at the real details… you’d hit the red. I call this the Watermelon Green effect.
To be clear, I have no idea if that PowerPoint slide exists at Microsoft (it probably does, based on what I’m told as a customer by MS sales about Microsoft’s amazing internal security) — it’s just a made up example of that I think every org can learn from. If people aren’t bringing out their dead, too, you’re looking at death by compliance and governance, especially where everything is incentivised to be green.
Breaking down the pillars
Protect identities and secrets — the identity teams and managers at Microsoft are really good, in my experience. Microsoft Entra ID is also a really good product, along with the product security offerings.
What they’ve grappled with in the identity space internally includes parts of the org trying to rapidly bolt on services upon services — e.g. consumer Xbox authentication etc — and they’ve reached a point of a two decade long game of Jenga that has got a bit out of hand. Some of the things bolted on included clear documentation on how not to use the features, which internal developers promptly ignored. It feels right Microsoft go back and concentrate on this area, as Jenga has a habit of toppling down.
Notably, the announcement mentions HSMs — think of this as digital vaults. I’ve had a few people at Microsoft call me and be like ‘Kevin, we do use HSMs’. Reality check: Microsoft don’t use HSMs for everything, including some of the most important things.
There’s lots of certificate keys and such which simply haven’t been securely handled properly. I know a bunch of this infrastructure is really old, but.. well.. the world grew up, and the steps other orgs had to take a decade ago are also steps Microsoft needs to take about a decade ago. HSMs are hard to implement and use, in some cases incredibly so — I’m 25 years into corporate security and have the scars. But… uh… either make a better solution, or suck it up and implement ’em 100% of the time with deeply sensitive material.
On the other hand, the story with secrets at Microsoft is… they are not so secret. This is common across lots of orgs, and to Microsoft’s credit they’ve tried (and tried and tried) to get internal developers under control with putting API keys in source code and such. I think they need to keep trying.
For a long period Microsoft dealt with a guy or girl called The Walking Cat (_h0x0d_) on Twitter who constantly leaked Microsoft’s commercial plans publicly (notably Xbox, but others too), and he/she was just.. er… largely walking in the front door.
None of this should be reflection on Entra ID, I think. It isn’t. It’s issues around how Microsoft themselves use their own systems and have superglued things together like they’re on Season 37 of The A-Team. Mr T’s looking at bit too old for this.
Protect tenants and isolate production systems — you might look at this and think ‘Microsoft has the best security in the world, of course they already did this’. There’s always work to be done, and the path Charlie outlines in his blog is spot on for focus areas. Does everybody need to be local admin? Can attack surface be reduced for many people? If it was me I’d bring in somebody like PWC and say, don’t red team us, just roast us on our day to day security practices in the trenches… before China does.
Protect networks — raise your hand if you’ve worked for a company where the network is too flat, and people have convinced themselves it is okay because Zero Trust Or Whatever? Every company should try to fix this.
Protect engineering systems- obvious and good.
Monitor and detect threats — I often made the joke at Microsoft that Microsoft’s security products are great, and Microsoft should use them. It wasn’t a joke, also. Microsoft dogfood a whole bunch of their product stack, but they need to turn the features used up to 11 and really deploy everything. Microsoft Threat Experts? It should be enabled on Microsoft’s own corporate tenant. When I was there, it wasn’t… and had that service been enabled, they would have spotted significant nation state activity several months before Mandiant did. Microsoft own the product and the team and can afford their own services — that applies across the stack.
And yes, logging of corporate systems for two years is very much needed (and relatively standard amongst many industries). Azure IRM classification of every new doc created, DLP, the whole shebang should be there. If it’s too hard internally, improve the product for customers too (and sell more as a result).
Accelerate response and remediation- “Increase transparency of mitigated cloud vulnerabilities through the adoption and release of Common Weakness Enumeration™ (CWE™), and Common Platform Enumeration™ (CPE™) industry standards for released high severity Common Vulnerabilities and Exposures (CVE) affecting the cloud.” <- one thing with this, it should not just apply to CVEs, as CVEs exclude just cloud services. Microsoft, I think, need to lead the way in detailing fixed vulnerabilities in their cloud stack that they currently don’t list. Why? Because all cloud providers should do this, and it forces hands I think. This may be what they mean, but the wording leaves a bit of wriggle room I think compared to the CSRB report intentions.
Another pullout — Instituting new governance
I want to talk about the “Instituting new governance” heading from Charlie’s blog as I think it’s really important, and he’s hit upon an area of concern as a customer.
Microsoft has a bunch of different business units, who operate.. kinda separately? You’d have, say, Microsoft Security who operated differently to, say, Azure. Each of these would have overlapping areas — e.g. both Azure and Microsoft Security operated security products, but they had different senior leadership.
On top of this, Microsoft operated a security governance structure I’d not seen before — each business unit basically did its own thing. That’s over simplifying it and being glib, but the security governance was basically not aligned, and it led to situations like — for example — Microsoft Security, when I worked there, didn’t have certain features in Microsoft Defender for Endpoint enabled. But some non-security business units did. (This was fixed while I was there). It was pretty head scratching.
To quote the blog: “Microsoft is implementing a new security governance framework spearheaded by the Chief Information Security Officer (CISO). This framework introduces a partnership between engineering teams and newly formed Deputy CISOs, collectively responsible for overseeing SFI, managing risks, and reporting progress directly to the Senior Leadership Team. Progress will be reviewed weekly with this executive forum and quarterly with our Board of Directors.”
I think what’s happening here is they’re trying to centralise some of this, in a structure that works inside Microsoft’s existing operating model. That is smart, I think. But I’m bias:
Also smart: “We will instil accountability by basing part of the compensation of the company’s Senior Leadership Team on our progress in meeting our security plans and milestones.”
In other smart moves: “Finally, given the importance of threat intelligence, we are bringing the full breadth of nation-state actor and threat hunting capabilities into the CISO organization.”
Assuming this includes MSTIC, it’s a really good move. MSTIC are one of Microsoft’s threat intelligence teams, who track nation state threats and criminal groups. MSTIC is full of really smart people, many of whom I admire and do important work.. and is also an element of, I think, a loaded gun to the foot in terms of their level and scope of access to telemetry, as not everything is just the optics of cybersecurity. It’s certainly an area that needs oversight and diverse thinking. Frankly threat actors are transitioning to platforms like Mac and Linux anyway. MSTIC probably needs to find sustainable insight from outwards in, rather than looking out.
My overall thoughts
As always, the proof is in the pudding, not the vendor blog. I think these changes will take a few years to start to work through, and fully expect a few more clanger breaches in the mean time. And that’s annoying but okay, because hard work is hard.
I do think, though, Microsoft are on the right track here towards earning my trust back as a customer. They’re talking about real internal issues at Microsoft — in a corporate blog cosplay way of course — and actually heading straight at long standing and festering issues which need addressing.
I’d also say, people shouldn’t take the CSRB report to be criticisms of Microsoft’s security product offerings. They aren’t. Microsoft’s security products are largely very good, in my view — having both used them at scale at large organisations for long periods of time while being the accountable manager, to actually working on a few. Microsoft Defender for Endpoint? Great. Microsoft Defender for Identity? Great. Both have got me and my organisations out of multiple security incidents unscathed. Same with Entra ID Protection.
Have there been tangible security failings with Microsoft 365 services? Yeah.
Do I understand the customers who get frustrated about receiving malware from Microsoft 365 hosted services? Hell yeah, I’m one. I also resent being sold Microsoft Defender for Endpoint… then being told to buy Microsoft Defender for Endpoint P2 addon, then Microsoft Sentinel etc to get the right logs in the right format. I get growth top right is always needed for a publicly traded business, but… uh… the upsell is real and often a turn off.
It’s particularly hard to justify nowadays as something in business you’ll really hear is ‘Microsoft should be giving us this for free, it’s our own data’ — I think the woes over the last few years have really damaged trust and the E5 upsell may be a problem for Microsoft’s sales people, frankly.
I think challenges remain in Microsoft Office, Microsoft Exchange, Microsoft SharePoint, ADFS.. some of the core products. Things should be more secure out of the box. Microsoft Windows shouldn’t allow .VBS, .VBE files to do anything in the space year 2024. I know, I know, Smart App Control and all that — but it feels like a lot of engineering work for a conversation which should probably become more ‘…should we just sunset some features and break a small amount of AppCompat, to protect hospitals?’.
I know there’s been some significant engineering resource on things like Application Guard for Office, Edge etc — but my personal view, from the trenches, is that if features require new hardware, things manually enabling etc — it isn’t going to work. Lots of orgs still have nobody responsible for cybersecurity, that’s just how it is in the trenches. Every day is spent talking to suppliers who have been hit with ransomware and have no idea how to begin to respond. And largely, it started with Windows PCs and servers. And it’s usually threat actors repeating the same basic things over and over again.
I think a whole bunch of the solutions Microsoft should look towards should be as simple to engineer and use as possible.
For example, the changes which hardened Office macros started due to, as I understand it, somebody in the Office team making the brave call to just disable macros from the internet, and take the heat it would generate. That relatively small pivot changed the email security market, and changed the overall threat landscape — it’s evident in data from email server and endpoint logs. In my opinion, that’s the template of what should be happening across Microsoft’s product offerings, driven by threat intelligence. Office macros were left to fester for far too long, and essentially spawned a large part of the ransomware industry problem we are now left with (and has become very difficult to put out — we’re in the age of the ransomware wildfire).
The pivot to attackers looking at edge network devices is, I think, partly driven by that. Now, you may say — Kevin — let’s just leave ransomware actors using Office macros, as we know how to defend against that. And it’s an okay argument. But it doesn’t help libraries, UK councils, old people’s homes etc in the immediate term. Disabling macros from the internet definitely did help them. Yes, attackers may be finding new ways — but that’s a cost we should, overall, be imposing I think. Every vendor should be better, and Microsoft should aim to be the best.
I’ve long said ransomware groups are a competitor for security vendors. I think we’re seeing that play out. It’s completely flown under the radar, but I recently published a private exploit that Akira ransomware group had been using against Cisco ASA VPN devices, written by @Naproxen.. the exploit looks simple, but finding it required looking through ~100k lines of code.
When you’ve got ransomware groups, largely driven by teenagers, working on edge network product exploits alongside nation state activity in China.. it’s a wild and escalating situation. I think we’ll continue to security vendors, not just Microsoft, having to reprioritise security to be at the heart of their operations — due to the real threat of being plunged into chaos by anime loving ransomware fans.
Microsoft has the ability to redefine Windows, Office, Microsoft 365 and other products as the most secure solutions available out of the box — to be the true security leader for small to medium size businesses and large enterprise, by taking some calculated decisions on legacy compatibility.
Being an open to use operating system platform should not mean being an insecure one due to legacy code owned end to end by Microsoft itself (hi, wscript).
Microsoft has history with this. When I flash back to the beginnings of my career, as I was right there at the front of the IIS Unicode issue, where a Unicode directory traversal vulnerability allowed IIS to spawn any command. It was a really daft vulnerability. One of the earliest emails in my personal inbox is me and Microsoft’s security team talking about that, as it happens. A bunch more gaffes were made.
Microsoft’s then CEO, Bill Gates, reset the agenda on security, and in fairness — if you look at many Microsoft subsequent server offerings (such as IIS), they’re rarely the cause of security incidents in organisations. They really tightened up shop in many areas.
Did Microsoft later fumble it? Yeah. The Department of Homeland Security literally handed the US President a report (the CSRB one) saying Microsoft have made big mistakes.
To spell it out, security debt has piled up and internal corporate security at Microsoft isn’t where it needs to be for the world’s most important and targeted software company. The risk pivot was wrong. Hindsight is easy, doing the right thing is hard.
Microsoft products and services should be the safest choice to use for business leaders and community groups.
I think that’s a competitive advantage to end up at.. but frankly and more importantly, I think it’s the right thing to do. And I think if Microsoft delivers on this strategy, it can deliver that vision.
Breaking down Microsoft’s pivot to placing cybersecurity as a top priority was originally published in DoublePulsar on Medium, where people are continuing the conversation by highlighting and responding to this story.