Russinovich: Microsoft is “All-in” on Rust (Premium)

Rust may be controversial over in Linux-land, but there’s no such controversy at Microsoft, which has embraced the memory-safe language without condition. I’m glad to hear that, and am not surprised that the adults over in the Azure group have a firm grip on the rudder. But I’m also vaguely happy to discover that the software giant is going down a path I outlined when first learned about Rust and Microsoft’s initial adoption. It is working to a future in which it will use AI to rewrite its core software kernels in this relatively new language.

Many things need to happen before that simple-sounding but technically complex miracle occurs, perhaps more than is obvious. Among the issues are developer familiarity with the language, modern tools for working on Rust, C++ interoperability–a key issue for a platform that dates back to the 1990s and Microsoft’s fixation at the type with Object-Oriented Programming (OOP) and C++ specifically–and advances in AI coding capabilities. But let’s get real here. We know all that is happening. And we know that the AI piece of this puzzling is happening quite quickly.

So yes, one day, an engineer at Microsoft will wave a magic wand and all the code will just get better, better in this case meaning a guarantee of safety, improved performance, and all the other benefits that come from lesser code refactoring efforts. But today, we’re just at the beginning of this journey. And a recent talk by Azure CTO Mark Russinovich provides a timely update to his company’s use of Rust across the stack, and how it’s evolved over time.

? An update on Linux and Rust

This is timely because of a recent flare-up in the Linux community. I told about half that story in From the Editor’s Desk: The Problem is You (Premium), but there have been some important developments since.

Here’s a quick recap: In October 2022, Linus Torvalds merged the first Rust code into the Linux kernel. This wasn’t the first use of Rust in a major computing platform–Google announced it would use Rust in the Android kernel in April 2021–but Linux is basically the infrastructure for the planet, so this was big news. The problem, of course, is people: Some of the old-school graybeards that maintain the Linux kernel have resisted this shift, for reasons practical (Rust and C/C++ interoperability) and personal (their expertise is being undermined by an unfamiliar technology).

Because Linux is developed in the open, the internal fights over Rust are public. Most recently, a Linux kernel maintainer refused to accept a Rust code submission, trigger a battle in which both sides were wrong. The net result was that the code submitter dropped out contributing code to Linux and then, after Torvalds crapped on him for trying to work around the system he put in place, he dropped out of an open source project he had founded, too.

That’s where I had left things when I wrote that article linked above, as things were happening in real time and what I really wanted to focus on is that resistance to change that I see so often in our more Microsoft-focused world. But I was delighted to see that, in the wake of that event, several Linux kernel maintainers spoke up about their support of Rust, reiterating all the usual claims about improved security and performance. And Torvalds himself lashed out at the maintainer who had refused to accept the Rust code submission, explaining to him that his job was not to refuse code that wasn’t even tied to his area of responsibility in the first place.

“If you as a maintainer feel that you control who or what can use your code, YOU ARE WRONG,” Torvalds wrote on the Linux kernel mailing list in his characteristically blunt style. “As a maintainer you are in charge of your code, sure — but you are not in charge of who uses the end result and how.” Three days later, the maintainer Torvalds went after quietly stepped down from maintaining the module at the center of this controversy. One assumes there were some private email-based discussions that resulted in a “you can leave or be let go” discussion in which that maintainer was left with no real choice. Not that it matters. For all the wrong on both sides, the important correction occurred: Someone in a position of power who resisted a positive change for selfish reasons was pushed aside.

? Meanwhile, in Redmond

When I wrote about this flare-up originally, I noted that Microsoft’s kernel development is not quite as transparent as that of Linux. And that it was possible there were similar flare-ups within the software giant, and that we would never be privy to most of them because that development occurs largely in secret. (I joked of imagining that Microsoft’s kernel development happened in a secret underground lair similar to that seen in the opening credits of Get Smart.)

But Microsoft is transparent in many ways, much more so than companies like Apple and Google, which don’t have the same freewheeling policies as Microsoft does when it comes to employees engaging with the public via social media, blogs, and the like. And Microsoft executives routinely describe internal machinations during public talks, at Microsoft and third party events big and small. Such was the case with Russinovich this past week: He appeared at Rust Nation UK last week in London, delivering a closing keynote to the show called Microsoft is Getting Rusty: A Review of Successes and Challenges. And that talk was the very definition of transparent in that it didn’t just highlight the successes, but also honestly discussed some of the challenges of using Rust as well.

Heading into this talk, I knew that Microsoft had announced in April 2023 that it was rewriting parts of the Windows kernel in Rust. This is a tricky area, but Microsoft often commingles the terms “Windows” and “Azure” despite the latter dropping the Windows brand many years ago. So while it’s sometimes unclear where Windows ends and Azure begins, the two do share a core architecture and kernel, at least to some degree. And so there is overlap, with the same core team responsible for both low-level codebases.

Since then, we saw the first Windows kernel code written in Rust go live in stable, and Microsoft later expanded its use of Rust to include drivers as well. Rust will only improve the quality and security of Windows, so that’s fantastic. But I’d like to see more frequent discussions of this work. And it’s fair to say that Azure, which sits at the backbone of the world’s infrastructure–most of the platforms hosted on Azure are Linux, not Windows Server–and so its use of Rust is in some ways more profound.

“It’s time to halt starting any new projects in C/C++ and use Rust for those scenarios where a non-GC language is required,” Russinovich tweeted in September 2022. “For the sake of security and reliability. the industry should declare those languages as deprecated.”

Those were–and still are–fighting words.

“When I saw Rust show up, I became convinced just based off its simplicity, its elegance, the fact that it actually enforces correctness, that it was a language that was designed for somebody like me that likes systems programming, likes the performance, the reliability, and wants a language that actually does what I want it to do,” Russinovich said in his talk this past week. “And one day, I was looking at programming language stats … and seeing C and C++ still on that list of commonly used languages. And I’m just thinking, Rust is ready. We should just be switching to Rust. And in a burst of emotion, I tweeted this is and it just seemed non-controversial to me.”

This was, of course, controversial. But to Russinovich, the engineer, the storied software developer who once reverse engineered the Windows NT kernel, it was also pragmatic, common sense.

“I actually got a call from [Microsoft CEO] Satya Nadella after this tweet saying, ‘Really?’ and I said, yes, really,” Russinovich revealed. “I’m convinced this is it. And he’s like, ‘OK.’ So Microsoft has been on a journey. Not just from my tweet, but [from] grassroots efforts across the company to adopt Rust. My tweet was just part of a larger movement towards recognizing the problems in C and C++ that led to things like the NSA declaring just a couple of months later that people should get off C and C++ and other any other type of unsafe memory languages because of the problems they were seeing.”

This is sharply different from the flare-up we just witnessed in the Linux community. Again, it’s heartening to know that there are real adults in charge of Microsoft’s kernels, and that they can put aside years of tradition and familiarity to do the right thing for their employer, the products they oversee, and the customers who will later use those products.

? Examples of Microsoft’s use of Rust

Russinovich added some crucial details to the story of Microsoft’s Rust adoption this past week, too. He noted that Rust is a key component of the software giant’s Secure Future Initiative push, as it will impact the kernels of its two most important platforms, Azure and Windows (and already has). He admitted that SFI was “a response to [Microsoft] being breached by two nation state actors that took advantage of vulnerabilities like unsafe code to penetrate [its] infrastructure.” And that, at that time, the company “broadly acknowledged” it should get off unsafe languages. So, controversial to some degree, yes. But not really.

He then provided specific examples of Microsoft’s use of Rust, noting that the goal now is to “get Rust everywhere.” These examples are the most prominent and mature examples, he said, and that Microsoft started small and then expanded its adoption of Rust with each success. Interestingly, he was unable to cite any failures. Microsoft’s adoption of Rust has been incredibly positive.

Those examples are:

Project Mu (Rust UEFI). One of the “very first” Rust implementations inside Microsoft was to rewrite its UEFI firmware, “security critical code” that’s used both by its Azure datacenters and its Surface “consumer line of laptops.” The source code for Project Mu is open source and available on GitHub. It’s running now on Surface PCs, and Microsoft “encourages” its PC maker partners to adopt it as well.

DirectWrite Core. This is the Windows App SDK implementation of the DirectWrite text rendering APIs in Windows, which date back to the early 2000s and Longhorn/Vista. (You may recall that it replaced the old Win32 GDI/GDI+ libraries from the earliest days of Windows.) I believe this was literally the first work Microsoft did with Rust (despite the above example). It was selected because it was “fairly self-contained” and its fonts parser was the source of numerous security vulnerabilities. Two developers spent 6 months to port 154,000 lines of C and C++ code to Rust, about two-thirds of the DirectWrite component, but 100 percent of the surface area impacted by vulnerabilities. The Rust version of this component is 5 to 15 percent faster than its predecessor (more on that below). More to the point, an entire class of the most secure security issues was eliminated.

Win32 GDI Region. This was Microsoft’s first push into kernel mode. And its tied to an architectural decision from the mid-1990s, when the NT team moved graphics into the NT kernel to speed performance (a decision that had reverberations in the modern era when the CrowdStrike outage of 2024 made headlines.) “I consider this the underground oil deposit that keeps leaking oil in the form of [security] vulnerabilities. There is literally more than 1 or 2 every month … it’s the Achilles Heel of security for Windows.” Here, Microsoft took one component, GDI Regions, and ported it to Rust. This time, two developers ported 6,000 lines of C/C++ code to Rust in three months, and the only reason it took that long is C/C++ interop issues that are a common Rust gripe. (More on that below as well.) This work was so successful that Microsoft will continue “chipping away” at the other kernel mode parts of Win32, Russinovich said, adding that there were “no performance regressions at all.”

Cryptography. Microsoft ported a core part of the security-critical cryptography engine in Windows to Rust via a library called rustls-symcrypt, the SymCrypt Provider for Rustls. It also open sourced the code, which Russinovich says is “critical to Microsoft moving to a Rust-based crypto stack across not just Windows, but the rest of the company.”

Microsoft Office. The Office team–which I guess is technically called the Microsoft 365 team now, but I’m delighted that Russinovich stuck to the traditional name–has been looking at how it can take advantage of Rust. And this one is particularly interesting because its using Rust to eliminate C#, Microsoft’s language, which is a managed language with automatic garbage collection. That’s a good thing, of course, but it also brings performance overhead. And “there’s a lot of performance critical code in Office that has been written in C#,” he said, which surprises me. The example here was the semantic indexer and search algorithm that powers semantic search in Office. This is part of the “substrate” of Office, he says, and it scales from local apps to the biggest SharePoint implementations. Working with the CosmosDB and PostreSQL teams, Office ported this algorithm to Rust and obtained dramatically improved performance and scalability using about 60 percent the RAM. So Office, too, will expand its use of Rust thanks to this success.

Azure. This is, of course, Mark’s area of expertise, and here he added some personal anecdotes that amplify the message nicely. Before he sent the tweet quoted earlier–two to three years earlier–Russinovich issued an edict to Azure’s developers internally: “No more systems code in C++ in Azure. Enough is enough. Rust is ready. Let’s just use Rust instead of creating more [technology] debt for ourselves.” He provides several examples of Rust usage in Azure, from the Hardware Root of Trust that sits at the heart of the platform to Azure’s Hardware Security Modules (HSMs), Azure Boost Agents for networking and storage, the Hyper-V virtual machine (VM) hypervisor, and more. In that latter case, Microsoft wrote its Arm64 emulator for Hyper-V in Rust, and that team has been converting more and more of Hyper-V to Rust, as well as related technologies like OpenVMM, the open source, Rust-based implementation of its Virtual Machine Monitor (VMM) that it released earlier this year. There’s also a new lightweight in-app VMM called HyperLight that was originally written in C# but rewritten in Rust and open sourced. One of these projects has over 350,000 lines of code, Russinovich said.

⚖️ The edict

Tied to the Azure bit above, Russinovich expanded on the seemingly apocryphal story I simplified in From the Editor’s Desk: The Problem is You (Premium) as him “declaring an end to new C and C++ development” in Azure.

“I declared a few years ago in whatever authority I have as [Azure] CTO to say, ‘no more C/C++,” he said. “To my pleasant surprise, all I had to do was say it in a few meetings, and suddenly, it was like the law of the land. And since then, we’ve had no more C/C++ components. The new components have all been in Rust. In fact, several of the teams took their existing C++ code and migrated it to Rust.”

Universally, the experience at Microsoft has been positive and each Rust conversion has been successful. The safety and security was no surprise. But the performance scalability benefits were unexpected.

? Performance

“The performance impact of moving to Rust is a common theme across everything done,” Russinovich said, “When we moved from C/C++ to Rust, we saw a 5 to 15 percent performance Improvement.”

Addressing the obvious complaint from the never-Rusters–you might see similar gains if you refactored the same code in C/C++ because code improvements often provide performance and reliability gains regardless of language–Mark had a quick comeback.

“The fact is, we did not intend to get a performance Improvement,” he said. “This was purely a porting exercise. And we saw this [gain]. And the other aspect of this is that we never see performance regressions when we port to Rust. So this is something else that’s very encouraging.”

?️ The feedback

Coincidentally, Microsoft recently announced its first Azure SDK for Rust, in beta. This SDK includes Rust-based libraries for Identity, Key Vault secrets and keys, Event Hubs, and Cosmos DB, each of which is open source, and with more to come. “Because of customer demand, Rust is a first class language for the Azure SDK,” Mark said, not just internally at Microsoft, but for all developers. This feedback is important for understanding Rust and its pros and cons.

“A lot of what we’re hearing from people [who adopted Rust internally at Microsoft]” is overwhelmingly positive, Russinovich explained. When Rust code compiles, it works properly, unlike C/C++. The reduced friction is motivational to developers, leading to accelerated adoption. Dependency management is streamlined. Performance is better. Memory and data race bugs are reduced, with Rust putting memory management front and center, forcing developers to be conscious of mistakes they were previously blind to.

But there are issues, of course.

The biggest issue with Rust is C/C++ interoperability, the underlying technical excuse for the recent Linux kernel flare-up. “That’s been a topic of conversation here [at the conference],” Russinovich said, noting it was not a surprise that this was the number one issue.

Rust tooling is still behind that of more established languages, frameworks, and SDK in the Microsoft space. This is why the aforementioned Azure SDK is such a big deal, but the “ecosystem is a little immature” and this is something even Microsoft is still trying to figure out.

The language is still evolving and some features aren’t yet stable. The dynamic linking that Windows relies on is “a problem with Rust.”

So it’s not all rosy. But what it is, Russinovich said, was “rosy enough. We are all-in at this point.”

According to Mark, almost all Microsoft developers who adopt Rust have nearly identical experiences. At first, the unfamiliarity makes Rust feel complicated. But after about two months–“that’s very common”–they become converts. “People flip from ‘I don’t get it’ to ‘I love it’.” he said.

? To the future

Concluding his talk, Russinovich discussed how Microsoft is now accelerating its adoption of Rust. There are two big areas here, one niche–verified crypto algorithms, including for post-Quantum computing crypto–and the other more general. One of the goals here is to make porting to Rust automatic. And it is here that Russinovich hints at how, if not outright confirms that, my contention that this company will one day use AI to rewrite and recompile the Windows kernel in Rust isn’t science fiction.

“We’re more ambitious,” he said. “We want more automated translation of C and C++ to Rust. And we’re just getting started on this. And guess what tool we’re using to do this, or try to do this. LLMs.”

He then launched into a discussion of generative AI, what it can do, and how Microsoft is using something called GraphRAG that uses the Retrieval Augmented Generation (RAG) capabilities of LLMs to extract knowledge from raw text in structured forms. It’s hierarchical RAG, which is much more precise when it comes to translating software code. And Microsoft has already used this technology in limited ways. Russinovich showed a demo of a game, written in Python, that it converted to Rust. Which sounds simple enough, but traditional LLMs don’t work well with multiple source code files, and GraphRAG can handle that structure and duplicate it in Rust.

The simple game Russinovich showed in no way matches the complexity of even a single Windows kernel component. But the end game here is, I think, obvious and inevitable. All journeys start with a single step.

“We’re 100 percent behind Rust,” Mark added.