Orchestration (Premium)

Microsoft technical fellow Steven ("Stevie") Bathiche followed up his epic Build 2023 appearance with a short talk about Copilot+ PCs and NPUs at Build 2024 this past month. If you care about AI and how it is disrupting personal computing in unprecedented ways, both these discussions are worth your time. And each stands as a slice in time overview of what we can expect in the coming year.

I've discussed Stevie's Build 2023 appearance many times, but it's worth summarizing. He explained that Microsoft and third party developers would add AI capabilities to their platforms, apps, and services using three "application structures," and that these would coexist for a time until the ways in which we interact with personal computing resources were fundamentally altered.

The first application structure, "beside applications," is common today with Copilot being the obvious example: This is an obvious way to add AI capabilities to existing (legacy) apps, and it can happen external to the app (Copilot in Windows 11) or be added to an app, like Copilot in Microsoft Word or Excel.

The second structure is "inside applications," where AI is embedded inside an app, resulting in simplified user interfaces but powerful capabilities. He cited Clipchamp and Microsoft Designer as examples of this type of app. But the implication here is interesting: Where apps with hundreds or thousands of commands—like those in Office—have historically required busy user interfaces, making it difficult to find what you want, "inside applications" apps are an almost magical combination of simple and powerful. My experiences with Clipchamp certainly bear this out.

"Outside applications," the third structure, is the most futuristic. It may also be the end-game, if you will, for that coming generation of personal computing interactions, one that breaks us out of the monolithic standalone app model that we're so used to today. In this model, AI capabilities are exposed as agents (services) controlled by an underlying orchestrator. Instead of using explicit apps, users will ask the AI to perform some task and the orchestrator will determine the optimal combination of apps, services, plugins, and whatever else is needed to accomplish that task.

I want to focus on the orchestration bit here, as I feel this is perhaps the most important concept to understand. Orchestration also plays a role—or will, I think—in the way that software of the future, be it Windows, apps, services, whatever, will interoperate with the NPU, GPU, and CPU components in modern PCs. Which, interestingly enough, was the topic of Stevie's Build 2024 talk. But he didn't mention the term "orchestrate" even once.

We'll get to that. For now, consider how Stevie described orchestration a year ago.

"The AI will orchestrate across the multiple apps, plugins, and services, functioning more as an agent," he said. "If you take a step back, the Windows shell itself is an orchestrator. In fact, m...