Microsoft wants to bring generative AI to the forefront of Windows — and the PCs running it.
At a pair of keynotes during its annual Build developer conference this week, the company unveiled a new lineup of Windows machines it’s calling Copilot+ PCs, plus generative AI-powered features like Recall, which helps users find apps, files and other content they’ve viewed in the past. Copilot, Microsoft’s brand of generative AI, will soon be far more deeply integrated into the Windows 11 experience. And new Microsoft Surface devices are on the way.
We’ve rounded up all the major announcements from Monday and Tuesday here.
Volumetric Apps
Microsoft is bringing Windows Volumetric Apps — basically spatially aware, interactive VR apps — to Meta Quest headsets. Through a partnership with Meta, Microsoft says that it’ll deliver Windows 365 and local PC connectivity to Quest headsets, enabling developers to extend their apps into 3D space.
During Tuesday’s keynote, Microsoft showed off a digital exploded 3D view of an Xbox controller from the perspective of a Meta Quest 3 headset — a digital object the wearer could manipulate with their hands. “We’re deepening our partnership with Meta to make Windows a first-class experience on Quest devices,” Pavan Davuluri, CVP of Windows and devices at Microsoft, said during the demo.
Developers can sign up for a preview to receive access to Microsoft’s new volumetric API.
Copilot+ PCs
Copilot+ PCs are Microsoft’s vision of AI-first, flagship Windows hardware. All include dedicated chips called NPUs to power AI experiences like Recall. And they ship with 16GB of RAM minimum, paired with SSD storage.
The first Copilot+ PCs will pack Qualcomm’s Snapdragon X Elite and Plus chips, which Microsoft claims delivers up to 15 hours of web browsing and 20 hours of video battery life. Chipmakers Intel and AMD are also committed to building processors for Copilot+ devices in partnership with a range of manufacturers, including Acer, Asus, Dell, HP, Lenovo and Samsung.
Copilot+ PCs start at $999, and some are available for preorder today.
Surface Pro and Surface Laptop
Microsoft’s newly unveiled Surface devices, the Surface Laptop and Surface Pro, focus on performance and battery.
The latest Surface Laptop — available with a 13.8- or 15-inch display — has been redesigned with “modern lines” and thinner screen bezels. It lasts up to 22 hours on a charge and is up to 86% faster than the Surface Laptop 5, the company says. It also supports Wi-Fi 7 and has a haptic feedback touchpad.
As for the new Surface Pro, Microsoft says it’s up to 90% faster than the previous-gen Surface Pro (the Surface Pro 9), and it sports a new OLED with HDR display, Wi-Fi 7 (and optional 5G) and an upgraded ultrawide front-facing camera. Also, its detachable keyboard — which has been reinforced with additional carbon fiber — now has haptic feedback.
Recall
Windows 11’s forthcoming Recall feature can “remember” apps and content a user accessed on their PC weeks or even months ago, for example helping them to find a Discord chat where they were discussing clothes they were considering buying. Users can use Recall’s timeline to “scroll back” to see what they were working on in the recent past and drill down in files like PowerPoint presentations to surface info potentially relevant to their searches.
Microsoft says that Recall can create associations between colors, images and more to let users search for practically anything on their PCs in natural language (not dissimilar to startup Rewind’s tech); developers will be able to improve Recall by adding contextual information to their apps. And Microsoft claims all user data associated with Recall is kept private and on-device — and not used to train AI models, importantly.
Here’s more from Microsoft: “Your snapshots are yours; they stay locally on your PC. You can delete individual snapshots, adjust and delete ranges of time in Settings, or pause at any point right from the icon in the System Tray on your Taskbar. You can also filter apps and websites from ever being saved.”
Image editing and live translations
There’s now more AI in Windows than ever and some of it exclusively on the new Copilot+ PCs.
A new feature called Super Resolution can restore old photos by automatically upscaling them. And Copilot can now analyze images to give users ideas for creative compositions. Through a feature called Cocreator, users can generate images and also ask the AI model to follow what they’re drawing to change or restyle the image.
Elsewhere, Live Captions with live translations translates any audio that passes through a PC — whether from YouTube or a local file — into the language of the user’s choosing. Live translations will initially support around 40 languages, including English, Spanish, Mandarin and Russian.
A separate but related new feature in Microsoft Edge offers real-time video translation on sites like LinkedIn, YouTube, Coursera, Reuters, CNBC, Bloomberg and more. Set to become available in the near future, the feature — which supports the translation of Spanish into English and English to German, Hindi, Italian, Russian and Spanish — translates spoken content through both dubbing and subtitles live.
Team Copilot and extensions
Team Copilot is the latest expansion of Microsoft’s growing Copilot suite of generative AI tech. It integrates with Teams, the company’s videoconferencing app, to help manage meeting agendas and take notes that anyone in a meeting can co-author. And it extends to Loop and Planner, Microsoft’s collaboration and planning platforms, to create and assign tasks, track deadlines and notify team members when their input’s needed.
In somewhat related Copilot news, Microsoft has launched (in private preview) Copilot Extensions, which allow developers to extend GitHub’s code-generating tool GitHub Copilot with third-party apps and skills. Launch partners include DataStax, Docker and LambdaTest; extensions will live in the GitHub Marketplace, but developers will also be able to create their own private extensions to integrate with their internal systems and APIs.
Windows Copilot Runtime
Powering capabilities such as Recall and Super Resolution is the Windows Copilot Runtime, a collection of ~40 generative AI models that make up what Microsoft describes as “a new layer” of Windows. In tandem with the semantic index, a vector-based system local to an individual Copilot+ PC, the Windows Copilot Runtime allows generative AI-powered apps — including third-party apps — to run without necessarily needing an internet connection.
“[The runtime] consists of ready-to-use AI APIs like Studio Effects, Live Captions translations, OCR, Recall with user activity and [more], which will be available to developers in June,” Davuluri said on Tuesday.
Microsoft says that CapCut, the popular video editor from TikTok owner ByteDance, will use the Windows Copilot Runtime and accompanying new Windows Copilot Library, a set of APIs and AI dev tools, to speed up its AI features. And Meta will add the aforementioned Studio Effects to WhatsApp to deliver features such as background blur and eye contact during video calls.
Upgraded bot builders
Azure AI Studio, the toolset within Microsoft’s Azure OpenAI Service that lets customers combine an AI model and build an app that “reasons over” that data, will soon allow developers to create apps using pay-as-you-go inference APIs — the APIs through which developers can access and fine-tune generative AI models hosted on Azure infrastructure. Microsoft calls this “model-as-a-service,” and it’s launching with models from Nixtla and Core42 to start.
In the adjacent Copilot Studio product suite, Microsoft is launching Copilot agents, which the company describes as AI bots that can “independently orchestrate tasks tailored to specific roles and functions.” (Copilot Studio provides tools to connect Copilot for Microsoft 365, the AI-powered “copilot” in apps like Excel and Word, to third-party data.) Leveraging memory and knowledge of context, Copilot agents can navigate various types of business workflows, learning from user feedback and asking for help when they encounter situations they don’t know how to handle.
Snapdragon Dev Kit
There’s a new dev kit from Qualcomm aimed at developers building apps for Arm-chip-packing Copilot+ PCs
The $899.99 Snapdragon Dev Kit for Windows — which measures around the same width, height and length as Apple’s Mac Mini, incidentally — houses Qualcomm’s Snapdragon X Elite chip paired with 32GB of RAM, 512GB of storage and lots of I/O. The Dev Kit supports Wi-Fi 7 and Bluetooth 5.4, and, via its various USB-C and HDMI ports, it can drive up to three 4K monitors at once.
Phi-3
Microsoft announced an addition to its generative AI model family Phi, Phi-3-vision, that can do general visual analysis and reasoning tasks, such as answering questions about charts and images. The model can read both text and images and is efficient enough to run on a mobile device.
Phi-3-vision is available in preview, while the model’s previously announced text-only counterparts — Phi-3-mini, Phi-3-small and Phi-3-medium — are now generally available.
Partnership with Khan Academy
Microsoft is teaming up with Khan Academy to donate access to cloud compute infrastructure, allowing Khan Academy to offer educators in the U.S. free access to Khan Academy’s AI-powered tools. The two companies will also collaborate to explore opportunities to improve AI apps for math tutoring through generative AI, Microsoft said on Tuesday.
We’re launching an AI newsletter! Sign up here to start receiving it in your inboxes on June 5.
Comment