Measuring Microsoft 365 Copilot Adoption: A Strategic Approach
Microsoft 365 Copilot represents a groundbreaking technological advancement, demanding a novel approach to adoption. Microsoft Digital, the company’s IT organization, has taken the lead in Copilot implementation, with a critical focus on measuring the value it provides to internal users. This article explores their strategy, the evolution of their metrics, and offers insights for tracking your own Copilot rollout.
Peter Varey, Tom Heath, and Tara Suan are instrumental in augmenting Microsoft 365 Copilot adoption efforts with data and insights.
The Unique Challenges of Copilot Adoption
Several factors distinguish Microsoft 365 Copilot, posing unique challenges for adoption leaders. Unlike a single application, Copilot integrates into multiple facets of the Microsoft technology stack, functioning as an intelligent assistant. Users interact with Copilot through natural language prompts, presenting a new paradigm for many.
“Copilot is unlike anything we’ve launched before, requiring us to rethink our strategies,” explains Peter Varey, director of employee insights on the Employee Experience Success team within Microsoft Digital. “Instead of simply promoting a new tool, we’re fundamentally changing how we work.”
Furthermore, the value of Copilot varies significantly depending on an employee’s role, making a robust measurement strategy essential for capturing and reinforcing its benefits.
Strategizing for Effective Insights
While tracking basic usage can be tempting, Microsoft Digital recognized the need for a deeper understanding. Tom Heath, a senior business program manager driving AI transformation, notes, “People only maintain habits when they become part of their identity. Therefore, we base our adoption strategies around ‘sticky metrics’ that measure consistent, habitual use.”
Microsoft Digital’s telemetry workstream for Microsoft 365 Copilot adoption focuses on four key objectives:
- Providing clarity on usage metrics, reporting options, and target audience for each.
- Ensuring the access and visibility needed to effectively support other adoption initiatives.
- Offering valuable Customer Zero feedback on telemetry sources and dashboards.
- Leveraging reporting to gather insights for refining the adoption approach.
Quantitative app telemetry is only part of the equation. Supplementing this data with qualitative insights through listening campaigns, which track satisfaction, adoption drivers, and product feedback, is essential. Microsoft’s UX experts played a critical role in planning and implementing this aspect of the measurement strategy by grounding their efforts in UX principles and AI-human interaction guidelines to ensure the right questions were being asked.
“Understanding the user’s experience puts us in a better position to support them,” says Tara Suan, a senior UX research lead with Microsoft Digital. “It’s less about identifying problems and more about shaping adoption to meet their needs.”
Key Metrics That Drive Adoption
A crucial lesson learned during the ongoing Copilot adoption measurement efforts is that the most effective metrics evolve as the rollout progresses. Varey states, “We’re on a continuous journey, striving for a more thorough understanding of what success looks like. We began with high-level OKRs [Objectives and Key Results] focused on usage, and we’ve advanced to more granular metrics.”
Initially, the focus was on maximizing employee experimentation with Copilot. Monthly active usage (MAU) was a critical starting point, and the measurement cycles lasted six months before new key metrics were instituted. Microsoft Digital captures these metrics via internal and commercial tools including the Copilot Dashboard in Viva Insights, the Copilot Adoption Report in the Analyst Workbench, and the Experience Insights Dashboard in the Microsoft 365 admin center. These tools allow for tracking weekly and daily usage, with data segmented by role, organization, and region. The team also assesses Copilot distribution by app penetration to understand how users access the tool through different applications such as Word, Excel, and Teams.
On the qualitative front, Microsoft utilizes surveys and in-app sentiment checks. Listening campaigns, conducted through collaborative bug bashes, satisfaction surveys, community engagement via Viva Engage, focus groups, and “prompt-a-thons”, help gather feedback across different facets of the employee experience, including:
- Net satisfaction: The overall positive or negative experience with Copilot.
- Favorability: Whether Copilot enhances productivity or efficiency.
- AI-assisted hours: A measure of time saved through Copilot use.
Results and Insights that Inform Adoption
Growing Copilot usage is a continuous journey, says Stephan Kerametlian, director of employee experience in Microsoft Digital.
Copilot adoption metrics are offering valuable insights that inform further usage across the company. For instance, app telemetry shows that summarization is the most frequently used Copilot feature. It also reveals that most users access Copilot through Teams rather than more comprehensive pathways like Graph-grounded Chat, which suggests that the prominence of in-app prompts, such as meeting summarization reminders, leads users to intuitive access to Copilot capabilities.
Microsoft can use this data to report to the product group about user behavior. They can also signal the adoption team might need to supplement less visible Copilot onboarding initiatives with learning programs. The maturity of measurement methods has made this adoption guidance possible.
From a qualitative perspective, employee satisfaction with Copilot is high, with 76% reporting satisfaction and 85% using the tool regularly. By digging deeper, insightful adoption patterns emerge. For example, early-career employees show a distinct adoption pattern: initial enthusiasm, a dip in engagement, and a resurgence around the 11-week mark. This understanding allows for proactive interventions, such as targeted skilling initiatives and reminders, to address the midstream dip.
Analyzing trends carefully is another important element of the process. Kerametlian says, “When we see what the data’s telling us, the trends that emerge, and how our employees feel about Copilot, it puts us in a position to shape both our technology and how we implement it—with people at the center.”
Recommendations for Measuring Copilot Impact
Here are some key suggestions for measuring the impact of Copilot within your company:
- Adapt OKR cycles: Keep Objectives and Key Results lifecycles relatively short to accommodate rapid evolution. Use an OKR while it’s useful, and then develop targeted performance metrics for the subsequent cycle.
- Establish product usage benchmarks: Gain a solid understanding of how employees are using existing Microsoft products like Word and Teams to provide a baseline before adding Copilot telemetry.
- Embrace negative results: Negative findings are an expected part of any research and measurement effort. You should not be discouraged when a specific line of investigation does not produce positive outcomes.
- Ensure quality employee metadata: Maintain a well-structured employee metadata system to enable effective segmentation and analysis of the workforce.
- Let the data guide the questions: Analyze usage vs. non-usage, areas of use, and employee satisfaction to inform your inquiries. Base your questions on trends shown by the data.
- Focus on desired outcomes: Identify key benefits that employees value and barriers to adoption and implement adoption strategies centered around those factors.
As Microsoft continues to track its Copilot adoption journey, it will continue to dig into new data, ask more directive questions, and refine its approach to maximize the technology’s value. By focusing on employee experience and tailoring strategies based on data-driven insights, Microsoft is paving the way for successful Copilot implementation.
