Like many large enterprise organizations, Microsoft’s support team manages thousands of troubleshooting guides, self-help resources, knowledge base articles, and process documentation. To improve customer support, they’re increasingly leveraging AI and ChatGPT to assist support engineers with technical issues. However, with continuous product updates and new features, these document repositories had become incredibly large, and sometimes contained outdated information.
“It was fascinating how fast things were changing,” says DJ Ball, senior escalation engineer, Modern Work Supportability. “OpenAI was so new and features growing so fast from GPT 2.0 to 3.0 to ChatGPT to 4.0 almost overnight. Keeping up with the technology is a challenge and big opportunity.”
In July 2022, Microsoft’s Modern Work Supportability team developed a concept for semantic search. This would allow support engineers to use a single search query across vast support document repositories to find relevant results. After exploring different solutions and discussing ideas with others, the team began exploring generative AI. Soon after, ChatGPT was announced and drastically changed the landscape.
“People thought we planned it, which we didn’t, but for once we felt we were ahead of the game,” says Sam Larson, a senior supportability PM, Modern Work Supportability.
The team secured subscriptions to Microsoft Azure OpenAI and found an internal GPT playground focused on handling enterprise content. This playground was similar to what the Supportability Team was designing, which made collaboration easy. The development of an AI-based solution accelerated. The Modern Work Supportability team provided continuous feedback to the engineering team, which helped shape the product that later became Microsoft Azure AI Studio. This studio simplifies the process of integrating external data sources into Microsoft Azure OpenAI Service. Consequently, the team created a private chat workspace, Modern Work GPT (MWGPT).
The Modern Work Supportability Team initially curated content from various sources related to the Teams product and integrated it into the large language model (LLM). By utilizing Azure Cognitive Search to inject and chunk the documentation, they were able to test the results with subject matter experts (SMEs) across the Teams support business. They’ve since expanded to include all Modern Work Technology support documentation—an estimated 300,000-plus pieces of content for 34 products. They’ve learned a lot about content management, prompts, use case scenarios, and the functionality of LLMs along the way. Now, they work with over 450 SMEs across the Modern Work Support business to continue refining the content and testing the solution for accuracy.
“There is no question that there are a lot of variables that come into play that we are exposing our engineers to, and quality is a non-negotiable factor. We owe it to our customers who turn to our support engineers to help them solve their most challenging technical problems,” says Mayte Cubino, Modern Work support director, Office and Project Products.
The 6 Ds Framework
This documentation process led to the 6 Ds Framework, a roadmap for deploying enterprise content on private LLMs.
1. Discover
Discover is the initial phase. The team identifies problem goals and objectives. This stage involves learning, research, exploration, and analysis.
- Content curation
- Assess user needs and data sources
- Review existing data for accuracy and readiness
- Prepare data for security and reduce bias
2. Design
The design phase uses requirements developed during the discovery phase to make specific design choices. Ideation, testing, and prototyping occur here.
- Integrate AI with existing technology
- Manage content creation
- Data collection pipelines
- Database selection
- Data mining and analysis
- Responsible AI review
- Security
- Microsoft’s responsible AI principles (fairness, reliability and safety, privacy and security, inclusiveness, and transparency and accountability).
- Efficient technology use
- Plan how to train AI model
3. Develop
During the development phase, grounding data sets are created and tested. This is a highly iterative process, which includes:
- Content preparation
- Accuracy, bias, completeness, uniqueness, timeliness, validity, and consistency are key
- Content ingestion
- Chunk larger documents
- Prompt engineering
- Iterative process to elicit a desired response from the model
4. Diagnose
Rigorous testing and training are essential before deployment. Check that the model delivers the expected results since old, weak data can cause model drift, leading to inaccurate results. Consider these elements in this phase:
- Responsible Development and Diagnosis
- Data collection, fairness, transparency, security, and accountability are important
- Validate chatbot deployment
- Test to ensure accurate and reduce hallucinations
- Testing throughout the process mitigates issues
- Overfitting and underfitting
- Prompt tuning
5. Deploy
Deployment involves integrating a machine learning model into a production environment to make effective business decisions based on data. This phase uses live data for the AI model to make predictions.
- SME and validation team signoff process
- Evaluate the AI model to meet ongoing business objectives
6. Detect
Include AI model monitoring after deployment to create a feedback loop.
- Monitoring
- Monitor regularly and ensure accurate results. Align responses with the organization’s principles and values.
- Implement a feedback loop
- Learning from Reinforcement Learning from Human Feedback (RLHF) is a powerful technique.
Mayte Cubino, director of Modern Work support, joined the project as a volunteer and immediately added value by helping to create the structure for the 6 Ds framework. Cubino was interested in what she heard across the business about using ChatGPT to support customers. She knew the project needed someone with delivery experience. She addressed two questions:
- How can we ensure the new technology is successfully deployed with all support engineers?
- How can we make the technology as helpful as possible without adding extra work?
Cubino documented the content process and helped the team see that some things were nonnegotiable. She outlined steps to focus on to ensure accuracy and responsible engagement with the model.
Ross Smith, the leader of supportability for Modern Work technologies at Microsoft, spearheaded the team’s creation of Modern Work GPT. He explains that they were the first in the world to do this on support content, and they have the responsibility to get it right.
“We’re one of the first in the world to do this on support content and we recognize the opportunity and responsibility we have to get this right,” Smith says. “We hope others can build on our lessons, learn, and improve this for everyone.”
Starting to approach content support in this way prompted Smith to have many conversations across Microsoft to discover what teams were doing, and they focused on building a responsible LLM. The team aimed to create a responsibly AI (RAI) ready model. This required an understanding of the potential impacts on people and society and appropriate measures to mitigate predicted harms and prepare responses to unanticipated ones.
Jason Weum is a director of supportability for OneDrive and SharePoint support, and he led strategy development for Modern Work GPT. Another consideration was measuring success and the impact on the business. Knowing that support engineers would use the model for administrative and technical tasks helped the team develop a set of metrics. The application of the 6 Ds framework has the potential to extend beyond tech support to disciplines like human resources, finance, sales, and legal.
“Inputting all of your content into a chat model is like taking a flashlight and shining it in every dark corner of your content. You quickly realize what’s outdated,” says Jason Weum, director of supportability on the Modern Work Supportability team.
To ensure a successful deployment, the team found these points critical:
- Know where your documents are
- The curation and creation of documents ingested into the model is key
- Text or markdown formats tend to work best
- Review and retrain
- Gather feedback from SMEs
- Prompt
- Change management matters
Shakil Ahmed, general manager of support for Modern Work technologies at Microsoft, leads a team of over 3,500 support professionals focused on helping customers solve their toughest technology issues every day. The team is looking forward to increased use of Copilot by all Microsoft support engineers. They aim to continue to refine their models to deliver better results to support engineers, which in turn, will improve customer support experiences. The team believes AI will revolutionize how companies serve customers.
“As we dive into the world of Modern Work GPT support indexes, refining our models and embracing the AI revolution, we are like a team of tech-savvy superheroes, ready to take customer support experiences that are out of this world to the next level,” says Shakil Ahmed, general manager of the Modern Work Support team. “The future is here, and we’re excited to be on this wild ride of innovation.”
