Microsoft Modernizes Security Information and Event Management with Microsoft Sentinel

Modernizing Security with Microsoft Sentinel

Microsoft’s internal security team works tirelessly around the clock to safeguard the company’s intellectual property, its employees, and overall business health. To enhance its security posture, the team recently transitioned from an on-premises security information and event management (SIEM) solution to Microsoft Sentinel.

With Microsoft Sentinel, the team has significantly increased its capacity to ingest and respond to cybersecurity events, now handling over 20 billion events daily. This cloud-scale SIEM solution integrates seamlessly with crucial systems, ensuring timely and accurate responses to security threats. The team is responsible for maintaining robust security and compliance standards across Microsoft. Successfully managing the massive influx of security-related data is essential to the company’s operational well-being.

Previously, the team used a third-party SIEM tool hosted in Microsoft’s on-premises datacenters. However, they identified several areas where the service could be improved by adopting a next-generation SIEM tool. Challenges presented by the legacy tool included:

• Limited Scalability: The older system struggled to keep up with the ever-increasing volume of incoming traffic.
• Slow Data Ingestion: Transferring data into the previous SIEM tool was time-consuming and inefficient due to restricted ingestion capabilities.
• On-Premises Limitations: The on-premises nature of the solution hampered scalability and agile response to evolving business and security needs.
• Extensive Training Requirements: Substantial resources were needed for employee training and onboarding, particularly for new hires unfamiliar with the complex, customized on-premises architecture.

As part of Microsoft’s ongoing digital transformation, the team is migrating to cloud-based solutions with proven track records and strong customer-facing development efforts, enabling the technology stack to evolve at the speed of the business.

Feature Assessment and Planning

To address these challenges, the team began evaluating potential SIEM replacements, which would enable them to keep pace with the evolving cybersecurity landscape. In collaboration with the Microsoft Sentinel product team, the internal security division assessed whether Sentinel could effectively replace the existing solution. Sentinel, a cloud-native SIEM solution developed by Microsoft, leverages the cloud’s agility and scalability to facilitate rapid threat detection and response, using:

• Elastic Scaling: Easily adjusts to handle fluctuating data volumes.
• AI-Powered Detection: Incorporates artificial intelligence to enhance threat detection capabilities.
• Extensive Data Integration: Offers a broad range of pre-built connectors for data ingestion and connectivity.

To ensure a smooth transition, the team verified that all necessary features and capabilities from the previous environment were available in Microsoft Sentinel. Security teams across Microsoft were consulted to confirm all requirements were met, including teams which had mature monitoring and detection systems in place. Issues regarding throughput, agility, and usability significantly informed whether Sentinel could meet the company’s requirements. This close collaboration allowed the Microsoft Sentinel product team to receive feedback and provided the team with the ability to quickly identify and resolve customer needs.

Defining and Refining SIEM Detections

As the team developed standards to align with the requirements of the new SIEM, the team analyzed the functionality of their previous SIEM tool to determine a transition strategy into Microsoft Sentinel. During this review, three core aspects of incoming security data were examined:

• Data Source Validity: Reviewing the sources of incoming SIEM data identified the data sources that were helpful, and which were no longer providing relevant data.
• Detection Importance: The team examined the event-detection definitions used in the previous SIEM solution, understanding how detections were performed, which detection definitions generated alerts, and the volume of alerts. This helped the team prioritize key detection definitions for the migration process.
• Detection Validity: By evaluating the detections within the SIEM environment, invalid or redundant detection definitions were removed to create a more streamlined detection set within Microsoft Sentinel. The process of removal included combining multiple detection definitions and removing others.

Throughout this process, the Microsoft Security Operations team collaborated to evaluate detections end-to-end.

Implementation

After confirming feature parity and throughput capabilities, the migration process to Microsoft Sentinel began. Based on initial testing, certain implementation steps were added to support Microsoft’s security environment, which included:

• Onboarding Data Sources: Properly onboarding data sources was critical to implementation and one of the biggest benefits of the Microsoft Sentinel environment. With access to a massive inventory of default connectors, the ability to connect to data sources without customization became accessible. This includes typical data sources such as Microsoft Azure Active Directory, Microsoft Defender for Cloud, and Microsoft Defender. It also included connections to on-premises data sources, such as Windows Events and firewalls. Connections were established using playbooks.
• Refining Detections: Further testing allowed them to refine their detection definitions in Sentinel to support better alert suppression and aggregation. Suppression logic was included when a notification wasn’t required, and aggregation grouped related events to prevent security teams from being overwhelmed.
• Scaling with the Cloud: Dedicated clusters were deployed for Microsoft Azure Monitor Log Analytics to support the required data-ingestion capabilities. Moving from 10 billion to greater than 20 billion events per day was possible by adding additional data sources.
• Customizing Functionality: Customization was a necessity with the implementation and, by leveraging standard Microsoft Sentinel features, was easily done to help build the functionality that was necessary. Implementations included:
  • Integration with their IT service-management system: This extended Sentinel information and provided teams with the information they needed.
  • Implementing Microsoft Defender for Cloud to support scale: A playbook was used to automate the addition of more than 20,000 Azure subscriptions.
  • High-volume ingestion with Microsoft Azure Event Hub and Microsoft Azure Virtual Machine scale sets: A custom solution was built to ingest the large volume of events from their firewall systems, exceeding 100,000 events per second.

Benefits of Microsoft Sentinel

Microsoft’s team experienced several important benefits after the implementation of Microsoft Sentinel, including:

• Faster Query Performance: Query speeds improved significantly, with performance 12 times faster, on average, and up to 100 times faster for certain queries.
• Simplified Training and Onboarding: The cloud-based SIEM solution has simplified employee onboarding and training. Security engineers do not need to understand a complex on-premises architecture.
• Greater Feature Agility: The Sentinel platform provides features and capabilities that iterate at a much faster rate than a comparable on-premises solution.
• Improved Data Ingestion: Sentinel’s out-of-the-box connectors and Microsoft Azure integration make it much easier to include data from anywhere and extend functionality.

Reexamining and refining of the team’s approach to SIEM continued throughout the implementation of Microsoft Sentinel. Discoveries included:

• More testing enables more refinement: Through continuous testing, the team improved their test results and refine their implementations as a result.
• Customization is necessary but achievable: Flexibility became a cornerstone of the Microsoft Sentinel implementation, as the out-of-the-box features often needed the team to create customizations and integrations to meet their security requirements. Because of their size, enterprise customers may also need to dedicate a cluster.

With the first phase of the migration complete, ongoing development includes leveraging new ways to interact and engage with connected datasets and using machine learning to manage some of their most complex detections. As a result, the team can meet the security needs at an enterprise level.