Modernizing Enterprise Integration with Microsoft Azure Integration Services
Microsoft Digital Employee Experience (MDEE) recently modernized its on-premises integration platform, significantly boosting its capabilities, performance, and resilience. The team achieved this by adopting Microsoft Azure Integration Services, building a cloud-based integration platform as a service (iPaaS) solution. This migration allowed for impressive gains in data-transaction throughput, expanded integration capabilities, and enhanced platform reliability.
Business-to-business (B2B) and app-to-app (A2A) integration are crucial in modern software solutions, which depend on middleware technology to facilitate secure communication and data exchange across diverse enterprises and applications. At Microsoft, this means seamless integration between various independent software systems with diverse message formats like EDIFACT, X12, XML, JSON, and flat files. Modern integration requires multifaceted connectivity, including the ability to connect:
- Two or more internal applications.
- Internal applications with business partners.
- Internal applications with software-as-a-service (SaaS) applications.
Building on a Foundation of Enterprise Integration
Microsoft has a long history of integrating its business data internally and with its partners, vendors, and suppliers. For years, BizTalk Server has been a standard for integration services, providing a reliable and configurable foundation. The ongoing digital transformation is driving a shift towards cloud adoption, moving business resources out of traditional datacenters. With the evolution of data storage and application development, cloud-native solutions based on SaaS and PaaS models have become prevalent. To meet growing demands for scalability, reduce maintenance overhead, and decrease the total cost of ownership, the Platform Engineering team has increasingly turned to cloud-based solutions for enterprise integration.
Transforming Integration with Microsoft Azure
To modernize their infrastructure, the Platform Engineering team began exploring Microsoft Azure Integration Services. This service combines several Azure services–Logic Apps, API Management, Service Bus, Event Grid, and Azure Functions–into a comprehensive platform to seamlessly integrate business applications and data sources. The team started by testing scenarios on the platform to gauge feasibility and plan for enterprise-scale integration capabilities.
Collaborating to Improve Microsoft Azure Integration Services
Throughout development, the Platform Engineering team worked directly with the Integration Services product group, helping to enhance and build connectors. This close collaboration led to improvements in existing Integration Services functionality, including two new Logic Apps connectors (SAP with Secure Network Communication (SNC) and Simple Mail Transport Protocol (SMTP)) and enhancements to two current connectors (EDIFACT and X12).
Examining our Azure Integration Services architecture
MDEE utilizes all the components of Microsoft Azure Integration Services in its architecture to provide end-to-end integration. Each component contributes a critical part of the overall solution, including:
- API Management: For APIs, policies, rate limiting, and authentication.
- Logic Apps: For business workflows, message decoding and encoding, schema validations, transformations, and integration accounts to store B2B partner profiles, agreements, schemas, and certificates.
- Microsoft Azure Event Grid: For event-driven integration to publish and subscribe to business events.
- Microsoft Azure Functions: For writing custom logic tasks, including metadata and config lookup, data lookup, duplicate checks, and namespace and segment replacements.
- Microsoft Azure Data Factory: For low-volume, large-payload messages, ETL processes, and data transformation.
The team uses Microsoft Azure Front Door as the entry point for all inbound traffic and secures endpoints using Microsoft Azure Web Application Firewall, assigning permissions for allowed IP addresses. Additionally, API Management abstracts the authentication layer to improve security and simplify processing. The entire solution is deployed to an integration service environment, which provides a fully isolated and dedicated integration environment with benefits such as autoscaling, increased throughput limits, larger storage retention, improved availability, and a predictable cost model.
Figure 1: Microsoft Azure Integration Services architecture for Microsoft Digital Employee Experience.
The solution’s architecture follows several key design principles and goals:
- Pattern-based workflows that enable dynamic decisions using partner information.
- Self-contained extensible workflows that can be modified and improved without affecting existing components.
- A gateway component to store and forward messages.
- Publish-and-subscribe services for data pipeline output.
- Complete B2B and A2A pipeline processing with a throughput of 100 transactions per second and message handling up to 100 megabytes (MB) per message.
Designing Dataflow Pipelines
The dataflow pipelines perform critical processing for most business-data transformation and movement tasks. The team designed the B2B and A2A processing pipelines using Logic Apps and Microsoft Azure Functions, processing documents in their native format and delivering them to line-of-business (LOB) or enterprise resource planning (ERP) systems such as Finance, HR, Volume Licensing, Supply Chain, and SAP.
-
B2B Pipeline: Electronic data interchange (EDI) documents like purchase orders are brought in using AS2, processed using X12 standards, transformed, decoded and encoded using Logic Apps and Azure Functions, and then sent to the LOB app using the Logic Apps HTTP adapter.
-
A2A Pipeline: Documents such as XML/JSON come in using adapters including SAP, File, SQL, SSH File Transport Protocol (SFTP), or HTTP. The documents are debatched, transformed, decoded, and encoded using Logic Apps and Azure Functions, and then sent to the line-of-business system using the appropriate Logic Apps adapter.
This integration solution is used in practical business scenarios across several lines of business at Microsoft. For example, when a hardware manufacturer that includes Windows or Microsoft Office submits an order for licenses, the ordering system sends the order details to the integration suite. The suite validates the messages, transforms them to IDoc format, and routes the IDoc to SAP using a data gateway for taxation and invoice generation. SAP generates an order acknowledgment in IDoc format, which the integration suite then transforms into a format that the Microsoft ordering system recognizes.
In another scenario, from Microsoft Finance, an employee uses a corporate credit card and the financial institution sends a transaction report to the integration solution. The integration solution validates the message, performs currency conversion, and sends it to Microsoft’s expense-management system for approvals. After approval, the remittance transaction goes through the integration suite back to the banking system for payment.
Capturing End-to-End Messaging Telemetry
The solution is designed to monitor message flow across the pipeline. Each transaction injects data into the telemetry pipeline using Microsoft Azure Event Hubs. This pipeline synthesizes and correlates data to identify end-to-end processing status and recognize any runtime failures. A custom tracking service monitors critical metrics for end-to-end workflows, providing visual indicators on a dashboard. The readily available and accurate telemetry creates a more robust and reliable integration environment, improving the customer experience across the pipelines.
Realized Benefits
The iPaaS solution has delivered numerous benefits, including:
- Increased Scalability: The integration solution processes millions of monthly transactions, including 10 million B2B, 2.5 million A2A, and 74 million hybrid cloud transactions.
- Improved Quality of Service: Cross-region deployment with active-active configuration and thorough fault handling has produced 99.9 percent availability and reliability metrics.
- Reduced Total Cost of Ownership: Using this iPaaS solution, monthly costs in Microsoft Azure have been reduced by more than 40 percent.
- Increased Customer Engagements: The team is working to increase Microsoft Azure Integration Services adoption by promoting this solution to partners, vendors, and suppliers.
Microsoft Azure Integration Services has created an improved and more efficient integration environment for Microsoft. The increased scalability, reliability, and cost-effectiveness of Azure Integration Services has placed Microsoft in a better position to actively collaborate with and operate alongside its partners, suppliers, and vendors. The company continues to transform its integration services landscape with Azure Integration Services to remain competitive in the fast-paced modern business environment.
