Today's enterprise applications demand high levels of reliability, scalability, availability, and interoperability. These demands have fueled the growth of complex service-oriented architectures (SOA) built on industry standards such as Web services and XML-based messages. The added complexity of these types of applications has increased the requirement for powerful tools and platforms.

With the release of BizTalk Server 2004 Microsoft has provided a powerful, cost-effective platform for developing and implementing message-based architectures, B2B applications, and enterprise application integration (EAI) projects. BizTalk tools are hosted within Visual Studio .NET, so applications can now leverage the full power of the .NET Framework, providing nearly endless extensibility.

In this article we will look at the components of BizTalk Server 2004 that are required to build and implement message-based solutions, including:

• Defining message structures using XML Schema
• Transforming message types using BizTalk maps
• Transporting messages
• High-performance messaging using filters

Messages
Overview
In the SOA world, data is passed between services as messages. These messages can take almost any form: e-mail messages, flat-file constructs, XML, HTML, even raw data from a database. The particular form and format of messages is largely determined by the type of solution. For example, integration of legacy systems will often require direct access to various databases, or the use of FTP to transfer various file formats between systems; B2B applications commonly use flat files like EDI sent over a proprietary network called a value-added network; and many new applications are using standard Internet technologies like XML and HTTP.

BizTalk Server 2004 provides a number of powerful tools to help us with defining the structure of messages, transforming messages from one type to another, transporting messages, and orchestrating the flow of messages between services (BizTalk Server 2004 has two main functional components: orchestration and the messaging engine. This article will focus on the messaging engine).

Defining Message Structure
The first step in building a messaging solution using BizTalk Server 2004 is to define the structure of the messages that are exchanged between the services. This is done by creating an XML Schema document that describes the physical structure of the data as well as defining certain characteristics of the data (e.g., types, ranges, etc.). Message specifications, which can define any type of message form and format, have two primary uses within BizTalk: validation of the message data, and definition of the rules used to construct instances of the message.

You can add a new message definition to a BizTalk project by right-clicking the project and selecting Add|Add New Item and selecting Schema. BizTalk defaults to defining the structure of an XML message; later I will discuss how to define the structure of non-XML messages. If you have used Visual Studio .NET you will notice that the XML Schema editor looks different after installing BizTalk. It is replaced during the installation to make the creation of document specifications easier.

Figure 1 shows a simple document specification for an XML file format. The tree view pane on the left is used to manipulate nodes in the specification, the schema display pane in the center is read-only and displays the resulting XML Schema, and the top right pane is the property page that provides the ability to change the attributes of the selected nodes.

Creating a document specification for a non-XML format is only slightly more complicated, with the primary difference being the additional configuration required to define element positions and delimiter characters or sequences. You can add the flat-file or other extensions to the current schema through the schema editor extensions property. When you add the extensions an additional page is added to the schema display pane, which allows you to view field information for each node (e.g., start position, stop position, etc.).

The addition of full support for XML Schema allows us to use namespaces to effectively manage document types. In BizTalk a unique document is defined by the combination of the namespace and root node type and is denoted as the concatenation of the two, in the following form:

mynamespace#myrootnodetype

It is common practice to set the root node name and root node type to the same value to make type identification more evident. We will use the document type information later when we look at using filters for high-performance messaging.

Once a specification is complete you can create instances of the document described by the schema, or you can validate sample data against the schema. Instance documents can speed development by providing sample documents to developers of target systems. Validating sample data allows you to spot format and data errors early, thus reducing the impact on dependent components. You access this functionality through the context menu of the selected schema, and you can select the file path for instance documents and input documents from the properties page.

Importing Documents
It is more common to have samples of message data than it is to have the specifications already in XML Schema form. BizTalk provides mechanisms for importing well-formed XML documents, XDR (XML-Data Reduced) documents, and DTD (Document Type Definition) documents. This can be a considerable time-saver when working with large numbers of document types.

In the current release of the product only the DTD Import wizard is installed. To install the other import wizards, run the two install scripts found in <install drive>\Program Files\Microsoft BizTalk Server 2004\SDK\Utilities\ Schema Generator. These scripts will add the Well-Formed XML Document and XDR Schema wizards to the Generate Schemas selection dialog box.

You can access the import wizards by right-clicking on any BizTalk project from the Visual Studio .NET solution explorer window, selecting Add|Add generated items, and then selecting Generate Schemas from the dialog box. Once in the Generate Schemas wizard, you can select the appropriate document (if you see only DTD schema in the dropdown menu, make sure the scripts mentioned earlier ran correctly), point to the input file, and BizTalk will convert the document to an XML Schema and add it to your project.

If you find yourself working with file formats other than XML, you may find it beneficial to invest in a tool such as XMLSPY, which can convert flat-file and database formats to XML. Once in XML format you can use the Well-Formed XML wizard to create an XML Schema from your document.

Transforming Message Forms
Systems and services rarely share identical interfaces, so once we have defined some of the messages that will be used to exchange data between our systems and services we can shift our efforts to creating the components that allow us to transform these messages into different formats. In BizTalk, transformation components are called maps. A map is simply an XSLT document that, when applied to a document matching the source schema, produces a document matching the destination schema. This is possible because all messages within BizTalk are in XML format.

Maps can get quite complex and often include data translation functions, loops, indexing, and lookup functions. BizTalk simplifies map creation by providing a powerful GUI and toolbox of canned functions, known as "functoids". BizTalk ships with over 70 canned functoids, and allows you to write your own so that you can reuse those special functions.

Since we are working with XSLT behind the scenes it is important to keep the following things in mind: maps can support only one-to-one transformations (with an orchestration you can do one-to-many transformations), and while you can perform basic indexing and looping functions, more complex programming constructs are not supported. To address these constraints and give us the flexibility to map virtually everything, BizTalk provides a scripting functoid designed to call an external .NET assembly or insert C#, VB .NET, JScript .NET, XSLT, or an XSLT template directly inline.

Once a map is created it can be applied to a message either in an orchestration or directly in a send port or receive port. I will discuss the direct application of a map in a send port later when I cover high-performance messaging.

Transporting Messages
A key feature of the BizTalk platform is the separation of physical message transport functions from other system functions. This allows us to configure message delivery points once and reuse them with different business logic. In addition, the responsibility for the physical interfaces is moved from the development domain and into the system administrator domain, where they can be managed more effectively.

BizTalk offers a number of transport mechanisms, including HTTP, FTP, FILE, SMTP, MSMQT, MQSeries, EDI, SOAP, and SQL, and can also be extended through the use of the adapter framework. With over 300 adapters available from third parties, BizTalk can connect to virtually any system. This is a considerable advantage, as a large portion of effort in integration is often spent on the development and configuration of physical interfaces.

Transport endpoints, known as ports, are divided into two types: send and receive. While port configuration is outside the scope of this article, I will discuss the use of filters and maps with the send port later.

High-Performance Messaging
As we have seen, BizTalk provides a plethora of powerful tools designed to make the development and operation of message-based solutions straightforward. There are instances, however, when common design patterns do not meet certain requirements, and a change in the design pattern is required. High-performance systems often require deviation from nominal design patterns.

The BizTalk architects considered high-performance solutions during the development of BizTalk and provided some additional features to improve pure high-perfor-mance messaging solutions. These features are inherent in the design of the BizTalk messaging components that implement the publish-subscribe design pattern.

BizTalk uses a universal message box construct to which all messages are published (this is actually implemented in SQL Server). Entities that wish to process the message data simply subscribe to the target message. A subscription agent component is responsible for fulfilling the subscriptions and deleting the message instances once all subscriptions are filled, enabling a one-to-many relationship to be created between message publishers and subscribers. This design pattern is very powerful and allows us to create solutions in which a single message is processed by multiple send ports, each using a unique physical transport, and possibly applying a unique map.

Configuration of send ports to take advantage of this "shortcut" is straightforward and can be performed through the port configuration dialog box as shown in Figure 2.

Adding a filter expression such as

BTS.MessageType == mynamespace#myrootnodetype

will create a subscription between this send port and the identified message type. When a message of this type is published into the universal message box, the message agent will fulfill this subscription by passing an instance of the message to the send port. Filter expressions can get quite complex, containing multiple rows in the filter dialog; you can even use information in the message itself by using promoted properties. This configuration dialog is also used to apply an outbound map to the message; map selection is found under the Outbound Map property.

Be careful when configuring messaging systems using filters; such a configuration can cause the generation of multiple outputs. Remember that the message agent fulfills all subscriptions before the message is discarded, so if you have multiple send ports all subscribing to the same message type, each send port will process the message.

Another common element in high-performance systems is the asynchronous design pattern, which optimizes performance by performing tasks in parallel. One example of this pattern is to split a large message composed of many repeating records into many messages, each containing a single record. This pattern allows each of the single record messages to be processed in parallel, providing a highly scalable solution.

You can split a message in BizTalk by using an envelope, which is simply a standard BizTalk XML Schema with the Envelope property of the <Schema> node set to true, and the Body XPath property of the root node set to the parent node of the repeating element you wish to split on. Splitting messages requires two schemas, one for the composite message and one for the individual messages. For example, suppose we have an XML message that contains the root node <Trades>, under which we have one or more <Trade> nodes. We would construct a schema in which <Trades> was the root node, and we would construct another schema in which <Trade> was the root node. We would then set the properties of the schema for the "trades" message as described above to indicate that it is an envelope. When the "trades" message is submitted to BizTalk, BizTalk will find the associated schema, which in this case is an envelope, and BizTalk will create a new "trade" message for each of the trade records found. Each of the resulting message instances will be processed in parallel as defined by the trade schema.

Summary
In this article you have learned how to create message-based solutions using BizTalk Server 2004. You have seen how to create XML Schema to define message structure, transform message formats using maps, and configure transport components. Finally, we looked at specific features of BizTalk that allow us to create very high-performance messaging solutions.

Microsoft has raised the bar with this release of BizTalk Server, giving us a powerful platform to take advantage of the explosive growth of collaborative, integrated enterprise applications.