SOA

Mobile devices, such as the Microsoft Windows CE-based PocketPC, have grown significantly in popularity over the last several years. In the car, on a plane, or out in the middle of nowhere, applications on the device can operate without a connection to any other computers, the Internet, or an intranet. Devices can then partner with a desktop system either via modem or network card, or simply by placing the mobile device in a cradle. New or modified data on either the device or server system is then automatically migrated and synchronized. More and more businesses, however, want to enable users with applications beyond basic e-mail and contact management. Businesses want to bring functionality from their enterprise applications down to the small and easily portable mobile devices. Microsoft CRM Mobile (CRM Mobile for short) is a mobile client for the Microsoft CRM (MSCRM) business application. It is a .NET-based rich client targeted at PocketPC 2003 (see Figure 1).

CRM Mobile Stats

• Completely developed in C#
• 150K lines of code ~40K on PPC with NETCF
• Metadata-driven business logic, user interface, and message schemas
• 4 SQL CE databases ~40 Tables; transactions coordinated across databases; schemas updated on the fly when metadata changes.
• Uses Web services and asynchronous messaging built on SQL CE RDA Protocol

CRM Mobile users have the ability to work in either a connected or disconnected mode. Like its desktop counterpart, MSCRM, CRM Mobile is easy to use, customize, and maintain. It offers users rich account and opportunity management functionality.

This article explores the CRM Mobile implementation based on a service-oriented architecture, specifically submitting transactions to and synchronizing data from MSCRM.

Service-Oriented Architecture
Before diving into the CRM Mobile's architecture, it is useful to understand the principles of service-oriented architecture (SOA). SOA aims to solve the problem of distributed application development. Generally, such architecture is defined as a set of service providers that expose their functionality through public interfaces. The general theme is one that has been popular for a long time in software development circles: separate interfaces from implementation. A powerful theme in SOA is the definition of individual service characteristics. These characteristics are what make SOA highly suitable for mobile applications, where client and server typically share very little in terms of hardware and software capabilities. A service can be defined as an application that exposes a message-based (asynchronous) interface that encapsulates its data and the associated ACID (database style 2PC) transactions within its data sources.

Services are typically constructed in four tiers: interface, business process facade, business rules, and data access. The interface is the bridge between clients of a service and the service's business process facade. A single service can have multiple interfaces such as Web services, queuing systems, or simple file shares. An example of the difference between a business process facade and a business rule is that accepting an order document from a customer is a business process whereas checking the availability of line items and calculating the sales tax are business rules. Services typically expose only coarse processes and not discrete rules.

What Defines a Service?

• It is a trust boundary. It does not trust clients and enforces all business and data integrity rules.
• It provides a coarse stateless interface i.e., UpdateCustomer (Customer) instead of Update CustomerName(id, string).
• It has a service-level agreement defining such things as up time and performance that are independent of other services and applications in the system.
• It is the system of record for some data.
• It encapsulates business processes and rules.
• It encapsulates one or more data sources.

There is a cost associated with constructing services with coarse interface and fully encapsulated business logic and data access layers. Building client applications becomes more difficult. Business logic is hidden from client applications and coarse interfaces are cumbersome to use. A solution comes in the form of agents, which can be defined as "smart proxies" that can serve as intermediaries between services and clients. One example of agent functionality is the Web shopping basket. The shopping basket functionality in the agent shields the client from having to build a complicated order request with multiple line items. The agent provides a very granular interface to the application that is easy to program against. It then aggregates multiple inbound application invocations into a single service invocation.

CRM Mobile Design
How does one take an existing business application like MSCRM and mobilize it? How can service-oriented architecture be effectively used? The following steps show one way to approach the problem:

1. Identify the application to be mobile enabled.
2. Build or identify a service interface that is accessible with the communication mechanisms available to the mobile device.
3. Identify which functions in the service interface make sense in a disconnected environment. Build these as one-way calls.
4. Identify which functions in the service interface make sense only in a connected environment. Build these as request response.
5. For the functions that are available in the disconnected scenario build an agent:
       a. Identify the subset of business logic, data, and reference data that the agent needs.
       b. Build a granular interface into the agent that hides the coarse service interface.
6. Make sure the service interface provides the ability to retrieve the data and reference data needed by the client in disconnected mode.
7. Build a user interface that uses the granular interface provided by the agent to execute business functions.

The resulting parts of a disconnected mobile application can be factored as shown in Figure 2.

Note that the service agent can mimic the service. It can have a subset of the business logic and data available in the service to help the disconnected application avoid errors and provide the end user with a better experience.

How does this apply to CRM Mobile? Starting at the back end, MSCRM exposes its business processes with a well-defined SDK but hides its business rules and data. The SDK for MSCRM supports Web service and COM interfaces, but does not currently support reliable delivery semantics. These semantics are a critical foundation without which disconnected applications are very difficult to build. The first work item for CRM Mobile was a reliable delivery service interface. The messaging API, called the Message Bus, is built using Microsoft SQL Server on the server and Microsoft SQL Server CE on the client. The RDA protocol available in SQL Server CE is used to transport bits reliably over HTTP from source and destination queues built with database tables. There is a programmatic interface on each side that supports sending and receiving messages from the queues. Messages are routed using custom implementations of standards schemes like WS-Routing and WS-Addressing. The full details of this API are beyond the scope of this article, but it can be thought of as a bidirectional queuing system that supports reliable, once, and in-order delivery of messages.

The layer above the message bus is the proxy/stub layer. Proxies and stubs are defined for each of the business process APIs defined by MSCRM. These proxies and stubs serve the same purpose as their Web service counterparts; they serialize a method invocation into the Message Bus format and then deserialize it back into a method invocation.

Transactions are sent to the server via the message bus proxies as method arguments. They are constructed as an XML document consisting of a verb and a Diff-Gram. The verb in the transaction identifies which operation to invoke on the server like Account-Create while the Diff-Gram portion is used to communicate before and after images of the modified records to facilitate optimistic concurrency.

On the client side, there is a service agent on top of the proxy/ stub layer that mimics some of the business logic implemented in MSCRM. Data validation rules are run by the agent as well as business rules that depend on single record state. The agent maintains an offline store that houses the same data as MSCRM, but whose schema is significantly different from the MSCRM offline store. The client store is implemented with SQL Server CE. The interaction diagram in Figure 3 is an example of how the agent forms a bridge between the client application and MSCRM.

In Figure 3, the service agent provides the following functionality:

• Construction of a dataset: This dynamic instantiation is necessary because MSCRM schema can be changed on the fly.
• Validation of a dataset: This is necessary because validation rules can be customized in MSCRM.
• Creation and submission of a transaction: The UI developer is shielded from the complexities of a service-oriented interface.
• Local simulation of remote business logic for a better user experience: The real state of the created account will not be known until the client synchronizes its offline store with the server.

Once transactions have been submitted, the next task CRM Mobile faces is to update its offline store based on the current state of MSCRM data. Because the schemas on the client differ from the schemas on the server and because MSCRM does not make its database publicly visible, the SQL Server CE Merge replication functionality is not used to replicate data. A custom data synchronization service exists on the server. It uses two synchronization mechanisms: global version?based synchronization and object tracker?based synchronization. The first schema is used for reference data, in which all clients sync the same data and the second schema is used for business data, where each client subscribes to a different set of data. Figure 4 shows how the MSCRM and Synchronization services exist as two separate entities on the server.

CRM Mobile defines the setting of data subscriptions as a strictly connected and interactive scenario. Users can only change subscriptions while connected because an integral part of the operation is a real-time estimate of what the new data size will be. The probability that the new subscribed data will overflow the device must be estimated to avoid wasted data downloads. Because of this and the fact that the process has to be interactive, standard [unreliable] Web services are used instead of the asynchronous reliable message bus. Figure 5 shows how CRM Mobile allows a user to create data subscriptions.

Once subscriptions have been set, data based on subscriptions must be synchronized. Figure 6 outlines this process.

Although simplified in the diagram, data synchronization is a complex problem. There are various flavors of synchronization, i.e., single master and multi-master. When developing disconnected mobile applications, data synchronization is usually a one-way operation from the server to the client. The basic problem in synchronization is in determining what deltas to send to the client. The deltas can be of three basic types:

1. Tombstones (delete a record)
2. Updates
3. Creates

There are many scenarios of data usage that can affect how deltas are calculated:

1. Do all clients synchronize the same data?
2. Does each client have a different view of the data?
3. How often does the data change?
4. How big is the data?

If all of the clients synchronize the same data, then data can be versioned at the record level and a global version can be kept. Each record can be assigned to create, update, and delete versions. The versions can be created and kept up-to-date using the schema and business logic extensibility mechanisms available in most server application platforms. There are, however, scenarios where such a replication schema is not feasible. One example of this is where the service does not support schema extensibility or transacted system callouts. Data synchronization by maintaining server copies of client data is another way to synchronize data that does not need schema and business logic alterations. This mechanism is also useful when each client subscribes to different sets of data. Here is how the synchronization is done:

1. The end users define subscriptions for their clients. These subscriptions are queries that the service can process to return a set of data when the client requests synchronization.
2. A client object tracker database is defined that contains a client key, a unique key for a record, and a tombstone bit.
3. When the client requests synchronization, the saved queries are run and a data set is returned.
4. Each record in the data set returned is compared with the client object tracker representation of that record.
   a. Before processing begins, the tombstone bit for all records in the object tracker is set to one. This means that it is assumed that all records are deleted.
   b. If the unique key for a record in the returned dataset is not found in the object tracker, then the key and the modified time is copied into the object tracker. A delta in the form of a create is sent to the client. The tombstone bit for the object tracker record is not set.
   c. If the unique key for the record is found in the object tracker, then the modified time in the query result is compared with the object tracker record. If the modified time has not changed then no delta is sent to the client. If the modified time has changed then an update is sent to the client. In either case, the tombstone bit for the record is set to zero.
   d. After all of the records have been processed, those that still have their tombstone bit set are sent as tombstones to the client. Calculating tombstones in this way takes into account when objects are deleted, when they move out of subscription i.e., the stored queries no longer select them or when the client is no longer authorized to select them.

Conclusion
Developing Microsoft CRM Mobile using service-oriented architecture was a challenging task, as the descriptions of the transaction submission and data synchronization show. The benefits of this approach, however, were that the entire project was completed without any mobile-specific lines of code in the MSCRM server application. Because of the chosen architecture, CRM Mobile solely relies on public interfaces of MSCRM and can be maintained separately from the server application. This fact allows CRM Mobile and MSCRM to be maintained by two completely different and separate teams at Microsoft. While using service-oriented architecture is not trivial, its ability to force the correct factoring of applications can be invaluable. CRM Mobile also shows that the Microsoft software stack on the PocketPC and other CE Devices is ready for prime time.