Industry Commentary

XML is changing the business world by making integration easier. You can't pick up a computer industry or business magazine without seeing an article about the impact of XML on large software infrastructure companies such as Microsoft, Oracle, and SAP.

The stories are similar: a company representative describes a proprietary form of XML messaging that makes integration with other products easier, followed by a pledge to support a new global standard. The key points of the article are usually: their product is the center of your IT universe; they recognize your need for functionality they don't currently sell; and this functionality is in their next version.

Improving enterprise applications and easing integration is a huge step in the right direction, but I still need a pundit from one of these major software firms or a consulting and system integration firm, such as KPMG, Price- waterhouseCoopers, or EDS, to help me understand MI-XML. How does easy integration help me get beyond the limits of my current application infrastructure?

Consider how a company like Crossmark, a Plano, Texas-based sales and marketing organization, must integrate their services into each of their 1,100-plus clients' enterprise applications to complete seamless supply-chain management for each client.

Crossmark is part of the consumer product goods (CPG) industry. Their target market consists of major retailers and product suppliers to whom they provide customized sales and marketing services such as headquarters sales, decision support, and in-store representation. These services are supplemented by administrative support for selling and marketing activities, such as order processing, as well as local area and in-store marketing programs. A key characteristic of Crossmark's service offering is the use of proprietary technology systems to improve existing applications and create new processes.

Headquarters sales is a service in which Crosssmark provides the sales team for consumer products companies such as Pillsbury. This outsourced sales force is focused on selling products to large retailing companies such as Kroger, Safeway, and Wal-Mart. Headquarters sales service spans more than 60 offices in four countries and consists of approximately 1,000 salespeople. In-store support teams provide sales and merchandising support at more than 150,000 retail outlets each month. There are approximately 4,000 employees to coordinate with client and outlet-specific objectives every 28 days.

The order processing service handles 1.5 million sales transactions annually, representing approximately $12 billion in gross sales. This service is a bit of a misnomer since Crossmark's role in order processing is to perform price and deal verification as well as order reconciliation. This service requires that Crossmark keep up with the product, price, promotion, trading terms, and transmission capabilities of its clients and their customers. This is accomplished by managing the pertinent information for more than 30,000 store- keeping units (SKU), which are sold by their 1,100 clients and purchased by roughly 300 retailers.

It probably wouldn't surprise you to know that Crossmark supports countless versions of electronic data interchange (EDI) as well as fax, telephone, e-mail, regular mail, and many new forms of intercompany communication, all in the pursuit of e-commerce. Most industry participants manage their information regarding the sale, payment, and movement of their products on proprietary product numbers, not the industry standard universal product code (UPC). The usual issues of ordering, unit translation, and lead time, and the customary difficulties associated with shipping, also create difficulties in performing this service.

The communication and coordination requirements associated with Crossmark's business are daunting, but with the major software vendors extending the interoperability of their products through traditional APIs and COM- or Java-based components, and adding XML-based messaging, most of the integration problems are easily solved.

In this new world of easy integration that extends the scope of automated business processes, what does the structure of a database have to look like to accommodate the data sets that facilitate functionality between so many applications? How can XML be easily and fluidly stored and managed?

Most major companies have implemented some enterprise resource planning (ERP) modules. Crossmark implemented General Ledger, Finance, and Human Resources. The data structure supporting these three modules alone is 1,281 tables containing 25,775 columns. Imagine what the numbers would balloon to if we implemented CRM, Procurement, Forecasting, Asset Management, Risk Management, Fleet, Training, and Logistics.

Atomization of Business
Organizations like Crossmark are fast becoming the rule, not the exception. The CPG industry has been in a state of consolidation for years. During the last few years, globalization of retailers and product manufacturers has accelerated. These larger entities are relying on an ever-growing number of service firms to provide specialization. Smaller firms operate much the same way a software component does. They usually perform a limited number of functions and need to be used in combination with other components to provide solutions. This effect has been referred to by Cap Gemini Ernst & Young (CGE&Y) as the "atomization" of business. Atomized supply chains require an increase in the amount and type of information communicated up and down the supply chain.

Is it any wonder Crossmark has a performance group (PG) division, whose sole purpose is to investigate emerging technologies? As the president of this division, my mission is to find technologies that can support rapid development methodologies needed to provide customized services on a global basis. In order to support the vision of mass customization, we look at software development in much the same way an oyster makes a pearl: accretion, or by gradually adding calcium carbonate to a grain of sand. The word accretion is closely related to incremental. Incremental design, building, and testing are the software development concepts used.

The problem with this thinking is that rapid and incremental designs aren't easily achieved with today's technology, no matter what language you use or what development methodologies you believe in. Using today's relational databases as the foundation for enterprise applications means lots of up-front time, understanding, and planning for all possible issues surrounding the application. You have to make the design as accurate as possible so you don't waste time and money fixing mistakes that could have been avoided.

Data from a study conducted in 1988 by Boehm and Pappacio shows that a change in requirements or architecture in the early stages of a project costs 50-200 times less than the same changes made later in construction or maintenance. The cost balloons to over 1,000 times once the application is deployed. Another study performed by Boehm in 1981 indicated change is typical. Boehm looked at roughly a million instructions and found that an average project changes 25% during development. The reason planning is such a large factor in the cost of software is because building a software application is a learning process for both the programmer and the users.

Mission impossible is finding a technology that allows you to use rapid development techniques without incurring the time required for database design and redesign. Think about being able to take an idea for an application and creating the most basic version of the software using only the data elements required to support minimal functionality. You deploy the basic application and the user generates a list of improvements that require a complete redesign of the database. However, the user wants to continue with the application in its present form until the improvements are developed and deployed. You probably think this approach is dead on arrival. I'd have agreed with you before I discovered the...

NeoCore XML Information Server
NeoCore has adapted and extended its core technology to deliver a database server that stores, retrieves, and modifies XML data. This unique and exciting product manages XML as patterns of information instead of traditional structures such as rows, columns, objects, or DOMs. The current server implementation supports functions that:

• create a new XML document in the store.
• Insert a well-formed XML fragment into an existing document.
• Modify the textual contents of an existing attribute or text element(s).
• Delete an existing XML fragment(s) or document(s).
• Copy an existing XML document.
• Query an XML document(s) or fragment(s).
• Allow different XML tags with the same semantics to be treated together with the use of aliases.

The XML Information Server is based on NeoCore's patented Digital Pattern Processing (DPP) technology, a new information-processing technique that generates icons for raw data and associates them with known patterns at up to 10-million matches per second. Hundreds of thousands of transactions per minute can be made on lower-cost equipment, reducing the cost per transaction to pennies on the dollar. The NeoCore Information Server is designed from the ground up as an XML purebred. Its performance, fluid handling, and scalability for XML are unparalleled.

The server uses DPP to manage information in a new way. Rather than using rows and columns or predetermined indexing instructions to manage data, DPP allows information to be managed in an entirely dynamic way. Pattern matching is used to allow data and data types (metadata) to be managed in real time; traditional databases treat metadata in a static way, requiring it to be defined in advance. The need to design the database to accommodate applications can be a significant portion of any rapid application development project. Subsequent changes can be a major headache. The NeoCore XML Information server eliminates the need to design or configure the database management system altogether because data types are treated in the same dynamic manner as data elements. Changes require no reconfiguration of the database whatsoever.

The server allows data types to be allocated on an individual basis. A new data type can be added to a single record without having to be globally added to all records. Because DPP locates information by matching patterns (and patterns can be arbitrary), there are no restrictions imposed on the amount of data - or the number of data types - that can be managed.

The NeoCore XML Information Server was introduced in October 2000. In September EE Times honored Chris Brandin, NeoCore's CTO and DPP's inventor, in a special edition of the magazine entitled "Architects of the Internet."

NeoCore minimizes XML's two largest weaknesses:

• Speed: Computers are not particularly good at string manipulation. A particularly amazing feature of the NeoCore XML Information Server is that for direct-access single queries, search performance remains essentially flat regardless of the size of the XML store. This has interesting implications for high-volume transaction processing.
• Accessibility: A lot of customized programming is required to make XML data structures behave like relational database management systems (RDBMS). In contrast to an RDBMS, the NeoCore XML Information server is schema-independent. The dynamic nature of XML is preserved. No prior definition is required before a developer can store and access arbitrary XML data and retrieve it in arbitrarily sized chunks, from single elements to complete or multiple documents. In fact, XML can change on the fly without breaking the database. This allows transactions and workflow to change harmoniously as required by day-to-day business.

NeoCore allows Crossmark to build applications specifically for the one client (either internal or external) who requests them, without considering the needs of other clients. Using the XML Information Server, we can modify the underlying data structure of any application without taking the application out of service or creating a duplicate infrastructure.

The first product Crossmark is building using the XML Information Server is a knowledge management system. It's intended to be a framework that can be freely and easily expanded upon with additional building blocks, functionality, and data sets that might or might not yet be conceived. This ability to append additional functionality and information, whatever form they might take, without breaking existing functionality or impairing performance is just one of the goals of the application.

The main goal is to create a system that can aggregate, classify, link, and categorize information from myriad sources (including documents, free-form text, XML, e-mail messages, web pages, postings on Internet bulletin boards, and structured and unstructured data records). This will provide useful actionable information and conclusions. Some additional goals of the project are to test the validity of my three assertions regarding NeoCore's product:

1. To create an XML data repository with capabilities that can grow without breaking
2. To ascertain how easily and rapidly development against a relatively free-form XML data repository can be as opposed to a structured relational database
3. To investigate the limits of NeoCore's XML Server with respect to speed and scalability

This article is the first in a three-part series. Part 2 will detail our application development effort and describe the experience of working with the XML Information Server. Part 3 will and detail the results of this effort.