In the five years that I have worked in Web solutions practices, a typical business problem has changed from "we need a new Web site" to "we need to regain control over our existing sites." It's not uncommon for large corporations to have hundreds or even thousands of different Web sites spread over various service lines, geographies, and organizational boundaries. This presents challenges ranging from logistical and technical, to creative, business, and legal. This article focuses on solving the problem of ubiquitous navigation across diverse Webscapes.

In the Beginning There Was the RFP
Let's start with a real-life problem. A global financial services company whose products you probably carry in your wallet identified a need for a ubiquitous navigation utility. They have to manage myriads of intra- and inter-site references spanning nearly 1,000 of their Internet, intranet, extranet, and partner Web sites across the globe. It should maintain the infrastructure of navigation assets, such as familiar top, left, and bottom navigation bars, site maps, navigable taxonomies, features and highlights, inline ads, and cross references, while ensuring that all content is reachable and there are no broken links.

Being a large company, it issued an RFP and received numerous responses, mostly from product vendors, including a Web testing tool and several portal and content management platforms. Unfortunately, none of them would satisfy all the requirements. CMS is able to publish the assets and can do some initial validation, but cannot maintain the validity of links at runtime. Portals don't have that problem, but can only maintain links to their own pages. Site crawlers can detect broken links only after they appear on the sites.

My team was the only services organization that responded to the RFP. To differentiate against stronger competition, we needed to come up with a perfect solution that would satisfy all the requirements, while delivering comparable or better scalability, maintainability, and, ultimately, value. Thus we came up with the idea of a target-centric navigation solution.

Target-Centric Navigation Solution
The only way to ensure that the site never displays a broken link is to validate all the links at the page rendering time, suppressing the ones that point to unavailable resources. The simplest way to do it is to use classes from package java.net as shown in the following code:

public static boolean isBrokenLink(String linkURL)
{
  try
  {
   // Throws MalformedURLException if //invalid syntax
   URL toTest = new URL(linkURL);
   // Throws IOException for some URL //types
   URLConnection c = toTest.openConnection();
   // Throws IOException when site not //found
   c.connect();
   // Throws IOException when page not //found
   c.getContent();
   return false;
  }
  catch (MalformedURLException ex) {}
  catch (IOException ex) {}
  return true;
}

However, a single page can contain dozens even hundreds of links, many of them pointing to dynamic pages, so validating each one using the above method would result in very poor performance. Furthermore, this method will not work at all if the target site intercepts its own 404 errors and redirects them to a not found page. A better solution would be to create a Target Registry containing records about every navigable target, including pages, documents, navigation nodes, and media streams. Every navigation link on the page can then be validated just before rendering with a single query to the registry, ensuring optimum combination of performance and reliability.

Sometimes it is not sufficient to remove a broken link by itself - when it is accompanied by the contextual information, such as text or images, the entire page element should be removed. A good way to implement navigation assets is to assemble them from contentlets, or elementary units of dynamic content presentable on a Web page, as introduced in the article I co-wrote with Alexey Yakubovich about Vitrage framework, "Development of Component-Oriented Web Interfaces" (JDJ, Vol. 10, issue 3). A contentlet has a number of core attributes: Name, Description, Image, and Display Order. Additional attributes, such as Date, Type, and Source, may be represented as needed for a particular problem domain or implementation. All of these attributes are optional. Content Reference is a type of contentlet that points to a specific URL. As shown in Figure 1, there are several concrete types of content references pointing to different types of targets, such as dynamic pages, documents published through CMS, external links, and media streams.

Due to the diversity of possible targets, the target registry can be realized as a single entity or as a logical collection of target-specific registries. PageRegistry would contain the information about all dynamic pages in the system. It usually has to be implemented from scratch; however, it is well worth the effort because it can be used not only to support navigation, but also for search, security, automatic site maps and indices, supporting page life cycles, etc. Document Registry or CMS_Cache usually comes as a part of the Content Management System and does not require any custom implementation. Approved External Links store can be implemented as a part of PageRegistry or on its own, and similarly has uses beyond navigation. Regardless of the implementation details, use of target registries would allow bulk validation of all navigation assets on a page in a single though possibly complex query.

Implementation Considerations
The Links
Content References are flexible enough to implement any navigation assets found on today's Web sites. They have a built-in validation mechanism to prevent rendering of broken links at runtime while maintaining the correct structure of the remaining page. When used in conjunction with a compartment-oriented presentation framework such as Vitrage, which allows bulk loading of all contentlets used on a page in a single database hit, content references have the same or better performance than any other dynamic navigation mechanism.

Navigation compartments can be easily extended to non-Java technologies, such as static, PHP, and even ASP pages. You can define a special type of compartment, which, instead of generating HTML into a ServletResponse, would rewrite a predefined region of an external file.

If you opt to not use compartment-oriented presentation framework, you can still implement a deferred contentlet-based validation through the use of a Response Filter from package javax.servlet. To use this mechanism described in detail in "Adding Internationalization to Business Objects" (WLDJ, June 2005), each contentlet should be rendered surrounded with meta-information tags containing references to the target registries. The response filter then parses ServletResponse, finds all the tags, validates them against the registries, and removes the tags along with invalid content references.