Often-Overlooked Problems

part of Perl for the Web

While performance concerns are paramount in this book, you also want to do real work with a Web application. This often means that the Web application has to cover areas that other applications can ignore; after all, a Web site devoted to workflow or other high-level concepts requires all pieces of the puzzle, not just a few.

One difficult aspect of complex computer environments is legacy applications that need to be brought into the Web site framework. Mainframe and custom applications are designed to be accessed from specific hardware systems with narrow requirements; thus, applying a more robust interface on these systems for use by Web applications can range from difficult to expensive to impossible.

Licensing issues also come up when developing Web applications because offering a service that relies on other servers can create additional usage and incur costs that weren't anticipated when designing the Web application. If Web developers aren't aware of licensing concerns, they can incur unanticipated costs for an organization simply by making a connection between applications or by making an application available to a new class of users. Some licenses protect against such concerns, but it's usually not the Web developer's role to determine whether this is the case.

The general idea behind enterprise integration and automation overarches these concerns. Every technical advance tends to promise integration of all business processes into one efficient structure, but implementing such a system often results in almost as much work created as is automated away.

Legacy Applications

Existing applications can be a real pain to integrate into a Web environment. It's not so bad when the applications can be accessed over the network through an established protocol, but applications that assume a human interface can be nearly impossible to access in any other way. Nevertheless, many organizations still use legacy applications for critical systems, so you often need to add these systems into the mix when designing a robust Web application. Although it takes some custom programming and a little creativity to solve these kinds of tricky integration problems, it is possible to solve them.

Mainframe applications are some of the hardest to integrate. Many were developed at a time when all users accessed the system with the same proprietary interface, so there was no need to provide alternate interfaces, let alone automation. Most mainframe development environments assumed a pre-Internet network that had no reason to integrate with other network services because those services were all running as part of the mainframe environment itself. The interface to such systems was simple–a screen-by-screen display that was dictated verbatim by the mainframe application, with input provided by a user at a terminal. To add a Web perspective to this environment, you had to build an additional interface over the human-centric text interface, with a simulated terminal for input and data skimmed from the screen-by-screen results.

Other custom applications can be just as difficult to integrate because they're completely opaque and have only one thin interface. In some cases, the application is compiled, and it provides few clues about how it could be modified for use in a new environment.

Terminal-Based Mainframe Applications

Mainframe applications, the hard-working dinosaurs of enterprise server environments, still can be found operating critical systems at large organizations around the globe. Financial applications, power regulation systems, and a whole host of other mission-critical applications have been written for environments such as CICS, which was developed for use with IBM mainframes. These environments have been updated over the years to interact with more modern hardware and software, but they're still not easy to integrate into Web applications.

Although some mainframe applications are designed to be accessible by common network protocols such as Telnet or remote procedure call protocols, others use proprietary interfaces that require emulation to provide any automated interface at all. Some IBM mainframe applications use the 3270 terminal protocol, for instance, which has many emulators available for use as direct-access clients, but few ways to integrate such applications into a more robust automated interface. The assumption made when developing these systems was that all access would come interactively by way of terminals, not automatically by way of client programs.

To integrate with mainframe applications from nonstandard client systems, developers commonly use text-filtering systems that process the screen-by-screen results of terminal interaction. These systems are generally called screen scrapers because they have to extract meaningful data from a screen's worth of undifferentiated information–an inefficient method at best. Screen scraper interfaces are some of the least robust interfaces possible because of the many assumptions that have to be made when determining what data from each screen is important. Very little can be done to catch errors or provide for unusual situations with a screen scraper because the interface definition is so arbitrary. If the third through fifth characters of the second line of the fourth screen are used to indicate the three-digit product code of a widget, for instance, the entire interface could be thrown off by having four-digit product codes come into use after the interface is developed.

With this kind of environment, it would be too much to ask for any sort of data markup, let alone adherence to standards such as Extensible Markup Language (XML). This is unfortunate because the data contained in these legacy systems is sometimes the richest source of integration potential an organization has available. In addition, making that information available by encoding it in XML would be a boon to many future applications in a wide variety of uses.

Custom Applications

Custom applications are usually written to fit one purpose within an organization, often because no other software performs that task. A custom application to capture technical support information from phone technicians, for instance, could be written in C++ or Visual Basic with networked access to a central support database that is custom-written for the purpose. Such an application could be a poor fit for off-the-shelf server environments such as Oracle or Lotus Notes if it needs to interact with phone systems or employee time clocks. It might be more cost-effective to build the application from scratch than it would be to assemble it from expensive server products.

Like mainframe applications, custom applications generally have no interface other than the user-centric controls provided for the environment in which the application was developed. A point-of-sale (POS) application, for instance, is usually developed to run on specific cash register hardware that has only basic network connections to a POS server on the same network. It might be difficult to abstract the interface to system functions away from code that operates the hardware, and it can be virtually impossible to simulate the terminal environment because the interface is written specifically for touchscreen terminals.

Even worse, there is often no vendor to contact for support for adapting a custom application for Web use because the application is usually developed in-house. Because the pace of information technology changes causes frequent shifts in the workforce, programmers who no longer work there might have written the application. In some cases, little documentation is available for extending the application because emphasis was placed on covering user-interface issues instead. Whatever the cause, it can be difficult or unreasonable to adapt an application for use on the Web because the application itself would be too difficult to modify.

Database Connections and Licensing

Licensing issues can be tricky even when a server application is being used in a traditional framework. However, Web applications provide additional twists that make licensing even more difficult to keep track of. With traditional application use, it's possible to keep track of users who access an application and restrict access to only those who are authorized in terms of the end-user license purchased from the vendor. Functionally, this is usually rolled into the same server restrictions that provide security and access control by tracking named users that have limited access to a central system. Server licensing is usually provided in terms of either the total number of named users or the number of users allowed to access the server simultaneously, with some servers enforcing license restrictions on both.

When using Web applications to access these same servers, however, the distinction between users becomes blurred and the need for simultaneous access can become more of a performance issue than overall server usage. In many Web application cases, large numbers of anonymous users access the application sporadically with varying frequency, so it's difficult to separate these users into named groups, let alone named users. Even if that were possible, it would probably lead to an explosion in the number of named users registered for system access. This explosion would occur even though most of the users would have accessed the system only once for a short period of time. As a result, most Web applications provide their own layers of system access and conglomerate all Web access into a single named user with multiple simultaneous connections to each server system. This goes against the vendor's licensing model, though, so many vendors have changed their policies to reflect Web usage more realistically, with varying results.

Microsoft and MySQL have two different ways of looking at the solution, but both approaches are opportunistically trying to get the most out of Web environments. Microsoft changed its SQL Server database product from a user-centric licensing model to an enterprise model to capture more licensing dollars from Web use. They did this after realizing that most Web application developers were getting by with the least expensive licensing option, regardless of the amount of Web traffic they were processing using the product. MySQL AB, on the other hand, went in a different direction with their MySQL database server. MySQL was originally licensed based on platform and use, but concerns by Web developers convinced the company to make the server available under an open source license.

Microsoft's View of the Web

When Web applications started using databases as backend storage, Microsoft didn't realize that this kind of usage presented a special concern for their SQL Server product. Like Oracle, Microsoft was licensing SQL in terms of a per-processor cost, plus an additional cost for each named user accessing the system at any time. Each named user would have a Client Access License (CAL), so a Web application developer had to purchase additional licenses as the server was used by increasing numbers of people within the organization.

This licensing model assumed that client applications accessing the server would each require a CAL from the user operating the application, but Web applications were designed to use the same named connection for all requests. Thus, the Microsoft pricing model enabled Web sites to operate using the cheapest licenses for SQL Server because only one named user was ever invoked at any one time. The server enabled a single named user to open multiple simultaneous connections from an application. In addition, the Web application could submit all its requests from the same user, so there was no reason to purchase a license for more than one user. Because the price for these licenses ran into the thousands of dollars, it was possible to save a considerable amount of money by routing more information through the Web application. Also, new e-commerce businesses were getting more mileage out of cheaper licenses by purchasing only a single processor license and a single CAL, and then spending the money left over on improving the performance of the server hardware itself.

Microsoft was concerned that a new class of applications in a growing market was capable of bypassing its standard pricing model. To close the pricing loophole and make up for the additional usage being seen by SQL Server in Web environments, Microsoft introduced a pricing addition called the Internet Connector. The Internet Connector provided no additional functionality for SQL Server, but it let Microsoft differentiate between servers that were allowed to process Web requests and those that were not. Simply put, it was an Internet tax levied by Microsoft against any application accessing SQL on behalf of an Internet user. Servers required an Internet Connector license for each processor in addition to the base license and CAL user licenses. Thus, Web use was clearly defined as something that required additional payment to Microsoft, regardless of whether other licenses had been purchased.

Later, Microsoft changed the pricing model to mask the tax-like nature of Internet licensing of SQL Server by offering an additional tier of per-processor pricing. Now it's possible to purchase either a traditional license with per-server and CAL costs or a per-processor license for SQL Server that does not limit the number of clients. The latter license covers both traditional users and Internet use. The per-processor license is considerably more expensive, however, so the tax has turned into a multiplier that makes SQL considerably more expensive for Internet use than it is for traditional use.

For Web application designers, a difficult aspect of these licensing issues is the fact that the software always operates the same, regardless of whether it's violating its license. In addition, because the licenses have little effect on the software that's actually installed, Web applications can access SQL Server, regardless of whether the Internet Connector or later licenses have been purchased. This puts Web application designers in a precarious position; it's possible that adding an integration layer to a Web application using systems that are already fully licensed might violate the license of those systems or incur additional costs above those of Web application development. Because most Web developers are not part of the license procurement process, it's possible for a developer to inadvertently incur thousands of dollars worth of additional licensing simply by making a server more widely available or using it as part of a Web application.

MySQL's View of the Web

MySQL is a database server designed to provide a lightweight, high-performance alternative to Oracle or SQL Server. Using MySQL for basic Web applications is a natural choice because many of the environments used for Web development–including Linux distributions and Perl modules–either included noncommercial copies of MySQL or assumed it was available.

Unfortunately, licensing concerns came to light when applications based on MySQL were put into production. The first concern involved the fact that MySQL was freely licensed only on a noncommercial basis; this meant that development use of the MySQL server was free, but that the software had to be purchased as soon as Web applications based on it went into production on commercial Web sites. Making matters worse, the line between commercial and noncommercial had been fuzzy on the Web because some noncommercial sites were offered by commercial entities and some noncommercial sites ended up raising money for one purpose or another.

The second concern involved the fact that MySQL was licensed differently for Windows and UNIX platforms, so adapting Web applications for use on a new platform had the potential to incur new licensing fees. This made Web application developers wary because the possibility existed that the free license under which they developed would go away at some point in the future; the free license would be replaced by a paid license if they crossed some boundary between platforms or between commercial and noncommercial use.

MySQL AB, makers of the MySQL database, answered the licensing concerns of Web application designers in a different way than did Microsoft; however, the motives behind each were rooted in profiting from increased Web use. While Microsoft offered unlimited Internet usage at a premium, MySQL AB offered their database product as open source under the GNU General Public License (GPL). This unified the licenses for Windows and UNIX platforms and enabled both commercial and noncommercial use of the software on any platform, without additional fees or licenses.

This change opened up MySQL to even broader groups of application designers because it now could be included with all sorts of application environments and used by any Web application, with no fear of later restrictions or unnatural distinctions between platforms. MySQL AB hopes that the increased use of MySQL fuels the need for support licenses and consulting contracts, which eventually should offset the costs of developing and marketing the software.

Although this particular situation turned out for the best for Web developers, it underscores the broad range of licensing possibilities each integrated application presents over its lifetime. As Web services and application syndication become more prominent, the line between an application user and a product reseller will blur even more; thus, it will be more difficult for Web application developers to consider all the licensing problems that might crop up if an application gets used in a previously unanticipated fashion.

Business-To-Business Automation

Enterprise Application Integration (EAI) is a hot topic, especially with companies trying to Web-enable their applications and capitalize on the success of Web interfaces. With each new Web interface comes the need to integrate it with other applications' Web interfaces; thus, more and more applications are coming under the domain of the Web developer. This unprecedented access to enterprise applications from a central interface makes it possible to integrate business processes that were previously inaccessible within an enterprise application framework. This makes the overall process more efficient.

Vertical markets, however, pose a big challenge when creating an integrated Web application to bring partners and suppliers together. Because control can't reasonably be exerted over the software and hardware choices made by each company, the need for standards and common implementations becomes more crucial. At the same time, the special needs of some vertical markets pose particular challenges to Web developers trying to fit a robust infrastructure into a Web interface.

On-Site Integration

EAI creates an interesting opportunity for Web applications. Its goal is to integrate and automate all applications used by a business, including business logic, data storage, communications, and groupware applications. Because Web applications provide a good network interface to traditionally server-bound applications, the whole is greater than the sum of its component applications.

Web applications can perform tasks that were previously possible only with lots of custom programming. In addition to integrating applications that are already being accessed individually, Web applications can also integrate network processes (such as email) and physical processes (such as fax or print). With these processes, it's possible to use a Web application to bring people into the process that aren't otherwise using the enterprise applications.

As an example of integration, consider an ordering system. It can take orders from a Web interface, use business logic to determine a pricing model, record the order in an order database, make inventory requests to a stock management system, create a customer record in a Customer Relationship Management (CRM) application, email an invoice to the customer, and fax a packing slip to the fulfillment company. Because each step happens within a system provided by a different vendor or internal department–and some of them happen outside the system entirely–Web-based EAI can provide a new level of continuity and reduce the need for human intervention in each step of the processes.

Unfortunately, many Web application development environments are not built with EAI in mind. They're designed to build dynamic consumer Web sites and are sometimes capable of little else. Some support is generally provided for network activities such as email or socket access, but rarely is there support for Web-client simulation or off-network systems such as voice or fax. Most Web development environments have started to support databases for backend data storage, but integration with other enterprise systems usually requires more custom programming, if the development environment allows it. This widens the gap between what can be done within the framework of mainline Web application development and what will require more in-depth programming, which adds additional time and effort to a process that usually has little to spare.

As a result, developers who want to add enterprise applications to a Web environment usually wait for the Web environment or the application vendor to provide an interface between the two. Enterprise application vendors are starting to provide Web interfaces to their products, but in many cases, the vendor confuses Web integration with the mere integration of a Web server and the provision of an HTML interface to the application. For Web applications looking to integrate enterprise systems, interfaces such as these are nearly impossible to automate, and thus are as useless as having no interface to the application at all.

Vertical Integration

Integration with business partners and suppliers is another focus of many businesses on the Web. After internal business processes are integrated into a coherent Web application, it's useful to automate the relationships that the application has with other companies. Much attention has been given to supply-chain integration for companies producing and selling products. The integration provides automated interfaces for each supplier, shipper, reseller, and support organization to streamline the order fulfillment process.

Other types of companies and organizations also benefit from automated interaction with their suppliers and consumers. For instance, news agencies and other content syndicators on the Web were early adopters of this idea, offering content to aggregators and portals through automated network interfaces. Companies that haven't used the Web for internal applications to date are starting to take notice of the potential this idea creates for their partners' applications and associated data.

Reams of data are being passed from company to company and from company to consumer, so there's a pressing need for systems to process and deliver it all. Just a few years ago, it was unusual for a company to provide a Web interface to its order system, even if the company had a Web site; now, companies and consumers are used to the availability of Web applications for business-to-consumer use, so they want the same ease of use when dealing with partners and suppliers.

Some vertical markets have unusual needs that require more than just interaction with suppliers and partners. Medical invoice and insurance tracking, fault-tolerant messaging for infrastructure monitoring servers, and similar high-importance systems are being developed to enable any market to benefit from the same kind of integration and ease-of-use that existing Web applications enjoy. With these advances come difficulties for Web application designers, however, because traditionally lax standards for Internet data security and robustness are clashing with the accountability and security needed by these new markets.

XML is a promising technology that seems to provide a standard way to encode and distribute application data, but few vendors seem to have a handle on what to do with XML and how to deal with it from an integration standpoint. Many companies pay lip service to the XML idea, but few are actually implementing solutions by marking up data and dealing with other marked-up data in a standard fashion. The flux surrounding XML implementations is creating additional uncertainty for Web application developers because vague standards and changing implementations create an environment that is both difficult to avoid and tough to work with. Further discussion of the future of XML for application integration is available in Chapter 18, "XML as a B2B Interface."

Summary

Although integration issues are usually overlooked in favor of user-interface concerns and feature sets, developers have their work cut out for them when attempting to integrate enterprise systems into Web applications. Some enterprise systems, such as mainframe applications or custom programs, pose technical problems because the interfaces are archaic and difficult to adapt. Other systems are easy to integrate technically, but provide licensing difficulties because of the special circumstances Web applications create. Solutions to these issues are pressing, though, because the need for application integration is growing. Users are getting acquainted with the possibility of a single Web interface to all enterprise systems, and the promise of even more integration with business partners and application syndicators is just around the corner.