Software Magazine - The Personalization Equation

The Personalization Equation

E-BUSINESS INTELLIGENCE (eBI) applications help e-businesses identify important customers and how to keep them.

By Curt Hall

The pervasiveness of the Internet means that the first interaction most businesses have with a potential customer is via the company Web site or e-commerce application. Consequently, the ability to "personalize" (match and modify) Web site content to individual customer or partner preferences is now a necessity as all companies—whether operating on a business-to-consumer (B2C) or business-to-business (B2B) model—strive to make their online operations more profitable.

Business intelligence (BI) and analytical applications are playing a decisive role in enabling Web site personalization. In fact, the application of customer analytics to Web operations has become so vital that it has prompted a new analytical domain: e-business intelligence (eBI).

Personalization, though, requires considerably more than just buying a BI tool and mining a Web site's data. In practice, personalization involves a number of considerations, including extracting, combining, and analyzing data taken from multiple sources, and integrating the results into the Web store (and other customer-facing channels).

The business case for applying personalization is found in all the buzzwords associated with e-commerce, including turning first-time visitors into paying customers and facilitating greater cross-sell and up-sell activities. In reality, many of the benefits obtainable from personalization are realized over the long term, yet they are just as important to the success of a business, and include:

The "Components of Personalization" provides an overview of the various processes necessary to facilitate Web personalization (indicated with circles). It also shows the components and platforms (products and services) supporting these processes. Personalization involves four main steps, conducted in a cyclical manner:

Customer Interaction. Personalization requires incrementally interacting with online visitors to gradually generate and assemble data that can be analyzed to determine individual preferences and tastes. This can be as simple as having visitors fill out a form stating their favorite types of music, books, software products, and so on, or applying real-time data mining in the form of consumer profiling systems. These systems employ rule-based or collaborative filtering techniques to automatically segment them into "communities" of customers with similar tastes.

Data Collection and Integration. This process, which includes extraction, transformation, and loading (ETL), varies greatly depending on company needs. For simple personalization efforts, some companies may only need to analyze Web visitor clickstream data in order to determine customer interests, and then tailor their Web sites accordingly. However, more in-depth customer analysis requires integrating data extracted from multiple sources and loading it into a customer information store.

Business Intelligence. Analysts apply various analytical techniques to data collected in the customer information store to determine customer preferences and segment customers into different categories based on individual preferences. A company may conduct analysis internally, or have it done offsite by an analytical application service provider (ASP).

Customer Interaction Personalization. In customer interaction personalization, the results of the analyses are tied back into an organization's e-business operations, effectively "closing-the-loop" between analysis and operations. This consists of generating personalization rules, which are incorporated into the Web store or the e-commerce platform's personalization engine. These rules target visitors with specific content based on their behavioral profiles.

Personalization requires implementing various Web components and the necessary data integration infrastructure that will enable the following components to interact with one another:

There are a variety of approaches for analyzing and personalizing data, as well as various data sources within the company's architecture from which to cull.

Clickstream Analysis. The first option a company has for conducting Web site personalization is to analyze its clickstream data. "Clickstream" refers to lines of code that are written to a flat file (or Web log) each time a visitor views a Web page or clicks on a hyperlink. Clickstream data provides a detailed activity path that is generated when a visitor interacts with a Web site. Analyzing clickstream data lets organizations track visitors as they navigate through the site, determining what Web pages and ads visitors viewed, what they clicked on, and how long they stayed on a certain page.

Clickstream analysis is useful for revealing the most popular Web site content areas by tracking customer behavior (i.e., What specific pages are preferred by what specific kinds of customers? Or, What are the most frequently traveled routes visitors follow when navigating the site?). Such information is valuable for determining how to redesign and optimize the Web site so that visitors are immediately greeted with offerings tailored to their preferences and tastes.

In addition, Clickstream analysis is useful for determining the optimum number and frequency of ad placements; for example, to determine the correct ratio of marketing-to-educational-to-merchandise promotions; or to test how visitors are receiving new ads (i.e., How many clicked on the new Victoria's Secret ad?). However, to really develop a comprehensive view of visitors and target more potential customers, organizations need to conduct more detailed analysis. This requires analyzing more than just clickstream data.

Customer Information Store. The foundation for in-depth personalization is the customer information store—a data warehouse designed for blending and analyzing data taken from Web operations (e-commerce transaction systems, server logs, etc.) with data from other sources. Such sources can include enterprise resource planning (ERP), call center, sales force automation (SFA), customer resource management (CRM), and other customer-transaction systems. Another important source is external data purchased from third-party demographic, marketing, and psychographic data suppliers such as Acxiom, Experian, D&B, Polk, and Naviant. The trick is to get all these differently formatted data into the customer information store. This requires a data integration infrastructure that can handle the extraction and transformation of disparate data.

Ideally, the customer information store will also serve as a company's enterprise data warehouse. Using an enterprise data warehouse is important for several reasons. One, the volume of data associated with Web site operations is gigantic; many companies report that the sheer volume of clickstream data they collect when combined with other Web transaction data can quickly grow to a terabyte or more. When other data is added to the mix, organizations may find themselves literally overwhelmed with unmanageable data volumes. Another reason to make an enterprise data warehouse the online customer information store is so that in the future, it can serve as the main repository for applying personalization techniques across all customer-facing channels (Web, call center, kiosk, ATM, bank branch teller, etc.) by feeding into enterprise-class CRM applications.

The reality, however, is that many companies are still defining and implementing their e-business strategies. Consequently, the customer information store will actually be a narrowly focused data mart designed specifically for personalizing online operations.

E-business Intelligence for Comprehensive Web Data Analysis. Consolidating various data in the customer information store provides a rich repository for analysts to explore and uncover customer patterns and preferences, which are used to segment customers into categories and to create profiles. Organizations frequently use these profiles for marketing purposes, such as predicting propensity to buy and customer lifetime value.

There are four basic categories of analysis and segmentation techniques: online analytical processing (OLAP), data mining, statistical analysis, and advanced data visualization.

Data analysis takes place in an iterative manner, in which analysts generate ad hoc queries to uncover customer preferences, demographics, and transaction patterns. For example, to identify the most frequent shoppers, analysts might query the customer information data store. Next, they might use data mining by applying a Self-Organizing Map (SOM) neural network to identify the company's most profitable customers. (SOM algorithms are useful for automatically clustering data into naturally occurring clusters, making them ideal for segmenting customers into similar categories). Next, analysts might apply statistical analysis to determine the average income level, age, and sex of a company's most profitable customers.

To identify the best cross-sell and up-sell opportunities, analysts typically segment customers into different categories based on user profiles, Web site behavior, and traditional customer interactions such as sales transactions and call center records. As an example, an online retailer might segment customers into one of 10 categories, obviously depending on the business. For instance, basic segmentation could place a customer in the category of graphical designer. A subcategory might also include her under artist. The use of third-party demographic and household data might indicate that she is married, has two grade-school children, operates her own design business, and belongs to a four-member family household with an overall income in excess of $300,000 a year.

Once organizations have analyzed customer data, they need to apply the results of the findings to personalize Web operations.

Consequently, more in-depth analysis of this rich blend of data would result in this customer (along with thousands of others) being segmented into a high-income category. In short, the objective is to fine-tune categorization schemes to earmark content, advertising, and promotions to support the categories a company defines.

Applying Data Analysis Results to Personalization. Once organizations have analyzed customer data, they need to apply the results of the findings to personalize Web operations. This process depends largely on the type and sophistication of a company's e-commerce platforms, BI tools, and analytical applications.

Different e-commerce platforms use personalization engines based on various approaches to deliver personalized content. For example, BroadVision Inc., Redwood City, Calif., and Vignette Corp., Austin, Texas, both provide template-based approaches that allow marketers to earmark content, promotions, and ads for various categories of visitors that they have defined. Other platforms use rule-based (inference engine-based) personalization approaches, such as platforms from Blue Martini Software, San Mateo, Calif., and Art Technology Group, Cambridge, Mass. Still others, like Net Perceptions Inc., Edina, Minn., offer outsourced personalization services based on collaborative filtering techniques. (Collaborative filtering consists of analyzing customers' historical and current buying behavior. Based on this behavior, customers are sorted into a "buying community" consisting of other customers that have established similar profiles. The more a customer shops, the more likely the algorithms will produce effective results. In addition, collaborative filtering algorithms can learn in near-real time.)

Some analytical products offer tight integration with popular e-commerce platforms like those from BroadVision and Vignette. This means an organization can use its output (analytical models) to generate personalization rules, which are then fed directly back into the Web store, thereby automating the process of matching content to users. For example, the Broadbase Enterprise Performance Management analytical application from Broadbase Software Inc., Menlo Park, Calif., can generate rules that are populated inside the BroadVison e-commerce server's personalization rules engine to personalize content.

Blue Martini features a data mart, data mining, and data analysis tools that are tightly integrated with its rules-based server that runs on the Web store. This allows merchandisers to enter rules detailing customer behavior uncovered by data mining directly into the Web store to dynamically update the platform with specific personalization rules and content that drive promotional, advertising, and other merchandising activities targeted at online visitors.

There are a number of options for companies seeking to personalize their e-business operations. These include purchasing one of the many BI tools on the market, implementing a packaged analytical application, or outsourcing to one of the new analytical ASPs offering personalization services.

All the major BI tool vendors (including OLAP, data mining, data visualization, etc.) have enhanced their products with new capabilities for analyzing clickstream data. Organizations that decide to go with a tools approach should go for a product that provides prebuilt Web analysis templates. Templates provide a "road map" that can help get a Web analysis application up and running more quickly. Products for analyzing clickstream data and combining it with other data sources are available from vendors such as Hyperion Solutions, Sunnyvale, Calif.; Cognos Inc., Burlington, Mass.; Angoss Software Corp., Toronto; SPSS Inc., Chicago; and SAS, Cary, N.C.

Packaged applications can accelerate implementing advanced analytical applications that provide a platform for applying personalization across all customer and partner channels.

Another choice is to use a packaged analytical application that bundles application-specific data models, prebuilt interfaces (for extracting and transforming data), a metadata repository, and predefined reports and metrics in the form of key performance indicators tailored to specific industries and domains. Packaged applications can accelerate implementing advanced analytical applications that provide a platform for applying personalization across all customer and partner channels. Their main drawback is that they are initially more expensive than buying a BI tool. And, although they provide considerably more functionality, many organizations may not require all of their features.

The most visible products in this category include those from Broadbase; E.piphany Inc., San Mateo, Calif.; Informatica Corp., Palo Alto, Calif.; Sagent Technology Inc., Mountain View, Calif.; MicroStrategy Inc., Vienna, Va.; Business Objects Inc., San Jose, Calif.; and Blue Martini. In reality, the market for packaged analytical applications is overcrowded with products.

Another option is to outsource Web analysis requirements to an analytical ASP such as WhiteCross Systems Inc., San Francisco; Interelate, Eden Prairie, Minn.; digiMine Inc., Kirkland, Wash.; WebMiner.com, New York City; or WebTrends Corp., Portland, Ore. The benefit to outsourcing is that an organization avoids having to buy, install, and manage the necessary hardware and software, and can leverage the ASP's expertise in conducting and applying data analysis to online operations. The drawback is that a company is essentially putting its trust in a third party. In addition, by outsourcing, the company will not gain the long-term benefit of building up data analysis expertise within the organization.

Personalization consists of various processes, and the degree of personalization that can be performed depends on the data an organization collects or has access to. Personalization may take place in an automated near-real-time manner immediately at the initial point of first customer contact by applying collaborative filtering and rules-based data mining techniques. "Deeper" personalization requires using richer customer data culled from multiple sources, which are typically stored in a data warehouse and analyzed using BI techniques.

BI plays a key role in personalization because it provides the analytical capabilities to help determine the important customers and what an organization should do to keep them. BI isn't cheap; however, neither is going out of business. Without BI, an organization is just guessing as to what makes its marketing operations successful or a flop. Lastly, BI and personalization will not substitute for a well-defined business plan, as too many dot-coms have come to realize. However, they can provide business managers with the ammunition needed to measure and interpret how well the company is operating and what areas need attention.

Curt Hall is editor of Business Intelligence Advisor newsletter, a monthly newsletter covering the application of, and market for, data warehousing and BI products and services. He is also a senior consultant with Cutter Consortium's eBusiness Intelligence Advisory Service. E-mail him at cchall4@home.com.