Information Advantage
http://www.theinformationadvantage.com/wp-content/plugins/sociofluid/images/digg_32.png http://www.theinformationadvantage.com/wp-content/plugins/sociofluid/images/reddit_32.png http://www.theinformationadvantage.com/wp-content/plugins/sociofluid/images/stumbleupon_32.png http://www.theinformationadvantage.com/wp-content/plugins/sociofluid/images/delicious_32.png http://www.theinformationadvantage.com/wp-content/plugins/sociofluid/images/newsvine_32.png http://www.theinformationadvantage.com/wp-content/plugins/sociofluid/images/technorati_32.png http://www.theinformationadvantage.com/wp-content/plugins/sociofluid/images/google_32.png http://www.theinformationadvantage.com/wp-content/plugins/sociofluid/images/facebook_32.png http://www.theinformationadvantage.com/wp-content/plugins/sociofluid/images/twitter_32.png

The industry took a decade (most of 90’s) to absorb the meaning of a ‘datawarehouse’ and just when it got over the war between Kimball’s conformed (shared) dimensions and Inmon’s heavy reliance on data marts, the term ‘real-time datawarehousing’ was introduced in 2000. The debate has now shifted to three fundamental points:

  1. What is ‘real-time’?
  2. Is there ‘real’ business demand for this technology?
  3. Are there technologies and technical architectures that can enable a ‘real-time datawarehouse’?

I am going to focus on the first two points in this blog and deal with the third point later.

So what is ‘real-time’?  Real-time can be defined as having no lag between the occurrence of an event to the time it is recorded and shared. But the words ‘no lag’ are a misnomer as what may be perceived as ‘no lag’ to a human eye can be captured as hundreds of snapshots by a high-powered camera. Hence, knowing the nuances in precision, the terms ‘near real-time’ or ‘on time’ are gaining more popularity. ‘Near real-time’ refers to having a lag of couple of minutes to an hour where as ‘on time’ is geared to meet service level agreements (SLA’s) and policies where data is anywhere from two hours to a day old.

So now that we have settled on what is real-time, is there real demand for these technologies?  Before answering that question, we have to step back and think about the speed at which businesses are conducted, and one can argue that very few business need data on a real-time basis. But when the question is posed in terms of business value, a different perspective emerges.

There has always been demand for ‘on time’ datawarehouses, but is there demand for ‘near real-time’ datawarehouses? Yes, there is. A recent article talks about how Fashion retailer Elie Tahari is using ‘five-minute’ refresh rates for her retail stores to allow customer purchase behavior drive the inventory.. “One of our regional managers told me that as soon as the system went up she realized we were not buying correctly across size levels per store,” says Aytaman. “Before, they had ordered the same size breakdown for all stores. While the customer would not notice, they’re now more likely to find the sizes and styles they prefer at their location. This is a great improvement for stocking and logistics that also reduces returns.”

Another practical application of ‘near real-time’ datawarehouses is in the area of fraud detection. Couple of years back, Continental implemented a ‘near-real time’ datawarehouse and in the first year alone $7 million in fraud was recognized and eliminated, and the airline realized a $41 million reduction in costs.  Just after the tragic 9/11 event, the team sorted through 35 different data marts and loaded the airline’s booking system into XML so senior airline officials and the FBI could monitor customers booking flights in real time.

Fraud detection hinges on using standard identifiers (e.g. Social Security, Credit Card Accounts, Bank Accounts, Passport Numbers, Driving License ID’s etc.) to analyze transactions and tease out inconsistencies in usage patterns. ‘Near-real time’ datawarehouses are great for this function as they collate information from myriads of sources and can churn out exception analyses on a timely basis.

A few years back, I was personally involved in implementing an enterprise datawarehouse for a large insurance company. It took more than two years to get it done, but the benefits were quick and tangible – Of the many reports produced to measure the health of the auto insurance division, couple of them focused on using the standard identifiers to highlight claim frauds. For the first time in the company’s long history, they were able to automate a report which could list people using the same social security or driving license but under different names, addresses and claiming losses on different automobiles within a short time frame. Regardless to say the executives were more than pleased.

Most of us have personally experienced receiving automated messages from our credit card companies informing us that our credit card has been blocked based on some suspicious activity. And even though we may get ruffled by the extra steps to get our credit card back to a ‘functional’ state, nobody can deny the ‘peace of mind’ knowing that one will not be liable for some random high-valued charges – and all of this rests on ‘triggers’ generated through timely analysis of millions and millions of transactions.

In looking through the uses of this technology, there are clear business value themes:

  • Cost Reduction – through consolidation of information and automation of analyses
  • Risk Identification / Mitigation – by alerting the customers on a timely basis and resolving the exception within a short span of time
  • Revenue Opportunities – improving the customer experience by empowering the channels (and customer reps) with accurate information on a timely basis

It is clear that ‘information’ can drive a significant ‘advantage’ to a company’s profitability, especially if it can drive insights faster than the rest of the competition. But one needs to identify and size the business value at play before spending a lot of money on the technology. More businesses will ultimately realize the power of ‘near real-time’ data, but can they afford such sophisticated technology?  Well, I will save that discussion for my next blog…

{ 1 comment }

http://www.theinformationadvantage.com/wp-content/plugins/sociofluid/images/digg_32.png http://www.theinformationadvantage.com/wp-content/plugins/sociofluid/images/reddit_32.png http://www.theinformationadvantage.com/wp-content/plugins/sociofluid/images/stumbleupon_32.png http://www.theinformationadvantage.com/wp-content/plugins/sociofluid/images/delicious_32.png http://www.theinformationadvantage.com/wp-content/plugins/sociofluid/images/newsvine_32.png http://www.theinformationadvantage.com/wp-content/plugins/sociofluid/images/technorati_32.png http://www.theinformationadvantage.com/wp-content/plugins/sociofluid/images/google_32.png http://www.theinformationadvantage.com/wp-content/plugins/sociofluid/images/facebook_32.png http://www.theinformationadvantage.com/wp-content/plugins/sociofluid/images/twitter_32.png

Theme 4: Statistical techniques and tools are not likely to provide competitive advantage

I read this interesting post from Sijin describing his journey to master a video game (emphasis added by me)

All this kind of reminded me of my experiences with finding the perfect weapon while playing Call of Duty 4 over the past year. I spent 3 hours a day almost every day for the past one year playing this game, reaching the max prestige level (the “elite” club) in the multi-player version. I became really good at it… no matter what weapon I was using. But I remember when I started out and I really sucked, I became obsessed with finding the perfect weapon with the perfect set of perks and add-ons. I used to wander the forums asking people about which weapons and perks to use on which map and what the best tips were etc. Thinking that having the perfect weapon would make me a good player. In the end, the only thing that mattered was all the hours I put in to learn all the maps, routes, tricks and my ability (I like to think). The surprising thing was that once I mastered the game, it didn’t really matter what weapon I chose, I was able to adapt any weapon and do a decent job.

This story captures the essence of the theme of this post.

The popular statistical techniques frequently used in business analytics like linear regression and logistic regression are more than half-a-century old. System dynamics was developed in 1950s. Even neural networks have been around for more than 40 years. SAS was founded in 1976 and the open source statistical tool R was developed in 1993. The point is that popular analytical techniques and tools have been around for some time and their benefits and limitations are fairly well understood.

An unambiguous definition of the business problem that will impact a decision, a clear analysis path leading to output, thorough understanding of various internal and 3rd-party datasets are all more important aspects of a predictive analytics solution than the choice of the tool. Not to mention having a clear linkage between the problem, the resulting decision, and measurable business value.  The challenge is in finding an expert user who understands the pros and cons and adapts the tools and techniques to solve the problem at hand. Companies will be better served by investing in the right analytical expertise rather than worrying about the tools and technique as the right analytical team can certainly be a source of competitive advantage.

While this theme is fairly well understood within the analytics practitioner community, the same cannot be said about business users and executives. It is still easy to find senior executives who believe that ‘cutting edge’ techniques like neural networks should be used to solve their business problem or predictive analytics tools are a key differentiator while selecting analytics vendors.  The analytics community needs to do a better job in educating the business user and senior executives about this theme.

You can read the previous installments of the series here (part 1, part 2, and part 3).

{ 9 comments }

Beyond Reporting: The Organizational Side of BI

February 25, 2010

In our work we frequently address questions about the organizational aspects of information strategy, data architecture, and business intelligence efforts.  Several large insurance and CPG companies have BI functions that are deliberately moving from a “reporting” focus to encompass analytics and, more broadly, effective decision making that seeks to capture more value from information. [...]

Read the full article →

Predictive Analytics: 8 Things to Keep in Mind (Part 3)

February 25, 2010

Theme 3: Integrating third-party data into predictive analysis
This is the third installment of the eight part series on predictive analytics (see part 1, part 2).
Perhaps one of the most significant opportunities for organizations using predictive analytics is incorporating new relevant third-party data into their analysis and decision making process.  Investment in a targeted and [...]

Read the full article →

Predictive Analytics: 8 Things to Keep in Mind (Part 2)

February 18, 2010

Theme 2:  Modeling Strategic vs. Operational Decisions
In the first post of this eight-part series, I wrote about the importance of understanding the cost of a wrong decision prior to making an investment in a predictive modeling project.
Once we determine the need for the investment, we need to focus on the type of modeling approach. [...]

Read the full article →

First Principles: The Role of Information Systems

February 16, 2010

The purpose of any information system is to generate reliable information for decision making.  Financial accounting systems exist to provide reports on financial position and performance to outside parties, largely providers of capital.  Management accounting systems provide information to enable planning and control activities.  In complex environments, definition and maintenance of a definitive system [...]

Read the full article →

Predictive Analytics: 8 Things to Keep in Mind (Part 1)

February 8, 2010

I along with two of my colleagues (Anand Rao & Dick Findlay), recently conducted a workshop at the World Research Group’s Predictive Modeling conference at Orlando.
Many of our clients have been asking about predictive analytics, and it is a timely example of how companies are capturing value from information.  As part of the workshop, [...]

Read the full article →

3 Paths to Understanding Your Demand Curve

February 8, 2010

The New York Times recently announced its intent to start charging some frequent online readers a flat fee.  For visitors that aren’t print subscribers, a certain number of articles per month will be free; once the articles-per–month threshold is reached, a flat fee for unlimited access kicks in.
This is an attempt to extract surplus [...]

Read the full article →

RIP – IBM Acquires Initiate Systems

February 3, 2010

As I predicted in my earlier post on the recent Informatica – Siperian acquisition, the Master Data Management (MDM) / Customer Data Integration (CDI) market was poised for more takeovers and turmoil and so it has happened. Today IBM announced the takeover of Initiate Systems and introduced a new layer of confusion.
While it was [...]

Read the full article →

Informatica acquires Siperian – What is next in MDM ?

February 1, 2010

With the Jan 28th announcement of Informatica’s acquisition of MDM pure play vendor Siperian for $130m in cash the red hot master data management (MDM) market is poised for more interesting things to come.  Siperian has been one of the last standing independent MDM vendors with a strong footprint in the Pharmaceutical and Life [...]

Read the full article →