Data agility in Master Data Management

Recently, I came across few master data management implementations and database performance problems related to them. 

What is master data management (MDM): It is a set of processes to centralize the management of “master data” i.e, the reference data for a given business. It typically consists of Customers, Agents, and Products for an insurance company.  Typically an MDM instance focuses on one data domain. If we focus on Customer domain the implementation is called as Customer Data Integration or when the domain is Products it is called as Product Information Management.

When it comes to Insurance industry, even though the “customer centricity” has been the latest trend, traditionally it has been a “agent” driven business. So, one key subject area for MDM is “Agent Data” which actually sells “Products” to “Customers”. For generalization we can call this as “Channel Master Data.” So we have three subject areas to deal with in a multi-domain-MDM implementation.

The processes include:

1.       Data acquisition (Input)

2.       Data standardization, duplicate removal, handling missing attributes (Quality)

3.       Data mining, analytics and reporting  (Output)

The industry tools have standard data and process models for customers and products to achieve the required functionality of MDM. We can adopt some customer model to model the agent domain or keep both in a “party” domain.

The problem with multi-domain MDM:

Here comes the problem. When it comes to agent domain, the customer requirement states “get me the agent and his top n customers acquired” or “get me the agent and his top n products sold”

In a renowned MDM product an SQL query need to join 20+ tables (involving some outer joins) to achieve the requirement and even after caching all the tables in-memory the processing for each agent id is taking 20+ seconds on an Oracle 11.2 database on a decent production sized hardware. 

a)      Dynamically querying for such data and building the structure of data into target structure puts more load/ adds latency to the screen. SLA is not met.

b)      Pre-populating the data with a periodic refresh in a materialized view does not make the data current to the real-time.

As we can’t have both requirements of getting the data in 2sec and it is to be current in traditional design, we need to design a hybrid approach.

Any ideas or experience in this are welcome. Have anyone implemented a MDM solution on a NoSQL database yet?

Measure, Metric and Key Performance Indicator (KPI)

In a BI project it is very important to design the system to confidently visualize the indicators that matter for business. Even senior BI professionals sometime get confused with the terms like measure, metric and KPI.

This post is to give some simple definitions for the hierarchy of KPIs:

Measure: is something that is measured like value of an order. Number of items on the order or lines of code or the number of defects identified during the unit testing. Measures are generally captured during the regular operations of the business. These are the lowest granularity of the FACT in the star schema.

Metric: What is metric? Metrics are generally derived from one or more measures. For example the defect density is the number of defects for 1000 lines of code in a COBOL program. It is derived from lines of code and number of defects measures. A metric can also be max, min, average of a single measure. Generally metrics are computed or derived based on the underlying facts / measures.

Performance Indicator: brings in the business context into the metric. For example reduction of defect density after introducing more rigorous review process (review effectiveness) could be a performance indicator in a software development context. In this case it reduces the rework effort of fixing the unit test bugs and re-testing thereby improving the performance of a software development organization.

A careful selection of KPIs and presenting them with suitable granularity to the right level of management users within the business makes a BI project be successful.

Bringing the data / measures from multiple sources into a star schema is typically ETL cycle of the warehouse. Once the data is refreshed, incrementally summarizing and computing all the metrics is the next stage. Finally visualizing the KPIs is the art of dash boarding.

I have seen several BI projects having problems mixing up all the three steps; trying to clean data during loading, trying to summarize the data into metrics during ETL or trying to summarize during the reporting phase causing multiple performance problems. A good design should try to keep the stages independent of each other and take care of issues like missing refreshes and duplicate refreshes from feeding source systems. Also need to consider parallelization of tasks to take advantages of multiple cores of processors and large clusters of computing resources.

Cloud Computing workshop slides

Cloud Computing - Foundations, Perspectives and Challenges workshop slides.

Present to BITES state level faculty development program at Mangalore today to around 60 faculty members from regional engineering colleges. 

Cloud prasad chitta_8th_feb2013 from Prasad Chitta

Few thoughts on Data Preparation for #Analytics

It is Friday and it is time for a blog post.

Typical analysis project spends 70% (- 80%) of time in preparing the data. Achieving the right Data quality and right format of data is a primary success factor of success of an analytic project.

What makes this task very knowledge intensive and why is a multifaceted skill required to carry out this task?

I will give a quick/simple example of how the “Functional knowledge” other than the technical knowledge is important in the preparation of the data. There is a functional distinction between missing data and non-existing data.

For example consider a customer data set. If the customer is married and the age of spouse is not available this is missing data. If customer is single, age of spouse is non-existing. In the data mart these two scenarios need to be represented differently so that the analytic model behaves properly.

Dealing with the missing data (data imputation techniques) within the data set while preparing the data impacts on the results of the analytical models.

Dr. Gerhard Svolba of SAS has written extensively on Data Preparation as well as Data Quality (for Analytics) and this presentation gives more details on the subject.

I have made a blog post earlier dealing with these challenges in the “Big data” world - http://technofunctionalconsulting.blogspot.in/2012/10/data-munging-in-big-data-world.html

Machine Learning Algorithms – Classification, Clustering and Regression

For a Data Scientist among the other skill sets, good fundamentals on “data mining” or “machine learning” is the icing over the cake. These algorithms are also used in predictive analytics. It is immaterial if the data is “big data” or not!

We will end up having several variables either containing numeric or nominal attributes describing an entity. The numeric variables could be continuous or discreet like ordinals.  Nominal variables are of the form of known list of values having binary (two i.e., yes or no; male or female etc.,) or more possible values.

Given that a data set consisting of a millions of records, each record containing some 100s of variables an analyst’s job is to derive some insights to solve the known or unknown business problems! This is where the application of machine learning algorithms comes into play.

Broadly Machine Learning can be put into two groups.

1.       Predicting a target variable for a given instance of data record. We will have a set of records with the known values for target variable by which we can develop a model and train the model, test and put that into production – This is called supervised learning

a.       If the target variable is nominal then these algorithms are called classification.

b.      If the target variable is a continuous numeric variable then we need to apply regression

2.       There is no target variable; we need to group the data records into distinct groups based on multiple variables within the dataset – This is called unsupervised learning. Clustering and Association Analysis algorithms are used to achieve this. 

Formulating the problem, preparing the data, visualizing the data, training the model, testing the model and interpreting the results to generate insights and them implementing the derived knowledge to the business operations require multi disciplinary skills in business domain, operations management and technology.

A real business analytic solution consists of using multiple techniques involving machine learning to achieve Customer Segmentation, Cross-selling, Customer behavior analysis, Customer retention, Marketing Analytics and campaign management, fraud detection, optimization of profits etc.,

A recent quick read through the book Machine Learning in Action by Peter Harrington prompted me to write this tech capsule on this Friday… 

A journey through Grid and Virtualization leading to Cloud computing

In computing these are two trends I have seen crisscrossing throughout my career.

1.       Making a computing node look like multiple nodes using virtualization.

2.       Making multiple computing nodes work as a single whole called a cluster or grid

So, there was an era where the computing was really a “Big-Iron” and the computers are mainframe size and provided a lot of capacity of computing. One computer handled multiple users and multiple operations at the same time.  The Virtual Machine is included in operating system of IBM and the mainframes as well as from DEC VAX mainframes had similar concepts. Of late, we see the trend even in desktops with hypervisors like vmware etc., coming out.

With the advent of mid-range servers with limited capacity, there is a need to put them together to get the higher computing power to deal with the demand.  The first commercial cluster developed by DEC ARCnet even though there is always been a fight between IBM and DEC on who invented clusters.  Clustering also provides high availability / fault tolerance along with higher computing capacity.  Oracle was the first database to implement parallel server on ARCnet cluster for VAX operating system.

This trend of cluster computing has achieved supercomputing to break the complex task in to multiple parallel streams and execute them on multiple processors. What is the fundamental challenge in “clustering” – the process coordination and access to the shared resources. This leads to be locally networked and connected with high performance local network.

Another concept of this is grid computing where the administrative domain can connect loosely coupled nodes to perform a task. So, we have more and more cores, processors, nodes in a grid to provide low cost, fault tolerant computing. This is making smaller components put together to look like a giant computing capacity.

Finally, what I see today is “Cloud” which creates a grid of elastic nodes that look to appear like a single (large) computing resource and gives a slice of virtualized capacity to each of multiple tenants of that computing resource.  

Designing solutions in each of these technologies of big-iron, virtualization, clusters & grid and in Cloud world has really been challenging and keeps the job lively…

Small & Big Data processing philosophies

In this first post of 2013, I would like to cover some fundamental philosophical aspects of “data” & “processing”.

As the buzz around “Big Data” going on high, I have classified the original structured, relational data as “small data” even though some very large databases I have seen having 100+ Terabytes of data with an IO volume of 150+ Terabytes per day.  

Present day data processing predominantly uses Von-Neumann architecture of computing in which “Data” and its “processing” are distinct and separated into “memory” and “processor” connected by a “bus”.  Any data that need to be processed will be moved into processor using the bus and then the required arithmetic or logical operation happens on it producing the “result” of the operation. Then the result will be moved to “memory/storage” for further reference.  Also, the list of operations to be performed (the processing logic or program) is stored in the “memory” as well. One needs to move the next instruction to be carried out into the processor from memory using the bus.

So in essence both the data and the operation that needs performing will be in memory which can’t process data and the facility that can process data is always dependent on the memory in the Von-Neumann architecture.

Traditionally, the “data” has been moved into a place where the processing logic is deployed as the amount of data is small when compared to the amount of processing needed is relatively large involving the complex logic. In the RDBMS engines like Oracle read the blocks of storage into the SGA buffer cache of running database instance for processing. The transactions were modifying small amounts of data at any given time.

Over a period of time “analytical processing” that required to bring huge amounts of data from storage into processing node which created a bottleneck on the network pipe. Add to that there is a large growth in the semi-structured and unstructured data that started flowing which needed a different philosophy towards data processing.

There comes the HDFS and map-reduce framework of Hadoop which took the processing to the data. During the same time comes Oracle Exadata which took the database processing to storage layer with a feature called “query offloading”

In the new paradigm, the snippets of processing logic are being taken to a cluster of connected nodes where the data mapped with a hashing algorithm resides and results of processing then reduced to finally produce result sets. It is now becoming economical to take the processing to data as the amount of data is relatively large and the required processing is fairly simple tasks of matching, aggregating, indexing etc.,

So, we now have choice of taking small amounts of data to complex processing with structured RDBMS engines with shared-everything architecture of traditional model as well as taking processing to data in the shared-nothing big data architectures. It purely depends on the type of “data processing” problem in hand and neither traditional RDBMS technologies will be replaced by new big data architectures, nor could the new big-data problems be solved by traditional RDBMS technologies. They go hand-in-hand complementing each other while adding value to the business when properly implemented.

The non-Von-Neumann architectures still need better attention by the technologists which will probably hold the key to the way human brain processes and deals with the information seamlessly either it is structured or non-structured streams of voice, video etc., with ease. 

Any non-Von-Neumann architecture enthusiasts over here?

Where Art Thou Information? - Lost in Technology!

As another year of technology hype draws to a close, I wonder where the focus on the Information is. Information is the subject matter of “IT”, where the technology enables a proper capture, processing and presentation of the Information. Most of the focus is shifted to technology led transformation of businesses…..

Appification – a word coined to make an “App” (short for application!) that can be downloaded to perform a generic task on a mobile device.

Gamification – Again a word coined to create a simulated / augmented reality that “Engages” the user with the subject under consideration in a generic manner.

Cloudification – Deploying an elastic, self-provisioned, pay-per-use model of computing performing generic functions.

“Appliance”ification – pre-configured hardware + software bundles pushed in by the vendors to provide “optimized” & “efficient” solutions to problems following a generic pattern.

“Package”fication – deploying vendor provided generic package to provide predefined information flows by the vendor.

With all these “ifications” we have ended up with several suites of “products” that are branded as ERP, EIM, ECM, BPM etc., also we have “anything” as a service (saas, paas, iaas, dbaas, what not aaS!) providing everything on the move i.e, mobile.

So, what is the point I am trying to bring up? With all the possibilities of customization and promises of near zero time to market, and all the solutions around, we lost the focus on the “problem” that we are trying to solve!

If all these are solutions then what is the problem?

All the past year I have been working with the information management of the Insurance companies across the globe and interacting with the different levels employees representing the IT and business functions of those enterprises, the problems seems to lie in fundamental definitions of information flows. Legacy policy administration systems, multiple systems holding the data about quotations, pricing, products, policies & claims causing the “entropy” of not having the correct information to the different operational units within the business in the right time to make decisions seems to me the key problem.

As long as the real problem is not rightly understood with respect to the information flows between the organization units and within the operating environment, implementing newer technology solutions will not solve the problem.

I hope the new year 2013 gets the focus right on the problem of information and transforms the business based on facts i.e., what I call “Information led transformation” rather than the current trend of “technology led transformation”

Happy New Year 2013!!

Uncommon Sense, Common Nonsense - A review

Being a technologist, I rarely read management books. Even when I start reading a management book, I hardly complete reading it as it makes little sense to me. Recently, I came across this book “Uncommon Sense, Common Nonsense: Why Some Organisations Consistently Outperform Others” ( ISBN:9781846686009 ) and read till the end. 
 
This is a book which I could make some sense of business strategy and providing a leadership vision to an organization.
The book has four key parts dealing with:
1.        Winners and losers
2.        Strategy and tactics
3.        Organisation and management
4.        Biases and remedies
The fifth part gives Application and Examples from the author’s work.

In essence, there is a lot of “Common Nonsense” (in the form of big-data in today’s world) which is visible to us as well as the competitors (Same with “Common Sense!”).  There is an amount of “uncommon nonsense” about our own organisation known to us alone and that of the competition which is only known to them. But the key thing differentiates is the “Uncommon Sense” that makes organizations constantly outperform. The strategy is about deliberating means of constantly deriving and applying the “uncommon sense”

So, “Without changing our patterns of thought, we will not be able to solve the problems we created with our current patterns of thought.” Albert Einstein’s wisdom comes to rescue.


Overall, this is a good read for all the strategists and leaders in my honest opinion.

Aphorisms on Information Technology & Systems

Today, coincidentally it is 8888 days from 8/8/88 (The day I started studying Information Technology) and 1000 weeks from 11-Oct-1993 (The day I started working!)

Aphorisms on Information Technology &  Systems from Prasad Chitta

Some theoritical fundamentals on Information Systems & Technology as I see....

100 th post on this blog......