Few thoughts on Data Preparation for #Analytics

It is Friday and it is time for a blog post.

Typical analysis project spends 70% (- 80%) of time in preparing the data. Achieving the right Data quality and right format of data is a primary success factor of success of an analytic project.

What makes this task very knowledge intensive and why is a multifaceted skill required to carry out this task?

I will give a quick/simple example of how the “Functional knowledge” other than the technical knowledge is important in the preparation of the data. There is a functional distinction between missing data and non-existing data.

For example consider a customer data set. If the customer is married and the age of spouse is not available this is missing data. If customer is single, age of spouse is non-existing. In the data mart these two scenarios need to be represented differently so that the analytic model behaves properly.

Dealing with the missing data (data imputation techniques) within the data set while preparing the data impacts on the results of the analytical models.

Dr. Gerhard Svolba of SAS has written extensively on Data Preparation as well as Data Quality (for Analytics) and this presentation gives more details on the subject.

I have made a blog post earlier dealing with these challenges in the “Big data” world - http://technofunctionalconsulting.blogspot.in/2012/10/data-munging-in-big-data-world.html

Machine Learning Algorithms – Classification, Clustering and Regression

For a Data Scientist among the other skill sets, good fundamentals on “data mining” or “machine learning” is the icing over the cake. These algorithms are also used in predictive analytics. It is immaterial if the data is “big data” or not!

We will end up having several variables either containing numeric or nominal attributes describing an entity. The numeric variables could be continuous or discreet like ordinals.  Nominal variables are of the form of known list of values having binary (two i.e., yes or no; male or female etc.,) or more possible values.

Given that a data set consisting of a millions of records, each record containing some 100s of variables an analyst’s job is to derive some insights to solve the known or unknown business problems! This is where the application of machine learning algorithms comes into play.

Broadly Machine Learning can be put into two groups.

1.       Predicting a target variable for a given instance of data record. We will have a set of records with the known values for target variable by which we can develop a model and train the model, test and put that into production – This is called supervised learning

a.       If the target variable is nominal then these algorithms are called classification.

b.      If the target variable is a continuous numeric variable then we need to apply regression

2.       There is no target variable; we need to group the data records into distinct groups based on multiple variables within the dataset – This is called unsupervised learning. Clustering and Association Analysis algorithms are used to achieve this. 

Formulating the problem, preparing the data, visualizing the data, training the model, testing the model and interpreting the results to generate insights and them implementing the derived knowledge to the business operations require multi disciplinary skills in business domain, operations management and technology.

A real business analytic solution consists of using multiple techniques involving machine learning to achieve Customer Segmentation, Cross-selling, Customer behavior analysis, Customer retention, Marketing Analytics and campaign management, fraud detection, optimization of profits etc.,

A recent quick read through the book Machine Learning in Action by Peter Harrington prompted me to write this tech capsule on this Friday… 

A journey through Grid and Virtualization leading to Cloud computing

In computing these are two trends I have seen crisscrossing throughout my career.

1.       Making a computing node look like multiple nodes using virtualization.

2.       Making multiple computing nodes work as a single whole called a cluster or grid

So, there was an era where the computing was really a “Big-Iron” and the computers are mainframe size and provided a lot of capacity of computing. One computer handled multiple users and multiple operations at the same time.  The Virtual Machine is included in operating system of IBM and the mainframes as well as from DEC VAX mainframes had similar concepts. Of late, we see the trend even in desktops with hypervisors like vmware etc., coming out.

With the advent of mid-range servers with limited capacity, there is a need to put them together to get the higher computing power to deal with the demand.  The first commercial cluster developed by DEC ARCnet even though there is always been a fight between IBM and DEC on who invented clusters.  Clustering also provides high availability / fault tolerance along with higher computing capacity.  Oracle was the first database to implement parallel server on ARCnet cluster for VAX operating system.

This trend of cluster computing has achieved supercomputing to break the complex task in to multiple parallel streams and execute them on multiple processors. What is the fundamental challenge in “clustering” – the process coordination and access to the shared resources. This leads to be locally networked and connected with high performance local network.

Another concept of this is grid computing where the administrative domain can connect loosely coupled nodes to perform a task. So, we have more and more cores, processors, nodes in a grid to provide low cost, fault tolerant computing. This is making smaller components put together to look like a giant computing capacity.

Finally, what I see today is “Cloud” which creates a grid of elastic nodes that look to appear like a single (large) computing resource and gives a slice of virtualized capacity to each of multiple tenants of that computing resource.  

Designing solutions in each of these technologies of big-iron, virtualization, clusters & grid and in Cloud world has really been challenging and keeps the job lively…

Small & Big Data processing philosophies

In this first post of 2013, I would like to cover some fundamental philosophical aspects of “data” & “processing”.

As the buzz around “Big Data” going on high, I have classified the original structured, relational data as “small data” even though some very large databases I have seen having 100+ Terabytes of data with an IO volume of 150+ Terabytes per day.  

Present day data processing predominantly uses Von-Neumann architecture of computing in which “Data” and its “processing” are distinct and separated into “memory” and “processor” connected by a “bus”.  Any data that need to be processed will be moved into processor using the bus and then the required arithmetic or logical operation happens on it producing the “result” of the operation. Then the result will be moved to “memory/storage” for further reference.  Also, the list of operations to be performed (the processing logic or program) is stored in the “memory” as well. One needs to move the next instruction to be carried out into the processor from memory using the bus.

So in essence both the data and the operation that needs performing will be in memory which can’t process data and the facility that can process data is always dependent on the memory in the Von-Neumann architecture.

Traditionally, the “data” has been moved into a place where the processing logic is deployed as the amount of data is small when compared to the amount of processing needed is relatively large involving the complex logic. In the RDBMS engines like Oracle read the blocks of storage into the SGA buffer cache of running database instance for processing. The transactions were modifying small amounts of data at any given time.

Over a period of time “analytical processing” that required to bring huge amounts of data from storage into processing node which created a bottleneck on the network pipe. Add to that there is a large growth in the semi-structured and unstructured data that started flowing which needed a different philosophy towards data processing.

There comes the HDFS and map-reduce framework of Hadoop which took the processing to the data. During the same time comes Oracle Exadata which took the database processing to storage layer with a feature called “query offloading”

In the new paradigm, the snippets of processing logic are being taken to a cluster of connected nodes where the data mapped with a hashing algorithm resides and results of processing then reduced to finally produce result sets. It is now becoming economical to take the processing to data as the amount of data is relatively large and the required processing is fairly simple tasks of matching, aggregating, indexing etc.,

So, we now have choice of taking small amounts of data to complex processing with structured RDBMS engines with shared-everything architecture of traditional model as well as taking processing to data in the shared-nothing big data architectures. It purely depends on the type of “data processing” problem in hand and neither traditional RDBMS technologies will be replaced by new big data architectures, nor could the new big-data problems be solved by traditional RDBMS technologies. They go hand-in-hand complementing each other while adding value to the business when properly implemented.

The non-Von-Neumann architectures still need better attention by the technologists which will probably hold the key to the way human brain processes and deals with the information seamlessly either it is structured or non-structured streams of voice, video etc., with ease. 

Any non-Von-Neumann architecture enthusiasts over here?

Where Art Thou Information? - Lost in Technology!

As another year of technology hype draws to a close, I wonder where the focus on the Information is. Information is the subject matter of “IT”, where the technology enables a proper capture, processing and presentation of the Information. Most of the focus is shifted to technology led transformation of businesses…..

Appification – a word coined to make an “App” (short for application!) that can be downloaded to perform a generic task on a mobile device.

Gamification – Again a word coined to create a simulated / augmented reality that “Engages” the user with the subject under consideration in a generic manner.

Cloudification – Deploying an elastic, self-provisioned, pay-per-use model of computing performing generic functions.

“Appliance”ification – pre-configured hardware + software bundles pushed in by the vendors to provide “optimized” & “efficient” solutions to problems following a generic pattern.

“Package”fication – deploying vendor provided generic package to provide predefined information flows by the vendor.

With all these “ifications” we have ended up with several suites of “products” that are branded as ERP, EIM, ECM, BPM etc., also we have “anything” as a service (saas, paas, iaas, dbaas, what not aaS!) providing everything on the move i.e, mobile.

So, what is the point I am trying to bring up? With all the possibilities of customization and promises of near zero time to market, and all the solutions around, we lost the focus on the “problem” that we are trying to solve!

If all these are solutions then what is the problem?

All the past year I have been working with the information management of the Insurance companies across the globe and interacting with the different levels employees representing the IT and business functions of those enterprises, the problems seems to lie in fundamental definitions of information flows. Legacy policy administration systems, multiple systems holding the data about quotations, pricing, products, policies & claims causing the “entropy” of not having the correct information to the different operational units within the business in the right time to make decisions seems to me the key problem.

As long as the real problem is not rightly understood with respect to the information flows between the organization units and within the operating environment, implementing newer technology solutions will not solve the problem.

I hope the new year 2013 gets the focus right on the problem of information and transforms the business based on facts i.e., what I call “Information led transformation” rather than the current trend of “technology led transformation”

Happy New Year 2013!!

Uncommon Sense, Common Nonsense - A review

Being a technologist, I rarely read management books. Even when I start reading a management book, I hardly complete reading it as it makes little sense to me. Recently, I came across this book “Uncommon Sense, Common Nonsense: Why Some Organisations Consistently Outperform Others” ( ISBN:9781846686009 ) and read till the end. 
 
This is a book which I could make some sense of business strategy and providing a leadership vision to an organization.
The book has four key parts dealing with:
1.        Winners and losers
2.        Strategy and tactics
3.        Organisation and management
4.        Biases and remedies
The fifth part gives Application and Examples from the author’s work.

In essence, there is a lot of “Common Nonsense” (in the form of big-data in today’s world) which is visible to us as well as the competitors (Same with “Common Sense!”).  There is an amount of “uncommon nonsense” about our own organisation known to us alone and that of the competition which is only known to them. But the key thing differentiates is the “Uncommon Sense” that makes organizations constantly outperform. The strategy is about deliberating means of constantly deriving and applying the “uncommon sense”

So, “Without changing our patterns of thought, we will not be able to solve the problems we created with our current patterns of thought.” Albert Einstein’s wisdom comes to rescue.


Overall, this is a good read for all the strategists and leaders in my honest opinion.

Aphorisms on Information Technology & Systems

Today, coincidentally it is 8888 days from 8/8/88 (The day I started studying Information Technology) and 1000 weeks from 11-Oct-1993 (The day I started working!)

Aphorisms on Information Technology &  Systems from Prasad Chitta

Some theoritical fundamentals on Information Systems & Technology as I see....

100 th post on this blog......

A generation of “infonauts” in the world of “augmented reality”

With the news around the new augmented reality (AR) game from Google called ingress, the mobile augmented reality seems to be coming mainstream. It is all about “engaging” the person with the device and taking the person into the device.  Once the person inside the device collect the behavior and finally “sell” the person something by making the person feel happy having that “thing”! Or it makes the person feel a loss without the “thing”!!
  
Google with its new massively multiplayer map-based mobile alternative reality game (MMMMARG as abbreviated by a Forbes article!) called “ingress” which has objectives to “check-in” to “portals” that may be designer shops etc., guides the “players” to accomplish the “objectives” 
 
So, the current generation is not just happy with the “reality” of the world. They need to augment the reality with information. Location aware, mood aware, temporally aware information to keep themselves busy (I mean really busy!) and active (I mean only mentally active!).  As the word rightly fits them is “infonauts” who are passing through the information. What is this information? It is all about things, likes and dislikes, deals and offers.  All these strive to get a “consumer” to a place where the producer makes money! 
 
Each move within the “augmented reality” is being recorded, modeled and used for making profits by the businesses. Just be aware. 
 
But, it generates a lot of “big-data” for me to handle and hopefully I continue to design solutions to manage that heaps of information and derive some meaningful (that make business sense!) insights for business to make profits!!

Data Munging in the Big Data world

Recently, when NASA announced a competition on a large US government data sets' data munging problem. It is called NITRD Big Data Challenge series @ http://community.topcoder.com/coeci/nitrd/

The first of the challenges was primarily about "How to create a homogeneous big-data dataset from large soloed data sets available with the multiple government departments such that some meaningful societal decisions can be derived from the knowledge generated from big data analytics"

So, the word coined almost a decade back "Data Munging" has come back into a key skill in today's world of "Data Science" discipline.

What is data munging?

In simplest possible terms, it is making data that is generated in heterogeneous platforms/formats to a common processable format for further munging or analytics!

How is it deferent from ETL/data integration?

Data integration and ETL are fully automatic and programmed where as the munging involves semi automatic; based on human assisted machine learning algorithms.

Why is it important now?

As the massively parallel processing paradigm based on map reduce and other so called "big data" technologies, it is a key thing now how the existing vast amounts of "data" be made available for such kind of processing to derive the knowledge by the means of analytic and machine learning algorithms.

There is an emergence of start-ups trying to generate platforms and tools for data munging are now in the market. In my opinion, this is going to be a key "skill" in future big data based "Data Science" discipline.

So, if you have good skills in data and algorithms based on assisted machine learning for manipulation then go for it!

Oracle database 12c

I have been seeing Oracle database form 6, 7, 8i, 9i, 10g, 11g and now it is going to be called 12c - c as in Cloud. OOW12 revealed the new architecture. (there was no tail for 6 and 7.... i stood for internet, g for Grid..... )

The new release with a fundamental architectural change to the data dictionary that gives a concept of "Pluggable Database" - PDB over the standard oracle system "Container Database" - CDB.

The obj$ of Oracle database dictionary will contain all the information of objects in the current architecture.  It is being split into CDB and PDB going forward for easing the "multi-tenant" private cloud databases.

This fundamental seperation of the data dictionary provides
a. Better security of multiple "databases" on the same instance
b. Seperation of CDB from other PDBs will allow easy upgrades.
c. All PDBs will share the instance and overall management of the consolidated database should be much simpler.

Excited about the cloud enabled, multi-tenant, pluggable database from Oracle! 

So, let us wait and see when the stable 12.2 will come out to roll out the new database into production....