Techno Functional Consulting

Pensions Regulation in UK

2011-12-30T06:01:00.000Z

As the year 2011 draws to a close, we are entering into a new year that is very important for UK workplace pensions. The new regulation around workplace pensions is coming into force in 2012.

Being purely technical for few years, I just want to test my skills around understanding regulatory documents and extract "Functional Information Needs" for a business.

A good review published in October 2010 can be accessed here: http://www.dwp.gov.uk/docs/cp-oct10-full-document.pdf

Basically, all the employers in UK have to automatically enroll all the eligible workers falling in a AGE range and EARNINGS range to a suitable pension scheme. They also need to "certify" the selected pension scheme meets the required quality criteria. (Refer to section 6.5 of the review document above) The requirement is to have at least 8% of earnings are paid towards a pension fund.

Regulation defines "Qualifying Earnings" as gross earnings that include commissions, bonuses, overtime etc., but most of the employers have the pension contribution basis as the "Basic Pay" i.e a pensionable pay. So, employer has primarily following options to "Certify" the pension scheme and comply with it.

Pseudo logic in plain English
1. IF pension contribution basis IS qualifying earnings (within the band) THEN pay the contributions of 8% and no certification is required.
2. IF pension contribution basis IS pensionable pay THEN
2a. CASE pensionable pay is 100% of gross pay THEN pay contributions of 7%
2b. CASE pensionable pay is at least 85% of gross pay THEN pay contributions of 8%
2c. CASE pensionable pay is less than 85% of gross pay THEN pay contributions of 9%
AND self certify pension scheme for all the employees participating in the pension scheme.

So, core information needs to implement this regulation is employee payroll data that covers age, all components of qualifying earnings of all employees. A bit of intelligence is needed to "model" the best possible grouping of employees and assign them to a suitable pension scheme(s) with one (or more) of the pension providers in the market.

The overall goal for a techno-functional consultant like me is to optimize the value of new pension regulations for employees, employers, pension providers and IT consulting companies by optimizing the information flows across various stakeholders!!

Wishing everyone a great new year 2012. Let there be peace, security and prosperity be with one and all.

Storing Rows and Columns

2011-12-04T04:42:00.001Z

A fundamental requirement of a database is to store and retrieve the data. In Relational Database Management Systems (RDBMS) the data is organized into a table that contain the rows and columns. Traditionally the data is stored into blocks of rows. For example a "sales transaction row" may have 30 data items representing 30 columns. Assuming a record occupies 256 bytes, a block of 8KB can hold 32 such records. Again assuming a million such transactions that need to be stored in 32150 blocks per day. All this works well as long as we need the data as ROWS! We want to access one row or a group of rows at a time to process that data, this organization has no issues.

Let us consider if we want to get a summary of total value of type x items that are sold in past seven days. This query need to retrieve 7million records that contain 30 columns each to just process the count of items of types x. All that we need is two columns item type and amount to process this. This type of analytical requirement lead us to store the data in columns. We group the columns together and store them in blocks. It improves the speed of retrieving the columns from the overall table quickly for the purpose of analyzing the data.

But the column storage has its limitations when it comes to the write and update

With a high volume of social data, where there is high volume of write is needed (like messages and status updates, likes and comments etc.,) , highly distributed, NOSQL based column stores are emerging into mainstream. Apache Cassandra is the new breed of NOSQL column store that was initially developed by Facebook.

So, we have a variety of data base / data stores available now, a standard RDBMS engine with SQL support for OLTP applications, A column based engies for OLAP processing and noSQL based key value pair stores for in-memory processing, highly clustered Hadoop style big data with map/reduce framework for big data processing and noSQL based column stores for high volume social write and read efficiencies.

Making right choice of data store for the problem in had is becoming tough with many solution options. But that is the job of an architect; Is it not?

numbers and counters

2011-11-11T05:25:00.001Z

There is some amount of hype on the 11-11-11 i.e., 11-November-2011... I see the numbers as just counters and they themselves do not make much sense unless identified with some meaningful thing.

It is 14581 days since i was born, 8495 days since i was associated with software/computers, 6240 days since i started working etc., etc., in all these counters "I" remains constant while the numbers move on...

Several other numbers, top 10s, fortune 500s etc., etc., also make some hype around from time to time; but it will be continuously replaced in the flow of the numbers.

Especially in the current era of very high importance to the numbers and counters the true importance of the "Identity" and "Intelligence" seems to have lost...

Hope the 11:11AM IST of 11-11-11 bring some common sense around in the world in general and Information Technology world in particular.....

All the best flocks!

I would like to add one memorable item from year 2001 on this occasion. We have completed our first "Consolidation" project a decade back and for that we got a 500 Million years old natural slate piece printed with a small message as a memento.

Most efficient multi-set Cartesian join in C

2011-10-14T07:11:00.001+01:00

At the beginning of my career, with Indian Space Research Organization, I have been posed with a challenge that required implementing a multi-set Cartesian product with absolutely minimum memory usage to solve an optimization problem. (see my old post on description of the problem: Simple looking Complex problem )

As a tribute to Dennis M Ritchie (also known as dmr) the creator of C language, who passed away yesterday, I am posting my implementation of this algorithm in C language.

I consider the above seven "highlighted" lines of C code as one of the earliest and most notable achievements of my career!

If there is any better implementation to solve the stated problem please let me know by posting a comment.....

ACID and BASE of data

2011-10-09T02:43:00.000+01:00

I am completing my 18 years of working in the field of Information Technology.

All these days an enterprise data store generally provides the four qualities Atomicity, Consistency, Isolation and Durability (ACID) to the transactions. Oracle has emerged as a leader in providing enterprise class ACID transactional capabilities to the applications.

Recently in the Open World 2011, Oracle announced a noSQL database which typically characterized by the BASE acronym. Basically Available, Soft state, Eventually consistent (BASE)

I see a lot of debate on SQL vs NoSQL, ACID vs BASE and Shared Everything vs Shared Nothing architectures of data stores of late; and with Oracle getting on to the NoSQL bandwagon, this debate is just took up additional momentum.

Oracle has posted this paper nicely explaining their NoSQL database. http://www.oracle.com/technetwork/database/nosqldb/learnmore/nosql-database-498041.pdf

In my opinion, SQL and NOSQL choice is straight forward to make:-

big query: Are we storing data or BIG-DATA (read my old post on transactional data vs machine generated big data - http://technofunctionalconsulting.blogspot.com/2011/02/analytics.html)

With the new trends in 'BIG DATA' all the data almost become key, value pair with read and insert only operations with minimal or no updates to the data records. NoSQL/BASE is best suited to handle this type of data. Still the traditional transactional databases of OLTP nature, needs ACID complaint transactions.

So, when designing the big data solutions, an architect should surely look at the NoSQL dataBASE. Is it not?

Publishing this post on 09/10/11 (dd/mm/yy) and this is my 85th post to this blog.

User Experience - HTML5

2011-09-28T17:35:00.000+01:00

Approximately two years back, I have made a post on "Rich Internet Applications" where the development of user experience focused on the plug-ins on the browser or run-time environments like Adobe AIR for rich experience. (See Desktop Widgets post.... )

As the technology progressed in last two years, the new/emerging HTML 5 seems to take on the web user experience design to the standard based, plug-in independent mode.

Some examples can be found on : http://www.apple.com/html5/

Another dimension today is the "mobile devices" along with the browser on desktop/laptop.

When it comes to the mobile devices and integration with the specific device capabilities, one should develop "native applications" to take full advantage of the native hardware of the device. Standards are good but Native Applications can do better. On the other hand, using the standards we can develop once and deploy on multiple devices where as native applications development requires "effort/time" on each platform...

So, there is no silver bullet for the problem of Rich User Experience needs of ever changing world!

Ancient advice applicable for projects & tasks

2011-09-24T17:46:00.001+01:00

I have been writing this techno functional consulting blog for past 4 years and I like to bring some ancient touch for modern Project Management:

अफलानि दुरन्तानि समव्ययफलानि च
अशक्यानि च कार्याणि नारभेत विचक्षणः

aphalAni = Those without fruit, durantAni = those with a bad ending (i.e., ends in failure), sama-vyaya-phalaani ca = and, those who are equal in effort and result (i.e., that do not end in either profit or in loss!), aSakyAni ca = and, those which is beyond the capability (i.e., impossible ones!) kAryANi = activities, projects, tasks na+ArabhEta = should not be started or initiated by vicakshaNaH= the wise man.

In my view there is only a 50% success rate in the Information Technology projects. So, it is wise to start only the projects that are sure to be successful. so, the ancient scholar of the above verse saying:

Don't take up a task/project if it is known to be:
a. meaningless or fruitless,
b. sure to land in a bad-end,
c. is of no-gain; no-loss
d. impossible or beyond one's own capability

We will improve the "success rate" if we follow this basic advice before taking up the projects & tasks!!

Given a chance I will put this in the beginning of PMP and PRINCE 2 certification material.

Tiered data storage

2011-09-10T17:32:00.000+01:00

Hierarchical Storage Management
Not too long ago (about 10 years back), I have done a strategy for "Data Archival" options for a system that has lots of data which need to be preserved for 25 years due to legal reasons. (sort of Records Management requirement) The requirement is to have it fully query-able fine grained data in the system. The key challenge was keeping all data in on-line storage with the technology available at that time. So, we need to have a clear "Archival Strategy" to move the data off from the disk to tape and preserve the "Tapes" in a way they can be retrieved (by methods of proper labeling etc.,) on-demand within the given service levels. This technology later named as Hierarchical Storage Management. Overall strategy included manual tiering of data between the disks and tapes sometimes using a mechanical robotic hands and associated software around them.

Information Life-cycle Management
As the technology advanced, the disk storage evolved to multiple bands of cost/functionality. The database software like Oracle came up with options like table partitioning and advanced compression. Combining these advances in the database management systems and the storage a new strategy emerged as Information Life-cycle Management. Logically partitioning the tables and putting them in the different types of storage like Enterprise Flash Disks (EFD), Fiber Channel (FC) and SATA disks using an automated storage tiering is the trend of the day.

Thin provisioning technologies like EMCs Fully Automated Storage Tiering - Virtual Pooling FAST VP and Hitachi's Dynamic Tiering etc., when used with Oracle's ASM and the partitioning & advanced compression options gives the best flexibility, performance and value for money. There is a good whitepaper from EMC with published few months back that can be found here.

Conclusion:
Most of the storage vendors now have the Tiered storage technology embedded in the disk controller software layer that can automate the data migration or intelligently cache and tire the data across multiple types of storage. Using the available technology with right mix of logical features of database and storage virtualization leads to better data availability at the optimal cost. Still the "right solution" is a job of a knowledgeable Architect! (who can understand the Business and Technology well!!)

web age of WWW

2011-08-08T06:46:00.002+01:00

As the WWW turns 20 years over the weekend (Link to the first webpage), my association with the computers turns 23 years today. The WWW is estimated to have approx. 20 billion pages as of today.

The information hungry world started making "Assets" out of information. Information has been classified as confidential, sensitive, internal, limited circulation, public etc., and some companies purely live only on "Informational Assets" today...

Protecting these information assets in the current day scenario of (operation shady RAT and reports stating that the claims of shady RAT themselves are shady!! ) hacking is truly a challenge. The information storage and its regulated flow to different end points need to be fully governed and secured.

My past blog posts related to the Information Security:
1. Data Security Technologies
2. Maximum Security Architecture
3. Identity and Access Management

with all these technology still there is a lot of "insecurity" among the technologists. Why?
Originally the information is published by the owner of that information and he/she would secure it with necessary proven authentication. Overall the information flow is between two known entities. (e-mail etc.,)
OR
Public information is broadcasted to reach maximum number of recipients. (spam mails etc.,)

As the WWW advanced to "Social" media the information is now being published by individuals for consumption by different like minded individuals who are directly known or unknown to the original publisher. This mode of information flow makes the whole process of information security very complex.

Technology surely can live up to the challenges that are posed by the trends in the information management area. Only thing needed now is cleaver brains to tackle the threats... It is all in the proper implementation of the available technology...

On this 8400 day of my association with computers and software, I am working on securing the information in the financial industry... Let us all hope we will have another 20 years flourishing, safe and secure WWW....

SQL Plan Stability

2011-07-10T03:19:00.003+01:00

Recently I came across a "performance problem" on Oracle database. A fairly innocent looking insert statement is intermittently taking "hours" to complete.

Problem:
INSERT INTO TARGET_TABLE ("Column List")
SELECT "values"
FROM SOURCE_TAB1, SOURCE_TAB2, SOURCE_TAB3
WHERE "All necessary join conditions and other conditions"

As the statement performing will in some instances and giving problem only in some cases, I have looked at the plans. It was generating two different plans - one with a simple nested loops and another with a Cartesian Join. When the second execution path is executed, it needed a lot of CPU and memory resources.

When I looked at the source of this query, it is originated in a job which is uploading a set of files into the database. Table stats are collected just before running the job. The development team has tested it several times in their database and they never had a problem with performance even in UAT environment. This behavior is only in the new environment that was built for the purpose of to-be production!

Reason:
As the optimized statistics are highly fluctuating from one file load to another file load, Oracle Database is generating different plans when the query is executed with different bind variables.

Solution:
Plan Stability can be achieved by
1. Importing the statistics from a stable environment and LOCKed.
2. By the way of providing Hints
3. By creating SQL Profiles
4. By generating stored outlines
5. By SQL plan Management (SPM) functionality in 11g.
We took a simple approach of importing and locking the statistics for the intermediate file upload schema to achieve the stability which worked well in the environment. but the most sophisticated SQL Plan Management functionality in 11g can solve most of plan stability issues. See this Oracle TWP - http://www.oracle.com/technetwork/database/focus-areas/bi-datawarehousing/twp-sql-plan-management-11gr2-133099.pdf for more details.

Notes:
As there is no silver bullet, one should be careful in implementing new features. This blog post explains the flip side of SPM and how to be careful with it.

To put it simply, be careful with setting optimizer_capture_sql_plan_baselines to TRUE. One can enable this parameter at a session level and capture the needed baselines and use them to get a consistent performance!

finally, however cleaver the RDBMS engine becomes, it can still commit blunders! an experienced DBA can never be replaced while dealing with performance issues!!

data consolidation (ETL) and data federation (EII)

2011-06-16T04:46:00.004+01:00

Operational IT systems focus on providing the support for the business operations & enable capture, validation, storage and presentation of transactional data during normal running of the operations. They contain latest view of the organization's operational state.

Traditionally, the data from various operational systems is extracted, transformed and loaded into a central warehouse for historical trending and analytic purposes. This ETL process will need a separate IT infrastructure to hold the data as well as it introduces some time lag in making the information in the OLTP systems available in the central data warehouse.

When the costs/resources required for consolidating data in the traditional way is not suitable due to the latest trends of acquisitions etc., there is a need for a different mechanism of data integration. The relatively different way of looking at this problem is to provide a semantic layer that can be used to access the data across heterogeneous sources for analytical purposes. This new way is called as "Data Federation" or "Data Virtualization" or EII - Enterprise Information Integration.

Key advantages of EII are quick delivery and lower costs. Key disadvantage is the performance of the solution and dependence on the source systems.

A good use case of data virtualization in my view is to consolidate different enterprise data warehouses due to mergers/acquisitions.

Traditional ETL and data warehouse technology vendors are coming up with data federation tools. Informatica Data Services uses a consolidate data integration philosophy where as Business Objects data federator uses a virtual tables in the BO universes for providing same functionality. Composite Integration Server is the independent technology provider in this area.

Key considerations in selecting the data federation and associated technologies are
1. native access to the heterogeneous source systems
2. capabilities of access method optimization
3. caching capabilities of the federation platform
4. metadata discovery capabilities from various sources
5. ease of development

A carefully chosen hybrid approach of consolidation and federation of data is required for a successful enterprise in the modern world.

SQL performance tuning

2011-04-30T06:50:00.003+01:00

Having seen several performance problems within IT systems, I have a methodology for performance tuning. When it comes to a SQL query tuning, it should be slightly different.

9 out of 10 cases of performance problems on relational database systems relate to a bad SQL programming. Even with the latest "optimizers" within the commercial database management core execution engines, it is the skill of the developer to make use of the facilities effectively to get the best out of the RDBMS.

I have recently came across a typical problem with a "junction table" design.
A set of tables represent USER and all are connected by a USERID
Another set of tables represent ACCOUNT and all are connected by ACCOUNTID

The software product implements the one to many relationship using a junction table called USER_ACCOUNTS which contains (USERID, ACCOUNTID) with the composite primary key USERID, ACCOUNTID.

Now there are 30K users 120K accounts and 120K USER_ACCOUNTS and a query that need to get data from USER tables involving some outer joins on itself and ACCOUNT tables which joins multiple tables to get various attributes; all these tables linked in a join using the junction table. That query runs for 18 hours.

When the query is split into two inline views with all the required filtering in each side of data access on USER and ACCOUNT individually and then joined using the junction table it completes in 43 seconds.

So, FILTER and then JOIN is better than JOIN and then FILTER in terms of resource consumption. Hence the performance tuning is all about doing the sequence of actions in the right order to minimize the consumption of resources to perform the job!

Data Serialization, Process Parallization!

2011-04-26T17:08:00.003+01:00

There is a lot of buzz around: going away from traditional data processing i.e., a relational database and persistent data in relational form being processed by a set of processes that capture, process (validate, summarize, re-format etc.,) and present (display on multiple format displays over multiple channels in verbal and multimedia formats) that data.

But where are we going? Object orientation of encapsulating data with its own operations to make loosely coupled application services those can be orchestrated to form business services with in an enterprise.... Those enterprise business services further choreographed to form a business to business flows across common interfacing models...

The traditional computer architecture that has a Processor that can process the data which is stored in a distinct Memory of the computer. The processor and memory are two distinct components of the basic architecture of the modern computer. When an "object" needs to be stored or shared between two different applications one should "serialize" we have Hibernate and JSON etc., formats developed for this data serialization...

At the same time, there is a trend that takes over to process the data more and more in parallel streams in the shared nothing style clusters to break the typical task into smaller pieces and summarize the results in a hierarchical fashion to arrive at final result. This can happen when the data becomes more and more unstructured with the help of objects!

Overall the trend means we are slowly going away from structured data stores in traditional relational databases and going nearer to natural language, fault tolerant and predictive data capture and processing (e.g., you can type any spelling on Google and it will return results for the right word!) and more visual and multimedia presentation of the information (in mashups, maps) with bi-directional interaction (like social, I can "like", "comment" etc on the presented data as feedback!)

That is just my view.... Businesses have to gear-up quickly to adapt to these trends!

Leaving IBM...

2011-03-31T09:19:00.003+01:00

Having worked with IBM for few months I have decided to move on. It has been a great experience to work with the world's no 1 software company. It is very special as IBM is celebrating its centenary as an organization.

I have been working in smart energy initiatives of Application Innovation Services division. Theme of my work at IBM is "Smarter Grids" (Follow the link to see what is a smart grid!)

I would take this opportunity to wish all my colleagues "All the best" until we cross our paths in this small IT world sometime, somewhere!

Integration - Data or Information, Application

2011-03-22T11:59:00.008Z

Having written a post on topological differences (EAI vs ESB) of integration long time back, and management approaches (centralized vs distributed) I am making an attempt to look at approach in terms of data integration vs application integration.

We have slightly touched upon this subject of Application Integration on one of the earlier post - Binding energy in software systems.

So, what is data integration? A typical Extract-Transform-Load - ETL, data migration, Change Data Capture - CDC, Master Data Management scenarios are classified as data integration. There are several platforms from big and small vendors to achieve this. (IBM InfoSphere/DataStage, Informatica, Microsoft, Oracle, Pervasive have suites/products under this head)

How is this different from traditional Application Integration? Application Integration focuses on integrating business process supporting the information workflows. Data Integration focus is primarily on the propagation and synchronization of data across the enterprise system landscape sometimes spanning into Cloud..

When the volumes are high, a data integration based approach has an advantage; If the process/workflow complexity is high and orchestration is needed to achieve the integration then one should look for an ESB/SOA style integration.

Enterprise Data Integration platforms are comparably similar to EAI platforms in their nature of technical architecture (hub/spoke based EAI topology) with a ETL hub that loads the data into a warehouse.

Of late, the buzz is toward real-time data integration based on CDC etc., let us see how it goes and changes the game!

Architectural Approaches

2011-02-25T08:26:00.004Z

Having written a small post on "Enterprise Solution Architecture" some time back on this blog now I want to touch up on the Architectural Styles/Approaches.

Typically the architecture discipline is about making "models" and naturally the architecture is thereby model-driven. Model Driven Architecture or MDA is an approach for building various models using the UML (Unified Modeling Language) for defining "structure" and "behavior" of a system being modeled.

But, a "model" can only represent a specific view point of the "system" being modeled. So, there should be a standard set of view points fit into a framework to describe a typical Enterprise. The open group's TOGAF and ADM tries to do that by defining specific "architectural layers" i.e, business, information, application and technology with multiple view points as functional view, security view, user view, communications view, management view etc., The ADM gives a methodology to select the key stakeholders and required viewpoints the architecture needs to be developed.

IEEE 1471 gives a recommended practice of "Architecture Descriptions" that generalizes specific frameworks in generating these models within the system's context, stakeholders and their specific needs.

But there is never been an architecture that related to Information Technology that is purely greenfield. There is always an "As-Is" architecture and a "To-Be" architecture that will be built based on current problem domain. SEI's ATAM is a tradeoff methodology to evaluate the architecture and evolve the architectures.

So, architecture discipline develops models that go out of sync by the time the solution goes live into production and starts solving the problem. It is really difficult to keep the models in sync with what is the reality on the field.

The Service Oriented Architecture i.e, SOA looks at the Enterprise as a set of loosely coupled services that interact to run the enterprise in its environment. This gave a rise to deploy and host the services in a marketplace like environment called "Cloud" that changes the paradigm of architecture into "Cloud Computing".

While a majority of architectures even in SOA are flow based, there is a different approach that is available is Event Driven Architecture or EDA. This approach looks at the events processing and events triggering various workflows in a business environment. With the complex-event-processing that correlates events over cause-effect, spacial or temporal dimensions it has specific uses in service management and Business Performance Management areas.

While solving complex problems all the above different architectural approaches can be used based on their fitment and availability of time and resources.

But the key is to have the right set of people, processes and tools in developing these architectural views!

Analytics

2011-02-16T11:44:00.009Z

Analytics is the "science of analysis" i.e., dividing a large set of data into certain "themes" and understanding various relationships between these to make out some business sense and take it for strategic advantage or for efficient operations is the field of Analytics.

In olden days, data that used to be stored in the computers is predominantly "Human generated transactional data" i.e., when an transaction happens between a producer and consumer, the data related to such transactions was stored in the software systems. This is relatively small amount of data.

As the days progressed, various "machine generated events" like a customer's website views, different clicks, data from automated sensors (like RFID etc., ) and various system log events are being stored for analyzing the behaviors.

This post is to enumerate different mathematical models and their uses in the field of business and web analytics.

1. Descriptive models: Used to classify the data into different groups. For example deriving the age of a person based on the first driving license date. Determining the sex based on height and weight etc., Focus is on as many variables as possible.

2. Predictive models: Used to find the causal relationships between the themes of data. Focus is on specific variables. These models give a probability of a set of outcomes.

3. Optimization/decision models: Used to derive the definite impact of certain decision and optimize the result within a set of constraints based on the data.

PMML = predictive model markup language from dmg is the xml based standard that can be used to exchange the models across multiple supporting applications.

The trend is in-database analytics that brings the data analytics into the database core engine and databases that are specifically built for the purpose of analytics based on columnar storage that makes the database an "analytical database" instead of a mere data storage and retrieval engine.

Oracle has published a good reference paper on this subject that can be found here -Predictive Analytics: Bringing tools to data.

over and above the thematic analysis there is an increasing demand for spacial and temporal analysis of the data. The field of analytics will converge into a single set of tools where one can analyse the data using the slicing and dicing functionality on all the dimensions of themes, spacial characteristics and temporal analysis at the same time with loads and loads of machine generated data is not far in the future....

Recently, I came across this paper that presents a framework for thematic, spacial and temporal analytics that can be possibly combined with data mining option....

Accountability and Authority

2011-01-11T11:54:00.003Z

Back to fundamentals on the project management on this 11/1/11

The RACI matrix is a well known tool to identify the

Responsible
Accountable/Authority
Consulted and
Informed

parties involved in successful completion of a "Task".

Out of these four types of parties, there can be many responsible, many consulted and many informed about it but there should be one and ONLY ONE finally accountable for every given task within a project plan.

Why?
There should be one person in complete authority of a task who is empowered to take decisions during the task execution and makes sure the task in completed with the required quality within the time and within the effort allocated to the task. The person should have the final 'authority' to sign-off the deliverable.

In these days, technical architects are made accountable for tasks without necessary authority assigned.

Influencing people without authority and ability to be accountable for tasks is one of the key (soft) skills of a technical architect.

Integration - Centralized or Distributed?

2010-12-17T14:04:00.005Z

Let us imagine a big greenfield IT program that implements multiple packaged products, custom developed applications for an enterprise business.

There are multiple software vendors, package implementation partners and custom development teams, that are involved in the complete life cycle consisting of requirements analysis, architecture/design, code & test, integrate and deploy to production.

In such a scenario, how to deal with the interface development and integration?

Option 1: Centralized Integration Development
1. Agree the interfaces as step 1 of the program.
2. Let an independent group of people (let us call center of excellence) do the integration/interface work.
3. As the different products, applications are deployed the integration team will integrate them.

Option 2: Distributed Integration Development
1. Only have a integration back end (say ESB) be controlled by central team.
2. Each team working on different projects of the program will have their own interface development team.
3. As the different products, application are deployed the respective team will integrate them with the central ESB.

Which of the above options is better?

In my personal opinion, all the integration related activities are better done from a central COE.

Central COE can share the best practices, identify core patterns and leverage the skills.

Challenges in the COE model are acquiring and building the right skills and right sizing the COE for the program.

Any thoughts or ideas?

Historize or Roll-Up

2010-12-06T13:26:00.007Z

Information Technology is all about acquiring, processing, storing and presenting the "data" to the right people at right time to enable them to derive some meaningful information and in some cases useful intelligence or insight out of that data.

In usual business, a data point is captured only when there is a transaction that changes the data. So, each change to the data is captured, validated and stored. Such systems are called OLTP or On-Line Transaction Processing systems.

But, when the data is acquired at regular intervals in typical process control systems, not all the data points required to be stored; For example
a. the utilization of a system processor at every 5 seconds interval
b. Temperature of a steam turbine scanned at every 400 ms interval

In a typical process control system there will be several thousands of such data points scanned at very high frequency, typically every second.

What to do with all this data?
In a standard relational database storing all this data in raw format will be simply impractical. So, there are two methods to make some sense out of such "time series" data.

1. Historize the data using a process historian. A process historian uses a compression algorithm that only stores a data point only when there is a deviation beyond a set limit using variety of algorithms like straight line interpolation (SLIM1, 2, 3) or swinging door compression etc., to achieve a high degree of compression in storing the time series data.

2. Roll-up data on a periodical (i.e., hourly data for few weeks) basis to store max, min, average, standard deviation etc., values in one single record per data point. A next level roll-up of data can happen for a longer time interval (i.e., daily data for several years) This multi-level roll-up data can be used for historical trending purposes.

There are advantages of both methods. Recently I have seen a patent on dynamic compression of system management data which is interestingly putting the Historization with multiple compression algorithms for storing the systems management data.

Energy Storage - Pumped Storage Hydro

2010-11-28T14:45:00.005Z

Q: How to store the energy?
A: In its potential form...

When electricity is produced from an hydro electric plant, during off-peak hours of demand, the produced electricity can be used to pump the water to a higher reservoir using a reversible turbine/generator assembly.

The non-conventional "wind turbines" or "solar cells" can also be used to pump the water to a higher reservoir, thereby making the electricity production from them more reliable to connect the generated energy to the electricity Grid!

Once the water is available in higher reservoir, electricity can be generated as and when needed depending on the demand.

This is the largest capacity form of stored (potential) energy available for a Grid Connected storage.

In India we have one 1000MW plant at Tehri Dam.

In my personal opinion, this is one of the best and environmental friendly solution to store the electricity!

smarter grids

2010-11-18T10:30:00.002Z

I have been at a conference related to Smart Grids at Delhi last week. There was a question asked by a BA student to me "What is this smart grid all about?"

I will try to answer this question as simply as I can in this post.

Electricity traditionally supplied to consumers from its generating plants using the "power GRID". The interconnected network of generating plant -> transmission grid (high voltage) -> distribution grid (medium voltage) -> Consumer (at 220Volts) is the power grid.

Due to the amount of investment into this core infrastructure of GRID, it has been a public sector (government) agency that built the grid. The way consumers buy or pay for the energy is dictated by the utility company.

As the infrastructure development slowly got de-regulated and more and more private participation is into power generation and "open access" to the power across the grid is being made available for consumers.

"Energy" is freely available in the nature in the form of Sun light, wind, tidal, flow of a river, coal, nuclear etc., forms but making it available in a house to the lights, fans, ACs and several electrical appliances like pump sets etc., costs money.

Energy has an initial "conversion" cost i.e., converting fuel to heat in a boiler and heat to mechanical movement of a turbine and the mechanical energy to electricity by a generator. Then it need to pass through the transmission and distribution channels to the consumer which adds a "transportation and distribution" cost. Finally the consumption at each end point is measured by a Meter that involves some costs of metering, billing and collection of bills through operating some channels like collection kiosks, bank direct debit etc., from the final consumer.

How is all this related to smartness of the GRID?

A customer has not much a role in the whole process... He uses the Electricity and pays the bill monthly... It is a one-way flow of information and energy!

As the demand for energy increases and the fossil fuel like coal decreases in availability one should think of methods to wisely use the energy. A smarter grid is an enabler for wiser generation, distribution and consumption of the energy.

A "smart meter" that can measure more accurate consumption and power quality at more frequent interval (e.g every one hour) gives a better understanding of consumption patterns for the consumer and utility, there by the energy pricing can be more dynamic as different price can be applied for peak hours and another price for non-peak hours.

A "smart meter" enables micro grids. Where ever possible small commercial customer can have their own "solar" or "wind" generation capability and when they have excess of energy they can reduce the consumption from the grid or even feed the energy back to the grid. So, the meter can measure two way i.e., import and export of the energy from and to the grid.

This makes the distribution grid to be more responsive to the demand and supply scenario. A distribution grid should instrument each transformer, feeder and substation to measure power quality and other key parameters for efficient use of the equipment by minimizing the asset maintenance costs and failures of the equipment. It will also ensure the energy lost over the transportation to a minimum. (Currently globally a 50% of generated energy never reaches any customer!!)

An interconnected set of devices they can exchange information along with the energy in both directions for the purpose of making intelligent decisions for wise consumption of energy/utility makes up a smart grid.

A customer is more informed about the prices of energy he is using at different times of the day, Utility can control some high power consuming equipment at customer premise remotely, information exchanged between the customer, distribution, transmission and generation entities in bi-directional, automated mode.

It will finally lead to conserving the energy, managing the demand and supply balance in a wise manner and finally make the planet EARTH more greener! That is the whole idea of smarter grids. Advancements in telecommunication and information technology enable the transformation of power grids into smarter grids.

I hope this post is simple and smart enough to explain what is a "Smarter Grid" to a novice.

Entering 888th work week

2010-10-10T05:43:00.004+01:00

"If you can't explain it simply, you don't understand it well enough." — Albert Einstein

I have completed my 17 years working in Information Technology industry. In past 17 years I have realized the essence of above statement by Einstein.

Most of the times, people can't explain things simply. The problem IT industry is seeing today is the focus on "Information" has been lost completely and the whole industry is running behind "Technology"!

I wish all the best to the industry on this special day of 10/10/10 at 10AM IST.

Review of "Quality Reviews"

2010-10-08T06:43:00.007+01:00

Review means a "critical inspection" of a product or a process by someone who was not involved in the creation or regular operation of it.

Review is considered to be one of the important methods of Quality Assurance (QA) of software systems.

Review in software product / development:
Now, let us take a "review" of the review process in software product.

1. A software deliverable is architected as per the business requirement. The software architecture is reviewed by a reviewer to assure that architecture meets the functional and non-functional (scalability/performance/interoperability etc.,) requirements of the business.

2. Software deliverable-s are designed as per the architecture. The design is reviewed by a reviewer again to assure the design meets the architecture.

3. The design is implemented into software program-s (code) and the code is reviewed with reference to the design.

4. Each unit of code is also tested by the developer or an independent tester to make sure code has met the design specifications!

5. All the different units of software are integrated to realize the overall architecture of the software product. The integrated product is tested for functional and non-functional requirements.

6. The product is released to the customer.

So, review is preventive measure of quality assurance that helps in avoiding injection of bugs where as testing is a reactive method of finding and fixing the bugs after the bugs are injected into the product.

Why software still have bugs?

In my opinion, the primary cause is lack of "self-review" of the work products delivered by the individuals. Each individual should be encouraged to develop a culture of reviewing the work-product (let it be the architecture, a design, a unit of code, a test spec, or anything else) before it goes out into the software development process chain. Once the organization is successful in this mechanism, the quality will automatically improve and the cost of QA will considerably reduce.

Note: QA managers need not worry... They still have their jobs safe!! Guess WHY?

Review in process outsourcing:
Looking from a different perspective of BPO or ITO scenarios, the quality of a process where development of a "deliverable unit" costs nearly same as the review of that unit, the review adds a 100% overhead and not a solution for the quality assurance. The philosophy should be driving the workforce to "Do it right first time"
within the execution of the process.

But, a Business Process Review and transformation to:
a. Identify possible scope of improvement
b. Evaluate the impact of change
c. implementing the proposed transformation in a smooth manner

will help the process to be more efficient with an improved productivity and customer satisfaction.

Conclusion:
a better process --> a better quality product.

Hence one should "review" the process first to improve the process efficiency and then focus on "review" of individual products produced by the process for a better quality.

Software development is also a process! Increased focus on a review of a wrong thing will not result in the improved quality.

Binding Energy of Software Systems

2010-10-01T10:56:00.008+01:00

When we study basics of atomic physics, we came to know that "A bound system has typically lower potential energy (i.e., mass) than its constituent parts" To make it into simple words, the total mass of all nucleons is more than the mass of nucleus formed by them. This mass deficit when converted to energy equivalent is called binding energy. That is the "force" which keeps the system together and not let the different constituent components fall apart!

So, In software terms, this is the effort that has gone into the "integration" of the different system components into its final form of Business Application.

Traditionally the integration has followed different models in the software systems.

Silos: Multiple software components were developed on a specific technology / programming language like COBOL, C etc., and they are integrated vertically using the procedure calls and RPCs. It is difficult to integrate with a component that is outside the technology.

Star/Spaghetti i.e., point to point integration: In this method, different components of a business application talk to each other using the flat files, or other methods. As a need for integration arises, the necessary interface should be developed and deployed on both the interfacing components of the business application. Soon, we will have a very complex spaghetti created that is very difficult to maintain.

Hub and Spoke based EAI: To overcome the standardization problems of point to point interfaces, each component should talk a "common language" with a central HUB that mediates all the integration between the enterprise business systems in that common language. This technology has developed several standard adaptors for common business components.

Enterprise Service Bus (ESB): This is the most modern integration technique available today. The key difference is that the central HUB is replaced with a more open set of protocols that can integrate the business components beyond a single enterprise. It is more open and allows more loosely coupled, heterogeneous components to talk to each other by providing more sophisticated "translation" services to them.

So, it is important that sufficient "binding energy" is in the Enterprise Business System and the CORRECT structure/method is used for the integration to keep the software strong (for operations) and flexible (for tackling the changes) all the way through its lifetime.

For more information on binding energy (atomic physics) : http://en.wikipedia.org/wiki/Binding_energy

For more on Patterns in Business Service Choreography using ESB (IBM redbook): http://www.redbooks.ibm.com/redpapers/pdfs/redp3908.pdf

Condition Based Maintenance

2010-09-20T14:27:00.004+01:00

CBM as it is well known in the Enterprise Asset Management terms is a buzzword for sometime now.

This post aims at making CBM simple to understand.

Any "Asset" would need maintenance to give its optimal performance. Traditionally there are two strategies to manage the "Maintenance"

1. Wait till the equipment fails. As soon as it fails, fix the equipment/Asset. This is "reactive" maintenance. (Corrective Maintenance) The disadvantage is the unplanned downtime. You do not like the car to be broken down before the maintenance is done. is it not? But the advantage is we fix only what is required making it very "cost effective" mode of maintenance.

2. Maintain the Asset in a periodic schedule (Preventive Maintenance). Just like a car going for a service after every 5000 km run or after every 6 months. This will make the Asset perform without a breakdown, but the maintenance is expensive, we will change the oils etc., even when they can be used for further period. It amounts for more costs for maintenance.

So, there should be a more "smarter" approach for asset maintenance. Is it not? that is called a "predictive maintenance" or Condition Based Maintenance. Let us assume if we instrument the asset with more sensors they can exactly "measure" some key parameters, that can be used to compute the exact condition of the asset health and recognize certain "conditions" before they happen. Using this information, we can carryout the required "maintenance" of the asset. This approach will surely help reducing the maintenance costs and asset failures.

But, it adds additional instrumentation of sensors, data collection and processing the data to "intelligently" identify the conditions. When all this additional instrumentation, data collection and processing is considerably cheaper than the maintenance expenditure and it avoids a breakdown it is worth implementing it. Achieving this in a "smart" way is the key!

EAM with CBM will surely improve the operations of any "Asset Based" organization.

But when we look at the God's own creation, we see CBM everywhere... Macrocosmic System of the "World" and the microcosmic system of the "body" both implement the CBM to its nirvana. Both the systems sense the conditions, process them, present the conditions in a meaningful way and they also self HEAL the condition naturally which is the nirvana of "Asset Management"!!

My Technical White Paper on oracle.com

2010-09-14T04:46:00.002+01:00

This was the last piece of work I did at Oracle before I left the company in June 2010. The link to the published paper http://www.oracle.com/technetwork/oem/app-mgmt/twp-next-gen-siebel-monitoring-164300.pdf

It is always a pleasure to see the customer happy after implementing the solution and makes a comment like this

“We use RUEI on technical administration to get information, how the application “feels” on customer side and for management point of view to get the information of SLA agreements in fact of guaranteeing a dedicated response time to our customers. We verify the SLA with RUEI KPIs which are implemented in a management dashboard.”

I randomly came across this paper on o.com and thought of sharing on my blog!

Enterprise Solution Architect

2010-09-13T05:23:00.007+01:00

My teenage son sometimes wonders about what I do in the office to make my living....

Let me try to define the phrase "Solution Architecture". To do so, I got to first define the words that constitute the phrase in the correct context.

Solution :
A SYSTEM continuously experiences environmental change! Change in the system environment is a natural process. Sometimes the change is gradual and other times the change is sudden.

In any case a system (an enterprise itself is a SYSTEM!) has to have a built in mechanism to tackle the change.

When a "change" can't be handled by the system within its scope of operation, this change is called as a "problem" that needs a "Solution"

Architecture:
This is the tricky part of the phrase "Solution Architecture"!

Architecture is a discipline, a set of principles, methods, guidelines that can be used to create a "model" of a "system" from multiple view points of different stakeholders of the system. Once the model is constructed, it can be used to get a holistic view of the system by different stakeholders. It greatly helps the understanding of the problem and channelize the thought towards the challenge that is caused by the "problem"

Overall, the solution architecture is a discipline of finding the opportunities to improve the system gently to tackle the challenges posed by the environmental changes as well as making the system more "responsive" to future challenges by creating a multi-dimensional model of the system!

Enterprise Architecture

In the modern day every business organization is seen as a SYSTEM. So, Enterprise Architecture (EA) discipline is divided into four layers as

Business, (what makes up the business - people, processes and tools; Its goal, structure and functioning)
Data/Information,(what are the key data comes in and information needed for business functions both operational and strategic)
Applications (How the data is captured, processed, stored and converted into the useful information and knowledge and presented to the right people at the right time!)
Technology/Infrastructure Architectures. (What physical servers, network, storage components are required to run the applications within the budget meeting service levels)

With the "Cloud Computing Paradigm" is in, the business is seen as a loosely coupled services and each of the service can have three layered "clouding" in SaaS - Software as a Service, PaaS - Platform as a Service, and IaaS - Infrastructure as a Service. This cloud computing has changed the way we look at architecture in the Data/Information, Application and Technology/Infrastructure layers. An architect should consider the possible options of public (external supplier hosting the cloud), private (hosting a cloud within enterprise datacenter0 or hybrid (a combination of public and private) deployment models of the Cloud in these layers.

To make it simple, as an Enterprise Solution Architect, I draw different shapes and name them differently; I will connect those shapes to form a smooth looking flow between multiple layers of the enterprise and convince the key stakeholders that they have understood their current problem and different possible solutions to solve their problem. I will help them select the best solution option suitable for their budget and needs.....

It is FUN as I enjoy this!!

Risk Management

2010-09-05T12:19:00.007+01:00

Risk indicates a future event with a probability of occurrence that could cause an additional cost or a reduced revenue for a portfolio.

In a trade where the producer with stock in hand selling the product directly to a consumer who is buying the product for cash, there is no risk involved.

But, when a producer is selling the product to a consumer on a monthly installment basis, producer evaluates and manages a possible "credit risk".
Similarly when a consumer is paying cash in advance for product that will be delivered later, the consumer is evaluating and managing the "operational risks" of the producer.

When the transaction do not involve either producer or consumer and dealt by the "traders" then the risk is on its peak! When a trader purely buys a commodity or an instrument for the sake of selling it for profit in future and not for his own consumption; then it calls in for a methodical evaluation and management of "RISK".

In the modern day markets, there are multiple layers of traders and exchanges that are involved, making the "risk management" highly complex. There are spot, index based, options and future contract type trades that happen in the financial and commodity sectors, which made "risk management" a key tradition in the trading business!

We have several mathematical simulation models and supporting IT solutions in the Trading Risk Management models today. Several organizations are in the process of implementing and modernizing the "risk management" solutions to make their business make better sense.

But there is a "risk" in choosing and implementing the right "risk management" solution itself. How to manage that risk? Sounding tricky??

Warren Buffet's quote on this goes like this: "Risk comes from not knowing what you're doing." Taking this simple guiding principle, one can tackle the risk easily!

The impact of risk can be surely reduced by right KNOWLEDGE of what is being done! If you do not have the right knowledge, transfer the risk to someone who has the right knowledge!

Personally, I have successfully transferred all the risks to the most knowledgeable one thereby became completely risk free!

Identity and Access management philosophy

2010-08-17T11:07:00.002+01:00

Security of Information in the Information Technology world is very important component.

IT always revolves around making necessary information accessible to the authorized person as and when required to carryout his/her duties and help taking crucial decisions.

So, this post is about "identity management" in the IT world.

Identity can be established by one or more of the three factors:
1. What the requester "knows" e.g., password
2. What the requester "has" e.g., a token that generates random numbers every few seconds
3. What the requester "is" e.g., finger print or retina scan

Before letting the requester access a resource of IT infrastructure, the first thing to be done is to establish the identity of the requesting entity. Once the identity of the requester is established, the second step is to make sure the requester has necessary access to the resource.

The requester may be authorized to
a. read or view the information resource
b. make some modification to the information resource
c. create a new information resource
d. delete one of the resource

Some times the access is dependent on the value of the resource (a bank officer can only approve a transaction of value less than "some max limit")

Separation of duties:
One requester can only create a table and maintain it, but will not be able to read the data from it.

so, the "access policy" could be very complex to define, maintain and enforce it on the IT infrastructure. One very evolved method is Role based access control - RBAC.

There are multiple vendors who provide the solutions around Identity and Access Management in the IT world.

But, interestingly this problem was ancient and in the Great Indian epic of Ramayana, Hanuman establishes his identity to Mother Seetha using a two factor model with a token and a pass phrase! I think we are only applying an ancient solution in a modern way with IAM solution stacks...

Some modern architectural patterns are on this RedPaper..... for those who are interested!

8035 days or 22 years!

2010-08-09T16:17:00.003+01:00

It is 22 years since I have associated myself with computers and software. So what?

The key point to make is my recent move from "Grid Control" to "Smart Grid" ...

Having worked for past 4 years with Oracle working with computing grid management software, I have moved over to IBM into consulting solutions around the power grids!

It is just few weeks into the new role, and currently focusing on the domestic market. A new role, in a new place with only thing common to my immediate past role is "GRID" but in the both contexts, this same word means completely different!

With a new learning experience everyday, "Grid" is going to keep me active for several more cycles...

Is it??

1 - n relationship from 1 - 1 relationship

2010-07-12T13:40:00.001+01:00

Comments?

The "Pull" mode process!

2010-06-23T06:16:00.004+01:00

This is my favorite subject that is applicable in Marketing, Production, Consulting and also in Philosophy.

The question here is what is the best method of process control?

a. should a process push its output to the next stage? or

b. should a process pull its input from the previous stage?

In marketing:

Do we sell the product to customer or

Do we make the customer buy our product?

Who should be an active participant of a transaction?

Is it the giver?

the taker?

Pull mode is the natural and sustainable model in most of the situations. A process should be designed to make the next stage "pull" the product of the previous stage.

A customer sees the "value" in the product and starts the buying process.

Once the "buy" in initiated, all the necessary steps will be executed to fulfill the request.

The skill of designing such a process lies in reducing the "waiting time" of the customer when he requires the product. Depending on the type of the product, the sufficient inventory should be maintained. But even in the case of customer to wait for a reasonable period, it is fine. (As the value is attractive, customer will wait!)

Pull will gently fill a gap and add value to the customer.

Push will generally "replace" something at the customer. (or cause over consumption or wastage!)

Present World human population is suffering from a "Great PUSH Syndrome" in all the production to consumption processes. Customer has really no time even to "think" about it. That is the force of PUSH in this world as of today.

I can only hope customers and the producers will understand the gentle nature of "pull" process as explained and make the world a better place by using it!

Send-off to Prasad (Good Luck Prasad)

2010-06-16T16:52:00.001+01:00

Today, after having spent 4+ years at Oracle in Enterprise Manager Grid Control Strategic Customer program, worked with internal and external customers, I am moving on. My Oracle colleagues have given me a wonderful send off treat at Little Italy. This picture is when Gagan is presenting me the gift of "Kautilya's Arthasastra (Three volume set!)"

I would thank everyone of my colleagues who made my stay at Oracle a really memorable one. It has just gone in a jiffy starting with internal customers, training the Japan team, working with customers in Australia, Korea, Hongkong, China, India, Germany and North America. The trips to Oracle HQ for Open World 2006 and 2009 and CAB are very memorable. It is the demo booths, sessions and Hands on Labs and various activities around helping customers achieve their goals and enable them serve their customers better!

I wish all my Oracle colleagues all the best!

Maximum Security Architecture

2010-05-01T15:01:00.003+01:00

In one of the past posts, I have just listed different technologies that are available in the Data Security area. (in Feb 2008)

With Oracle 11g database, the security focus has taken more methodical and architectural approach.

To put things together data security is placed under the following broad (four) categories:
1. User Management
2. Access Control
3. Encryption and Masking
4. Auditing/Monitoring

Just like Maximum Availability Architecture for Highly available architectural patterns, we can call this as Maximum Security Architecture for highly secure architecture....

One should choose the required options and implement it properly to really make the data SECURE!

This Link gives more details of MSA on Oracle 11g database.

Enterprise Data Fabric - data grids

2010-04-10T08:46:00.004+01:00

Enterprise Data Fabric or "in memory data grid" can improve the distributed or clustered application performance dramatically.

The Problem:
Distributed applications need to share the "application objects" across multiple processing nodes. As the application objects are not "relational" there is a need for the Object Relational Mapping (ORM) to share the application objects using a relational database.

The ORM technology involves converting the object into a set of relational (table/column) information and and turns slow.

The Solution:
Using an in-memory data grid to store the application objects on distributed multiple node cache and managing the transactions in once and only once manner.

This will dramatically improve the performance and scalability of the application.

This is also going to be a key enabler of the Private PaaS Cloud Computing in future.

IBM WebSphere eXtreme Scale and Oracle Coherence are the examples of Distributed Caching Platforms that provide Enterprise Data Fabrics or in-memory data grid solutions.

Let us see how this technology shapes up the future!

migration, upgrade, enhancement

2010-03-23T09:12:00.003Z

Migration of a software application from one platform (32bit OS1 on xxx hardware) to another platform (64bit OS2 on yyy hardware)

Upgrade of a software application involves upgrading the software from a lower version to a higher version (like XX version of database to YY version of database)

Both the above activities are technical and no business benefit is directly attributed to this type of activity on the software applications.

An application enhancement will involve providing a new feature or functionality to the end-user of the application. Sometimes when the application is majorly enhanced, the new enhanced application can get migrated to a new platform and/or the underlying technology stack upgraded to higher version.

The key question I am dealing in here is:
When there is no "enhancement" involved, purely a migration and/or upgrade is planned:
a. is it good to do both together in a big bang? OR
b. is it good to migrate first to the new platform and upgrade the software later? OR
c. is is good to upgrade the software first and then migrate to new platform?

a. One can do it together but if there is an issue it is difficult to find the cause of the problem due to new platform or new software version.... each can point fingers at the other....
b. is a good choice if the platform migration is more complex than software upgrade.
c. is a good choice if the software is more stable on the higher version and if any specific platform related issues are fixed in the higher version.

If the software application has multiple tiers with database, application server etc., one should carefully choose the overall path of migrate and upgrade. Making sure the impact on the business SLAs is minimized; It is also important to have a very good "back-out" plan, if something badly goes wrong!!

Performance Analysis & Tuning

2010-03-11T07:20:00.003Z

Performance of any system is defined as "accomplishment of a 'task' measured against agreed quality, cost and time."

The software tasks are primarily divided into "transactional" tasks and "analytical" tasks.

One big area in Software Systems management is the Performance Analysis and Tuning.

How to measure the performance?
The basic thing needed for "Performance" is the "Measurement" - What is to be measured and How?
Generally people measure the "response time" (time taken to complete the task) if the task is transactional
and
throughput (number of tasks performed in a fixed unit of time) if the task is analytical.

So, in both the cases what is measured is "time". What happened to the original definition that consisted the "quality" and "cost"?

In Software systems the "tasks" are automated and once the automation is tested well, "quality" is guaranteed for the task.
The "cost" of performing the automated task of a software application is the computing resources (CPU and memory) that are allocated to the application and how well the application can use additional resources to carryout the tasks.

Assuming the application can take full advantage of available CPU and Memory resources allocated to it, "TIME" becomes the best measure to analyze the performance of a software system.

How to Tune the system?
Reduce the amount of time taken to perform the 'task'. As simple as that.

Is it that simple?
Let us take a close look at this approach. Can we really reduce the time taken to perform a task?
Answer is "NO". If a task takes "x" amount of seconds to do, it takes "x" seconds to finish it.
But,
Every Software task has two components of this "time". What are they?
a. The time taken to actually DO the work
b. The time taken while waiting for a dependent sub-task to get completed.

So, Now if we ask the same question Can we reduce the time taken to perform a task?
The answer is YES - by reducing the time "waiting"!

Performance Tuning is all about scheduling and executing the sub-tasks of a "task" optimally such that the waiting time is minimized (if not completely removed) thereby using the available resources optimally and reducing the "cost" of performing the "task" That is the secret of performance tuning!

Platform As A Service - Private Cloud

2010-02-17T12:36:00.002Z

Sometime back, Oracle has published this white paper.

This gives a clear vision of where Oracle is heading in the technological direction of Cloud Computing.

By standardization on the technology stack for enterprise application and fusion middleware enhancements, the PaaS seems the natural direction for enterprises in medium term.

Link to my past blog post on Private Clouds Here.

SOA Governance standalone? or merge with IT Management?

2010-02-09T07:52:00.006Z

"Governance" in general and SOA Governance in particular has been a buzz word for sometime now.

My definition for Governance is making sure that a "thing" works as it is supposed to work. Replace the "thing" with SOA, IT, state, country etc.,

One of the emerging trends for past few years is "Business Transaction Management" for managing the composite IT applications configuration, performance and "Governance"

Traditionally the IT management tools and frameworks have looked at the IT as operations management, quality management, performance management, configuration management etc., disciplines.

In this trend the problem of IT management is from a business point of view. That is good. So, what is happenning in the technology?

1. Application Performance Management
2. Composite Application Management using run time discovery of relationships between the components (services) of the application
3. Business transaction management tools

Here is the question:

Will the new trend merge in traditional IT Management tools?
or
A standalone set of new SOA Governance tools emerge due to the new trend?

Based on my "UNIX" philosophy, I think an integrated set of tools that consist of traditional IT management with the new SOA Governance would be a best fit to tailor to the needs of contemporary corporates!

Let us wait and see....

Hybrid Columnar Compression (HCC)

2010-02-03T09:24:00.002Z

Oracle has introduced a new data storage technology in Exadata V2 called Hybrid Columnar Compression. This method uses a "compression unit" that is made of several rows; it is physically stored in compressed "column vectors" in multiple blocks of storage.

There are two flavors of columnar compression that are optimized for "QUERY" (warehouse compression better for executing queries) and "ARCHIVE" (best data storage for archive compression)

Each of the QUERY and ARCHIVE come with another modifier to set the "LOW" or "HIGH" compression mode.

This technical white paper gives an overview of the new technology.

This article on Oracle Magazine may also be useful in understanding this new feature.

More data can be stored in "less" space. Good!

sizing a data center management solution

2009-12-09T12:02:00.003Z

Introduction:

A typical data center management solution will have a "data store" for storing
a. configuration data
b. monitoring data

On regular intervals it will collect the data and upload it to the central data store.

A host, database, listener, application server instance etc., are the managed entities that will need monitoring and management.

Before implementing the solution, every organization will try to "size" the infrastructure requirements for the solution.

The sizing involves estimating the resource consumption for the solution.
1. disk storage
2. memory
3. cpu cycles required
4. network bandwidth requirements

The Problem:
The complexity involved in sizing a data center management solution is mainly due to lack of clarity of definition.
for e.g., customer want to monitor 10 databases on 10 servers. (one DB on each server)
Depending on the database version and weather it uses ASM for storage or it is on a cold failover cluster and several other considerations the number of managed entities will vary. A database can have only one tablespace and a single datafile or a database can have 10K tablespaces with 100K datafiles. If customer want to monitor the database space usage by tablespace a database with 10K tablespaces will produce 10K times more data when compared to a database with 1 tablespace.

Each metric (the monitored datapoint) can also be collected once in every 5 minutes or once in an hour. The collection frequency may vary based on customer requirements.

A simple formula is number of metrics ( number is at lowest granule) * number of collections per day gives the total number of metric values per managed entity. Multiplying this number with bytes required to store an average metric value gives the bytes required to store the metric values per managed entity.

If customer wants to keep this data at this granularity for a week, then the storage requirement is number of days of retention at raw granularity * bytes required to store one day data. This multiplied by the number of managed entities gives the total storage requirement for this type of managed entitiy (e.g., database)

The same exercise need to be repeated for all types of managed entities to get the storage requirement.

Collecting that many bytes over a day means transferring that data over the network from the host where the managed entity resides to the host where the management data store resides. That gives the network bandwidth requirement.

This data need to be rolled up to a hourly average and daily average for keeping it for historical trending. One need to calculate the space requirements and processing requirements in this rolling up process.

The old data need to be purged away from the data store. It needs processing cycles.

All the managed entities should also have a set of configuration data. That need to be collected, compared on a regular basis and kept up-to-date. One need to compute the resources required to collect, compare and store the original and changed values of different configurations as they evolve.

Managing all the "heartbeats" between the central management solution and the monitored hosts requires a proactive two way mechanism. This needs processing capacity.

Monitoring the websites with synthetic transactions by periodically playing a pre-recorded transaction to make sure the application works require additional set of processing, storage, memory etc., This need to be added to the above estimate.

A provisioning solution that is used to deploy new instances of databases, application servers will need the GOLD IMAGES to be stored somewhere. Patching the enterprise applications needs considerable amount of patch download space and prorogation on regular intervals. This also need to be considered.

The other thing to consider is about the periodic tasks (jobs) that get executed to perform routine operations. Each job may produce few KB of output that need to be collect and stored in the central store for a period of time.

The next thing to consider is the number of users that use the application, number of concurrent users and the amount of data they retrieve from the data store for performing their operations.

Collecting all this information, adding the block header, segment header etc., overheads in the phisical structure of the target solution is a complex task. By the time this exercise is almost complete, the tool vendor would have released the next version of the data center management tool with some modifications!!

The solution:
It is nearly impossible to "size" any application to the last byte accuracy for storage, memory, Network and cpu utilization.

No managed entity should generate more than 25MB of data in the monitoring store. No managed entity should have more then 100kb of configuration data.
So, taking a 250GB storage for 1000 monitored entities and configuring the solution in such a way that it will not exceed this requirement is a wise man's solution.

Considering a 1000 monitored entities will have a maximum 100 administrators and at most 30 of them concurrently loggedin, An average db oltp application with 250GB database and 24 * 4 (considering 4 txn's an hour i.e., 15 min collection frequency) * 1000 (no of entities) would require a 2 processor DB server and a 2 processor Application server with 4GB RAM each.

Starting with this configuration, implementing the solution using an iterative model is the best approach to balance between the up-front sizing vs not impacting the service level of this critical service.

Conclusion:

In the current world with virtualization and cloud technology it is easier to build scalable Grid like applications and scaling the solutions horizontally is the trend that is going to stay and further pick-up more momentum. The data center management solutions are not exempt from this trend. For every next 1000 monitored entities we will have to add a node to the DB grid and another node to the AppServer Grid along with additional storage to the storage grid.

IS IT NOT??

Improved Application Performance Management

2009-12-02T10:12:00.002Z

In April 2008 I have written this post : Application Performance Management

With
a. Oracle Enterprise Manager 10.2.0.5 Service Level Management,
b. REUI 6.0,
c. SOA mamagement pack and
d. Diagnostic Pack for middleware the solution is available.

A corporate can now implement a completely integrated APM solution based on the above four components of Oracle Enterprise Manager.

The new capabilities of REUI 6.0 include:
a. Full capture and replay of end user sessions.
b. Ability to link the slow performing JSP to the overall configuration in SOA management pack and then jump directly into the diagnostic tool (AD4J)
c. Customizable dashboards from the collected data.

What is REUI? "Real End User Experience Insight"! and What is 6.0? The new version released on 1st December 2009.

The ACTIVE MONITORING functionality is already provided by Enterprise Manager Service Level Management pack.

We hope these new tools will truly improve the Real End User Experience of the web applications.

Exadata V2 - World's First OLTP database machine

2009-10-20T10:22:00.003+01:00

Last year the Exadata Database machine was announced for making the data warehouse performance improvement by moving the database query processing to the storage layer.

This year, the new Exadata V2 is proven to be the fastest database machine for OLTP applications.

The trick lies in the Flash solid state storage and the 40GBPS infiniband internal network fabric used between the DB servers and the Storage servers. The full rack contains 8 database servers and 14 storage servers. There are other configurations available.

All the flash storage (600 GB per storage server) is viewed as the extended memory of the servers and it has 72GB of DDR3 memory as the cache per database server....

Guess what, an open challenge to IBM can be found at http://www.oracle.com/features/exadatachallenge.html

So, why not try out for a $10 Million prize?

Oracle Open World 2009 a mega event

2009-09-04T13:22:00.006+01:00

Three years back I was at Oracle Open World 2006 staffing one demo pod on "Deployment Best Practices for Enterprise Manager"

Again this year, back to this event where I am presenting a session and three hands on lab sessions to attendees.

Link to the Sessions

S309533 - Applications Management Best Practices with Oracle Enterprise Manager
Oracle Enterprise Manager Hands-on Lab: Oracle Application Testing Suite
Oracle Enterprise Manager Hands-on Lab: Oracle Data Masking Pack
Oracle Enterprise Manager: Oracle Real Application Testing

Event dates: October 11-15 2009
Catch me there!!

Oracle Database 11g R2 released

2009-09-02T06:51:00.002+01:00

Oracle has released the latest version of the worlds largest used database!

Key new features are:
"Shared Grid" with server pools for consolidating multiple applications into a single RAC cluster
"RAC One Node" for consolidating less critical application systems transparently get the benefits of easy movement etc.,
"ASM Cluster File System (ACFS)" in the storage management
"Instance Caging" for confining the database to specific cores in a SMP environments
"In-Memory Database Cache (IMDB Cache)" for moving the OLTP processing to the middle tier
capabilities to use the Disaster Recovery (Idle) infrastructure for Reposting and Backup operations.

It has several security enhancements as well.
This Oracle White paper has more details.

It is nice to hear about the new features from Tom Kyte. Being with oracle I had the opportunity to hear about the features directly from the "Architect" of those features!

Another Year....

2009-08-08T07:15:00.002+01:00

Another year into the time making it 20 + 1 = 21 years of association with the computers.

This year is mostly focussed on the data center software infrastructure provisioning and public and private clouds. Application quality management and Application performance Management solutions.

Operations Management vs Quality Management

2009-07-21T05:50:00.004+01:00

One of the challenges Information Technology faces eternally is to:

a. reduce the operational costs of a system/application in production
b. improve the quality of system/application in development

They are inter related. If any company manages one of these areas perfectly, the other area is redundant.

Let us assume company A has a perfect operations management. They can operationally manage any sort of product implemented, let the product has unlimited bugs, the operational management process is so strong that the business is never impacted due to the bugs in the IT due to its efficiency of operations management process and techniques. This is a nirvana of IT Asset operations management. All the systems are proactively monitored, the capacity problems are predicted upfront and fixed before hitting the limit; all the business KPAs are always met that too within the limited budget of operations management.

Let us assume company B has a perfect quality management. Right from the architecture up till implementation all the quality processes are perfected and the product goes live with absolutely no defects. Once implemented there is no need for any operational management. Absolutely no monitoring is required, no performance issue will be there and no bugs to fix.

The "Perfection" in one of the areas means the "Redundancy" of the other area.

Company A's method of perfecting the operations is a "short term solution" until all the applications are replaced by Company B's method of perfect quality.

The FACT is, none of the today's corporates have patience to perfect any one of them. They concentrate on "improving" both the areas slightly in cycles. This attitude leads to never ending cycles of replacing the applications and continuing to spend a lot of money in maintaining them.

When the management starts to reduce the operational costs without any improvement of QUALITY of new IT implementations, the quality of operations will go down and impact the business KPAs.
So, sure method of reducing the operational expenditure is to Perfect the quality of IT systems/applications. That is the only way....

Private Clouds Again

2009-06-17T07:49:00.003+01:00

As mentioned in the previous blog post on Private Clouds, the vendors are coming out with some products on this technology.

IBM launched the CloudBurst and HP BladeSystem Matrix.

All the three types of private clouds look promising at the current moment; IaaS, PaaS and SaaS. Service oriented, self-provisioning model of Cloud and the pay-as-you-go billing are key factors of success for Clouds.

Google Fusion Tables

2009-06-15T06:51:00.003+01:00

Q. Is Google experimenting with "Clouding" the "database"?

It looks to be.....

One can now upload upto 100MB table with a total limit of 250MB as of now to this new experiment.

It is surely a step towards taking the spreadsheets to the cloud. But will it pose a problem to BIG database vendors too in future?

Let us wait and watch.....

To know more about Fusion Tables go through FAQs and play around.

Next dimension in search - Google Squared

2009-06-04T10:16:00.003+01:00

Google has released a new search tool - Google Squared. This provides a structure to the unstructured data. It searches for all the related information and puts the data into a "Square" - a typical spreadsheet format. One can add rows and columns to it and save the Square.

It looks very good at the first look. All the best Google!!

http://www.google.com/squared

Private Clouds

2009-05-13T06:30:00.004+01:00

An interesting trend emerging is the concept of "Private Cloud" in the Data Center Management Area.

This trend eventually combines the strengths of
a. Standardization of overall infrastructure based on open standards
b. Virtualization of server resources
c. Seemless Provisioning of applications
d. Pay-as-you-go billing models and charge back of IT resources based on consumption
e. Services oriented architecture (SOA)

Their main advantages over the "PUBLIC clouds" are
1. less risk of security related issues
2. more control over the network bandwidth and availability

In my opinion, some big IT houses will adopt to "Private Clouds" to get the benefits of cloud computing while keeping the control in-house in the short term. As the technology advances and matures, it will also benefit the PUBLIC CLOUD providers!

Desktop Widgets

2009-04-23T07:51:00.002+01:00

"Desktop Widgets" is an emerging design pattern in delivering desktop applications with a Rich Interface to the Internet applications running on different client environments like Windows, Mac and Linux.

AIR - provides an runtime environment for running such applications.

How will it compare with the Browser? - As per Adobe, the comparison is here.

As per the AIR blog it claims it has passed 100 Million installations few months back!!!

Oracle Enterprise Manager 10g Release 5

2009-03-03T04:29:00.002Z

The latest release of Oracle Enterprise Manager 10g called Release 5 or OEM 10gR5 released... It is more stable, manageable with new features in overall enterprise management.

New Technology and its Manageability

2009-02-25T04:41:00.002Z

New Technology and its Manageability

However great the new technology is, it requires a manageability solution along with it. "Manageability" is the overall ease of the technology starting at its commissioning until its decommission.

Even with great manageability features, the Oracle RAC took some time to move into mainstream production. [Ref. Gartner Report on Oracle RAC claims it took nearly eight years]

In that case, what is the importance of "Manageability" of a "Enterprise Manageability Solution"???

DBA 2.0

2008-12-10T04:30:00.002Z

It is just over an year after the word "DBA 2.0" was coined in the 2007 open world. Just thought of writing my opinion.

DBAs are people, they use Processes & Tools. Upgrading the tools, processes and the application of the upgraded tools and processes to manage the next generation data center is truly the DBA 2.0.

In my opinion the DBA 1.0 has to continuously upgrade on the point versions and finally end up at the major upgrade DBA 2.0. A sudden upgrade to DBA 2.0 is just next to impossible.

An upgrade doesn't necessarily mean obsolescing good old processes or tools - it is only improving on the strong basics and fundamentals. So, a DBA 2.0 is still able to fall back on to the well known tool SQL*Plus and manage the database.

So, How many "DBA 2.0" are available on Market today?

Difference between EAI and ESB

2008-11-21T10:00:00.004Z

What really is ESB (SOA term)? What is its main difference with EAI (slightly older term)?

EAI is generally a HUB and SPOKE model of integrating the applications. When we remove the HUB from EAI and replace with a BUS (called ESB - Enterprise Service Bus) it becomes the SOA.

"Web Services" can be developed using multiple technological options can then be put on to this BUS and used by multiple consumers.

The SOA gives the marketplace to buy and sell services as commodity. The standards around the service description, service discovery and service invocation makes it all possible.

ESB is a concept, there are several software tools enabling the implementation of ESB are available in the market. They generally run on a middleware server as an application providing some core functionality around the service choreography and orchestration.

There are also BPEL engines which can do all the functionality of ESB and provide more complex work flow modeling of business processes.

It is an important decision to make between the ESB and BPEL processor manager while providing solutions.

There are instances a classical EAI tool would be sufficient to solve a specific problem where the BPEL based solutions are being implemented!!!!

Links:
Good Link found on the web on this subject: http://emergingtech.ittoolbox.com/documents/soa-vs-eai-vs-esb-12998

Parallel Data Processing in Grid

2008-11-11T01:27:00.002Z

Functional programming Languages, emphasizing the evaluation of mathematical processing without regarding the state change are on rise to do different analytic operations on the several Petabytes of data especially in search engine and web analytic functions.

The old "lisp" has slowly transformed into the new generation scripting "sawzall" incorporating massive parallel processing using map and reduce functions.

Sawzall is used at Google the famous search engine.

Exadata - Database Processing Moves into Storage

2008-09-25T06:08:00.004+01:00

For the data warehouses having Terabytes of data, the key problem is moving the data from the large storage arrays into servers performing the processing - This is termed as the "Data Bandwidth Problem"

Oracle and HP came together and unveiled the new technology called Exadata - The key difference here is the query processing has moved nearer to the storage technology in the "programmable storage server"

This technology should eventually improve OLTP performance too....

Technology Transformation and Hiding

2008-09-04T06:59:00.002+01:00

The latest trend seems to be "Hiding Technology" from business.
Various layers of virtualization, Service Oriented architecture and virtual appliances, automated transformation tools and testing tools are making this trend dominate the current technology market.

The custom business applications and ERP based applications turn more and more service oriented and reduce the dependency on the underlying technical infrastructure.

There will be more and more automation of testing and migration tools at the infrastructure layers let it be application server or the database technology. Does it mean are we going to have a virtual database and virtual application server?

The pay-as-you-go model of technology sourcing is already in. Along with the virtualization, renting model and fast transformation tools, each layer will hide the internal implementation and move towards a OPEN standard based model.

May be we are not very far from service providers providing, CPU capacity, MEMORY/Disk Storage, network bandwidth on a pay as you go model
also we should see further service providers putting these basic services together and providing technology like DB and application on the same model etc., as well as the top end business services like BI and analytics.

Let us wait and watch...

Quick Recap of 20 years 8/8/88 till 08/08/08

2008-08-08T07:10:00.006+01:00

My association with computer software is just turned 20 years. I have joined my graduate course in computer sciences on 8/8/88.

My first BASIC program
10 PRINT Hello Prasad
20 GO TO 10
went into an infinite loop.

An algorithm to interchange values of A & B without using a third temporary variable.
A = A + B
B = A - B
A = A - B
This proves a program can be highly optimized to use least possible memory or least of processor but can never achieve both at the same time. Optimize for bet processor utilization needs more memory usage and vice-versa.

Fascinated by the fact that the knowledge of direction of flow of current by the diode can make basic logic of AND, OR and NOT making a microprocessor consisting of highly sophisticated processing logic and studied digital electronics and microprocessor fundamentals with a great regard. Von Neumann Architecture of stored program computers, digital electronics and Boolean algebra were my favorite subjects.

In 1991 joined my Masters in Computer Sciences studying Algorithms, data structures, operating systems, assemblers. Developed a Intel 8085 two pass assembler using C language.

Interested in image processing, expert systems and operational research and optimization elected Operations Research and expert systems in my masters. Did a project using expert systems to provide training and advice for purchase process.

In 1993, joined Indian Space Research Organization and worked on a "Scheduling" problem of Low earth Orbiting satellites operations scheduling and successfully completed developing algorithm and implementing the algorithm in c language to produced optimized operations schedules at ISRO telemetry, tracking and command network.

In 1996, Moved out of scientific research and joined Tata Consultancy Services and provided Automated Branch Automation software implementation at various Banks in Bangalore. The main achievements during this period include installing a 1st ATM machine at UTI Bank Bangalore and implementing Voice response system at the same bank.

In 1997, traveled to Belgium on a complex design involving multi lingual HR system where an employee registered with one language, work for an employer speaking another language and the payroll data being processed by the processing company who would like to see the error messages in third language. Successfully implemented the solution using a CASE tool then called IEF which was later called Composer and cool:gen.

Traveled to Citibank Singapore, and Transco in UK on different assignments. Involved in implementation of Domestic Competition phase III at BG Transco for providing a supplier of choice to all 21Million households in UK.

In 1999, worked on providing integrated information systems to a TPML and BEML at Bangalore using varied technologies on UNIX and Windows platforms.

In 2000, back in UK at National Grid Transco involved in consolidating the UK-Link system architecture which was originally partitioned with 9 Oracle Databases. All the partitions consolidated to a single Oracle database. Key challenges included CoolGen and database migration to newer technology as well as getting the whole business process work as before after the technology change.

Worked on Reforms of Gas Metering arrangements and metering unbundling projects for Transco.

Technically lead a team involving multiple organizations and people from multiple nationalities and cultures to achieve one of the most complex spacial data migration project. The project involved migrating legacy vector data in 56 layers to new Oracle Spacial based ArcGIS system. A total of over 108 Million geographical features including over 28 Million Transco's gas pipeline network assets were migrated. This being a safety critical project, the work got internally and externally audited to make sure the accuracy of this migration.

In 2004, working as a Technical Architect for the Unbundled metering companies, TMS and OnStream provided a complete AS IS and TO BE architectures and a high level transformation strategy to the customer. Provided tactical improvements along with the strategic road map which has improved the SAP back end system performance by 300%

In 2005 after returning to India, lead the offshore team of TCS implementing important changes to metering business in meter asset management area and successfully delivered the changes to best customer satisfaction (98%)

In 2006, left Tata Consultancy Services and Joined Oracle Corp. in Enterprise Management product suite as a Technical Lead.

Worked with internal and external customers of Oracle to quickly adopt the data center management solutions realizing the benefits of a centralized management console and simplifying, standardizing and automating several monitoring and management activities.

Overall it has been a great experience, learning and working across various technologies and people from different cultural backgrounds and corporate business processes.

It is 7305 days since 8/8/88 and every day is a new learning experience for me....

Oracle Database - Real Application Testing

2008-08-05T05:40:00.003+01:00

When a pure technology upgrade is planned without any application change, or some performance improvements via patches or parameter changes are done to a critical production database it is very important to test the patch, upgrade or the parameter change.

This includes a test infrastructure setup with database, application and web tier components.
This is not only expensive but also tedious to build, validate and use it for testing.

The solution is RAT – Real Application Testing which can be used to capture the workload from the production database and replay on the test database.

Advantages:
1. No application or web tier infrastructure need to be setup
2. The Testing can entirely be done by DBAs
3. Much superior to testing the application with simulated load or with scripts

Workaround - Best Practices

2008-06-26T08:39:00.003+01:00

I find myself in a lot of situations to provide a "workaround"

Few quick properties of workarounds:

Workaround generally leaves the root cause not attended. A conscious effort required to get the development identify and fix the root cause of the problem.

Workaround generally forgotten in the documentation and would lead to an unknown problem at a future time. (In most cases another workaround nullifying the original workaround is suggested)

so,

Document the workaround, possible root cause and follow the "process" to get the original problem identified and fixed in the base product. also,

clearly communicate to the customer and make sure they mention the workaround when logging a new related problem (or bug) against the product in future.

Otherwise, customer will have "workarounds", the support team a "nightmare" and the product "a lot of bugs"

Of Memory Leaks and Core Dumps

2008-06-13T07:38:00.003+01:00

Even a very well tested product have these basic problems when finally put into the production.

This statement is not an exaggeration of truth. It is Truth. There are multiple software components and multiple platforms on which these components are made to run. For example an embedded JVM may process the logoff signal from a remote desktop connection on a Windows server.

The Java Runtime Bug cause the embedded application to crash and dump core on a Windows platform.

Errors like this are hard to debug and fix.

Memory leaks are another complex thing to debug and fix when it happens on a Live/Production environment. It is best to identify the leaks during the development/testing phases. Example of such errors are the ORA-04030 errors showing the out of process memory on an oracle database running a production application.
When applications use the connection pooling and hold the connection for a long duration and application carries out complex and divergent transactions through these connections it gets quite challenging to identify and fix the problems.

Bottom line: The rigor of performance and scalability testing is very crucial in a product development life cycle.

Avoid Gobbledygook

2008-05-22T10:09:00.003+01:00

Especially when reading the proposals and technical documents I came across a lot of Gobbledygook.

Some examples from the http://www.unfg.org/

"The relevant modules of the project are, at the inter-governmental level, mapped onto local non-donor-driven poverty reduction and sustainable growth strategies."

"In the context of improving the legal and regulatory framework, the Member States will develop inter-agency-coordinated administrative targeted policy implementation aimed at mitigating negative effects of globalization."

It is best to make things simple and written in plain English. The plain english campain website has more info: http://www.plainenglish.co.uk/

...

Managing Data Center With Multiple Networks

2008-05-08T08:31:00.006+01:00

In the world of mergers/acquisitions and consolidation, large application hosting companies are being formed by the merger of several small hosting providers. This leads to having a diverse networks connected & separated by multiple firewalls and DMZs.

The challenge here is to manage and standardize the whole of the data center into one consolidated view.

Possible Monitoring/management Solution Options:

1. Centralize Monitoring Solution which can see all the Assets using necessary policies on firewalls/proxy etc.,
==>Complex network management and simple administration of the management solution.
==>Additional Load on the WAN

2. Distributed Management solution on each network with a data consolidation to a central management repository
==>Management solution need to provide a data synchronization option and possible "time lag" between the view of local and central consoles.
==>Maintainability of the solution is complex but involve minimum changes on Network /Security policies.

An Ideal Data Center Management Solution product should provide ability to configure both the above options.

Service Oriented Analysis and Design

2008-04-14T11:11:00.004+01:00

Currently the Business Process Modeling (BPM), Enterprise Architecture (EA) and the object oriented analysis and design (OOAD) being put together to make the overall life cycle of software architecture development into a service oriented analysis and design (SOAD)

The focus is to analyze the business from a service oriented perspective and get the business aligned, loosely coupled, coarse grained services identified and documented.

The complete SOA life cycle involves
Service Identification - Analysis
Service Specification - Design
Service Realization - Development & Testing
Service Operation - Implementation and Maintenance

The concepts come from Object Orientation but quickly getting adopted to "Service Orientation"

The life cycle is supported by the Service Governance and Service Security and other manageability aspects as the supporting "technical" SOA framework.

Application Performance Management

2008-04-01T10:55:00.003+01:00

End user experience monitoring and management is becoming a key for the websites. As the "Software as Service" increases it is more important to mange the services in an end to end perspective. End to End performance monitoring and management have two primary methods.

Active or synthetic monitoring:
This will need a simulated user action recorded and played back from different geographies. One can install multiple hosts across the globe and run the simulated tests periodically to monitor and identify possible availability and performance problems with a web application.

This method can be used to identify common problems with the availability and detect any degradation of performance actively.

Disadvantage is one can only monitor the "predefined access paths" through the application. Also need to have the simulated "agent"s across the different geographies and modify the access paths as needed when the application changes.

Real end-user monitoring or passive monitoring:
This method involves mining the access logs of the application and analyzing the data to identify the performance problems. It can also be done using the "network sniffing" method. Whatever is the method used to get the real end-user access data to the application, this data is very valuable for business to analyze both the performance and the general usage patterns on the application.

As our focus is performance monitoring, the disadvantage is we will get to know the issue after it impacts the users.

There are several products available in the market to do both the active and passive application performance management and one should develop a correct strategy with a right combination of both methods to archive the required service levels with right balance.

A holistic strategy should also include diagnostic tools within the solution.

Thinking Tools

2008-03-03T09:50:00.003Z

Thinking Tools? Yes. There are a few tools which can be used in organized thinking. I came across a many but trained and used the following few:

CEDAC: Cause and Effect Diagrams with addition of Cards this is a modification to the Fishbone diagram (a standard QC tool) by Dr Ryuji Fukuda. It is a very effective tool to get the ideas from all layers of the organization quickly and a wonderful aid to the divergent and convergent thinking technique. Has been very effective if used along with the other Fukuda techniques. (Ref. Building Organizational Fitness by the same author)

Six Thinking Hats: Edward de Bono's six thinking hats technique is very effective when conducting meetings involving people with varied interest. White, red, yellow, black, green and blue hats represent various aspects of a problem and they can be used in a suitable sequence within a meeting for pre-determined durations to channelise the thoughts of all the participants to the specific aspect of the discussion and finally summarizing the meeting with a blue hat.

Mind Maps: Tony Buzan's mind mapping technique uses colours, shapes, keywords in a multi channel thinking technique which will enhance memory and also bring a natural hierarchical thought process towards any problem.

Applying any one or more of the above techniques to the appropriate situations will surely yield better results than just going with the gut feel.

Data Security and Related Technologies

2008-02-13T10:22:00.003Z

This post is my review of the technology available in the area of "Data Security" - Security in used in a context of access control and threat protection.

All the information technology is about making the right data available to the right user and the security means it should not be available to a wrong user. This is especially true for the sensitive and personal data stored on the IT assets.

The focus is on Oracle Technologies.

Virtual Private Database & Oracle Label security
Used to restrict regional/role-based access to the data using the data access policies or labels. A user with right role in right region can access their own data. They will not be able to see any other data, which they are not authorized to see.

Transparent Data Encryption – Disk level encryption
Used for Disk level encryption of data. Data is not stored in plain text on the disk, Prevents data visibility to some low level disk reading type attacks

DBMS_CRYPTO – Application Encryption
Application managed encryption. Only the application knows how to use the data.

Database Vault
Role separation and Insider threat.
Good for IT/DBA outsourcing. Protection from privileged users accessing the sensitive application data.

Data Masking or Scrambling
The internal development/test by masking sensitive data or provide the data to service providers for billing or telemarketing etc by scrambling sensitive data.

Satellite Scheduling & Genetic Algorithms

2008-01-23T15:03:00.000Z

During the beginning of my career, I have spent approximately two years on a complex scheduling problem.

This link is a paper related to the original work: http://track.sfo.jaxa.jp/spaceops98/paper98/track2/2b002.pdf

Later, the same problem was analyzed and improved to use Genetic Algorithms by IIT professors. This link for the Genetic paper on the same subject: http://www.aiaa.org/Spaceops2004Archive/downloads/papers/SPACE2004sp-template00515F.pdf

After studying both the papers, I feel the original work has indeed used a "Genetic Algorithm"
!!

Data, Information & Knowledge

2008-01-22T06:19:00.000Z

Data - a precise fact mostly numerical, used to be processed to carryout some calculations like the interest on a deposit or a loan, or the wages of an employee in an automated way during the beginning of the computer age. EDP departments of large businesses used to buy the computers and use them in the data processing.

Information and analysis of the underlying data for improving the business decisions is the advancement due to the availability of enormous amounts of past data collected by the data processing machines. The processed data further analyzed, aggregated, segmented and used to aid the business decisions. Decision support systems and OLAP have emerged out of this need.

Knowledge is to provide some latent patterns and inference from the analytical information. To relate two or more seemingly unrelated facts to arrive at a relation is known as knowledge. The computing industry now working on this type of business intelligence using several pattern recognition and mining techniques to take advantage over the competition.

Most of the processing capacity is used to outsmart the competition. The real business intelligence is not in computing capacity; it is intuition. Artificial can never be intelligence. Intelligence can never be Artificial.

The question is "Can New Knowledge come out of Computers?"

Cloud Computing & Virtual Software Appliances

2008-01-09T06:37:00.000Z

Virtualization and Web Services merged together to form the new form of cloud computing.

Virtual hosts offered as service using a fully automated monitoring, management and provisioning are called as Cloud.

There is no publicly available service as of date, but Amazon Web services is coming up with the Elastic Compute Cloud (EC2) in this model.

Once stabilized the clouds with virtual software appliances should remove all the hassles of deployment & provisioning. - Hope the best!!

Infrastructure Lifecycle Management

2007-12-06T15:42:00.000Z

The end to end IT infrastructure life cycle management includes the following:

1. Strategy
Covering the Portfolio & Financial Management
2. Design
Availability/Business Continuity/Capacity/Security Design
3. Transition
Change/Release/Configuration Management including provisioning and patching of technology stack
4. Operation
Incident/Problem Management involving the service monitoring and proactive management
5. Continual Improvement
Service Level Management and Improvement and Reporting

As per latest ITIL V3

Testability - A key feature

2007-11-28T10:44:00.000Z

When we define quality, there are several attributes of quality are talked about - Usability, functionality, performance, maintainability, interoperability, flexibility etc., several desired abilities of the given product. In a software product the testability is also an important feature.

The "Testability" is equally important measure for a software product as well as for the testing methodology.

When the product specification was drawn up, the testability requirements need to be given sufficient importance during the analysis & design phases. The early the testing requirements are drawn the better measure of quality can be done and finally a better product can be delivered.

There will be several phases of testing unit, system, integration, scalability, performance, load, disaster recovery etc., with different environments etc., during a software development project.
If the overall effort is not co-ordinated well the final product may not completely satisfy the customer!

Business - IT alignment

2007-11-22T15:25:00.001Z

Business of an enterprise will normally need certain capabilities from their Information Technology resources. Different users at different levels of the enterprise would need access to right information at the right time. The customers and suppliers should be able to get the access to the right information as and when required. The information should accurately captured, stored and presented to the users as necessary.

In this process, IT resources are built in form of hardware - storage and processors, Network infrastructure. Software - OS, Databases, OLTP apps, BI tools.

Historically all this IT resources were procured, configured and maintained by the enterprises as a centralized department, then they started outsourcing certain functions. of late there are several pay per use solutions, hosting solutions and software as service solutions are coming up.

Coming to the topic of Business - IT alignment: At what level the alignment should start? CEO, COO, CIO/CTO should have this alignment done on a regular basis cascading the impact down to enterprise and closing the loop by getting the feedback from all levels.

It is a process (continuous) rather than a project (one time activity)!!!

Virtualization and Grid Computing

2007-11-07T05:16:00.001Z

1. Virtualization: Using hardware resources across virtual hosts making the hardware utilization better across a data center. Also provide quick turnaround times to build new hosts for the development and test infrastructures.
2. Grid Computing: Use low cost hardware resources to accomplish production tasks with high redundancy and performance.

"1" is actually partitioning a physical host into multiple virtual hosts and use the physical resources more effectively where as "2" is to combine multiple smaller physical hosts behave as if it were a big single host.

These different solutions are best utilized to solve different problems. All these multiple technology options when used in a correct manner surely reduce the cost and improve the utilization of the data center resources.

Best of Breed

2007-10-26T17:27:00.000+01:00

In today's marketplace there are several specialized products catering for specific business function. As each product is trying to extend into other areas, there is a lot of overlap of functionality.
Architectural decisions are
1. Build - Tailor made solution for the business problem
2. Buy - Buy one product - Implement, configure and customize the product for the business needs
3. Buy best of breed products and integrate to provide solution for the business problem

There is a little catch in the option 3. Some times, the technology stacks of these best of the breed products end up incompatible and cause major issues to business after integration. The problem resolution takes generally longer than single product solution.

Services and BPEL

2007-10-19T16:14:00.000+01:00

Of late there is a lot happening on the service oriented architectures (SOA)

Approximately two years back, when the UK gas distribution networks have been sold to different private owners by the regulated business there was a need to "integrate" the business processes uniformly for the metering business work management applications where I first got into the this.

Is this new? when the c language programs were written a function used to provide a specific "service" to the calling function. It was call by value or call by reference and use of pointers that time. Later in multi processing systems the interprocess communication (ipc) was used to coordinate between the processes running on the same computer. Further to that the remote procedure call (rpc) came up. Multiple standards have come up from the rpc technology and with the advent of xml it finally transformed as web services.

SOA is not actually a software thing it is more of an architectural pattern. designing the business process as a loosely coupled services and orchestrating the web services using the business process managers is synonymously called as SOA.

The beauty of SOA's success is in the appeal of legacy wrapping. No need of re-writing the legacy application. One can still use the core COBOL back-end program with a wrapped adapter in a web service orchestrated SOA environment.

Care must be taken while designing and defining services and the work flow. Properly designed business processes with the aid of BPEL looks very promising for the future.

Quality Measurement

2007-10-10T16:25:00.000+01:00

Quality is very important for the success of any software project. This is an accepted fact The issue here is how to measure the quality.

Time, Cost and Quality is the well known triple constraint. How to workout this is taught and practiced by several project management methodologies like PMP and Prince2 etc.,

It is a general feeling that Quality is all about Measurement, defining the measure, collecting the data, plotting the data against an upper and lower bound and analyzing the results when there are cases of data falling outside the tolerance limits.

What is wrong with this? Nothing. But the key lies in defining the measure. There are several factors of the quality :- Functional Fit, Usability, Performance, Security, Maintainability, Testability, Flexibility, interoperability, and so on the list goes on unending with several abilities.

Most of the times, Quality measurement goes with a standard template developed by some quality champion with in the organization. There will be a lot of thumb rules applied. So, projects mechanically collect the data and fill in the spreadsheets required by the Quality people.
No one really cares about the figures as long as the customer is happy. When customer is unhappy there will be an additional audit done and few NCRs (non conformance reports) logged against the project. The show continues.

Quality Measurement should be done religiously to improve the quality of the project - Not just to fulfill the requirements of the Quality department and filling up spreadsheets just for the sake of it.

It starts right at the planning stage when identifying work items and the quality goals. Be realistic when doing this part of the planning to achieve the best results for both the service provider and the customer.

Patterns

2007-10-01T11:13:00.000+01:00

Patterns - well known problem - solution pairs. They are good to solve the common problems - only if the problem exists.

The designers tend to get carried away by the patterns and apply different patterns to the general templates only to complicate the design. Sometimes over applying the pattern results in performance and maintainability issues to the solution rather than solving the problem.

So, I know the common solution so I will create the common problem.

Even in solutioning a consultant tend to have certain patterns and provides solutions using those patterns. This should be only applied after careful thought process, considering all other parameters of solution being same.

Each business situation will have its own Uniqueness - So, be careful when applying the past knowledge (pattern) while providing solutions.

Flexibility & Performance

2007-09-28T11:03:00.000+01:00

There are known pairs of opposites called architectural considerations. If there is all the money and time in the world given to a project, then the final result of the project (= product ) will not have any bugs, will be highly flexible and highly performant.

But, the reality is - "there are only limited resources - time and cost" available for building the solution. The "QUALITY" is dependent on the available resources.

It is very important to understand which of the contrasting qualities are important for the product. If the solution should be highly flexible it may not be performing at the highest level. To add flexibility one should introduce additional processing.

A real solutioning process is to strike the right balance between the various demands of the expected solution.

Technology Driven Change

2007-09-25T13:34:00.000+01:00

Sometimes, customer want to get on to the bandwagon too quickly and want to use a new technology with a huge percieved benifit. There is no real need from a business for a change, the project involves creating a completely new technology stack involving comparitively new technology. (which may not be fully mature and undergoing a lot of instability)

It is always good to experimant with a small proof of concept, understand the impacts of new technology after thouroughly evaluating the benifits. The justification and the business case should include tangible and measurable benifts and there should be full involvement from the business in such cases.

If a programme gets approved without complete buy in from the business it suffers from lack of commitment from the business and eventuall it fails to deliver the percieved benifits.

Ideally any change should be driven by the Business -> technology. What about the new technology completely changes a specific business? That is where the evaluation of new technology carefully fronted by the technologists and business specialists called "STRATEGY" team comes for rescue. Still the change is driven by business not by the technology.

Thumb rule ----- Business Change (whatever the reason including the technology advancement ) should lead Technology Change

The Unix command I like most

2007-09-21T11:01:00.000+01:00

The UNIX shell command "who am i" is what I like most. When I type it on a shell prompt I get something like

PrasadChitta pts/57 Jul 23 21:36 (hostname.domainname.com)

Why is this important on a blog named techno functional consulting? Some times when on a consulting assignment one should remember this command.

A consultant should help achieve the goal of the project without any bias. Sometimes the consultants cause more confusion especially during the technology selection, vendor selection phases. There is no fixed rule that only one vendor or technology is the best suitable for a given problem ---

All the business problems are more are less similar - to over simplify - "read, insert, update or delete data from/to the storage".

There are a lot of environmental, operational, security, performance, cost, ... type of non functional considerations which will drive the selection process along with the functional requirements of a given situation. It is really difficult to get an "independent consultant".

That is where I use my favorite UNIX command "who am i" to help me play the correct role as an independent consultant when on an assignment.

Other than my consulting career, I search for the answer for the same question spiritually - that is another reason for the liking of this command.

Simple looking Complex problem

2007-09-18T08:07:00.002+01:00

Approximately a decade back, after spending nearly two years on an optimization problem, I have ended up with the following problem. This is a variation of combination algorithm.

Assume that there are 'm' groups with each group will have 'nelem(i)' elements where i = 1 to m and nelem(i) > 0 (zero)

The problem is to find out all the possible combinations picking exactly one element from each group. [each element in a group is distinct and imagine it is numbered 1, 2, ... nelem(i)]

It looks very simple in the first look but takes time to solve this problem. If you are a programmer try to solve it for yourself. Just try to write an algorithm for this and you will realize the complexity of this problem.

So, there are certain things look very simple when you first see them. Only when coming to implement the solution the real complexity will appear.

Example output 1:
Number of Groups : 4
Number of Elements in Group 1 : 3
Number of Elements in Group 2 : 2
Number of Elements in Group 3 : 1
Number of Elements in Group 4 : 2
G1:1 G2:1 G3:1 G4:1
G1:1 G2:1 G3:1 G4:2
G1:1 G2:2 G3:1 G4:1
G1:1 G2:2 G3:1 G4:2
G1:2 G2:1 G3:1 G4:1
G1:2 G2:1 G3:1 G4:2
G1:2 G2:2 G3:1 G4:1
G1:2 G2:2 G3:1 G4:2
G1:3 G2:1 G3:1 G4:1
G1:3 G2:1 G3:1 G4:2
G1:3 G2:2 G3:1 G4:1
G1:3 G2:2 G3:1 G4:2

Example Output 2:
Number of Groups : 3
Number of Elements in Group 1 : 5
Number of Elements in Group 2 : 3
Number of Elements in Group 3 : 2
G1:1 G2:1 G3:1
G1:1 G2:1 G3:2
G1:1 G2:2 G3:1
G1:1 G2:2 G3:2
G1:1 G2:3 G3:1
G1:1 G2:3 G3:2
G1:2 G2:1 G3:1
G1:2 G2:1 G3:2
G1:2 G2:2 G3:1
G1:2 G2:2 G3:2
G1:2 G2:3 G3:1
G1:2 G2:3 G3:2
G1:3 G2:1 G3:1
G1:3 G2:1 G3:2
G1:3 G2:2 G3:1
G1:3 G2:2 G3:2
G1:3 G2:3 G3:1
G1:3 G2:3 G3:2
G1:4 G2:1 G3:1
G1:4 G2:1 G3:2
G1:4 G2:2 G3:1
G1:4 G2:2 G3:2
G1:4 G2:3 G3:1
G1:4 G2:3 G3:2
G1:5 G2:1 G3:1
G1:5 G2:1 G3:2
G1:5 G2:2 G3:1
G1:5 G2:2 G3:2
G1:5 G2:3 G3:1
G1:5 G2:3 G3:2

I have posed this problem to several programming experts. Still I did not get a better solution yet!

Simple vs Complex

2007-09-14T10:25:00.000+01:00

"There is no simple and correct solution to a complex problem." - but there are always multiple simple and wrong solutions to a complex problem.

Multiple simple (wrong) solutions make up a complex mess over a period of time and actually make the final solution more complicated to work with. Eventually it will end up in multiple technologies, data duplication, lot of backed scripts and unknown programs running to accomplish the task to some extent. Manual work - not to say several spreadsheets and personal databases on the end user computers performing a business critical report generation for company's CEO after getting the data from a high end data warehouse using a best possible OLAP tools.

So, giving a through thought about the solution architecture with a strategic perspective is very essential to a consultant.

Tactical Trap

2007-09-13T11:11:00.000+01:00

People fall into this trap generally after the confusion. Result is a Spaghetti. What do I mean by the tactical spaghetti architecture created by so called "tactical consultants"?

Tactical = Quick and Dirty

For a confusing change people will quickly find a tactical solution making the overall information systems more and more confusing.

Reasons for the tactical solutions are many, but they can be grouped into

a. time and cost constraints and

b. lack of strategic view point

Eventually these tactical solutions will run years in an organization and become critical applications without proper documentation.

Confusion and Consulting

2007-09-13T08:29:00.000+01:00

Consulting generally starts with confusion. Some call it as
gap between the requirement and what is available, some call it as a scalability issue, technology advancement, sometimes business change, merger or de-merger, or even sometimes as regulatory requirement or as auditing or safety measure or statutory change ---> In a nutshell CHANGE.

Change causes confusion in choosing the future course of action.

Why this blog?

2007-09-12T16:41:00.000+01:00

According to me CONSULTING is one art and science. There are no sub-divisions in it. Today in the IT world there are technical consultants, functional consultants, business consultants, architectural consultants and this list goes on and on....

After looking at the Information Technology in research, banking, insurance, Public utility transportation, distribution, metering, manufacturing and government sectors closely being a part of the consulting community, I want to share some of my thoughts on technology areas like c, Unix, CASE tools, Mainframe, Thick Client, Thin Client, databases, file organisations, expert systems, business intelligence, scheduling and optimization, GIS, Asset Management, Work management, ERP and Back office, CRM and front office, Field Force automation, Open Systems, Scripting, Automation, EAI, Standards, Frameworks, Compliance, 6sigma, Enterprise Architecture transformation programmes, business change management, programme management, ITIL, Systems management, benchmarking, configuration management, business intelligence, OLTP and OLAP, data mining, warehousing, ETL, OOAD, UML and Modelling, XML, SOAP, Web services and few more...