Data and Analytics - some thoughts on consulting

On this technical blog, I have not been very regular now-a-days primarily due to the other writing engagements on artha SAstra and on Medium.

Yesterday, I have been asked to address a group of analytic enthusiasts in an interactive session arranged by NMIMS at Bangalore on a theme "Visualize to Stratagise". Having spent around three hours several topics on Analytics and specifically on Visual Analytics were discussed.

I thought of writing on two aspects of Analytics which I have seen in past few months on this post to give a little food for thought to those who are consulting on Analytics.

Let "data" speak.
Few weeks back one of my customers had a complaint on database. Customer said, we have allocated a large amount of storage to the database and in one month time all the space was consumed. As per the customer's IT department at the maximum of 30K business transactions were only performed by a user group of 50 on the application which is supported by this database. So, they have concluded there is something wrong on the database and hence an escalation to me to look into it.

I have suspected some interfacing schema that could be storing the CLOB/BLOB type data and there could be missing cleanup and asked my DBA to give me a tablespace growth trend. The growth is on the transaction schema and across multiple transaction tables in that schema. I have ruled out some abnormal allocation on a single object with this observation.

We thought of running a simple analytics on the transaction data to see the created user on those transactions to verify if someone has run any migration script that could have got a huge amount of data into transaction tables or some other human error.

For our surprise we have seen 1100 active users who have created 600,000+ transactions in the database. All through the different times and most regular working day, working hour pattern. No nightly batch or migration user created the data. We went ahead with a detailed analytics on the data which has mapped all the users across geography of the country of operation.

We created a simple drill down visualization of the data and submitted to business and IT groups at the customer with a conclusion that the data indeed valid and created by their users and there is no problem with the system.

So, the data spoke for itself and the customer's business team said to the IT team that they have started using the system across the country for the last month and all the users were updating transactions on this system. This fact the IT team was not aware of. IT team is still thinking it is running pilot mode with one location and 50 users.

Let the data speak. Let it show itself to those who need it for decision making.Democratize the data.

The second point which came up evidently yesterday was

"If you torture your data long enough, it will confess anything"
Do not try to prove the known hypothesis with the help of data. It is not the purpose of analytics. With data and statistics you can possibly infer anything. Any bias towards a specific result will defeat the purpose of analytics.

So, let the data with its modern visualization ability be an unbiased representative which shows the recorded history of the business with all its deficiencies, with all its recording errors and all possible quality problems; in the process of decision making and strategising..... 

Hope I made clear my two points while consulting on Analytics....

F1 Database from Google: A scalable distributed SQL database

 

This world is a sphere. We keep going round and round. After a great hype around the NoSQL highly distributed databases, now Google presented a paper on how they have implemented a SQL based highly scalable database for supporting their "AdWords" business in 39th VLDB conference.

---

The news item: http://www.theregister.co.uk/2013/08/30/google_f1_deepdive/
and the paper: http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/41344.pdf

---

The key changes I liked:
1. Heirarchically clustered physical schema model: I always thought heighrarchical model is more suited for real life application than a pure relational model. This implementations is proving it.

2. Protocol Buffers: Columns allowing structured types. It saves a lot of ORM style conversions when moving data from a storage to in-memory and vice versa.

---

A quote from the paper's conclusion:

In recent years, conventional wisdom in the engineering community has been that if you need a highly scalable, high-throughput data store, the only viable option is to use a NoSQL key/value store, and to work around the lack of ACID transactional guarantees and the lack of conveniences like secondary indexes, SQL, and so on. When we sought a replacement for Google's MySQL data store for the Ad-Words product, that option was simply not feasible: the complexity of dealing with a non-ACID data store in every part of our business logic would be too great, and there was simply no way our business could function without SQL queries.

So, ACID is needed and SQL is essential for running the businesses! Have a nice weekend!!

All (that I know) about Exadata

Having associated with Exadata in one or other way from its release ( see: http://technofunctionalconsulting.blogspot.in/2008/09/exadata-database-processing-moves-into.html ) I have tried consolidating key points related to Exadata for a session to technical audience.


Related Posts: 

http://technofunctionalconsulting.blogspot.in/2009/10/exadata-v2-worlds-first-oltp-database.html

http://technofunctionalconsulting.blogspot.in/2010/02/hybrid-columnar-compression-hcc.html

http://technofunctionalconsulting.blogspot.in/2012/06/exadata-performance-features.html

Accelerating Analytics using “Blink” aka “BLU acceleration”

This Friday marks completion of my 2 years in the second innings with TCS ‘s Technology Excellence Group and it is time for a technical blog post.

During this week, I have seen IBM announcing new “BLU acceleration” enabled DB2 10.5 that claims a 10 to 20 times performance improvement out of box.  (Ref: http://ibmdatamag.com/2013/04/super-analytics-super-easy/ )

This post aims at giving a brief summary of the Blink Project which has brought in this acceleration to the analytic queries.

The Blink technology has primarily two components that achieve the said acceleration to the analytic processing:

1.       The compression at the load time

2.       The query processing

Compression & Storage:

At load time each column is compressed using a “Frequency Partitioning” order preserving fixed length dictionary encoding method. Each partition of the column has a dictionary of its own making it to use shorter column codes. As it preserves order the comparison operators/predicates can be applied directly to the encoded values without needing to uncompress them.

Rows of are packed using the bit aligned columns to a byte aligned banks of 8, 16, 32 or 64bits for efficient ALU operations. This bank-major storage is combined to form blocks that are then loaded into the memory (or storage.) This bank-major storage exploits SIMD (Single Instruction, Multiple Data) capability of modern POWER processor chips of IBM.

Query Processing:

In Blink there are no indexes, no materialized views nor a run-time query optimizer. So, it is simple. But the query must be compiled to take care of different encoded column lengths of each horizontal partition of the data.

Each SQL is split into a series of single-table queries (STQs) which does scans with filtering. All the joins are hash joins. These scans happen in an outside-in fashion on a typical snowflake schema creating intermediate hybrid STQs.

Blink executes these STQs in multiple blocks to threads each running on a processor core. As most modern ALUs can operate on 128bit registers all the operations are bit operations exploiting SIMD which makes the processing fast.

For more technical details of Blink project refer to - http://sites.computer.org/debull/A12mar/blink.pdf

Hope this will bring “Analytics” a boost and some competition to Oracle’s Exa- appliances. Views, Comments?

Small & Big Data processing philosophies

In this first post of 2013, I would like to cover some fundamental philosophical aspects of “data” & “processing”.

As the buzz around “Big Data” going on high, I have classified the original structured, relational data as “small data” even though some very large databases I have seen having 100+ Terabytes of data with an IO volume of 150+ Terabytes per day.  

Present day data processing predominantly uses Von-Neumann architecture of computing in which “Data” and its “processing” are distinct and separated into “memory” and “processor” connected by a “bus”.  Any data that need to be processed will be moved into processor using the bus and then the required arithmetic or logical operation happens on it producing the “result” of the operation. Then the result will be moved to “memory/storage” for further reference.  Also, the list of operations to be performed (the processing logic or program) is stored in the “memory” as well. One needs to move the next instruction to be carried out into the processor from memory using the bus.

So in essence both the data and the operation that needs performing will be in memory which can’t process data and the facility that can process data is always dependent on the memory in the Von-Neumann architecture.

Traditionally, the “data” has been moved into a place where the processing logic is deployed as the amount of data is small when compared to the amount of processing needed is relatively large involving the complex logic. In the RDBMS engines like Oracle read the blocks of storage into the SGA buffer cache of running database instance for processing. The transactions were modifying small amounts of data at any given time.

Over a period of time “analytical processing” that required to bring huge amounts of data from storage into processing node which created a bottleneck on the network pipe. Add to that there is a large growth in the semi-structured and unstructured data that started flowing which needed a different philosophy towards data processing.

There comes the HDFS and map-reduce framework of Hadoop which took the processing to the data. During the same time comes Oracle Exadata which took the database processing to storage layer with a feature called “query offloading”

In the new paradigm, the snippets of processing logic are being taken to a cluster of connected nodes where the data mapped with a hashing algorithm resides and results of processing then reduced to finally produce result sets. It is now becoming economical to take the processing to data as the amount of data is relatively large and the required processing is fairly simple tasks of matching, aggregating, indexing etc.,

So, we now have choice of taking small amounts of data to complex processing with structured RDBMS engines with shared-everything architecture of traditional model as well as taking processing to data in the shared-nothing big data architectures. It purely depends on the type of “data processing” problem in hand and neither traditional RDBMS technologies will be replaced by new big data architectures, nor could the new big-data problems be solved by traditional RDBMS technologies. They go hand-in-hand complementing each other while adding value to the business when properly implemented.

The non-Von-Neumann architectures still need better attention by the technologists which will probably hold the key to the way human brain processes and deals with the information seamlessly either it is structured or non-structured streams of voice, video etc., with ease. 

Any non-Von-Neumann architecture enthusiasts over here?

Oracle database 12c

I have been seeing Oracle database form 6, 7, 8i, 9i, 10g, 11g and now it is going to be called 12c - c as in Cloud. OOW12 revealed the new architecture. (there was no tail for 6 and 7.... i stood for internet, g for Grid..... )

The new release with a fundamental architectural change to the data dictionary that gives a concept of "Pluggable Database" - PDB over the standard oracle system "Container Database" - CDB.

The obj$ of Oracle database dictionary will contain all the information of objects in the current architecture.  It is being split into CDB and PDB going forward for easing the "multi-tenant" private cloud databases.

This fundamental seperation of the data dictionary provides
a. Better security of multiple "databases" on the same instance
b. Seperation of CDB from other PDBs will allow easy upgrades.
c. All PDBs will share the instance and overall management of the consolidated database should be much simpler.

Excited about the cloud enabled, multi-tenant, pluggable database from Oracle! 

So, let us wait and see when the stable 12.2 will come out to roll out the new database into production....

Data Replication

The need for data replication is evergreen. The use-case may vary from "High Availability" to "Operational Reporting (BI)" to "Real-time Analytics"; whatever may be the case, the need for replicating the data exists in the information systems and solutions.

I try to summarize the evolution of data replication technology from a Oracle Database standpoint in this post.

1. Looking back at the Oracle based data replication technologies the first one is "Database Link". One can create a database link and pull or push the data directly into a remote database starting Oracle 5 or 6. This is the very first method of replication where the application need to push or pull the data from a remote database and apply necessary logic to identify what data has changed and what to do with those changes.....
2. The next improvement in the replication is around Oracle 8 - one can setup a trigger based replication. That means whenever a transaction changes the data in a table one can trigger a function that can handle the replication without changing the application. So, database started giving a method to replicate data using triggers....
3. Then the next improvement has come around 9.2 with streams and log-based replication. The capability is to mine the redo logs and move the committed transactions to the target systems. (DBMS_CDC_PUBLISH and DBMS_CDC_SUBSCRIBE packages were introduced)
4. Oracle Advanced Queuing enhanced the streams to have robust publish and subscribe model replication that has enqueue and dequeue based communication. I was involved in a very large project that set up a custom Changed Data Capture to migrate data from a legacy system into SAP ERP involving large tables having 100Milion records.... 
5. Later the log-mining technology was used for physical and logical standby dataabses and evolved to a Active Data Guard technology......
6. With GoldenGate, the heterogeneous log-based data replication solution is complete from Oracle that has capabilities to extract, replicate, synchronize data in bi-directional movement.  

Depending on the need one should choose the right technology to achieve the needed data replication....

in memory computing

Approximately two years back I made a post on Enterprise Data Fabric technology. The aim of the data grid or "in memory data store" is to remove the movement of data in and out of slower disk storage for processing. Instead the data in kept in "pooled main memory" during the processing.

To get above the physical limitations on the amount of main memory, the data grid technologies will create a pooled memory cluster with data distributed over multiple nodes connected using a high bandwidth network.

With SAP bringing HANA, an in memory database that has option to store data in traditional row store and column store (read storing data in rows and columns) within an in-memory technology and Oracle bringing the Exaletics appliance, the in-memory computing is getting more attention.

So, the claims are that the in memory technology will boost the performance by multiple degrees. But the truth is it can only remove the time taken to move the data out of disk into main memory. If there is a query that is processing the data using a wrong access method, even if all the data is moved into a memory store the processing will still take as long to provide the answer!

In memory computing would need re designing the applications to use the technology for better information processing. OLTP workload will surely improve the performance due to memory caching but the consistency of the data need to be managed by the application moving to a event based architecture.

OLAP and Analytical workloads would also improve the performance by using memory based column stores with a good design of the underlying structure of data that suits the processing requirements.

Overall, in memory computing is promising at the moment but without the right design to use the new technology, the old systems will not just get the performance boost just by moving the data store into the main memory

Let us wait and see how the technology shapes further in future.....

Storing Rows and Columns

A fundamental requirement of a database is to store and retrieve the data. In Relational Database Management Systems (RDBMS) the data is organized into a table that contain the rows and columns. Traditionally the data is stored into blocks of rows. For example a "sales transaction row" may have 30 data items representing 30 columns. Assuming a record occupies 256 bytes, a block of 8KB can hold 32 such records. Again assuming a million such transactions that need to be stored in 32150 blocks per day. All this works well as long as we need the data as ROWS! We want to access one row or a group of rows at a time to process that data, this organization has no issues.

Let us consider if we want to get a summary of total value of type x items that are sold in past seven days. This query need to retrieve 7million records that contain 30 columns each to just process the count of items of types x. All that we need is two columns item type and amount to process this. This type of analytical requirement lead us to store the data in columns. We group the columns together and store them in blocks. It improves the speed of retrieving the columns from the overall table quickly for the purpose of analyzing the data.  

But the column storage has its limitations when it comes to the write and update

With a high volume of social data, where there is high volume of write is needed (like messages and status updates, likes and comments etc.,) , highly distributed, NOSQL based column stores are emerging into mainstream. Apache Cassandra is the new breed of NOSQL column store that was initially developed by Facebook.

So, we have a variety of data base / data stores available now, a standard RDBMS engine with SQL support for OLTP applications, A column based engies for OLAP processing and noSQL based key value pair stores for in-memory processing, highly clustered Hadoop style big data with map/reduce framework for big data processing and noSQL based column stores for high volume social write and read efficiencies. 


Making right choice of data store for the problem in had is becoming tough with many solution options. But that is the job of an architect; Is it not?

ACID and BASE of data

I am completing my 18 years of working in the field of Information Technology.

All these days an enterprise  data store generally provides the four qualities Atomicity, Consistency, Isolation and Durability (ACID) to the transactions. Oracle has emerged as a leader in providing enterprise class ACID transactional capabilities to the applications.

Recently in the Open World 2011, Oracle announced a noSQL database which typically characterized by the BASE acronym. Basically Available, Soft state, Eventually consistent (BASE)

I see a lot of debate on SQL vs NoSQL, ACID vs BASE and Shared Everything vs Shared Nothing architectures of data stores of late; and with Oracle getting on to the NoSQL bandwagon, this debate is just took up additional momentum.

Oracle has posted this paper nicely explaining their NoSQL database. http://www.oracle.com/technetwork/database/nosqldb/learnmore/nosql-database-498041.pdf

In my opinion, SQL and NOSQL choice is straight forward to make:-

big query: Are we storing data or BIG-DATA (read my old post on transactional data vs machine generated big data - http://technofunctionalconsulting.blogspot.com/2011/02/analytics.html)

With the new trends in 'BIG DATA' all the data almost become key, value pair with read and insert only operations with minimal or no updates to the data records. NoSQL/BASE is best suited to handle this type of data. Still the traditional transactional databases of OLTP nature, needs ACID complaint transactions.

So, when designing the big data solutions, an architect should surely look at the NoSQL dataBASE. Is it not?

Publishing this post on 09/10/11 (dd/mm/yy) and this is my 85th post to this blog.

SQL Plan Stability

Recently I came across a "performance problem" on Oracle database. A fairly innocent looking insert statement is intermittently taking "hours" to complete.

Problem:
INSERT INTO TARGET_TABLE ("Column List")
SELECT "values"
FROM SOURCE_TAB1, SOURCE_TAB2, SOURCE_TAB3
WHERE "All necessary join conditions and other conditions"

As the statement performing will in some instances and giving problem only in some cases, I have looked at the plans. It was generating two different plans - one with a simple nested loops and another with a Cartesian Join. When the second execution path is executed, it needed a lot of CPU and memory resources.

When I looked at the source of this query, it is originated in a job which is uploading a set of files into the database. Table stats are collected just before running the job. The development team has tested it several times in their database and they never had a problem with performance even in UAT environment. This behavior is only in the new environment that was built for the purpose of to-be production!

Reason:
As the optimized statistics are highly fluctuating from one file load to another file load, Oracle Database is generating different plans when the query is executed with different bind variables.

Solution:
Plan Stability can be achieved by
1. Importing the statistics from a stable environment and LOCKed.
2. By the way of providing Hints
3. By creating SQL Profiles
4. By generating stored outlines
5. By SQL plan Management (SPM) functionality in 11g.
We took a simple approach of importing and locking the statistics for the intermediate file upload schema to achieve the stability which worked well in the environment. but the most sophisticated SQL Plan Management functionality in 11g can solve most of plan stability issues. See this Oracle TWP - http://www.oracle.com/technetwork/database/focus-areas/bi-datawarehousing/twp-sql-plan-management-11gr2-133099.pdf for more details.

Notes:
As there is no silver bullet, one should be careful in implementing new features. This blog post explains the flip side of SPM and how to be careful with it.

To put it simply, be careful with setting optimizer_capture_sql_plan_baselines to TRUE. One can enable this parameter at a session level and capture the needed baselines and use them to get a consistent performance!

finally, however cleaver the RDBMS engine becomes, it can still commit blunders! an experienced DBA can never be replaced while dealing with performance issues!!

SQL performance tuning

Having seen several performance problems within IT systems, I have a methodology for performance tuning. When it comes to a SQL query tuning, it should be slightly different.

9 out of 10 cases of performance problems on relational database systems relate to a bad SQL programming. Even with the latest "optimizers" within the commercial database management core execution engines, it is the skill of the developer to make use of the facilities effectively to get the best out of the RDBMS.

I have recently came across a typical problem with a "junction table" design.
A set of tables represent USER and all are connected by a USERID
Another set of tables represent ACCOUNT and all are connected by ACCOUNTID

The software product implements the one to many relationship using a junction table called USER_ACCOUNTS which contains (USERID, ACCOUNTID) with the composite primary key USERID, ACCOUNTID.

Now there are 30K users 120K accounts and 120K USER_ACCOUNTS and a query that need to get data from USER tables involving some outer joins on itself and ACCOUNT tables which joins multiple tables to get various attributes; all these tables linked in a join using the junction table. That query runs for 18 hours.

When the query is split into two inline views with all the required filtering in each side of data access on USER and ACCOUNT individually and then joined using the junction table it completes in 43 seconds.

So, FILTER and then JOIN is better than JOIN and then FILTER in terms of resource consumption. Hence the performance tuning is all about doing the sequence of actions in the right order to minimize the consumption of resources to perform the job!

Maximum Security Architecture

In one of the past posts, I have just listed different technologies that are available in the Data Security area. (in Feb 2008)

With Oracle 11g database, the security focus has taken more methodical and architectural approach.

To put things together data security is placed under the following broad (four) categories:
1. User Management
2. Access Control
3. Encryption and Masking
4. Auditing/Monitoring

Just like Maximum Availability Architecture for Highly available architectural patterns, we can call this as Maximum Security Architecture for highly secure architecture....

One should choose the required options and implement it properly to really make the data SECURE!

This Link gives more details of MSA on Oracle 11g database.

Hybrid Columnar Compression (HCC)

Oracle has introduced a new data storage technology in Exadata V2 called Hybrid Columnar Compression. This method uses a "compression unit" that is made of several rows; it is physically stored in compressed "column vectors" in multiple blocks of storage.

There are two flavors of columnar compression that are optimized for "QUERY" (warehouse compression better for executing queries) and "ARCHIVE" (best data storage for archive compression)

Each of the QUERY and ARCHIVE come with another modifier to set the "LOW" or "HIGH" compression mode.

This technical white paper gives an overview of the new technology.

This article on Oracle Magazine may also be useful in understanding this new feature.

More data can be stored in "less" space. Good!

sizing a data center management solution

Introduction:

A typical data center management solution will have a "data store" for storing
a. configuration data
b. monitoring data

On regular intervals it will collect the data and upload it to the central data store.

A host, database, listener, application server instance etc., are the managed entities that will need monitoring and management.

Before implementing the solution, every organization will try to "size" the infrastructure requirements for the solution.

The sizing involves estimating the resource consumption for the solution.
1. disk storage
2. memory
3. cpu cycles required
4. network bandwidth requirements

The Problem:
The complexity involved in sizing a data center management solution is mainly due to lack of clarity of definition.
for e.g., customer want to monitor 10 databases on 10 servers. (one DB on each server)
Depending on the database version and weather it uses ASM for storage or it is on a cold failover cluster and several other considerations the number of managed entities will vary. A database can have only one tablespace and a single datafile or a database can have 10K tablespaces with 100K datafiles. If customer want to monitor the database space usage by tablespace a database with 10K tablespaces will produce 10K times more data when compared to a database with 1 tablespace.

Each metric (the monitored datapoint) can also be collected once in every 5 minutes or once in an hour. The collection frequency may vary based on customer requirements.

A simple formula is number of metrics ( number is at lowest granule) * number of collections per day gives the total number of metric values per managed entity. Multiplying this number with bytes required to store an average metric value gives the bytes required to store the metric values per managed entity.

If customer wants to keep this data at this granularity for a week, then the storage requirement is number of days of retention at raw granularity * bytes required to store one day data. This multiplied by the number of managed entities gives the total storage requirement for this type of managed entitiy (e.g., database)

The same exercise need to be repeated for all types of managed entities to get the storage requirement.

Collecting that many bytes over a day means transferring that data over the network from the host where the managed entity resides to the host where the management data store resides. That gives the network bandwidth requirement.

This data need to be rolled up to a hourly average and daily average for keeping it for historical trending. One need to calculate the space requirements and processing requirements in this rolling up process.

The old data need to be purged away from the data store. It needs processing cycles.

All the managed entities should also have a set of configuration data. That need to be collected, compared on a regular basis and kept up-to-date. One need to compute the resources required to collect, compare and store the original and changed values of different configurations as they evolve.

Managing all the "heartbeats" between the central management solution and the monitored hosts requires a proactive two way mechanism. This needs processing capacity.

Monitoring the websites with synthetic transactions by periodically playing a pre-recorded transaction to make sure the application works require additional set of processing, storage, memory etc., This need to be added to the above estimate.

A provisioning solution that is used to deploy new instances of databases, application servers will need the GOLD IMAGES to be stored somewhere. Patching the enterprise applications needs considerable amount of patch download space and prorogation on regular intervals. This also need to be considered.

The other thing to consider is about the periodic tasks (jobs) that get executed to perform routine operations. Each job may produce few KB of output that need to be collect and stored in the central store for a period of time.

The next thing to consider is the number of users that use the application, number of concurrent users and the amount of data they retrieve from the data store for performing their operations.

Collecting all this information, adding the block header, segment header etc., overheads in the phisical structure of the target solution is a complex task. By the time this exercise is almost complete, the tool vendor would have released the next version of the data center management tool with some modifications!!

The solution:
It is nearly impossible to "size" any application to the last byte accuracy for storage, memory, Network and cpu utilization.

No managed entity should generate more than 25MB of data in the monitoring store. No managed entity should have more then 100kb of configuration data.
So, taking a 250GB storage for 1000 monitored entities and configuring the solution in such a way that it will not exceed this requirement is a wise man's solution.

Considering a 1000 monitored entities will have a maximum 100 administrators and at most 30 of them concurrently loggedin, An average db oltp application with 250GB database and 24 * 4 (considering 4 txn's an hour i.e., 15 min collection frequency) * 1000 (no of entities) would require a 2 processor DB server and a 2 processor Application server with 4GB RAM each.

Starting with this configuration, implementing the solution using an iterative model is the best approach to balance between the up-front sizing vs not impacting the service level of this critical service.

Conclusion:

In the current world with virtualization and cloud technology it is easier to build scalable Grid like applications and scaling the solutions horizontally is the trend that is going to stay and further pick-up more momentum. The data center management solutions are not exempt from this trend. For every next 1000 monitored entities we will have to add a node to the DB grid and another node to the AppServer Grid along with additional storage to the storage grid.

IS IT NOT??

Improved Application Performance Management

In April 2008 I have written this post : Application Performance Management

With
a. Oracle Enterprise Manager 10.2.0.5 Service Level Management,
b. REUI 6.0,
c. SOA mamagement pack and
d. Diagnostic Pack for middleware the solution is available.

A corporate can now implement a completely integrated APM solution based on the above four components of Oracle Enterprise Manager.

The new capabilities of REUI 6.0 include:
a. Full capture and replay of end user sessions.
b. Ability to link the slow performing JSP to the overall configuration in SOA management pack and then jump directly into the diagnostic tool (AD4J)
c. Customizable dashboards from the collected data.

What is REUI? "Real End User Experience Insight"! and What is 6.0? The new version released on 1st December 2009.

The ACTIVE MONITORING functionality is already provided by Enterprise Manager Service Level Management pack.

We hope these new tools will truly improve the Real End User Experience of the web applications.

Exadata V2 - World's First OLTP database machine

Last year the Exadata Database machine was announced for making the data warehouse performance improvement by moving the database query processing to the storage layer.

This year, the new Exadata V2 is proven to be the fastest database machine for OLTP applications.

The trick lies in the Flash solid state storage and the 40GBPS infiniband internal network fabric used between the DB servers and the Storage servers. The full rack contains 8 database servers and 14 storage servers. There are other configurations available.

All the flash storage (600 GB per storage server) is viewed as the extended memory of the servers and it has 72GB of DDR3 memory as the cache per database server....

Guess what, an open challenge to IBM can be found at http://www.oracle.com/features/exadatachallenge.html

So, why not try out for a $10 Million prize?

Oracle Open World 2009 a mega event

Three years back I was at Oracle Open World 2006 staffing one demo pod on "Deployment Best Practices for Enterprise Manager"





Again this year, back to this event where I am presenting a session and three hands on lab sessions to attendees.

Link to the Sessions

S309533 - Applications Management Best Practices with Oracle Enterprise Manager
Oracle Enterprise Manager Hands-on Lab: Oracle Application Testing Suite
Oracle Enterprise Manager Hands-on Lab: Oracle Data Masking Pack
Oracle Enterprise Manager: Oracle Real Application Testing

Event dates: October 11-15 2009
Catch me there!!

Oracle Database 11g R2 released

Oracle has released the latest version of the worlds largest used database!

Key new features are:
"Shared Grid" with server pools for consolidating multiple applications into a single RAC cluster
"RAC One Node" for consolidating less critical application systems transparently get the benefits of easy movement etc.,
"ASM Cluster File System (ACFS)" in the storage management
"Instance Caging" for confining the database to specific cores in a SMP environments
"In-Memory Database Cache (IMDB Cache)" for moving the OLTP processing to the middle tier
capabilities to use the Disaster Recovery (Idle) infrastructure for Reposting and Backup operations.

It has several security enhancements as well.
This Oracle White paper has more details.

It is nice to hear about the new features from Tom Kyte. Being with oracle I had the opportunity to hear about the features directly from the "Architect" of those features!

New Technology and its Manageability

New Technology and its Manageability

However great the new technology is, it requires a manageability solution along with it. "Manageability" is the overall ease of the technology starting at its commissioning until its decommission.

Even with great manageability features, the Oracle RAC took some time to move into mainstream production. [Ref. Gartner Report on Oracle RAC claims it took nearly eight years]

In that case, what is the importance of "Manageability" of a "Enterprise Manageability Solution"???