SQL Plan Stability

Recently I came across a "performance problem" on Oracle database. A fairly innocent looking insert statement is intermittently taking "hours" to complete.

Problem:
INSERT INTO TARGET_TABLE ("Column List")
SELECT "values"
FROM SOURCE_TAB1, SOURCE_TAB2, SOURCE_TAB3
WHERE "All necessary join conditions and other conditions"

As the statement performing will in some instances and giving problem only in some cases, I have looked at the plans. It was generating two different plans - one with a simple nested loops and another with a Cartesian Join. When the second execution path is executed, it needed a lot of CPU and memory resources.

When I looked at the source of this query, it is originated in a job which is uploading a set of files into the database. Table stats are collected just before running the job. The development team has tested it several times in their database and they never had a problem with performance even in UAT environment. This behavior is only in the new environment that was built for the purpose of to-be production!

Reason:
As the optimized statistics are highly fluctuating from one file load to another file load, Oracle Database is generating different plans when the query is executed with different bind variables.

Solution:
Plan Stability can be achieved by
1. Importing the statistics from a stable environment and LOCKed.
2. By the way of providing Hints
3. By creating SQL Profiles
4. By generating stored outlines
5. By SQL plan Management (SPM) functionality in 11g.
We took a simple approach of importing and locking the statistics for the intermediate file upload schema to achieve the stability which worked well in the environment. but the most sophisticated SQL Plan Management functionality in 11g can solve most of plan stability issues. See this Oracle TWP - http://www.oracle.com/technetwork/database/focus-areas/bi-datawarehousing/twp-sql-plan-management-11gr2-133099.pdf for more details.

Notes:
As there is no silver bullet, one should be careful in implementing new features. This blog post explains the flip side of SPM and how to be careful with it.

To put it simply, be careful with setting optimizer_capture_sql_plan_baselines to TRUE. One can enable this parameter at a session level and capture the needed baselines and use them to get a consistent performance!

finally, however cleaver the RDBMS engine becomes, it can still commit blunders! an experienced DBA can never be replaced while dealing with performance issues!!