Blog Moved

Future posts related to technology are directly published to LinkedIn
https://www.linkedin.com/today/author/prasadchitta

Saturday, September 10, 2011

Tiered data storage

Hierarchical Storage Management
Not too long ago (about 10 years back), I have done a strategy for "Data Archival" options for a system that has lots of data which need to be preserved for 25 years due to legal reasons. (sort of Records Management requirement) The requirement is to have it fully query-able fine grained data in the system. The key challenge was keeping all data in on-line storage with the technology available at that time. So, we need to have a clear "Archival Strategy" to move the data off from the disk to tape and preserve the "Tapes" in a way they can be retrieved (by methods of proper labeling etc.,) on-demand within the given service levels. This technology later named as Hierarchical Storage Management. Overall strategy included manual tiering of data between the disks and tapes sometimes using a mechanical robotic hands and associated software around them.

Information Life-cycle Management 
As the technology advanced, the disk storage evolved to multiple bands of cost/functionality. The database software like Oracle came up with options like table partitioning and advanced compression. Combining these advances in the database management systems and the storage a new strategy emerged as Information Life-cycle Management. Logically partitioning the tables and putting them in the different types of storage like Enterprise Flash Disks (EFD), Fiber Channel (FC) and SATA disks using an automated storage tiering is the trend of the day.

Thin provisioning technologies like EMCs Fully Automated Storage Tiering - Virtual Pooling FAST VP and Hitachi's Dynamic Tiering etc., when used with Oracle's ASM and the partitioning & advanced compression options gives the best flexibility, performance and value for money. There is a good whitepaper from EMC with published few months back that can be found here

Conclusion:
Most of the storage vendors now have the Tiered storage technology embedded in the disk controller software layer that can automate the data migration or intelligently cache and tire the data across multiple types of storage. Using the available technology with right mix of logical features of database and storage virtualization  leads to better data availability at the optimal cost. Still the "right solution" is a job of a knowledgeable Architect! (who can understand the Business and Technology well!!) 

No comments: