Monday, June 19 2023
A consumer electronics company with global manufacturing and marketing operations relied on a large and complex data warehouse to support daily operations as the single system of record. The system architecture specified that Hot Storage, used to store mission-critical data for real-time data processing and analytics, should consume a maximum of 60% of system memory. The remaining 40% is reserved for the Operating System and Calculation Workspace to support daily business operations and accounting period closures. As one of the world’s largest companies with extremely large amounts of data, a formal data governance model is in place which provides guidance on the handling of multi-temperature data (Hot, Warm, Cold) and Archived, and Deleted Data.
A growing problem emerged as multiple systems were consolidated and caused data to grow at a rapid rate which caused the Hot Storage to exceed its allocated portion of memory capacity and consume memory intended to be used as Calculation Workspace. The net effect was the workspace for calculations was reduced every month impacting overall system stability and performance.
Design and build a cost-effective solution that allows the system to facilitate business users to support operations and closing accounting periods. In addition, the solution should accomplish the business directive without increasing fees for data, with adherence to the established data governance policies of the company.
SAP HANA Native Storage Extension (NSE) was selected as one of the core elements of the solution. NSE makes possible the use of Warm Storage in SAP HANA to compliment Hot Storage. This allows less-frequently accessed data to be accessible while preventing it from fully using hot system memory. This is achieved by utilizing disk-based database technology alongside SAP HANA in-memory database. By off-loading data from Hot Storage to Warm Storage on the disk, the amount of memory available for the Calculation Workspace was restored. This solution offered a better price-performance ratio than continually increasing the size and required bandwidth of the database.
With this architecture, when a query attempts to access data from warm storage, only the required data is brought into hot storage in the form of pages. This approach avoids the need to bring the entire partition into hot storage, ensuring efficient memory usage and query performance.
Through the successful implementation of NSE, we have efficiently addressed the issue of tables consuming excessive amounts of system memory.