Business Problem & Impacts

A consumer electronics company with global manufacturing and marketing operations relied on an unprecedentedly large and complex data warehouse to support daily operations. The problem emerged as the execution of reports containing large sets of data for analysis slowed down the system response time and disrupted daily operations. The short-term workaround consisted of limiting the size of all extracts, which forced analysts to expend numerous hours on the performance of small extracts that required aggregation prior to analysis. The resulting delays impacted financial closing, and tactical decision making by leaders and the executive team.

Solution

As the team looked for proven solutions on how to accomplish the requirement, it became clear that no other company had successfully solved this problem given the scale of data. Unique IT Solutions collaborated with AWS and Client Architects to create a scalable platform for data extraction providing timely access to critical information used for decision making across the enterprise.

Project Methodology

A cross-functional team was created with individuals from each organization and an Agile methodology was used to create an MVP (Minimal Viable Product). Success criteria were defined as scheduled delivery of specific big data sets without performance degradation of the enterprise technical systems.

Solution Overview

Designed and built a low-code platform to efficiently extract data and make it available for analysis. Subsequent improvements have incorporated Modular Coding with Smart Logic that facilitate rapid changes in existing extracts and reduce the development time of future data targets.

The platform consists of three distinct areas:

• Data Ingestion
  • The framework performs the rapid extraction of small amounts of data from existing source systems using AWS Glue, PySpark, Lambda functions, and smart logic to avoid performance degradation, and moves it to a specially configured AWS S3 landing zone.
• Data Curation
  • The partial extracts are combined by leveraging AWS Lake Formation Catalog to create a large data set in specified formats. The result is a set of curated data that is made available for consumption by analysts and downstream systems in a transformation zone.
• Data Consumption
  • The resulting big data is made available for use by business analysts using various technologies such as AWS Athena, Power BI, Tableau, and SAP BOBJ. Data Science and advanced analytics efforts utilize tools such as AWS SageMaker and other ML models.

Results

• Produced the targeted big data sets in under 90 days
• The memory footprint of the Data Warehouse was reduced by over 30%
• Reduced the effort for Financial Close by Senior Financial Analysts by over 300 hours every month
• Modular Coding reduced subsequent extract development by over 40%

About Unique IT Solutions

Unique IT Solutions is focused on helping leaders of companies make faster and better decisions through insight into their existing data.