Data Warehousing

Customer: Origin Energy
Industry: Oil & Energy
Services: Amazon Data Warehouse

Download success story

Challenges

Design and deliver a scalable solution to enable aggregated data to be queried and provide business intelligence
Ongoing monitoring and management of raw data for analytics
Support operational and engineering services in large and complex data services environment

Outcome

Successful design and delivery of data warehouse – Redshift implementation
A data warehouse and data lake that is fully integrated with existing platforms and infrastructure
Trusted partnership between Origin and CyberCX to drive future cloud technology outcomes

Origin Energy

Origin Energy originally engaged CyberCX to undertake the initial design and delivery of one of Australia’s first Amazon Redshift Cloud Data Warehouse implementations.

Since inception (over four years ago) this environment has grown to become one of the largest data lakes in the country. It provides real time analytics capabilities that enable Origin customers to effectively query and model their energy usage and projected spend. It has also expanded to include Snowflake and AWS Aurora.

The challenge

The Origin Energy data services landscape is extremely large and complex with stakeholders and business units spread across three Australian states. There are a wide number of products and services with multiple support teams from a range of vendors.

The solution

CyberCX has established management of the entire data lake infrastructure as well as provide architectural consulting, development and DBA support for a range of platforms (Postgres, MySQL, SQLServer).

The environment consists of a number

of thirdparty integrations and tools such as Matillion ETL, BryteFlow, AlexSolutions, Tableau BI and SAS. We have recently been involved in the incorporation of several native AWS services to improve data loading, scalable compute and the ability to query unstructured data. A number of machine learning components have also been added to the platform including AWS Sagemaker and Databricks.

CyberCX provides 24x7 support,

monitoring and reporting for the data and analytics components as well as the AWS infrastructure. We also provide maintenance, recommendations and PoCs for prospective new technology.

The outcome

CyberCX’s relationship with Origin Energy has been extremely successful over the last 4 years and is now an established and respected part of their data services ecosystem. We have extended the relationship to new support contracts with the Integrated Gas and Virtual Power Plant teams.

In addition to support, CyberCX is the preferred partner for the design and delivery of projects covering a number of areas from the design and development of Matillion ETL flows, to the implementation and migration of Snowflake warehouses and the design and development of BI dashboards.

We are also constantly involved in the onboarding and integration of new data sets into a data ingestion/processing pipeline that we built using cloud native services, established CI/CD tools and Apache Airflow.

AWS services used

CyberCX’s data services and managed services teams have been providing level 2 (operational) and Level 3 (engineering) support for several analytics tools and technologies and have experience/skillset in the following:

Data analysis

with Python and ANSI SQL

Creating Tableau

dashboard and stories

Creating data

repository and visualisation with ELK stack

Data analysis

with Amazon Athena