A market leader, internationally renowned media and broadcasting company

Founded in 2002, our client has been around for over two decades and is an internationally known company dealing with broadcasting data for commercial use. With a mission of making high-quality technology and content affordable for everyone, they have established themselves as a market leader.

Date reporting delays

The client deal with real-time broadcasting and commercial data, and the data needs to be available in near real-time. Their existing ETL infrastructure built on AWS technologies such as Redshift was not able to meet their expectations and requirements. Our highly experienced engineers presented the solution of leveraging Databricks for the whole ETL process for the many benefits it offers including parallel processing with multiple clusters.

Immense real-time data

The client was dealing with considerably large datasets including the likes of 215 million records of commercial data being processed on an hourly basis, and 9 million records of content data being processed hourly. This data created eighty reports which need to be produced quickly and efficiently.

Parallel processing on multiple clusters with Databricks

To significantly improve the speed of data availability and reporting, the client required a solution which would enable the parallel processing of multiple data clusters. The ETL pipeline infrastructure was built in line with customer requirements and the data flowed through multiple stages:

• Raw data, extracted from the source

• Staging

• Silver

• Gold

Why Databricks?

Databricks is a brilliant technology used for ETL processes with user-friendly dashboards to track the entire process and spot any errors. Our highly skilled data engineering team are proficient in Databricks and has utilised it for many client projects, therefore they were able to make the recommendations to help the client overcome their challenges.

Automated error resolving and reporting

The solution was built with automated error resolution in place which is offered by Databricks. Our engineers were able to set a number of tries that the system would make before the error was reported for manual intervention. This helps the data engineering team ensure that there are no data drops and delays and errors can be resolved quickly and efficiently. There are two main errors that there is a potential for occurring and these include:

• Data drops – This would mean the batch processing is incomplete therefore the processing of that particular data batch would be repeated.

• Data gaps – In this case, there are gaps in the data which would require a backfill to when the error first occurred.

Ongoing operational monitoring and support of the ETL

Our client are thrilled with the high performance of the new and improved ETL infrastructure built on Databricks which is driving a new speed and efficiency for their reporting. Our expert data engineering continues to provide an ongoing operational monitoring and support service to the client to ensure data availability and accessibility at all times.

Ensuring timely data availability for real-time, mission-critical data with Ardent data engineering service

Ardents' team of highly skilled data engineers are proficient in world-leading data technologies and can make recommendations based on your unique needs and requirements. Whether you have a preferred tech stack or want expert guidance on the technologies right for your data and business, we can help. Are you facing any of these challenges:

• Delayed data reporting turnaround

• Data delays, gaps and dropouts

• Slow data performance and speed