Leading entertainment Fortune 500 company

Headquartered in California, our client are a well-established Fortune 500 company worth over a few billion as of October 2022. They deal with a large scale of various broadcasting data including audience and commercial data. We have worked with them on a number of projects to help unlock the potential of data by continuously improving and optimising data performance.

Data delays and slow reporting

Our client deal with huge volumes of data and were having delays in their reporting. They were running around 80 reports and each report took around 4 to 5 minutes to be produced. The data reports delay of each report adds a considerable amount of time to the reporting time of the full 80 reports. Therefore, our client were looking for an alternative solution to significantly improve their data reporting turnaround.

Improving data turnaround by 80% with Databricks

Our clients existing data structure was built on Amazon Redshift, which is a powerful technology, however, it presented data delay challenges for our clients' data. Therefore, we recommend Databricks as the alternative to processing data quickly and efficiently. Databricks offered scalable, efficient and quicker processing of data with the use of independent clusters which can run parallel.

Databricks clusters

Our highly experienced data engineers created three clusters on Databricks including Cluster A for storing all data, Cluster B to set up ETL, and Cluster C for any issues and delays, which could then be moved to a new cluster to enable parallel processing. One of the biggest benefits on offer with data bricks is the ability to create as many clusters as required to process data in parallel. This enabled much more efficient and quick processing of data improving data reporting speed by 80%.

215 million rows of data processed hourly

Cluster B where the ETL process is running had 16 nodes and a huge amount of data is being processed. Approximately 9 million rows of viewing data and 215 million rows of commercial data are processed on an hourly basis and around the clock, every day.

Errors and optimisation

As the streams of data are constantly flowing, our engineers provide operational monitoring and support to spot errors, resolve issues and continuously make recommendations to improve and optimise data performance. PagerDuty is employed for error alerts, which are then resolved by the Ardent data engineers.

Ardent ongoing monitoring and support

Overall, our clients can significantly reduce the data processing and reporting time with the adoption of Databricks. This offers them many benefits from improving productivity to a better data turnaround time for end clients. They have peace of mind with the operational and monitoring support as any errors and issues that may arise will be resolved quickly and efficiently.

Additionally, both the Ardent team for this project and the client's data science team have regular meetings to discuss progress and optimisation suggestions.