Global media market research

Our client are a global media market research company based in California, USA. They analyse millions of social media conversations and thousands of consumer surveys to understand audience reactions to products and services. Our client provides media market research to businesses allowing them to create targeted, impactful marketing campaigns. They provide insights to some of the biggest companies in the world including YouTube, NFL, Instagram and more.

Large volumes of variety of data

Our client acquires large volumes of data in different formats from multiple sources. They deal with data coming in from three main sources, this includes survey data from Decipher API which vary from provider to provider, real-time data coming in from social media channels such as Twitter, Instagram, Pinterest and Facebook, as well as survey data from SharePoint which comes in different file formats and has data from different locations and clients.

The main challenge that our client faced was that they lacked a single repository of data where their data could be stored, organised, and then output with a single viewpoint.

They needed a robust, scalable, and secure data lake that could allow a smooth stream of incoming data to be organised and stored in a structured way while being quick and efficient to data queries from the end-user. The data would be used by the company's data scientists in order to analyse and report on the data which would then be provided insights to the end client.

1.3 million user survey data and over a billion social media records

Our team of experienced data experts came on board and collated the vast amount of real time social media data consisting of over a billion records and survey data of around 1.3 million users, into a data lake. The data was organised in tables to ensure that it was categorised and structured for it to be useful and be legible when it was queried to gauge insights and understanding.

The project challenge was the amount of data coming in real-time from Twitter. As the data was constant and varied, it was difficult to store the data in an organised, structured way. However, our highly skilled data engineers were able to face the challenge and create dynamic tables, which allowed the data to be checked over on an hourly basis with new categories being added to store data that did not fit within the pre-defined categories.

Ardent ensures that we deliver a solution that will grow with your organisation. Therefore, the data lake for our client was created with scalability in mind to ensure that the data lake could handle an increase in data load.

Complex data lake optimised for scalability and performance

Our client can effectively provide real-time analytics to their end clients from the vast volumes of data. The single point data capture ensures that the data is understandable for the end-user, allowing quick and efficient data queries. The data can then be used for data science, analysis and reporting and it can also be sent to third parties with structured, organised data pipelines.