Building Application Streaming for Real-Time Ad Metrics with Databricks

A large advertising technology company needed to monitor online advertisements to ensure compliance, track performance, and collect metrics on views and clicks. They needed to create real-time dashboards accessible via customer portals, enabling their clients to monitor ads in real time. The company selected Databricks for its robust, cost-effective streaming capabilities. They aimed to expedite the initiative to save money by phasing out their inefficient and expensive legacy data platform.

Aimpoint Digital took a comprehensive approach to address the client's needs. Our team developed four main streaming pipelines into a centralized data warehouse. Then, we implemented streaming transformations to join and aggregate data in real-time as it passed through the data warehouse, making it faster to generate metrics and reports on the portal side.

Our team also established a systematic method to filter and secure data for row-level access. This ensured that only relevant customers could access their specific data, with no access to other data points. Furthermore, the Aimpoint Digital team created deployable scripts that allowed for easy creation of additional streaming pipelines into Databricks with minimal effort from the existing team. Terraform was used to enable infrastructure-as-code, streamlining the process of setting up new pipelines.