Data & AI Products (Data Mesh) on Databricks: Making Data Engineering and Consumption Self-Service Driven for Data Platforms
In a quest to establish “Data as a Product” for both internal and external stakeholders, our client, a prominent IT and business consulting firm, partnered with Infosys. This collaborative project adopted a data platform approach, utilizing Delta Sharing, API endpoints, and Unity Catalog to bring to life the Data and AI Products (Data Mesh) architecture.
During this session, three key design patterns will be presented, offering valuable insights for organizations seeking to evolve towards a no-code/low-code approach. These design patterns serve as practical examples of how to effectively leverage the power of data platforms to deliver data-driven products.
The first design pattern focuses on Delta Sharing, a technology that facilitates secure data sharing across organizations. By implementing Delta Sharing, our client was able to seamlessly exchange data with their stakeholders, enabling collaborative data-driven initiatives.
The second design pattern highlights the utilization of API endpoints to expose data and functionality to internal and external users. This approach enables easy and controlled access to data products, promoting self-service capabilities and empowering stakeholders to make informed decisions based on real-time data.
Lastly, the session will delve into the implementation of Unity Catalog, a tool that simplifies data discovery, access, and governance. By leveraging Unity Catalog, our client achieved a centralized and unified view of their data assets, streamlining data management processes and ensuring compliance with governance policies.
