- Vadym Humeniuk - Resident Architect
- Datapunkt Enterprise Architecture
- Hits: 478
Medallion Strategy in Datapunkt
Introduction
In the intricate landscape of modern business, the effective management and utilization of data have become paramount. Enterprise Data Architecture (EDA) emerges as the master plan, weaving together diverse data sources, ingestion methods, processing layers, and consumption avenues. Let's embark on a journey through the layers of this architectural tapestry, uncovering its key elements and their role in shaping a data-driven future.
Data Sources: Foundation Stones
1. Relational DB:
- Type: Anchoring Structured Data
- Characteristics: ACID-supported, Consistency
- Utilization: Handling transactional data with precision.
2. Object Storage:
- Type: Reservoir for Unstructured Riches
- Utilization: Providing scalable storage for images, videos, and static content.
3. Streaming:
- Type: Dynamic Real-time Flow
- Utilization: Navigating the swift currents of real-time data streaming.
Data Ingestion: Building Bridges
Efficient data ingestion serves as the connecting bridge, bringing diverse data into the enterprise fold.
- Batch Data Processing from Object Storage
- Batch Data and Change Data Capture from Relational DB
- Streaming Real-time data from Streaming Data
Data Processing: Navigating the Layers
1. Landing Layer:
- Data Types: Handling initial data touchpoints.
- Focus: Ensuring consistency in Object Storage and facilitating real-time streaming.
2. Raw Layer:
- Data Types: Raw data in its purest form.
- Focus: Consistency in Object Storage and real-time streaming.
3. Curate Layer:
- Data Types: Shaped and refined data.
- Focus: Ensuring consistency in Object Storage and Data Warehouse while facilitating real-time streaming.
4. Business Layer:
- Data Types: Business-ready data.
- Focus: Consistency in Object Storage and Data Warehouse, facilitating real-time streaming, and supporting relational databases.
Data Consumption: Unveiling Insights
Data, having undergone processing and curation, transforms into a valuable asset for diverse forms of consumption.
- BI Tools
- Query Engines
- AI/ML
- Apps
Enabling Components: Pillars of Strength
- Metastore
- Data Catalog
- Data Management
- Data Lineage
- Central Access Layer
- Data Modeling
- Orchestration
- Secret Storage
- IAM Policies
- API Management
- Cost Management
Conclusion: Navigating the Data Frontier
Enterprise Data Architecture stands as the compass guiding organizations through the intricate realm of data. As technologies evolve and data volumes surge, a well-architected data framework becomes not only a necessity but a strategic asset, propelling organizations toward a future where data is a catalyst for growth and innovation. In this journey, the data architecture serves as the lighthouse, illuminating the path toward a data-driven future.
Learn more about Datapunkt Operational Model
An Operational Model is a structured framework that defines how an organization's resources, processes, and strategies work in concert to achieve its objectives. It serves as the operational backbone, aligning various facets of the business to optimize efficiency, foster innovation, and deliver value to stakeholders.
