Is this how the data architecture of the future works?

When it comes to the importance of data, two things are beyond question: the amount will continue to increase and it has long been a business-critical asset. In order to be able to gain insights from data that are relevant throughout the company and to use them strategically for growth, a standardized structure and defined principles for handling data are essential. In short: a sensibly organized data architecture. It ensures that processes run smoothly, data is accessible and compliance guidelines are observed.

Organizations have to decide on the architecture that suits them: data lake, data warehouse, data fabric – companies are now spoiled for choice when it comes to their data management.

Data-Mesh offers a new approach to data management and distribution. This even goes a step further than managing data: data mesh is a socio-technical approach that empowers employees to analyze and manage their data themselves.

Basically, a higher-level data architecture is required for successful data management. It defines how data is collected, stored, managed and used and sets standards for how different systems interact. A well-constructed data architecture leads to improved data quality and creates transparency in the handling of data.

Companies often have different data sources. The classics of data management here are data warehouses, which already existed in the late 1980s. There, the processed data is easily accessible, like in a “warehouse”. They find use in reporting and decision making using data science and business intelligence (BI) applications. In this case, the usual data setup of companies consists of a powerful database, one or more data warehouses based on it, integrated analysis tools and dashboards and possibly AI/ML tools.

As data grew worldwide, so did the requirements. Data lakes therefore appeared on the scene in the mid-2000s. There companies can store all kinds of data in large quantities. The data is neither prepared nor sorted – it swims in a “data lake” and is primarily used for data backup and recovery as well as advanced analyzes that require a large number of raw data.

Due to the different functions, both data management systems can stand alone or be part of a data architecture at the same time.

Data architectures, in turn, can be classified into two types: centralized or decentralized. Data-Fabric represents a central approach that makes data management possible regardless of storage location. The “data fabric” connects data from the cloud and on-premises environments.

However, central administration also creates a discrepancy between where the data is created and where it is used. IT data specialists often have data sovereignty, since they “sit” directly with the data that is located in data warehouses or data lakes. However, the actual data consumers – the people who want to draw conclusions from their own data – are in a different company silo and therefore do not have direct access.

The data mesh approach provides an alternative here: It enables a decentralized and distributed architecture in which data consumers are closer to their data. In this way, he empowers employees without a technical background to generate insights from the data relevant to them. In order for this to succeed, the inventor of the data mesh approach, Zhamak Dehghani, defines four principles:

1. data domains: The prerequisite for a decentralized architecture are clearly defined data domains and the associated responsibilities. In modern companies, the domains often already exist in the form of departments. With data mesh, they take responsibility for their own data – from accounting to sales to marketing.
2. Data as Product: A core principle of data mesh is the classification of data as a product with certain attributes, so-called “data products”. They should correspond to the needs of the “customers” (consumers).
3. self service: A self-service infrastructure is required so that employees can access the data products relevant to them in their domain. It offers an interface for the entire company via an easy-to-use platform.
4. governance: Despite the independent domains, teams and data products, these must be interoperable. In order for this to succeed, a uniform governance model is required that defines central, global guidelines for decision-making authority. At the same time, there are separate, federated rules at the domain level.

Through these principles, data mesh represents a cultural change that contains not only technical but also social components. A current PwC study shows that only in eight percent of the companies surveyed do all employees have the opportunity to independently create data analyses.

Data-Mesh makes exactly that possible and thus contributes to a real democratization of data and the establishment of a data culture. This also includes training employees in data competence and access to all the correct data that is relevant to them.

Many companies are already recognizing the benefits of data mesh. According to the PwC study, 36 percent have discussed the concept and plan to implement it. 32 percent want to introduce individual elements. Since it is not just about the introduction of technical components, but about a corporate “data transformation”, the following should be observed:

1. Develop data mesh strategy: With every transformation, the strategy comes first. In accordance with the company goals and the general data strategy, fields of action, goals and principles should be defined for the introduction of data mesh.
2. Review existing technology: The basis for decentralized data processing is a very powerful central system that can process different types of access. Therefore, before data mesh can be implemented, the existing technologies such as databases and data warehouses should be checked for performance, flexibility and scalability.
3. Bring management team on board and inform employees: In order to successfully introduce data mesh at scale, misinformation must be eliminated. An important piece of the puzzle is the management level, which can ensure that a better understanding of this is developed company-wide.
4. Introduce data mesh principles: Once the strategy and technology are defined and ready, it’s time to build the foundation. This means delimiting data domains, setting up these teams with data specialists accordingly, training employees and developing central governance guidelines.
5. provide analytics: Once the foundations of data mesh have been laid, more advanced features such as DataOps or Advanced Analytics are introduced to maximize the value of the architecture.

With decentralized data management on the one hand and strong central governance to control compliance on the other, Data-Mesh offers the best of both worlds and enables users to better exploit the potential of their data. Above all, companies that have a diverse data landscape with different business areas can benefit here.