Lesetips Data Mesh: Beneath the surface

With the rapid advancement of digitalization needs that the world has experienced the past couple of years, data has risen as the ultimate champion, for organizations to become the leaders of their field and achieve their goals. Data driven and AI enabled are just a couple of examples of the buzzwords that are dominating our everyday life. So, if you are someone either working with or highly interested in data, you have probably heard the term Data Mesh quite frequently, especially the last year.

Data Mesh is, according to its inventor Zhamak Dehghani, an architectural paradigm that aims to challenge the already established data platform concepts of Data Warehouse & Data Lake, and provide a more domain driven approach that is better suited for the fast pace, cloud focused data environment we live in. The four pillars of Data Mesh are a domain-oriented distributed data ownership & architecture, data as a product, self-serve data infrastructure as a platform, and federated computational governance. (Dehghani)

These principles sound very appealing when examined at first, and they are indeed given the right context. However, it is within our human nature, to most often only examine things on the surface and avoid diving to see the whole iceberg that resides beneath. Many organisations are feeling left behind in the data revolution, and they end up being driven by the “fear of missing out” phenomenon. They are overly focused on the newest technology or the newest architectural trend, looking for a quick fix or a silver bullet that will take them to the data driven paradise. Alas, reality is very different from the promises of technology vendors or the benefits of a new type of data architecture.

Even though this article talks about Data Mesh, it is essentially not about Data Mesh. It is about some of the most basic principles of proper Data Management practices. It is about understanding the problem before applying the solution. It is about resisting our innate voice asking for a quick fix that will essentially hide a problem instead of addressing it properly. It is about understanding your needs, assessing your capabilities, working through to a solution and applying it in the best way that fits your goals and general strategy, based on your available resources. It is about “Fit for purpose” in all things in life, Data included.

Data mesh and other distributed architectures are here to address the need to move to a more agile way of working with data and keep up with hugely increasing data volumes and the scaling potential of the cloud platforms. They all carry value and merit analysis for possible use and implementation. What organizations fail to understand, though, is that in terms of managing your data there is no one size fits all. I would argue the exact opposite. Each organization is unique and thus its Data Strategy and Data Management methodology should be as well. It is great for organizations to aspire to reach the data operational excellence of some of the world’s the greatest companies and want to learn from them, but it is equally important to keep a clear perspective of the journey it takes to get there.

Data mesh requires above all a high level of data maturity across all parts of an organization as well as a well-structured and established Data Management framework that aims to be further refined to take the organizations data journey into the next level. Even in such cases, however, it is important to evaluate both the benefits and the disadvantages of moving from a more centralized model to a more distributed one. By moving to a higher agility & flexibility environment, sacrifices need to be made in terms of governance principles. Abandoning governance all together for the sake of speed and agility might seem very appealing at first, but it always carries almost long-term consequences that when they surface, they can expose critical parts of the operational model and require disproportionately high resources to be remediated. Apart from the time and resources needed to remedy these issues, the most important consequence, as well as the most impactful one, that most organizations seem to forget, is the loss of trust to the data itself.

As with most things, the truth often resides somewhere in the middle. No one wishes a heavy moving super-centralized data governance model nowadays, but you need to make sure that you don’t end up with a domain siloed organizational structure either. Both sides will result in a non-optimal Data Management practice. It is therefore very important to find the correct balance between agility and governance, between centralization and distribution, with the organization’s overall strategy as your guide. This is the greatest challenge for all Data Management professionals out there and the reason I find each client’s case unique.

The bottom line is that Data Management is extremely beneficial for your organisation, and vital if you want to use Data as a strategic asset and wish to excel in areas like Machine Learning or Artificial Intelligence. However, in order to reap the benefits, a long and arduous journey awaits. There are no shortcuts or easy quick way to do this, and it is not a single project but an integral part of the organizations every day function throughout its lifecycle. I believe our mission as Data professionals is to more to guide organizations into navigating through these waters as they embark on this long journey.