The Data Mesh and the Hub-Spoke: A Macro Pattern for Scaling Analytics

In my day-to-day conversations with CDOs and data leaders, a perennial challenge discussed is enabling analytics at scale for complex and large organizations. As organizations evolve and grow, they become complex. These organizations are spread across the globe and lead multiple products and services with a plentitude of the line of businesses (LoBs) with their micro-cultures, motivations, and skills. Such organizational complexities inherit the challenges of harnessing value from data.

Organizations have been using Operational Data Stores (ODS), Enterprise Data Warehouses (EDW), Data Lakes, and Data Lakehouses for their decision support systems. However, these decision support systems have their focus and purpose, and such simplistic patterns cannot wholly cater to the requirements of large and complex organizations. Therefore, new scalable patterns need to be explored with the right balance between

The crux of the problem is managing the governance-flexibility trade-off. The fundamental question to answer is the following:

How can they ensure that their decision support systems are governed appropriately yet provide them the flexibility to innovate at their own pace?

Two macro patterns of Data Mesh and Hub-Spoke have emerged recently. However, these patterns are nascent and have layered complexities that need to be addressed systematically to adopt these patterns.

This blog is the first blog in the series. It establishes the need for these macro-patterns, clarifies the underlying concepts, and provides the conceptual architecture for these patterns.

The Two Concepts

Firstly, I would want to break this complex problem into manageable parts. There are two conceptual building blocks for managing this trade-off:

1. The structure of a domain
2. The governance-flexibility spectrum

Let us discuss these concepts briefly.

The structure of a domain

The first concept to discuss is a domain. The following figure depicts a domain concept from an organizational point of view.

Typically, a complex organization has two organizational units: the central and subunits.

• Central Unit: This is the unit that is at the organizational level. They may prescribe guidance that is expected to be followed by the subunits. For example, they may hold budgets distributed for various initiatives across the subunits. They may also have platforms that fulfill group-level requirements.
• Subunits: It is not uncommon for organizations to have many subunits. The subunits may have differing levels of independence from the central unit. This degree of autonomy is based on the organizational structure and its culture.

Typically, these subunits can belong to three categories:

• The first category comprises a different organization (entity) within a group organization in the same or another geography.
• The second category comprises an independent business unit within the same organization.
• The third category is an intra-organizational department.

A domain is a derivative of central units and subunits.

A domain is defined as any logical grouping of organizational units to fulfill a functional context subjected to organizational constraints.

• The functional context implies the task that the domain is assigned to perform. The functional context is the raison d’être for the domain.
• The organizational constraints can be business constrained imposed on the domain like regulations, people and skills, operational dependencies.

Typical examples of domains are:

• A department like marketing or sales focuses on a specific function within a business.
• A product group that focuses on creating a specific product or service.
• A subsidiary of a parent company.

Each domain requires technical capabilities that are fulfilled by a . As an example, for fulfilling the technical capability of a decision support system, a node can have components like an Operational Data Store (ODS), a Data warehouse, a Data Lake, or a Data Lakehouse along with its peripheral components like data catalog, data processing, machine learning, etc.

Each domain may have its node or rely on another domain for its node. For example, the following figure depicts the formation of a node derived from layers of a Data Lakehouse.

It is important to note that the node is a technical construct.

Nodes fulfill a specific technical capability (e.g., decision support) for a particular domain.

The governance-flexibility spectrum

The next concept is the governance-flexibility spectrum. Let us first define what each of these terms means in the context of the topic.

• Governance: A system through which the organization directs and controls a domain.
• Flexibility: The degree of freedom provided to the domain for decision making.

Governance and flexibility have an inverse relationship. The more governed you are, the less flexible you become and vice-versa. Therefore, we could investigate the trade-off between governance and flexibility as a spectrum. The goal is to find that sweet spot. The following figure provides a view of the governance-flexibility spectrum.

The spectrum can be divided into three zones:

1. The zone of anarchy: At one end of the spectrum is the zone of anarchy where the need for governance is traded-off for flexibility. This zone has limited or no governance that implies greater or unlimited flexibility for the domains. As the name suggests, anarchy is not optimal. Organizations in this zone see a proliferation of technology and data, confusion in decision support, and a lack of coherence. Unfortunately, this zone is commonplace across large organizations.
2. The zone of rigidity: The other end of the spectrum is the zone of rigidity, where flexibility is traded-off for governance. This zone has extreme governance that stifles flexibility. As a result, organizations in this zone soon stagnate and cease to innovate. The central unit controls every minor or crucial decision, budget, and skill in this zone. Eventually, the zeal for innovation is lost without enough flexibility, and the organization stagnates.
3. The zone of governed-flexibility: Somewhere in between is the sweet spot of the zone of governed-flexibility. This zone maintains a healthy balance of governance yet provides enough flexibility for domains to innovate. In this zone, organizations thrive and innovate at a natural pace.

The Data Mesh and the Hub-Spoke pattern strive to exist in the zone of governed-flexibility.

Now that we have clarified the two concepts let us discuss how the two concepts fuse to form the conceptual architecture of the data mesh and the hub-spoke.

The Data Mesh Pattern

The first pattern that we want to discuss is the data mesh pattern. ThoughtWorks, a global technology company, popularized the concept of data mesh as a new paradigm that takes inspiration from modern distributed architecture and treats data as a product. I have borrowed some ideas from their concepts and adapted them to a more practical flavor.

The following figure depicts the conceptual architecture of a data mesh pattern.

The data mesh pattern has the following characteristics:

• First, the data mesh is holistically governed by an organizational data governance framework that provides a blueprint for governance.
• Each domain is as independent as any other domain in a data mesh pattern.
• Each domain has a domain node that fulfills the technical requirement.
• There is a governed and seamless mechanism of data sharing between each domain.
• Apart from data sharing, each domain has access to the sharable data catalog that every other domain can access.

It is important to note that in this pattern, each domain is as independent as any other domain in the data mesh.

The Hub-Spoke Pattern

The second pattern that we want to discuss is the hub-spoke pattern. The next figure depicts the conceptual architecture of a hub-spoke pattern.

The hub-spoke pattern has the following characteristics:

• First, an organizational data governance framework holistically governs the Hub-Spoke pattern. These are policies and principles that provide a blueprint for governance.
• Each domain has a that fulfills the technical requirement. It is an optional component for spoke domains. They can entirely depend on the hub domain for their decision support. However, the node for the hub domain is a must-have.
• The hub-spoke pattern has a central domain that acts as the hub, and one or many spokes are linked with the hub.
• The hub governs the spokes and ensures that the spoke domain follows the prescribed governance framework created by the hub domain.
• The data is shared between the hub and the spokes in a governed manner, including a shared data catalog.

The Hybrid Pattern

A common misconception is that organizations should either implement a Hub-Spoke pattern or a Data Mesh pattern. This thinking is impractical as organizations are not simplistic entities. Large organizations are evolving organically and are complex. Typically, a hybrid approach works the best. The subsequent figure shows the conceptual architecture of a hybrid approach.

An organization can have multiple domain networks. For example, there are two domain networks in the conceptual architecture depicted above. Domain network 1 is a data mesh, and domain 2 is hub-spoke. A shared domain is part of both networks. The shared domain in this example is part of both domain network 1 (data mesh) and domain network 2 (hub-spoke).

Another configuration of the hybrid pattern is where one of the domains of the data mesh is the hub domain of another hub-spoke network. The following figure depicts that example.

Now that the concept of the Data Mesh and Hub-Spoke is clear. Let us introduce the idea of the governance-flexibility spectrum to determine whether a domain should be part of a data mesh or a hub-spoke.

Choosing the placement of domain between Data Mesh and Hub-Spoke

Once the organizations have identified their domains, choosing between a hub-spoke or a data mesh depends on where the domain falls in the governance-flexibilityspectrum. The following figures show the placement of the domain in a data mesh or a hub-spoke pattern.

The placement of the domain depends on where it is placed on the governance-flexibility spectrum. Suppose the governance-flexibility trade-off is tilted towards lesser governance/greater flexibility. In that case, the domain is a suitable candidate for being a part of the data mesh. Inversely, suppose the governance-flexibility trade-off is tilted towards greater governance/lesser flexibility. In that case, the domain is a viable candidate for being a part of hub-spoke.

The degree of relative domain independence determines how one places the domain in the spectrum.

Five key parameters determine relative domain independence:

1. Functional Context: As mentioned earlier, the functional context implies the task that the domain is assigned to perform. The degree of autonomy the domain has for fulfilling its functional context determines its governance flexibility.
2. People and Skills: The degree of independence the domain has for hiring, skilling, and managing its people to fulfill its functional context.
3. Regulations: The degree of independence the domain has in adhering to internal or external regulations. For example, a regulatory reporting function in a bank is subject to enormous external regulations.
4. Operations: The degree of independence the domain has in controlling its operations and budgets to fulfill its functional context.
5. Technical Capabilities: The degree of independence the domain has in choosing, implementing, and managing its technology and related services to fulfill its functional context.

The way these parameters play out in an organization depends entirely on the organization’s context. However, suppose the domain has more independence in most of these parameters. In that case, it is an ideal candidate for a data mesh pattern. If not, it can be a better candidate for adopting the hub-spoke pattern.

This blog was the first part of the series of blogs on these patterns. The top three takeaways from this blog are as follows:

1. As organizations grow and become complex, data Mesh and Hub-Spoke macro patterns are required to fulfill analytical requirements.
2. The crux of the problem is managing the governance-flexibility trade-off.
3. A hybrid approach between Data Mesh and Hub-Spoke is a more practical approach towards implementation.

In the next part of this blog, we will focus on the logical architecture for these patterns and explore the components required to fruition these patterns.

Originally published at https://www.linkedin.com.