The Data Mesh and the Hub-Spoke: A Macro Pattern for Scaling Analytics

How can they ensure that their decision support systems are governed appropriately yet provide them the flexibility to innovate at their own pace?

Two macro patterns of Data Mesh and Hub-Spoke have emerged recently. However, these patterns are nascent and have layered complexities that need to be addressed systematically to adopt these patterns.

The Two Concepts

Firstly, I would want to break this complex problem into manageable parts. There are two conceptual building blocks for managing this trade-off:

  1. The governance-flexibility spectrum

The structure of a domain

The first concept to discuss is a domain. The following figure depicts a domain concept from an organizational point of view.

  • Subunits: It is not uncommon for organizations to have many subunits. The subunits may have differing levels of independence from the central unit. This degree of autonomy is based on the organizational structure and its culture.
  • The second category comprises an independent business unit within the same organization.
  • The third category is an intra-organizational department.

A domain is defined as any logical grouping of organizational units to fulfill a functional context subjected to organizational constraints.

  • The functional context implies the task that the domain is assigned to perform. The functional context is the raison d’être for the domain.
  • The organizational constraints can be business constrained imposed on the domain like regulations, people and skills, operational dependencies.
  • A product group that focuses on creating a specific product or service.
  • A subsidiary of a parent company.

Nodes fulfill a specific technical capability (e.g., decision support) for a particular domain.

The governance-flexibility spectrum

The next concept is the governance-flexibility spectrum. Let us first define what each of these terms means in the context of the topic.

  • Flexibility: The degree of freedom provided to the domain for decision making.
  1. The zone of rigidity: The other end of the spectrum is the zone of rigidity, where flexibility is traded-off for governance. This zone has extreme governance that stifles flexibility. As a result, organizations in this zone soon stagnate and cease to innovate. The central unit controls every minor or crucial decision, budget, and skill in this zone. Eventually, the zeal for innovation is lost without enough flexibility, and the organization stagnates.
  2. The zone of governed-flexibility: Somewhere in between is the sweet spot of the zone of governed-flexibility. This zone maintains a healthy balance of governance yet provides enough flexibility for domains to innovate. In this zone, organizations thrive and innovate at a natural pace.

The Data Mesh and the Hub-Spoke pattern strive to exist in the zone of governed-flexibility.

Now that we have clarified the two concepts let us discuss how the two concepts fuse to form the conceptual architecture of the data mesh and the hub-spoke.

The Data Mesh Pattern

The first pattern that we want to discuss is the data mesh pattern. ThoughtWorks, a global technology company, popularized the concept of data mesh as a new paradigm that takes inspiration from modern distributed architecture and treats data as a product. I have borrowed some ideas from their concepts and adapted them to a more practical flavor.

  • Each domain is as independent as any other domain in a data mesh pattern.
  • Each domain has a domain node that fulfills the technical requirement.
  • There is a governed and seamless mechanism of data sharing between each domain.
  • Apart from data sharing, each domain has access to the sharable data catalog that every other domain can access.

It is important to note that in this pattern, each domain is as independent as any other domain in the data mesh.

The Hub-Spoke Pattern

The second pattern that we want to discuss is the hub-spoke pattern. The next figure depicts the conceptual architecture of a hub-spoke pattern.

  • Each domain has a that fulfills the technical requirement. It is an optional component for spoke domains. They can entirely depend on the hub domain for their decision support. However, the node for the hub domain is a must-have.
  • The hub-spoke pattern has a central domain that acts as the hub, and one or many spokes are linked with the hub.
  • The hub governs the spokes and ensures that the spoke domain follows the prescribed governance framework created by the hub domain.
  • The data is shared between the hub and the spokes in a governed manner, including a shared data catalog.

The Hybrid Pattern

A common misconception is that organizations should either implement a Hub-Spoke pattern or a Data Mesh pattern. This thinking is impractical as organizations are not simplistic entities. Large organizations are evolving organically and are complex. Typically, a hybrid approach works the best. The subsequent figure shows the conceptual architecture of a hybrid approach.

Choosing the placement of domain between Data Mesh and Hub-Spoke

Once the organizations have identified their domains, choosing between a hub-spoke or a data mesh depends on where the domain falls in the governance-flexibilityspectrum. The following figures show the placement of the domain in a data mesh or a hub-spoke pattern.

The degree of relative domain independence determines how one places the domain in the spectrum.

Five key parameters determine relative domain independence:

  1. People and Skills: The degree of independence the domain has for hiring, skilling, and managing its people to fulfill its functional context.
  2. Regulations: The degree of independence the domain has in adhering to internal or external regulations. For example, a regulatory reporting function in a bank is subject to enormous external regulations.
  3. Operations: The degree of independence the domain has in controlling its operations and budgets to fulfill its functional context.
  4. Technical Capabilities: The degree of independence the domain has in choosing, implementing, and managing its technology and related services to fulfill its functional context.
  1. The crux of the problem is managing the governance-flexibility trade-off.
  2. A hybrid approach between Data Mesh and Hub-Spoke is a more practical approach towards implementation.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store