The Data Mesh and the Hub-Spoke: A Macro Pattern for Scaling Analytics

How can they ensure that their decision support systems are governed appropriately yet provide them the flexibility to innovate at their own pace?

The Two Concepts

  1. The structure of a domain
  2. The governance-flexibility spectrum

The structure of a domain

  • Central Unit: This is the unit that is at the organizational level. They may prescribe guidance that is expected to be followed by the subunits. For example, they may hold budgets distributed for various initiatives across the subunits. They may also have platforms that fulfill group-level requirements.
  • Subunits: It is not uncommon for organizations to have many subunits. The subunits may have differing levels of independence from the central unit. This degree of autonomy is based on the organizational structure and its culture.
  • The first category comprises a different organization (entity) within a group organization in the same or another geography.
  • The second category comprises an independent business unit within the same organization.
  • The third category is an intra-organizational department.

A domain is defined as any logical grouping of organizational units to fulfill a functional context subjected to organizational constraints.

  • The functional context implies the task that the domain is assigned to perform. The functional context is the raison d’être for the domain.
  • The organizational constraints can be business constrained imposed on the domain like regulations, people and skills, operational dependencies.
  • A department like marketing or sales focuses on a specific function within a business.
  • A product group that focuses on creating a specific product or service.
  • A subsidiary of a parent company.

Nodes fulfill a specific technical capability (e.g., decision support) for a particular domain.

The governance-flexibility spectrum

  • Governance: A system through which the organization directs and controls a domain.
  • Flexibility: The degree of freedom provided to the domain for decision making.
  1. The zone of anarchy: At one end of the spectrum is the zone of anarchy where the need for governance is traded-off for flexibility. This zone has limited or no governance that implies greater or unlimited flexibility for the domains. As the name suggests, anarchy is not optimal. Organizations in this zone see a proliferation of technology and data, confusion in decision support, and a lack of coherence. Unfortunately, this zone is commonplace across large organizations.
  2. The zone of rigidity: The other end of the spectrum is the zone of rigidity, where flexibility is traded-off for governance. This zone has extreme governance that stifles flexibility. As a result, organizations in this zone soon stagnate and cease to innovate. The central unit controls every minor or crucial decision, budget, and skill in this zone. Eventually, the zeal for innovation is lost without enough flexibility, and the organization stagnates.
  3. The zone of governed-flexibility: Somewhere in between is the sweet spot of the zone of governed-flexibility. This zone maintains a healthy balance of governance yet provides enough flexibility for domains to innovate. In this zone, organizations thrive and innovate at a natural pace.

The Data Mesh and the Hub-Spoke pattern strive to exist in the zone of governed-flexibility.

The Data Mesh Pattern

  • First, the data mesh is holistically governed by an organizational data governance framework that provides a blueprint for governance.
  • Each domain is as independent as any other domain in a data mesh pattern.
  • Each domain has a domain node that fulfills the technical requirement.
  • There is a governed and seamless mechanism of data sharing between each domain.
  • Apart from data sharing, each domain has access to the sharable data catalog that every other domain can access.

It is important to note that in this pattern, each domain is as independent as any other domain in the data mesh.

The Hub-Spoke Pattern

  • First, an organizational data governance framework holistically governs the Hub-Spoke pattern. These are policies and principles that provide a blueprint for governance.
  • Each domain has a that fulfills the technical requirement. It is an optional component for spoke domains. They can entirely depend on the hub domain for their decision support. However, the node for the hub domain is a must-have.
  • The hub-spoke pattern has a central domain that acts as the hub, and one or many spokes are linked with the hub.
  • The hub governs the spokes and ensures that the spoke domain follows the prescribed governance framework created by the hub domain.
  • The data is shared between the hub and the spokes in a governed manner, including a shared data catalog.

The Hybrid Pattern

Choosing the placement of domain between Data Mesh and Hub-Spoke

The degree of relative domain independence determines how one places the domain in the spectrum.

  1. Functional Context: As mentioned earlier, the functional context implies the task that the domain is assigned to perform. The degree of autonomy the domain has for fulfilling its functional context determines its governance flexibility.
  2. People and Skills: The degree of independence the domain has for hiring, skilling, and managing its people to fulfill its functional context.
  3. Regulations: The degree of independence the domain has in adhering to internal or external regulations. For example, a regulatory reporting function in a bank is subject to enormous external regulations.
  4. Operations: The degree of independence the domain has in controlling its operations and budgets to fulfill its functional context.
  5. Technical Capabilities: The degree of independence the domain has in choosing, implementing, and managing its technology and related services to fulfill its functional context.
  1. As organizations grow and become complex, data Mesh and Hub-Spoke macro patterns are required to fulfill analytical requirements.
  2. The crux of the problem is managing the governance-flexibility trade-off.
  3. A hybrid approach between Data Mesh and Hub-Spoke is a more practical approach towards implementation.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Pradeep Menon

Pradeep Menon

3.6K Followers

Creating impact through Technology | #CTO at #Microsoft| Data & AI Strategy | Cloud Computing | Design Thinking | Blogger | Public Speaker | Published Author