In the article, Making Data Lakehouse real…yet effective, the concept of Data Lakehouse was introduced. Its Architectural paradigm was discussed and the components that weave the Architecture together were explained.

In this follow-up article, the Data Lakehouse Architecture will be fruition using Microsoft Azure services.

This article will explain various Azure services that will fit into different components of the Data Lakehouse Architecture.

Data Lakehouse Conceptual Architecture

As a recap in the precursor article, the conceptual Data Lakehouse Architecture was introduced. The seven components that form the conceptual Data Lakehouse Architecture are as follows:

1. Data ingestion services: The components to get the data in.
2. …

Once upon a time, decisions were supported by decision support systems called as the Data Warehouses.

It neatly designed structured data into different schemas: stars, snowflakes, and normal forms.

Then came the 2010s. Three things culminated in a perfect storm:

1. The definition of data changed. There were more types of data. There were no good names for it. It was then called unstructured or semi-structured data.
2. Cloud computing started to make innovations. It was innovative and was making economic impacts. The cost of storing data decreased.
3. Processing capabilities rose. …

Artificial Intelligence (AI) is set to change the way the world works. It is the engine that fuels the digital transformations. Organizations and societies are optimistic about AI, its potential, and its ability to transform the world we live in.

Like the breaking of atoms can be harnessed to light cities or to destroy one so is AI’s promise marred by its perils.

It is time to harness AI technologies for their goodness and creates a framework for them to be responsible. …

Introduction

Today is the age of data. Every organization is becoming a “data company.” Today, if an organization is not thinking about its data as a strategic asset, then it has already missed the bus.

Data has evolved over the past decade. The rate of evolution has been exponential. But have the Data Architecture practices managed to keep up with the same pace?

McKinsey recently published an article that formulates the building of modern data architecture to drive innovation. The article explains the foundational shifts for modern data architecture.

Being a practitioner in this field, I found Mckinsey’s view interesting. …

There should no doubts in anyone’s mind about how Big Data and AI are fueling the next revolution. Data is the new oil, and AI refines the oil. The questions to ask is the following:

• How well are you placed to use data as a strategic asset?
• Are your technologies equipped to harness data effectively? At the right speed and in the proper manner?
• Are you able to reduce the data to insight turnover time and ensure that the right insight is available to the right stakeholders promptly?

Data Engineering has become vital for any organization that is serious about…

Is Artificial Intelligence (AI) the new elixir of all modern problems? Or is it a double-edged sword, sometimes destructive and at other times life-saving?

The fact is this: With the right framework, AI has the potential to be harnessed.

According to Gartner, around 37% of organizations are implementing some form of AI. Yet, according to a survey conducted by EY, only about 20% of firms are seeing have strategic AI capabilities. Very few organizations have successfully managed to harness the real power of AI to create meaningful impact.

How to harness AI? What should be the framework? The paper published…

Concept

Machine learning algorithms have a method of learning patterns from data. The method is intuitive. The model determines the underlying pattern from a given data set. This process is called training the model. The trained model is tested on another data set that it has not seen before. The goal is always to find the optimal model. The endeavor is to hit the sweet spot where the model performs satisfactorily on both training set and test set.

The test error is the mean error that occurred when the model on the new observation makes the prediction. This new observation is…

Evolution of Retail

Like any industry, retail has gone through its share of transformations. Now the exciting thing about retail is that its development has been symbiotic to consumer behavior. It had to adapt rapidly based on how consumer behavior changed from time to time. For example, between 1900–1940s, cars were not mainstream; the retail shop formats focused on catering to the neighborhood mom and pop shops.

However, as cars became mainstream, large format stores started opening as the consumers preferred living away from cities and drive. Fast forward 1990s-2000s, as the internet became mainstream, e-commerce rose into prominence. Companies like Alibaba and…