Data Fabric: Where are my data sources?

By Aditya Abeysinghe

In a previous article, I explained about *data management in multiple cloud environments. Multiclouds, as described in that article are used to distribute applications in multiple cloud environments. However, when applications are distributed in multiple clouds, monitoring, managing and providing a uniform flow of data is often time consuming, costly and requires additional resources. Therefore, a platform from which all these data can be integrated is useful. A data fabric is a data management model where all data endpoints of applications in multiple hosts can be integrated.

What advantages do data fabrics provide?

As I explained in a previous article about *Self-service integrations, they (Self-service integrations) enable users who are in non-technical teams to integrate apps into existing systems. With self-service integrations, these teams can access data and perform tasks they require with less time rather than requiring technical teams to provide access to data. However, when self-service apps are integrated into systems, a common issue is the difficulty in monitoring these apps and the data shared between these apps. To overcome such issues, a data fabric could be used to monitor all apps and their systems in one common model. This ensures faster data access leading to high returns, less cost in monitoring and the ability to analyze multiple systems at lesser time.

Data engineering is where data is analyzed and decisions are made based on them. In hybrid cloud environments where multiple datasets need to be analyzed, the common method is to use several analysis tools to analyze data in each cloud. However, in many cases where data are distributed in several clouds, such analyses could cause errors in analytics and predictions. Also, modern businesses require real-time data processing and task automation to serve customers with their queries. Data integration is a key when real-time data needs to be analyzed rather than analyzing stored data by data analysts. For such needs, a data fabric could provide the integration platform to provide faster data access for processing tools.

When services and data are distributed across clouds, governing policies and rules for clouds is often a difficult procedure. Policies for each cloud should be separately enforced in a multi cloud or hybrid cloud. Compliance issues are another type of issue when multiple clouds are used, as complying rules of one cloud may be different from other clouds. However, with a data fabric, governance of data, policies and other issues can be reduced as one platform is used to manage all distributed apps.

Features of a data fabric

A data fabric observes existing data as well as new data and then suggests alternatives for decisions taken on data. This enables decision makers to focus on more innovative tasks rather than analyzing data. Semantic data and active metadata management, embedded AI (Artificial Intelligence) in data management and monitoring both hot and cold data are some technologies in data fabrics that provide an effective as well as faster decision making process.

Knowledge graphs are one of the core concepts of data fabrics. Multiple data sources are integrated to form a topology of data graphs which have interlinks with other entities such as objects and events. Therefore, data, relationships between data, events which trigger certain scenarios etc. are analyzed. The semantic layer in the knowledge graph makes it easy to justify decisions formed using links.

The data transition between build, test, deploy and monitoring is one single model for all sources in a data fabric due to its intelligent integration methods. This is also enhanced by the automation of processes which automatically enables data transition between build, test, deploy and monitoring after certain conditions. With automatic transitions between these modes, many repetitive tasks done by operational teams could be automated.

*Data Management in Multiple Clouds Article: https://www.elanka.com.au/data-management-strategies-in-multiple-clouds-by-aditya-abeysinghe/

*Self-service integrations Article: https://www.elanka.com.au/building-your-own-app-self-service-application-integration-by-aditya-abeysinghe/

Image courtesy: https://itprotoday.com/