I am referring Information Fabric to systems delivering comprehensive capabilities to enable dynamic, real time data services.

Based on the technology trends, personal experiences and demands from our customers  I consider below four components as essential for building new Modern Information fabric.

1. Real time Insight built in system of engagements (hybrid transactional and analytical systems).
2. Serverless Applications. –Micro-services and serverless architecture would become the essential part of information fabric.  This would promotes quickly,  creating scalable data services modifying action sequences to meet the evolving demands of ever changing business requirements.

3. Scale by Design, Run Anywhere. –Business today can acquire petabytes of data in a short span and are dealing with the dilemma of security vs. elasticity of the cloud. While the industry responds the need of the hour is to build data services that can scale by design and run anywhere.
4. Predictive Analytics, Machine Learning, and AI. – While above three component helps to meet the data services “must have”,  this one is “differentiating “ factor. Machine Learning and AI capabilities would differentiate two business competing in the same space.

In this series, I would share my experience building each one of them starting with how to build an event driven enterprise that leverages HTAP for a real-time data services.

Part 1– Real-time Insights using hybrid transactional, analytical processing (HTAP) systems.

The need for real time insight both on structured, unstructured data and instant customer gratification is no brainer these days and had become essential to any businesses. If you need a quick reference, please visit.

Top industry use cases for real-time analytics

Over last decade, the approach used by an enterprise for real time Insight has evolved. Depending on the maturity, the execution differs. For the most enterprise, they kicked their journey through the process of creating an operational data store (transaction replication) and building a data warehouse.

This approach though helped to provide access to event(transaction) however any actionable insight still required substantial time lag due to the ETL/ELT processes, and that doesn’t provide a business impact with the “now” generation customers.  Some IT vendor tried to bridge the gap with tools like Change data capture, In-memory Analytics etc. . Technology savvy enterprises embarked on to a journey of optimising this with real time streaming analytics and later converging to a lambda architecture for real time and batch processing.

While some enterprises today managed to improve or influence customer experience through the above mechanism, the success of the above projects and ROI has been challenging mainly due to the complexity it brings and skill set. Most of the successful enterprises in this endeavour, are the one who take an application centric or use-cases based approach.

From an enterprise perspective, the challenge with the above approach is that still they are different applications, and apart from the IT challenges (plumbing) and complexity of the system it doesn’t provide a mechanism to have a comprehensive enterprise data services.

Introducing the new IBM Project EventStore –

IBM EventStore is an platform to provide a next generation transactional and hybrid transactional, analytical processing (HTAP) systems. By “next generation”, I am referring to modern, new, operational systems that are built for tomorrow’s systems of engagement, as opposed to traditional systems of record.
The primary goal of the event store is to provide a platform, where real-time analytics and transaction processing techniques are woven together in the same application and optimise the execution of transactional processes (In-Process HTAP). It also simplifies and bridge the gap with traditional deployment by providing a platform for Point-of-decision HTAP whereby the transaction processing and analytic aspects is segregated into distinct, independently designed applications but without moving the data. Such pattern allows advanced analytics to be performed on “live” transactional data, something that is very hard to achieve in traditional architectures.

Project EventStore is a unique platform, proactive effort from IBM to meet the next generation data-services requirement for a modern data fabric without added complexity. With benchmark proving million events/sec per node enterprise can cater to the new real-time insight at a low cost.

Project EventStore is exciting as this is based on the best of the open source technologies available today and thus enables wider adoption.

References :

IBM Project EventStore

The Lambda Architecture, simplified