Events Sourcing and Command Query Responsibility Segregation Based Fast Data Architecture ()
1. Introduction
The proliferation of data has caused a lot of concern for some companies since the advent of digital science. Those companies often need these data in the implementation of a decision-making strategy. It often happens that the data, because they originate from various sources, they represent a real challenge for the companies. In order to efficiently exploit the data with a very low latency time, certain real-time architectures have emerged. These architectures are much more used in the management of raw Big Data generated from various sources. These Big Data are characterized by their volume, variety and velocity; which means that their storage and operation require capacities that surpass those of traditional computer systems [1] .
At the origin of big data, the lambda architecture was proposed for the work of integration and exploitation of massive data [2] . This architecture is still widely used in Big Data industrialization activities in companies and organizations. However, real-time application requirements, particularly with the advent of the Internet of Things, have led to the proposal of smack architectures as the first viable industrial approach [3] . Thus, the lambda and smack architectures are intended to be references in the development of applications exploiting Big Data in particular, data integration applications.
However, all these proposed so-called existing architectures proposed have the disadvantage of not taking into account application development activities as well as the deployment of Machine Learning solutions. They are therefore incomplete for the creation of an integrated and coherent enterprise architecture. This is why, in this paper, we propose a so-called Payi architecture that addresses all these limits. From a technical point of view, the Payi architecture is based on Command Query Responsibility Segregation (CQRS) and Event Sourcing approaches, integrating Software Engineering, Data Engineering and Machine Learning model deployment activities simultaneously.
The first is to present a literature review of CQRS and Event Sourcing approaches. Then, this will then lead to the presentation of the new proposed architecture. And finally we end with a conclusion.
2. Existing Architectures
2.1. Lambda Architecture
At the origin of the industrialization of Big Data, the lambda architecture was proposed. Figure 1 below gives the structure of the lambda architecture:
![]()
Figure 1. Lambda architecture (Source: [4] ).
The lambda architecture is composed of four (4) parts namely the data source, the batch layer, the streaming layer and the data presentation layer.
The data source: this notion is used as a parameter for queries and concerns structured, unstructured and semi-structured data. It comes in various forms and makes it possible to determine the data to which the request must relate [5] .
The batch layer: This layer has the role of storing the constantly growing and immutable basic data in a file system such as the HDFS. It also pre-calculates batch views of distributed data using the MapReduce function of this layer. Batch views are commonly used to respond to incoming requests with low read latency [6] [7] .
The streaming layer: its role is to route the data to the data processor. Then, it processes them in real time and produces a result within a short time [8] . The output of this processing is passed to the service layer where this data is displayed to the end user as part of a web application, dashboard, report or event used by another system [9] .
The presentation layer: it consists of frontend components as well as micro-services in order to serve the frontend. This layer is implemented in a pluggable manner so that the non-functional requirement of extensibility is satisfied [10] .
The specificity of this basic architecture is that it separates batch data processing from real-time processing.
In the quest for better architecture, Miguel et al. proposed an improved version of the Lambda architecture. Their approach is to standardize and simplify data extraction. To do this, they propose to insert a data ingestion layer between the data source and the batch and real-time data integration layers [2] . In this same vision, Tarik Hachad et al. also proposed a new Big Data architecture, based on the lambda architecture, used in the deployment of Machine Learning models. In their architecture, the data source is replaced by the new data to be used for predictions thanks to an already trained machine learning model. Their proposal was used to detect the level of attention of students be it in a batch context as well as in a real time application [3] .
2.2. The SMACK Architecture
In the professional context of Fast Data, an architecture based on Scala technologies such as Spark, Mesos, Akka, Cassandra and Kafka (SMACK architecture) has been very popular in companies. The following Figure 2 shows the SMACK architecture:
The SMACK architecture is composed of 4 layers namely the data source, the data propagation layer (Kafka), the processing layer (Spark) and the data storage layer (Cassandra). These four (04) layers use two additional technologies to accelerate the distribution of real-time data. These are the Akka actor-model programming paradigm and the Mesos machine cluster management technology.
Spark: is a processing engine within the Big Data architecture. It performs
![]()
Figure 2. SMACK architecture (Source: [11] ).
analytical work on real-time data. This engine offers flexibility from a development perspective and is available on scala, Java, R, Python and SQL. This engine offers the infrastructure and operation of the worker in a program [12] . It can also be used as a data science tool, capable of handling large datasets and performing operations on them. It works with resilient distributed datasets (RDD), which provides fault tolerance, efficiency, speed, and in-memory data storage [13] . RDDs enforce immutability and have no negative effects of interfering parallel running tasks. When he this one runs on a cluster, a Spark driver program delegates work as tasks to its subsidiary worker nodes. This allows for scalability and is perfect for working in the cloud environment. Spark can work with different types of cluster managers, but in the SMACK stack.
My bones: is used to manage and coordinate cluster resources. It extracts all the computing resources (CPU, memory, storage) from the different machines. It is not only the core of distributed systems, but it is easy to build and runs efficiently on distributed systems. It is elastic and fault tolerant. Mesos orchestrates all components and manages computational resources [14] . Mesos is based on the principles of the Linux kernel and is the basis of three environments (Aurora Apache, Chronos and Marathon) [15] . This tool allows you to manage and organize an infinite number of machines quickly and reliably.
Akka: a library of the SMACK architecture based on the actor model. It represents a tool for developing distributed, fault-tolerant and message-driven applications. Akka uses the Java Virtual Machine (JVM) as its runtime environment, which allows development in Java and Scala programming languages. This serves as the basis for Akka in the actor model, which divides a program into simultaneous actors who are exclusively exchanging information with each other through messages [16] .
Cassandra: a non-relational and distributed database manager. This handler is part of NoSQL databases [17] [18] [19] . It is scalable and fault tolerant for large amounts of data. Unlike relational databases, the database is column-oriented, which is particularly advantageous for applications that work primarily with column-based queries such as aggregations of individual columns. In the SMACK architecture, Cassandra is used to store operational data and can be used as a data source for the presentation layer [20] .
Kafka: a stream processing platform it represents the data ingestion point in the SMACK architecture [21] . It is responsible for publishing and subscribing to messages. Kafka takes data from applications and streams to process it inside the stack. Kafka inspects the incoming data volume to partition it and distribute it across nodes. It is packed with several features like Automatic Fault Tolerance, high performance in distributed messages, partitioning and distribution between cluster nodes. It is also independent on the data pipeline, supports a large number of users, and processes large amounts of data [22] .
The SMACK architecture essentially aims to standardize the separate batch and real-time layers in the Lambda architecture. This architecture is more of an alternative to the lambda architecture whose role was to become a standard for real-time or near-real-time Big Data applications.
3. CQRS and Event Sourcing
Events Sourcing and CQRS are methodological approaches that have been the subject of several works in the field of Software Engineering in general, and in the development of micro-services in particular. Event sourcing is a methodology in which each action generates events which are stored in an appropriate database called an event database. As for the data, they are separately in a dedicated database. In this approach, all actions on the data are not done directly via the user interface, but rather by analyzing the event matches by an event engine.
Called event header whose function is to update the database according to the generated event [23] . Command and Query Responsibility Segregation, abbreviated as CQRS, is a methodology that separates write and read operations. In this approach, the databases in which the data is written are dissociated from the databases on which the read requests are made. For a writing database, multiple read databases are generated based on business rules. And it is the latter that are used for read operations [24] [25] .
In practice, event sourcing and CQRS are complementary and can be used to develop more robust micro-services capable of processing large volumes of data. It is in this context that FANSHA et al. implemented a microservices architecture based on the CQRS model, Events sourcing on OpenAPI, a pilot API and a pilot Event. In order to evaluate the performance of the proposed architecture, they carried out some tests according to the response time, the error rate and the throughput. Indeed, this test proved that micro-services with CQRS and Event Sourcing models have much faster performance than those of the pilot API, i.e. 3.7%. Moreover, it appears that the communication between the services has no effect on the error rate and the throughput [26] . KLJUN et al. in the quest of implementing a micro-service, made an in-depth analysis of the architecture of micro-services and the CQRS model. Thus, this inspection allowed the implementation of a micro-service based on the extension of the KumuluzEE framework. This extension allowed the integration of the Axon framework in the CQRS and Event Sourcing (ES) model [27] . Akre et al. have developed a CQRS and ES system that supports both event sourcing and order sourcing. Thus, they implemented multiple mechanisms of logarithmic reduction (pruning) and persistence. This made it possible to test and measure the performance of various CQRS+ES configurations. By doing so, they better understood the design principles and performance of CQRS + ES systems [28] . As for Vlček, Lukas explores the area of Enterprise Application Integration (EAI) models in combination with the CQRS architectural model. This is to specify the prerequisites for using a combination of CQRS and the design pattern of mediation or federated integration of EAI. The result of this work provides prerequisites in the appropriate use of model combinations by a given company when deciding on the final form of the EAI [29] .
4. PAYI Architecture
The essential components of the Payi architecture in Figure 3 are:
· Commands: these are basically the actions of inserting, modifying, deleting and reading data or the results of data processing;
· Bus: these are messaging brokers allowing the transport and distribution of data and events;
· Handlers, Engines and Generators: these are the data and event processing engines;
· Stores: these are the technologies for storing data and events;
· Platforms: these are technologies specific to the exploitation and analysis of data.
The architecture has the advantage of taking into account all activities related to data; from the development of data storage applications to the development of learning machine. It can therefore be used for software engineering as well as for data engineering, data science and machine learning engineering activities.
One of the advantages of the Payi architecture lies in the fact that it allows the development of batch solutions and real-time or near-real-time solutions with the same technological bricks. Indeed, the architecture unifies the development of batch and real-time solutions both at the software engineering level and at the data engineering level.
Another advantage of the Payi architecture is that it allows the processing of extreme voluminous data and the development of applications using Big Data. Indeed, event and data storage solutions can be distributed file systems (HDFS, etc.), object storage systems (S3, MinIO, etc.) or NoSQL technologies (Elasticsearche, MongoDB, etc.). Also, data and message buses can be distributed messaging brokers (Kafka, etc.) and event processing engines can be distributed computing engines (Spark, Flink, etc.).
The last advantage of the Payi architecture consists in simplifying Software Engineering thanks to Data Engineering. Indeed, all the backend consisting in
![]()
Figure 3. CQRS and events sourcing based fast data architecture.
the processing of events, they can be automated by Data Engineering technologies. Therefore, the work of Software Engineering is essentially limited to the development of the frontend of applications
5. Conclusion
We have proposed an architecture that unifies software engineering and data engineering. In reality, it is an architecture that integrates management applications with decision-making applications. Its strength is that it can be used both for development for classic applications and for real-time applications such as those of the IoT.
This architecture has the particularity of being suitable for the development of big data and fast data applications in a software engineering context. Its flexibility makes it possible to use several different big data technologies, unlike the SMACK architecture which focuses on scala technology (Spark, Mesos, Akka, Cassandra Kafka).