3.4 NexusDS – Flexible Data Stream Processing 85
Figure 3.3:NexusDS Layer Architecture
is a distributed stream processing middleware targeting the requirements levied in Section 2.4, thus providing enhanced support for complex application scenarios which require specialized techniques.
The main purpose of NexusDS is to sustain aflexible and seamless extensibility of the system by means of application or domain specific functionality. This especially refers to the integra-tion of operaintegra-tional extensions in terms of operators, and functional extensions in terms of servicesinto NexusDS. NexusDS consists of four layers as depicted in Figure 3.3. Each layer is built upon its underlaying layers. The architecture combines an adaptive service platform and an extensible operator framework. Operators (displayed as dashed boxes) are push-based and are used to integrate custom data-processing into the system by means of the push-paradigm.
In contrast, services (displayed as solid boxes) follow a request-response paradigm and are used to implement customized system behavior in terms of service functionality by means of the pull-paradigm. Starting from the bottom of thefigure, a brief explanation of the single layers is provided to get an overall picture of the system. Thereby, the lowest layer is mandatory and is part of each NexusDS node providing basic NexusDS functionality. The higher the layers in Figure 3.3, the more specific they are.
3.4.1 Communication and Monitoring Layer
TheCommunication and Monitoring Layerrelies on communication and execution primitives.
More sophisticated communication mechanisms are defined within this layer. The
Communi-3.4 NexusDS – Flexible Data Stream Processing 87 cation and Monitoring Layer inherits the flexibility and scalability of P2P networks [12] and is inspired by the idea of advanced resource handling and monitoring mechanisms from grid computing systems [54, 75]. This layer contains two fundamental services which are part of each NexusDS node: The Monitoring Service (MS) and the Service Manager (SM). They constitute the basic functionality that each NexusDS node has. The Monitoring Service is the component that monitors the system and keeps track of the computing nodes and their specific characteristics as well as the runtime behavior of executed operators. The Service Manager per-mits service publication and makes services available within the system. Multiple instances of both services—the Service Manager and the Monitoring Service—are created and distributed across multiple NexusDS nodes to increase availability and to reduce possible bottlenecks. An instance of the Monitoring Service is started on each NexusDS node to collect runtime statis-tics of the respective node. Besides this, the service also provides statistical data by means of historical data.
The Service Manager is described in more detail in Section 3.5.1 and the Monitoring Service with its related infrastructure is shown in Section 3.5.2.
3.4.2 Nexus Core Layer
TheNexus Core Layercontains the core components, i. e. operators and services of NexusDS.
Services are instantiated and distributed among NexusDS nodes and its life cycle is managed by the Service Manager. Not every participating node provides all kinds of services, however.
This can be adjusted to meet certain requirements such as avoiding a single point of failure or restricting service execution to a certain administrative domain. The core components pro-vide fundamental data stream processing functionality of a DSPS. The Core Graph Service accepts a SP graph in NPGM2format and takes the necessary actions for SP graph deployment.
The Core Graph Service fragments and deploys the SP graph to meet certain QoS require-ments. The fragmentation and deployment process is presented in Chapter 6. Each SP graph fragment is deployed on an Operator Execution Service which controls the execution of the operators in the assigned SP graph fragment. Details on the Operator Execution Service are shown in Section 3.5.4. TheCore Operatorsrepresent the basic operators available in NexusDS and correspond to DB operators commonly known, such as Selection and Projection. These operators—actually the physical operators—are stored in the Operator Repository Service. The Operator Repository Service provides a query interface to query operators. The repository also counts sources and sinks as operators, having either only outputs or inputs. As with opera-tors, services also need a place to be stored which is represented by the Service Repository Service (SRS). Like the Operator Repository Service, the Service Repository Service also stores the service implementations and provides an interface to query services. Finally, the Access Control Service takes care to enforce the access conditions defined in the system. The Access
2A NPGM SP graph is a directed graph which consists of interconnected operators, forming a processing pipeline.
The NPGM and NEGM SP graphs are described in the next chapter.
Control Service is needed to check if all necessary permissions are valid to start processing of the requested data streams.
Details on the individual components are discussed later in this chapter.
3.4.3 Nexus Domain Extensions Layer
The Nexus Domain Extension Layer builds on the Nexus Core Layer and clusters services and operators for a certain domain of interest. There may exist many of these Nexus Domain Extension Layers in parallel, each providing functionality tailored to a particular domain. The idea of the Nexus Domain Extension Layers is to group functionality needed by many applica-tions and share this functionality with them thus avoiding redundancy. For example we can imagine many visualization applications performing e. g. volume visualization. For the domain of volume visualization appropriate operators are implemented and published via the Operator Repository Service mentioned beforehand. Analogously, a query service—we call this service the volume visualization service—is implemented and made available providing a domain spe-cific language which is tailored to this domain. The volume visualization service translates the visual queries to SP graphs which the Core Graph Service can process. The service eliminates unnecessary complexity for the inquirer and helps focusing on relevant facts related to this domain.
3.4.4 Nexus Application Extensions Layer
Analogously to the Nexus Domain Extensions Layer theNexus Application Extensions Layer enables services or operators that are specific for a single application. Thus, parts of the ap-plication logic which is not already covered by domain-specific extensions can be outsourced to NexusDS. Consequently, this outsourced functionality is highly application-dependent and not treated as a domain extension. A reason for such an application-specific extension is that a machine the application is currently running on is not able to process certain tasks in a satisfac-tory manner, thus making outsourcing of the functionality under concern mandasatisfac-tory. Again thinking of the visualization scenario, current mobile clients still have shortcomings regarding processing power and would rapidly cause an overload situation. E. g., the rendering process is not reasonably executable on a mobile device with limited capabilities. In such a case it is highly recommendable to outsource application-specific parts to a more powerful machine and just receive the final outcome. In this particular case, an additional encoding operator which takes the rendered pictures from the rendering operator, transforms them into an image and provides a video stream of scaled and compressed images (in order to save bandwidth) rather than the full-sized results.