With the rapidly increasing number of mobile devices equipped with GPS sensors and mo-bile Internet connections, the use of data stream processing is increasing in many application areas. This chapter has presented a security framework and its integration into NexusDS . The security framework deals with the requirements of modern applications relying on the data stream processing paradigm. Thereby, the security framework proposes different security con-trol patterns, i. e. AC policies, PC policies, and GC policies, which can be assigned to different system entities. The defined security control patterns are exploited to ensure a safe process-ing of sensible data. This is achieved by augmentprocess-ing SP graphs with the correspondprocess-ing AC policies, PC policies, and GC policies. By the proposed security framework it is possible to adjust the density of information that is going to be processed as well as to limit access to data.
The following chapter highlights M-TOP. M-TOP performs the deployment of SP graphs by exploiting the respective annotations.
173
6
Stream Processing Graph Deployment
I
n the past decade, DSPSs gained great attention. They are well suited to address the chal-lenges in processing high-volume and real-time data. In particular, DSPSs such as [2, 40, 57, 78]have been in the focus due to their inherent ability to distribute load among different partici-pants. This is a key feature for scalability, as it avoids bottlenecks when processing potentially unbound data streams. In general, these systems provide a declarative query language and support the continuous distributed processing of incoming data streams. In some DSPSs, the application developer creates SP graphs directly. A topic of interest is how to distribute a SP graph across different computing nodes w.r.t. certain objectives, e. g. to avoid single points of failure or guarantee a certain Quality of Service (QoS). This is important as the initial dis-tribution has a big impact on the run time behavior of a DSPS. An inappropriate initial SP graph distribution degrades execution and may lead to big overhead during run time, e. g. by migrating operators with heavy state.
In this chapter, the multi-target operator placement (TOP) of SP graphs is presented. M-TOP is a QoS-aware multi-target operator placement algorithm. M-M-TOP applies to the operator placement problem in data stream processing environments and considers a set of application-specific QoS targets for operator placement. Provided there is a set of nodes to place a set of operators, M-TOP searches for an operator placement that fulfills the specified QoS tar-gets. The M-TOP heuristics thereby aim at eliminating placements that do not lead to suitable solutions.
For the operator placement process different approaches are feasible. We assume application developers to create corresponding SP graphs. Our hands-on experience has shown that devel-opers are interested in explicitly providing the QoS targets to influence directly the SP graph distribution and so the operator placement process. By this, the application QoS requirements are best reflected to the process of operator placement in DSPSs.
Sections 6.1 and 6.2 provide a problem description for the operator placement in a distributed environment. Then, in Section 6.3, related work is presented and discussed. Section 6.4
intro-duces the multi-target operator placement problem which is the focus of M-TOP and is pre-sented in detail in Section 6.5. The sections 6.6 and 6.7 present details of the mapping step of M-TOP. In Section 6.8 we discuss the experimental evaluations performed to show the effec-tiveness of M-TOP. Finally, in Section 6.10, this chapter is concludes by a short summary.
6.1 Problem Description
As argued in Section 3.2 there is an adaptation problem in DSPSs which makes it difficult for applications—relying on the same processing paradigm, i. e. push-based stream processing—to integrate custom functionality seamlessly and moreover to influence the deployment process.
Tight application integration in DSPSs is beneficial because data processing is tightened and the actual processing happens close the actual data. This improves data processing and allows reuse of already existing components, which avoids isolated solutions and redundancy.
The application of an interactive visualization application as the one presented in Section 2.3.1 visualizes the result of a SP graph operating on data streams and exploiting the func-tionality of DSPSs. The DSPS topology is utterly heterogeneous with devices ranging from desktop computers to mobile devices. The SP graph abrasively consists of three processing steps: filter, map and render. The filter operator is reusable by existing DSPSs, whereas the map and render operators implement custom logic and need to be implemented by the specific application developer. However, in contrast to the custom map operator, the render operator has additional specific requirements in order to run properly, e. g. a GPU. Hence, operators be-ing deployed might have inherent restrictions that must be considered for operator placement when selecting devices going to execute the specific operator.
Beside the inherent restrictions, operators might also have restrictions which are application-specific (defined by their respective developers) or even user-application-specific. Assuming customers are classified in amwith gold, bmwith silver, and cmwith bronze status1, each one has its own priorities when executing the application:
• Customer amwants the bandwidth utilization maximized and requests a minimum of 10 MBit/ssince no additional accounting will occur. Furthermore latency must be mini-mized but should not be more than500 ms.
• Customer bmwants the bandwidth utilization maximize requesting a minimum of 1 MBit/s. Latency should be minimized not exceeding 2 s. However, costs should be minimized not exceeding1 cost unit.
• Customer cmwants just to maximize bandwidth utilization but requests a minimum of 500 kBit/s, his applications needs to run properly. Utilization costs should not occur at all.
1Here gold stands forall inclusive, silver for100% accounting on QoS utilization exceeding a certain threshold otherwise 50% accounting, and bronze for100% accounting on QoS utilization.