Multi-channel VideoStreaming Technology for Processing Based on Distributed Computing Platform

With the rapid development of the Internet, sensor network, and mobile Internet technology, a large number of data sets are continuously generated in the form of streaming in various application fields. At the same time, the processing of stream data has drawn more and more attention because of its application in different situations. To meet the urgent need for the processing of streaming data, there are many computing engines on the market, such as Spark and Flink. Traditional data-flow analysis has many problems in the process of streaming data, such as the high time delay, weak extensibility, and bad adaptability. In order to improve them, people use Kafka as the intermediate-cache and the computing engine that uses Spark Structured Streaming as the streaming data to realize the effective process of multichannel flow data. Considering the present demand for low latency and high flux that are analyzed by urban management in traffic video, processing the data of multichannel traffic video on distributed computing platform realized the tracking and search on motor vehicles, non-motor vehicles, and pedestrians of multiple roads and intersections.


Introduction
With the rapid development of the Internet, sensor network, and mobile Internet technology, a large number of data sets are continuously generated in the form of streaming in various application fields.
Due to its wide applications in various fields, such as analysis of real-time transactions, analysis of Internet of Things data, intelligent advertising, log processing, and analysis of measurement, convective data processing has drawn more and more attention. In order to meet the needs of applications, frameworks of streaming data processing have emerged, such as Apache Flink [1], Apache Spark [2], and Apache Storm [3]. Using these big data processing platforms to change the processing mode of traditional streaming data, such as video streaming, is conducive to making some applications better deal with the challenges brought by the era of big data.
In recent years, with the popularization of network technology, the original scattered video resources are being prompted towards the direction of unified and centralized processing. The increasingly intensive deployment of cameras in industries such as smart cities means that new video sources will increase at every moment. Changes of video resolutions from 360p, 720p to 1080p, and even of 4K show that video information is getting larger and larger. These increasing video data not only brings the demand for video storage but also poses a challenge to the video processing technology [4].Large-scale data, diverse data sources, and application requirements for video timeliness demand efficient technology for processing a large amount of multi-channel videos.
A large amount of video data needs to be processed in real-time to provide large-scale and real-time video analysis solutions. The following challenges are faced: (1) How to select a feasible systematical design to realize a feasible computing architecture oriented to the multi-channel video stream to meet the real-time requirements, while taking both the huge video stream and network communication cost of large-scale control terminals into account; (2)How to build a platform with good scalability and flexibility; (3) How to make full use of distributed resources. In this work, we put forward a feasible multi-channel video streaming technology for processing based on a distributed computing platform by using Spark structured flow and Kafka. And by dealing with traffic video streams in the field of urban management, we can make the tracking and search on motor vehicles, non-motor vehicles, and pedestrians of multiple roads and intersections. Moreover, we can obtain real-time traffic flow and forecast the congestion situation of these intersections to realize urban automation management further.

Related work
Many kinds of research have been conducted on multi-channel video stream processing. [5][6][7].
First, cloud computing can traditionally provide low-cost and flexible computing resources with almost unlimited capacity, so some traditional focuses are to use cloud computing to solve the problem 3 of multi-channel video processing. However, the video stream is basically used as the basic scheduling unit, and video processing capacity and physical resources are highly coupled [5]. Because every task of video stream processing will result in huge resource consumption of a server, the traditional scheduling scheme of the coarse-grained task will lead to a lot of resource fragmentation and underutilization. The video analysis system based on the Hadoop computing framework used by Zhao et al. [6] first stores the video to HDFS and then uses MapReduce for distributed processing. Streaming [8] and Kafka cache as a solution to process video streaming. However, Spark Streaming uses the microbatch processing mode, where the data flow is divided into microbatches at a specified time interval and then the data is batched for processing, mainly for bounded batch data processing.
But a real flow should be continuous and have no boundaries [9]. Spark Structured Streaming [10], which eliminates the concept of the batch, can truly support continuous stream processing and can be processed immediately as soon as data items arrive. Compared with other Streaming computing frameworks, its outstanding advantage is that it can realize fault-tolerant recovery and seamless integration of data analysis, machine learning, and other tools in an efficient way.
In this paper, we mainly use Spark Structured Streaming and Kafka message-oriented middleware to process multi-channel traffic video and conduct real-time detections of vehicle flow and a pedestrian on traffic roads, so as to provide a reliable basis for traffic management, improve the work efficiency of traffic management and maintain good traffic order. At the same time, the platform has obvious advantages in extensibility and fault tolerance.

System Architecture
For high flux streaming data, clustering is the preferred way to ensure it. For data segmentation, balanced task allocation, cluster communication, cluster resource management, and other functions, distributed big data processing frameworks, such as Spark, Hadoop, and Flink can be used. They provide a general distributed parallel computing platform for data processing and greatly reduce the complexity of system development [11]. It can also directly increase the computing resources to improve the processing performance and computing scale of videos. Calculating based on storage, the Spark is faster than the Hadoop that calculates based on the file system and can directly use Spark Standalone mode to manage the cluster in a unified way, reducing the difficulty of the deployment of the Spark in the cluster, making it easier to use to add new computing resources, supporting the Spark Structured Streaming in using advanced DAG scheduler, the query optimizer and physical execution engine and also realizing the high performance of batch and flow data processing. Spark also supports batch processing, interactive queries, machine learning, and graph computing, and has a complete ecosystem that can significantly reduce system update and maintenance costs. So we use Spark as the basic resource management and task execution framework. The architecture of the whole system is shown in Fig.1. Lightweight distributed Kafka message queue is selected as stream data cache in memory, and Zookeeper is used to coordinate the Kafka cluster, so as to realize load balance and fault recovery of the Kafka cluster. Collectors of the data flow will store data collected by different cameras into different partitions of the same Topic.
The Processor of data flow is composed of Spark as a resource management cluster, which is divided into Master Node as task allocation and resource coordination Node and Worker Node task execution Node. The Master will allocate different partitions of the same topic according to the use of each Worker's usage of resources, and then each Worker will continuously read data from the specified partition. As long as the data reach the Kafka, Spark Structured Streaming immediately reads data, and then the user Trigger (Trigger) reads the interval of the data stream into a chart, which is to convert the data stream data item in the Tab of a line. Next, use SparkSQL execution engine Stream Execution to deal with each line of data flow analysis and optimization, and the results will be updated to the DataStorage for storage.

Traffic video stream processing
In the development of the smart city, the increase of camera deployment can timely analyze data such as detection and tracking of motor vehicles, non-motor vehicles, and pedestrians and traffic congestion from multi-channel video stream data, which plays an important role in urban management decision.
However, the large traffic flow, multiple sources, and high timeliness requirements of traffic video data pose great challenges to the load balancing, scalability, and data cache of the data processing system. Therefore, we analyze and process traffic video data for this problem. The specific process is as follows.

Stream data acquisition
Traffic video data collection is carried out by each server to get data from the designated IP cameras.
Then the video is parsed and is stored in the Kafka message queue. Apache Kafka provides high-throughput data processing capability and supports the process of stateful data flow and low-latency distributed queries. However, the Kafka message queue has certain performance advantages in processing text-type data. So we must convert continuous video data into a series of video frames and then encapsulate it into JSON format. The specific data format is shown in Fig.2.

Fig.2 Data encapsulation format
Since different cameras may correspond to different specifications, such as encoding method, resolution, number of frames per second, and position information generated by frames, we must identify the source of each frame data for further analysis. So we introduce CameraID as the identifier for the different cameras.
TimeStamp is the time when the video collector processes the video frame. In the process of sending the video to Kafka, there may be a delay. TimeStamp is used to ensure that the video processor can restore the original video stream data when processing the video.
FrameData is the information of video frames. We use VideoCapture in the OpenCV video processing library to decode the video stream and convert it into a continuous Mat frame structure, which is the basic frame structure in OpenCV. In order to simplify Kafka's storage and data acquisition, the data resolution (e. g. 640x480) was pre-adjusted uniformly. FrameData is the base64-encoded string that is converted from the Mat byte array for that frame after adjusting sizes.
Since the Mat needs to keep the details of the frames (that is, rows, columns, and types) constant, then

Stream data processing
At present, the mainstream distributed streaming data patterns is mainly divided into two categories: one is the microbatch processing system that divides streaming data into tiny batches for processing, and the other is the streamlined flow data processing. Both models have effective fault tolerance, one-time semantics, and analysis of batch and stream data [11].
In the streamlined flow data processing model, the next batch processing can be carried out only when the processed data items are ready without forming the whole batch processing. Therefore, the model implements low latency. Apache Flink implements a pipelined flow processing model to support a true native flow processing engine. But this also means that it is difficult to achieve fault tolerance without affecting throughput because for each record we need to trace and checkpoints after processing. But state management is easy because long-running processes can easily maintain the required state.
In the batch model, time intervals that are predefined split the input stream into microbatches.
Spark Streaming and Spark Structured Streaming [9] process the flow data in this way. They split the stream data into distributed data sets (RDD), which allows users who use the same API to switch between batches and streams. Because this model has to cache a batch of data before a batch operation can take place, the latency of the task is high. But it is convenient to provide fault tolerance because it is essentially a batch process and throughput is high and because the processing and checking points will be done in one set of records. However, this will cost some delay and efficient state management will be a maintenance challenge.
Flow processing is a powerful tool, and all kinds of flow computing platforms have their own advantages. However, many streaming system platforms such as Flink which needs to consider a variety of complex API semantics. For example, the user needs to specify window mode, trigger mode, and trigger refinement mode for each operation. Considering engineering implementations, we prefer the expressions that are easy to use, operate, and integrate into an application's Spark Structured Streaming using the incremental query model to simplify an application. And the main aim is to support end-to-end real-time applications that integrate flows with batch processing and interaction analysis. Moreover, the structured code generation engine circulating through Spark SQL achieves high performance and can exceed Apache Flink by up to 2 times [12]. In the traffic video stream processing environment, it is mainly responsible for taking the encapsulated frame data out of the memory Kafka to track vehicles and pedestrians and predict congestion. The specific data flow diagram is shown below.  Fig.5 The processing effect picture Fig.6 The specific data flow diagram

Summary
Real-time data flow analysis of multi-channel video massive data streams requires high capacity and throughput and data processing engines. We built a collector component of a video stream by using Spark Structured Streaming, which can effectively process multi-channel traffic video streams. When processing video streams, we found that different applications have different needs in delay and performance. For example, the streaming video display can receive a certain delay as a buffer and have no strict requirement of time delay. But when processing warning applications of video analysis, we found that it has higher requirements on time. So we will make different programming models for different needs of different applications in the next step.