Query-aware adaptive windowing for spatiotemporal complex video event processing for internet of multimedia things

Abstract

With the evolution of the Internet of Things (IoT), there is an exponential rise in sensor devices that are deployed ubiquitously. Due to the extensive usage of IoT applications in smart cities, smart homes, self-driving cars, and social media, there is enormous growth in multimedia data streams like videos and images. We are now transitioning to an era of the Internet of Multimedia Things (IoMT), where unstructured data like videos are continuously streamed from visual sensors like CCTV cameras and smartphones. Video data is highly expressive and has traditionally been very difficult for a machine to interpret. Middleware systems such as Complex Event Processing (CEP) mine patterns from data streams and send notifications to users in a timely fashion. Current CEP systems have inherent limitations to mine event patterns from video streams due to their unstructured data model, lack of expressive query language, and resource intensiveness. This work introduces VidCEP, a data-driven, distributed, on the fly, near real-time complex event matching framework for video streams with five key contributions: 1) Expressive Video Event Query Language- Current event query languages are highly focused on temporal reasoning. A SQL-like declarative Video Event Query Language (VEQL) is proposed which enables state-based spatiotemporal video event matching without needing to focus on low-level video features. The VEQL enables the creation of robust spatiotemporal operators using a hybrid approach. 2) Structured Video Stream Representation- The work introduces Video Event Knowledge Graph (VEKG), a knowledge graph driven model of video data streams using an ensemble of deep learning models and VEQL operators. VEKG creates a semantic knowledge representation of video data by modeling video objects as nodes and their relationship interaction as edges over time and space. 3) Query and State-aware stream summarization- Objects coexist across multiple frames, leading to the creation of redundant nodes and edges at different time instances that result in high memory usage and increased matching time. The work introduces two novel state and multi-query based spatiotemporal graph summarizations of VEKG streams- Time Aggregated Graph (VEKG-TAG) and Event Aggregated Graph (VEKG-EAG) for an efficient state-based event matching. 4) Resource constraint distributed content-driven windowing- An adaptive and content-driven windowing technique VID-WIN is proposed to improve Quality of Service (QoS). VID-WIN instances are deployed over edge and cloud to perform efficient state-based video event matching. The VID-WIN adopts resource and query-aware runtime optimization strategies to improve the CEP matching performance under limited available resources and application requirements. 5) VidCEP Complex Event Processing Framework: The above techniques are integrated into Video Complex Event Processing framework (VidCEP). A prototype of the framework is developed which performs query-based video event pattern matching in a distributed setting. The VidCEP capability is demonstrated using a real-work traffic estimation service for OpenStreetMap. A total of 18 event rules are defined from the traffic management and activity recognition domain. Extensive experiments have been performed across 20 datasets consisting of more than 3900 video clips to evaluate the performance and efficacy of proposed techniques. Results of this study show that VidCEP achieves a throughput of approximately 70 frames per second (fps) for five parallel streams (at 17fps) with sub-second matching latency. The system successfully detects different spatiotemporal video event patterns with good F-scores (0.51-0.90). The optimization techniques proposed in the thesis improves ~5X search time, ~2.3X throughput, and ~99% bandwidth savings.