The problem of events detecting and classifying in the continuous video stream of information and telecommunication security systems is solved. As a basic algorithm, it is proposed to use the ensemble of deep neural networks - convolutional networks and feedback networks, in particular, for classification and annotation, which makes it possible to recognize abnormal emergency situations. A training set of abnormal situations was collected, various architectures of deep neural networks were trained and tested. It is shown that the use of the indRNN layers achieves up to 70% when recognitions multi-class events in the video stream. The new is strengthening of classification estimation of a video segment by extracting the keywords from the automatic annotation. The developed software package can be implemented in the integrated security system of abnormal situations recognition in real time.