DEEP Open Catalogue: Massive Online Data Streams

This is a service to analyze online data streams in order to generate alerts in real-time. The core part is built as ML application using ML/DL techniques for modelling in co-function with underlying Intrusion Detection Systems (IDS) supervising traffic networks of compute centers. The service is running on TensorFlow backend. Further information on the package structure and the requirements can be found in the documentation in the git repository.

Data Science application ML/DL, NN, RNN
Deep Learning Framework(s) Tensorflow, Keras
Programming language Python
GPU version yes
CPU version yes
DEEP DS template yes
DEEP-Nextcloud access yes

Keywords: ML/DL, NN, RNN, time-series data, cyber-security, Tensorflow

DEEP-OC DockerHub image:

DEEP-OC Dockerfile:

Application source code:


The principle of the security/anomaly detection is proactive time-series prediction adopting artificial NNs to build prediction models capable to predict next step(s) in near future based on given current and past steps. The discrepancy between the prediction and the reality gives an indication of warning levels when supervising networks activities.

The challenge of the solution is also it aims to scalable edge technologies to support extensive data analysis and modelling as well as to improve the cyber-resilience by adopting an heuristic approach combining misuse detection in real-time with the building intelligence module using ML, NN and DL techniques.


The described workflow supposes usage of downloaded from DEEP Open Catalog Docker images, i.e. you need either docker or udocker tool.

1. Data preprocessing

ML/DL learns from data, then the first important thing is to have the data correctly set up.

1.1 Feature building and ML/DL data pool


The description of this step from massive online data through raw private datasets and consequently to ML/DL data using in-memory operations of Apache Spark will be available later due to raw data sensitiveness!

1.2 Prepare ML/DL data

Put your ML/DL dataset (i.e. already prepared time-series data file) in the ./data/ folder. The location of your data can be set also in the ./mods/ file. Currently, the accepted ML/DL format is either .tsv or .csv. You don’t have to use all features in your dataset, the selected features are specified in ./mods/

You can find an example of ML/DL dataformat here.

2. Train the model

Before training the model you can customize the default parameters of the configuration file ./mods/ This step is optional and training can be launched with the default configurarion parameters and still offers reasonably good results.

Once you have customized the configuration parameters, you can launch the train model using default configuration by running ./mods/models/ --method train [args ...].


Work-in-progress to be consistent with DEEP DS Template and DEEPaaS API. After training, the trained model is packed together with the model scaler and the model configuration in one .zip file located in the ./models/ folder.

3. Test the model

You can test the created model using default configuration by running ./mods/models/ --method test [args ...].


Fig. 1 Train and test on 6 month monitoring dataset. Blue=dataset, green=prediction on train dataset, red=prediction on test (unseen) dataset.


Fig. 2 Train and test on three day dataset for better visualisation (monitoring of two aspects simultaneously). Blue=dataset, green=prediction on train dataset, red=prediction on test (unseen) dataset.


Work-in-progress to be consistent with DEEP DS Template and DEEPaaS API.

Launching the full DEEPaas API

1. Prediction/inference through DEEPaaS

  • You can easily try the default configuration by start the container as:

    $ docker run -ti -p 5000:5000 deephdc/deep-oc-mods
  • Direct your web browser to

  • Go to POST /models/mods/predict , click Try it out button

  • Go to Data file, select some .tsv file containing observations like here.

  • Click Execute and get predicted values in JSON format.

The prediction using the created model goes through DEEPaaS API ./mods/models/ --method predict_data [args ...]


The model scaler and model configuration are required for prediction using the trained model. All available MODS models are packed in .zip with all three files.

2. DEEPaaS API functionality

To access this package’s complete functionality (both for training and predicting) through the DEEPaaS API you have to follow the instructions here: Run a module on DEEP Pilot Infrastructure