Cognite AS, a Norwegian software company, recently launched the Open Industrial Data portfolio/cognite. It composes a live stream of real industrial data from Aker BP’s Valhall oil platform in the North Sea. The data comes from the first stage compressor and consists of 1:

  • timeseries
  • maintenance history
  • process and instrumentation diagrams

The first stage compressor (23-KA-9101) is an electrically-driven, fixed-speed centrifugal compressor which receives gas from the separators at approximately 3 barg pressure. The gas is compressed to approximately 12 barg and flows through a pair of discharge coolers (23-HA-9114/9115) before entering the second stage of compression. 2

“What is the typical difference between the input and output pressure of the compressor?”

The objective is to calculate and visualize the pressure difference across the compressor at any point in time within the last 90 days. Pressure measurements from seven sensors installed around the compressor is available at a one-minute frequency. Every day new data is loaded and old date is deleted.


An application was developed that allows to calculate and visualize the pressure difference between any two sensors. In addition, the location of selected sensors is highlighted in a flowchart.

This exercise serves to demonstrate how a reliable data processing architecture based on microservices can be built using open-source technology.

How to use the app

The application is hosted on Google Cloud Platform and the complete code is published on GitHub. Give the app a try by following this link or by navigatying to the address with your web browser. The video below shows a 30-second demo of the application:


The data is made available through the Cognite Python SDK. The architecture is composed of three components: the data pipeline, database and the dashboard.

All components are hosted on an E2-Medium VM instance on Google Cloud Platform. The data pipeline and dashboard are deployed as separate microservices in two Docker containers. The database runs directly on the virtual machine.

Data pipeline

The data pipeline is defined as a Directed Acyclic Graph or DAG defined in Apache Airflow and is executed daily at midnight UTC. The graph consists of four consecutive steps:

  1. A new batch of daily data is queried through the Cognite Python SDK
  2. The data is written to a PostgreSQL database
  3. Data older than 90 days is deleted
  4. Disk space occupied by the deleted data is released


The PostgreSQL relational database serves the dashboard with up to 90 days of sensor data. Data is stored in tidy (long) format in a single table with a composite primary key:

CREATE TABLE compressor_pressure (
  asset_id BIGINT NOT NULL,
  sensor_name VARCHAR (25) NOT NULL,
  pressure REAL,
  PRIMARY KEY (timestamp, asset_id)


The dashboard consists of a Shiny Dashboard served by four Shiny modules. A JSON file allows to configure the entire app.

  "dashboard": {
    "theme": "flat_red",
    "title": "Pressure dashboard"
  "columns": {
    "left": {
      "width": 9
    "right": {
      "width": 3
  "dropdown": {
    "info": {
      "icon": "info",
      "circle": "FALSE",
      "size": "lg",
      "label": "About",
      "width": "20vmax",
      "right": "TRUE"

Care was taken to design the dashboard to support a wide range of screen sizes.


Full documentation.


Distributed under the MIT License
Copyright (c) 2020 Denis Gontcharov

  1. Cognite, Open Industrial Data portfolio/cognite, about 

  2. Cognite, open Industrial Data portfolio/cognite, system description