Skip to content

History application

Experimental feature (added in 2022.1.71)

Background

History application

The history application consists of a scheduled job, which triggers batched transfers of historical data from the engine data store to the historical data store. You can configure the periodicity of the scheduler, HISTORY_SCHEDULER_CRON_EXPRESSION, as well as the batch sizes, HISTORY_MOVER_BATCH_LIMIT.

Note that transferring data means also removing them from the engine's data store. You can, however, choose to keep the most recent events and states by configuring their retention period ("expiry"), HISTORY_MOVER_EXPIRY_DURATION.

Setup scenarios

Use of the history application is optional. The following service configurations are possible:

  1. No history application running, meaning historical data are not moved anywhere and remain available from within the engine's data store
  2. The history application running against the same database schema as the engine
  3. The history application running against its own database schema

In order to deploy the history application, the following conditions must be met:

  1. The history application must use the same database as the engine
  2. Both history and engine applications must have the same platform version
  3. A history database user must exist

See example setup.

Configuration

Docker image

The history application is provided as a docker image which includes a default configuration. The history application uses the same versioning as the rest of the platform (including, most importantly, the engine).

API

The history application runs on port 12010 by default. The history admin endpoints are exposed on port 12710 by default. The history management endpoints are exposed on port 12410 by default.

Database user

A dedicated database user for the history application should be created and configured for the application to start. If you choose to operate the history application within its own database schema, make sure to provide the history database user with GRANT CREATE ON DATABASE privileges.

Environment variables

Deprecated environment variables are documented here.

Engine

These are the environment variables for the engine application, needed to enable the history application setup (disabled by default).

Environment variable Description Default value
ENGINE_DB_HISTORY_USER History store user null
ENGINE_DB_HISTORY_PASSWORD History store password null
ENGINE_DB_HISTORY_SCHEMA History store schema history

History

The following environment variables can be configured for the history application:

Environment variable Description Default value
HISTORY_DEV_MODE History running mode - indicates whether application runs in development mode false
HISTORY_STARTUP_HEALTH_CHECK_ATTEMPTS History startup - number of check attempts 10
HISTORY_STARTUP_HEALTH_CHECK_RETRY_DURATION History startup - time before retrying a check PT20S
HISTORY_SPRING_ACTUATOR_EXPOSE History monitoring - exposed Spring actuator endpoints info,health,service-health,metrics,prometheusmetrics
HISTORY_PORT History API port 12010
HISTORY_ADMIN_HOST History Admin API host 127.0.0.1
HISTORY_ADMIN_PORT History Admin API port 12710
HISTORY_MANAGEMENT_HOST History Management API host 127.0.0.1
HISTORY_MANAGEMENT_PORT History Management API port 12410
HISTORY_MOVER_BATCH_LIMIT History mover - batch limit 5000
HISTORY_MOVER_EXPIRY_DURATION History mover - expiry duration PT24H
HISTORY_SCHEDULER_CRON_EXPRESSION History scheduler - CRON expression * */15 * * * *
HISTORY_SCHEDULER_ENABLED History scheduler - scheduled process initially enabled true

Database

Environment variable Description Default value
HISTORY_DB_URL History DB - URL jdbc:postgresql://localhost:5432/platform
HISTORY_DB_USER History DB - user history
HISTORY_DB_PASSWORD History DB - password null
HISTORY_DB_SCHEMA History DB - schema history
HISTORY_DB_ENGINE_SCHEMA History DB - Engine schema noumena
HISTORY_DB_SSL History DB - SSL false
HISTORY_DB_POOL_MAXIMUM_POOL_SIZE History DB - maximum size of the connection pool 3
HISTORY_DB_POOL_MAXIMUM_CONNECTION_LIFETIME History DB - maximum lifetime of a connection in the pool PT40S

Logging

Environment variable Description Default value Example value
HISTORY_LOG_CONFIG Logging configuration (file path) null classpath:/logback-json.xml
HISTORY_ROOT_LOG_LEVEL Logging level (root) INFO
HISTORY_LOG_LEVEL Logging level - History application INFO
HISTORY_REQUEST_LOG_LEVEL Logging level - Http requests WARN
HISTORY_SPRING_LOG_LEVEL Logging level - Spring INFO

Swagger UI

The following environment variables can be configured to set up the history application's Swagger UI:

Environment variable Description Default value
SWAGGER_HISTORY_ADMIN_URL Swagger UI – URL to connect to the History Admin API http://${HISTORY_ADMIN_HOST}:${HISTORY_ADMIN_PORT}/
SWAGGER_HISTORY_MANAGEMENT_URL Swagger UI - URL to connect to the History Management API http://${HISTORY_MANAGEMENT_HOST}:${HISTORY_MANAGEMENT_PORT}
SWAGGER_SECURITY_AUTH_URL Swagger UI - URL to connect to the Authentication API http://localhost:11000/realms/noumena
SWAGGER_SECURITY_CLIENT_ID Swagger UI - IAM client ID nm-platform-service-client
SWAGGER_SECURITY_CLIENT_SECRET Swagger UI - IAM client secret 87ff12ca-cf29-4719-bda8-c92faa78e3c4

Example setup

Database user setup

The following SQL script creates a dedicated database user for the history application.

create role history login password <pwd>;
alter role history createrole noinherit;
-- needed if history data are to be stored in separate DB schema:
grant create on database mydatabase to history;

Docker configuration

The engine will provide all database-level privileges needed for the history database user. For this to happen, make sure to configure the variables that are related to the engine, i.e. ENGINE_DB_HISTORY_%.

Make sure to always deploy the same application versions of both the engine and the history docker images. If you don't, the history application will refuse to start, or it will report failing health checks and stop transferring data to its data store.

# engine application (listed below is only the configuration specific to the history application)
engine:
  image: ghcr.io/noumenadigital/packages/engine:latest
  depends_on:
    - postgres_db
  # ...
  environment:
    # ...
    ENGINE_DB_URL: jdbc:postgresql://postgres_db/platform
    ENGINE_DB_SCHEMA: engine-schema
    # enable history application
    ENGINE_DB_HISTORY_USER: history
    ENGINE_DB_HISTORY_PASSWORD: <pwd>
    ENGINE_DB_HISTORY_SCHEMA: history-schema

# history application
history:
  image: ghcr.io/noumenadigital/packages/history:latest
  depends_on:
    - postgres_db
    - engine
  ports:
    - "${HISTORY_PORT:-12010}:12010"
    - "${HISTORY_ADMIN_PORT:-12710}:12710"
    - "${HISTORY_MANAGEMENT_PORT:-12410}:12410"
  environment:
    HISTORY_ADMIN_HOST: 0.0.0.0
    HISTORY_MANAGEMENT_HOST: 0.0.0.0
    HISTORY_DB_URL: jdbc:postgresql://postgres_db/platform
    HISTORY_DB_USER: history
    HISTORY_DB_PASSWORD: <pwd>
    HISTORY_DB_SCHEMA: history-schema
    HISTORY_DB_ENGINE_SCHEMA: engine-schema

For further configuration options, see environment variables (history).

A complete docker compose file example can be found here.

Health check

Once the history application is deployed, you can test its healthcheck endpoint with:

curl localhost:12010/actuator/health

Swagger UI

You can explore the APIs provided by the history application via Swagger UI, available at http://localhost:12010/swagger-ui/.

History states (API)

New feature (added in 2023.1.94)

The history application API provides an endpoint to retrieve protocol states for a given protocol from the history data store. Note that the history data endpoint is not secured and is available on the admin host/port.

You can try it out as follows:

curl localhost:12710/admin/history/streams/states/{protocolId}

History management (API)

New feature (added in 2023.2.26)

The history application management API provides endpoints that allow to control the application and its data. It is available on the management host/port.

You can try it out as follows:

curl -X GET    'localhost:12410/management/history/scheduler/status'
curl -X PUT    'localhost:12410/management/history/scheduler/status' -H 'Content-Type: application/json' -d '{"enabled": "false"}'

curl -X POST   'localhost:12410/management/history/data/move?batchLimit=100&synchronous=true'
curl -X DELETE 'localhost:12410/management/history/data'