History application

Background

History application

The history application consists of a scheduled job, which triggers batched transfers of historical data from the engine data store to the historical data store. You can configure the periodicity of the scheduler, HISTORY_SCHEDULER_CRON_EXPRESSION, as well as the batch sizes, HISTORY_MOVER_BATCH_LIMIT.

Note that transferring data means also removing them from the engine's data store. You can, however, choose to keep the most recent events and states by configuring their retention period ("expiry"), HISTORY_MOVER_EXPIRY_DURATION.

Setup scenarios

Use of the history application is optional. The following service configurations are possible:

1. No history application running, meaning historical data are not moved anywhere and remain available from within the engine's data store
2. The history application running against the same database schema as the engine
3. The history application running against its own database schema

In order to deploy the history application, the following conditions must be met:

1. The history application must use the same database as the engine
2. Both history and engine applications must have the same platform version
3. A history database user must exist

See example setup.

Configuration

Docker image

The history application is provided as a docker image which includes a default configuration. The history application uses the same versioning as the rest of the platform (including, most importantly, the engine).

API

The history application runs on port 12010 by default. The history admin endpoints are exposed on port 12710 by default. The history management endpoints are exposed on port 12410 by default.

Database user

A dedicated database user for the history application should be created and configured for the application to start. If you choose to operate the history application within its own database schema, make sure to provide the history database user with GRANT CREATE ON DATABASE privileges.

Environment variables

Deprecated environment variables are documented here.

Engine

These are the environment variables for the engine application, needed to enable the history application setup (disabled by default).

Environment variable	Description	Default value
ENGINE_DB_HISTORY_USER	History store user	null
ENGINE_DB_HISTORY_PASSWORD	History store password	null
ENGINE_DB_HISTORY_SCHEMA	History store schema	history

History

The following environment variables can be configured for the history application:

Environment variable	Description	Default value
HISTORY_DEV_MODE	History running mode - indicates whether application runs in development mode	false
HISTORY_STARTUP_HEALTH_CHECK_ATTEMPTS	History startup - number of check attempts	10
HISTORY_STARTUP_HEALTH_CHECK_RETRY_DURATION	History startup - time before retrying a check	PT20S
HISTORY_SPRING_ACTUATOR_EXPOSE	History monitoring - exposed Spring actuator endpoints	info,health,service-health,metrics,prometheusmetrics
HISTORY_PORT	History API port	12010
HISTORY_ADMIN_HOST	History Admin API host	127.0.0.1
HISTORY_ADMIN_PORT	History Admin API port	12710
HISTORY_MANAGEMENT_HOST	History Management API host	127.0.0.1
HISTORY_MANAGEMENT_PORT	History Management API port	12410
HISTORY_MOVER_BATCH_LIMIT	History mover - batch limit	5000
HISTORY_MOVER_EXPIRY_DURATION	History mover - expiry duration	PT24H
HISTORY_SCHEDULER_CRON_EXPRESSION	History scheduler - CRON expression	* */15 * * * *
HISTORY_SCHEDULER_ENABLED	History scheduler - scheduled process initially enabled	true

Database

Environment variable	Description	Default value
HISTORY_DB_URL	History DB - URL	jdbc:postgresql://localhost:5432/platform
HISTORY_DB_USER	History DB - user	history
HISTORY_DB_PASSWORD	History DB - password	null
HISTORY_DB_SCHEMA	History DB - schema	history
HISTORY_DB_ENGINE_SCHEMA	History DB - Engine schema	noumena
HISTORY_DB_SSL	History DB - SSL	false
HISTORY_DB_POOL_MAXIMUM_POOL_SIZE	History DB - maximum size of the connection pool	3
HISTORY_DB_POOL_MAXIMUM_CONNECTION_LIFETIME	History DB - maximum lifetime of a connection in the pool	PT30M

Logging

Environment variable	Description	Default value	Example value
HISTORY_LOG_CONFIG	Logging configuration (file path)	null	classpath:/logback-json.xml
HISTORY_ROOT_LOG_LEVEL	Logging level (root)	INFO
HISTORY_LOG_LEVEL	Logging level - History application	INFO
HISTORY_REQUEST_LOG_LEVEL	Logging level - Http requests	WARN
HISTORY_SPRING_LOG_LEVEL	Logging level - Spring	INFO

Swagger UI

The following environment variables can be configured to set up the history application's Swagger UI:

Environment variable	Description	Default value
SWAGGER_HISTORY_ADMIN_URL	Swagger UI – URL to connect to the History Admin API	http://${HISTORY_ADMIN_HOST}:${HISTORY_ADMIN_PORT}/
SWAGGER_HISTORY_MANAGEMENT_URL	Swagger UI - URL to connect to the History Management API	http://${HISTORY_MANAGEMENT_HOST}:${HISTORY_MANAGEMENT_PORT}
SWAGGER_SECURITY_AUTH_URL	Swagger UI - URL to connect to the Authentication API	http://localhost:11000/realms/noumena
SWAGGER_SECURITY_CLIENT_ID	Swagger UI - IAM client ID	nm-platform-service-client
SWAGGER_SECURITY_CLIENT_SECRET	Swagger UI - IAM client secret	87ff12ca-cf29-4719-bda8-c92faa78e3c4

Example setup

Database user setup

The following SQL script creates a dedicated database user for the history application.

create role history login password <pwd>;
alter role history createrole noinherit;
-- needed if history data are to be stored in separate DB schema:
grant create on database mydatabase to history;

Docker configuration

The engine will provide all database-level privileges needed for the history database user. For this to happen, make sure to configure the variables that are related to the engine, i.e. ENGINE_DB_HISTORY_%.

Make sure to always deploy the same application versions of both the engine and the history docker images. If you don't, the history application will refuse to start, or it will report failing health checks and stop transferring data to its data store.

# engine application (listed below is only the configuration specific to the history application)
engine:
  image: ghcr.io/noumenadigital/packages/engine:latest
  depends_on:
    - postgres_db
  # ...
  environment:
    # ...
    ENGINE_DB_URL: jdbc:postgresql://postgres_db/platform
    ENGINE_DB_SCHEMA: engine-schema
    # enable history application
    ENGINE_DB_HISTORY_USER: history
    ENGINE_DB_HISTORY_PASSWORD: <pwd>
    ENGINE_DB_HISTORY_SCHEMA: history-schema

# history application
history:
  image: ghcr.io/noumenadigital/packages/history:latest
  depends_on:
    - postgres_db
    - engine
  ports:
    - "${HISTORY_PORT:-12010}:12010"
    - "${HISTORY_ADMIN_PORT:-12710}:12710"
    - "${HISTORY_MANAGEMENT_PORT:-12410}:12410"
  environment:
    HISTORY_ADMIN_HOST: 0.0.0.0
    HISTORY_MANAGEMENT_HOST: 0.0.0.0
    HISTORY_DB_URL: jdbc:postgresql://postgres_db/platform
    HISTORY_DB_USER: history
    HISTORY_DB_PASSWORD: <pwd>
    HISTORY_DB_SCHEMA: history-schema
    HISTORY_DB_ENGINE_SCHEMA: engine-schema

For further configuration options, see environment variables (history).

A complete docker compose file example can be found here.

Health check

Once the history application is deployed, you can test its healthcheck endpoint with:

curl localhost:12010/actuator/health

Swagger UI

You can explore the APIs provided by the history application via Swagger UI, available at http://localhost:12010/swagger-ui/.

History states (API)

The history application API provides an endpoint to retrieve protocol states for a given protocol from the history data store. Note that the history data endpoint is not secured and is available on the admin host/port.

You can try it out as follows:

curl localhost:12710/admin/history/streams/states/{protocolId}

History management (API)

The history application management API provides endpoints that allow to control the application and its data. It is available on the management host/port.

You can try it out as follows:

curl -X GET    'localhost:12410/management/history/scheduler/status'
curl -X PUT    'localhost:12410/management/history/scheduler/status' -H 'Content-Type: application/json' -d '{"enabled": "false"}'

curl -X POST   'localhost:12410/management/history/data/move?batchLimit=100&synchronous=true'
curl -X DELETE 'localhost:12410/management/history/data'