History application

Background

History application

The history application consists of a scheduled job, which triggers batched transfers of historical data from the engine data store to the historical data store. You can configure the periodicity of the scheduler, HISTORY_SCHEDULER_CRON_EXPRESSION, as well as the batch sizes, HISTORY_MOVER_BATCH_LIMIT.

Note that transferring data means also removing them from the engine's data store. You can, however, choose to keep the most recent events and states by configuring their retention period ("expiry"), HISTORY_MOVER_EXPIRY_DURATION.

Setup scenarios

Use of the history application is optional. The following service configurations are possible:

No history application running, meaning historical data are not moved anywhere and remain available from within the engine's data store
The history application running against the same database schema as the engine
The history application running against its own database schema

In order to deploy the history application, the following conditions must be met:

The history application must use the same database as the engine
Both history and engine applications must have the same platform version
A history database user must exist

See example setup.

Configuration

Docker image

The history application is provided as a docker image which includes a default configuration. The history application uses the same versioning as the rest of the platform (including, most importantly, the engine).

API

The history application runs on port 12010 by default. The history admin endpoints are exposed on port 12710 by default. The history management endpoints are exposed on port 12410 by default.

Database user

A dedicated database user for the history application should be created and configured for the application to start. If you choose to operate the history application within its own database schema, make sure to provide the history database user with GRANT CREATE ON DATABASE privileges.

Refer to the example setup for more details.

Environment variables

Deprecated environment variables are documented here.

Engine

These are the environment variables for the engine application, needed to enable the history application setup (disabled by default).

Environment variable	Description	Default value
ENGINE_DB_HISTORY_USER	History store user	n/a (optional value)
ENGINE_DB_HISTORY_SCHEMA	History store schema	`history`

History

The following environment variables can be configured for the history application:

Environment variable	Description	Default value
HISTORY_DEV_MODE	History running mode - indicates whether application runs in development mode	`false`
HISTORY_STARTUP_HEALTH_CHECK_ATTEMPTS	History startup - number of check attempts	`10`
HISTORY_STARTUP_HEALTH_CHECK_RETRY_DURATION	History startup - time before retrying a check	`PT20S`
HISTORY_SPRING_ACTUATOR_EXPOSE	History monitoring - exposed Spring actuator endpoints	`info,health,service-health,metrics,prometheusmetrics`
HISTORY_PORT	History API port	`12010`
HISTORY_ADMIN_HOST	History Admin API host	`127.0.0.1`
HISTORY_ADMIN_PORT	History Admin API port	`12710`
HISTORY_MANAGEMENT_HOST	History Management API host	`127.0.0.1`
HISTORY_MANAGEMENT_PORT	History Management API port	`12410`
HISTORY_MOVER_BATCH_LIMIT	History mover - batch limit	`5000`
HISTORY_MOVER_EXPIRY_DURATION	History mover - expiry duration	`PT24H`
HISTORY_SCHEDULER_CRON_EXPRESSION	History scheduler - CRON expression	`0 /15 * * *` (CRON expressions)
HISTORY_SCHEDULER_ENABLED	History scheduler - scheduled process initially enabled	`true`

Database

Environment variable	Description	Default value
HISTORY_DB_URL	History DB - URL	`jdbc:postgresql://localhost:5432/platform`
HISTORY_DB_USER	History DB - user	`history`
HISTORY_DB_PASSWORD	History DB - password	n/a (mandatory value)
HISTORY_DB_SCHEMA	History DB - schema	`history`
HISTORY_DB_ENGINE_SCHEMA	History DB - Engine schema	`noumena`
HISTORY_DB_SSL	History DB - SSL	`false`
HISTORY_DB_POOL_MAXIMUM_POOL_SIZE	History DB - maximum size of the connection pool	`3`
HISTORY_DB_POOL_MAXIMUM_CONNECTION_LIFETIME	History DB - maximum lifetime of a connection in the pool	`PT30M`

Logging

Environment variable	Description	Default value	Example value
HISTORY_LOG_CONFIG	Logging configuration (file path)	n/a (optional value)	`classpath:/logback-json.xml`
HISTORY_ROOT_LOG_LEVEL	Logging level (root)	`INFO`
HISTORY_LOG_LEVEL	Logging level - History application	`INFO`
HISTORY_REQUEST_LOG_LEVEL	Logging level - Http requests	`WARN`
HISTORY_SPRING_LOG_LEVEL	Logging level - Spring	`INFO`

Swagger UI

The following environment variables can be configured to set up the history application's Swagger UI:

Environment variable	Description	Default value
SWAGGER_HISTORY_ADMIN_URL	Swagger UI – URL to connect to the History Admin API	`http://${HISTORY_ADMIN_HOST}:${HISTORY_ADMIN_PORT}/`
SWAGGER_HISTORY_MANAGEMENT_URL	Swagger UI - URL to connect to the History Management API	`http://${HISTORY_MANAGEMENT_HOST}:${HISTORY_MANAGEMENT_PORT}`
SWAGGER_SECURITY_AUTH_URL	Swagger UI - URL to connect to the Authentication API	`http://localhost:11000/realms/noumena`
SWAGGER_SECURITY_CLIENT_ID	Swagger UI - IAM client ID	`nm-platform-service-client`
SWAGGER_SECURITY_CLIENT_SECRET	Swagger UI - IAM client secret	`87ff12ca-cf29-4719-bda8-c92faa78e3c4`

Example setup

Database user setup

The following SQL script creates a dedicated database user for the history application.

-- the use of 'noinherit' is regarded good practice, history user privileges are obtained by explicit 'set role'
create role history login password <pwd> noinherit;
grant create on database mydatabase to history;

Docker configuration

The engine will provide all database-level privileges needed for the history database user. For this to happen, make sure to configure the variables that are related to the engine, i.e. ENGINE_DB_HISTORY_%.

Make sure to always deploy the same application versions of both the engine and the history docker images. If you don't, the history application will refuse to start, or it will report failing health checks and stop transferring data to its data store.

# engine application (listed below is only the configuration specific to the history application)
engine:
  image: ghcr.io/noumenadigital/packages/engine:latest
  depends_on:
    - postgres_db
  # ...
  environment:
    # ...
    ENGINE_DB_URL: "jdbc:postgresql://postgres_db/platform"
    ENGINE_DB_SCHEMA: engine-schema
    # enable history application
    ENGINE_DB_HISTORY_USER: history
    ENGINE_DB_HISTORY_SCHEMA: history-schema

# history application
history:
  image: ghcr.io/noumenadigital/packages/history:latest
  depends_on:
    - postgres_db
    - engine
  ports:
    - "${HISTORY_PORT:-12010}:12010"
    - "${HISTORY_ADMIN_PORT:-12710}:12710"
    - "${HISTORY_MANAGEMENT_PORT:-12410}:12410"
  environment:
    HISTORY_ADMIN_HOST: 0.0.0.0
    HISTORY_MANAGEMENT_HOST: 0.0.0.0
    HISTORY_DB_URL: "jdbc:postgresql://postgres_db/platform"
    HISTORY_DB_USER: history
    HISTORY_DB_PASSWORD: <pwd>
    HISTORY_DB_SCHEMA: history-schema
    HISTORY_DB_ENGINE_SCHEMA: engine-schema

For further configuration options, see environment variables (history).

A complete docker compose file example can be found here.

Health check

Once the history application is deployed, you can test its healthcheck endpoint with:

curl localhost:12010/actuator/health

Swagger UI

You can explore the APIs provided by the history application via Swagger UI, available at http://localhost:12010/swagger-ui/.

History states (API)

The history application API provides an endpoint to retrieve protocol states for a given protocol from the history data store. Note that the history data endpoint is not secured and is available on the admin host/port.

You can try it out as follows:

curl localhost:12710/admin/history/streams/states/{protocolId}

History management (API)

The history application management API provides endpoints that allow to control the application and its data. It is available on the management host/port.

You can try it out as follows:

curl -X GET    'localhost:12410/management/history/scheduler/status'
curl -X PUT    'localhost:12410/management/history/scheduler/status' -H 'Content-Type: application/json' -d '{"enabled": "false"}'

curl -X POST   'localhost:12410/management/history/data/move?batchLimit=100&synchronous=true'
curl -X DELETE 'localhost:12410/management/history/data'