Engine application

Background

The Operating Engine consists of the engine application, and its companion applications:

This document describes the setup for the engine application.

Configuration

Docker image

The engine application is provided as a docker image which includes a default configuration.

API

The engine application runs on port 12000 by default. The engine admin endpoints are exposed on port 12700 by default. The engine management endpoints are exposed on port 12400 by default.

Database user

A dedicated database user for the engine application should be created and configured for the application to start.

Environment variables

Deprecated environment variables and changes in default values of feature flags are documented here.

Feature flags

The following environment variables can be configured to enable or disable feature flags:

Environment variable	Description	Default value
FEATURE_NON_STANDARD_NPL_TYPES	Feature flag - enables recognition of non-standard types as NPL types to prevent redefinition	`false`

Engine

The following environment variables can be configured for the engine application:

Environment variable	Description	Default value
ENGINE_DEV_MODE	Engine running mode - indicates whether application runs in development mode	`false`
ENGINE_SPRING_ACTUATOR_EXPOSE	Engine monitoring - exposed Spring actuator endpoints	`info,health,service-health,metrics,prometheusmetrics`
ENGINE_PORT	Engine Core API port	`12000`
ENGINE_ADMIN_HOST	Engine Admin API host	`127.0.0.1`
ENGINE_ADMIN_PORT	Engine Admin API port	`12700`
ENGINE_MANAGEMENT_HOST	Engine Management API host	`127.0.0.1`
ENGINE_MANAGEMENT_PORT	Engine Management API port	`12400`
ENGINE_MANAGEMENT_ARCHIVE_SIZE_LIMIT	Engine Management API archive size limit	`10000000`
ENGINE_NPL_MIGRATION_DIRECTORY_PATH	NPL migrations - path to the folder where the NPL migration artifacts reside	`/migrations`
ENGINE_NPL_MIGRATION_RUN_ONLY	NPL migrations - environment identifier tag to be matched against change descriptor attribute 'run-only' values
ENGINE_SESSION_STATE_READ_CACHE_MAX_ENTRIES	Runtime session - state read cache size	`500`
ENGINE_STREAMS_BUFFER_MAX_SIZE	Streams - buffer streams until maximum size has been reached	`50`
ENGINE_STREAMS_BUFFER_MAX_TIMEOUT	Streams - buffer streams until timeout has elapsed	`PT0.5S`

Database

Environment variable	Description	Default value
ENGINE_DB_URL	Engine DB - URL	`jdbc:postgresql://localhost:5432/platform`
ENGINE_DB_USER	Engine DB - user	`platform_owner`
ENGINE_DB_PASSWORD	Engine DB - password	N/A, does not have a default value
ENGINE_DB_SCHEMA	Engine DB - schema	`noumena`
ENGINE_DB_HISTORY_USER	Engine DB - History user	`null`
ENGINE_DB_HISTORY_PASSWORD	Engine DB - History password	`null`
ENGINE_DB_HISTORY_SCHEMA	Engine DB - History schema	`history`
ENGINE_DB_POSTGRAPHILE_USER	Engine DB - Postgraphile user	`null`
ENGINE_DB_POSTGRAPHILE_PASSWORD	Engine DB - Postgraphile password	`null`
ENGINE_DB_UPSHIFT_SCHEMA	Engine DB - Upshift schema	`null`
ENGINE_DB_SSL	Engine DB - use SSL	`false`
ENGINE_DB_POOL_MAXIMUM_POOL_SIZE	Engine DB - maximum size of the connection pool	`10`
ENGINE_DB_POOL_MAXIMUM_CONNECTION_LIFETIME	Engine DB - maximum lifetime of a connection in the pool	`PT30M`

AMQP publication

The following environment variables can be configured to set up the engine's publication of notifications to an AMQP broker:

Environment variable	Description	Default value
AMQP_BROKER_URL	The URL of the AMQP broker. AMQP publication will be completely disabled if this is left unset.
AMQP_QUEUE_NAME	The name of the queue to publish notifications to.	`messages`
AMQP_USERNAME	The username to use when connecting to the broker.	`guest`
AMQP_PASSWORD	The password to use when connecting to the broker.
AMQP_POLLING_PERIOD_SECONDS	The period between polling the internal publication queue for new notifications to publish.	`5`
AMQP_CONNECTION_TIMEOUT_SECONDS	The timeout for establishing a connection to the broker.	`5`
AMQP_BATCH_SIZE	The maximum number of notifications to retrieve from the internal postgres queue. Does not affect the client publication behaviour.	`10`

NPL Contributor Libraries

Environment variable	Description	Default value
NPL_CONTRIB_PATH	The default location of NPL contributor libraries.	`/npl-contrib`

Authentication

The following environment variables can be configured to set up the engine's integration with the authentication server(s):

Environment variable	Description	Default value
ENGINE_ALLOWED_ISSUERS	Comma-delimited list of trusted JWT issuers. JWTs are only accepted if the value in the `iss` field matches one of these values exactly.	`http://localhost:11000/realms/noumena`
ENGINE_ISSUER_OVERRIDE	Instead of extracting the issuer from the `iss` field, always use this URL as the issuer; for development purposes only

Logging

The following environment variables can be configured for finer control of the engine application's logging:

Environment variable	Description	Default value	Example value
ENGINE_LOG_CONFIG	Logging configuration (file path)	`null`	`classpath:/logback-json.xml`
ENGINE_ROOT_LOG_LEVEL	Logging level (root)	`INFO`
ENGINE_LOG_LEVEL	Logging level - Engine application	`INFO`
ENGINE_AUTH_LOG_LEVEL	Logging level - Authentication	`INFO`
ENGINE_SPRING_LOG_LEVEL	Logging level - Spring	`INFO`
ENGINE_REQUEST_LOG_LEVEL	Logging level - Http requests	`WARN`

Swagger UI

The following environment variables can be configured to set up the engine application's Swagger UI:

Environment variable	Description	Default value
SWAGGER_ENGINE_URL	Swagger UI - URL to connect to the Engine Core API	`http://localhost:${ENGINE_PORT}`
SWAGGER_ENGINE_ADMIN_URL	Swagger UI - URL to connect to the Engine Admin API	`http://${ENGINE_ADMIN_HOST}:${ENGINE_ADMIN_HOST}`
SWAGGER_ENGINE_MANAGEMENT_URL	Swagger UI - URL to connect to the Engine Management API	`http://${ENGINE_MANAGEMENT_HOST}:${ENGINE_MANAGEMENT_PORT}`
SWAGGER_SECURITY_AUTH_URL	Swagger UI - URL to connect to the Authentication API	`http://localhost:11000/realms/noumena`
SWAGGER_SECURITY_CLIENT_ID	Swagger UI - IAM client ID	`nm-platform-service-client`
SWAGGER_SECURITY_CLIENT_SECRET	Swagger UI - IAM client secret	`87ff12ca-cf29-4719-bda8-c92faa78e3c4`

Example setup

Database user setup

The following SQL script creates a dedicated database user for the engine application.

create role engine login password <pwd>;
alter role engine createrole noinherit;
grant create on database mydatabase to engine;

Docker configuration

# engine application (here is a sub-selection to get you started)
engine:
  image: ghcr.io/noumenadigital/packages/engine:latest
  depends_on:
    - postgres_db
  ports:
    - "${ENGINE_PORT:-12000}:12000"
    - "${ENGINE_ADMIN_PORT:-12700}:12700"
    - "${ENGINE_MANAGEMENT_PORT:-12400}:12400"
  environment:
    ENGINE_ADMIN_HOST: ${ENGINE_ADMIN_HOST:-0.0.0.0}
    ENGINE_MANAGEMENT_HOST: ${ENGINE_MANAGEMENT_HOST:-0.0.0.0}
    ENGINE_DB_URL: jdbc:postgresql://postgres_db/platform
    ENGINE_DB_SCHEMA: engine-schema
    ENGINE_DB_USER: engine
    ENGINE_DB_PASSWORD: <pwd>
    ENGINE_DB_POSTGRAPHILE_USER: postgraphile
    ENGINE_DB_POSTGRAPHILE_PASSWORD: <pwd>
    ENGINE_DB_HISTORY_USER: history
    ENGINE_DB_HISTORY_PASSWORD: <pwd>
    ENGINE_DB_HISTORY_SCHEMA: history-schema
    SWAGGER_ENGINE_ADMIN_URL: http://localhost:12700/
    SWAGGER_ENGINE_MANAGEMENT_URL: http://localhost:12400/
    ENGINE_ALLOWED_ISSUERS: http://localhost:11000/realms/noumena
    # ...

For further configuration options, see environment variables.

A complete docker compose file example can be found here.

Health check

Once the engine application is deployed, you can test its healthcheck endpoint with:

curl localhost:12000/actuator/health

Swagger UI

You can explore the APIs provided by the engine application via Swagger UI, available at http://localhost:12000/swagger-ui/.