System topology

High level overview

The image below sketches out the components of the NOUMENA environment. At the center we have the three components that constitute the NOUMENA runtime: the Engine, the (optional) History application and the (optional) Read Model. These components all connect to a PostgreSQL database.

Applications are deployed in the engine, after which they become available through all three components. Authorization is managed using an Identity and Access Management of choice, as long as it supports OIDC.

We typically suggest running a reverse proxy in front of the runtime.

Example topology

Depending on individual preferences and requirements there are several ways to realize the environment. The diagram below shows an example of a possible configuration.

Example configuration

Docker compose file

Realising such a set-up can be achieved with a docker compose file. What follows is a minimal working example:

version: "2.17.0"

services:

  # the engine requires a PostgreSQL database
  engine-db:
    image: postgres:14.10-alpine
    mem_limit: 256m
    environment:
      - POSTGRES_DB=platform
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - ENGINE_DB_USER=engine
      - ENGINE_DB_PASSWORD=engine_pwd
      - HISTORY_DB_USER=history
      - HISTORY_DB_PASSWORD=history_pwd
      - POSTGRAPHILE_DB_USER=postgraphile
      - POSTGRAPHILE_DB_PASSWORD=postgraphile_pwd
    volumes:
      # refer to database user setup script example (below) for details on how to set up database users
      - ./bin/init-db-users.sh:/docker-entrypoint-initdb.d/init-db.sh
    healthcheck:
      test: pg_isready -U postgres
      interval: 1s
      timeout: 5s
      retries: 50

  engine:
    # refer to Dockerfile example (below) for what would usually get wrapped around an engine image
    build: .
    ports:
      - "12000:12000"
    mem_limit: 512m
    environment:
      ENGINE_DB_URL: "jdbc:postgresql://engine-db/platform"
      ENGINE_DB_USER: engine
      ENGINE_DB_PASSWORD: engine_pwd
      ENGINE_DB_POSTGRAPHILE_USER: postgraphile
      ENGINE_DB_POSTGRAPHILE_PASSWORD: postgraphile_pwd
      ENGINE_DB_HISTORY_USER: history
      ENGINE_DB_HISTORY_PASSWORD: history_pwd
      ENGINE_DB_HISTORY_SCHEMA: history
      ENGINE_ALLOWED_ISSUERS: "http://keycloak:11000/realms/noumena"
    depends_on:
      engine-db:
        condition: service_healthy
      keycloak:
        condition: service_healthy

  history:
    image: ghcr.io/noumenadigital/packages/history:latest
    ports:
      - "12010:12010"
    mem_limit: 512m
    environment:
      HISTORY_DB_URL: "jdbc:postgresql://engine-db/platform"
      HISTORY_DB_USER: history
      HISTORY_DB_PASSWORD: history_pwd
    depends_on:
      engine-db:
        condition: service_healthy
      engine:
        condition: service_healthy

  postgraphile:
    image: ghcr.io/noumenadigital/packages/postgraphile:latest
    ports:
      - "5555:5555"
    environment:
      POSTGRAPHILE_DB_URL: "postgres://postgraphile:postgraphile_pwd@engine-db:5432/platform"
      POSTGRAPHILE_DB_USER: postgraphile
      POSTGRAPHILE_ALLOWED_ISSUERS: "http://keycloak:11000/realms/noumena,"
      POSTGRAPHILE_ENGINE_HEALTH_ENDPOINT: "http://engine:12000/actuator/health"
    depends_on:
      engine-db:
        condition: service_healthy
      engine:
        condition: service_healthy

  # the engine requires an OIDC provider for authentication
  # Keycloak provides a lot of options to get this up and running
  keycloak:
    image: quay.io/keycloak/keycloak:20.0
    command: start --hostname-strict=false --hostname-strict-https=false --metrics-enabled=true --db=postgres
    environment:
      KEYCLOAK_ADMIN: admin
      KEYCLOAK_ADMIN_PASSWORD: Keycloak123!
      KC_DB_URL: jdbc:postgresql://keycloak-db/postgres
      KC_DB_USERNAME: postgres
      KC_DB_PASSWORD: testing
      KC_HEALTH_ENABLED: "true"
      KC_HTTP_ENABLED: "true"
      KC_HTTP_PORT: 11000
      KC_HOSTNAME: keycloak
    healthcheck:
      test: curl -s localhost:11000/health || exit 1
      interval: 1s
      retries: 60
    depends_on:
      keycloak-db:
        condition: service_healthy

  keycloak-db:
    image: postgres:14.10-alpine
    mem_limit: 256m
    environment:
      POSTGRES_PASSWORD: testing
    healthcheck:
      test: pg_isready -U postgres
      interval: 1s
      timeout: 5s
      retries: 50

To run this set-up, execute the following command:

docker compose up --wait --build

Refer to the docker configuration example sections of the engine, history, and Postgraphile applications for more details about setting up individual applications.

Dockerfile

To deploy an engine image with NPL source code, you can wrap the engine image and add NPL, which gets deployed during engine startup.

# For a production setup, we advise to use the actual version number instead of latest
FROM ghcr.io/noumenadigital/packages/engine:latest

# Files in npl/npl-1.0.0 contain NPL source code
COPY npl/migration.yml migration.yml
COPY npl/npl-1.0.0 npl-1.0.0

# Copy migration files to the /migrations folder
COPY npl/npl-1.0.0/* /migrations/npl-1.0.0/

Database setup

We recommend setting up distinct database users for the different applications. What follows is the example script used in the docker compose file, init-db-users.sh:

#!/bin/sh
set -e

psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
    CREATE ROLE "$ENGINE_DB_USER" LOGIN PASSWORD '$ENGINE_DB_PASSWORD';
    ALTER ROLE "$ENGINE_DB_USER" CREATEROLE NOINHERIT;
    GRANT CREATE ON DATABASE $POSTGRES_DB TO "$ENGINE_DB_USER";
    --
    CREATE ROLE "$POSTGRAPHILE_DB_USER" LOGIN PASSWORD '$POSTGRAPHILE_DB_PASSWORD';
    ALTER ROLE "$POSTGRAPHILE_DB_USER" CREATEROLE NOINHERIT;
    GRANT CONNECT ON DATABASE $POSTGRES_DB TO "$POSTGRAPHILE_DB_USER";
    --
    CREATE ROLE "$HISTORY_DB_USER" LOGIN PASSWORD '$HISTORY_DB_PASSWORD';
    ALTER ROLE "$HISTORY_DB_USER" CREATEROLE NOINHERIT;
    GRANT CREATE ON DATABASE $POSTGRES_DB TO "$HISTORY_DB_USER";
EOSQL