Deploying migrations
You can deploy your migrations either at engine startup or via the management API.
Deploying during engine start up
The time it takes to apply migrations can and does vary, and applications will experience downtime while they are being applied. The duration of the downtime will depend on the number of protocols that exist in the system. It is therefore better to have the engine be offline while a migration occurs. The files which constitute a migration (NPL sources, Kotlinscript files, and migration descriptor) can be placed in any local directory or system mount. It will need to be configured in the docker setup as a volume mount.
Listed below are the step-by-step instructions to apply migrations at startup.
- Create any needed migration files. Ascertain that the migration does not break existing systems, or ensure that existing systems have been adapted accordingly. Pay attention to changing protocol interfaces within or outside of NPL, and existing protocol references that may no longer be present.
- Place all migration files within the desired location. The files can be organized within sub-directories as long as they are contained under the configured directory, which acts as a root directory.
- All migrations are transactional, so the changes will not persist unless the migration was a success. However, to ensure that there are no other issues, it is strongly encouraged to try the migration on a test system first.
- Configure a volume mount linking the local directory containing the migration artifacts to its container counterpart.
- Set the named
ENGINE_NPL_MIGRATION_DIRECTORY_PATH
environment variable with the absolute path to the container location of the migration components. This is an optional step,/migrations
is the default value. - Start the platform, wait for it to complete the startup process, and inspect the output. The engine logs will provide information about any migrations that have been applied, and any potential issues.
Upon success, the engine will run as normal using the newly migrated system.
Upon failure, the engine startup will halt with an error status. The engine logs will contain any migration errors that have occurred. If a failure occurred, fix any errors in the migration and then restart the engine.
Migration considerations
In order to migrate pre-existing CLINT logs to the new migration tables, you need to provide the schema where the logs
reside (this is assumed to be in the same database). To do this, set the environment variable ENGINE_DB_UPSHIFT_SCHEMA
to the correct name when migrating for the first time.
Example migration using the Seed Project
If you don't already have it, clone the Seed Project and follow the README instructions.
Example Docker config
FROM registry.noumenadigital.com/noumena/platform/engine:latest
# Set migration artifact directory, and set feature flag to "true"
ENV ENGINE_NPL_MIGRATION_DIRECTORY_PATH="/npl"
# Configure docker mount point
COPY src/main/yaml /npl/yaml
COPY src/main/npl /npl/npl-1.0.0
Start the engine (and initiate the migration)
Run the following command:
make run
Once the process is finished, run the command below to see the logs. The Dockerfile is already configured.
docker-compose logs engine
Engine log excerpt of a successful migration
---------------------------------------------------------------------
NPL Migration Log (id: 94c23cc4-9b34-49cf-8376-7f0d1c047155, state: OPENED):
┌─────┬────────────────┬────┬──────────┬──────────────────┬─────┬─────────────────────────────
│ │Platform version│Type│Result │Id │Entry│When
├─────┼────────────────┼────┼──────────┼──────────────────┼─────┼─────────────────────────────
│00001│2021.1.175 │run │SUCCESSFUL│1.0.0::0::npl-load│npl │2022-09-01T13:47:33.921240Z -
│ │ │ │ │ │ │2022-09-01T13:47:38.297954Z
└─────┴────────────────┴────┴──────────┴──────────────────┴─────┴─────────────────────────────
2022-09-01 13:47:38 INFO Completing migration 94c23cc4-9b34-49cf-8376-7f0d1c047155}
2022-09-01 13:47:38 INFO NPL migrations were successfully applied.
2022-09-01 13:47:49 INFO Exposing 5 endpoint(s) beneath base path '/actuator'
2022-09-01 13:47:50 INFO Initializing Spring DispatcherServlet 'dispatcherServlet'
2022-09-01 13:47:50 INFO Initializing Servlet 'dispatcherServlet'
2022-09-01 13:47:50 INFO Completed initialization in 4 ms
2022-09-01 13:47:51 INFO Started task scheduler.
2022-09-01 13:47:51 INFO Starting notifications handler.
2022-09-01 13:47:51 INFO Listening to PostgreSQL notifications.
Deploying via the Application Management API
Given all the files necessary for a migration (NPL sources, Kotlinscript files, and migration descriptor), they can be deployed as a zip file directly onto a running engine using the Application Management API's deployment endpoint
For example, the following directory
> tree deployment
deployment
├── kotlin-script
│ ├── S0001_migration.kts
│ ├── S0002_test.kts
│ └── S0003_migration.kts
├── npl-1.0.0
│ ├── Contract.npl
│ └── Entry.npl
├── npl-1.0.1
│ ├── Contract.npl
│ └── lib
│ └── Calc.npl
├── npl-1.0.2
│ ├── experimental
│ │ └── ExperimentalFeature.npl
│ └── prod
│ ├── Contract.npl
│ └── lib
│ └── Calc.npl
└── migration.yml
can be zipped up and deployed as follows:
> cd deployment
> zip -r deployment.zip *
> curl -X PUT \
'http://localhost:12400/management/application' \
-H "Authorization: Bearer ${IAM_TOKEN}" \
-H 'Content-Type: multipart/form-data' \
-F 'archive=@deployment.zip;type=application/zip'