Write a migration

Migrations are written in Kotlin Script, using an NPL specific DSL.

Migration files

Migrations are written in a Kotlin DSL. Its contents are stored in separate .kts-files for each individual migration.

File names follow a mandatory naming pattern, containing both the sequence number and a description that explains what the purpose or intent of the migration is. Its format is a S followed by an actual migration sequence number, followed by an underscore and the descriptive name. Examples of valid file names include S1_Name.kts, S1_Hello_world.kts, S1_v1-to-v2.kts, and S1_Smörgåsbord.kts.

Describing a migration

New migrations are instantiated using migration(descriptor, enableCompilerOptimisations), where the descriptor serves as a means of identifying this particular migration when applied to the system. The enableCompilerOptimisations parameter (false by default) enables automatic code removal for statements which don't impact code execution or produce side effects.

Subsequent operations may be added using its fluent API. These operations fall in one of two categories: read operations, and mutating operations. All operations pertain to type identifiers (typeIds), which represent a fully qualified identifier of the type to be targeted.

migration("description")
    .read(
       typeId = "/app-1.0.0?/example/Contract"
    ) {
        // <ProtocolReader>
    }
    .transformProtocol(
       currentTypeId = "/app-1.0.0?/example/OldAgreement",
       targetTypeId = "/app-2.0.0?/example/NewAgreement"
    ) {
       // <ProtocolTransformer>
    }
    .transformStruct(
       currentTypeId = "/app-1.0.0?/example/OldStruct",
       targetTypeId = "/app-2.0.0?/example/NewStruct"
    ) {
       // <StructTransformer>
    }
    .transformIdentifier(
       currentTypeId = "/app-1.0.0?/example/OldId",
       targetTypeId = "/app-2.0.0?/example/NewId"
    )
    .transformEnum(
       currentTypeId = "/app-1.0.0?/example/OldEnum",
       targetTypeId = "/app-2.0.0?/example/NewEnum",
       mappings = mapOf("OldVariant1" to "NewVariant1")
    )

Reads

Read operations are intended to accumulate existing protocol data, and are always executed before mutating operations. The migration API does not allow the definition of a read operation after a mutating operation to help reflect this fact. You can use as many read operations as necessary.

Use case: Collecting data prior to transformations

Let's take the following protocol as an example.

protocol[p] Contract(var amount: Number) {};

Imagine that as part of our migration, the value amount must be read for every Contract instance.

val collectedData = mutableMapOf<ProtocolReferenceValue, NumberValue>()

migration("collect data")
    .read(typeId = "/app-1.0.0?/example/Contract") {
        collectedData[reference] = get("amount")
    },

At the start of the transformation phase of the migration, the collectedData map will have been populated completely.

Operations

`read`

The only available read operation is read.

migration("read")
    .read(typeId = "/app-1.0.0?/example/Contract") {
        // <ProtocolReader>
    }

ProtocolReader

The ProtocolReader exposes various ways of collecting data from an existing protocol instance.

Method/property	Explanation
`id`	protocol instance identifier
`reference`	protocol instance reference value
`parties`	protocol parties (in provided order)
`get<Value>(propertyName)`	obtain a property or argument value

Transformations

Transformations are types of changes to existing protocol instances. They are always executed after read operations and may safely assume that all read operations have been run against all known protocols at the time of execution.

Oftentimes the protocols you are migrating will themselves be part of other protocols, collections, or structs. Migrating these can become a tedious procedure, trying to traverse such a graph is error-prone, and ultimately cycles will emerge such that no clear dependency graph may be constructed. It is therefore possible to migrate protocols out of order, and recursively apply protocol, struct, and identifier migrations, even when they are within other protocols and collections or structs. They may for that reason be defined in an arbitrary order, and are applied as encountered: transformations cannot be assumed to run in a particular order.

For many operations you can therefore just leave it entirely to the migration to map an old instance to a new instance if no special rules are required to work out how to do so. This is the case for protocols that are structurally identical. In such cases you can use the retag transformation (in combination with the convenience function mapPrototypes that can generate a concise list of all your NPL prototypes).

So why are transformations required at all?

Use case 1: Changing a protocol definition

Take the following as an example. A year ago, the following NPL was uploaded, and various protocol instances exist in the system.

struct Data {
    amount: Number
};

enum CustomerClass { A, B };

protocol[p] Contract(var data: List<Data>, var class: CustomerClass) {};

An important feature is going to be added, requiring the storage of a discount on the Data struct. Furthermore, CustomerClass is now going to be ModifiedCustomerClass. The new NPL is now as follows.

struct ExtendedData {
    amount: Number,
    discount: Number
};

enum ModifiedCustomerClass { X, Y };

protocol[p] NewContract(
    var data: List<ExtendedData>,
    var class: ModifiedCustomerClass) {};

If existing protocols need to be moved from Contract to NewContract, this also means that Data needs to be changed to ExtendedData. This requires an explicit rule for how to set the discount field. Furthermore, values of CustomerClass need to be mapped to ModifiedCustomerClass.

A migration to move from Contract to NewContract looks as follows.

migration("description")
    .transformProtocol(
        currentTypeId = "/app-1.0.0?/example/Contract",
        targetTypeId = "/app-2.0.0?/example/NewContract",
    )
    .transformStruct(
        currentTypeId = "/app-1.0.0?/example/Data",
        targetTypeId = "/app-2.0.0?/example/ExtendedData",
    ) {
        put("discount") {
            NumberValue(0)
        }
    }
    .transformEnum(
        currentTypeId = "/app-1.0.0?/example/CustomerClass",
        targetTypeId = "/app-2.0.0?/example/ModifiedCustomerClass",
        mappings = mapOf("A" to "X", "B" to "Y"),
    ),

For the transformStruct step here, we have to explicitly use put to set a value for discount. So why do we not have to do something similar for data in the transformProtocol step that transforms Contract to NewContract? Because the field is left untouched: it is still a List, it is still called data, and therefore the transformStruct transformation is applied recursively. (When the value at hand is touched, withTransformations provides a similar mechanism to accomplish this manually.)

Use case 2: Modifying a protocol's values

Take the following protocol as an example.

protocol[p] Contract(var name: Text, var discount: Number) {};

The discount for a person named "Frank" has been set wrongly: Frank's discount should be 10 rather than 0. A migration to change this is as follows.

migration("change Frank's discount")
    .transformProtocol(
        currentTypeId = "/app-1.0.0?/example/Contract",
        targetTypeId = "/app-2.0.0?/example/Contract",
    ) {
        if (get<TextValue>("name") == TextValue("Frank")) {
            replace<NumberValue>("discount") {
                NumberValue(10)
            }
        }
    },

Because no mutating rule is defined for name, its transformation is handled automatically.

Operations

transformProtocol

Transform protocol applies a transformation to each protocol that matches the currentTypeId.

In all cases the new protocol takes the ID of the old one (meaning existing references from other protocols will point to the new protocol after any migration).

The framework will attempt to construct the state of the new protocol entirely from that of the old one. Constructors or init blocks are not called, and require conditions are not enforced.

The transformation logic does not try to be clever - it does not discard data that doesn't seem to be needed for the new protocol. If the result of applying the deletes, replaces and puts does not result in exactly the arguments and fields declared by the new protocol then it will fail the transformation.

migration("transformation")
    .transformProtocol(
       currentTypeId = "/app-1.0.0?/example/OldAgreement",
       targetTypeId = "/app-2.0.0?/example/NewAgreement"
    ) {
        // <ProtocolTransformer>
    }

rebindTokenContext

In a Multinode context, it is possible for the migration of a protocol to affect the TokenContexts of existing@multinode-annotated permissions. There are three (3) scenarios under which TokenContext migrations can be performed:

A protocol transformation that does not affect its @multinode-annotated permission(s): token contexts will be migrated automatically as long as the permissions' action name and party name don't change between the migration from protocol A to protocol B. Changing the underlying party (claims + access) of the party name is permitted.
A protocol transformation that affects its @multinode-annotated permission(s): token contexts will have to be migrated manually by using the rebindTokenContext operation and specifying the target protocol type, the current protocol action name, the target protocol action name, and the party caller name.
A retag migration: token contexts will be automatically migrated to the new protocol version.

// automatic migration of token contexts
migration("transformation")
    .transformProtocol(
       currentTypeId = "/app-1.0.0?/example/OldAgreement",
       targetTypeId = "/app-2.0.0?/example/NewAgreement"
    ) {}

// manual migration of token contexts
migration("transformation")
    .transformProtocol(
       currentTypeId = "/app-1.0.0?/example/OldAgreement",
       targetTypeId = "/app-2.0.0?/example/NewAgreement"
    ) {}
    .rebindTokenContext(
       "/app-2.0.0?/example/NewAgreement",
       "oldPermission1",
       "newPermission1",
       "partyName"
    )
    .rebindTokenContext(
       "/app-2.0.0?/example/NewAgreement",
       "oldPermission2",
       "newPermission2",
       "partyName"
    )

// automatic migration of token contexts
migration("transformation")
    .retag()

transformStruct

Transform struct applies a transformation to each struct that matches the currentTypeId.

migration("transformation")
    .transformStruct(
       currentTypeId = "/app-1.0.0?/example/OldStruct",
       targetTypeId = "/app-2.0.0?/example/NewStruct"
    ) {
        // <StructTransformer>
    }

transformIdentifier

Transform identifier applies a transformation to each identifier that matches the currentTypeId.

migration("transformation")
    .transformIdentifier(
       currentTypeId = "/app-1.0.0?/example/OldId",
       targetTypeId = "/app-2.0.0?/example/NewId"
    ) // there is no transformer block

transformEnum

`transformEnum(String, String, Map<String, String>?)`

Applies a transformation to each enum that matches the currentTypeId.

Variant mappings define how variants of currentTypeId map to those in targetTypeId. If no mappings are specified, an identity mapping is assumed (where each variant in currentTypeId is mapped to the same targetTypeId variant). If currentTypeId contains variants that have not been defined in targetTypeId and no explicit mapping has been provided for these variants, the migration will fail with an exception. Note that identity mappings are always added automatically.

Consider the enums OldEnum and NewEnum.

// original enum definition
enum OldEnum { A, B, C };

// target enum definition
enum NewEnum { B, X };

The following migration applies.

migration("enum transformation")
    .transformEnum(
        currentTypeId = "/app-1.0.0?/example/OldEnum",
        targetTypeId = "/app-2.0.0?/example/NewEnum",
        mappings = mapOf("A" to "X", "C" to "X"),
    )

Note that the variant B is present in both enums and therefore does not need to be handled explicitly (but may be overridden if it needs to map to something that is not B in NewEnum). Variants A and C are removed: any values of A or C are now values of X.

`transformEnum(StatesEnumTypeId, StatesEnumTypeId, Map<String, String>?)`

Applies a transformation to the States derived enum of a given protocol typeId, while also automatically issuing a state call on the ProtocolTransformer of a transformProtocol.

Variant mappings define how variants of currentDerivedTypeId map to those in targetDerivedTypeId. If no mappings are specified, an identity mapping is assumed (where each variant in currentDerivedTypeId is mapped to the same targetDerivedTypeId variant). If currentDerivedTypeId contains variants that have not been defined in targetDerivedTypeId and no explicit mapping has been provided for these variants, the migration will fail with an exception. Note that identity mappings are always added automatically.

While this enum transform requires a transformProtocol to be meaningful, the operation will be a valid no-op if a transformProtocol is omitted.

You may override the automatic state call with your own custom state or state { } inside a transformProtocol context. When overriding this state call, calling transforEnum is still required – otherwise the migration will fail. The mappings parameter of the transformEnum call is also required.

Consider the following protocols and their States:

// original protocol definition
protocol[p] One() {
    initial state myFirstState;
    var a: One.States = One.States.myFirstState;
};

// target protocol definition
protocol[p] Two() {
    initial state mySecondState;
    var a: Two.States = Two.States.mySecondState;
};

A typical migration to migrate the States of Protocol One to the States of Protocol Two would be:

migration("transformation")
    .transformProtocol(
        currentTypeId = "/app-1.0.0?/example/One",
        targetTypeId = "/app-2.0.0?/example/Two",
    )
    .transformEnum(
        currentDerivedTypeId = StatesEnumTypeId(
            protocolTypeId = "/app-1.0.0?/example/One",
        ),
        targetDerivedTypeId = StatesEnumTypeId(
            protocolTypeId = "/app-2.0.0?/example/Two",
        ),
        mappings = mapOf("myFirstState" to "mySecondState"),
    ),

A migration with a custom state instead of the automatic one issued by transformEnum could be:

migration("transformation")
    .transformProtocol(
        currentTypeId = "/app-1.0.0?/example/One",
        targetTypeId = "/app-2.0.0?/example/Two",
    ) {
        // `state` call override
        state { if (it == "myFirstState") "mySecondState" else "anotherState" }
    }
    .transformEnum(
        currentDerivedTypeId = StatesEnumTypeId(
            protocolTypeId = "/app-1.0.0?/example/One",
        ),
        targetDerivedTypeId = StatesEnumTypeId(
            protocolTypeId = "/app-2.0.0?/example/Two",
        ),
        mappings = mapOf("myFirstState" to "mySecondState"),
    ),

Finally, a valid no-op migration with an omitted transformProtocol would be:

migration("transformation")
    .transformEnum(
        currentDerivedTypeId = StatesEnumTypeId(
            protocolTypeId = "/app-1.0.0?/example/One",
        ),
        targetDerivedTypeId = StatesEnumTypeId(
            protocolTypeId = "/app-2.0.0?/example/Two",
        ),
        mappings = mapOf("myFirstState" to "mySecondState"),
    ),

createProtocols

Migration operation that allows for the creation of protocols via a lambda (ProtocolCreateOperation) -> Unit. This approach can be useful for a variety of situations without necessitating prior transforms. Protocols created via method cannot depend on protocols created by the ProtocolTransformer.createProtocol method.

migration("create protocol migration without transform")
    .createProtocols { op ->
        for (i in 0..9) {
            op.createProtocol(
               typeId = "/app-2.0.0?/example/Contract",
               parties = listOf(PartyValue("p")),
               arguments = listOf(argumentValue),
            )
        }
    }

migration("create protocol migration with transform")
    .createProtocols { op ->
       for (i in 0..9) {
           op.createProtocol(
              typeId = "/app-2.0.0?/example/Contract",
              parties = listOf(PartyValue("p")),
              arguments = listOf(argumentValue),
           ) {
               put("name") { TextValue("my name $i") }
           }
       }
    }

retag

Retag applies a remapping of prototypes based on provided lists of ID pairs. It is functionally equivalent to writing one of the above transforms specifying the IDs but omitting the transform block. The goal of retag is to make the intent of the migration easier to grasp by reducing the amount of error-prone boilerplate elements in a script. Retag must come last in a migration.

The retag operation takes lists of ID pairs wrapped up into a PrototypeIdPairs instance. If any pair involves IDs for which a custom transformation has already been specified (with transformProtocol etc.) then retag silently ignores such pairs and the custom transformation takes precedence.

val prototypes = PrototypeIdPairs(
    enums = listOf<IdPair>(),
    structs = listOf<IdPair>(),
    identifiers = listOf<IdPair>(),
    protocols = listOf<IdPair>(IdPair("/app-1.0.0?/example/Car","/app-2.0.0?/example/Car"),
    current = "app-1.0.0", // current tag
    target = "app-2.0.0" // target tag
)

Although PrototypeIdPairs instances can be hand coded, it is expected that they be generated by helper functions provided by the Platform or associated tooling (see next section) or by project specific logic.

PrototypeIdPairs generation

The convenience function mapPrototypes can be used to automatically generate a PrototypeIdPairs instance for you. It does this by discovering all prototypes in a source directory. If you also provide the desired current and target tags then it will automatically prefix the IDs with these tags.

val prototypes = mapPrototypes(path)

// When using tags to version the NPL.
val prototypes = mapPrototypes(path, "app-1.0.0", "app-1.0.1")

Note: Additionally the convenience function mapPrototypesInMigration is provided which will construct the prototype ID pairs based on the migration definition file.

The list of prototypes returned by either of the above functions can also be used in transformations, to map the current and target prototype IDs, and to create type reference.

val prototypes = mapPrototypes(path, "app-1.0.0", "app-1.0.1")

val color = prototypes.match("Color")
val features = prototypes.match("Features")
val car = prototypes.match("Car")

migration("${prototypes.current} to ${prototypes.target}")
    .transformEnum(
        color.current,
        color.target,
        mapOf("r" to "red", "g" to "green", "b" to "blue")
    )
    .transformStruct(features.current, features.target) {
        put("color") {
            createOptional(color.targetAsType())
        }
    }
    .transformProtocol(car.current, car.target) {
        put("features") {
            createOptional(features.targetAsType())
        }
    }
    .retag(prototypes)

Refer to the match utility function for more details.

StructTransformer and ProtocolTransformer

There are two transformers: StructTransformer exposes operations for performing struct-to-struct mappings, whereas ProtocolTransformer exposes operations for state-to-state mappings.

See Transformer operations for a list of fields and methods available in these contexts.

withTransformations

In the section on transformations it was explained that under most circumstances all specified transformations are applied automatically, rather than having to respecify these operations for each specific field. For example, in a migration in which all protocols of type OldAgreement are turned into NewAgreement any values of OldAgreement found on any other protocols are converted as well.

This feature can also be invoked explicitly whenever more involved transformations include not just such already specified migrations, but require additional transformative steps. For example, an additional protocol OldAgreementBundle may contain List<OldAgreement> that is now a Set<NewAgreement>. This transformation not only relies on OldAgreement being turned into NewAgreement (a transformation that is already specified), but also on an additional step of converting the List into a Set that cannot be performed automatically.

Rather than having to respecify the conversion from OldAgreement to NewAgreement when specifying the conversion of List to Set, there is withTransformations which applies all specified automatic transformations to the provided value. In our example above, the variable newAgreementListBundle in withTransformations(oldAgreementListBundle) { newAgreementListBundle -> ... } would be a List<NewAgreement>. The logic within the braces should then only concern itself with converting List<NewAgreement> into Set<NewAgreement>.

migration("withTransformations-example")
    // a transformation from `OldAgreement` to `NewAgreement`.
   .transformProtocol(
      currentTypeId = "/app-1.0.0?/example/OldAgreement",
      targetTypeId = "/app-2.0.0?/example/NewAgreement"
   ) {
      // transformation logic.
   }
    // another transformation wishing to make use of the above.
   .transformProtocol(
      currentTypeId = "/app-1.0.0?/example/OldAgreementBundle",
      targetTypeId = "/app-2.0.0?/example/NewAgreementBundle"
   ) {
      replace<ListValue>("bundle") { oldList ->
         withTransformations(oldBundle) { transformedList ->
            // Existing transformations applied, 'transformedList' is
            // now List<NewAgreement>, but not yet a Set<NewAgreement>.
            // All that needs to be done is convert it into a set.
            createSet(transformedList.value)
         }
      }
   }

Specifically, this allows additional transformations to be performed on top of the automatic transformations. Such function composition may be denoted as (withTransformations ∘ additionalTransformation)(oldValue).

type

Given a type ID for a user-defined type (union, struct, identifier, or enum), function type(typeId: String) returns the corresponding type reference.

For example, an empty list that would (eventually) hold protocols of type Example can be created like this:

createList(emptyList(), type("/app-2.0.0?/example/Example"))

A more extensive example can be found here.

Note that while symbols are user-defined type, they do not currently have their own prototypes, and type will therefore not be able to resolve them. TypeRef.name(typeId) can be used instead for symbols.