Documentation for a newer release is available. View Latest

Payment Entries Processor Execution Data Purging

Purging removes documents from the executionInfos collection. This process ensures that, after all the Transactions for an Instruction are processed successfully, the Execution Info record for the Instruction is eventually deleted from the database.

To enable Purging, the following actions are required:

Create TTL (Time-To-Live) indexes: This step is mandatory as it allows the database to automatically purge items in the collections. Without these indexes, no purging will occur.
Set ttl and expiryDate fields: The application itself sets the ttl field (required for CosmosDB) and the expiryDate field (required for MongoDB) when an Instruction is finished processing successfully. These fields determine the retention period for the items before they are purged; the retention period is configurable and is explained in detail in the Configurable Time To Live (TTL) section.

TTL indexes must be created because the database relies on these indexes to purge entries. Without TTL indexes, Execution Infos will not be purged from the DB.

By default, all indexes are created automatically at application startup and support both MongoDB and Azure CosmosDB for MongoDB. For manual index creation, refer to MongoDB or Azure CosmosDB for MongoDB.

The following configuration controls Execution Info Purging:

ipf.core.payment-entry-processor.storage {

  # This flag determines if indexes should be created during application startup.
  # The default value is true.
  create-indexes = true (1)
  # If ipf.mongodb.create-indexes is specified, it will take precedence over this setting.
  create-indexes = ${?ipf.mongodb.create-indexes} (2)

  # Specifies the database type the application is using. This is crucial as each database requires a slightly different configuration.
  # The default mode is "mongo". Available options are: "mongo" for MongoDB and "cosmosdb" for Azure CosmosDB for MongoDB.
  database-mode = mongo (3)
  # If ipf.mongodb.database-mode is specified, it will take precedence over this setting.
  database-mode = ${?ipf.mongodb.database-mode} (4)

  # PaymentEntries triggered for 'housekeeping' will be deleted after the configured time-to-live duration has elapsed.
  time-to-live = 30d (5)

  # The commit quorum to use when creating indexes,
  # see https://www.mongodb.com/docs/manual/reference/command/createIndexes/#create-index-with-commit-quorum for more info
  commit-quorum = "votingMembers" (6)
  # If ipf.mongodb.commit-quorum is specified, it will take precedence over this setting.
  commit-quorum = ${?ipf.mongodb.commit-quorum} (7)

}

1	`create-indexes` is enabled by default. This will create all indexes required, including the TTL index required for purging. If indexes are to be created manually (e.g. via deployment scripts), this can be disabled by setting to `false`.
2	`ipf.mongodb.create-indexes` can be used instead of the `ipf.htm.mongodb.purging.create-indexes` property.
3	`database-mode` is set to 'mongo' by default. This specifies which supported database you are using. Set this to `mongo` for MongoDB or `cosmosdb` for Azure CosmosDB for MongoDB. The application will create the appropriate TTL indexes based on this property.
4	`ipf.mongodb.database-mode` can be used instead of the `ipf.htm.mongodb.purging.database-mode` property.
5	`time-to-live` is set to 30 days by default. This specifies the retention period of a successfully processed Instruction’s Execution Info record. The duration format is any format supported by HOCON.
6	`commit-quorum` dictates how many replicas must acknowledge index creation before it is considered successful. For more details, refer to creating indexes with commit quorum.
7	`ipf.mongodb.commit-quorum` can be used instead of the `ipf.htm.mongodb.purging.commit-quorum` property.

Configurable Time To Live (TTL)

Execution Infos in the Payment Entries Processor have a configurable TTL. The value is set using the ipf.core.payment-entry-processor.storage.time-to-live configuration key.

The TTL is applied right after all Transactions for Instruction are processed successfully.

For example, if the TTL is 3d and the processing for Instruction is finished on 01/01/2025, then the related Execution Info will expire (removed from the DB) on 04/01/2025.

The default TTL is configured for 30 days.

Manual Creation of TTL Indexes for MongoDB

For MongoDB, create TTL indexes on the expiryDate field, which indicates the exact date an item should be purged. For more details, refer to Expire Documents at a Specific Clock Time.

db.getCollection("executionInfos").createIndex({ "expiryDate": 1 }, { expireAfterSeconds: 0 })

ipf.core.payment-entry-processor.storage.time-to-live is used to calculate the future date when the document will be deleted.

Manual Creation of TTL Indexes for Azure CosmosDB for MongoDB

For Azure CosmosDB for MongoDB, create TTL indexes on the internal _ts field, which contains a timestamp of the last modification. Additionally, documents need a ttl integer field representing the retention period in seconds. For more information, check out Time to Live (TTL) in Azure Cosmos DB and Expire data with Azure Cosmos DB’s API for MongoDB.

db.getCollection("executionInfos").createIndex({"_ts":1}, {expireAfterSeconds: -1})

ipf.core.payment-entry-processor.storage.time-to-live is converted to seconds and sets the ttl field. Azure CosmosDB for MongoDB will purge the document after this period.

How to disable purging?

Purging can be disabled by not creating the TTL indexes on executionInfos collections.

Make sure that automatic index creation is disabled by setting ipf.core.payment-entry-processor.storage.create-indexes to false, otherwise indexes will be created on next application startup.

Even though indexes are dropped, the Payment Releaser will continue to set ttl and expiryDate fields which has no effect on purging until indexes are recreated.