Bulk File Creation

The Bulker is part of the core functionality of IPF. It provides a persistent store that processing flows are able to submit components to, over a given period of time. Using a pre-configured trigger, the Bulker will stop accepting new components and write all the components it has to a structured file.

Within the SEPA DD CSM, the Bulker is configured to accept Direct Debit Transactions and write these to one of many messages types. Prior to the daily SEPA DD Core Collection Scheme Cutoff time, a trigger will be configured to close all the open messages, update their headers and add them to an IDF. Once the IDF is written to file, a notification will be sent to the configured destination to inform downstream systems that the IDF is ready for delivery to the EBA scheme. New Bulks will be opened automatically when the full Bulks close, so that there is no down-time or disruption to the service.

The Bulker is configured to hold Bulks and create batch files for the following outbound files and messages:

  • IDF

    • pacs.003

    • pacs.007

    • camt.056

Multi Level Bulking

Multi-level bulking is supported for pacs.003, pacs.007 and camt.056.

In order to group individual payments into the correct messages to pass scheme validation, the Bulker is configured to use multi-level bulking, auto-generated, recurring Bulks.

Each message within the Bulker is a separate Bulk with all the functionality that comes with a Bulk (add components, remove components, query components etc). In order to use the Multi Level Bulking functionality, however, these bulks are not configured to stream to a File when finalised. A Multi-level bulk is linked as a "Child" to a "Parent Bulk" (ie the IDF file), this means that when the Parent is ready to be streamed to a File, the whole of the Child will be included as a single component within it. An IDF file will end up containing multiple messages that have all been built separately from related transactions.

The Bulks are configured to use the auto-generation functionality. It is not possible to pre-define every possible message that could be needed and have a Bulk for them, ready in the Bulker waiting to be filled. When a request is made to put a transaction into a bulk, if the appropriate bulk is not present it will be automatically generated using the defined parameters. Subsequent transactions to the given message will be added as per any normal Bulk.

IDF files are needed for each LAC and can be predefined; as a recurring Bulk. This means each time an IDF Bulk is closed a new Bulk is automatically created to the same specification and ready to accept notifications as soon as a new child bulk is created.

The combination of auto-generation and recurring functionality gives seamless availability, thus ensuring no transactions are missed.

IDF File Generation per Instructing Agent

IDF file is generated based on the following grouping rules:

  • File Level:

    • File Type: IDF

    • Instructing Agent (each BIC results in a separate IDF file)

  • Message Level:

    • Creditor Agent at the transaction level

    • Settlement Date (Pacs.003/Pacs.007 only)

This means that for clients using the SDD scheme pack across multiple Instructing Agents, the system will generate a separate IDF file for each unique Instructing Agent.

Each IDF file contains one or more messages, where transactions are grouped according to Creditor Agent and in the case of pacs.003 and pacs.007 also Settlement Date.

These files are generated automatically by the Bulker system, which applies multi-level bulking logic fully aligned with SEPA scheme requirements.

Limiting Size Of Produced Bulk File

STEP2 scheme has a constraint regarding size of the file it can process.

In order to fulfill this requirement, when new IDF bulk is produced, a task to periodically check its size is scheduled. This task will check the current size of IDF bulk and all its child bulks, and it will trigger closing of that bulk when size reaches configured maximum file size.

Since this check is performed periodically, and bulker is continuously receiving transactions, there is no guarantee that the Bulk will be closed as soon as the value reaches maximum file size. Because of that, maximum file size should be set to a value lower than actual maximum file size allowed by the scheme.

Property Name Type Comment Default

ipf.csm.sepa.bulk-size.maximum-file-size

Integer

Maximum size of the bulk, when size of the bulk reaches this value, bulk will be closed and data will be streamed to a file

480MB

ipf.csm.sepa.bulk-size.size-check-cron

String

Cron expression for checking if the size of the bulk reached 'maximum-file-size'

*/30 * * ? * *