Documentation for a newer release is available. View Latest

S3 File Manager

The S3 file manager is suitable for integration with remote storage systems that are compatible with the Amazon S3 API.

Maven Dependency

To utilise the S3 File Manager in your application include the following dependency:

    <dependency>
        <groupId>com.iconsolutions.ipf.filemanager</groupId>
        <artifactId>ipf-file-manager-s3</artifactId>
    </dependency>

Configuration

Property Grouping: ipf.file-manager.s3

Key Description Default Value

ipf.file-manager.s3.resiliency-settings.attempt-timeout

The maximum time allowed for each individual attempt before timing out. Shorter timeouts fail faster but might not allow enough time for slow operations.

 
"5s"

ipf.file-manager.s3.upload-parallelism

Controls the maximum number of parallel upload operations that can occur simultaneously. Higher values may improve throughput but increase resource usage.

 
1

ipf.file-manager.s3.checksum-algorithm

The algorithm used to verify data integrity during transfers. CRC32 provides a good balance between performance and reliability.

 
"CRC32"

ipf.file-manager.s3.resiliency-settings.retryable-status-codes

HTTP status codes that should trigger a retry. Typically includes server errors like 500 (Internal Server Error) and 503 (Service Unavailable).

 
[
    500,
    503
]

ipf.file-manager.s3.credentials.secret-access-key

AWS secret access key for authentication when using static credentials provider.

 
"secretAccessKey"

ipf.file-manager.s3.resiliency-settings.max-attempts

Determines the maximum number of retries to be made. Note that this includes the first failed attempt.

 
5

ipf.file-manager.s3.region

AWS region is mandatory but will be automatically provided by the environment, see docs.aws.amazon.com/sdkref/latest/guide/feature-region.html

 
"us-east-1"

ipf.file-manager.s3.reader.buffer-size

Larger buffer size for better throughput with large files

 
"32kB"

ipf.file-manager.s3.enabled

S3 file manager is disabled by default when the maven dependency is included. Can be enabled by setting this property to true

 
false

ipf.file-manager.s3.credentials-provider

Values:
- static: Uses fixed access/secret keys specified in configuration
- sts-web-identity: Uses AWS IAM OIDC/Web Identity token provider for EKS/Kubernetes environments
- default: Uses AWS Default Credentials Provider Chain (docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html#credentials-default)
Follows this order: environment variables, AWS credentials file, IAM role for EC2/ECS
Falls back to DefaultCredentialsProvider if not specified or any other value

 
"static"

ipf.file-manager.s3.credentials.access-key-id

AWS access key ID for authentication when using static credentials provider.

 
"accessKey"

ipf.file-manager.s3.resiliency-settings.call-timeout

The overall timeout for the entire operation, including all retry attempts. Sets an upper bound on how long client code will wait for completion.

 
"3s"

ipf.file-manager.s3.chunk-encoding-enabled

Controls whether to use chunked transfer encoding for uploading files. When disabled, the entire file content is buffered before upload.

 
false

Error Handling

List of System Events

The below table documents the system events that can be raised by S3 File Manager. Payload schemas are documented as JSON Schema draft 4. Use a tool such as this one for a visual representation of the payload schemas.

All events contain some basic top-level information, and a payload. The top-level information is represented as members of the IPFSystemEvent type, and their payloads are documented below.

The schema for the top-level IPFSystemEvent type is as follows:

Top-level Schema 
{
  "$schema" : "http://json-schema.org/draft-04/schema#",
  "title" : "IPF System Event",
  "type" : "object",
  "additionalProperties" : false,
  "properties" : {
    "name" : {
      "type" : "string"
    },
    "level" : {
      "type" : "string",
      "enum" : [ "DEBUG", "INFO", "WARN", "ERROR" ]
    },
    "createdAt" : { },
    "version" : {
      "$ref" : "#/definitions/EventVersion"
    },
    "type" : {
      "type" : "string",
      "enum" : [ "TECHNICAL", "FUNCTIONAL" ]
    },
    "processingContext" : {
      "$ref" : "#/definitions/ProcessingContext"
    },
    "source" : {
      "type" : "string"
    },
    "metadata" : {
      "type" : "object",
      "additionalProperties" : {
        "type" : "string"
      }
    },
    "payload" : {
      "$ref" : "#/definitions/Object"
    }
  },
  "definitions" : {
    "EventVersion" : {
      "type" : "object",
      "additionalProperties" : false,
      "properties" : {
        "major" : {
          "type" : "integer"
        },
        "minor" : {
          "type" : "integer"
        },
        "patch" : {
          "type" : "integer"
        }
      },
      "required" : [ "major", "minor", "patch" ]
    },
    "ProcessingContext" : {
      "type" : "object",
      "additionalProperties" : false,
      "properties" : {
        "associationId" : {
          "type" : "string"
        },
        "checkpoint" : {
          "type" : "string"
        },
        "unitOfWorkId" : {
          "type" : "string"
        },
        "clientRequestId" : {
          "type" : "string"
        },
        "processingEntity" : {
          "type" : "string"
        }
      }
    },
    "Object" : {
      "type" : "object",
      "additionalProperties" : false,
      "properties" : { }
    }
  }
}
S3 File Manager System Events
Name Description Type Level Payload schema

S3FailureEvent

Event raised when S3 initialisation or operation fails

TECHNICAL

ERROR

 
{
  "$schema" : "http://json-schema.org/draft-04/schema#",
  "title" : "Payload",
  "type" : "object",
  "additionalProperties" : false,
  "properties" : {
    "message" : {
      "type" : "string"
    },
    "component" : {
      "type" : "string"
    },
    "failureType" : {
      "type" : "string",
      "enum" : [ "CREDENTIALS_ERROR", "CLIENT_INITIALISATION_ERROR" ]
    },
    "exceptionMessage" : {
      "type" : "string"
    },
    "exceptionType" : {
      "type" : "string"
    }
  }
}

Examples

Example Events

S3 Client Initialisation Error

Field Descriptions
Client Initialisation Error Event
Field Description

name

Unique identifier of the event type, always 'S3FailureEvent' for S3-related failures

level

Severity level of the event (possible values: ERROR, WARN, INFO)

type

Classification of the error (TECHNICAL for system-level issues)

source

Origin of the event, identifies the component raising the event

Metadata Fields

metadata.component

Name of the factory component responsible for client initialisation

metadata.failure_type

Specific categorization of the failure (CLIENT_INITIALISATION_ERROR for setup issues)

metadata.region

AWS region specified in the configuration

Payload Fields

payload.message

Human-readable description of the initialisation error

payload.component

Component identifier matching the metadata component

payload.failureType

Detailed failure classification

payload.exceptionMessage

Detailed error message describing the initialisation failure

payload.exceptionType

Fully qualified name of the Java exception

payload.configurationDetails

Configuration parameters that led to the initialisation failure

payload.configurationDetails.endpoint

AWS S3 endpoint URL that was configured

payload.configurationDetails.region

AWS region specified in the configuration
Example
{
  "name": "S3FailureEvent",
  "level": "ERROR",
  "type": "TECHNICAL",
  "source": "S3FileManager",
  "metadata": {
    "component": "S3ClientFactory",
    "failure_type": "CLIENT_INITIALISATION_ERROR",
    "region": "eu-central-1"
  },
  "payload": {
    "message": "Failed to initialise S3 client",
    "component": "S3ClientFactory",
    "failureType": "CLIENT_INITIALISATION_ERROR",
    "exceptionMessage": "Unable to initialise AWS S3 client: Invalid endpoint configuration",
    "exceptionType": "com.amazonaws.SdkClientException",
    "configurationDetails": {
      "endpoint": "https://s3.invalid-region.amazonaws.com",
      "region": "eu-central-1"
    }
  }
}

S3 Credentials Error

Field Descriptions
Credential Error Event
Field Description

name

Unique identifier of the event type, always 'S3FailureEvent' for S3-related failures

level

Severity level of the event (possible values: ERROR, WARN, INFO)

type

Classification of the error (TECHNICAL for system-level issues)

source

Origin of the event, identifies the component raising the event

Metadata Fields

metadata.component

Name of the system component that encountered the error

metadata.failure_type

Specific categorization of the failure (CREDENTIALS_ERROR for authentication issues)

metadata.region

AWS region where the operation was attempted

metadata.bucket

Name of the S3 bucket involved in the operation

Payload Fields

payload.message

Human-readable description of the error

payload.component

Component identifier matching the metadata component

payload.failureType

Detailed failure classification

payload.exceptionMessage

Original error message from AWS service

payload.exceptionType

Fully qualified name of the Java exception

payload.bucket

Target S3 bucket name

payload.key

Path to the file in S3 that was being accessed
Example
{
  "name": "S3FailureEvent",
  "level": "ERROR",
  "type": "TECHNICAL",
  "source": "S3FileManager",
  "metadata": {
    "component": "S3FileManager",
    "failure_type": "CREDENTIALS_ERROR",
    "region": "eu-central-1",
    "bucket": "example-bucket"
  },
  "payload": {
    "message": "Failed to authenticate with AWS S3",
    "component": "S3FileManager",
    "failureType": "CREDENTIALS_ERROR",
    "exceptionMessage": "The security token included in the request is invalid",
    "exceptionType": "com.amazonaws.services.s3.model.AmazonS3Exception",
    "bucket": "example-bucket",
    "key": "path/to/file.txt"
  }
}