S3 bucket File Ingestion Configuration

Overview

This is default configuration which will work, but some settings are mandatory as well (as credentials which must be set).

Configuration Options

Key Description Default Value

ipf.csm-reachability.file-ingestion.s3.enabled

Toggles the s3 bucket file ingestion.

false

ipf.csm-reachability.file-ingestion.notification-service.kafka.producer.topic

Name of the topic for producer

FILE_PROCESSED_NOTIFICATION

ipf.csm-reachability.file-ingestion.notification-service.kafka.producer.restart-settings

Restart settings to tell the connector what to do on failover.

default connector values

ipf.csm-reachability.file-ingestion.notification-service.kafka.producer.kafka-clients.group.id

Group ID for the producer.

file-processing-notification-group

ipf.csm-reachability.file-ingestion.notification-service.kafka.consumer.topic

Name of the topic for consumer.

FILE_AVAILABLE_NOTIFICATION

ipf.csm-reachability.file-ingestion.notification-service.kafka.consumer.restart-settings

Restart settings to tell the connector what to do on failover.

default connector values

ipf.csm-reachability.file-ingestion.notification-service.kafka.consumer.kafka-clients.group.id

Group ID for the consumer.

file-available-notification-group

ipf.file-manager.s3.endpoint-url

Endpoint url for AWS S3 bucket

{bucket-name}.s3.{region}.amazonaws.com

For the list of all other S3-related configurations, please refer to: S3 File Manager

Example of hocon configuration:

The csm-reachability-file-notification-s3 configuration example:

ipf.file-manager.s3 {
  region = "us-east-1"
  upload-parallelism = 1
  credentials {
    access-key-id = "accessKey"
    secret-access-key = "secretAccessKey"
  }
  resiliency-settings {
    # Determines the maximum number of retries to be made. Note that this includes the first failed attempt.
    max-attempts = 2
    # Retry if HTTP error code is in the list
    retryable-status-codes = [500, 503]
    attempt-timeout = 2s
    call-timeout = 3s
  }
}

ipf.file-manager.s3 {
  region = "us-east-1"
  upload-parallelism = 1
  credentials {
    access-key-id = "accessKey"
    secret-access-key = "secretAccessKey"
  }

  resiliency {
    retry = 3
    api-call-timeout = 10s # duration
  }
}