Documentation for a newer release is available. View Latest

IPF Application Data Considerations

This page is written to help explain the components of an IPF solution, where data is present and where that data is exposed for client usage:

Diagram - shows a typical IPF solution and the data touch points, each labelled/numbered.
Table - the table which follows gives a description of each data type, intended purpose and references to IPF developer docs for further information.

Example IPF Solution

Data Types and Usage

Label	Data	Purpose	Good For	Access	Technology	Reference
1	Event Journal	Persistence of Akka state by persisting events for recovery/restart	Internal only	IPF Flows only	MongoDB	Should not be accessed outside of IPF Core software processing
2	Processing Data	Data belonging, or resulting from, a transaction flow, such as a payment, is egressed by IPF processing nodes throughout its lifecycle.	Sending domain event data to other systems/DBs - for audit, warehousing, monitoring, MI, reporting, analytics	Processing Data interface to other infrastructure / DBs	Kafka & Java Impl	IPF Processing Data
3	ODS	Perform activities like tracking & tracing of payments, exception handling (e.g. understand reason for failure) as well as overall monitoring of payment processes.	Payments Business and Ops activities - query state of payment, domain event info, payment details	ODS Ingestion & Inquiry API via UI (or client app)	MongoDB	IPF Operational Data Store - ODS
4	Technical Metrics	Exposed for the purpose of Prometheus and Grafana integration using sampling over a set interval	Understanding/monitoring processing metric trends	Metrics are exposed via a Prometheus HTTP server	Prometheus/Grafana	Time Series Metrics
5	System Events	Technical events- BAU and error	Access to step by step processing within an IPF flow, technical level info	Processing Data or System Events Exporter	Kafka & Java Impl	System Events & IPF System Event Exporter Spring Boot starter packs
6	Application Logs	Log level integration to banks monitoring tools - e.g. ELK / Dynatrace	Log driven alerting & query	Log integration	Logs to ELK or Splunk	Logging

Label

Data

Purpose

Good For

Access

Technology

Reference

Event Journal

Persistence of Akka state by persisting events for recovery/restart

Internal only

IPF Flows only

MongoDB

Should not be accessed outside of IPF Core software processing

Processing Data

Data belonging, or resulting from, a transaction flow, such as a payment, is egressed by IPF processing nodes throughout its lifecycle.

Sending domain event data to other systems/DBs - for audit, warehousing, monitoring, MI, reporting, analytics

Processing Data interface to other infrastructure / DBs

Kafka & Java Impl

IPF Processing Data

ODS

Perform activities like tracking & tracing of payments, exception handling (e.g. understand reason for failure) as well as overall monitoring of payment processes.

Payments Business and Ops activities - query state of payment, domain event info, payment details

ODS Ingestion & Inquiry API via UI (or client app)

MongoDB

IPF Operational Data Store - ODS

Technical Metrics

Exposed for the purpose of Prometheus and Grafana integration using sampling over a set interval

Understanding/monitoring processing metric trends

Metrics are exposed via a Prometheus HTTP server

Prometheus/Grafana

Time Series Metrics

System Events

Technical events- BAU and error

Access to step by step processing within an IPF flow, technical level info

Processing Data or System Events Exporter

Kafka & Java Impl

System Events & IPF System Event Exporter Spring Boot starter packs

Application Logs

Log level integration to banks monitoring tools - e.g. ELK / Dynatrace

Log driven alerting & query

Log integration

Logs to ELK or Splunk

Logging

More on Data Processing

You may also be interested in looking at the Data Processing & Persistence section of the site for more information on Data Identifiers, Processing Data, System Events, ODS and Data Models within IPF.