kafka-to-postgresql

The technical documentation of the microservice kafka-to-postgresql, which consumes messages from a Kafka topic and writes them up to a PostgreSQL database.

By default, it sets up two Kafka consumers, one for high throughput and one for high integrity.

High throughput

This Kafka listener is usually configured to listen in on the processValue topics.

High integrity

This Kafka listener is usually configured to listen in on the other topics.

Environment variables

Variable nameDescriptionTypePossible valuesExample value
LOGGING_LEVELDefines which logging level is used, mostly relevant for developers. If logging level is not DEVELOPMENT, default logging will be usedstringanyDEVELOPMENT
DRY_RUNIf set to true, the microservice will not write to the databasebooltrue, falsetrue
POSTGRES_HOSTHostname of the PostgreSQL databasestring
POSTGRES_USERThe username to use for PostgreSQL connectionsstring
POSTGRES_PASSWORDThe password to use for PostgreSQL connectionsstring
POSTGRES_DATABASEThe name of the PostgreSQL databasestring
POSTGRES_SSLMODEIf set to true, the PostgreSQL connection will use SSLstring
KAFKA_BOOTSTRAP_SERVERURL of the Kafka broker used, port is requiredstringalllocalhost:9092
KAFKA_SSL_KEY_PASSWORDKey password to decode the SSL private keystringanychangeme
MEMORY_REQUEST``

Program flow

The graphic below shows the program flow of the microservice.

Kafka-to-postgres-flow

Data flow

High Integrity

The graphic below shows the flow for an example High Integrity message

high-integrity-data-flow

Last modified November 23, 2022: Feat/linkfixes (#176) (120bad7)