kafka-to-postgresql

The technical documentation of the microservice kafka-to-postgresql, which consumes messages from a Kafka topic and writes them up to a PostgreSQL database.

less than a minute

By default, it sets up two Kafka consumers, one for high throughput and one for high integrity.

High throughput

This Kafka listener is usually configured to listen in on the processValue topics.

High integrity

This Kafka listener is usually configured to listen in on the other topics.

Environment variables

Variable name	Description	Type	Possible values	Example value
`LOGGING_LEVEL`	Defines which logging level is used, mostly relevant for developers. If logging level is not `DEVELOPMENT`, default logging will be used	`string`	any	`DEVELOPMENT`
`DRY_RUN`	If set to true, the microservice will not write to the database	`bool`	true, false	true
`POSTGRES_HOST`	Hostname of the PostgreSQL database	`string`
`POSTGRES_USER`	The username to use for PostgreSQL connections	`string`
`POSTGRES_PASSWORD`	The password to use for PostgreSQL connections	`string`
`POSTGRES_DATABASE`	The name of the PostgreSQL database	`string`
`POSTGRES_SSLMODE`	If set to true, the PostgreSQL connection will use SSL	`string`
`KAFKA_BOOTSTRAP_SERVER`	URL of the Kafka broker used, port is required	`string`	all	localhost:9092
`KAFKA_SSL_KEY_PASSWORD`	Key password to decode the SSL private key	`string`	any	changeme
`MEMORY_REQUEST`		``

Program flow

The graphic below shows the program flow of the microservice.

Kafka-to-postgres-flow

Data flow

High Integrity

The graphic below shows the flow for an example High Integrity message

high-integrity-data-flow

Last modified October 5, 2022: Historian article (#107) (5f30dc0)