v0.9.14: Streamlining Operations with Redpanda & Optimized Resource Usage

Jeremy Theocharis -

Major Changes

Our main focus in this update is to boost the system's performance and make it easier to maintain. We've made two significant changes in this area:

1. Replacement of Apache Kafka with Redpanda

We have replaced Apache Kafka with the fully Kafka-compatible Redpanda. This change aims to simplify the setup and maintenance processes. Redpanda maintains compatibility with Kafka protocols, APIs, and clients, but offers enhanced ease of use. The transition from Apache Kafka to Redpanda will not affect your day-to-day operations, but will offer a smoother and more streamlined experience.

2. Resource Usage Optimization

In order to deliver more value, we have optimized resource usage. You can now send at least 500 messages per second with just 4 cores and 8GB of RAM. This adjustment will ensure more efficient operations, even with limited resources.

Backend Work

There's been substantial work done behind the scenes to future-proof the United Manufacturing Hub:

Microservices Migration

Our microservices have been migrated to use the Sarama Kafka library. This move is part of our efforts to support ARM-based devices in the future.

Container Registry Move

In order to better manage our container images and provide an improved experience for our developers, we have moved our container registry from Docker Hub to GitHub Container Registry. For now, we will continue publishing images to Docker Hub, but will eventually deprecate the old registry.

Other Enhancements

Additional improvements have been made to enhance the system's stability and functionality:

  • A new test build has been implemented to detect race conditions in the codebase, contributing to the overall stability of the United Manufacturing Hub.
  • All our custom images now run as non-root by default, except for the ones requiring root privileges.
  • The custom microservices now allow changing the type of Service used to expose them by setting the 'serviceType' field.
  • An SQL trigger function has been added to delete duplicate records from the 'statetable' table after insertion.
  • The validation of environment variables in the codebase has been enhanced.
  • It's now possible to set the aggregation interval when calculating the throughput of an asset.
  • We have updated various dependencies to their latest versions.

We believe these changes will significantly contribute to the system's overall functionality and usability. Stay tuned for more updates and enhancements.