Open-Source Docker Container for Seamless OPC-UA Integration with Unified Namespace (MQTT / Kafka)

Connecting PLCs to a Unified Namespace — by subscribing to change events or polling data at regular intervals and pushing it into a MQTT or Kafka broker — can be a daunting task.

At the United Manufacturing Hub, we prioritize building on existing solutions rather than reinventing the wheel. After evaluating commercial offerings and finding them unsatisfactory in terms of performance and limitations, we decided to create the OPC-UA plugin for benthos.

In this article, we will share our journey in developing this solution, introduce the plugin's features and provide a usage guide . We will also discuss our future plans for this plugin.

Our Journey to Developing the OPC-UA Plugin for Benthos

Our team at United Manufacturing Hub sought a reliable and efficient method to transfer data from OPC-UA to a MQTT or Kafka broker. After evaluating multiple commercial solutions, we were disappointed with their limitations. This prompted us to develop our own solution—the OPC-UA plugin for benthos.

Requirements

We based our requirements on customer needs and our experience in configuring and managing large-scale IT / OT infrastructures. In summary, the solution must meet our criteria for Reliability, Scalability and Maintainability:

General

• Must be a single, independent building block; complex "frameworks" like Node-RED are too intricate and change too rapidly.
• Must handle the entire complexity of OPC-UA—simply point it to the server, specify what data it should read, and the data should appear in the Unified Namespace (MQTT / Kafka).
• (Optional) Should include options to transform and normalize messages incl. Report-by-Exception (RBE) / Message deduplication
• (Optional) Should include a browsing functionality.

Reliability

• Clustering for reliability during upgrades/configuration changes (rolling upgrades).
• Guaranteed at-least-once data processing to ensure data integrity at all times.
• Compatibility with non-standard-conforming OPC-UA servers.

Scalability

• A plan for future scalability, partitioning, and defined maximum limits.

Maintainability

• Integration with modern IT landscapes (Docker) and compatibility with edge, on-premise, and cloud environments (Docker container image available, support for Kubernetes).
• Remote configuration, versioning, and disaster recovery for all components. (see also our approach to DevOps and immutable infrastructure). Best-case is that the Docker container is only configurable via a single configuration file, which then can be versioned and archived.
• Proper metrics and logging (e.g., to stdout) for connection with our Management Console and alerting in case of issues (service crashes, data disruptions, connection loss, etc.).

Testing Existing Solutions

At the United Manufacturing Hub, we prefer using existing software and avoid reinventing the wheel. We evaluated numerous commercial solutions, hoping to find one that met our requirements. However, we encountered various compatibility and functionality issues, as outlined below:

1. Softing edgeAggregator: Softing has a promising vision which is very much in line with our thinking and the market need. Therefore, we were very excited to learn more about it and try it out. edgeAggregator is a Docker app that helps streamlining OPC UA application architectures. Despite having a promising vision, we could unfortunately not get the edgeAggregator to read data from our OPC-UA simulator. Several attempts with their support team did not help solve the technical challenges either. We’ll keep a close eye on the product and will try it out in the near future again. If you want to try it out yourself, check out our tutorial on how to deploy it in Kubernetes. Upon reaching out for comment on the article, Softing started investigating further. We'll update the article if we have more information.
2. Kepware: Although functional, Kepware was not suitable for us or our customers due to its Windows dependency - see also our blog article on Kepware and how to overcome its limitations.
3. ThingWorx Kepware Edge: Despite being Linux-compatible, this solution seemed outdated and didn't work with Docker. Also, we have not been able to get it to start in our Linux Virtual Machine (WSL). We appreciate input and feedback in case someone made it work.
4. Matrikon MQTT Publisher: Though it came in a Docker container, it only worked with Matrikon software, rendering it unsuitable for extracting data from a non-Matrikon OPC-UA server.

We needed a standalone building block, not an entire framework (such as OPC Router or HighByte). Node-RED, despite working well, didn't fit that requirement either.

The closest tool we discovered is Telegraf from InfluxData, which is also open-source. It offers both polling and subscribing capabilities, and it can transmit data to MQTT and Kafka. However, it lacks an "auto-browsing" feature that allows users to simply direct the tool toward the desired nodes, which it would then browse and publish the data to the Unified Namespace. Instead, users must specify each node individually, which can be rather cumbersome.

Other tools exist, such as Neuron, which we did not consider further for various reasons, but which could be an option for other users.

Developing Our Own Solution

Faced with the lack of suitable options, our team chose to create a bespoke solution utilizing benthos, a tool we have employed extensively for data contextualization and stream processing within the United Manufacturing Hub.

Although it lacks a user interface (with the exception of Benthos Studio), benthos is remarkably user-friendly and dependable, offering not only a wide range of connectors but also numerous message processing features, such as report-by-exception, deduplication, schema enforcement, compression, caching integration with redis/memcached/etc. and multiple failure paths.

Leveraging the open-source OPC-UA library (https://github.com/gopcua/opcua, same as telegraf), which was jointly developed and maintained by multiple companies, we created then a benthos plugin.

Features

The benthos-umh Docker container offers the following features:

• Simple deployment in Docker, docker-compose, and Kubernetes
• Connects to an OPC-UA server, browses selected nodes, and forwards all sub-nodes in 1-second intervals
• Supports a wide range of outputs, from the Unified Namespace (MQTT and Kafka) to HTTP, AMQP, Redis, NATS, SQL, MongoDB, Cassandra, or AWS S3. Check out the official benthos output library
• Fully customizable messages using the benthos processor library: implement Report-by-Exception (RBE) / message deduplication, modify payloads and add timestamps using bloblang, apply protobuf (and therefore SparkplugB), and explore many more options.
• Integrates with modern IT landscape, providing metrics, logging, tracing, versionable configuration, and more
• Entirely open-source (Apache 2.0) and free-to-use

How to use

The benthos-umh image can be used in standalone mode, together with the United Manufacturing Hub, or in advanced mode. The only requirement is that you need to be able to deploy Docker containers.

Standalone ("docker run" & MQTT)

To connect to the OPC-UA broker, read out nodes and send them to MQTT, follow these instructions:

1. Create a new file called benthos.yaml with the provided content

   ---
   input:
     opcua:
       endpoint: 'opc.tcp://localhost:46010'
       nodeIDs: ['ns=2;s=IoTSensors']
   
   pipeline:
     processors:
       - bloblang: |
           root = {
             meta("opcua_path"): this,
             "timestamp_unix": timestamp_unix()
           }
   
   output:
     mqtt:
       urls:
         - 'localhost:1883'
       topic: 'ia/raw/opcuasimulator/${! meta("opcua_path") }'
       client_id: 'benthos-umh'
   
   

2. Execute the docker run command to start a new benthos-umh container

   docker run --rm --network="host" -v '<absolute path to your file>/benthos.yaml:/benthos.yaml' ghcr.io/united-manufacturing-hub/benthos-umh:latest
   

With the United Manufacturing Hub (Kubernetes & Kafka)

To use the Benthos-UMH image with the United Manufacturing Hub and its OPC-UA simulator, simply deploy the provided Kubernetes manifests in UMHLens/OpenLens: (if you are unsure how, check out this tutorial):

apiVersion: v1
kind: ConfigMap
metadata:
  name: benthos-1-config
  namespace: united-manufacturing-hub
  labels:
    app: benthos-1
data:
  benthos.yaml: |-
    input:
      umh_input_opcuasimulator: {}
    pipeline:
      processors:
        - bloblang: |
            root = {
              meta("opcua_path"): this,
              "timestamp_unix": timestamp_unix()
            }
    output:
      umh_output: 
        topic: 'ia.raw.${! meta("opcua_path") }'
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: benthos-1-deployment
  namespace: united-manufacturing-hub
  labels:
    app: benthos-1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: benthos-1
  template:
    metadata:
      labels:
        app: benthos-1
    spec:
      containers:
        - name: benthos-1
          image: "ghcr.io/united-manufacturing-hub/benthos-umh:latest"
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 4195
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /ping
              port: http
          readinessProbe:
            httpGet:
              path: /ready
              port: http
          volumeMounts:
            - name: config
              mountPath: "/benthos.yaml"
              subPath: "benthos.yaml"
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: benthos-1-config

Summary and Next Steps

Our goal is to provide IT and OT engineers with a reliable and efficient solution that can be trusted for their production processes and infrastructure management. By using the benthos-umh container and the United Manufacturing Hub in general, engineers can enhance their integration of IT and OT tools while avoiding vendor lock-in and streamlining their data management processes.

In the weeks and months ahead, we plan to enhance our OPC-UA plugin for benthos with the following improvements:

• Exposing more OPC-UA settings from the library (e.g., certificates)
• Adding a customizable poll interval (currently fixed at 1 second)
• Switching from the current "polling" to a "subscribe" data extraction method, which fits better into the overall Pub/Sub pattern of the Unified Namespace
• Test it on more OPC-UA servers (this is also where you can come in)

Furthermore, we are planning to do this with other manufacturing protocols such as Siemens S7, Modbus and many more as well.

By offering our solution as open-source, we aim to address any skepticism that may arise from evaluating other providers before developing our own product. Our ultimate goal is to merge our plugin into the main benthos project, further solidifying its position as a reliable and efficient solution.

Feel free to provide us feedback on our LinkedIn post or in our Discord channel. And make sure to star the benthos-umh project on GitHub.

Please note that we reached out to the vendors of the tested software (PTC and Matrikon) for comments on our evaluation, but we have not received any responses to date.