Request for comment: new protocol converter configuration

This is a Request for Comment (RFC) in order to get feedback from the community on a potential new way of configuring protocol converters. The idea is to provide a "configuration only" interface for OT people, without loosing any of the flexibility that the current bloblang logic provides you. It would be a new "processing" plugin for benthos-umh.

💡

Scroll further down for a full example!

Configuration Overview

The configuration is divided into several sections:

Basic Settings: Essential configurations for beginners.
Conditions: Advanced rules for tag processing.
Custom Protocol Settings: Protocol-specific overrides.
Advanced Processing: For custom schemas and complex processing.

1. Basic Settings

These settings are intended for beginners and cover the essential configurations required to get started.

1.1. Default Protocol Settings

tagProcessor:
  defaultProtocolSettings: opcua  # Set the default protocol (e.g., opcua, modbus)

Purpose: Specifies the default protocol to apply standard settings automatically.
Usage: Set to the protocol of the devices you are interfacing with.

1.2. General Options

tagProcessor:
  defaultProtocolSettings: opcua

  # Basic grouping
  groupTo:
    enterprise: "YourEnterprise"  # Set your enterprise name
    site: "DefaultSite"           # Set your site name

Purpose: Defines the basic grouping hierarchy for your tags.
Usage: Replace "YourEnterprise" and "DefaultSite" with your actual enterprise and site names.

1.3. Renaming Tags

tagProcessor:
	renaming:
	  "OriginalTagName1": "NewTagName1"
	  "OriginalTagName2": "NewTagName2"

Purpose: Simplifies complex or vendor-specific tag names.
Usage: Provide a mapping of original tag names to new, simplified names.

1.4. Regex-Based Replacements

tagProcessor:
	replaceAll:
	  tagFolder:
	    "/[^a-zA-Z0-9_\\/]/g": ""  # Removes special characters except '/'

Purpose: Sanitizes tag folders or names using regex patterns.
Usage: Specify the regex pattern and replacement string.

2. Conditions

This section allows advanced users to define conditional rules for tag processing.

2.1. Conditions Structure

conditions:
  - if: <condition_expression>
    groupTo:
      <level>: <value>
    tagOverrides:
      <tagProperty>: <value>

Purpose: Applies specific processing rules when certain conditions are met.
Components:
- if: A bloblang condition expression that evaluates to true or false. Boolean operators such as && and || can be used to apply multiple conditions.
- groupTo: Specifies the grouping hierarchy based on the condition.
- tagOverrides: Optionally overrides tag properties such as the tagFolder or the tagName.
Edge Cases: the conditions are executed from top to bottom and a message / a tag will be matched against all conditions. It starts with the “basic grouping” (see tagProcessor.groupTo) and then every condition and its overrides are executed in order.

2.2. Examples

2.2.1. Grouping by Modbus Slave ID

conditions:
  - if: meta("modbus_slave_id") == 10
    groupTo:
      area: "AreaA"
      line: "Line1"

Explanation: If the modbus_slave_id is 10, group the tags under AreaA and Line1.

2.2.2. Overriding Tag Names

conditions:
  - if: meta("modbus_address") == "B123"
    tagOverrides:
      tagName: "Temperature"

Explanation: If the modbus_address is "B123", override the tag name to "Temperature".

3. Custom Protocol Settings

Advanced users can override default protocol settings with custom configurations.

3.1. Custom Protocol Settings Structure

customProtocolSettings:
  tagFolder: meta("<protocol_specific_metadata>")
  tagName: meta("<protocol_specific_metadata>")
  tagType: meta("<protocol_specific_metadata>")

Purpose: To specify protocol-specific metadata for tag properties.
Usage: Replace <protocol_specific_metadata> with the appropriate metadata keys for your protocol.

3.2. Example for OPC UA

customProtocolSettings:
  tagFolder: meta("opcua_tag_group")
  tagName: meta("opcua_tag_name")
  tagType: meta("opcua_tag_type")

Explanation: Overrides the default OPC UA tag folder, name, and type with the specified metadata.

4. Advanced Processing

For scenarios requiring custom schemas or complex processing logic, advanced users can enable Bloblang scripting. When advancedProcessing is enabled, all other configurations are ignored.

4.1. Enabling Advanced Processing

advancedProcessing:
  enabled: true
  script: |
  
    # Add your Bloblang script here
    
    meta kafka_topic = "umh.v1.<yourEnterprise>._historian" # required value
    
    root = { # required value as well
	    "timestamp_ms": 123,
	    "yourTagName": this.content().string()
    }

Purpose: Allows the use of Bloblang scripts for custom processing.
Usage: Set enabled to true and provide your Bloblang script under script.

4.2. Important Notes

Override Behavior: When advancedProcessing is enabled, all other settings in the configuration are ignored.
Fallback Option: Use this option if you need to implement custom schemas not supported by the default configuration.

5. The Output

5.1. Expected Inputs

The tag processor outputs messages in the _historian schema, formatted as JSON.

Message Structure:

this.timestamp_ms: Unix timestamp in milliseconds.
this.<tagName>: The tag value, enforced with <tagType>.
this.<metadata_name>: Additional metadata key-value pairs.
meta("kafka_topic"): The Kafka topic, structured according to ISA-95 hierarchy (enterprise, site, area, etc.).

Example Output:

{
  "timestamp_ms": 1633072800000,
  "Temperature": 75.5,
  "serial_number": "12345",
  "meta": {
    "kafka_topic": "enterprise/site/area/line"
  }
}

Example Metadata:

kafka_topic: "umh.v1.<enterprise>.<site>.<area>._historian.<tagFolder>.<tagName>

Explanation:
- timestamp_ms: The timestamp of the data point.
- Temperature: The tag name with its value.
- serial_number: An example of additional metadata.
- meta.kafka_topic: The topic where the message will be published.

5.2. Custom Schemas

If you require a custom schema that differs from the _historian schema, you must use Advanced Processing with Bloblang scripting.
Note: When using advanced processing, ensure your script outputs the data in your desired format.

6. Configuration Flow

To make the configuration intuitive, it's arranged in the following order:

Basic Settings: Start with default protocol settings and basic grouping, renaming and regex replacements.
Conditions: Define any conditional processing rules.
Custom Protocol Settings: Override protocol defaults if necessary.
Advanced Processing: Use Bloblang scripting for custom needs (overrides all previous settings).

7. Example Configuration

Here is an example that incorporates all the discussed elements:

tagProcessor:
  defaultProtocolSettings: opcua  # Basic Setting

  groupTo:                        # Basic Grouping
    enterprise: "MyEnterprise"
    site: "MainSite"

  renaming:                       # Tag Renaming
    "ComplexTagName1": "SimpleName1"
    "ComplexTagName2": "SimpleName2"

  replaceAll:                     # Regex Replacements
    tagFolder:
      "/[^a-zA-Z0-9_\\/]/g": ""   # Sanitize tag folders
    tagName:
      "/[^a-zA-Z0-9_]/g": ""      # Sanitize tag names
  
  conditions:                     # Advanced Conditions
    - if: always()
      groupTo:
        area: "DefaultArea"
      tagOverrides:
        tagFolder: meta("opcua_tag_group")
        tagName: meta("opcua_tag_name")
        tagType: meta("opcua_tag_type")
        tagMetadata:
          serial_number: "SN12345"

    - if: meta("modbus_slave_id") == 10
      groupTo:
        area: "ProductionArea"
        line: "AssemblyLine1"

  customProtocolSettings:         # Protocol Overrides
    tagFolder: meta("opcua_tag_group")
    tagName: meta("opcua_tag_name")
    tagType: meta("opcua_tag_type")

  advancedProcessing:             # Advanced Processing (Disabled)
    enabled: false
    script: |
      root = this
      # Custom Bloblang script

8. Templating and Custom Variables

To enhance flexibility and usability, the configuration supports templating using the Go templating language. This allows IT personnel or system integrators to create templates with custom variables. OT personnel or end-users can then fill in these variables via the Management Console UI under the Connections section.

By utilizing templating, you can create more complex configurations that remain maintainable by non-IT personnel, ensuring that advanced setups are still accessible and manageable.

8.1. Purpose of Templating

Simplify Configuration for End-Users: Templating abstracts the complexity of the configuration file, allowing end-users to input only the necessary variables.
Reuse and Scalability: Templates can be reused across multiple deployments, promoting consistency and reducing configuration errors.
Customization: Enables the creation of customized configurations tailored to specific use cases without modifying the underlying logic.

8.2. How Templating Works

Template Creation: An IT professional or system integrator creates a configuration template using Go templating syntax, defining placeholders for variables.
Variable Definition: OT personnel or end-users provide values for these variables via the Management Console UI.
Template Rendering: The system processes the template, replacing placeholders with actual values to generate the final configuration.

8.3. Defining Custom Variables

Custom variables are defined by the template creator and are intended to be filled out by the end-user.

Example of Custom Variables Input in Management Console UI:

customVariables:
  siteName: "TestSite"
  opcuaFolderNodeID: "ns=2;i=5567"

Explanation:
- siteName: The name of the site, provided by the end-user.
- opcuaFolderNodeID: The OPC UA folder Node ID to filter tags.

8.4. Creating a Protocol Converter Template

Here's how a template might look with custom variables:

tagProcessor:
  defaultProtocolSettings: opcua  # Basic Setting

  groupTo:                        # Basic Grouping
    enterprise: "MyEnterprise"
    site: "{{ .customVariables.siteName }}"

  conditions:
    - if: meta("opcua_nodeid") != "{{ .customVariables.opcuaFolderNodeID }}"
      tagOverrides:
        tagValue: deleted()

Template Components:
- {{ .customVariables.siteName }}: Placeholder that will be replaced with the actual siteName provided by the user.
- {{ .customVariables.opcuaFolderNodeID }}: Placeholder for the opcuaFolderNodeID variable.
- Condition Logic: Deletes tags that do not match the specified OPC UA Node ID, effectively filtering tags.

8.5. Using the Template

Steps for End-Users:

Select Template: Choose the appropriate template in the Management Console under Connections.
Provide Variables: Input values for the custom variables as prompted by the UI.
Save Configuration: The system renders the template with the provided variables, generating the final configuration.

Example:

siteName: "FactoryA"
opcuaFolderNodeID: "ns=2;i=5567"

8.6. Benefits of Templating

User-Friendly: Allows non-IT personnel to configure complex settings without dealing with YAML syntax.
Consistency: Ensures standardized configurations across different deployments.
Efficiency: Saves time by reusing templates for similar setups, reducing duplication of effort.
Maintainability: Updates to the template automatically apply to all instances using it (when re-rendered), simplifying maintenance.

8.7. Advanced Templating Features

Templating supports advanced features that can further enhance configuration flexibility.

8.7.1. Conditional Logic

You can include conditional statements in your templates.

Example:

{{ if .customVariables.enableFiltering }}
conditions:
  - if: meta("opcua_nodeid") != "{{ .customVariables.opcuaFolderNodeID }}"
    tagOverrides:
      tagValue: deleted()
{{ end }}

Explanation: The conditions section is included only if enableFiltering is true.

8.7.2. Looping Constructs

You can iterate over lists or maps.

Example:

groupTo:
  enterprise: "MyEnterprise"
  site: "{{ .customVariables.siteName }}"
  {{ range $key, $value := .customVariables.additionalGroupings }}
  {{ $key }}: "{{ $value }}"
  {{ end }}

Explanation: Adds additional grouping levels based on user-provided key-value pairs.

8.8. Example Scenario

Scenario: A company has multiple sites with different OPC UA configurations but wants to maintain a consistent data processing structure.

Template Creation: IT creates a template with variables for siteName and opcuaFolderNodeID.
End-User Input: OT personnel at each site input their specific siteName and opcuaFolderNodeID via the UI.
Outcome: Each site has a customized yet consistent configuration, and OT personnel didn't need to understand the underlying YAML or templating syntax.

8.9. FAQs on Templating

Q1: Can end-users modify the template?

A: Generally, end-users provide variable values via the UI but do not modify the template itself. Template modifications are intended to be performed by IT personnel or system integrators.

Q2: What happens if a variable is missing or invalid?

A: The system should validate inputs and prompt the user to correct any missing or invalid variables before rendering the final configuration.

Q3: Can templates include advanced processing sections?

A: Yes, templates can include any configuration sections, including advancedProcessing. However, keep in mind that enabling advancedProcessing will override other configurations.

Note: For detailed guidance on Go templating syntax, refer to the Go Templating Documentation.

10. Additional Notes

Order of Processing: The configuration is processed in the order presented. Enabling advancedProcessing will override all other settings, including those defined in templates.
Error Handling: Ensure all metadata keys used in meta("<key>") and all custom variables are available and correctly formatted; otherwise, processing may fail.
Documentation: Refer to the UMH Management Console documentation for detailed explanations of each configuration option and templating guidelines.

11. FAQs

Q1: What happens if both `customProtocolSettings` and `advancedProcessing` are enabled?

Answer: If advancedProcessing.enabled is set to true, all other configurations, including customProtocolSettings, are ignored. Only the Bloblang script in advancedProcessing.script is executed.

Q2: Can I use `conditions` to perform tag renaming and replacements?

Answer: Yes, you can use conditions along with tagOverrides to rename tags or modify tag properties based on specific conditions.

Q3: How do I ensure that the outputs conform to the expected schema?

Answer: By default, the tag processor outputs messages in the _historian schema. Ensure your tag names, types, and metadata are correctly configured. If using advanced processing, your Bloblang script should output data in the desired format.

We are waiting for your comments! You can find the full discussion here on Discord