By clicking “Accept All Cookies”, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts. View our Privacy Policy for more information.

B) Data Modeling in the Unified Namespace: From Topic Hierachies over Payload Schemas to MQTT/Kafka

ℹ️
This is still work in progress. Feedback is appreciated.

In manufacturing, the Unified Namespace (UNS) is a powerful tool for facilitating communication between nodes in a network. This event-driven architecture operates on the principle that all data should be made available for consumption, regardless of whether there is an immediate consumer. This means that any node in the network can act as either a producer or a consumer, depending on the system's needs at a given time.

💡
ADR: Adoption of Unified Namespace in UMH

Context: Establishing a cohesive IT/OT architecture within UMH.

Challenge: Integrating a system that can handle the complex, data-intensive, and long-lived nature of manufacturing applications while promoting ease of use and adaptability.

Decision: Implementing a Unified Namespace (UNS) using a message broker as a core component of our system architecture​​.

Neglected Alternatives:
- Dataflow through Databases: Utilizing long-lived databases like Historians for data storage, which lacks real-time stream processing capabilities.
- Dataflow through Service Calls: Relies on inter-service communication, which can become unwieldy and lead to "spaghetti diagrams" in complex systems.

Objective: To develop an architecture that is reliable, scalable, and maintainable, allowing for real-time data processing and easy integration or removal of components​​.

Trade-off: Introducing a new component (message broker) into the system stack, potentially increasing complexity and the likelihood of system failure​​.

Mitigation Strategy: Ensuring robustness through hardware and software redundancy, and fostering maintainability with a system that is simple to operate, understand, and evolve.

Rationale:
- Reliability: Mitigating faults at hardware, software, and human levels.
- Scalability: Handling varying loads from numerous data sources and message sizes.
- Maintainability: Ensuring operational ease, system simplicity, and the flexibility to evolve without disruptions

Any manufacturing company intending to implement a UNS-based architecture should follow a series of steps:

Step 1: Connecting to Operational Technology (OT)

The data utilized in manufacturing can be broadly categorized into three types: relational, time-series, and semi-structured or unstructured.

Time-series data can be dispatched through two primary methods:

  1. Transmitting at consistent intervals to the message broker. This method includes details such as device uptime, but it can generate a substantial volume of data.
  2. The "report-by-exception" approach. Here, data is transmitted only when changes occur, effectively reducing the data volume. However, this method necessitates additional uptime information. For instance, it's crucial to determine whether the value didn't change or if the device was offline.

The conversion of relational data into events for transmission via a Unified Namespace (UNS) is recommended. This process entails subscribing to all changes in the existing relational table (inserts, updates, deletes) and transmitting these changes to the UNS. Examples of these events are "addOrder," "changeOrder," and "deleteOrder."

SQL triggers or Change Data Capture (CDC) tools like Debezium can be utilized to generate data change events. In some cases, Programmable Logic Controllers (PLCs) or shop floor systems may directly emit these change events. This strategy enables event replay, allowing for the reproduction of the exact SQL table state.

Just like relational data, semi-structured / unstructured data should be transmitted when changes occur. Contrary to popular belief, there are no restrictions on sending images through MQTT. The MQTT standard allows up to 256 MB per payload, while Kafka's default is 1MB, extendable to 10MB without significant performance issues. This capacity is sufficient for most semi-structured or unstructured data.

If data size exceeds these limits, consider splitting larger data, like videos, into smaller segments to ensure compatibility. Alternatively, if large payloads are required, store the data in blob storage like file disk or S3 compatible systems. Then, process only the data reference, such as the file path, via Kafka.

Ensure that no data modification occurs between the OT producer and the message broker to maintain modularity. Some industries, like pharmaceuticals, even mandate this due to regulatory requirements, such as GxP compliance. Downsampling or altering the data before sending it to the UNS could create issues. For example, if someone wants to access the raw data for a different analysis or a new AI model, modification would render this impossible.

💡
ADR: Building a Unified Namespace with MQTT and Kafka

Context: Developing scalable data processing in Industrial IoT within the United Manufacturing Hub (UMH).

Challenge: Creating an architecture that addresses the unique challenges in IIoT environments, such as real-time data processing, integration of diverse data sources, ensuring system resilience, and accommodating various data formats.

Decision: Implementing a hybrid architecture using MQTT for data collection and Kafka for data processing.

Neglected Alternatives:
- MQTT as a Service from Cloud Providers: Services like Azure IoT Hub were considered but not chosen due to their limitations in edge computing scenarios.
- NATS, and others: Despite its (potentially) technical superiority in messaging, NATS was passed over due to its weak side in processes and people. It is more difficult to use for OT engineers and will mostly require protocol converters (PLCs, IoT devices, etc. can speak MQTT natively). There is also potential skepticism from conservative non-IT managers about its longevity as there is no big company behind it.

Objective: To establish a robust, scalable, and user-friendly data processing architecture that can handle the complexities of IIoT environments, including variable data sources, high data throughput, and the need for reliable, real-time processing.

Trade-off: Increased complexity in integrating MQTT with Kafka.

Mitigation Strategy: Balancing MQTT's simplicity and widespread adoption in OT environments with Kafka's advanced processing capabilities and existing enterprise support.

Rationale:
- MQTT was chosen for its ease of use and wide acceptance among OT professionals, with many PLCs and sensors natively supporting it.
- Kafka addresses the process-heavy requirements of the architecture, benefiting from existing enterprise support and infrastructure.
- The hybrid approach addresses both the human (ease of use, familiarity) and process (scalability, fault tolerance) aspects, ensuring a robust solution for UMH’s diverse data processing needs.

The long version can be found here: https://learn.umh.app/blog/tools-techniques-for-scalable-data-processing-in-industrial-iot/

Step 2: Topic Hierarchy

The second step in the process requires the creation of a topic hierarchy. Ideally, this hierarchy should closely mirror the physical structure of the manufacturing plant or any existing asset naming system for two main reasons:

  1. Enhancing data point visibility and facilitating data browsing for OT engineers.
  2. Bolstering security. For instance, in multi-plant scenarios, access of a certain device could be restricted to specific plants or even more granular levels.

A large number of enterprises adopt the ISA-95 model, thus forming their topic structures based on this standard. However, it's important to note the existence of Sparkplug-B, a standard that outlines potential payload and topic structures. Despite its benefits, Sparkplug-B focuses solely on "Device Management" and doesn't align with the ISA95 model. Also, it relies on ProtoBuf, which, while straightforward for IT professionals, may complicate matters for those in OT. Given that most manufacturing devices are connected via Ethernet cables, the bandwidth savings provided by Sparkplug-B hardly justify the increased complexity.

💡
ADR: Unified Namespace Data Modeling Schema

Context: In the Unified Namespace for data modeling.

Challenge: Addressing the complexities of topic hierarchy and payload format.

Decision: We opted to develop our proprietary schema instead of adopting Sparkplug-B.

Neglected Alternative: Sparkplug-B was considered but ultimately not chosen.

Objective: Our goal is to ensure ISA95 compatibility and simplify the payload format to JSON, making it more accessible for OT professionals.

Trade-off: We acknowledge that deviating from established standards might initially cause discomfort among some users.

Mitigation Strategy: To alleviate this discomfort, we are committed to providing detailed explanations and justifications for our decision.

Rationale: The choice to develop a bespoke schema was driven by the need for a more intuitive and easily understandable structure for OT professionals, aligning closely with the ISA95 standards. While we recognize that this might challenge users accustomed to existing standards, our thorough explanatory approach aims to bridge this gap and demonstrate the benefits of our tailored solution.

Consider including the following elements in your topic structure for enhanced functionality:

  1. A version number to facilitate future changes.
  2. The client ID for granular access control, allowing a device to send data only to its own topics.
  3. The client ID for tracing and lineage purposes to pinpoint the exact source of each message.
  4. The tag name, and for larger quantities, tag groups, which can improve performance. For example, if a device sends large amounts of varied data, a specific microservice may only require a subset of this data. By filtering through tag groups, unnecessary burdens can be avoided.

Example from the United Manufacturing Hub

In the United Manufacturing Hub (UMH), we organize data using a specific topic structure that adheres to the ISA95 model. This structure looks like this:

umh/v1/enterprise/site/area/productionLine/workCell/originID/_schema

Topic Names and Rules: All parts of this structure, such as 'enterprise', 'site', 'area', etc., are flexible in terms of their naming. They can include letters (a-z, A-Z), numbers (0-9), and certain symbols (- and _). However, symbols like ., +, # or / are not used as they have special meanings in MQTT or Kafka.

Versioning Prefix: The umh/v1 at the beginning is obligatory. It ensures that the structure can evolve over time without causing confusion or compatibility issues.

ISA95 Compliance: The terms like 'enterprise', 'site', 'area', etc., are aligned with the ISA95 model, which is a standard for industrial systems. 'Enterprise' is the only mandatory term; others can be skipped if they don't apply, e.g., a room temperature sensor for a specific area.

Origin ID: This part identifies where the data is coming from. It could be a serial number, a MAC address, or even a software component like a Docker container. If multiple sources are involved, they're separated by underscores. Examples of originIDs: E588974, 00-80-41-ae-fd-7e, VM241_nodered_mes

The _schema Field: This is where IT and OT professionals need to pay close attention. The '_schema' field, identified by its leading underscore, is crucial for defining the type of data or the format being sent. In the UMH, we have default schemas like '_historian', '_analytics', and '_local', but users can add more as needed. The underscore is important for clarity in parsing the structure, especially when some elements like 'workCell' might be omitted.

  1. Schemas _historian and _analytics:
    • Validation of Messages: The United Manufacturing Hub (UMH) is programmed to process messages under the _historian and _analytics schemas only if they adhere to a valid schema format (see further below)
    • Handling Invalid Messages: Any message that is not in JSON format or does otherwise not meet the schema, even if sent to these schemas, will not be saved in the database nor forwarded to another broker. This ensures data integrity and consistency in processing.
  2. Schema _local:
    • Non-Processing Policy: Messages sent under the _local schema will not be processed by UMH. This schema is reserved for data that is intended to remain local and is not meant for forwarding or storing in the database.
  3. Other Schemas:
    • Forwarding without Storage: Messages falling under schemas other than _historian, _analytics, and _local will be forwarded to other brokers via bridges.
    • Independence from Structure and Payload: This forwarding occurs regardless of the specific topic structure following the _schema marker and irrespective of the payload format. However, UMH will not store these messages in its database.
💡
ADR: Topic Structure and Schema Management in UMH

Context: In the United Manufacturing Hub (UMH), managing data communication and storage.

Challenge: Establishing a robust and flexible topic structure for data handling that aligns with industrial standards.

Decision: We have chosen to structure our topics according to the ISA95 model with a specific format umh/v1/enterprise/site/area/productionLine/workCell/originID/_schema.

Neglected Alternative: Other less structured or non-standardized approaches to topic organization.

Objective: Our goals are to ensure data integrity, ease of understanding for OT professionals, and adaptability to future changes.

Trade-off: This decision might limit flexibility in some use cases, where a less structured approach could be beneficial.

Mitigation Strategy: By allowing customization within the '_schema' field and providing clear documentation on topic structure and rules, we aim to balance structure with flexibility.

Rationale:
- Versioning and Flexibility: The inclusion of a versioning prefix (umh/v1) ensures that our topic structure can evolve over time. This approach provides a balance between maintaining consistency and allowing for future enhancements.
- ISA95 Compliance: Aligning with the ISA95 model ensures that our system is in line with industry standards, making it more intuitive for professionals in the field.
- Origin ID and Schema Definition: The inclusion of an origin ID and a clear '_schema' field allows for precise data source identification and format definition, enhancing data management and processing.
- Schema Specific Handling:
- _historian and _analytics: These schemas are strictly processed only if messages comply with the defined format, ensuring data integrity and relevant processing.
- _local: This schema is designated for data that remains local, not intended for forwarding or storing, thereby providing flexibility for internal data handling.
- Other Schemas: Messages under other schemas are forwarded but not stored, allowing for data distribution without overloading the system database.

Default schema: _historian

Data under this schema gets stored in TimescaleDB, useful for time-series data analysis aka the Historian Feature.

Topic Structure:

  • The basic topic format under this schema is .../_historian or .../_historian/<tagGroup1>/<tagGroup2>/.../<tagGroupX>/<tagname>.
  • The use of tag groups (<tagGroupX>) is optional and allows for better organization of large numbers of tags (e.g., 10,000 tags from an asset). This organization facilitates easier navigation and reduces system load.
  • The <tagname> part of the topic is optional and can be used for more specific data categorization.

Payload Format:

  • Payloads in this schema are in JSON format.
  • Essential keys include:
    • timestamp_ms: A Unix timestamp in milliseconds, indicating when the message was created.
    • At least one additional key representing the data value, which can be Number, boolean or string. Note: the boolean must NOT be in quotes as it will be treated a string instead

Tag Grouping Methods:

  1. Using Underscores in Key Names: For example, spindle_axis_x. This results in the tag 'x' being categorized under the group 'axis', which is part of the 'spindle' group.
  2. Using Tags / Tag Groups in the Topic: The tag is placed before the key name, allowing for multiple group formations.
  3. Combining Both Methods: For instance, a message to .../_historian/spindle/axis with key x_pos will categorize 'pos' under 'x', which is under 'axis' and 'spindle'.

Examples:

  • Example 1: Sending a message with payload { "timestamp_ms": 1680698839098, "x_pos": 123, "x_speed": 456, "y_pos": 789, "y_speed": 001 } to the topic umh/v1/dcc/aachen/shopfloor/wristband/warping/_historian/spindle/axis will result in storing four tags for the given timestamp: spindle_axis_x_pos, spindle_axis_x_speed, spindle_axis_y_pos, and spindle_axis_y_speed
  • Example 2: A message with payload { "timestamp_ms": 1680698839098, "temperature": 23 } to umh/v1/dcc/aachen/_historian stores the temperature tag for the site 'aachen' for the given timestamp.
💡
ADR: Implementing the _historian Schema in UMH for Time-Series Data

Context: Within the United Manufacturing Hub (UMH), managing and storing time-series data.

Challenge: Developing an efficient and structured approach for organizing large volumes of time-series data.

Decision: We adopted the _historian schema, which utilizes TimescaleDB for data storage.

Neglected Alternative: Simpler, less structured data handling approaches.

Objective: Our goal is to facilitate detailed time-series data analysis while ensuring efficient data organization and reduced system load.

Trade-off: This structured approach might be more complex to set up initially compared to simpler methods.

Mitigation Strategy: We provide clear guidelines and examples for users to understand and effectively utilize this schema.

Rationale:
- Topic Structure: The _historian schema employs a flexible topic format (.../_historian or .../_historian/<tagGroup1>/.../<tagname>), allowing for both general and detailed data categorization.
- Tag Grouping Flexibility: The optional use of tag groups and tag names offers users the ability to organize large numbers of tags effectively. This flexibility enhances data management, especially for assets with numerous data points.
- Payload Format: Adopting JSON for payloads, with essential keys like timestamp_ms and at least one data value key, ensures a standardized and accessible format for data representation.
- Tag Grouping Methods:
- Underscore in Key Names: This method allows for intuitive categorization of data tags within hierarchical groups.
- Tags in Topic: Placing tags before key names in the topic string enables further grouping possibilities.
- Combination Approach: Merging both methods offers a comprehensive structure for complex data categorization.
2. _analytics

This is under heavy development - use at your own risk

The _analytics use-case should be used when you wish to leverage the Analytics Feature of the UMH (**https://umh.docs.umh.app/docs/features/analytics/**). This could be used to create production dashboards with automated calculated OEEs, drill-downs into stop-reasons, order overviews, machine states, and more. It is recommended to use for certain time-series data as well as for most relational data.

The payload always comes in JSON format, but the schema will vary based on the tag specified.

Jobs

TODO: definition of a job according to ISA-95. a job is an order to produce target-amount of a certain product-type (which has a target cycle time speed). It can be added, started and ended. Products, when produced, are connected with the job, so that one can get later an overview over the produced/scrapped pieces or amounts. In process, this is also called a batch.

  • tag: job/add (previously addOrder)
  • job-id
  • product-type
  • target-amount
  • tag: job/delete
  • job-id
  • job/start (previously startOrder)
  • job-id
  • timestamp-begin
  • job/end (previously endOrder)
  • job-id
  • timestamp-end

product type

  • product-type/add (previously addProduct)
  • product-id
  • cycle-time-in-seconds

product

  • product/add (previously count)
  • product-type-id
  • We do not know the product-type-id
  • It is already specified in the job
  • But when inserting into database, we need to know it
  • timestamp-end
  • timestamp-end is the “primary key” to identify a product for a workCell
  • note: there can only be one product produced per millisecond.
  • (optional) id
  • (optional) timestamp-begin
  • (optional) total-amount
  • (optional) scrap
  • product/overwrite (previously modifyProducedPieces)
  • timestamp-end
  • (optional) id
  • (optional) timestamp-begin
  • (optional) total-amount
  • (optional) scrap

Shifts

  • shift/add (previously addShift)
  • timestamp-begin
  • timestamp-end
  • shift/delete (previously deleteShift)
  • timestamp-begin

States

  • state/add (previously state
  • timestamp-begin
  • state
  • see also our state list, that one stays the same
  • state/overwrite (previously modifyState)
  • timestamp-begin
  • timestamp-end
  • state
  • state/activity (previously activity)
  • timestamp-begin
  • activity
  • note: there needs to be a microservice autoamtically calculating the state from activity and detectedAnomaly
  • state/reason (previously detectedAnomaly)
  • timestamp-begin
  • reason
  • note: there needs to be a microservice autoamtically calculating the state from activity and detectedAnomaly

Note: Items related to digital shadow are removed for now (uniqueProduct, scrapUniqueProduct, addParentToChild productTag, productTagString ). Recommendation is also removed for now (recommendation)

Step 3: Payload Structure

This step involves determining the structure of your payload data, which is subject to your production environment's specific needs such as speed, compatibility, simplicity, and capacity to handle complex data structures. Payload data can be defined using various data formats, including JSON, Protobuf, XML, or any other suitable data format.

Binary formats like Protobuf or Avro offer a structured and compressed approach, leading to bandwidth transmission savings. However, for instances where direct readability is crucial, XML, JSON, or YAML structures are preferred.

We recommend using JSON payloads, as in manufacturing bandwidth is rarely a concern. JSON enhances the capability of OT professionals to work with and understand messages. It enables you to directly open and interpret the messages in tools like MQTT Explorer or Node-RED. With formats like protobuf, such direct understanding isn't possible.

Example from the United Manufacturing Hub

See also the example further up

💡
ADR: Selection of JSON for Payload Structure in UMH

Context: In the United Manufacturing Hub (UMH), managing and processing data payloads.

Challenge: Choosing an optimal data format for payloads that balances speed, compatibility, simplicity, and the ability to handle complex data structures.

Decision: We have chosen JSON as the primary data format for payload structures.

Neglected Alternatives: Other data formats like Protobuf, XML, Avro, or YAML.

Objective: To enhance the accessibility and understanding of data payloads for Operational Technology (OT) professionals, while maintaining efficient data handling.

Trade-off: By opting for JSON, we forego the bandwidth efficiency and structured compression offered by binary formats like Protobuf or Avro.

Mitigation Strategy: Although JSON may not be as bandwidth-efficient as some binary formats, in manufacturing environments, bandwidth is typically not a primary concern. The ease of use and direct readability of JSON take precedence.

Rationale:
- Direct Readability and Simplicity: JSON’s format allows OT professionals to easily read and interpret messages directly in tools like MQTT Explorer or Node-RED, which is crucial in manufacturing environments.
- Manufacturing Environment Considerations: In the manufacturing context, the benefits of direct readability and simplicity often outweigh the need for the bandwidth efficiency provided by more compressed formats.
- Balance of Needs: JSON strikes a balance between the need for a format that can handle complex data structures and the need for simplicity and direct readability, making it a suitable choice for a wide range of applications within UMH.

Step 4: Bridging data efficiently between MQTT and Kafka

Because we opted in for using MQTT and Kafka, we also need to ensure that data is bridged between those. As this is an article about data modeling, we just leave the ADRs here, and proceed quickly to the other topics.

💡
ADR: Development of a Custom Bridge for UMH

Context: Essential bridging functionality required in the United Manufacturing Hub (UMH).

Challenge: Finding an appropriate bridge solution that aligns with UMH's open-source ethos.

Decision: Opting to develop an in-house bridge solution.

Neglected Alternatives: Off-the-shelf solutions with enterprise licenses, lacking community editions.

Objective: To ensure an open-source, seamlessly integrated bridging solution within UMH.

Trade-off: Potential reinvention of the wheel and challenges in achieving the reliability seen in established products.

Mitigation Strategy: Focusing on rigorous testing and integration practices to align the custom bridge's reliability with that of external solutions.

Rationale:
- Existing solutions did not offer open-source, community-driven versions, conflicting with UMH's open-source commitment.
- Custom development allows for tailored integration with other UMH components, offering a more cohesive system experience.
💡
ADR: Implementing a merge point feature in the Data Bridge for MQTT and Kafka Integration in UMH

Context: In the United Manufacturing Hub (UMH), managing data transfer between MQTT and Kafka brokers.

Challenge: Overcoming the limitations of both MQTT and Kafka in terms of topic management and scalability while ensuring data integrity and ease of use.

Decision: Implementing the Data Bridge microservice to transfer data between Kafka or MQTT brokers, adhering to the UNS data model.

Neglected Alternatives:
- Direct 1:1 Topic Mapping in Kafka: Not viable due to Kafka's limitations with a high number of topics.
- Simplified Topic Structure in MQTT: Rejected as it would hamper data browsing and understanding.

Objective: To create an efficient, scalable bridge that maintains a consistent topic strategy across MQTT and Kafka, catering to the operational needs of UMH while being accessible to OT professionals.

Trade-off: Balancing the granularity of topics to optimize performance and maintain clarity without complicating the system.

Mitigation Strategy: Utilizing the Data Bridge's capability to merge messages from multiple MQTT topics into a single Kafka topic, utilizing the message key to maintain source topic identification​​.

Rationale:
- The Data Bridge microservice serves as a crucial link between MQTT or Kafka brokers. It subscribes to topics using a wildcard pattern and processes messages according to the UNS data model. Where Kafka is the destination, the Data Bridge merges messages from multiple topics into a single Kafka topic, identified by the merge point configuration. For example, with a merge point of 4, it consolidates messages from various detailed topics like umh/v1/acme/anytown/foo/bar into a general topic umh.v1.acme.anytown while using the specific sub-topic paths (foo.bar, foo.baz, etc.) as message keys in Kafka. This design minimizes the overhead in Kafka associated with managing a large number of topics while retaining the granular detail within the message key, thus maintaining optimal performance and usability for both MQTT and Kafka within the UMH architecture.

Step 5: Enforcement

The final step in this process is enforcing the established standards and conventions throughout your system. This consistency ensures system integrity. The enforcement involves outlining security requirements, managing state and data flow, setting boundaries, and confirming that each device has the necessary authorization to exchange information.

Currently, enforcing message and topic schemas is not a feature available by default in MQTT. However, efforts are underway to introduce this functionality. HiveMQ, an MQTT broker, is actively working on providing such features. For more details, you can refer to their documentation: https://docs.hivemq.com/hivemq/4.15/data-governance-hub/index.html.

For Kafka, this feature is known as a 'schema registry'. It's an integral part of the protocol. If a message doesn't comply with the established standard, the MQTT/Kafka broker won't receive it, and it will be rejected.

A few other features in MQTT, such as last will and state, are less relevant for manufacturing contexts. These features are typically more applicable in use-cases involving globally scattered devices, such as connected cars. We recommend focusing on features that most directly apply to your specific manufacturing requirements to ensure the optimal setup for your system.

Read next

Share, Engage, and Contribute!

Discover how you can share your ideas, contribute to our blog, and connect with us on other platforms.