Many companies struggle when deciding which database type to use for their architecture. In this video, we break down the key differences between OLTP (Online Transaction Processing) and OLAP (Online Analytical Processing) systems, and explain how they can fit into the Unified Namespace architecture.
OLTP Databases
- Purpose: Designed for routine, day-to-day transactions.
- Common Use: Online banking, shopping, or similar applications.
- Users: End-users or customers interacting via web applications.
- Data Focus: Ensures data remains current at all times.
- Data Size: Typically handles gigabytes to terabytes of data.
- Examples: Most SQL databases are classified as OLTP systems.
OLAP Databases
- Purpose: Built for complex tasks like data mining, business intelligence, and deep analytical calculations.
- Common Use: Bulk data import (ETL), data streams, advanced analytics.
- Users: Primarily used by internal analysts for decision-making and support.
- Data Focus: Manages massive datasets, often in static or historical form.
- Data Size: Capable of handling data ranging from terabytes to petabytes.
- Examples: Data Warehouses, Data Lakes, Azure Synapse Analytics, AWS Redshift.
Chapters
00:00 - 00:30 Introduction
00:30 - 01:00 OLTP Systems
01:00 - 02:22 OLAP Systems
02:22 - 04:20 Relevance within Unified Namespace
Transcript
So what are OLTP and OLAP systems and how do they fit in with the unified namespace? A lot of companies, when they try to figure out how they should build the architecture, spend a lot of time about thinking about, uh, what exact database type do I need to go? Is it MongoDB, is it timescale, influx, whatever? What I think really helps is taking a look at the two base categories of, of databases. So first are, uh, OLTP systems, online transaction processing systems. And these systems are designed for routine transactions, like when you go shopping. So every supermarket has the OLTP database where you have, where they have the current inventory, for example, of their milk. Um, so every time you go there and the cashier goes there and scans a pack of milk, there is a small transaction that will reduce the inventory of the milk. One very simple transaction, but there will be a lot of them.M the other, uh, type of database are OLAP systems, online analytical processing systems, where the target is not a lot of simple routine transactions, but instead to have all the data there, in a big data warehouse, what you have then there is, you have all the small information pieces that you gather during routine transaction. For example, let's take the pack of milk. Example, um, just get a pack of milk. And then you would have like a timestamp, and it would say pack of milk was bought. And you would have it have really a lot of data there, like terabytes, petabytes of data.And then there would be a business analyst coming in and it would be like, okay, what's the perfect time for people to buy milk? Or what's the ideal price? And then they would do like long running transactions on those databases. So there are two different groups for entirely different purposes. And I think this really helps a lot of people to redecide what to go for.Um, so OLTP database, for example, is everything SQL based, postgres, etcetera. So if you have something that most ERP mes systems, they have just an OTP database in the background. But if you think about Snowflake and stuff like this, it's always for analytical purposes. It's not to control the production that I think is already really helpful. And now, what does it have to do with the unified namespace?See, if you take in the very traditional sense, ▪ um, how those databases need to be connected. So you have an ETL process, extract, transform, load. So, um, very traditionally, this is old nineties to two thousands technologies here, um, you do a daily or end of week, like a nightly job that takes all the data from the otpenous databases for all the transactions where MOOC was bought, and would push it into OLAP database so that there, the business analysts can work with it and try to improve m ▪ the pricing, whatever. And then over time, business analysts were like, I don't want to have like a nightly job. I want to have it in near real time.So what they did was they started. So instead of like duplicating the whole database, they started developing tools. I think it was around 2012, 2010, where, uh, Kafka came up. So they were developing tools like Debesium that would take a look at the databases, would subscribe to all the changes, would then push it into Kafka, and from Kafka it would go into OLAP databases. So there you would have then in Kafka a continuous stream of change events.And this is already very, very similar to the unified namespace to a message broker architecture. And it can also be used, uh, as such. So also in a message broker, you also have every time there is a change event, or in general, in an event driven architecture, every time there's a change event, you can take the data, uh, you have the data there in real time, and you usually push it somewhere else for analytical purposes. So that was the difference between OTP and OLAP databases and how they fit in with the unified namespace, or how they can be used to explain unified namespace architecture. So thank you for watching.And if you like this content, feel free to check out all the other stuff that we are doing here. Just check out the description and you will find their links to YouTube, to our blog articles, podcasts, and all the other stuff that we have.
