Believe it or not, we are getting to the end of this small series about a potential real-time data processing pipeline. In this final part I will show how Grafana can retrieve our pipeline data from PostgreSQL and visualize it as a graph. But before we dive into it, let’s have a quick recap of the previous topics.
As a quick reminder, here is the Scala code that I have used so far to retrieve the data from ActiveMQ and write it to a memory sink.
// create a named session
val spark = SparkSession
// read data from the OscStream topic
val mqttDf =
Welcome back to the second part of my series, showcasing a real-time data processing pipeline! In part 1, I explored visual real-time sensor data simulation, as the entry point into our pipeline. Now it’s time to find out, how we can get the generated data into Apache ActiveMQ, by transferring it via the OSC protocol.
Apache ActiveMQ™ is the most popular open source, multi-protocol, Java-based messaging server. It supports a variety of Cross Language Clients and Protocols, and therefore makes it an excellent choice for our pipeline.
This is the first part of my series to showcase a potential pipeline for real-time data processing. An overview about the different components that I am going to use can be found here. So let’s get started and find out how real-time sensor data can be simulated, as each pipeline needs to start somewhere.
There may be times when you need to generate continuous numeric data that allows you to test your real-time streaming processing pipeline. One common approach is to generate this data by code, which, however, can come with some drawbacks.