TimescaleDB Archive • 3rd man

14. September 2019 Big Data, PostgreSQL, Time Series

PostgreSQL & Grafana – Real-time data processing pipeline – Part 6

Believe it or not, we are getting to the end of this small series about a potential real-time data processing pipeline.
In this final part I will show how Grafana can retrieve our pipeline data from PostgreSQL and visualize it as a graph. But before we dive into it, let’s have a quick recap of the previous topics.

So the bit that is still missing, is the visualization of data.

Starting the docker image

To keep things simple, I … more

31. August 2019 Apache Spark, Apache Zeppelin, Big Data, Database, PostgreSQL, Scala

Spark to PostgreSQL – Real-time data processing pipeline – Part 5

Previously I have demonstrated how streaming data can be read and transformed in Apache Spark. This time I use Spark to persist that data in PostgreSQL.

Quick recap – Spark and JDBC

As mentioned in the post related to ActiveMQ, Spark and Bahir, Spark does not provide a JDBC sink out of the box. Therefore, I will have to use the foreach sink and implement an extension of the org.apache.spark.sql.ForeachWriter. It will take each individual data row and write it to PostgreSQL.

Preparing PostgreSQL

Even though I want to use PostgreSQL, I am actually

… more

8. June 2019 Apache Spark, Big Data, Time Series

Real-time data processing pipeline showcase – Series Overview

With the multitude of existing projects and solutions related to real-time data processing out there, it can be very easy to get lost in all the available options.

That is why I have started this blog series. I want to showcase an example pipeline that covers the topic of real-time data processing, from beginning (data generation) to the end (data presentation).

Generation on the left, data presentation on the right side

Below is an overview of the pipeline that I am going to use.

Here are the links to the related articles.

Part 1 – Visual