In this article I am going to describe how to use .NET for Apache Spark with EntityFrameworkCore to stream data to a Microsoft SQL Server. If you have tried this before, you probably stumbled upon the following exception: Microsoft.Data.SqlClient is not supported on this platform.
So let’s find out, how that can be fixed.
If you want to stream to an SQL Server, you obviously need to have access to an SQL Server instance first.
Using docker, it is very easy to fire up a related container. I’ve just named it sqlserver, as … more
This time I will show how this can be done in C#, by creating a ForeachWriter for .NET for Apache Spark.
To create a custom ForeachWriter, one needs to provide an implementation of the IForeachWriter interface, which is supported from version 0.9.0 onward. I am going to use version 0.10.0 in this article, however.
Documentation of the C# Interface is provided within the related source code:
Believe it or not, we are getting to the end of this small series about a potential real-time data processing pipeline. In this final part I will show how Grafana can retrieve our pipeline data from PostgreSQL and visualize it as a graph. But before we dive into it, let’s have a quick recap of the previous topics.
As mentioned in the post related to ActiveMQ, Spark and Bahir, Spark does not provide a JDBC sink out of the box. Therefore, I will have to use the foreach sink and implement an extension of the org.apache.spark.sql.ForeachWriter. It will take each individual data row and write it to PostgreSQL.
Even though I want to use PostgreSQL, I am actually