Database Archive • 3rd man

29. May 2021 .NET Core, C#, Database, Docker, Entity Framework, Informix, Visual Studio

IBM Informix and EntityFrameworkCore

Just recently I needed to explore how I could get IBM Informix and EntitiyFrameworkCore to work together with an existing database, and therefore decided to document my findings in this simple step-by-step walk-through.

Prerequisites

As mentioned here, the IBM® Data Server Provider for .NET IBM DB2® provider option is now the preferred Informix® provider for developing new applications.

Looking at the related site for the IBM Data Server Provider for .NET, it actually states that the former Informix specific provider (IBM.Data.Informix) is now deprecated.

Although this statement actually applies to the provider for .NET, … more

22. September 2020 .NET Core, Apache Spark, Big Data, C#, Database, Entity Framework, SQL Server, Visual Studio Code

.NET for Apache Spark – Stream to SQL Server

In this article I am going to describe how to use .NET for Apache Spark with EntityFrameworkCore to stream data to a Microsoft SQL Server. If you have tried this before, you probably stumbled upon the following exception: Microsoft.Data.SqlClient is not supported on this platform.

So let’s find out, how that can be fixed.

Preparation

If you want to stream to an SQL Server, you obviously need to have access to an SQL Server instance first.

Using docker, it is very easy to fire up a related container. I’ve just named it sqlserver, as … more

15. March 2020 .NET Core, Apache Spark, Big Data, C#, Database, Entity Framework, PostgreSQL

.NET for Apache Spark ForeachWriter & PostgreSQL

.NET for Apache Spark IForeachWriter implementation

Introduction

A couple of months ago I’ve described how to transfer data from Apache Spark to PostgreSQL by creating a Spark ForeachWriter in Scala.

This time I will show how this can be done in C#, by creating a ForeachWriter for .NET for Apache Spark.

To create a custom ForeachWriter, one needs to provide an implementation of the IForeachWriter interface, which is supported from version 0.9.0 onward. I am going to use version 0.10.0 in this article, however.

Documentation of the C# Interface is provided within the related source code:

https://github.com/dotnet/spark/blob/master/src/csharp/Microsoft.Spark/Sql/ForeachWriter.cs

The example project I am … more

14. September 2019 Big Data, PostgreSQL, Time Series

PostgreSQL & Grafana – Real-time data processing pipeline – Part 6

Believe it or not, we are getting to the end of this small series about a potential real-time data processing pipeline.
In this final part I will show how Grafana can retrieve our pipeline data from PostgreSQL and visualize it as a graph. But before we dive into it, let’s have a quick recap of the previous topics.

So the bit that is still missing, is the visualization of data.

Starting the docker image

To keep things simple, I … more

31. August 2019 Apache Spark, Apache Zeppelin, Big Data, Database, PostgreSQL, Scala

Spark to PostgreSQL – Real-time data processing pipeline – Part 5

Previously I have demonstrated how streaming data can be read and transformed in Apache Spark. This time I use Spark to persist that data in PostgreSQL.

Quick recap – Spark and JDBC

As mentioned in the post related to ActiveMQ, Spark and Bahir, Spark does not provide a JDBC sink out of the box. Therefore, I will have to use the foreach sink and implement an extension of the org.apache.spark.sql.ForeachWriter. It will take each individual data row and write it to PostgreSQL.

Preparing PostgreSQL

Even though I want to use PostgreSQL, I am actually

… more

Search Site

Category: Database