.NET for Apache Spark 2.0.0 released

.NET for Apache Spark 2.0.0 released

Microsoft just recently announced the release of .NET for Apache Spark 2.0.0.

It provides a couple of new features, like the support for the Apache Spark 3.1.0 APIs, for example.

Apache Spark 2.3.x support has now officially been dropped, however.

I have also updated my different docker images, so that you can try out all the different combinations of the supported Apache Spark versions along with the new .NET for Apache Spark 2.0.0 runtime.

As you might be aware, there are actually 3 different kinds of docker images provided.

Interactive Jupyter Notebook

This image allows you … more

My .NET for Apache Spark docker image at .NET Conf 2020

This year’s .NET Conference covered many interesting topics in various online sessions. One of them was titled “The Missing Piece – Diving into the World of Big Data with .NET for Apache Spark”.
Since I wrote about .NET for Apache Spark in the past, I obviously was very curious to find out what this session might cover.

I almost couldn’t believe my eyes when I saw my name mentioned on one of the slides. But the next surprise wasn’t long in coming. One of my .NET for Apache Spark docker images was even used during … more

.NET for Apache Spark 1.0 is available

Microsoft has released version 1.0 of .NET for Apache® Spark™. The official announcement provides a brief overview of the basic features, performance and the future of .NET for Apache Spark.

It even mentions my docker images, which are now available for version 1.0.0, too. Some of my articles are mentioned on the Open Source Blog as well. A big Thank You to everyone supporting me by trying out the images, providing feedback or contributing in any other way.

Speaking of docker images: In case you didn’t know yet, there are now three different types of … more

.NET for Apache Spark – interactive notebook Docker image

If you are reading this, you are probably aware of my .NET for Apache Spark Docker images that I’ve made available so far. Just recently I’ve added a development image that allows you to easily build .NET for Apache Spark with VS Code in a browser. Today I want to introduce you to the latest member of the family:

The .NET for Apache Spark interactive notebook Docker image.

Jupyter Notebooks

In case you are not aware of what Jupyter Notebooks are, here’s a quick summary quote from the Jupyter project site.

The Jupyter Notebook

more

Build .NET for Apache Spark with VS Code in a browser

Build .NET for Apache Spark with VS Code in a browser

My last article explained how you can use .NET for Apache Spark together with Entity Framework to stream data to an SQL Server. There is one caveat though. You have to build Microsoft.Spark.Worker yourself.
This time I’ll show you how you can actually build .NET for Apache Spark with VS Code in a browser yourself, including building and running the C# examples.

Setting up your own development environment to build and test .NET for Apache Spark can be tricky and time-consuming. However, as a regular reader, you are probably aware that I like to use docker … more

Scroll to top