Microsoft just recently announced the release of .NET for Apache Spark 2.0.0.
It provides a couple of new features, like the support for the Apache Spark 3.1.0 APIs, for example.
Apache Spark 2.3.x support has now officially been dropped, however.
I have also updated my different docker images, so that you can try out all the different combinations of the supported Apache Spark versions along with the new .NET for Apache Spark 2.0.0 runtime.
As you might be aware, there are actually 3 different kinds of docker images provided.
Interactive Jupyter Notebook
Use these images if you want to build .NET for Apache Spark yourself, make changes to the source code or contribute to .NET for Apache Spark.
You don’t even need to execute an interactive docker terminal (docker exec -it …), but can just use Visual Studio Code in your web-browser, as shown below.
For a more detailed introduction, check out this blog post.
The runtime images are intended for testing .NET for Apache Spark, without the need to install all the required bits and dependencies manually. Per default, the related container will start up a master instance and one slave instance of Apache Spark. Here’s a list of articles that might be useful to get you started.
Thanks again for trying out the images and have a great time.3