Microsoft just recently announced the release of .NET for Apache Spark 2.0.0.
It provides a couple of new features, like the support for the Apache Spark 3.1.0 APIs, for example.
Apache Spark 2.3.x support has now officially been dropped, however.
I have also updated my different docker images, so that you can try out all the different combinations of the supported Apache Spark versions along with the new .NET for Apache Spark 2.0.0 runtime.
As you might be aware, there are actually 3 different kinds of docker images provided.
Interactive Jupyter Notebook
This image allows you to explore .NET for Apache Spark interactively in your web browser, using Jupyter Notebooks. You can even give it a try without downloading the docker image by using binder.
Note: as mybinder.org is a free service that builds the image from its GitHub source, it may take a while until it is available in you web-browser. So please be patient.
Development
Use these images if you want to build .NET for Apache Spark yourself, make changes to the source code or contribute to .NET for Apache Spark.
You don’t even need to execute an interactive docker terminal (docker exec -it …), but can just use Visual Studio Code in your web-browser, as shown below.
For a more detailed introduction, check out this blog post.
Runtime
The runtime images are intended for testing .NET for Apache Spark, without the need to install all the required bits and dependencies manually. Per default, the related container will start up a master instance and one slave instance of Apache Spark. Here’s a list of articles that might be useful to get you started.
Thanks again for trying out the images and have a great time.
