.NET for Apache Spark ForeachWriter & PostgreSQL

.NET for Apache Spark IForeachWriter implementation

Introduction

A couple of months ago I’ve described how to transfer data from Apache Spark to PostgreSQL by creating a Spark ForeachWriter in Scala.

This time I will show how this can be done in C#, by creating a ForeachWriter for .NET for Apache Spark.

To create a custom ForeachWriter, one needs to provide an implementation of the IForeachWriter interface, which is supported from version 0.9.0 onward. I am going to use version 0.10.0 in this article, however.

Documentation of the C# Interface is provided within the related source code:

https://github.com/dotnet/spark/blob/master/src/csharp/Microsoft.Spark/Sql/ForeachWriter.cs

The example project I am … more

Using the htm.core-jupyter docker image

htm.core-jupyter

To make it easier to get started with some of my htm.core experiments or with htm.core in general, I thought it would make sense to provide a docker image with htm.core preinstalled. So here it is:
Please welcome the htm.core-jupyter image.

This image is using the scipy-notebook as foundation, with the htm.core package installed on top of it.

For a list of the other preinstalled python packages, just look here.
There are also a lot of other Jupyter docker images available. I recommend starting with the Jupyter Docker Stacks quick start page for a … more

Hierarchical Temporal Memory – part 1 – getting started

Hierarchical Temporal Memory - Neuron

Prologue

I came across the concept of Hierarchical Temporal Memory (HTM) and its implementation a while ago, and am still very fascinated about this approach to artificial intelligence.

When, about one year ago, the active development shifted towards the community fork named htm.core, which supports Python 3, it became finally time to have a closer look and try it out by myself.

BTW: According to this forum post, there are no plans to upgrade the older NuPIC library to Python 3.

A lot of documentation about the theory of HTM is available at numenta.orgmore

.NET for Apache Spark – VSCode with Docker on Linux and df.Collect()

.NET for Apache Spark docker image

Overview

My last article explained, how a .NET for Apache Spark project can be debugged in Visual Studio 2019 under Windows. I have also mentioned some limitation at the end of the article.
In this article I will extend the project a bit and demonstrate the aforementioned limitation using version 0.8.0 of my docker image for .NET for Apache Spark.
Furthermore, I will show a possible workaround that can be used, if you are running Docker and Visual Studio Code under Linux (Ubuntu 18.04).

The extended application

In order to demonstrate the issue, I have … more

Scroll to top