At university, I generated Markov chains of the solution space from a single neuron that was being used as a binary classifier. You take n samples, average them out and look at the decision boundary. The decision boundary itself is linear but the margin of error is not.

It was really cool. Attempting to implent Hamiltonian MCMC on a single neuron really forced you to learn what a gradient is in regards to NN.