After my recent playdate with Ruby, I decide to explore a little bit more. And I’ve been wanting to work on Neural Networks for quite some while, so that’s the pet project of the day. To guide me around, I used an introduction to neural networks
And before you tell me how Ruby is too slow, this is a learning project. I probably won’t build terribly advanced networks. In fact, for starters, it’s going to be a simple Perceptron.
In a nutshell, that thing computes a weighted sum of its inputs and compares it against a threshold. And thanks to Ruby’s simplicity, that’s done in a snap.
def output
val = @inputs.zip(@weights).inject(0) { |total, i|
total + i[0] * i[1]
}
val < @threshold ? false: true;
end
Now the usual test case is using those as a simple logic gate. Since I feel like it, here’s the code for an AND gate:
perc = Perceptron.new
perc.weights = [ 1, 1 ]
perc.threshold = 2
Of course, that’s boring. The whole appeal of neural nets is that the little buggers are supposed to learn – we don’t want to set the weights ourselves. The question is, how do we do that? The easiest way is a “fixed increment learning rule“. If the expected output is not the computed output, we nudge the perceptron’s weights into the proper direction by a fixed amount.
def train(inputs, expected_output)
if @perc.eval(inputs)!=expected_output then
new_weights = []
c = (expected_output == true ? @c : -@c)
inputs.each_with_index { |input,i|
new_weights[i] = @perc.weights[i] + c * inputs[i]
}
@perc.weights = new_weights
end
end
And indeed, when we train this perceptron with all the inputs for an ‘OR’ gate, we get a weight set of [2,2] after a single training run. Unfortunately, a single perceptron is rather limited in what it can do – so that will be the next step.
In the meantime, I came to an interesting realization. I skipped writing unit tests since the above is “unbelievably simple” (hah!), and got what was coming to me. All the bugs would have been caught in a static language, though – mistyped variables, forgotten ‘@’ signs, etc.
The conclusion from that is that type annotations in static languages are a very specific form of unit test. (Well, that, plus optimization hints for compilers). Now I’m wondering if you can annotate Ruby/Python/whatever dynamic language programs and then perform a static analysis on them. That’s pretty much a topic for a separate post, though