Neural Modelling

Here are some ramblings on "neural networks" I'd just like to set free ..

In the traditional neural network, neurons are typically modelled by computational nodes of various types. A node inevitably has some "output activation" level, and this is sometimes allowed to range from 0 to 1, in other models may range -1 to +1, and in so-called linear models can even take any value. At the same time, inputs to a node are weighted sometimes only with positive values, and in other models, positive or negative weights are permitted. While these seem overtly different, there is something they all have in common that is probably important.

What makes them all the same is this: you can take a snapshot in time, and at any single instant there will be some set of activation values for all the nodes in the network. Put another way, you don't have to observe over any finite length of time; an instantaneous observation provides knowledge about the activation of any and all nodes in the network. Since these observations are timeless (apart, maybe, from the fact of when the observation is made), I will refer to them as 'static' measurements.

This is not something you could do with real neural networks, i.e., real, biological neural tissue. The reason being that real neurons tend to fire in pulses. One could argue that the static measurements in artificial neural networks approximates real neurons by saying that "the (static!) activiation level of the artificial node at time T represents the average firing rate of the biological node in a narrow time window around time T." Well, maybe so, but a lot of information is lost this way. If you take the average firing rate over a sliding 100 millisecond window, you're really running the 'real' signal, the signal you're interested in investigating, through a low pass filter (LPF). Any effects based on the exact timing of individual pulses cannot be represented by such a "moving average". Furthermore, as the averaging window is narrowed, even though the LPF effect diminishes, the resolution, i.e., the number of distinct possible values diminishes, too. Widening the averaging window gives better resolution, but worsens the LPF effect. In the limit, as the averaging window is reduced to zero, the individual pulses reappear as items of consideration. It's not that the continuous-value, static activation approximation is not good for anything, it's just that it can't possibly capture ALL the possible behaviours that might be seen when individual firing pulses are considered.

For example, suppose you had a node with five inputs, with equal input weighting, and that each of these were firing at about the same LOW constant rate. Conceivably, you could have a node where, if all five of the inputs were to spike within a small interval, the node would fire its own output. Using the moving-average approximation, you'd be forced to say that "each input is at a fluctuating but SMALL value, and the node seems to fire somewhat randomly". If your averaging window was really small, you'd start to notice that the fluctuations would all be in the positive direction when the node fired and you'd have to conclude that the node was exquisitely sensitive to inputs. Well, with the averaging window made small, your 'fluctuations' would start to reveal individual input spikes, so you'd be onto something there, but in the static model, the necessary conclusion that the node is 'extremely sensitive' would force you to conclude that a single input, if amped up a bit, should suffice to tip the node into firing. Of course, amping up the 'level' of an input in the artificial model means increasing the frequency of firing in the biological model.

Well, I haven't worked out the details, but it seems that the static model isn't going to be able to produce all the behaviours of real neural systems, while real neural systems should be able to reproduce anything the static model is capable of. Sure, I need to put the math to it to make it a valid argument, but this is a BLOG isn't it? I'm just thinking out loud here. If someone does the math, or knows where it's already been done, and wants to tell me, "hey, yer full of it, look here's the math to prove it", that'd be welcome.