Hello everybody,

today I want to write few words about normalization in neural networks and why it is needed or can be needed, but not only from mathematical prospective. Here I mentioned some mathematical reasons why normalization and scaling can be needed.

First of all, you can be surprised, but your brain also does normalization quite often. As way of example give answers at the following questions:

1.     What is color of tree?

2.     What is color of banana?

3.     What is color of logo of Coca-Cola?

4.     In which quantities GDP is measured?

5.     What is color of skin of people in Africa?

I can’t give 100% guarantee, but most probably you answered that tree is green, banana is yellow, Coca-Cola is red, GDB is measured in billions and Africans are black skinned.

 But if to look at some pictures:

you can see that tree has other colors, not just green.

Banana has other colors, not just yellow.

Coca-Cola has other colors, not just red.

Also there are countries which have their GDP measured not in billions but in millions of dolars. I can't explain why our brain behaves like this, but that is interesting feature of human brain, that it makes some kind of normalization. Actually if to speak about any kind of good report to CEO, CTO, VP, your boss, etc., great report is something, that display in form of pictures not some raw data, but normalized data as some chart.

It's also possible to assume, that our body makes some kind of normalization of information which it gives to brain.  It means that it's possible to cut of hearing part of brain ( for example if it is affected by cancer ) , and switch hearing to some neurons in head which responsible for sight ( of course it's very complicated surgery ).

One more example of normalization which brain does is the following: imagine, that 800 of 1000 times you see, some observation. What will you conclude? You will make conclusion, that some observation has probablity 80%. And that statement can also can be considered as normalization which was perfomed by your brain.

Summary

So, as for neural network user it means the following, next time, when you'll have to normalize and scale in your input, you should remember that not only artifical neural network needs normalized data, but your brain also operates with normalized data.