Is it OK to increase validation checks and decrease min gradient while training neural network?

Is it OK to increase validation checks and decrease min gradient while training neural network?

• My input vector is a 130*85 matrix and my target vector is 130*26 matrix. I am using the below parameters for training the network with 60 hidden nodes. net.trainParam.max_fail = 50; net.trainParam.min_grad=1e-10; net.trainParam.show=10; net.trainParam.lr=0.01; net.trainParam.epochs=1000; net.trainParam.goal=0.001;` As you can see I have set the max_fail to 50 and min_grade to 1e-10. While the default values will be 6 and 1e-5 respectively. But with the default values, the training stops early with out reaching the performance goal. By setting these parameters,the training is stopped when the performance goal is reached. So. Is it OK to change these parameters?

• Answer:

  From http://www.mathworks.com/help/nnet/ug/train-and-apply-multilayer-neural-networks.html: The magnitude of the gradient and the number of validation checks are used to terminate the training. The gradient will become very small as the training reaches a minimum of the performance. If the magnitude of the gradient is less than 1e-5, the training will stop. This limit can be adjusted by setting the parameter net.trainParam.min_grad. The number of validation checks represents the number of successive iterations that the validation performance fails to decrease. If this number reaches 6 (the default value), the training will stop. (...) You can change this criterion by setting the parameter net.trainParam.max_fail. So yeah what you are doing is perfectly legitimate. I would suspect that most of the difference (improvement) actually comes form changing the property net.trainParam.min_grad as by reducing its value you keep optimizing while taking small steps you would otherwise not have taken rather than increasing net.trainParam.min_grad. net.trainParam.min_grad will probably allow your optimization not to terminate in the case is hits a small "plateau". One last thing: While you do not say what are you doing with your NN, if you using a sigmoid function and you use your NN for regression you might want to rescale your target data to be in a $[-0.8,0.8]$ interval so you avoid the saturation that comes to a sigmoid near its edges. (MATLAB, and most software I suspect you probably normalize this to $[-1,1]$ that while correct from a theoretical point of you it might now be optimal in some hard cases.)