Mathematical Neural Networks

Abstract

Financial support from the Spanish Ministry of Universities. "Disruptive group decision making systems in fuzzy context: Applications in smart energy and people analytics" (PID2019-103880RB-I00). Main Investigator: Enrique Herrera Viedma, and Junta de Andalucia. "Excellence Groups" (P12.SEJ.2463) and Junta de Andalucia (SEJ340) are gratefully acknowledged. Research partially supported by the "Maria de Maeztu" Excellence Unit IMAG, reference CEX2020001105-M, funded by MCIN/AEI/10.13039/501100011033/. ; ANNs succeed in several tasks for real scenarios due to their high learning abilities. This paper focuses on theoretical aspects of ANNs to enhance the capacity of implementing those modifications that make ANNs absorb the defining features of each scenario. This work may be also encompassed within the trend devoted to providing mathematical explanations of ANN performance, with special attention to activation functions. The base algorithm has been mathematically decoded to analyse the required features of activation functions regarding their impact on the training process and on the applicability of the Universal Approximation Theorem. Particularly, significant new results to identify those activation functions which undergo some usual failings (gradient preserving) are presented here. This is the first paper—to the best of the author's knowledge—that stresses the role of injectivity for activation functions, which has received scant attention in literature but has great incidence on the ANN performance. In this line, a characterization of injective activation functions has been provided related to monotonic functions which satisfy the classical contractive condition as a particular case of Lipschitz functions. A summary table on these is also provided, targeted at documenting how to select the best activation function for each situation. ; Spanish Government PID2019-103880RB-I00 ; Junta de Andalucía P12.SEJ.2463 SEJ340 ; "Maria de Maeztu" Excellence Unit IMAG - MCIN/AEI CEX2020001105-M