Neural Network
In quest for automating key activities and creating learning machines or embedding intelligence in machines (artificial intelligence (AI)), scientists and engineers started to analyze how the human brain recognizes and learns new objects, concepts, and patterns continuously. The research in the area of AI and machine learning lead to the concepts of what is well known as deep learning and neural network. The neural network concepts were first proposed by Warren McCullough and Walter Pitts, two researchers at the University of Chicago in 1944, who further continued their research in MIT.

  Blausen - Multi polar Neuron
  Photo Courtesy: Blausen, Wikipedia
 
A human brain consists of neurons - cells in the nervous system of the brain that communicate with each other. A neural network is loosely modeled to mimic human brain, which consists of millions of interconnected nodes that work in parallel. In recent years, nodes in neural networks (deep learning) are organized in multiple parallel layers and allow unidirectional flow of data as shown in diagram below.
 
Typical Neural Network
 
A neural network could also be designed to consist of millions of algorithms (acting like a node) that process data and achieve deep learning.
 
 
Types of Neural Network based on their architecture (ASIMOV Institute)
ASIMOV Institute - Neural Network Architecture
 
  1. Feed-Forward (FF) Neural Network (unidirectional)
  1.1 Radial Basis Feed-Forward Neural Network (RBF)
  1.2 Deep Feed-Forward Neural Network (DFF)
  2. Recurrent Neural Network (RNN)
  3. Long/Short Term Memory (LSTM)
  4. Gated Recurrent Unit (GRU)
  5. Auto Encoder (AE)
  5.1. Variational Auto Encoder (VAE)
  5.2. Denoising Auto Encoder (DAE)
  5.3. Sparce Auto Encoder (SAE)
  6. Marcov Chain (MC)
  7. Hopfield Network
  8. Boltzmann Machine
  8.1 Restricted Boltzmann Machine (RBM)
  9. Deep Belief Network (DBN)
  10. Deep Convolution Network (DCN) or Convolution Neural Network (CNN)
  10.1 Deconvolutional Network
  10.2 Deep Convolution Inverse Graphics Network (DCIGN)
  11. Generative Adversarial Network (GAN)
  12. Liquid State Machine (LSM)
  13. Extreme Learning Machine (ELM)
  14. Echo State Network (ESN)
  15. Deep Residual Network (DRN)
  16. Differential Neural Computer (DNC)
  17. Neural Turing Machine (NTM)
  18. Capsule Network
  19. Kohonen Network
  20. Attention Network


References
1. Larry Hardesty, MIT News Office, April 14th, 2017
2. The ASIMOV Institute Neural Network Zoo; Diagram 1400x2380
3. A Basic Introduction To Neural Networks, University of Winconsin, Madison
4. Sigmoid Function


Last Revised on: May 31st, 2023