Article summary of Perceptrons by Van der Velde - Chapter


The purpose of this article is to provide an overview of how perceptrons classify patterns, and to highlight the importance of squashing functions such as activation functions and the learning opportunities of perceptrons.

Basic principles of perceptrons

A perceptron is a neural network in which neurons from different layers are connected to each other. A basic perceptron is shown in Figure 1 of the article. The network consists of two input neurons (x and y) and one output neuron (U). In general, perceptrons can have multiple input neurons and multiple output neurons. The output of neuron x is given by its output activation which is also indicated by x (so x can refer to the name of a neuron or to its output activation). The output of neuron y is its activation y.

Normally a neuron transforms the activation it receives by using an activation function (AF). The input for the AF is generally given by (total input - Activation Threshold).

Activation function ('squashing function')

In general, an activation function AF is a so-called 'squashing function'. Two important squashing functions are:

  • logistic function

  • hyperbolic tangent function

These functions can be seen in Figure 2 in the article.

The input can vary over a long interval (from strongly negative to strongly positive). However, the output is limited, either between 0 and 1 or between -1 and 1. This is an important feature of activation functions. It shows that the logistic function and the hyperbolic tangent function are both squashing functions in the sense that they reduce ('squashing') a (potentially) large input to a relatively small output.

The 'squashing behavior' of both functions means that the most important input values ​​are those that are around 0 (around the threshold value). For both functions you can manipulate the steepness with which the function values ​​change around 0. Moreover, the logistic function only produces positive output activations.

The logistic function

The formula of the logistic function is:

Lf(x) = 1/1 + e-x

In this formula, the following applies:

  • Lf(x): logistic function

  • x: input variable

  • e-x: exponential function

Figure 3 shows ex. The behavior of e-x is the opposite. When x is strongly positive, e-x is small. The logistic function (Lf(x)) therefore approaches 1 for a strongly positive input. When x is strongly negative, e-x is large. The logistic function (Lf(x)) therefore approaches 0 for a strongly negative input. When x = 0, it applies that e-x = e0 = 1, so Lf (x) = 0.5. This means that the logistic function is 0.5 with zero input.

The hyperbolic tangent function

The formula for the hyperbolic tangent function tandh (x) is:

tanh(x) = ex - e-x / ex + e-x

Both ex and e-x play an important role in this. When x is strongly positive, ex is large and e-x small. The hyperbolic tangent function tanh(x) therefore approaches 1 for a strongly positive input. When x is strongly negative, ex is small and e-x is large. So the hyperbolic function approaches -1 for a strongly negative input. When x = 0, it applies that ex = 1 and e-x = 1. The hyperbolic function tanh(x) = 0 at zero input.

Threshold value

<

p>Activation neuron = AF (total input - Activation Threshold). Activation Threshold is indicated by θ. When θ > 0, a larger positive input activation is required to obtain an output of 1. It also applies that if θ

Classification in perceptrons

This paragraph illustrates how perceptrons can classify patterns. It also illustrates why squashing functions are such important activation functions in neural networks. Figure 5 of the article shows how the two-layer network of Figure 1 can be used to classify patterns in line with both the logistic AND-function. The figure shows the (x, y) patterns (0, 0), (1, 0) and (0, 1) that are classified as 0, and the pattern (1, 1) that is classified as 1. These points are called the 'input space'.

Classification of the AND-function can be achieved if we draw a line in the input space that separates points (0, 0), (1, 0) and (0, 1) from point (1, 1). Then all (x, y) inputs are classified to the left of the line. These are labeled as 0. All (x, y) inputs located to the right of the line are classified as 1.

EXOR classification

The EXOR problem is introduced in Figure 8 and is as follows:

x y xERORy

1 1 0

1 0 1

0 1 1

0 0 0

However, this problem cannot be classified in a two-layer network. It therefore cannot be classified with a perceptron. The reason for this is that the problem is not linearly separated in the (x, y) space. The fact that problems such as EXOR are not linearly separated was an important discovery in network theory. The solution is to solve the problem with a three-layer network. This is shown in Figure 9. In this, there is a 'hidden' layer (neurons a and b) within the input (x, y) layer and the output neuron U. The hidden layer performs two necessary intermediate classifications.

Learning with perceptrons

Figure 11 illustrates learning in a two-layer network (perceptron) that learns to classify patterns with the logistic AND- and NOR-rules. The learning procedure depends on the difference between the output of a neuron on each given example U and the output that the network needs to have a correct classification (the desired output D). The difference D-U is therefore a measure of the error that the network makes. Learning procedures that use the measure error are called 'supervised learning procedures'.

Check page access:
Public
Work for WorldSupporter

Image

JoHo can really use your help!  Check out the various student jobs here that match your studies, improve your competencies, strengthen your CV and contribute to a more tolerant world

Working for JoHo as a student in Leyden

Parttime werken voor JoHo

How to use more summaries?


Online access to all summaries, study notes en practice exams

Using and finding summaries, study notes en practice exams on JoHo WorldSupporter

There are several ways to navigate the large amount of summaries, study notes en practice exams on JoHo WorldSupporter.

  1. Starting Pages: for some fields of study and some university curricula editors have created (start) magazines where customised selections of summaries are put together to smoothen navigation. When you have found a magazine of your likings, add that page to your favorites so you can easily go to that starting point directly from your profile during future visits. Below you will find some start magazines per field of study
  2. Use the menu above every page to go to one of the main starting pages
  3. Tags & Taxonomy: gives you insight in the amount of summaries that are tagged by authors on specific subjects. This type of navigation can help find summaries that you could have missed when just using the search tools. Tags are organised per field of study and per study institution. Note: not all content is tagged thoroughly, so when this approach doesn't give the results you were looking for, please check the search tool as back up
  4. Follow authors or (study) organizations: by following individual users, authors and your study organizations you are likely to discover more relevant study materials.
  5. Search tool : 'quick & dirty'- not very elegant but the fastest way to find a specific summary of a book or study assistance with a specific course or subject. The search tool is also available at the bottom of most pages

Do you want to share your summaries with JoHo WorldSupporter and its visitors?

Quicklinks to fields of study (main tags and taxonomy terms)

Field of study

Access level of this page
  • Public
  • WorldSupporters only
  • JoHo members
  • Private
Statistics
996
Comments, Compliments & Kudos:

Add new contribution

CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Image CAPTCHA
Enter the characters shown in the image.