Je vertrek voorbereiden of je verzekering afsluiten bij studie**,** stage of onderzoek in het buitenland

**Study or work abroad**? check your insurance options with The JoHo Foundation

Article summary of Perceptrons by Van der Velde - Chapter

The purpose of this article is to provide an overview of how perceptrons classify patterns, and to highlight the importance of squashing functions such as activation functions and the learning opportunities of perceptrons.

A perceptron is a neural network in which neurons from different layers are connected to each other. A basic perceptron is shown in Figure 1 of the article. The network consists of two input neurons (x and y) and one output neuron (U). In general, perceptrons can have multiple input neurons and multiple output neurons. The output of neuron x is given by its output activation which is also indicated by x (so x can refer to the name of a neuron or to its output activation). The output of neuron y is its activation y.

Normally a neuron transforms the activation it receives by using an activation function (AF). The input for the AF is generally given by (total input - Activation Threshold).

In general, an activation function AF is a so-called 'squashing function'. Two important squashing functions are:

logistic function

hyperbolic tangent function

These functions can be seen in Figure 2 in the article.

The input can vary over a long interval (from strongly negative to strongly positive). However, the output is limited, either between 0 and 1 or between -1 and 1. This is an important feature of activation functions. It shows that the logistic function and the hyperbolic tangent function are both squashing functions in the sense that they reduce ('squashing') a (potentially) large input to a relatively small output.

The 'squashing behavior' of both functions means that the most important input values are those that are around 0 (around the threshold value). For both functions you can manipulate the steepness with which the function values change around 0. Moreover, the logistic function only produces positive output activations.

The formula of the logistic function is:

*Lf(x) = 1/1 + e ^{-x}*

In this formula, the following applies:

Lf(x): logistic function

x: input variable

e

^{-x}: exponential function

Figure 3 shows e^{x}. The behavior of e^{-x} is the opposite. When x is strongly positive, e^{-x} is small. The logistic function *(Lf(x))* therefore approaches 1 for a strongly positive input. When x is strongly negative, e^{-x} is large. The logistic function* (Lf(x))* therefore approaches 0 for a strongly negative input. When x = 0, it applies that e^{-x} = e^{0} = 1, so *Lf (x*) = 0.5. This means that the logistic function is 0.5 with zero input.

The formula for the hyperbolic tangent function tandh (x) is:

*tanh(x) = e ^{x} - e^{-x} / e^{x} + e^{-x}*

Both e^{x} and e^{-x} play an important role in this. When x is strongly positive, e^{x} is large and e^{-x} small. The hyperbolic tangent function *tanh(x) *therefore approaches 1 for a strongly positive input. When x is strongly negative, ex is small and e^{-x} is large. So the hyperbolic function approaches -1 for a strongly negative input. When x = 0, it applies that e^{x} = 1 and e^{-x }= 1. The hyperbolic function *tanh(x) *= 0 at zero input.

<

p>Activation neuron = AF (total input - Activation Threshold). Activation Threshold is indicated by θ. When θ > 0, a larger positive input activation is required to obtain an output of 1. It also applies that if θ

This paragraph illustrates how perceptrons can classify patterns. It also illustrates why squashing functions are such important activation functions in neural networks. Figure 5 of the article shows how the two-layer network of Figure 1 can be used to classify patterns in line with both the logistic AND-function. The figure shows the (x, y) patterns (0, 0), (1, 0) and (0, 1) that are classified as 0, and the pattern (1, 1) that is classified as 1. These points are called the 'input space'.

Classification of the AND-function can be achieved if we draw a line in the input space that separates points (0, 0), (1, 0) and (0, 1) from point (1, 1). Then all (x, y) inputs are classified to the left of the line. These are labeled as 0. All (x, y) inputs located to the right of the line are classified as 1.

The EXOR problem is introduced in Figure 8 and is as follows:

x y xERORy

1 1 0

1 0 1

0 1 1

0 0 0

However, this problem cannot be classified in a two-layer network. It therefore cannot be classified with a perceptron. The reason for this is that the problem is not linearly separated in the (x, y) space. The fact that problems such as EXOR are not linearly separated was an important discovery in network theory. The solution is to solve the problem with a three-layer network. This is shown in Figure 9. In this, there is a 'hidden' layer (neurons a and b) within the input (x, y) layer and the output neuron U. The hidden layer performs two necessary intermediate classifications.

Figure 11 illustrates learning in a two-layer network (perceptron) that learns to classify patterns with the logistic AND- and NOR-rules. The learning procedure depends on the difference between the output of a neuron on each given example U and the output that the network needs to have a correct classification (the desired output D). The difference D-U is therefore a measure of the error that the network makes. Learning procedures that use the measure error are called 'supervised learning procedures'.

Check page access:

Public

Join WorldSupporter!

How to use this summary?

Work for WorldSupporter

JoHo can really use your help! Check out the various student jobs here that match your studies, improve your competencies, strengthen your CV and contribute to a more tolerant world

How to use more summaries?

- Check out: Register with JoHo WorldSupporter: starting page (EN)
- Check out: Aanmelden bij JoHo WorldSupporter - startpagina (NL)

There are several ways to navigate the large amount of summaries, study notes en practice exams on JoHo WorldSupporter.

- Starting Pages: for some fields of study and some university curricula editors have created (start) magazines where customised selections of summaries are put together to smoothen navigation. When you have found a magazine of your likings, add that page to your favorites so you can easily go to that starting point directly from your profile during future visits. Below you will find some start magazines per field of study
- Use the menu above every page to go to one of the main starting pages
- Tags & Taxonomy: gives you insight in the amount of summaries that are tagged by authors on specific subjects. This type of navigation can help find summaries that you could have missed when just using the search tools. Tags are organised per field of study and per study institution. Note: not all content is tagged thoroughly, so when this approach doesn't give the results you were looking for, please check the search tool as back up
- Follow authors or (study) organizations: by following individual users, authors and your study organizations you are likely to discover more relevant study materials.
- Search tool : 'quick & dirty'- not very elegant but the fastest way to find a specific summary of a book or study assistance with a specific course or subject. The search tool is also available at the bottom of most pages

Do you want to share your summaries with JoHo WorldSupporter and its visitors?

- Check out: Why and how to add a WorldSupporter contributions
- JoHo members: JoHo WorldSupporter members can share content directly and have access to all content: Join JoHo and become a JoHo member
- Non-members: When you are not a member you do not have full access, but if you want to share your own content with others you can fill out the contact form

**Field of study**

- Communication & Media sciences
- Corporate & Organizational Sciences
- Cultural Studies & Humanities
- Economy & Economical sciences
- Education & Pedagogic Sciences
- Health & Medical Sciences
- IT & Exact sciences
- Law & Justice
- Nature & Environmental Sciences
- Psychology & Behavioral Sciences
- Public Administration & Social Sciences
- Science & Research
- Technical Sciences

Check other studie fields?

Promotions

Je vertrek voorbereiden of je verzekering afsluiten bij studie**,** stage of onderzoek in het buitenland

**Study or work abroad**? check your insurance options with The JoHo Foundation

More contributions of WorldSupporter author: Vintage Supporter:

- 1 of 2157
- next ›

## Add new contribution