Pandora: Documentation en: Alerts

From Pandora FMS Wiki
Jump to: navigation, search

Go back to Pandora FMS documentation index


Contents

1 Alert Configuration in Pandora FMS

1.1 Introduction

An alert is Pandora FMS's reaction to a module's value being incorrect. Such a reaction is configurable and can result in any action that can be triggered by a script configured in the Operating System where the Pandora FMS server that processes the module is running.

There are several alert types:

  • Simple alerts.
  • Event alerts.
  • SNMP trap alerts.

This chapter discusses the alert system as a whole and particularly the first two types.

1.2 Introduction to the alert system

In Pandora FMS, alerts work by defining some triggering terms, some actions selected for that alert, and finally the execution of some commands in the Pandora FMS server, which will be in charge of carrying out the configured actions.

The general alert system associates a single alert for each module, this alert can carry out one or more actions.

More information can be found in the video tutorial «Do not panic: we talk about alert systems».

1.2.1 The Alert Structure


Esquema-alert-structure.png


An alert consists on:

  • Commands: They specify what will be done. This will be the execution done by Pandora FMS server when triggering an alert. This could be writing in a log, sending an email or an SMS, executing a script, etc.
  • Actions : They specify how it will be done. They are the customizations of the command's arguments, they allow to customize the execution itself, adding to the command particular parameters like module name, agent, etc.
  • Templates: They specify when it will be done, defining the conditions for triggering the action(s). For example: when the module becomes critical.

1.2.2 The Alert System's Information Flow

When defining actions and templates, both have some general fields called Field1, Field2 and Field3 that are used to customize the alert's triggering.

These fields are applied according to a precedence order, "transferring" information from template -> action -> command, to finally be used as parameters in the execution of this command.

Said information is transferred as long as the following step does not have information defined in its Fieldn fields. That means in case of field or parameter overwriting , it overwrites the action to the template (for example, if the template has field1 defined and the action too,, the action's field1 prevails.

The following diagram shows this parameter transfer from the template to the command:


Esquema-parameters-carrying.png


This is an example for how template values are overwritten by the action's values:

Alertas esquema6.png


For example, this is a template that triggers an alert and sends an email, containing the following fields:

  • Template:
    • Field 1: myemail@domain.com
    • Field 2: [Alert] The alert was fired.
    • Field 3: The alert was fired!!! SOS!!!
  • Action:
    • Field 1: myboss@domain.com
    • Field 2: <left blank>
    • Field 3:<left blank>

The values transferred to the command are:

  • Command:
    • Field 1: myboss@domain.com
    • Field 2: [Alert] The alert was fired
    • Field 3: The alert was fired!!! SOS!!!


Field2 and Field3 keep the values defined in the template, but Field1, uses the value defined in the action.

1.3 The Alert Command

1.3.1 Introduction

Pandora FMS's actions in face of alert situations at the end mean server executions, in the form of commands.

Susi2.png

1.3.2 Command creation for an alert

When clicking on Create in the previous section:

Susi3 5.png


Name: The command's name. It must be descriptive but short.

Command: The command to be executed when the alert is triggered. Macros can be used to replace the parameters configured in alert declaration (see the following section).

Template warning.png

When creating the commands for alerts, it is necessary to take into account that these commands are executed by Pandora FMS server. The alerts are also executed with the privileges of the user that executes the Pandora FMS server.

 


Info.png

When defining a command, it is convenient to test, from the command line, that the execution of the command is successful and produces the desired result (sending an e-mail, generating an entry in a log file, etc.).

 



Group

Command group. This will determine which alerts the command can be associated to. User can only assign a group to which the user creating the alert command belongs, unless that user explicitly belongs to the ALL group.

Field description and field values: For each custom field it is possible to configure:


    • Field description: The label next to the text box in the configuration form of the command action.
    • Possible values: A collection of the possible values for that field. If the field is configured (it is not empty), it will be a selection combo instead of a text box. The combo needs a tag (the visible option) for each possible value and a value (the sent option). This is the supported syntax:
value1,tag1;value2,tag2;value3,tag3
    • Hide: If the field hosts some password, this options hides the content with asterisks .

Info.png

From version 6.0 it is possible to show an HTML code editor in a field of the command in the creation or edition of an action of an alert if that command field has as value the _html_editor_ special token.

 


Once all the parameters are porperly filled in, click Create to save.

1.3.2.1 Example

A simple field where it will be possible to choose the first four numbers:

1,Number one;2,Number two;3,Number three;4,Number four

The command field must be configured:

Possible values 1.png

When you go to the action you will see it this way:

Possible values 2.png

1.3.2.2 Command macros

The macros that can be used within command configuration are in the macro list at the end of this chapter.

1.3.3 Predefined Commands

There is a series of predefined commands ready to use in Pandora FMS alert system.

eMail

It sends an email from Pandora FMS server and uses the Perl "sendmail" command to do so. Emails are sent in HTML format, which allows creating more visually attractive templates. Notice that the receiver of the email should have access to the resources used in the template such as images.

Internal Audit

This generates a small entry within the internal Auditing System of Pandora FMS. It is saved in Pandora FMS database and it can be reviewed by the console's event viewer.

Pandora FMS Event

This creates a custom event within Pandora FMS Event Manager.

Pandora FMS Alertlog

This is a default alert to write alerts in a standard ASCII plain-text log file located in /var/log/pandora/pandora_alert.log.

SNMP Trap

It sends a parametrized SNMP trap with the arguments being used.

Syslog It sends an alert to the system registry and uses the system command named "logger" to do so.

Sound Alert

It plays a sound if an alert is received.

Jabber Alert

It sends a Jabber alert to a chat room on a predefined server (configure the file named .sendxmpprc first). It uses field3 for the text message, field1 for the user's alias, and field2 for the chat-room's name.

SMS Text

It sends an SMS to a specific cellphone. Define an alert and a gateway for sending configured and accessible SMS from Pandora FMS before being able to do so. It is also possible to install one using "Gnokii" to send an SMS directly by using a Nokia phone with an USB connection. Further information on the detailed procedure is provided below.

Validate Event

It validates all events regarding a module. The agent's and module's names will be given.

Remote agent control

This sends commands to the agents with the UDP server enabled. The UDP server is used to order agents (Windows and UNIX) to "refresh" agent execution: that means, to force the agent to execute and send data

Generate Notification

This command allows you to send an internal notification to any user or group.

1.3.4 Editing an Alert Command

You may edit the newly created alert commands by clicking on Alerts -> Commands.

Susi2.png

To edit an alert command, click on the command's name.

Susi6 5.png

Once the chosen alert has been modified, click on the 'Update' button.


Template warning.png

The commands named eMail, Internal Audit and Pandora FMS Event cannot be modified.

 


1.3.5 Operations of an alert command

Susi7.png

Delete: To delete an alert, click on the grey trash can icon to the right of the alert.

Copy: Alerts can be copied. It is especially useful to generate commands similar to other existing ones by changing some detail such as a field or group.


The alerts ”eMail”, “Internal Audit” and “Monitoring Event” cannot be deleted nor copied.

1.3.6 Examples of Commands

1.3.6.1 Sending alerts with Jabber

It is very easy to set up Pandora FMS to send alerts by using a Jabber Server. Jabber can be used as a system to get real time alerts as well as a history log, allowing a group of people to receive those alerts simultaneously.

1.3.6.1.1 Installing Jabber Services

Procedure for the Client:

  1. Please install a Jabber client like Pidgin.
  2. Register an account under 'Pidgin' by clicking on the 'Accounts' tab to configure the account.
  3. Login to that account.

Procedure for Pandora FMS Server:

  1. Install the package named sendxmpp. This tool enables sending Jabber messages.
  2. Create a file named '.sendxmpprc' within the '/home' folder.
  3. Edit that file and insert the following text:
  useraccount@jabber.org password
  1. Grant permissions to the folder:
 chmod 0600 .sendxmpprc

Private messages can be sent using the command line, for instance:

  $ echo "Hello" | sendxmpp -s pandora useracount@jabber.org 

To register the alert within the Pandora FMS Web Console and to add a new command and configure its variables, do the following:

  • Field_1: The Jabber address.
  • Field_2: The text to be sent.

The alert is defined as follows:

  echo _field2_ | sendxmpp -s pandora _field1_
1.3.6.1.2 Additional examples of Jabber usage

Send this to a chat room:

  $ echo "Dinner Time" | sendxmpp -r TheCook --chatroom test2@conference.jabber.org

Send the log entries to a Jabber destination:

  $ tail -f /var/log/syslog | sendxmpp -i sysadmin@myjabberserver.com


NOTE:|Be careful not to overload public Jabber servers or they will cut off access.

1.3.6.2 Sending Emails by the Expect Script

Sometimes, it is necessary to use an authenticated SMTP to send emails. Pandora FMS has everything necessary for normal email forwarding in general console configuration and there you may even send an email to check the forwarding system. However, it is probably easier and more versatile using a simple Expect script instead of configuring sendmail to use an authenticated SMTP.

Expect is a tool to automate interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc. Expect makes those things to be trivial and it is also useful to test those same applications. Expect can make all kinds of tasks that are usually impossibly difficult with any other tool easier. You will see Expect is an invaluable tool - you may automate tasks you never thought of automating before - and you will be able to do so easily.

This is an example using EXPECT to send emails by using an MS Exchange® Server:

A file called /etc/snmp containing the following script is created:

#!/usr/bin/expect -f
set arg1 [lindex $argv 0] 
set arg2 [lindex $argv 1]
set arg3 [lindex $argv 2]
set timeout 1 
spawn telnet myserver.com 25 
expect "220"
send "ehlo mymachine.mydomain.com\r"
expect "250"
send "AUTH login\r"
expect "334"
send "2342348werhkwjernsdf78sdf3w4rwe32wer=\r"
expect "334"
send "YRejewrhneruT==\r"
expect "235"
send "MAIL FROM: myuser@domain.com\r"
expect "Sender OK"
send "RCPT TO: $arg1\r"
expect "250"
send "data\r"
expect "354"
send "Subject: $arg2\r"
send "$arg3 \r\r"
send ".\r"
expect "delivery"
send "quit"
quit

File permissions are changed to allow execution:

chmod 700 /root/smtp 

Before trying to use it, make sure that /usr/bin/expect works properly, by by copying, saving, granting execution permissions to the following script:

#!/usr/bin/expect -f

spawn date
sleep 20
expect

To use this together with Pandora FMS, create a new command (or modify the already existing email alert-sending command) and specify the following fields within the Pandora FMS Alert Command definition in the field named Command. Write the following:

/root/smtp _field1_ _field2_ _field3_

The script can be located anywhere on the system. Just keep in mind that the alert script is launched by the server that processes the data. If it is a network data, it will be the network server, if it is a data coming from an agent, through an XML file, then the dataserver will be the one to launch it.

If you have several physical servers, it is possible that you need to copy the same script to the same location, along with the same permissions and the same owner on all the systems where you have a Pandora FMS Server running and want to execute this alert on. Remember that Pandora FMS Network Servers must be executed as 'root' (e.g. for being able to execute ICMP latency tests). However, dataservers can be started by any user without special privileges.

The alert will be executed by the user who is executing the Pandora FMS Server process.

1.3.6.3 Sending SMS through Gnokii

To use Gnokii, use a Nokia cellphone or one compatible with Gnokii (check the compatible hardware list on the Gnokii project page). You will also need a USB data cable connected the cellphone and a connection to the Pandora FMS Server you intend to send the SMS Alerts from.

Gnokii supports a large variety of Nokia cellphones and some models by other manufacturers. By using Gnokii, you may send SMS directly from the command line directly from a Pandora FMS Server, thereby avoiding the use of SMS forwarding gateways through the Internet or GSM hardware solutions for sending messages which are very expensive in some countries.

This is an example of sending an SMS from the command line using Gnokii:

echo "PANDORA: Server XXXX is down at XXXXX" | gnokii --sendsms 555123123

Gnokii is unable to send an SMS with images attached, but it can send a URL via HTTP or WAP for it to be displayed when receiving a message, for example:

echo "Image capture sample" | gnokii --sendsms 555123123 -w http://artica.homelinux.com/capture.jpg

It can also send an image's URL or one that leads to a 'light version' of the console in order to provide console access and data analysis to the user.

The development team has tested sending SMS alerts from a Nokia 6030 cellphone when Internet connection was not available. The Nokia 6030 cellphone uses the module's 6510 definition within the .gnokiirc file. It takes about four seconds to send an SMS.

An alternative to the use of Gnokii is the Gamu project. It is possible to install a much more powerful sending gateway using Gammu.

1.3.6.4 Executing a Remote Command on another System (UNIX)

Sometimes, it is pretty interesting to execute the command on another system by means of the ssh command. The system where the command is executed should be a UNIX system and have the ssh daemon installed, started and accessible.

To avoid storing the access password on the machine that executes the command within the Pandora FMS Console, copy the server's public key to where you intend to execute the remote command on the Pandora FMS Server.

Once done, execute the following command:

ssh user@hostname [_field1_]

By using _Field1_ as a variable, you may execute the command required for the alert.

1.4 Alert Actions

1.4.1 Introduction

Actions are the components of alerts where a command is related to the generic variables Field 1, Field 2,..., Field 10.

Actions allow defining how to launch the command.

1.4.2 Creating an Action

New actions are created by clicking on Alerts -> Action and Create.

Accion1.jpg

Once you have clicked on 'Create', this window will appear:

Accion2.jpg


Name
The action name.
Group
The action group. User can only assign a group to which the user creating the action belongs, unless that user explicitly belongs to the ALL group. If the associated command has a group other than All, only the group associated with the command or the All group can be set as the action group. If for some reason this is deferred, you will see a warning message for prompt correction by a user who has the necessary rights.
Alert-alert actions-group command and group action mismatch.png
Command
The command used in case of a triggered alert. You may choose among several predefined commands under Pandora FMS.
Threshold
The action's execution threshold.
Command preview
The command to be executed on the system appears here automatically.
Field 1-10
The values of the macros from '_field1_ through '_field10_' are defined here. They are intended to be used along with the command if necessary. These fields might be a text field or a selection combo if configured.

Once you have filled out the fields, click Create.

Boton1.jpg

To edit newly created actions, click on Alerts and Actions.

Template warning.png

When we assign a value to the fields "Field" in the triggered section, by default they will be the same values for recovery, unless we assign a different value.

 


1.4.2.1 Action macros

The macros that can be used in the actions are in the Macro list at the end of the chapter.

1.4.3 Editing an Action

Alert action.png

To edit an action, click on the action's name.

Alert action edit.png

Once you have completed the changes, update them by clicking on the Update button.


1.4.4 Deleting an Action

To delete an action, click on the gray trash icon which is located at the right side.

Sipo.jpg

1.5 Alert Templates

1.5.1 Introduction

Templates define alert triggering terms (when the action should be executed).

Alert templates are associated to modules, so theta ehen the template terms are met, the linked action(s) are executed.

Their design allows generating a small generic template group that is used for most cases in Pandora FMs.

1.5.2 Creating an Alert Template

Go to the Alerts menu > Templates. You can create a new template by clicking on the Create button.

Planti.jpg

1.5.2.1 Step 1: General

Templform.JPG

You may specify in the template wizard:

  • Name: The name of the template.
  • Group: The group to which the template will be applied. User can only assign a group to which the user creating the alert command belongs, unless that user explicitly belongs to the ALL group.
  • Description: It describes the template's function. It is useful to distinguish the template from others within the alert's general view.
  • Priority: The field which provides information about the alert. The generated event will inherit this priority whenthe alert is triggered. It is useful for filtering when searching for alerts.

You may choose among the following priorities:

    • Maintenance
    • Informational
    • Normal
    • Warning
    • Critical

1.5.2.2 Step 2: Conditions


Templform2.JPG


Days of Week
It sets the days the alert could possibly be triggered. Those that have the verification checkbox checked will be those affected by the alerts.
Set initial time
Time from which the alert action will be executed.
Set end time
Alert action time limit.
Time Threshold
Time required to reset the alarm counter. It defines the time interval where it is guaranteed that an alert will not be triggered more than the maximum number of alerts. After the defined interval, the counter will be reset. The restart of the trigger counter will not be restarted if the alert recovers when a correct value arrives, unless the value Alert recovery is activated, in which case the counter will restart immediately after receiving a correct value.
Min number of alerts
The minimum number of times the terms set on the template have to be met (counting from the number defined in the module's FlipFlop parameter) so that an alert is triggered. The default value is '0', which means the alert will be triggered when the first value that meets the terms is received. It is intended to work as a filter, which might be useful to ignore false positives.
Max number of alerts
Maximum number of alerts which could be sent consecutively within the same time interval (time threshold). It is the maximum alert counter value. No more alerts than the specified number will be received by interval.
Reset counter for non-sustained alerts
Activating this token depends on the number indicated in Min. number of alerts being higher that 0. Activating this token restarts the alert counter when the indicated condition is NOT repeated consecutively. For instance, if the field "Min. number of alerts" has a value of 2, it will mean that the module must go through the status assigned in "Condition type" 3 times to fire the alert. There are two options for this last token:
    • If the restart token is checked, it will be necessary for the number of critical states must be consecutive, otherwise the counter will restart.
 normal -> critical -> critical
    • If the restart token is not checked, the alert will fire after an alternative sequence or where there are continuous critical states:
 normal -> critical -> normal -> critical -> normal -> critical


Disable event
By checking this token, the event generated in the alert trigger event view will not be created.
Default Action
The default action the template will have is defined in this combo. This will be the action to be automatically created when the template is assigned to the module. You may assign one or none, but you cannot assign several default actions here.
Condition Type
The field where the type of condition to be applied on the alert is defined.
Regular.jpg

The required combos according to the selected type will be added. They are the following:

    • Regular Expression: A regular expression is used. The alert will be triggered if the module's value meets certain requirement. By choosing the 'regular expression' condition, the possibility of selecting the trigger box appears if it matches the value. If you select it, the alert will be fired if the value matches. If not, the alert will be fired if the value does not match.
Regular expression triggered.png
    • Max and Min: The used maximum and a minimum values. When checking 'Trigger when matches the value', the alert will be launched when the value is within the indicated range between maximum and minimum and, if not marked, the alert will be launched when the value is outside the selected range.
Notmaxmin.png
    • Max: The maximum value used. The alert will be triggered if the module's value is higher than the defined 'maximum' value.
Max.png
    • Min: The used 'minimum' value. The alert will be triggered if the module's value is lower than the defined minimum value.
Min.png
    • Equal to: Used to trigger the alert when a value must be equal to the received data. This condition, like max/min, is used only for numeric values, e.g. 234 or 124.35.
Equal.png
    • Not Equal to: It is the same as previous one but denying the condition (logic operator NOT).
Notequal.png
    • Module status: The module is used, whether its status (Critical status, Warning status, Unknown status, Not normal status) or its value change.(On change, however, when unchecking the verification checkbox Triggered when the value matches allows firing if the value is the same) or just Always, if necessary.
Status template.png

1.5.2.3 Step 3: Advanced fields

Click on the image to enlarge
Alert recovery
Combo where alert recovery is enabled or disabled. In the event that alert recovery is enabled, when the module no longer meets the terms specified in the template, the action linked to the arguments specified by the fields defined in this column will be executed.
Field 1 ~ Field 10
Here you may use a series of macros described later.

Once all appropriate fields have been filled out, click on 'Finish'.

1.5.3 Replaceable macros within Field1 ~ Field10

It is possible to use the macros within the list of macros at the end of this chapter in all instances of Field1,... Field10 (both in the alert template, as well as the command and the action). These are keywords that will be replaced if executed by a value. That value is depends on the time, value or agent that triggered the alert, etc.


1.5.3.1 Complete example of an alert containing replacement macros

Create a LOG entry where in each line the following format appears:

2009-12-24 00:12:00 pandora [CRITICAL] Agent <agent_name> Data <module_data> Module <module_name> in CRITICAL status

Command Configuration

echo _timestamp_ pandora _field2_ >> _field1_

Action Configuration

Field1 = /var/log/pandora/pandora_alert.log
Field2 = <left blank>
Field3 = <left blank>

Template Configuration

Field1 = <left blank>
Field2 = [CRITICAL] Agent _agent_ Data _data_ Module _module_ in CRITICAL status
Field3 = <left blank>

In the recovering section:

Field2 = [RECOVERED] [CRITICAL] Agent _agent_ Data _data_ Module _module_ in CRITICAL status
Field3 = <left blank>

If an alert is triggered, the following line will be added to the log:

2009-10-13 13:37:00 pandora [CRITICAL] Agent raz0r Data 0.00 Module Host Alive in CRITICAL status

And the this one to recover the alert:

2009-10-13 13:41:55 pandora [RECOVERED] [CRITICAL] Agent raz0r Data 1.00 Module Host Alive in CRITICAL status

1.5.4 Editing a template

Go to Alerts -> Templates and click on the template's name to edit it.


Plantilla.jpg

1.5.5 Deleting a template

To delete a template, click on the gray trash icon located at the right of the alert.

Cruz.jpg

1.6 Assigning alert templates to modules

Once the basic information about the alert system is known, we will show you the possible ways to assign alerts to the modules.

1.6.1 Alert management from alert submenu

1.6.1.1 Assigning alerts from alert submenu

Got to section Alerts > List of Alerts. From that section you may create new alerts by clicking on the pencil icon (Builder alert), configure the fields:


Pinar.jpg


These are the fields to fill in:

Agent
Smart auto-completion to choose the agent.
Module
Module list of the previously selected agent.
Actions
Action to be executed when the alert is triggered. If the template has a default action, Default may be kept.
Template
Template that contains alert triggering terms.
Threshold
The alert action to not be executed more than once every action_threshold seconds, regardless of the number of times the alert is triggered.

1.6.1.2 Modifying alerts from an alert's submenu

Once an alert has been created, it is only possible to modify the actions that have been added to the template's action.

It is also possible to delete the action that was selected when the alert was created by clicking on the gray trash icon located at the right side of the action, or to add new actions by clicking on the 'Add' ('+') button.


Modifica.jpg


1.6.1.3 Disabling alerts from an alert's submenu

Once the alert has been created, it is possible to deactivate it by clicking on the light-bulb icon located at the right side of the alert's name.


Desha.jpg


1.6.1.4 Deleting alerts from the alert submenu

It is possible to delete any alert by clicking on the gray trash icon located at the right side of the alert.

Filter.jpg

1.6.2 Managing alerts from within the agent

1.6.2.1 Alert assignment within the agent

From the agent management section, new alerts can be added by clicking on the corresponding tab.


Wiz agent alerts.png



These are the form's available fields:

Module
Agent module list.
Actions
Action executed when the alert is triggered. If the template has a default one, leave it as Default.
Template
Template that contains alert triggering terms.
Threshold
The alert action will not be executed more than once every action_threshold seconds, regardless of the number of times the alert is triggered.

1.6.2.2 Modifying alerts from within the agent

Once an alert has been created, it is only possible to modify the actions that have been added to the template's action.

It is also possible to delete the action selected at the moment the alert was created by clicking on the gray trash icon which is located at the right side of the action, or to add new actions by clicking on the 'Add' button.

Agent edit alerts.png


1.6.2.3 Deactivating alerts from the agent

Once an alert has been created, it is possible to deactivate it by clicking on the light bulb icon located at the right side of the alert's name.

Wiz agent disable alert.png

In the example image, the second alert is disabled (note that the font color and the disabled alert icon are light grey).


1.6.2.4 Deleting alerts from within the agent

It is possible to delete any alert by clicking on the gray trash icon which is located at the alert's right side.


Wiz agent delete alert.png


1.6.2.5 Alert detail

By clicking on the magnifying glass icon on the button panel of alert options, you will see a summary page of the effective alert setup.

This is the screen where each of the settings selected for the alert can be confirmed:


Agent alert summary.png

Select a specific action from the Actions dropdown to see an example of the final command:

Agent alert summary action.png


1.7 General alert overview

1.7.1 Defining a threshold

For a module called "CPU Load", critical and warning thresholds will be defined.


Cpu1.JPG


Access the module's editing form to set the following thresholds:

CpuTwo.png

Accept and save the changes. When the value of the module CPU Load is between 70 and 90, its status will change to WARNING, and when between 91 and 100, it will become CRITICAL, showing its status in red as seen here:


Cpu3.JPG


1.7.2 Configuring an action

Create an action that is "Send an email to the operator". Go to the menu: Alerts > Actions and click Create:

Qgcpu5.png

This action uses the command eMail and its fields Field1, Field2 and Field3 which correspond to the target address, email subject, and message body.




1.7.3 Configuring an alert template

A generic alert template will be created for any module in critical status, and its default action will be to notify the group of operators by email. Define the template from the Templates section:

Step 1:

Qgcpu6.png


The priority set here as Informational will be used to display the event in a certain color when the alert is triggered.

Step 2 specifies the parameters that determine the specific triggering terms, such as the state the module should have or the time intervals when the template will work.

Step 2:

Qgcpu7.png


The most important parameters in this step are:

Condition type
It determines whether the alert will be triggered by a status change, a value change, etc. It is the most important parameter for the alert to work as desired. The Critical status term would be used to trigger the alert when a module is in a critical status.
Default action
The action to be executed by default when the alert is triggered. It is optional.
Time threshold
Time during which the alert will not be repeated if the incorrect status is continuously kept. If left at one day (24 hours), it will only send the alert once every 24 hours even if the module remains longer in faulty status.
Min. and Max. Number of alerts
Minimal and maximal number of times the condition must be met (in this case, for the module to be in critical status) before Pandora FMS executes the actions associated to the alert's template. With minimal value set as 0, the firt time the module is wrong, the alert will be triggered.

Step 3:

Click to zoom in

In section 3, Field1, Field2, Field3, etc. will be used to transfer parameters from the template to the action, and from the action to the command. In addition, in this third section alert recovery can be enabled or disabled, which consists of executing another action when the situation goes back to normal.



1.7.4 Associating an alert to a module

Now just link the alert template to the module. To do so, go to the Alert tab within the agent where the module is located:

Qgcpu9.png

Here the module named cpu_user and the critical condition alert template have been linked. It will show the predefined action in this template ('Send email to XXX') by default.

1.7.5 Scaling alerts

Once a complete alert has been associated with a module, it is possible to add additional actions that are executed if the alert is repeated a certain number of times consecutively. That is called scaling alerts.

Add the additional actions and determine between which consecutive alert repetitions this action will be executed, as shown in the following capture:

Alert1.JPG

When an alert is recovered, all actions that have been executed up to that point will be re-executed, not just those that correspond to the Number of alerts match from current setting.

In addition, a second threshold on the basis of which it will not be possible to send an alert more than once within said interval.


1.7.6 Alert message forwarding through instant messaging

  1. Telegram is an instant messaging platform through which you may receive alert messages from Pandora FMS. Find out more in this video tutorial "Notifications Telegram: Pandora FMS". Through this video tutorial you may put into practice, with a guide (rom minute three), how to create and configure all components of a Pandora FMS alert.

1.8 Stand-By Alerts

Alerts can be defined as 'active', 'deactivated' or in 'standby'. The difference between 'deactivated' and 'standby' alerts is that 'deactivated' alerts are not fired at all. They will not be shown in the alert's view either. 'Standby' alerts will be shown in alert view. They will also work but only on the display level. They will show whether they are triggered or not, but they neither perform the assigned actions nor generate events.

Standby alerts are useful to see what happens, but they have notifications and actions disabled.

1.9 Cascade Protection

Cascade Protection is a Pandora FMS feature which allows to avoid alert overload if a group of agents cannot be reached due to a connection failure. These kinds of things tend to happen if an intermediate device such as a router or a switch is down and most of the network managed by Pandora FMS becomes unaccessible. Since network checks would not work in that situation, alerts would start being triggered due to down devices, despite not being true.

Recursive cascade protection ilustration.png

Cascade protection is enabled from agent configuration. Click on the Cascade protection option.

Down1.jpg

In order for the agent to work with cascade protection, the parent agent it depends on must be correctly configured. If the parent agent currently has any critical-state module alert triggered, the lower agent with cascade protection will not execute its alerts. This does not apply to module alerts in WARNING or UNKNOWN status.


1.9.1 Examples

These are the agents at your disposal:

  • Router: ICMP check module and SNMP check module, using a standard OID to verify the status of an ATM port. Latency towards your provider's router can be verified too.
  • Web Server: It has several modules executed by the agent: CPU, Memory, Apache process verification. It also has a four-step WEB latency check.
  • Database Server: It has several modules executed by the agent: CPU, Memory, MySQL process verification. It has some additional BBDD integrity checks. It can also verify connectivity to another database remotely, using a specific plugin that logs in, queries and exits, measuring the total time.

In WEB SERVER and DATABASE SERVER, ROUTER is defined as parent. Check the cascade protection check-box in WEBSERVER and DATABASE SERVER. Now define several simple alerts:

  • ROUTER

SNMP Check / CRITICAL -> Action, send MAIL. Latency > 200ms / WARNING -> Action, send MAIL.

  • WEB SERVER

CPU / WARNING MEM / WARNING PROCESS / CRITICAL -> Action, send MAIL. HTTP LATENCY / CRITICAL -> Action, send MAIL.

  • DATABASE SERVER

CPU / WARNING MEM / WARNING PROCESS / CRITICAL -> Action, send MAIL. SQL LATENCY / CRITICAL > Action, send MAIL.

If ROUTER connection fails, by means of which Pandora FMS connects with WEB SERVER and DATABASE:

  • If cascade protection is disabled, you will receive six alerts. Imagine what would happen if you had 200 Servers connected instead of six. That is the reason for why it is sometimes called an 'Alert Storm'. In worst-case scenarios, this problem may kill your Mail and Monitoring Servers or your cellphone, because they get flooded by lots of alerts or SMS messages.
  • However, if you have the cascade protection enabled, you will only receive one alert, saying that the router ATM interface is down. You will still see the Web and Database Servers in red, but you will not receive the alerts.

1.9.2 Service-based cascade protection

Info.png

Version NG 727 or higher.

 


Services can be used to avoid multiple source alerts reporting the same incident.


If service-based cascade protection is enabled, the service elements (agents, modules or other services) will not report problems, but the service itself will alert on behalf of the affected element.


Proteccion cascada1.png



If the element 192.168.10.149 goes into critical status without affecting the rest of the service, the operator will receive an alert pointing out that 192.168.10.149 is down, but the service Service works normally.



In order to receive this information, edit or create a new alert template, using the _rca_ macro for a root cause analysis.

_rca_

This macro will provide info about the affected 'path' in the service to the operator.

Proteccion cascada2.png


For example, the value of the macro _rca_ corresponding to the service status in the image would be:

[Service -> web_service -> 192.168.10.149]


Though the status of the service would be correct, since it does not exceed 50% of the components in critical status (you may get more information about this in the Services section.

Observation: The chain of events represented in root cause analysis represents the critical elements within a service, allowing to see what elements are affecting the service.



1.9.3 Cascade protection based on modules

The state of a module of a parent agent can be used to avoid agent alerts in case the module of the parent agent goes into critical state.

Alerta-modulo.png

1.10 Safe operation mode

Safe operation mode.png

Safe operation mode can be enabled in agent advanced configuration options.

If the selected module's status becomes critical, the rest of the agent's modules are disabled until it goes back to warning or normal again. This allows, for example, to disable remote modules if connectivity is lost.

1.11 List of special days

Pandora FMS allows to define a list of special days for holidays and vacations that can be used in the template configuration, so that during those days alerts are not triggered.

1.11.1 Creating a Special Day

New special days are created in the Alerts > List of special days section, by clicking on Create underneath the calendar.



Creating special day61-1.png



Once one of them has been clicked, fill in the following parameters:



Creating special day2.png



Date
The special day's date. The data format is YYYY-MM-DD. If you want to define the same day every year, you may use *-MM-DD.
Group
Select here the group to which the special day applies. User can only assign a group to which the user creating the special day belongs, unless that user explicitly belongs to the ALL group.
Same Day of the Week
Select a day. The date in field Date is dealt with the same as that weekday.

Once the ields have been selected, click Create.

Practical example 1

Suppose that May 1st, 2021 is holiday. If you define the 2021-05-01 as Sunday, that day becomes the same as Sunday, instead of the corresponding day.

Practical example 2
Considering May 1st of every year as a holiday, if you define *-05-01 as Sunday, that day will be considered as Sunday, instead of the corresponding day.
Practical example3

There is a template that alerts you from Monday to Friday from 8 am to 6 pm but on Saturdays and Sundays this template will not cause any alert to go off. The 15th August is Wednesday and it is holiday, so a special day is created and in the field Same day of the week Saturday or Sunday is chosen, so alerts of any kind will not be received on August 15th as it will be as if it was Saturday or Sunday, when the template is configured to not trigger any alerts.

Once the fields have been selected, click on "Create".

1.11.2 Creating special days in bulk from an .ics file

Special days can also be created using an iCalendar file (. ics). These can be imported at the top of the window. Once imported, the corresponding data will be recorded in the current month.



Creating special day ics.png



1.11.3 Editing a special day

You may edit special days created within the List of Special Days option by clicking on Alerts.



Editing special day61-1.png



To edit a special day, click on the wrench icon next to the corresponding special day.



Editing special day2.png



Once the changes are completed, click on Update.

1.11.4 Deleting a Special Day

In order to delete a Special Day, click on the gray trash icon located next to the Special Day wrench icon.



Deleting special day61.png



1.12 Complete Alert Examples

1.12.1 Sending SMS Alerts

You must have a tool that allows sending SMS installed, such as smstools. Suppose you have already configured your SMS account. Enter the following command:

> sendsms 


Enter two parameters destination and message:

<destination> 'Full message'

Enter the full destination number (e.g. 346276226223 for Spanish phone numbers) and the message between simple quote marks (' y ').


That way you may use a command for the alert to be created in Pandora FMS management interface:



Smsalert sample1.png



For this command Field 1 will be the dastination phone number and Field 2 the message itself. Remember these fields will be sent to the added to the alert and their values may be taken of replaced, for that readon in the previous image the destination phone number for the example was "123456789".

Now configure the action for that command:


Smsalert sample3.png

This action executes the previously defined command, replacing Field 1 and Field 2 by custom values. Field 1 will be the destination phone number ("346666666666" in the example of the previous image), and Field 2 the text defined for this action ("Hello" in the example of the previous picture).

Info.png

In Pandora FMS, you may use a word (alphanumeric) for the destination phone number, but bear in mind that some mobile operators do not manage alphanumeric identifications well.

 


For the following step you may use an existing alert template or a new one:

Smsalert sample5.png

In this case the alert template will only be triggeres when a module goes into critical status.

Once this is defined, configure the alert to be fired once a day at most. But if it recovers, it will be launched again each time it recovers and gets fired again; see the following picture.

Smsalert sample6.png

Now just assign a module to an alert template and an alert action:

Smsalert sample4.png

On a CPU workload module, set a value under 20 to be able to test message forwarding. Look at the following screen:

Smsalert sample7.png

Now, to finish off, you may "force" the alert's execution, that means, executing the agent's alert immediately; go to the agent's alert view and click on the green circle icon.

Smsalert sample8.png

You should receive an SMS in your mobile phone, as the following picture shows. Notice the test message was replaced by the text "aeryn"; in addition, the CPU workload value shows "N/A" because, when forcing an alert, it does not receive any real data by the module, since it has not had enough time to retrieve any data.

Smsalert sample9.png

1.12.2 Using alert commands different from email

Template warning.png

Pandora FMS is mainly flexible, so that it can always be useful. The following procedures is an advanced one and must always be taken as an exception to the rules. That is how real life is, sometimes you need to go beyond'.

 


The email is hosted as command in Pandora FMS, because Field 1, Field 2 and Field 3 are fields clearly intended to be used as addressee, subject and message text - but, what if you intend to execute a user-defined action?

This is how a new command is custom defined. Suppose you intend to generate a log file entry for each alert alert find. The format of the log file should be something like this:

DATE_HOUR - NAME_AGENT -NAME_MODULE -VALUE- PROBLEM DESCRIPTION

VALUE is the module's value at that specific moment. There will be several log file entries, depending on the action that calls the command. The alert will define the description and the file to which the events are added.


To accomplish this, first create a command like this one:


Qgcpu11.png

Then, define an action:


Qgcpu12.png


When the alerts are executed, the log file created will look similar to this:

2010-05-25 18:17:10 - farscape - cpu_user - 23.00 - Custom log alert #1

The alert was triggered at '18:17:10' within the agent named Farscape, in the module named cpu_sys containing the data 23.00 and the description entered when the action was defined.

As for command execution, the field order and the other things might make you not to understand very well how the command is eventually executed, the easiest way is activating the Pandora FMS Server's debug traces within the server's configuration file located at /etc/pandora/pandora_server.conf. Restart the server by entering /etc/init.d/pandora_server restart, look for the file named /var/log/pandora/pandora_server.log and look for the exact line which contains the execution of the user-defined alert command to see how Pandora FMS Server triggers it in detail.

1.12.2.1 Complete example of an alert with substitution macros

Suppose for a moment you intend to generate a log entry where each line is supposed to show its data in the following format:

2009-12-24 00:12:00 pandora [CRITICAL] Agent <agent_name> Data <module_data> Module <module_name> in CRITICAL status

Command configuration:

echo _timestamp_ pandora _field2_ >> _field1_

Action configuration:

Field1 = /var/log/pandora/pandora_alert.log
Field2 = <left blank>
Field3 = <left blank>

Template configuration

Field1 = <left blank>
Field2 = [CRITICAL] Agent _agent_ Data _data_ Module _module_ in CRITICAL status
Field3 = <left blank>

In the recovery section:

Field2 = [RECOVERED] [CRITICAL] Agent _agent_ Data _data_ Module _module_ in CRITICAL status
Field3 = <left blank>

When executing an alert, the following line is entered in the log:

2009-10-13 13:37:00 pandora [CRITICAL] Agent raz0r Data 0.00 Module Host Alive in CRITICAL status


And the following line to recover the alert:

2009-10-13 13:41:55 pandora [RECOVERED] [CRITICAL] Agent raz0r Data 1.00 Module Host Alive in CRITICAL status

1.13 Custom module alert macros

Any amount of macros may be added to an agent module.



Add module macros.png


These macros have the following characteristics:

  • They are defined in the module configuration section.
  • They store information in the database.
  • They may have any name, for example: _joey_
  • They do not affect local configuration. files(pandora_agent.conf).
  • They are exclusively used in alerts.
  • They cannot be defined at local component level.
  • They can be defined in policies.

These specific macros can be added by just extending the module macros section.



Module macros.png


Macro values can be used as part of the fields in alert definition.

For instance: To include a macro to the mail sending action, the field with the e-mail body must be configured in the following fashion:



Campos alertas.png


If a module is added to this alert without any defined custom macro, then no information will be displayed in the macro value section.

1.14 Quick guide to email configuration for alerts in Pandora FMS

Pandora FMS on it own has the ability to send emails as explained in general console settings.

However, its flexibility allows sending emails with different email platforms.

1.14.1 Email configuration with Gmail

For Pandora FMS server to send alerts through (Gmail) proceed to the console's general configuration or Pandora FMS server configuration and enter your credentials (Office365 web domain, usernames, password, etc.).

1.14.1.1 Action configuration

Add an action, for example with the name Mail to Admin, and to configure the email addressee use the command eMail:

GMAIL1.png
1.14.1.2 Action configuration

In this case, in the configuration of Module > Alerts, a new alert with the following module has been generated:

GMAIL2.png

Once the alert fires, you may check how it reaches the action-chosen email:

GMAIL3.png
GMAIL4.png

1.14.2 Email configuration with Office365

Pandora FMS can use Office365 by means of the following configuration:

  • You must have an Office365 account.

1.15 Alert correlation: event and log alerts

Info.png

Version NG 741 or higher.

 


From Pandora FMS version 741, alerts can be build based on received events or logs collected by means of the log collection system. Simple or complex alerts may be created based on a set of terms with logic relations. This feature replaces the previous one of event alerts.

Pandora FMS allows working from a much more flexible perspective, since alerts are not generated according to the status of a specific module, but on an event -which may have been generated by several different modules of different agents.

Event alerts are based on filtering rules using logical operators:

  • and
  • or
  • xor
  • nand
  • nor
  • nxor

These logic operators look for events/expressions in logs hat match the configured filtering rules, and if matches are found, the alert will fire.

They also use templates to define some parameters, such as the days on which the alert will work. However, in this case the templates do not determine when the event alert is triggered, but rather it is through the filtering rules that the events that match will be searched and the corresponding alerts will be triggered.

It is recommended to use a new rule editor that allows building alerts visually can be used. The old event alert editor will still be available for some time.

Info.png

Given the high number of events Pandora FMS Database is able to store, the server works on an maximal event window which is defined in the pandora_server.conf configuration file by a parameter named 'event_window'. Events generated outside the specified time range will not be processed by the server. So it does not make any sense to specify in a rule a time range wider than the one configured within the Server.

 



Template warning.png

To create event alerts it is mandatory to define the following parameters: agent, moduleand event.

 



1.15.1 Correlation alert creation

Template warning.png

For event correlation alerts to work, activate the event correlation server by means of the parameter eventserver 1 in Pandora FMS server configuration file.

 



1.15.1.1 Correlation alerts / templates

To configure a correlated alert you will have to access through the menu to the section Alert correlation.

Alertaslog2.png


In this global overview, you may see the fields:

Alertaslogprin.PNG



Sort
Marks the order of the evaluation of correlated alerts valuing if it is configured as pass or drop. The higher it is in the list the sooner the alert will be evaluated.
Name
Name of the alert.
Group
group where it has been organized. The user will be able to see only the groups to which he/she belongs, unless the user explicitly belongs to the ALL group.
Matched
how many times has been detected an event that coincides with the trigger rule in the current threshold.
Triggered
how many times the alert has been launched in the threshold that has been configured.
Action
It shows the actions configured in the alert.
Options
It allows to operate with the action disabling it, putting it in standby, adding more actions, editing or deleting.

Create a rule and define its performance (similar to Alert Templates creation):

Alertaslog12.PNG


Alertaslog13.PNG


The configuration parameters of the templates for correlation alerts are similar to those of a module alert. There are only two specific parameters for event alerts:

  • Rule evaluation mode: It can be Pass or Drop. Pass means that, in case an event coincides with an alert, the rest of the alerts are still evaluated. Drop means that, in case an event coincides with an alert, the rest of alerts are not evaluated.
  • Group by: It allows to group the rules by agent, module, alert or group. For example, if a rule is configured to jump when two critical events are received and it is grouped by agent, two critical events should arrive from the same agent. It can be deactivated.

{Warning|In case of alerts that contain log rules, it will only affect the grouping per agent. If you choose a different grouping, alerts based on log entries will never be fulfilled.}

Each rule is configured to jump to a certain type of event or log match; when the logical equation defined by the rules and their operators is met, the alert is fired.

1.15.1.2 Rules within a correlation alert

Alertaslog14.PNG


To define these rules, drag the elements from the left side to the drop area at the right to create your rule.

Configuration available items:

Alertaslog3.png


These elements will be enabled to guide the user to meet the rule's grammar. Hereon the grammar to be used is further explained:

S -> R | R + NEXUS +R
R -> FIELD + OPERATOR + C | FIELD + OPERATOR + C + MODIFIER
C -> VARIABLE

Where S is the set of rules defined for the correlated alert.

Drag the element to the rule definition area:

Alertaslog4.png


To clean up and undo all changes, the Cleanup and Reset buttons are available.

Template warning.png

No changes will be saved until you click on the Next button.

 


Info.png

REMEMBER: Blocks are simultaneous when meeting a term.

 


(A and B) 

It forces the analyzed item (whether event or log) to meet A and B simultaneously.

A and B 

It forces both rules (A) and (B) to be met in the evaluation window. This means that during the last couple of seconds (defined by log_window and event_window parameters), entries that meet both rules must exist.

1.15.1.3 Fields within a correlation alert

To understand the operation of this section, see "Alert system".

1.15.1.4 Triggering within a correlation alert

In this section we configure the actions that we are going to perform when the alert is triggered and indicate at what intervals and how often it is going to execute such action.

Globaltriggering.png


In this section we see a preview of the configuration that we have done in the section Conditions to take into account when configuring the execution of an action.

Configactionsalert.png


At the bottom we can configure the actions through the fields:

Actions
action that we want to execute.
Number of alerts match
number of intervals that have to pass since the alert is fired to execute the action. If we want it to be always we should leave these fields blank.
Threshold
Interval that has to pass for the action to be executed again once the alarm is fired.

Then we visualize the list of configured actions. In this list, the field triggering shows us in which intervals of the alert will be executed the action as we configured in number of alerts match. In addition, in the column Options we can delete or modify the configured actions.

1.15.1.5 Multiple correlated alerts

When you have multiple alerts, these have a specific evaluation order. They are always orderly evaluated, starting by the first of the list.

If the PASS rule evaluation mode is configured, if a correlated alert is run, the next ones will also be evaluated. Tha is the "normal" mode.

If you configure the DROP rule evaluation mode, if a correlated alrt configured through this mode is run, the evaluation of the next rules will come to a halt. This feature makes cascade alert protection possible.


For example:

  • Generic alert.
  • Specific alert.

If the general alert fails, there is no need to evaluate the specific one. Configure both of them with DROP.

Click on the order icon and drag to change rule evaluation order.


Alertaslog9.png


The rest of correlation rules (action fields and action application) work similarly to the rest of Pandora FMS alerts and do not require additional explanation.

1.15.1.6 Event Alert macros

The macros that can be used in the event alerts are those at the end of this chapter in list of macros.

1.16 List of macros

Command macros, action macros and event alert macros are common but for the following exceptions _modulelaststatuschange_, _rca_ and _secondarygroups_


_address_
Address of the agent that triggered the alert.
_adress_n_
Address of the agent that corresponds to the position indicated in "n" e.g: address_1_ , address_2_
_agent_
Alias of the agent that triggered the alert. If there is no alias assigned, the name of the agent will be used instead.
_agentalias_
Alias of the agent that triggered the alert.
_agentcustomfield_n_
Agent number n custom field (e.g. _agentcustomfield_9_).
_agentcustomid_
Agent custom ID.
_agentdescription_
Description of the agent that triggered the alert.
_agentgroup_
Agent group name.
_agentname_
Name of the agent that triggered the alert.
_agentos_
Agent's operative system.
_agentstatus_
Current agent status.
_alert_critical_instructions_
Instructions for CRITICAL status contained in the module.
_alert_description_
Alert description.
_alert_name_
Alert name.
_alert_priority_
Alert’s numeric priority.
_alert_text_severity_
Priority level, in text, for the alert (Maintenance, Informational, Normal Minor, Major, Critical).
_alert_threshold_
Alert threshold.
_alert_times_fired_
Number of times the alert has been triggered.
_alert_unknown_instructions_
Instructions for UNKNOWN status contained in the module.
<b>_alert_warning_instructions_</code>
Instructions for WARNING status contained in the module.
_all_address_
All addresses of the agent that fired the alert.
_data_
Module data that caused the alert to be triggered.
_email_tag_
Emails associated to the module’s tags.
_event_cfX_
(Only event alerts). Key of the event custom field that triggered the alert. For example, if there is a custom field whose key is IPAM, its value can be obtained using the _event_cfIPAM_ macro.
_event_description_
(Only event alerts) Textual description of the Pandora FMS event.
_event_extra_id_
(Only event alerts) Extra id.
_event_id_
(Only event alerts) ID of the event that triggered the alert.
_event_text_severity_
(Only event alerts) Priority text about the event that triggered the alert (Maintenance, Informational, Normal Minor, Warning, Major, Critical).
_eventTimestamp_
Timestamp in which the event was created.
_fieldX_
User defined field C.
_groupcontact_
Group contact information. Configured when the group is created.
_groupcustomid_
Group custom ID.
_groupother_
Other information about the group. Configured when the group is created.
_homeurl_
It is a link of the public URL this must be configured in the general options of the setup.
_id_agent_
Agent ID, useful for building a direct URL with to Pandora FMS console.
_id_alert_
Alert ID, used to correlate the alert with third party tools.
_id_group_
Agent group ID.
_id_module_
Module ID.
_interval_
Module execution interval.
_module_
Module name.
_modulecustomid_
Module custom ID.
_moduledata_X_
Using this macro ("X" is the module name) the last piece of data of this module is collected, and if it is a number it is returned with the decimals specified in the console and its unit (if it has it). This could be useful for example for sending an email once a module alert is triggered, and also send additional information about other modules of the same agent (which could be very relevant).
_moduledescription_
Module description.
_modulegraph_nh_
(Only for alerts that use the eMail command) It returns an image encoded in base64 of a module graph with a period of n hours (e.g. _modulegraph_24h_). A correct setup of the connection between the server and the console's API is required. This setup is done in the server configuration file.
_modulegraphth_nh_
(Only for alerts that use the eMail command) Same operation as the previous macro, but with the critical and warning thresholds of the module provided they are defined.
_modulegroup_
Module’s group name.
_modulestatus_
Module status.
_moduletags_
URLs associated to the module tags.
_name_tag_
Names of the tags related to the module.
_phone_tag_
Phone numbers associated to the module tags.
_plugin_parameters_
Module plugin parameters.
_policy_
Name of the policy that the module belongs to (if applies).
_prevdata_
Module previous data before the alert was triggered.
_rca_
Root cause analysis chain (only for services).
_server_ip_
Ip of server assigned to agent.
_server_name_
Name of server assigned to agent.
_target_ip_
IP address for the module’s target.
_target_port_
Port number for the module’s target.
_timestamp_
Time and date on which the alert was triggered (yy-mm-dd hh:mm:ss).
_timezone_
Timezone that is represented on _timestamp_.