Pulse Secure-Uac-5.1-Troubleshooting PDF
Pulse Secure-Uac-5.1-Troubleshooting PDF
Published: 2015-02-10
Pulse Secure and the Pulse Secure logo are trademarks of Pulse Secure, LLC in the United States. All other trademarks, service marks, registered
trademarks, or registered service marks are the property of their respective owners.
Pulse Secure, LLC assumes no responsibility for any inaccuracies in this document. Pulse Secure, LLC reserves the right to change, modify, transfer, or
otherwise revise this publication without notice.
Pulse Policy Secure Monitoring and Troubleshooting
The information in this document is current as of the date on the title page.
The Pulse Secure product that is the subject of this technical documentation consists of (or is intended for use with) Pulse Secure software. Use of such
software is subject to the terms and conditions of the End User License Agreement (“EULA”) posted at http://www.pulsesecure.net/support/eula. By
downloading, installing or using such software, you agree to the terms and conditions of that EULA.
Part 1 Overview
Chapter 1 Logging and Monitoring ............................................................................................ 3
Logging and Monitoring Overview....................................................................................... 3
Log File Overview .......................................................................................................... 3
Log File Severity Levels .............................................................................................................4
Custom Filter Log Files ....................................................................................................... 4
Dynamic Log Filters ............................................................................................................. 5
Chapter 2 System Status................................................................................................................ 7
Using the System Status Page ........................................................................................................7
Viewing System Capacity Utilization .................................................................... 7
Specifying Time Range and Data to Display in Graphs ........................................... 8
Configuring Graph Appearance ....................................................................................... 8
Downloading the Current Service Package ................................................................ 9
Chapter 3 Pulse Policy Secure Series Devices .................................................................................... 11
Troubleshooting Overview ............................................................................................................ 11
Chapter 4 TCP Dump Files and Filter Expressions ............................................................... 13
Using TCP Dump Files ...................................................................................................................... 13
Using TCPDump Filter Expressions ........................................................................................ 13
Chapter 5 Network Connectivity Tools................................................................................................ 15
Using Network Connectivity Tools .................................................................................................... 15
Using the Kerberos Debugging Tool ................................................................................................ 16
Chapter 6 RADIUS Diagnostic Log .............................................................................................. 17
About the RADIUS Diagnostic Log ............................................................................................... 17
Part 2 Configuration
Chapter 7 Logging and Monitoring .......................................................................................... 21
Configuring Log Monitoring Features ......................................................................................... 21
Configuring Events, User Access, Admin Access, and Sensor Logs........................ 21
Specifying Which Events to Save in the Log File....................................................... 22
Specifying Which Events to Save in the Log File....................................................... 22
Creating, Editing, or Deleting Log Filters ...................................................................... 23
Creating Custom Filters and Formats for Your Log Files .........................................24
Chapter 8 Clustering .................................................................................................................27
Configuring Cluster Group Communication Monitoring ............................................... 27
Configuring Cluster Network Connectivity Monitoring ..................................................28
Part 3 Administration
Chapter 9 User Sessions.......................................................................................................... 31
Viewing and Deleting User Sessions .....................................................................................31
Chapter 10 SNMP Agent ................................................................................................................. 33
Configuring the SNMP Agent ..................................................................................................33
Chapter 11 System Statistics........................................................................................................... 41
Viewing System Statistics....................................................................................................... 41
Displaying Hardware Health Status ...................................................................................... 41
Chapter 12 Active Users ........................................................................................................... 45
Monitoring Active Users ........................................................................................................... 45
Chapter 13 Clusters .......................................................................................................................... 47
Monitoring Cluster Nodes........................................................................................................ 47
Monitoring Cluster Group Communication .................................................................. 48
Monitoring Cluster Network Connectivity ....................................................................... 48
Part 4 Troubleshooting
Chapter 14 Events ............................................................................................................................. 53
Tracking Events ..........................................................................................................................53
Tracking Events Using Policy Tracing ......................................................................................... 53
Chapter 15 Snapshots .....................................................................................................................55
Creating Snapshots of the Pulse Policy Secure Series Device System State............ 55
Creating Snaphots .................................................................................................................... 56
Chapter 16 Dump Files ................................................................................................................... 59
Creating TCP Dump Files ........................................................................................................ 59
Chapter 17 Network Connectivity Tools................................................................................................... 61
Using UNIX Commands to Test Network Connectivity.................................................... 61
Chapter 18 Debugging Tools and Logs ...................................................................................... 63
Running Debugging Tools Remotely .................................................................................... 63
Creating Debugging Logs ........................................................................................................ 63
Part 5 Index
Index...............................................................................................................................69
Part 1 Overview
Chapter 4 TCP Dump Files and Filter Expressions ............................................................... 13
Table 3: Examples of TCPDump Filter Expressions...........................................................13
Chapter 5 Network Connectivity Tools................................................................................................ 15
Table 4: Useful UNIX Commands ...................................................................................... 15
Part 3 Administration
Chapter 10 SNMP Agent ................................................................................................................. 33
Table 5: Configuration Objects ...............................................................................................35
Chapter 11 System Statistics........................................................................................................... 41
Table 6: Hardware Status Information ............................................................................... 41
Table 7: RAID and Hard Drive Status for Pulse Policy Secure Series Devices ................ 42
Table 8: RAID and Hard Drive Status for the MAG-SM360 .......................................... 42
To obtain the latest version of all Pulse Secure technical documentation, see the
product documentation page at http://www.pulsesecure.net/support
Supported Platforms
For the features described in this document, the following platforms are supported:
IC4500
IC6500 FIPS
IC6500
MAG Series
Documentation Conventions
Table 2 on page x defines the text and syntax conventions used in this guide.
Bold text like this Represents text that you type. To enter configuration mode, type
theconfigure command:
user@host> configure
Fixed-width text like this Represents output that appears on the user@host> show chassis alarms
terminal screen.
No alarms currently active
Italic text like this Introduces or emphasizes important A policy term is a named structure
new terms. that defines match conditions and
Identifies book names. actions.
Junos OS System Basics Configuration
Identifies RFC and Internet draft titles.
Guide
RFC 1997, BGP Communities Attribute
Italic text like this Represents variables (options for which Configure the machine’s domain name:
you substitute a value) in commands or
configuration statements. [edit]
root@# set system domain-name
domain-name
Text like this Represents names of configuration To configure a stub area, include the
statements, commands, files, and stub statement at the[edit protocols
directories; configuration hierarchy levels; ospf area area-id] hierarchy level.
or labels on routing platform The console port is labeled CONSOLE.
components.
< > (angle brackets) Enclose optional keywords or variables. stub <default-metric metric>;
# (pound sign) Indicates a comment specified on the rsvp { # Required for dynamic MPLS only
same line as the configuration statement
to which it applies.
[ ] (square brackets) Enclose a variable for which you can community name members [
substitute one or more values. community-ids ]
}
}
}
> (bold right angle bracket) Separates levels in a hierarchy of J-Web In the configuration editor hierarchy,
selections. select Protocols>Ospf.
Documentation Feedback
Technical product support is available through the Pulse Secure Global Support Center (PSGSC).
If you have a support contract, then file a ticket with PSGSC.
For quick and easy problem resolution, Pulse Secure, LLC has designed an online self-service
portal called the Customer Support Center (CSC) that provides you with the following features:
To verify service entitlement by product serial number, use our Serial Number Entitlement
(SNE) Tool: http://www.pulsesecure.net/support
Overview
Logging and Monitoring on page 3
System Status on page 7
Pulse Policy Secure Series Devices on page 11
TCP Dump Files and Filter Expressions on page 13
Network Connectivity Tools on page 15
RADIUS Diagnostic Log on page 17
This topic provides an overview of log files and log features. It includes the following
information:
Pulse Policy Secure Series device log files are text files stored on an Pulse Policy
Secure Series device appliance that track system events. An Pulse Policy Secure
Series device appliance produces the following types of log files:
User Access log—Contains information about when users access the appliance,
including the number of simultaneous users at each one hour interval (logged on the
hour) and user sign-ins and sign-outs.
The Log and Monitoring pages let you specify which events are logged, the maximum
file size for the system log, and whether to log events to the syslog server in addition
to logging them locally. The Log and Monitoring pages also let you view the specified
number of events, save the log files to a network, and clear the logs.
When one of the logs reaches the configured maximum log file size (200 MB is the
default), the current data is rolled over to a backup log file. A new, empty, file is then
created for all subsequent (new) log messages. Using the log viewer, the administrator
can see the most recent 5000 log messages (the display limit). If the current log file
contains fewer than 5000 log messages, older log messages from the backup log file
are displayed, up to a total of 5000 log messages. This makes the log files appear as
one, even though they are stored separately.
When you save the log messages or use the FTP archive function, the backup log file
is appended to the current log file, and is then downloaded as one log file. If the log files
are not archived or saved by the time they are rolled over again, the oldest log
messages (saved in the backup log file) are lost.
Additionally, you can use a network management tool such as HP OpenView to monitor
an Pulse Policy Secure Series device appliance as an SNMP agent. The Pulse Policy
Secure Series device platform supports SNMPv2, implements a private MIB, and
defines its own traps. To enable the network management station to process these
traps, you need to download the Pulse Secure MIB file and specify the appropriate
information to receive the traps. You can configure some of the traps to suit your
needs.
To monitor vital system statistics, such as CPU utilization, load the UC-Davis MIB file
into your SNMP manager application. You can obtain the MIB file from
http://net-snmp.sourceforge.net/docs/mibs/UCD-SNMP-MIB.txt.
Critical (severity level 10)—When the Pulse Policy Secure Series device cannot
serve user and administrator requests or loses functionality to a majority of
subsystems, it writes a critical event to the log.
Major (severity levels 8-9)—When the Pulse Policy Secure Series device loses
functionality in one or more subsystems but users can still access the appliance
for other access mechanisms, the Pulse Policy Secure Series device writes a
major event to the log.
Minor (severity levels 5-7)—When the Pulse Policy Secure Series device
encounters an error that does not correspond to a major failure in a subsystem, it
writes a minor event to the log. Minor events generally correspond to individual
request failures.
Info (severity levels 1-4)—When the Pulse Policy Secure Series device displays a
notification message, when a user makes a request, or when an administrator
makes a modification, the Pulse Policy Secure Series device writes an informational
event to the log.
When you filter log files, the Pulse Policy Secure Series device saves only the
messages that are specified within the filter query. For example, you might create a
query that logs entries for a particular range of IP addresses or for users who are
signed in to a specific realm. To create a query, use the custom expression language.
When you format log files, the Pulse Policy Secure Series device simply changes the
“look” of the log messages based on your specifications. Log formats do not affect
which data the device saves. Formats affect only how the appliance displays the data.
An Pulse Policy Secure Series device includes standard, WELF, W3C log formats, but
you can also create your own custom format using log fields.
There is also a RADIUS Accounting Filter. This filter allows only the accounting log
message, and it puts the entire message in a comma separated list. The order of the
filtered message is: Date, Time, User, Realm, "List of Roles", NAS-ID, Acct-Status,
Auth-Type, Attr-Value1, Attr-Value2, Attr-Value3.
As with custom log filters, dynamic log filters change only the current view of the log —
not the data that the Pulse Policy Secure Series device saves. Although quick filters act
as temporary filter agents, the Pulse Policy Secure Series device gives you the option
of saving the temporary query strings as new custom filters.
System Status
When you sign in to the admin console, the Pulse Policy Secure Series device displays
the System > Status page, showing the Overview tab. This tab summarizes details
about the Pulse Policy Secure Series device server and system users. When you make
changes on other admin console pages, the Pulse Policy Secure Series device updates
corresponding information on the Overview tab. This tab is the home page for all
administrators, including delegated administrators without read or write access to the
System > Status tabs.
This topic describes how to use and customize the system status page. It includes the
following information:
To use this information for data reporting elsewhere, export it as an XML file using
options on the Maintenance > Import/Export > Configuration page.
These graphs are displayed in the System > Status > Overview tab when you open the
admin console, and allow you to easily view:
Concurrent Users—This graph shows the number of users signed into the Pulse
Policy Secure Series device. In clustered environments, the graph includes two lines.
The first line displays the number of local users signed into the node selected from
the list and the second line displays the number concurrent users signed into the
entire cluster.
Hits Per Second—This graph shows the number of hits currently being processed by
the Pulse Policy Secure Series device. In a clustered environment, you may select an
Pulse Policy Secure Series device from the list to determine which node’s data is
displayed in the graph. The graph includes four lines: number of hits, number of Web
hits, number of file hits, and number of client/server hits.
CPU and Virtual (Swap) Memory Utilization—This graph shows the percentage of the
CPU and available memory currently being used. In a clustered environment, you may
select an Pulse Policy Secure Series device from the list to determine which node’s
data is displayed in the graph.
Throughput—This graph shows the amount of data (in KB) currently being processed.
In a clustered environment, you may select an Pulse Policy Secure Series device from
the list to determine which node’s data is displayed in the graph. The graph includes
four lines: external in, external out, internal in, and internal out.
You may also use the Page Settings window to configure which graphs the Pulse Policy
Secure Series device displays in the dashboard and the period of time that the Pulse
Policy Secure Series device tracks.
2. Click the Download link that corresponds to the graph that you want to download.
3. Click Save, specify the directory where you want to save the XML file, and click Save.
4. Select the range of time that you want to plot in the graphs. Graphing intervals
range from 1 hour to 1 year.
2. Click the Edit link that corresponds to the graph that you want to modify.
3. Use settings in the Graph Settings dialog box to edit the background color, graph line
colors, text color, line color, and line width displayed in the graph.
4. Click Save Changes.
The dashboard for the Pulse Policy Secure Series device allows you to easily view the
last 10 critical system events. Using the Event Monitor window, you can quickly access
and address any critical system problems. Once you have opened the Event Monitor
window, you may keep it open and continually monitor system events while navigating
through the admin console to perform standard maintenance and configuration tasks.
2. Click Critical Events. The Event Monitor window displays the severity and message
of any critical events recorded in the system’s log file.
4. (Optional) Click See All to navigate to the System > Log/Monitoring > Events >
Log tab, where all events—ranging from informational to critical—are displayed.
3. Click Save.
5. Click Save.
Troubleshooting Overview
The Pulse Policy Secure Series device provides several troubleshooting utilities that
enable you to monitor the state of your system, including clusters. Follow the Related
Documentation links for an overview of the troubleshooting tasks you can perform with
the Pulse Policy Secure Series device.
The Maintenance > Troubleshooting > Tools > TCP Dump tab allows you to sniff
network packet headers and save the results in an encrypted dump file that you can
download to a network machine and then e-mail to Pulse Secure Support.
This feature uses the TCP/IP network stack to capture packets at the TCP layer. It
captures all communication that passes through the Pulse Policy Secure Series device.
However, certain encrypted higher level protocols cannot be decrypted. This feature is
useful for troubleshooting common customer problems. A TCP dump file helps the Pulse
Secure Support team observe the communication protocols used between Pulse Policy
Secure Series device and any other intranet server and how the intranet server
responded to requests from the Pulse Policy Secure Series device.
On the admin console, you can select which interface you want to capture packets from,
whether internal or external, you can select promiscuous mode, which increases the level
of detail in the dump file, and you can specify a filter.
tcp port 80 or port 443 and dst #.#.#.# and This example shows how to specify multiple
src #.#.#.# parameters to create a filter that sniffs on TCP port
80, or on TCP or UDP port 443, and on the
destination and source ports, where each #.#.#.#
represents a valid IP address.
The Commands tab allows you to run UNIX commands, such as arp, ping, traceroute,
and NSlookup, to test Pulse Policy Secure Series device network connectivity. You can
use these connectivity tools to see the network path from the Pulse Policy Secure
Series device to a specified server. Table 4 on page 15 describes these commands.
arp Use the arp command to map IP network addresses to the hardware addresses. The Address Resolution
Protocol (ARP) allows you to resolve hardware addresses.
To resolve the address of a server in the network, a client process on the Pulse Policy Secure Series
device sends information about its unique identity to a server process executed on a server in the intranet.
The server process then returns the required address to the client process.
traceroute Use the traceroute command to discover the path that a packet takes from the Pulse Policy Secure
Series device to another host. Traceroute sends a packet to a destination server and receives an ICMP
TIME_EXCEEDED response from each gateway along its path. The TIME_EXCEEDED responses and
other data are recorded and displayed in the output, showing the round-trip path of the packet.
1. In the admin GUI, select Maintenance > Troubleshooting > Tools > Kerberos.
3. Specify the realm name and the fully qualified domain name for the site in the
respective Kerberos Realm and Site text boxes.
4. Click Run.
The Access Control Service runs the probe and returns results to the Output region of
the page, for example:
The RADIUS Troubleshooting Log allows you to view the full suite of RADIUS logging
features, including traffic trace and debug-level messages. In releases earlier than 2.2,
these logs were available only in an encrypted format that required decryption by Pulse
Secure.
The RADIUS Troubleshooting Log monitors all requests that the Pulse Policy Secure
Series device receives from RADIUS clients. RADIUS requests that the Pulse Policy
Secure Series device initiates do not appear in the log.
Raw traffic is not available in the log. To view raw traffic, use the tcpdump feature.
You can configure the maximum size of the log. When the log fills up, logging stops.
You can clear the log to restart logging.
All events that appear in the log have an ID code, and all messages in a thread are
tagged with the same ID. This allows you to track individual logins or login attempts.
The RADIUS Troubleshooting Log is secure, because passwords are suppressed and
do not appear in the logs.
Performance of the Pulse Policy Secure Series device is affected with RADIUS logging
turned on.
Configuration
Logging and Monitoring on page 21
Clustering on page 27
This topic describes how to configure log monitoring features. Log monitoring features
enable you to monitor events, user access, and administrator access logs, which you
can filter and save for later review. Additionally, the Pulse Policy Secure Series device
allows you to use SNMP to monitor its activities and provides statistics and client-side
logs for applications such as Host Checker and OAC. This topic provides the following
information:
Configuring Events, User Access, Admin Access, and Sensor Logs on page 21
Specifying Which Events to Save in the Log File on page 22
Specifying Which Events to Save in the Log File on page 22
Creating, Editing, or Deleting Log Filters on page 23
Creating Custom Filters and Formats for Your Log Files on page 24
The events, user access, and admin access logs are three distinct files. Although the
basic configuration instructions for each is the same, modifying the settings for one
does not affect settings for another
2. Select Events, User Access, Admin Access, or Sensors, and then select Log.
3. In the View by filter list, select the custom filter that the Pulse Policy Secure
Series device should use to filter data.
4. Enter a number in the Show field and select Update to change the number
of log entries that the Pulse Policy Secure Series device displays at one
time.
5. Click Save Log As, navigate to the desired network location, enter a file
name, and then select Save to manually save the log file.
To save all log files, click Save All Logs. The Pulse Policy Secure Series device
prompts you for a location where it saves the log files in one compressed file. You can
access the Save All Logs button from any one of the three log tabs.
6. Click Clear Log to clear the local log and log.old file.
When you clear the local log, events recorded by the syslog server are not affected.
Subsequent events are recorded in a new local log file.
2. Select the Events, User Access, Admin Access, or Sensors tab, and then select
Log.
3. Click on any data log variable link in the current log. The log immediately
redraws based on the chosen variable.
4. (Optional) Continue adding variables in the same manner. Each data log variable
link you select adds an additional variable to the Edit Query text field and the log
updates with each added variable.
5. (Optional) Click the Reset Query button to clear the Edit Query text field and
reset the log to the view determined by the filter specified in the View by filter
field.
6. (Optional) Click the Save Query button to save the dynamic log query as a
custom filter. The Filters tab displays with the Query field prepopulated with the
variables you selected from the log. Next:
b. (Optional) the new filter the default filter by selecting Make default.
In the Start Date section, click Earliest Date to write all logs from the first
available date stored in the log file. Or, manually enter a start date
In the End Date section, click Latest Date to write all logs up to the last
available date stored in the log file. Or, manually enter a end date.
You may also use the Archiving page to automatically save the logs to an FTP accessible
location.
2. Select Events, User Access, Admin Access, or Sensors tab, and then select
Settings.
3. In the Maximum Log Size field, specify the maximum file size for the local log file. (The
limit is 500 MB.) The system log displays data up to the amount specified.
Maximum Log Size is an internal setting that most closely corresponds with the size
of logs formatted with the Standard format. If you use a more verbose format such as
WELF, your log files may exceed the limit that you specify here.
4. Under Select Events to Log, select the check box for each type of event that
you want to capture in the local log file.
If you disable the Statistics check box in the Events Log tab, the Pulse Policy Secure
Series device does not write statistics to the log file, but continues to display them in
the System > Log/Monitoring > Statistics tab.
5. (Optional) Under Syslog Servers, enter information about the syslog servers
where you want to store your log files:
b. Enter a facility for the server. The Pulse Policy Secure Series device
provides 8 facilities (LOCAL0-LOCAL7) which you can map to facilities
on your syslog server.
d. Click Add.
e. Repeat for multiple servers if desired, using different formats and filters for
different servers and facilities.
Make sure your syslog server accepts messages with the following settings: facility
= LOG_USER and level = LOG_INFO.
Standard (default)—This log filter format logs the date, time, node, source IP address,
user, realm, and the Pulse Policy Secure Series device event ID and message.
W3C—The World Wide Web Consortium’s extended log file format is a customizable
ASCII format with a variety of different fields. Visit http://www.w3.org for more
information about this format. Only the User Access log offers this filter as an option.
2. Select the Events, User Access, Admin Access, or Sensors tab, and then select Filters.
If you select a format and then create a new name for it in the Filter Name field, the
Pulse Policy Secure Series device does not create a new custom filter format that is
based on the existing format. Instead, it overwrites the existing format with the
changes you make.
5. Click Make Default to define the selected filter as the default for the log file type. You
may set different default filters for the events, user access, and administrator access
logs.
6. Use options in the Query section to control which subset of data the Pulse Policy
Secure Series device writes to the log:
a. In the Start Date section, click Earliest Date to write all logs from the first available
date stored in the log file. Or, manually enter a start date.
b. In the End Date section, click Latest Date to write all logs up to the last available
date stored in the log file. Or, manually enter an end date.
c. In the Query section, use the Pulse Policy Secure Series device custom expression
language to control which subset of data the IC Series device writes to the log.
Any string (including a * wildcard character) you manually enter in a query, must
be enclosed in double-quotes. For example, the query protocol="UDP" AND
sourceip=172.27.0.0/16 AND port=* must be presented as protocol="UDP" AND
sourceip=172.27.0.0/16 AND port=”*” or the logging component returns an error.
7. Use one of the options the Export Format section to control the format of the data in
the log:
Select the Standard, WELF, or W3C option to format the log entries using one of
these standardized formats.
Select the Custom option and enter the format you want to use in the Format field.
When entering a format, surround variables with percentage symbols (for example
%user%). All other characters in the field are treated as literals.
Clustering
3. If you want to monitor all cluster nodes from the current local node, select the
Monitor all cluster nodes from this node check box. If you do not select this
option, the group communication monitor gathers statistics only for the local node.
NOTE: If you select the Monitor all cluster nodes from this node option,
the cluster nodes must be able to communicate over UDP port 6543.
4. Select the Enable group communication monitoring check box to start the
monitoring tool.
1. Select the Enable cluster network troubleshooting server check box to enable the
server component.
6. Click the Details link below the fields to view the results.
Administration
User Sessions on page 31
SNMP Agent on page 33
System Statistics on page 41
Active Users on page 45
Clusters on page 47
User Sessions
The configuration page for most IC Series device authentication servers contain a User’s
tab that you can use to view and delete active IC Series device user sessions.
Authentication server types that do not display this tab include:
Anonymous server—The IC Series device cannot display individual session data about
users who sign in through an anonymous server, because it does not collect usernames
or other credentials for such users.
For all other types of authentication servers, you can view and delete active user
sessions by using these instructions:
Enter a username in the Show users named field and click Update to search for a
specific user.
Alternately, you can use an * character as a wildcard, where an * represents
any number of zero or more characters. For example, if you want to search for
all usernames that contain the letters jo, enter *jo* in the Show users named
field. The search is case-sensitive. To display the entire list of accounts again,
either enter an * character, or delete the field’s contents and click Update.
Enter a number in the Show N users field and click Update to control the
You can find several access statistics for any user account on the Users tab in the Last
Access Statistics columns. These columns appear on any of the Users tabs anywhere
they appear in the admin console. The statistics include the last sign-in date and time a
user successfully signed in as well as the browser type and version.
SNMP Agent
You can use a network management tool such as HP OpenView to monitor the Access
Control Service system as an SNMP agent. The system supports SNMP (Simple Network
Management Protocol) v2 and SNMPv3, implements a private MIB (management
information base), and defines its own traps. To enable your network management
station to process these traps, you need to download the Pulse Secure MIB file and
specify the appropriate information to receive the traps.
To monitor vital system statistics, such as CPU utilization, load the UC-Davis MIB file into
your SNMP manager application. You can obtain the MIB file from:
http://net-snmp.sourceforge.net/docs/mibs/UCDSNMP- MIB.txt. The system supports
standard MIB objects, including the system uptime (sysUpTime) object. The system
uptime (sysUpTime) object returns the time elapsed (in hundredths of a second) since the
SNMP agent was started.
The User-Based Security Model (USM) is the default Security Module for SNMPv3.
Access Control Service supports only one user at a time to be registered with an SNMP
engine. Editing the SNMPv3 user attributes overwrite any already registered SNMPv3
user. The SNMPv3 user must have read-only access on all MIBs supported by the
Access Control Service system. SNMPv3 user configuration attributes can also be used
for SNMP traps.
2. Click the Pulse Secure MIB file link to access the MIB file, and then save the file
from your browser to a network location. For descriptions of the Get and Trap
objects in the MIB file.
Enter information in the System Name, System Location, and System Contact
fields that describes the SNMP agent (optional).
Enter the community string (required only for SNMPv2c).
To query the system, your network management station must send it the
community string.
Select the desired security level and enter the following information based on
the level selected.
6. Under Trap Thresholds, set the values for the following traps (optional).
Setting a threshold value to 0 disables that respective trap.
Check Frequency
Log Capacity
Users
Memory
Swap Memory
Disk
Meeting Users
CPU
8. Under SNMP Servers, specify servers to which you want the system to send the
traps that it generates by entering information in the following fields, and then
clicking Add:
The community string required by the network management station (if applicable).
The community string is applicable only for SNMPv2c.
9. Click Save Changes. If the version is changed from v2 to v3, the system
generates two engine IDs. Both IDs are displayed.
b. Specify the community string required when querying the system (see step 4). The
community string is applicable only for SNMPv2c.
e. The agent engine ID for SNMPv3 queries and trap engine ID must be configured to
receive the SNMPv3 traps for the device, as outlined in Step 9.
logFullPercent Returns the percentage of the available file size filled by the current log
as a parameter of the logNearlyFull trap.
iveMemoryUtil Returns the percentage of memory utilized by the system at the time of
an SNMP poll. The system calculates this value by dividing the number
of used memory pages by the number of available memory pages.
clusterConcurrentUsers Returns the total number of users logged in for the cluster.
iveFileHits Returns the total number of file hits to the system since last
reboot.Incremented by the web server with each GET/POST
corresponding to a file browser request.
iveAppletHits Returns the total number of applet hits to the system since last
reboot.Incremented by the web server for each GET request for a Java
applet.
logName Returns the name of the log (admin/user/event) for the logNearlyFull
and iveLogFull traps.
diskFullPercent Returns the percentage of disk space used in the system for the
iveDiskNearlyFull trap. The system calculates this value by dividing the
number of used disk space blocks by the number of total disk space
blocks.
logID Returns the unique ID of the log message sent by the logMessageTrap
trap.
logDescription Returns a string sent by the logMessageTrap trap stating whether a log
message is major or critical.
iveLogNearlyFull The log file (system, user access, or administrator access) specified by
the logName parameter is nearly full. When this trap is sent, the
logFullPercent (%of log file full) parameter is also sent. You can
configure this trap to be sent at any percentage. To disable this trap, set
the Log Capacity trap threshold to 0%. The trap’s default value is 90%.
iveMaxConcurrentUsersSignedIn Maximum number or allowed concurrent users are currently signed in.
You can configure this trap to be sent at any percentage. To disable this
trap, set the Users trap threshold to 0%. The trap’s default value is
100%.
When the system sends this trap, it also sends the authServerName
(%of log file full) (name of unreachable server) parameter.
archiveServerUnreachable The system is unable to reach configured FTP or SCP Archive server.
iveDiskNearlyFull Supplies notification that the system disk drive is nearly full. When the
system sends this trap, it also sends the diskFullPercent parameter. You
can configure this trap to be sent at any percentage. To disable this trap,
set the Disk trap threshold to 0%. This trap’s default value is 80%.
logMessageTrap The trap generated from a log message. When the system sends this
trap, it also sends the logID, logType, and logDescription parameters.
cpuUtilNotify Supplies notification that the system has met the configured threshold
for CPU utilization. To disable this trap, set the CPU trap threshold to 0.
The threshold is 0%, by default.
iveFanNotify Supplied notification that the status of the fans has changed.
iveRaidNotify Supplies notification that the status of the RAID device has changed.
iveNetInternalInterfaceDownTrap (nicEvent) Supplies the type of event that brought down the internal interface. The
nicEvent parameter can contain values of “external” for an external
event and “admin” for an administrative action.
iveClusterChangedVIPTrap(vipType, currentVIP, Supplies the status of a virtual IP for the cluster. The vipType indicates
newVIP) whether the changed VIP was external or internal. The currentVIP
contains the VIP prior to the change, and newVIP contains the VIP after
the change.
iveClusterDelete(nodeName) Supplies the name of the node on which the cluster delete event was
initiated.
The options for sending SNMP traps for critical and major events are set to OFF by default,
for security purposes.
System Statistics
Every hour, the IC Series device logs the peak load of Web users.
The Statistics page displays that information for the past seven days. The IC Series device
writes that information to the system log once a week. Note that upgrading the IC Series
device clears all statistics. If you configure the system to log statistics hourly, however, old
statistics are still available in the log file after an upgrade.
You can use the Maintenance > System > Platform page to display the hardware health
status, including information about hard drives, fans, and power supplies.
Hard Disk Status Displays a health statement for the device disk drive. See Table 7 on page 42 and
Table 8 on page 42 for details.
Power Supply Displays a health statement for the device power supply.
Table 7 on page 42 lists the RAID status and hard drive status for IC Series devices.
Depending on your system, you may or may not see all these possible statuses.
Table 8 on page 42 lists all the possible RAID status and hard drive status for the
MAG-SM360. You can also view the RAID and hard drive status in log messages and in
SNMP.
42 © 2015 by Pulse Secure, LLC. All rights reserved
Table 8: RAID and Hard Drive Status for the MAG-SM360
Table 8: RAID and Hard Drive Status for the MAG-SM360 (continued)
Drive 1
Offline
Degraded Optimal
Active Users
You can monitor users signed in to the IC Series device. Each user’s name,
authentication realm, role, Host Checker status, and sign-in time are listed on the Active
Users page.
If a user signs into the IC Series device and then the user’s computer is placed on a VLAN
without an IP address, the IC Series device does not display an IP address under Signed
in IP for the user’s status on the Active Users page.
If there is a NAT device between the user’s computer and the Infranet Enforcer, the IC
Series device displays both the NAT device’s IP address and the endpoint's virtual source
IP address under Signed in IP for the user’s status on the Active Users page. For
example, if the NAT device’s IP address is 10.64.9.26, and the endpoint’s virtual source
IP address is 192.168.80.128, the following information is displayed under Signed in IP:
Additionally, there is a column for Host Checker status. Endpoint Security Status entries are
updated based on a users policy status.
Next to the security policy status is a‘ ’ button. Clicking this button dynamically lists the
passed and failed policies. A hyperlink that redirects to the policy configuration page is
displayed on policy names. A list of any roles that have been eliminated because of a failure
in security policies is also displayed. Also next to the security policy status is a hyperlink
named Logs. This link redirects to the User Access Logs page, which displays the log
messages for the corresponding session by automatically using the proper filters.
1. In the admin console, select System > Status > Active Users.
To forcibly sign out all end-users who are currently signed-in, click Delete
All Sessions.
To forcibly sign out all end-users who are currently signed-in and also
prevent any other users from signing in, click Disable All Users. To allow
users to sign in again after you disable all users, click Enable All Users.
If you want to sign out administrators, you must choose them individually and
use the Delete Session button.
To display a specific user, enter the username in the Show Users Named
field and click Update. If you do not know the user’s exact username, use the
* wildcard character. For example, if you have a usernamed “Joseph Jones,”
but you do not remember if the username is “Joe” or “Joseph,” enter Jo* in
the Show Users Named field. The IC Series device returns a list of all users
whose usernames start with the letters jo.
To control how many users and administrators are displayed in the Active
Users page, enter a number in the Show N users field and click Update.
To edit a user’s authentication realm, click the Realm link next to the name.
To edit a user’s role, click the Role link next to the name.
Clusters
If you have a problem with a cluster, a Pulse Secure Support representative may ask you
to create a snapshot that includes node monitoring statistics to assist with debugging the
cluster problem. When you enable the node monitor on the Node Monitor tab, the IC
Series device captures certain statistics specific to the cluster nodes on your system.
Using the resulting snapshot, the support team can identify important data, such as
network statistics and CPU usage statistics.
1. Select Maintenance > Troubleshooting > Monitoring > Node Monitor to enable the
node monitor.
3. Enter the interval, (in seconds) at which node statistics are to be captured.
4. Select the Node monitoring enabled check box to start monitoring cluster nodes.
5. For Maximum node monitor log size, enter the maximum size (in MB) of the log
file. Valid values in the range of 1 - 30.
6. Specify the interval (in seconds) that defines how often nodes are to be monitored.
If you have a problem with a cluster, a Pulse Secure Support representative might ask
you to create a snapshot that includes group communication statistics to assist with
debugging the cluster problem. When you enable the group communication monitor in the
Group Communication tab, the IC Series device records statistics related to all of the
cluster nodes on your system. As the local node communicates with other nodes in the
cluster, the IC Series device captures statistics related to intra cluster communication.
The Group Communication tab is displayed only when you enable clustering on your
system. On a standalone IC Series device, you do not have access to the Group
Communication tab.
You can also enable the cluster networking troubleshooting server on the Network
Connectivity page.
NOTE:
Performing excessive node monitoring can impact system performance
and stability. You should only perform extensive monitoring when directed
by your Pulse Secure Support representative.
If you have a problem with a cluster, a Pulse Secure Support representative might ask
you to enable the cluster node troubleshooting server. When you enable the server on the
Network Connectivity tab, the IC Series device attempts to establish connectivity between
the node on which the server resides and another node you specify. As the nodes
communicate, the IC Series device displays network connectivity statistics on the page.
The Network Connectivity tab is displayed only when you enable clustering on your
system. On a standalone IC Series device, you do not have access to the Network
Connectivity tab.
Use the Network Connectivity tab to enable the cluster node troubleshooting server and
to select a node on which to perform troubleshooting tasks. The troubleshooting tool
allows you to determine the network connectivity between cluster nodes.
The server component of this tool runs on the node to which connectivity is being tested.
The client component runs on the node from which connectivity is being tested. The basic
scenario for testing connectivity is this:
The administrator tests the connectivity to the server node from the Active node, by
starting the client component on the active node and then contacting the passive
node running the server component.
NOTE: The server component must be run on nodes that are configured as
either standalone or in a cluster but disabled. Cluster services cannot be
running on the same node as the server component.
Troubleshooting
Events on page 53
Snapshots on page 55
Dump Files on page 59
Network Connectivity Tools on page 61
Debugging Tools and Logs on page 63
RADIUS Diagnostic Logs on page 65
Events
Tracking Events
You can determine why your IC Series device does not allow you to accomplish a task by
tracking problematic IC Series device events using the Policy Tracing page accessible from
the admin console, guides you through all the realms, roles, and policies that are currently
configured in the IC Series device and lets you print log messages at various steps of the
authentication, authorization, and access process.
The events in question are related to authentication, authorization, and access for a
particular user. They are driven entirely by what happens during a user session.
These events do not include any other system related events. The IC Series device merely
uses the events as a filtering mechanism to reduce the number of logs and to highlight the
problem.
The IC Series device allows you to troubleshoot problems by tracking events when a user
signs into a realm. The Policy Tracing page allows you to record a policy trace file for an
individual user. The IC Series device displays log entries that list the user’s actions and
indicates why that user is allowed or denied access to various functions.
NOTE: User access logs are only reported for policies that are checked under
Events to Log.
Use this tab if your users are having problems accessing functions they expect to use in
their roles. The events logged in the policy trace file might help you diagnose these
problems.
1. In the admin console, select Maintenance > Troubleshooting > User Sessions
> Policy Tracing.
2. In the User field, enter the IC Series device username of the user you want to
trace. You can use a wildcard character (*) in place of a username. For
example, if your users are signing into an anonymous server, you can use the
wildcard character (*) because you cannot know the internal username that the
IC Series device assigns to the user.
3. In the Realm field, select the user’s realm. (The IC Series device does not
allow you to select a realm that maps to an anonymous authentication server.)
4. Under Events to log, select the types of events you want to write to the policy
tracing log file.
5. Click Start Recording. Ask the user to sign in to the IC Series device after
you start recording.
8. Review messages in the log file to identify what is causing the unexpected
behavior. If you cannot determine and fix the problem, click Save Log As to save
a copy of the log file to a location on the network. Then, send the file to Pulse
Secure Support for review.
9. Click Clear Log to clear the contents of the log file, or click Delete Trace to
clear the contents of the log file and to remove the default entries from the
username and realm fields.
Snapshots
The System Snapshot tab allows you to create a snapshot of the IC Series device system
state. When you use this option, the IC Series device runs various utilities to gather
details on the IC Series device system state, such as the amount of memory in use,
paging performance, the number of processes running, system uptime, the number of
open file descriptors, and the ports in use.
You can include or exclude system configuration and debug logs. However, debug logs
are particularly important in the event of a problem. You must set the debug log at a
certain level and add the events list as directed by your Support representative. Recreate
the problem or event and then take a snapshot and send it to Pulse Secure Support. The
debug log is encrypted, so you cannot view it.
When a RADIUS process on an IC Series device stops processing incoming
authentication requests for a time greater than 2 minutes, the RADIUS process aborts
and generates both a process snapshot and a system snapshot.
NOTE:
The IC Series device stores up to ten snapshots, which are packaged in an
encrypted “dump” file that you can download to a network machine and then
e-mail to Pulse Secure Support. If you take more than ten snapshots, the IC
Series device overwrites the oldest snapshot file with the new snapshot. If
the IC Series device runs out of disk space, the IC Series device does not
store the newest snapshot and logs a message in the Event log. Although
the IC Series device compresses the files first and then performs the
encryption to minimize file size, we recommend that you download the
snapshots to a network machine in a timely manner to avoid losing them.
Creating Snaphots
1. In the admin console, select Maintenance > Troubleshooting > System Snapshot.
2. (Optional) Select the Include system config check box to include system configuration
information in your snapshot.
3. Select the Include debug log check box to include log file created through the Debug
Log tab in your system snapshot.
NOTE: If the size of the snapshot exceeds the maximum file size you
specify, the snapshot fails and the IC Series device logs a message in the
Event log. The IC Series device compresses the files first and then
encryps them to minimize file size.
e. If you want to disable debug logs at the stop time you specified, select
Disable debug logs at stop time.
7. When the IC Series device finishes taking the snapshot, click the link for the
snapshot listed under Snapshot, click Save, navigate to the folder where you want
to store the snapshot file, and then click Save.
9. When you are finished, select the snapshot listed under Snapshot and then click
Delete to delete the snapshot.
NOTE: See Performing Common Recovery Tasks with the Serial Console.
This method is useful if you cannot get to the admin console and need to save
the system configuration.
Dump Files
1. In the admin console, select Maintenance > Troubleshooting > Tools > TCP Dump.
2. Select the IC Series device port on which you want to sniff network packet headers.
3. Turn off Promiscuous mode to sniff only for packets intended for the IC Series
device.
6. Click Stop Sniffing to stop the sniffing process and create an encrypted file.
1. In the admin console, select Maintenance > Troubleshooting > Tools > Commands.
3. In the Target Server field, enter the IP address of the target server.
The Pulse Secure Support team can run debugging tools on your production IC Series
device if you configure it to do so. To enable this option, you must work with Pulse Secure
Support to obtain a debugging code and host to which your IC Series device connects.
1. Contact Pulse Secure Support to set up the terms of a remote debugging session.
2. In the admin console, select Maintenance > Troubleshooting > Remote Debugging.
5. Click Enable Debugging to allow the Pulse Secure Support team to access the IC
Series device.
7. Click Disable Debugging when Pulse Secure Support notifies you that the remote
debugging session is over.
If you have a problem, a Pulse Secure Support representative might ask you to create
debugging logs to assist with debugging IC Series device internal issues. When you
enable logging, the IC Series device records certain events and messages based on
event codes you enter in the admin console Debug Log tab. Using the debug log that
results, the support team can identify the code flow for any discrepancies. Your
support representative gives you the information you need to create the log file,
including the debug detail log level and the event codes.
The IC4500 and IC6500 displays the current debug log size.
NOTE: Running debug logging can impact system performance and stability.
Generate debug logs only when directed by your Pulse Secure Support
representative.
1. In the admin console, select Maintenance > Troubleshooting > Monitoring > Debug Log.
NOTE: Setting the detail level to 0 displays only critical messages it does
not disable logging completely.
10. Click Take snapshot to create a file that contains the debug log. The IC Series device
compresses the files and then encrypts them to minimize file size.
12. Attach the snapshot file in an e-mail message and send it to Pulse Secure Support.
1. Select Troubleshooting > Monitoring > RADIUS from the left navigation bar of the
admin console.
3. Enter the maximum log size (up to 1,000 MB) in the Max Diagnostic Log Size box.
Index
Index on page 69
L
Index log capacity ....................................................................... 3
log file severity ................................................................... 4
log filters ...........................................................................23
log filters, dynamic ........................................................ 5
Symbols
log monitoring, configuring .................................................. 21
#, comments in configuration statements ...................xi
logging, monitoring overview.............................................. 3
( ), in syntax descriptions .............................................. xi
< >, in syntax descriptions ................................................x
M
[ ], in configuration statements .................................... xi
manuals
{ }, in configuration statements ................................... xi
comments on ............................................................ xi
| (pipe), in syntax descriptions .......................................xi
monitoring users.............................................................45
monitoring, logging overview.............................................. 3
A
administrator access log ..................................................... 3
P
parentheses, in syntax descriptions .............................. xi
B
braces, in configuration statements ................................xi
S
brackets
service package, downloading ...................................... 9
angle, in syntax descriptions .....................................x
sessions, deleting ................................................................. 31
square, in configuration statements ........................xi
severity, log files..................................................................... 4
snmp agent ..........................................................................33
C
statistics, viewing .................................................................. 41
capacity, log........................................................................ 3
support, technical See technical support
capacity, system.....................................................................7
syntax conventions ........................................................... x
comments, in configuration statements ........................xi
system status, viewing ...................................................... 7
conventions
text and syntax ...............................................................x
T
curly braces, in configuration statements ......................xi
technical support
custom filter log files......................................................... 4
contacting PSGSC ............................................................ xiii
customer support .............................................................xi
contacting PSGSC …………………...………….xiii
U
user access log ...................................................................... 3
D
user sessions, deleting ........................................................ 31
deleting user sessions ........................................................ 31
users, monitoring ...........................................................45
documentation
comments on ............................................................xi
dynamic log filters............................................................. 5
E
events log ............................................................................... 3
F
font conventions ................................................................x