Daren Matthews

When capturing log sessions during an outage or debug session it’s crucial to keep a history of what was done when and in what order. This helps in post mortem analysis after the event or session is over. There are a couple of standard things to do to help this such as:

• NTP time sync all devices in the network to get them on a single time source
• Enable syslog for all devices in the network
• Allocate some local log space on each local device based on a percentage of free memory available on the device
• Always have trend graphs from some polling mechanism for CPU levels, memory usage, traffic rates, etc.

In addition to these here are a few other tips that really help TAC analyse log and debug sessions that are collected:

• Annotate the local log and syslog output using the send log command

CE_#send log "Shutting down interface now!"

CE_#config t

Enter configuration commands, one per line.  End with CNTL/Z.

CE_(config)#int e 0/0

CE_(config-if)#shut

CE_(config-if)#end

CE_#sh log

%SYS-7-USERLOG: Message from 0(): "Shutting down interface now!"

00:00:46: %SYS-5-CONFIG_I: Configured from console by console

00:00:47: %LINK-5-CHANGED: Interface Ethernet0/0, changed state to

administratively down

00:00:48: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0,

changed state to down

By using the send log <message> the output is sent to the local log as well as to the syslog so there is a historical view of what was done and why. View it as “Thinking out loud and capturing the thought process” so it can be followed later. Without it it’s very hard to remember exactly what was done when and why.

• Annotate the logging or debug session as things are learned or events happen using the !<message> CLI

CE_#!Troubleshooting RIP routers being lost

CE_#!checking # of routers learned before cable is pulled

CE_#sh ip ro summ

IP routing table name is default (0x0)

IP routing table maximum-paths is 32

Route Source    Networks    Subnets     Replicates  Overhead    Memory

(bytes)

connected       0           2           0           104         344

static          0           0           0           0           0

rip             0           1           0           52          172

eigrp 1         0           0           0           0           0

internal        1                                               528

Total           1           3           0           156         1044

CE_#!Only one RIP route...wait some more time for network to converge

• Capture date, time, and CPU automatically when running CLI commands during a session by enabling terminal exec prompt timestamp

CE_#term exec prompt timestamp

CE_#sh user

Load for five secs: 3%/0%; one minute: 1%; five minutes: 1%

Time source is hardware calendar, *14:02:40.623 EST Mon Oct 20 2008

    Line       User       Host(s)              Idle       Location

*  0 con 0                idle                 00:00:00

  Interface    User               Mode         Idle     Peer Address

CE_#

Notice the Load and Time as the output for each command. It will be printed automatically every time a CLI command is entered.

This is very useful because it allows deltas of counters between outputs to have some rate context by comparing the time difference between outputs.

Without that when looking at a raw capture log it’s impossible to tell how much time elapsed between outputs.

When using the above in conjunction with “|” don’t forget to include the “Load for five|Time source is” to capture it while filtering other output.