Rebuilding the RPM Database

lsof | grep /var/lib/rpm

Remove /var/lib/rpm/__db* files to avoid stale locks:

# cd /var/lib
# rm __db*

Rebuild RPM database:

# rpm --rebuilddb
# rpmdb_verify Packages

Method # 2       

If you are still getting errors, then try your luck with following commands:

# mv Packages Packages-BAKUP
# db_dump Packages-BAKUP | db_load Packages
# rpm -qa
# rpm --rebuilddb

syslog and logrotate

Ø  /etc/syslog.conf

·         The Syslog daemon logs any notable events on your local system. It can store that logs in local file or send them to a remote log host for added security.

·         It can also accept the logs from  other machines when acting as a remote log host

*.info; mail.none; news.none; authpriv.none; crone.none   /var/log/messages

authpriv.*                   /var/log/secure      ---authprivilege

mail.*                          /var/log/maillog       ---mail logs

cron.*                          /var/log/cron             ---cron logs

*.emerg                      *                                   --everybody gets messages

uucp.news.crit          /var/log/spooler      

Ø  logrotate.conf

ü  logrotate.conf and the files within logrotate.d determine how often your log files are rotated by the logrotate program

ü  logrotate can automatically rotate, compress, remove and mail your log files.

ü  Log files can be rotated based on size or on time, such as daily, weekly or monthly

ü  Logrotate.conf

#rotate log files weekly

Weekly

#keep 4 weeks’ worth of backlogs

rotate 4

#create new empty log files after rotating old ones

Create

#your log files compressed

Compress

#RPM packages drop log rotation information into this directory

Include /etc/logrotate.d

#nopackges own lastlog or wtmp

/var/log/wtmp

{

                         monthly

                         create 0664 root wtmp

                         rotate 1

}

Ø  Logwatch searches log files for suspicious activity

·         /usr/share/logwatch/

·         /etc/logwatch/conf/logwatch.conf

·         /etc/logwatch/scripts

Ø  Increasing Loglevel from Info to Debug to aid with troubleshooting

awk

Ø  Syntax

awk options `selection criteria {action}` filename

Ø  Cat data.txt

ü  There are two servers:HP:IBM

Ø  awk –F: `/server/ {printf $5 $ 6}` data.txt

ü  HP:IBM

Ø  awk `/server/{printf $6}` data.txt

ü  IBM

Ø  Comparison operator

ü  awk ‘$2 == “director” || $3 == “chairman” {printf “%-20s %-20s \n”, $2, $3}’ datafile

Ø  awk `$2~/100/` test.txt

Ø  awk `$2 !~ /100/` test.txt

Ø  Number comparison

ü  awk `$2>200` test.txt

ü  awk `$2^200` test.txt  ---shows data which equal or above 200

Ø  awk –F: `$3 == 0 && $1 != “root”` /etc/passwd

Advanced head and tail

Ø  Show first x lines from file

ü  $head –n 10 data.txt

Ø  Show all lines in filename except the last x

ü  $head –n -15 data.txt

Ø  Show last x lines from file

ü  $tail –f –n 10 data.txt

ü  -f option show new data immediately

Ø  Show the entire contents of file

ü  $tail –n +10 data.txt

ü  $tail –f logfile

ü  $ps  -ef

ü  $ps –ef | tail –n+2

Log Analysis

Ø  /var/log/messages   --Read the log file first

Ø  If no information, follow below

#ls –lt  /var/log/         --its shows the last modified with the newest at the top

#ls –lrt /var/log/        --Use –r flag to reverse the ordering, i.e. newest at the bottom

Ø  dmesg

ü  kernel log buffer can be read by using the “dmesg”

ü  the size of this buffer is 128 KB in RHEL5

ü  the oldest log messages are dropped once the buffer fills up

ü  During boot process, the log buffer is saved to /var/log/dmesg near the end of the “rc.sysinit” script

ü  Otherwise, kernel messages will generally be logged to /var/log/messages (by klogd service)

Additional Information Gathering

Ø  Increase logging information

ü  Often when trying to debug problems it’s helpful to be able to increase the verbosity of output or logging

ü  Example, cups printing system, changing the “Loglevel Info” to “Loglevel Debug” in /etc/cups/cupsd.conf

ü  It will vastly increase the information logged under /var/log/cups

Ø  Verbose flags

ü  -v,-vv,-vvv

ü  --help

Best Practice

Ø  Backup files before editing like (filename_todaydate)

Ø  One change at a time

ü  Change the configuration

ü  Test the changes

ü  Solved

ü  Otherwise, undo and repeat

Ø  Comment files while editing

Technique for solving problems

1.       Information gathering

2.       Cross-reference with similar systems

3.       Isolate problem, test!

Ø  Information gathering

ü  Log files, Kernel output, Device output

ü  Use commands such as  grep, uniq, sort, less to finding errors and indentifying the problem

Ø  Cross-reference with similar systems

ü  Comparing logs and output with a similar healthy systems

Ø  Isolate problem, test!

ü  Once we have located the errors we can fix the problem and then test it.

Content

Ø  Unit 1-Intro to Troubleshooting Technique and Procedures

ü  Technique for solving problems

ü  Log Analysis

ü  Additional information gathering

ü  Runlevels

ü  Best practices

Ø  Unit 2-Common Troubleshooting Tools

ü  Advanced head and tail

ü  awk

ü  Syslog for logging

ü  Rebuilding the RPM Database

ü  Verifying with RPM