Showing posts with label script. Show all posts
Showing posts with label script. Show all posts

Friday, December 31, 2010

Command Line Init Script Generator For Linux Or Unix

Hey There,

To start this week off, we've got another updated script (now that we have over 600 some odd posts, we actually have about 400 some odd loose ends to finish up, as well ;) This script is an update of an earlier post we did to showcase a basic init script generator for Linux or Unix, again hearkening back to 2007. How time flies when you're not paying attention to what you're doing ;)

This script includes several updates (which haven't been tested to perfection - so feel free to write in with complaints ;) such as the ability to use command line arguments to specify a wider variety of options, set some to default and not even enter any (except three) if you don't want to. Actually, the only three required arguments are the name of the program you want to start with the init script you generate, the fully qualified name of the same program and the name of the init script itself. All the other options are easily displayed by just running the command without any arguments, like so:

host # ./rcscript.sh

Options -n -p and -f are required!

Usage: ./rcscript.sh [-h for this help screen]

Required switches:
[-n init script name] [-p name of program to control]
[-f controlled program's fully qualified name]

Optional switches:
[-i init directory] [-3 rc3.d directory] [-0 rc0.d directory
[-s start options for your program]
[-S stop options for your program]
[-k additional programs to kill on stop - space separated]
[-b common binary directory - defaults to /usr/bin]

Be sure to "double quote" any switch arguments with spaces!


Hope you enjoy the updates and this script helps you out in some way or fashion :)

Cheers,

Creative Commons License
rcscript.sh by Mike Golvach is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License.
Based on a work at linuxshellaccount.blogspot.com.
Permissions beyond the scope of this license may be available at http://linuxshellaccount.blogspot.com.

#!/bin/sh
#
# Generic Init Script Creator
# 2009 - Mike Golvach - eggi@comcast.net
#
#Creative Commons License
rcscript.sh by Mike Golvach is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License.
Based on a work at linuxshellaccount.blogspot.com.
Permissions beyond the scope of this license may be available at http://linuxshellaccount.blogspot.com.
#

trap 'rm -f ${init_dir}/${script_name} ${rc3_dir}/${script_name} ${rc0_dir}/${script_name} ${script_name}' 1 2 3 9 15

# Tips for a few command line arguments - setting an argument to zero will not be considered equivalent to not defining it
# a. set init_dir to 0 if you don't want any scripts and links setup - set rc3_dir, rc2_dir and/or init_dir to "0" to not create that link/script.
# b. If any of the init_dir, rc2_dir and/or rc3_dir variables are not set, they will default to the examples below.
# c. options that include 0 in the example do not need to be set and are entirely optional
#

# init_dir= /etc/init.d or 0
# rc3_dir= /etc/rc3.d or 0
# rc0_dir= /etc/rc0.d or 0
# script_name= myStartScript.sh
# program_name= testProgram
# fully_qualified_program_name= /usr/local/sbin/testProgram
# start_options= "start" or 0
# stop_options= "stop" or 0
# sure_kill= "space delimited list of extra programs to kill" or 0
# common_bin_directory= "place where common binaries are on your system" or defaults to /usr/bin
#

function usage()
{
echo
echo "Usage: $0 [-h for this help screen]"
echo
echo "Required switches:"
echo "[-n init script name] [-p name of program to control]"
echo "[-f controlled program's fully qualified name]"
echo
echo "Optional switches:"
echo "[-i init directory] [-3 rc3.d directory] [-0 rc0.d directory"
echo "[-s start options for your program]"
echo "[-S stop options for your program]"
echo "[-k additional programs to kill on stop - space separated]"
echo "[-b common binary directory - defaults to /usr/bin]"
echo
echo "Be sure to \"double quote\" any switch arguments with spaces!"
echo
exit 1
}

while getopts 0:3:b:f:hi:k:n:p:s:S: option
do
case $option in
'i')
if [[ "$OPT_ARG" == "0" ]]
then
init_dir=0
else
init_dir="$OPTARG"
fi
;;
'3')
if [[ "$OPT_ARG" == "0" ]]
then
rc3_dir=0
else
rc3_dir="$OPTARG"
fi
;;
'0')
if [[ "$OPT_ARG" == "0" ]]
then
rc0_dir=0
else
rc0_dir="$OPTARG"
fi
;;
'n')
script_name="$OPTARG"
;;
'p')
program_name="$OPTARG"
;;
'f')
fully_qualified_program_name="$OPTARG"
;;
's')
if [[ "$OPT_ARG" == "0" ]]
then
start_options=""
else
start_options="$OPTARG"
fi
;;
'S')
if [[ "$OPT_ARG" == "0" ]]
then
stop_options=""
else
stop_options="$OPTARG"
fi
;;
'k')
if [[ "$OPT_ARG" == "0" ]]
then
additional_kills=""
else
additional_kills="$OPTARG"
fi
;;
'b')
if [[ "$OPT_ARG" == "0" || -z "$OPT_ARG" ]]
then
common_bin_directory="/usr/bin"
else
common_bin_directory="$OPTARG"
fi
;;
'h')
usage
;;
*)
usage
;;
esac
done

if [[ -z $script_name || -z $program_name || -z $fully_qualified_program_name ]]
then
echo
echo "Options -n -p and -f are required!"
usage
fi

if [[ -z $common_bin_directory ]]
then
common_bin_directory="/usr/bin"
fi

if [[ -e $script_name ]]
then
rm $script_name
fi

cat << EOM >>$script_name

#!/bin/sh

#
# ${program_name} init script
# Usage: $script_name [start|stop|restart]
#

case \$1 in
'start')
echo
echo Starting ${program_name}....
echo
${fully_qualified_program_name} ${start_options} >/dev/null 2>&1 &
${program_name}_running=\`${common_bin_directory}/ps -ef|${common_bin_directory}/grep "${fully_qualified_program_name}"|${common_bin_directory}/grep -v grep|${common_bin_directory}/awk '{print \$2}'\`
if [ "\$${program_name}_running" == "" ]
then
echo "${program_name} start cannot be confirmed. Please check"
echo "system output and application logs for further detail"
else
echo "${program_name} started successfully"
fi
;;
'stop')
echo
echo Stopping ${program_name}....
echo
if [ "$stop_options" != " " ]
then
${fully_qualified_program_name} ${stop_options} >/dev/null 2>&1 &
fi
${program_name}_walking=\`${common_bin_directory}/ps -ef|${common_bin_directory}/grep "${fully_qualified_program_name}"|${common_bin_directory}/grep -v grep|${common_bin_directory}/awk '{print \$2}'\`
if [ "\$${program_name}_walking" == "" ]
then
echo "${program_name} does not appear to be running."
echo "Process not found. Not shut down."
else
counter=5
dead="alive"
echo "Shutting ${program_name} down."
echo "${fully_qualified_program_name} - pid ${program_name}_walking - stopping."
while [ \$counter -gt 0 ]
do
${program_name}_still_walking=\`${common_bin_directory}/ps -ef|${common_bin_directory}/grep "${fully_qualified_program_name}"|${common_bin_directory}/grep -v grep|${common_bin_directory}/awk '{print \$2}'\`
if [ "\$${program_name}_still_walking" != "" ]
then
echo "killing pid \$${program_name}_still_walking "
${common_bin_directory}/kill \$${program_name}_still_walking
counter=\`expr \$counter - 1\`
sleep 1
else
dead="dead"
echo "dead"
counter=0
fi
done
if [ \$dead = "alive" ]
then
echo "Could not kill process after 5 attempts."
echo "Process ${program_name}_walking still active."
${common_bin_directory}/ps -ef|${common_bin_directory}/grep "${fully_qualified_program_name}"|${common_bin_directory}/grep -v grep
fi
fi
EOM
if [[ $additional_kills && ! -z $additional_kills ]]
then
cat << EOM >>$script_name
for aks in `echo $additional_kills`
do
echo
echo Stopping \$aks....
echo
aks_walking=\`${common_bin_directory}/ps -ef|${common_bin_directory}/grep "\$aks"|${common_bin_directory}/grep -v grep|${common_bin_directory}/awk '{print \$2}'\`
if [ "\$aks_walking" == "" ]
then
echo "\$aks does not appear to be running."
echo "Process not found. Not shut down."
else
counter=5
dead="alive"
echo "Shutting \$aks down."
echo "\$aks - pid \$aks_walking - stopping."
while [ \$counter -gt 0 ]
do
aks_still_walking=\`${common_bin_directory}/ps -ef|${common_bin_directory}/grep "\$aks"|${common_bin_directory}/grep -v grep|${common_bin_directory}/awk '{print \$2}'\`
if [ "\$aks_still_walking" != "" ]
then
echo "killing pid \$aks_still_walking "
${common_bin_directory}/kill \$aks_still_walking
counter=\`expr \$counter - 1\`
sleep 1
else
dead="dead"
echo "dead"
counter=0
fi
done
if [ \$dead = "alive" ]
then
echo "Could not kill process after 5 attempts."
echo "Process \$aks_walking still active."
${common_bin_directory}/ps -ef|${common_bin_directory}/grep "\$aks"|${common_bin_directory}/grep -v grep
fi
fi
done
EOM
fi
cat << EOM >>$script_name
;;
'restart')
echo
echo ReStarting ${program_name}....
echo
\$0 stop
\$0 start
;;
*)
echo "Usage: \$0 [start|stop|restart]"
;;
esac
EOM

${common_bin_directory}/chmod 750 ${script_name};
${common_bin_directory}/cp ${script_name} ${init_dir}/${script_name}
${common_bin_directory}/ln -s ${init_dir}/${script_name} ${rc3_dir}/${script_name}
${common_bin_directory}/ln -s ${init_dir}/${script_name} ${rc0_dir}/${script_name}


, Mike


Banner for College Student-Oriented Sites (728 version 1)



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Sunday, June 14, 2009

msilaerruS nI tnemirepxE llamS A - dnEkooB A sA tsoP s'yadnuS

Wishing everyone a nice tasty Sunday!

If You haven't guessed Already, This is yesterday's post after being processed by our Backward Web Page Perl Script. Why? Why Not? ;)
Below is another collection of some funny, bizarre and interesting pictures I've found all over the web.

Since I collect these randomly, here and there, I don't remember where I got most of them by the time I set to put them up here. This week, however, I do remember that createdigitalmotion.com hosts the 3-D picture below. I haven't been able to test if it works (since I still don't know where to go to buy those old-style blue/red 3-D specs), but I can almost guarantee you that it will give you a splitting headache if you stare at it for too long ;)

Enjoy the amusing pictures. Try not to laugh out loud, or roll on the floor while laughing your ass off. Your safety is our primary concern ;)

Cheers,

, Mike






















Banner for College Student-Oriented Sites (728 version 1)



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Monday, June 8, 2009

NetBackup Backup Report Script

Hey there,

Today we've got an update that's been a while in the coming (mostly because no one ever requested it ;). Back in December of 2007, we put out a Perl script to report on NetBackup backup activity and, today, it's finally being updated.

But first, a few notes:

NOTE: If you don't care about what got updated or seeing any usage statistics and information, the script is tacked on to the end of this post and you can jump to it by clicking here.

VERY SPECIAL NOTE: This script's update has only been tested on Solaris (7,8,9,10) with the regular NetBackup backup and Oracle modules. If anyone runs this against another module (for Windows backups or any other module NetBackup has available) and it doesn't work, please send me (via the email link at the upper right of the page) your /usr/openv/netbackup/bin/admincmd/bpdbjobs -all_columns output (The location of your bpdbjobs binary may differ) for a successful job and a failed one. I'll be happy to update this script to include support for them. Of course, if you do send me that output, please alter any information that may be proprietary and get you in trouble! I promise not to do anything with it, but it's best to play it safe.

Some of the changes incorporated in this update include a check for "Catalog Backup" and "Backup Image Cleanup" jobs as well as checks to make the output more easily and quickly viewable. For instance, in the original script, no status was actually put in the Subject line. In this version, you'll get a status summary ("NO JOBS," "ALL OK" or "ERRORS") as well as the name of the server the report was generated on. Then you can just delete all the email reports you get that report that everything's fine (according to the reporter - Now, I've officially used that word enough to match my keywords. Let's just pray I don't have to use it again ;). And (since it was simple to do and not all of you may need to keep evidence that all was well at one point ;) we added a small - commented out for now - outer loop so you can remove those comments and have the script "only" mail you if there are errors to report(argh!).

Click on the picture below and maybe you'll be able to read those poorly doctored Subject lines ;)

My loyal Subjects

The script is, again, very simple to use. All you need to do is run it from the command line and let it know how far back you want to go for your reporting (Generally, putting this in cron to run once a day and setting the command line offset to 86400 is good enough). Of course, the script can only report (I've stopped counting, now ;) as far back as "bpdbjobs" is set to remember! If you forget how to use it, and don't want to read the first 10 or 15 lines of it, just run it without any arguments and you'll get a "Usage" message.

host # ./bpreport.pl
Usage: ./bpreport.pl TimeInSecondsToReportBack


The output generally looks like the output below and, under moderate load on a system that only does its own backups, it runs in under a second for a day's worth of information (again, this timing would be affected by how much activity is logged on your server in the time span you specify on the command line)

Click the picture below and maybe something really cool will happen... probably not. Most likely the picture will just get larger again ;)

Put out output

Below are some sample report outputs from the three types of reports shown in the "Subjects" picture from way up above, and the command line output directly above. After that, it's the script and we're out, baby ;)

Cheers,

P.S. Please forgive the exclamation marks. Typing in all caps is just so very exciting ;)

Click here to get directly to the script!



SAMPLE EMAIL REPORT WITH ERRORED JOBS - 24 HOUR SPAN!


NetBackup Status Report
-----------------------------------------------
Report Span = 06/06/09 ¦ 13:46:15 through 06/07/09 ¦ 13:46:15
-----------------------------------------------

Jobs With Errors To Report!

***************************************************
Server ¦ Policy ¦ Kb ¦ Min Elapsed ¦ Start Date ¦ Time ¦End Date ¦ Time - Error - File List
***************************************************

host123 ¦ warm_backup ¦ 0 kb ¦ 13.90 min ¦ 06/06/09 ¦ 23:24:02 ¦ 06/06/09 ¦ 23:37:56
ERROR: the backup failed to back up the requested files!!!
/usr/local/bin/mybackup.sh
***************************************************

-----------------------------------------------

Jobs Completed Successfully!

***************************************************

Server ¦ Policy ¦ Kb ¦ Min Elapsed ¦ Start Date ¦ Time ¦End Date ¦ Time - File List
***************************************************

host123 ¦ Catalog Backup ¦ 209056 kb ¦ 0.25 min ¦ 06/07/09 ¦ 02:04:35 ¦ 06/07/09 ¦ 02:04:50
***************************************************
host123 ¦ Catalog Backup ¦ 211872 kb ¦ 0.23 min ¦ 06/06/09 ¦ 23:37:52 ¦ 06/06/09 ¦ 23:38:06
***************************************************
host123 ¦ Default-Backup-Policy ¦ 12589911 kb ¦ 10.77 min ¦ 06/06/09 ¦ 23:00:20 ¦ 06/06/09 ¦ 23:11:06
/abcdef_1
***************************************************
host123 ¦ Default-Backup-Policy ¦ 12589911 kb ¦ 11.45 min ¦ 06/06/09 ¦ 23:24:09 ¦ 06/06/09 ¦ 23:35:36
/ghijkl_1
***************************************************
host123 ¦ Default-Backup-Policy ¦ 240761 kb ¦ 0.92 min ¦ 06/06/09 ¦ 23:12:31 ¦ 06/06/09 ¦ 23:13:26
/mnopqr_1
***************************************************
host123 ¦ Default-Backup-Policy ¦ 2759769 kb ¦ 3.77 min ¦ 06/06/09 ¦ 23:00:20 ¦ 06/06/09 ¦ 23:04:06
/stuvwxyz_1
***************************************************
host123 ¦ Default-Backup-Policy ¦ 2759769 kb ¦ 4.28 min ¦ 06/06/09 ¦ 23:24:09 ¦ 06/06/09 ¦ 23:28:26
/abcdef_2
***************************************************
host123 ¦ Default-Backup-Policy ¦ 548 kb ¦ 0.62 min ¦ 06/06/09 ¦ 23:36:49 ¦ 06/06/09 ¦ 23:37:26
/ghijkl_2
***************************************************
host123 ¦ Default-Backup-Policy ¦ 6640326 kb ¦ 8.43 min ¦ 06/06/09 ¦ 23:00:20 ¦ 06/06/09 ¦ 23:08:46
/mnopqr_2
***************************************************
host123 ¦ Default-Backup-Policy ¦ 6640326 kb ¦ 9.12 min ¦ 06/06/09 ¦ 23:24:09 ¦ 06/06/09 ¦ 23:33:16
/stuvwxyz_2
***************************************************
host123 ¦ Default-Backup-Policy ¦ 8547 kb ¦ 0.63 min ¦ 06/06/09 ¦ 23:11:28 ¦ 06/06/09 ¦ 23:12:06
/abcdefghijklm_3
***************************************************
host123 ¦ Default-Backup-Policy ¦ 8547 kb ¦ 0.63 min ¦ 06/06/09 ¦ 23:35:48 ¦ 06/06/09 ¦ 23:36:26
/nopqrstuvwxyz_3
***************************************************
host123 ¦ image catalog cleanup ¦ 0 kb ¦ 0.02 min ¦ 06/06/09 ¦ 15:19:14 ¦ 06/06/09 ¦ 15:19:15
***************************************************
host123 ¦ image catalog cleanup ¦ 0 kb ¦ 0.02 min ¦ 06/06/09 ¦ 23:38:08 ¦ 06/06/09 ¦ 23:38:09
***************************************************
host123 ¦ image catalog cleanup ¦ 0 kb ¦ 0.02 min ¦ 06/07/09 ¦ 02:04:51 ¦ 06/07/09 ¦ 02:04:52
***************************************************
host123-p ¦ Diff-Inc ¦ 3070549 kb ¦ 4.60 min ¦ 06/07/09 ¦ 02:00:00 ¦ 06/07/09 ¦ 02:04:36
/
***************************************************
host123-s ¦ Diff-Inc ¦ 359900 kb ¦ 1.77 min ¦ 06/07/09 ¦ 02:00:00 ¦ 06/07/09 ¦ 02:01:46
/
***************************************************

-----------------------------------------------



SAMPLE EMAIL REPORT WITH NO ERRORED JOBS - 12 HOUR SPAN!


NetBackup Status Report
-----------------------------------------------
Report Span = 06/07/09 ¦ 01:46:22 through 06/07/09 ¦ 13:46:22
-----------------------------------------------


Jobs Completed Successfully!

***************************************************

Server ¦ Policy ¦ Kb ¦ Min Elapsed ¦ Start Date ¦ Time ¦End Date ¦ Time - File List
***************************************************

host123 ¦ Catalog Backup ¦ 209056 kb ¦ 0.25 min ¦ 06/07/09 ¦ 02:04:35 ¦ 06/07/09 ¦ 02:04:50
***************************************************
host123 ¦ image catalog cleanup ¦ 0 kb ¦ 0.02 min ¦ 06/07/09 ¦ 02:04:51 ¦ 06/07/09 ¦ 02:04:52
***************************************************
host123-p ¦ Diff-Inc ¦ 3070549 kb ¦ 4.60 min ¦ 06/07/09 ¦ 02:00:00 ¦ 06/07/09 ¦ 02:04:36
/
***************************************************
host123-s ¦ Diff-Inc ¦ 359900 kb ¦ 1.77 min ¦ 06/07/09 ¦ 02:00:00 ¦ 06/07/09 ¦ 02:01:46
/
***************************************************

-----------------------------------------------



SAMPLE EMAIL REPORT WITH NO JOBS AT ALL - 6 HOUR SPAN!


NetBackup Status Report
-----------------------------------------------
Report Span = 06/07/09 ¦ 07:46:49 through 06/07/09 ¦ 13:46:49
-----------------------------------------------

No Activity To Report! Check If NetBackup Is Running!
-----------------------------------------------





Creative Commons License
bpreport.pl by Mike Golvach is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License.
Based on a work at linuxshellaccount.blogspot.com.
Permissions beyond the scope of this license may be available at http://linuxshellaccount.blogspot.com.

#!/usr/bin/perl
#
# bpreport.pl - 2009 Mike Golvach - eggi@comcast.net
#
#bpreport.pl by Mike Golvach is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License. Based on a work at linuxshellaccount.blogspot.com. Permissions beyond the scope of this license may be available at http://linuxshellaccount.blogspot.com.
#

$first_tmp = $$;
$offset = $ARGV[0];

if ( -f "tmp.$first_tmp" ) {
unlink("tmp.$first_tmp")
}

if ( $#ARGV != 0 ) {
print "Usage: $0 TimeInSecondsToReportBack\n";
exit(1)
}

system("/usr/openv/netbackup/bin/admincmd/bpdbjobs -all_columns >>tmp.$first_tmp");

open(FT, "<tmp.$first_tmp");
@all_jobs = <FT>;
close(FT);

if ( -f "tmp.$first_tmp" ) {
unlink("tmp.$first_tmp")
}

$global_error_flag = 0;
$separator = "***************************************************";
$hostname=`hostname`;

foreach $job (@all_jobs) {
@file_list="";
@job = split(/,/, $job);
chomp($job[14]);
if ( $job[14] =~ /^$/ ) {
$job[14] = "0";
}
$error_flag = 0;
$report_time = $job[8];
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($job[8]);
$today = parse_date($sec,$min,$hour,$mday,$mon,$year,$wday);
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($job[10]);
chomp($today_end = parse_date($sec,$min,$hour,$mday,$mon,$year,$wday));
if ( $today_end =~ /12\/31\/69/ ) {
$today_end = "JOB IN PROGRESS";
}
$today_span_pre = ( $job[9] / 60 );
$today_span = sprintf("%.2f min", $today_span_pre);
$actual_time = time;
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);
$right_now = parse_date($sec,$min,$hour,$mday,$mon,$year,$wday);
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time-$offset);
$back_then = parse_date($sec,$min,$hour,$mday,$mon,$year,$wday);
report_span($report_time, $actual_time, $offset);
next if $dont_report;
if ( $job[3] > 1 ) {
@error_condition = `/usr/openv/netbackup/bin/admincmd/bperror -S $job[3]`;
chomp($error_condition = $error_condition[0]);
$job[3] = $error_condition;
$error_flag = 1;
$global_error_flag = 1;
}
next if ( $job[6] eq "backuphost" && $job[14] == 0 && ! $error_flag );
$file_list=32;
$file_list_still_going = 1;
$just_starting = 1;
while ($file_list_still_going) {
if ( $job[$file_list] =~ /(\/¦\\¦ALL)/ ) {
if ( $just_starting ) {
push(@file_list, "$job[$file_list] ");
$file_list++;
$just_starting = 0;
} else {
push(@file_list, "¦ $job[$file_list] ");
$file_list++;
}
} else {
$file_list_still_going = 0;
}
}
if ( ! $job[5] && ! $job[6] ) {
if ( $job[42] =~ /^.*(image catalog cleanup).*$/ ) {
$job[6] = $job[25];
$job[5] = $job[42];
$job[5] =~ s/^.*(image catalog cleanup).*$/$1/;
} else {
$job[5] = "Server UNKNOWN";
}
} elsif ( ! $job[5] && $job[6] ) {
if ( $job[42] =~ /^.*(Catalog Backup).*$/ ) {
$job[6] = $job[25];
$job[5] = $job[42];
$job[5] =~ s/^.*(Catalog Backup).*$/$1/;
} else {
$job[5] = "Policy UNKNOWN";
}
}
if ( $#file_list == 0 ) {
push(@job_successes, "$job[6] ¦ $job[5] ¦ $job[14] kb ¦ $today_span ¦ $today ¦ $today_end\n$separator\n");
} elsif ( $error_flag ) {
push(@job_errors, "$job[6] ¦ $job[5] ¦ $job[14] kb ¦ $today_span ¦ $today ¦ $today_end\nERROR: ${error_condition}!!!\n@{file_list}\n$separator\n");
} else {
push(@job_successes, "$job[6] ¦ $job[5] ¦ $job[14] kb ¦ $today_span ¦ $today ¦ $today_end\n@file_list\n$separator\n");
}
}

$¦=1;

## Don't send out okay mails if everything is fine)
#if ( @job_errors ) {

open(MAIL, "¦/usr/lib/sendmail -t");
select(MAIL);
if ( ( $#job_errors < 0 )&& ( $#job_successes < 0 ) ) {
print "Subject: NO JOBS: $hostname NetBackup Status Report - $right_now\n";
} elsif ( @job_errors ) {
print "Subject: ERRORS: $hostname NetBackup Status Report - $right_now\n";
} else {
print "Subject: ALL OK: $hostname NetBackup Status Report - $right_now\n";
}
print "From: root\@yourcompany.com\n";
print "Reply-To: you\@yourcompany.com\n";
print "To: you\@yourcompany.com\n";
print "\n\n";
print "NetBackup Status Report\n";
print "-----------------------------------------------\n";
print "Report Span = $back_then through $right_now\n";
print "-----------------------------------------------\n";
print "\n";
if ( ( $#job_errors < 0 )&& ( $#job_successes < 0 ) ) {
print "No Activity To Report! Check If NetBackup Is Running!\n";
print "-----------------------------------------------\n";
} else {
if ( @job_errors ) {
print "Jobs With Errors To Report!\n\n$separator\n";
print "Server ¦ Policy ¦ Kb ¦ Min Elapsed ¦ Start Date ¦ Time ¦End Date ¦ Time - Error - File List\n";
print "$separator\n\n";
@job_errors = sort @job_errors;
$job_error_count = @job_errors;
if ( $job_error_count ) {
print "@job_errors\n";
} else {
print "NO ERRORS TO REPORT. REPORT MAY BE IN ERROR!!!\n\n"
}
print "-----------------------------------------------\n";
}
if ( @job_successes ) {
print "\nJobs Completed Successfully!\n\n$separator\n\n";
print "Server ¦ Policy ¦ Kb ¦ Min Elapsed ¦ Start Date ¦ Time ¦End Date ¦ Time - File List\n";
print "$separator\n\n";
@job_successes = sort @job_successes;
print "@job_successes\n";
print "-----------------------------------------------\n";
}
}
#} # End of possibly-no-mail section

sub parse_date {

my ($sec,$min,$hour,$mday,$mon,$year,$wday) = @_;
%months = qw(0 Jan 1 Feb 2 Mar 3 Apr 4 May 5 Jun 6 Jul 7 Aug 8 Sep 9 Oct 10 Nov 11 Dec);
%weekdays = qw(0 Sun 1 Mon 2 Tue 3 Wed 4 Thu 5 Fri 6 Sat);
if ( $sec < 10 ) {
$sec = "0" . $sec;
}
if ( $min < 10 ) {
$min = "0" . $min;
}
if ( $hour < 10 ) {
$hour = "0" . $hour;
}
if ( $mday < 10 ) {
$mday = "0" . $mday;
}
if ( $yday < 10 ) {
$yday = "0" . $yday;
}
$mon = $mon + 1;
if ( $mon < 10 ) {
$mon = "0" . $mon;
}
$wday = $weekdays{$wday};
$year = 1900 + $year;
$year =~ s/^..(.*)$/$1/;
$nice_date = "${mon}/${mday}/$year ¦ ${hour}:${min}:$sec";
}

sub report_span {

my ($report_time,$actual_time,$offset) = @_;
$dont_report = 0;
$more_than_offset_hours_old = $actual_time - $report_time;
if ( $more_than_offset_hours_old > $offset ) {
$dont_report = 1;
}
}


, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Thursday, April 16, 2009

Beginner's Shell Scripting On Linux And Unix: Why And How

Hey There,

Recently I was asked a very simple question that I'm pretty sure I've never addressed in this blog over the mass of posts already churned out. Shell scripting has been addressed, to a great degree, on this blog, but (even the beginner stuff) has included a presumption of certain knowledge. It's easy to forget what it was like back when I started working on Linux and Unix, so I often make the cardinal sin of assuming everyone has already asked this question and had it answered.

The question was simple (although, a three-parter) : I know how to use Unix/Linux enough to get around on the command line, but why would I want to write shell scripts? And how do I write a shell script, anyway? Isn't it just going to be more hassle than I already have to deal with?

The above was only-slightly paraphrased ;)

So, even though I could dump these questions, and their answers, off by conveniently linking to any number of other sites, I thought it might be a good idea to just answer them here. I've never posted on the subject before and it can't hurt to have this basic information available in our site search.

IMPORTANT NOTE FOR LINUX/UNIX USER'S AND SHELL SCRIPTERS WHO CAN ANSWER ALL THREE QUESTIONS: This post is probably going to be very boring for you ;) I'll do my best to see that it doesn't come off too dry, but I won't be writing about anything you don't know already, and I will be purposefully leaving out certain things that may drive you crazy ;)

So, to attack the question, I've decided to respond in two parts (since the final two naturally flow together) and keep this post as simple as possible. My feeling is that if you're reading this and aren't sure what shell scripting is and/or why you should even bother to make use of it, you're not looking to be weighed down by a ton of technical jargon and over-explanation which would probably just cloud the issue and be more difficult to understand than necessary. Here we go :)

Q: Why would I want to write shell scripts? Isn't it just going to be more hassle than I already have to deal with?

A: To answer the question "Why?," the most succinct answer would be "Why Not?" There are several good reasons to write shell scripts (for your own use or for the use of others), one of which addresses the "hassle" issue ;)

1. Shell scripting, at its most basic, is actually very easy to master (assuming that you can get around well enough on the Unix or Linux command line to accomplish whatever it is you need to do). In fact, your knowledge of the command line will be almost-directly commensurate with your ability to write shell scripts of varying complexity.

2. Writing a shell script is only slightly (and I mean "very" slightly) more of a hassle than doing the exact same thing on the command line. It can, very literally, be thought of as a way of saving your command line work so that you never have to worry about remembering it ever again.

3. If you've written a shell script once, you never have to write it again (unless you want to improve, or modify, it - well, you can have your own reasons. It's none of my business, anyway ;). If you hammer out a command line to get your chores done (assuming you don't have one of many other methods of saving command line history enabled), you're going to need to keep hammering out that command line every time you do your work.

4. For a very basic example of how writing simple shell scripts can "save" you from hassle. Consider the following situation:

Every day at work you come in, log in to your Unix/Linux terminal and type:

host # cd /my/main/work/directory
host # ls |grep myuserid.txt
host # rm myuserid.txt
host # find . -name core -print >>myuserid.txt
host # cat myuserid.txt|xargs rm -f
host # cd


While that example above illustrates more than a few opportunities for improvement, that's not today's issue ;) Now imagine if, instead of logging in to your Unix/Linux terminal and grinding out those six lines, all you had to do was log in and type the following (we're keeping this so basic and simple, we're not even going to deal with using the shebang line in your script or setting the script up to be executable on its own):

host # /bin/bash myscript

Much less hassle, yeah? :)

Q: How do I write a shell script, anyway?

A: It's actually extremely simple (and we're keeping it extremely simple for now). There are a lot more steps you can take to make the whole perception of "hassle" wither even further, as well as make your script writing even more convenient for you, but that's for another day. I'm fighting every instinct in my head to try and keep this as "basic" as possible and not introduce any "logical constructs" or "error checking" into your first script.

The first thing to consider, when answering this question, is another question. "What" is a shell script, anyway? It goes a long way to making the "how" obvious. A shell script, in its most pristine state, is simply a collection of command lines. No joking. That's all it has to be, and it can be just that simple.

Consider the example above (point number 4 of the answer to the first question). If you simply take all those lines that you type everyday and put them in a file, you will have written your first shell script! So, open up your favorite editor and either straight-type or cut-and-paste all those lines into it. It's very important, since we're not going to address issues of efficiency beyond the basic, that you type each command line on a separate line in your new file. It should look exactly like this (your commands may differ, of course ;)

cd /my/main/work/directory
ls |grep myuserid.txt
rm myuserid.txt
find . -name core -print >>myuserid.txt
cat myuserid.txt|xargs rm -f
cd


Save that file as "myscript" (or whatever else you'd rather call it) and you're all done. You've written your first shell script!

Now, instead of typing 6 lines of commands, you only have to type one. Since we're not getting into setting up your shell script to be executable on its own, you just need to invoke it with a shell (preferably the one you use to type those commands with every day). If you're not sure what shell you use, you can find out that information by simply typing the following at the command line:

host # echo $SHELL
/bin/bash


and, assuming that your environment variables are all set up correctly (arggh, losing focus... ;), that command will return the value of your shell. Now you can execute your shell script easily in either one of two ways.

a. If you're in the same directory as your script, you can execute it by running:

host # /bin/bash myscript

b. If you're not in the same directory as your script, or prefer to use full paths, you can invoke it like this:

host # /bin/bash /directory/where/the/script/is/myscript

and all of your commands will be executed, in order, as if you had manually typed them at the command line yourself.

That, in an nutshell, is the how and the why of writing a basic shell script. It should also address the "hassle" issue ;) Taking this simple example (and learning no more about how to create shell scripts than you've learned so far), you'll probably realize that there are several other things you do every day (all of which require prodigious typing) that you can put into shell scripts and stave off carpal-tunnel syndrome for at least a little while longer ;)

Over time, you should begin to experience the convenience shell scripting has to offer you and become curious about how much more it can help you by reducing the hassle of your workaday tasks and introducing automation of the mundane. Eventually, you may even become interested in more advanced concepts and begin to enjoy the full benefit of shell scripting.

The main thing to keep in mind (no matter how complex your shell script ever becomes) is that a shell script (shebang line excluded) is merely a collection of commands that you could run, yourself, from the command line. Keep in mind, also, that your shell includes many features that were not showcased here today. Even when you avail yourself of those, any shell script can be written on the command line and anything you can write on the command line can easily become a shell script.

I hope that long-winded answer to a fairly short (but definitely loaded ;) question was of some help to you. If you're interested in learning more, search around this blog or elsewhere on the Internet. If you can't find the answer to a question you have, feel free to email me via the link at the top right of the page. My response time isn't what it used to be (before I began inviting people to write me ;), but I will get back to you as soon as I can.

As a final exercise to demonstrate the simplicity of very-basic shell scripting (assuming, as above, that /bin/bash is your shell), try the following at your command line:

host # echo hi
hi
host # echo "echo hi" >>newscript
host # /bin/bash newscript
hi


Nice work :)

Cheers,

, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Tuesday, April 14, 2009

Compiling Perl Into Binary Code On Linux And Unix. Cool, But...

Hey there,

Since as long back as I can remember, the Perl compiler has always been a vague, yet distant objective. Something that, when I was really into it, wasn't fully supported and didn't work unless you rewrote it yourself (which kind of ran contrary to the reason you were using Perl in the first place: To get complicated tasks taken care of quickly!)

Over the years, I've seen different front-ends come and go (a few of them which looked pretty slick), but I never bothered to check any of them out to a great degree. If I absolutely "had" to write something in "C," then I would. Otherwise, I'd go with Perl. Actually, my coding philosophy is a bit more pliable. Basically, given that I've been monkeying around with Unix and Linux (for pay) for 13 years or so, I tend to go with whatever is easiest, most reliable and most portable depending upon the task at hand. If it can be written in a "find" statement, that's good enough for me. Shell script, Sed and Awk (which usually end up in a Shell script), tcl, tk, expect, Perl, Python, C; whatever is the simplest way to resolve a problem and not kill my machines or compromise security to a great degree; that's what I go with. I see no point in writing a Perl script to open up a file descriptor to read STDIN, parse that and spit out lines containing the word "hi," like in this example:

#!/usr/bin/perl

# perl.pl - A complete waste of time ;)

open(FILE, "<myfile.txt");
@file=<FILE>;
close(FILE);

foreach $x (@file) {
print $x if ( $x =~ /hi/ );
}


host # ./perl.pl
hi there how are you?
Say hi to the mrs. for me


When simply typing (or making this into a one-line shell script):

host # grep hi myfile.txt
hi there how are you?
Say hi to the mrs. for me


will get me the exact same results. Admittedly, my Perl script above was a bit bloated, but the excess goes toward the point I'm making. At least I hope it does ;)

I came across an interesting product (Read: Not Free) called perl2exe that caught my attention since it can compile code for different OS's (derived from your original Perl script) and (after testing), actually works quite well. You can check it out for a free 30 day trial, although you should be aware that all of your generated code will have an advertisement tacked on the end and a 2 second delay. It'll also help you to know (if you want to check it out) that you need to download all the different OS packages (not just the one for the OS on which you're going to install) if you want to use the cross-platform code generation feature. It will compile code for Linux on a Unix box, but you have to download both the Linux and Unix distro gzipp'ed tarballs in order to do this. I'm not sure why. It may be just be another pain-in-the-arse designed to convince you to buy; but I am by no means, and in no way, shape or form, suggesting that you do so. And I'll tell you why...

Although this is one of the nicest packages, for compiling Perl scripts into executable binary code, that I've come across in a while (ease of use, etc) it still fails to address the one "major" flaw of all Perl-To-Binary conversion tools on the market. When you compile your Perl Script, you've lost all the flexibility you began using Perl for in the first place! No more inline editing for you ;)

For instance, a while back, we did a post on getting the year from Solaris' wtmpx file. Taking that Perl script and converting it to binary was a snap, like so:

host # time perl2exe -o rip_wtmpx rip_wtmpx.pl
Perl2Exe V9.100 Copyright (c) 1997-2008 IndigoSTAR Software

This is an evaluation version of Perl2Exe, which may be used for 30 days.
For more information see the attached pxman.htm file,
or visit http://www.indigostar.com

Converting 'rip_wtmpx.pl' to rip_wtmpx

real 0m0.506s
user 0m0.382s
sys 0m0.102s


Done in about a half a second. And, much to my amazement, it worked perfectly:

host # ./rip_wtmpx
PROCESSING WTMPX FILE FOR CROSS-CHECKING: .....
PROCESSED 2393 ENTRIES

root Thu Nov 6 13:42:07 2008
root Thu Nov 6 13:48:00 2008
root Thu Nov 6 13:51:58 2008
root Thu Nov 6 13:54:12 2008
root Thu Nov 6 14:38:20 2008
root Thu Nov 6 14:39:52 2008
root Thu Nov 6 14:39:54 2008
...


Which is the same result I get from running the Perl script directly (The whole point of it was to get the "year" from wtmpx, since the "last" command, and others like it, don't seem to want to give out that information, even though it's in the wtmp/utmp struct and proves very useful if you don't rotate your logs every year like I obviously don't ;)

But, then, you may recall that we followed up that post with a post on how to get the year from lastlog on Linux since Linux's implementation of the wtmp(x) struct had a totally different "pack" template than the one used on Solaris. I was getting around this by coding a mixture of both scripts and having it evaluate the host OS and act accordingly. Even so, for the odd machine (or new OS implementation we rolled out) I'd have to make slight tweaks. And that (although it's a pitiful example, I know ;) is where the whole "convenience" of binary Perl falls apart. Now, everytime I want to make a simple change (in this extremely simple example) I have to update my Perl script, regenerate binary code from it, for all the OS's required, and then redistributed all of that. The alternative, using just Perl, would be to modify the Perl script and redistribute the one version everywhere.

True, in this instance, there doesn't seem to be that much difference (just multiply it by a larger number of machines on your server farm and it gets worse ;), but the "convenience" really makes a difference when you find yourself on a machine that's got some weird kink in it. If you're using a straight-up Perl script, you can just edit it slightly to adjust for however fugged-up that particular box is. You don't have to maintain a separate binary. Odds are, the goof you're compensating for will be fixed the next time you patch or upgrade the OS.

And if security is a concern, you can still avoid this by either implementing effective external security mechanisms (extended facls, role access databases, etc) or figuring out a way to achieve your desired end that doesn't pose a security risk. I don't know if Perl (to this day) actually "recommends" that you ever write any "suid" code (generally speaking, creating a script or binary that will be run as a different user - Set User ID - usually one with elevated system privileges) in Perl script unless you're protecting it by taking additional measures. Generally, if I "have" to write an "suid" program, I'll write it in "C" (like the one in our post on securing suid programs with a simple C wrapper, which doesn't even come close to the protection provided by the one-and-only Wietse Zweitze Venema's mother of all SUID C Wrappers.

Bottom line: It's up to you if you want to implement this sort of functionality. To my way of thinking, short of ensuring you don't get screwed on a work-for-hire at an institution in which you have no trust, there's really no reason to encrypt (I mean convert into binary ;) your Perl script. It's a pretty cool idea, but...

Cheers,

, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Sunday, April 12, 2009

An Easter Story: More ASCII Art For Linux And Unix

Hey There,

It's yet another holiday that we haven't ASCII'ed yet, and we've got another script to print a somewhat-tame Easter story, and picture, to your terminal. Yet again, I've gone back to Joan Stark's ASCII Art Gallery to find a really good picture. And I don't just pick her site because it comes up as Google search result number one for every query I run; there actually is an enormous quantity of high quality ASCII art there. Check it out if you still haven't.

If you're interested in any of our other ASCII art holiday script postings, just check out this page, which is a general search for ASCII Art on our site and you're sure to find most of them there.

If the pictorial representation of the script output below isn't large enough, click on it once to be smacked in the face with the HUMONGOUS version ;)

easter ASCII art

For this installment, I've attached two scripts to the post. The first one is the straight-up bash script. The second is the same script, except with all the spaces padded with "X"'s. If you're having problem maintaining the spaces when you copy/paste the original script, just copy/paste the padded one and then (in vi, or whatever your favorite editor is) substitute "X" with a space character for all occurrences. In vi, that would be an "ex" command like:

[esc]:g/X/s// /g

or

[esc]:%s// /g

Enjoy and Happy Easter :)


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License



REGULAR

#!/bin/bash

#
# easter.sh
#
# 2009 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

echo -en "...THE FIRST EASTER BUNNY... by Francine M. O'Connor\n (ASCII Art by joan stark)\n\n __ /^\\\\\n .' \ / :.\ This is the story of a long-eared rabbit\n / \ | :: \ who couldn't learn to do the bunny hop.\n / /. \ / ::: | His ears were floppy, his feet were sloppy,\n | |::. \ / :::'/ he'd hippity hop, then he'd trip and plop.\n | / \::. | / :::'/\n \`--\` \' \`~~~ ':'/\`\n / ( So this little rabbit developed the habit\n / 0 _ 0 \ of staying awake when the sun went down.\n \/ \_/ \/ He'd stay up all night, \n -== '.' | '.' ==- till the morning light, and \n /\ '-^-' /\ practice his hopping just outside of town.\n \ _ _ / \n .-\`-((\o/))-\`-. \n _ / //^\\ \ _ On the first Easter morn, \n.\"o\".( , .:::. , ).\"o\". just before dawn,\n|o o\\\\\ \:::::/ //o o| He was startled by a bright\n \ \\\\\ |:::::| // / and blinding light.\n \ \\\\\__/:::::\__// / And Jesus was there in the \n \ .:.\ \`':::'\` /.:. / shimmering glare,\n \':: |_ _| ::'/ smiling at that funny bunny's plight.\n jgs \`---\` \`\"\"\"\"\"\` \`---\`\n \n Don't worry, little lad, and don't be so sad, .-\"-.\n for humankind will celebrate this special day. .'=^=^='.\n You must bring the word to every beast and bird /=^=^=^=^=\\\\\n that I have risen and am in the world to stay. :^= HAPPY =^;\n |^ EASTER! ^|\n You should've seen that cottontail hop away, :^=^=^=^=^=^:\n feeling mighty proud to be the chosen one. \=^=^=^=^=/\n Though this story is quite old, it can now be retold \`.=^=^=.'\n to make little children smile on Easter morn. \`~~~\`\n"



PADDED
#!/bin/bash

#
# easter.padded.sh
#
# 2009 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

echo -en "...THEXFIRSTXEASTERXBUNNY...XbyXFrancineXM.XO'Connor\nXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX(ASCIIXArtXbyXjoanXstark)\n\nXXXXXX__XXXXXXXXXXXX/^\\\\\nXXXX.'XX\XXXXXXXXXX/X:.\XXXXXXXThisXisXtheXstoryXofXaXlong-earedXrabbit\nXXX/XXXXX\XXXXXXXXX|X::X\XXXXXXwhoXcouldn'tXlearnXtoXdoXtheXbunnyXhop.\nXX/XXX/.XX\XXXXXXX/X:::X|XXXXXXHisXearsXwereXfloppy,XhisXfeetXwereXsloppy,\nX|XXXX|::.X\XXXXX/X:::'/XXXXXXXhe'dXhippityXhop,XthenXhe'dXtripXandXplop.\nX|XXX/X\::.X|XXX/X:::'/\nX\`--\`XXX\'XX\`~~~X':'/\`\nXXXXXXXXX/XXXXXXXXX(XXXXXXXXXXXSoXthisXlittleXrabbitXdevelopedXtheXhabit\nXXXXXXXX/XXX0X_X0XXX\XXXXXXXXXXofXstayingXawakeXwhenXtheXsunXwentXdown.\nXXXXXX\/XXXXX\_/XXXXX\/XXXXXXXXHe'dXstayXupXallXnight,X\nXXXX-==X'.'XXX|XXX'.'X==-XXXXXXXXXtillXtheXmorningXlight,XandX\nXXXXXX/\XXXX'-^-'XXXX/\XXXXXXXXpracticeXhisXhoppingXjustXoutsideXofXtown.\nXXXXXXXX\XXX_XXX_XXX/XXXXXXXXXXXXX\nXXXXXXX.-\`-((\o/))-\`-.XXX\nXX_XXX/XXXXX//^\\XXXXXX\XXX_XXXXOnXtheXfirstXEasterXmorn,X\n.\"o\".(XXXX,X.:::.X,XXXX).\"o\".XXXXXjustXbeforeXdawn,\n|oXXo\\\\\XXXX\:::::/XXXX//oXXo|XXHeXwasXstartledXbyXaXbright\nX\XXXX\\\\\XXX|:::::|XXX//XXXX/XXXXXXandXblindingXlight.\nXX\XXXX\\\\\__/:::::\__//XXXX/XXXXAndXJesusXwasXthereXinXtheX\nXXX\X.:.\XX\`':::'\`XX/.:.X/XXXXXXXXshimmeringXglare,\nXXXX\'::X|_XXXXXXX_|X::'/XXXXXXsmilingXatXthatXfunnyXbunny'sXplight.\nXjgsX\`---\`X\`\"\"\"\"\"\`X\`---\`\nXXXXXXXXXXXXXXXXXXXXXXXXXX\nXXXXDon'tXworry,XlittleXlad,XandXdon'tXbeXsoXsad,XXXXXXXXXXX.-\"-.\nXXXXforXhumankindXwillXcelebrateXthisXspecialXday.XXXXXXXX.'=^=^='.\nXXXXYouXmustXbringXtheXwordXtoXeveryXbeastXandXbirdXXXXXX/=^=^=^=^=\\\\\nXXXXthatXIXhaveXrisenXandXamXinXtheXworldXtoXstay.XXXXXX:^=XHAPPYX=^;\nXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX|^XEASTER!X^|\nXXXXYouXshould'veXseenXthatXcottontailXhopXaway,XXXXXXXX:^=^=^=^=^=^:\nXXXXfeelingXmightyXproudXtoXbeXtheXchosenXone.XXXXXXXXXXX\=^=^=^=^=/\nXXXXThoughXthisXstoryXisXquiteXold,XitXcanXnowXbeXretoldXX\`.=^=^=.'\nXXXXtoXmakeXlittleXchildrenXsmileXonXEasterXmorn.XXXXXXXXXXX\`~~~\`\n"




, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Monday, March 30, 2009

Idle Process Time On Linux And Unix: How To Find It Again

Hey There,

In our final installment on "finding a process's idle time on Linux or Unix," (last touched upon in our post on echo debugging) we looked at whole lot of ways one could go wrong trying to find the idle time of a process on Linux or Unix. This post is a little more upbeat ;)

All of the previous issues with "who -T" have been worked out. Basically, this means that I've gone over it every which way and could find no good reason to use it, as opposed to "w." Of course, in our particular case, we are looking, specifically, for a single process's idle time (as opposed to a user process's idle time; reported by "who -T"). And, although it's a little bit of a pain (initially), short of programming in C (accessing the pstatus struct on Solaris, to be exact - the name and location may vary from distro to distro of proprietary, or free, Unix and/or Linux), linking the pty information from ps with the "idle time" information from w, seems to be the best way to get this information. So far, it's the most efficient way I could find using simple bash scripting.

Attached to today's post is the final "blog" version of this script. It comes with a few notes (possibly of caution) and may need to be modified for your system/OS (There's the first one ;)

The script runs very simply, and you only need to supply it with a PID. You can, optionally, supply a username as a second argument:

host # ./rip 17787

if you just run it with no arguments, you'll get a usage screen, which may or may not help ;)

host # ./rip
Usage: ./rip PID [user]
User defaults to the value
of $LOGNAME if not specified


Please see our previous post on echo debugging this script for more detailed sample output.

I hope you find some good use for this script, and, without further ado, the oft-dreaded notations of explanation ;)

1. This script has been rewritten to be self-contained. Please see the bottom line for any substitute command you may want to use. Actually, making this command a variable might be a good idea. Just call me "Lazy" ;)

2. You can remove the explicit PATH definition if you like. I put it in there specifically to make sure that the "which ps" variable assignment didn't accidentally grab /usr/ucb/ps on Solaris

3. You can comment out the SIGNAL variable as well, since plain old kill is a sig TERM or 15. The only real reason to set this would be if you wanted to always run kill with a different signal (like SIGKILL,, or 9, for example)

4. I changed the minimum idle time to 30 minutes from 45 (in the previous revisions)

5. All variables appearing in this work are fictitious. Any resemblance to real variables, living or dead, is purely coincidental ;)

Cheers,


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

#
# rip - Kill any processes that we know have been idle for more than 30 minutes
#
# 2009 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

PATH="/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin"
procowner="rftprocowner"
prog="\./rft"
sed=`which sed`
awk=`which awk`
ps=`which ps`
grep=`which grep`
kill=`which kill`
signal="-15"

while read a b c d
do
pid=$b
pid_not_var=$(echo $pid | $grep [A-z])

if [[ ! -z $pid_not_var ]]
then
echo "pid $1 contains non-numeric characters!"
continue
fi

pid="$b"
pid_pty="$c"

if [[ -z "$pid_pty" ]]
then
echo "pid $pid is either non-existent, not owned by \"$procowner\" or not attached to a p/tty!"
continue
elif [[ "$pid_pty" = "?" || "$pid_pty" = "console" ]]
then
echo "pid $pid is not attached to a pty!" # kill OR LEAVE IT?
else
pty_num=$(echo "$pid_pty"|$sed 's/^[^\/]*\///')
fi

proc_time=$(w -sh $procowner|grep $pty_num|grep -v grep|$awk '{if ( $2 == '"$pty_num"' && NF == 4 ) print $3;else if ( $2 == '"$pty_num"' && NF == 3) print "0"}')

proc_is_num=$(echo $proc_time | $grep [A-z])
if [[ ! -z $proc_is_num ]]
then
unset proc_time
fi

ext_proc_time=$(echo $proc_is_num | $grep [A-z])

if [[ ! -z "$ext_proc_time" && -z "$proc_time" ]]
then
echo "killing $pid - $d Up Over 24 Hours: $ext_proc_time $proc_time"
### $kill $signal $pid
elif [[ "$proc_time" = "0" ]]
then
:
else
proc_idle_time=$(echo $proc_time|$grep -v "[:]")
if [[ -z $proc_idle_time || $proc_idle_time -gt 30 ]]
then
echo "killing $pid - $d Up More Than 30 Minutes: $ext_proc_time $proc_time $proc_idle_time"
### $kill $signal $pid
fi
fi
done <<< "`$ps -fu $procowner -o procowner,pid,tty,comm|$grep "$prog"|$grep -v grep`"


, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Thursday, March 26, 2009

Simple, But Effective. Echo Debugging On Linux And Unix

Hey there,

I took some time and did some simple "echo debugging" and found that the warning I issued about yesterday's script to find a process's idle time was completely backward. Fortunately, it turns out that my mistaken judgement meant that I had a lot less to worry about, in terms of damage control, from the flaw I perceived in my script (I'm not saying there aren't others, of course ;)

It turns out that the problem was not that the script would sometimes consider an active process, that didn't have an idle column value in the "w -s" output, to be idle and worth terminating. The actual problem was that it would consider processes that had been up for more than a day to be active and not worth terminating. This was a much better situation. At least I wouldn't be killing off active processes!

A sample of just using the DEBUG statements I put in yesterday's script pointed out the error very obviously, like so:

DEBUG::::: PIDTTY 6892 pts/119
DEBUG::::: W user1 119 2days ./program
DEBUG::::: LONGTIME TIME
PID 6892 is OK - Not Idle At All - Remove this message!
------------------------
DEBUG::::: PIDTTY 581 pts/232
DEBUG::::: W user1 232 2days ./program
DEBUG::::: LONGTIME TIME
PID 581 is OK - Not Idle At All - Remove this message!
-----------------------------------


And, yes, I felt like a complete moron when I finally took a second to actually look at the output ;) It's a amazing what a few simple echo statements in a script can tell about what problem's it has :)

From that point, I found several other issues and worked on them accordingly:

1. ISSUE WITH IDLE ALPHA DAYS NOTATION = FIX BY CHECKING FOR NON-NUMERIC TYPES

2. ISSUE WITH NO-IDLE MISSING COLUMN = FIX BY SETTING EMPTY VALUE TO NULL PADDED

3. ISSUE WITH MISSING COLUMN ERROR OUTPUT = FIX BY CHECKING COLUMN COUNT IN TIME

4. MUCH BETTER - "NOT IDLE AT ALL" EXCEPTION NEVER CAUGHT - UNNECESSARY NOW - REMOVED

5. REWORKED TIME HANDLING AND SET TO AMBIGUOUS ALPHA MATCH


Pardon my hysterical notes ;) Most of my problem stemmed from the fact that I switched from full-fledged "w" to "w -s" and made some mistakes in updating the relevant columns that I needed to assign to variables.

I should note that I also considered using "who -T" to get around the one time-stealer in this script. Although it did bring the script down to under a second (processing approximately 100 records), "who" only reports on the "user process." This is a huge consideration, since the "user process" can (and usually is) the parent process of the process you want to check the idle time on. I ultimately decided to stick with "w" since using "who" would mean I'd have to check the parent process, cross reference that with the grep output associated with the pty and then end up back at "w" again to get the process's idle time. A lot of extra work for a lot of extra uncertainty. I didn't want to end up in a situation where the "user process" was idle because the user kicked off a script that ran for 6 hours and then terminate the user's main process (which would kill the kids) based on the idle time of the user's session. Sometimes, lack of precision like that can cause you headaches you never imagined you could have ;)

As you can see below, the updates weren't all that impressive, but I did get the execution time down to 30 seconds from 2 minutes. The only way I could get it lower (that I've figure out so far ;) was to compromise the integrity of the script and remove the one awk statement that was holding it back. Notice the last step I took, just to see what would happen, that proved the awk if/else conditional in the script was responsible for a majority of the execution time:

TRIMMED CODE - REMOVED DEBUG AND UNNECESSARY ECHO STATEMENTS - USING BASH TEST AND OPERATORS
OLD SCRIPT EXECUTION TIME FOR 178 PROCS = 1m27.430s
NEW SCRIPT EXECUTION TIME FOR 179 PROCS = 0m56.517s
NEW SCRIPT EXECUTION TIME FOR 110 PROCS = 0m48.940s
SELF-CONTAINED SCRIPT EXECUTION TIME FOR 111 PROCS = 0m51.048s
ADDED TTY TO PS SCRIPT EXECUTION TIME FOR 101 PROCS = 0m29.703s
REMOVING AWK TTY STATEMENT SCRIPT EXECUTION TIME FOR 100 PROCS = 0m29.991s
ADDED ?, console and "continue" SCRIPT EXECUTION TIME FOR 102 PROCS = 0m33.018s
REMOVED W HEADING (REM SED) AND EXPLICIT USER SCRIPT EXECUTION TIME FOR 101 PROCS = 0m33.382s
TEST - HARDCODED UPTIME AND REMOVED AWK STATEMENT - SCRIPT EXECUTION TIME FOR 170 PROCS = 0m8.512s!!!!!!!!!!!


I'm going to work on it some more, because I believe it can be improved tremendously, but - to satisfy any curiosity, here's some of the mid-work that fixed that issue and made the bash script report correctly. I'll post the one with the fixes noted above (and more, I'm sure ;) once I've thoroughly tested them and removed a lot of the redundancy in this script. Redundancy really gets under my skin. I mean it; redundancy really irritates me. Plus, I don't much care for redundancy ;)

Cheers,


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

#
# rip - Kill any processes that we know have been idle for more than 45 minutes - v2-alpha
#
# 2009 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

if [[ $# -lt 1 ]]
then
echo "Usage: $0 PID [user]"
echo "User defaults to the value"
echo "of \$LOGNAME if not specified"
exit 1
fi

PID=$1
ISITAPID=$(echo $PID | grep [A-z])

if [[ ! -z $ISITAPID ]]
then
echo "PID $1 contains non-numeric characters!"
echo "-----------------------------------"
exit 2
fi

PID="$1"
USER=${2:-$LOGNAME}

PIDTTY=$(/usr/bin/ps -fu $USER -o pid,tty |/usr/bin/grep -w $PID|/usr/bin/grep -v grep)

echo DEBUG::::: PIDTTY $PIDTTY

if [[ -z "$PIDTTY" ]]
then
echo "PID $PID is either non-existent, not owned by \"$USER\" or not attached to a p/tty!"
echo "-----------------------------------"
exit 3
else
TTYNUMBER=$(echo "$PIDTTY"|/usr/bin/sed '/TT/d'|/usr/bin/awk -F"/" '{print $2}')
fi

if [[ -z "$TTYNUMBER" ]]
then
echo "PID $PID is not attached to a p/tty!"
echo "KILL OR NOT-----------------------------------"
exit 4
fi

echo DEBUG::::: W $(w -s|/usr/bin/sed 1d|/usr/bin//awk '{if ( $2 == '"$TTYNUMBER"' ) print $0}')

TIME=$(w -s|/usr/bin/sed 1d|/usr/bin/awk '{if ( $2 == '"$TTYNUMBER"' && NF == 4 ) print $3;else if ( $2 == '"$TTYNUMBER"' && NF == 3) print "0"}')
#TIME=$(w -s|/usr/bin/sed 1d|/usr/bin/awk '{if ( $2 == '"$TTYNUMBER"' ) print $3}')
#WCOLUMNS=$(w -s|/usr/bin/sed 1d|/usr/bin/awk '{if ( NF == 4 ) print "4";else print "3"}')

ISITANUMBER=$(echo $TIME | grep [A-z])
if [[ ! -z $ISITANUMBER ]]
then
unset TIME
fi

LONGTIME=$(echo $ISITANUMBER | grep [A-z])

echo DEBUG::::: LONGTIME $LONGTIME TIME $TIME

if [[ ! -z "$LONGTIME" && -z "$TIME" ]]
then
echo "PID $PID is ancient - Idle for $LONGTIME... Killing $PID"
# KILLKILLKILL
elif [[ "$TIME" = "0" ]]
then
echo "PID $PID is OK - Not Idle At All - Remove this message!"
else
TIMEIDLE=$(echo $TIME|grep -v "[:]")
echo DEBUG::::: TIME $TIME
if [[ -z $TIMEIDLE ]]
then
echo "PID $PID has been idle way too long - $LONGTIME $TIME so far... Killing $PID"
# KILLKILLKILL
elif [[ $TIMEIDLE -gt 45 ]]
then
echo "PID $PID has been idle too long - $TIMEIDLE minutes so far... Killing $PID"
# KILLKILLKILL
else
echo "PID $PID is OK - Only idle for $TIME minute(s) - Remove this message!"
fi
fi
echo "-----------------------------------"


, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Wednesday, March 25, 2009

Finding A Process's Idle Time On Linux And Unix

Hey There,

Hopefully yesterday's rant on the simplicity of complexity wasn't too much of a bitter pill. If it was, here's hoping you didn't swallow it ;)

Today, I finally found some time to make a little headway on this project (which should be a lot simpler than it is). Basically, what I'm looking to do is create a way to track specific process's idle times at any given point in time on any given Linux or Unix system. As I mentioned yesterday, there are c structures in Solaris' /proc/PID/status C data files (for one example), but that's just another thing that ended up frustrating me more. As I noted, parts of the OS that are included, should be available for use. The structure is used by the OS, in some shape or fashion, to determine idle times (as we'll see below), but no specific "tool" exists to do what i wanted. Of course, this is limited "to my knowledge." If anyone out there knows of a standard program or command that's managed to elude me, please feel free to email me and tell me all about. I promise to not get offended if you feel the need to belittle me for not having the common sense to look for it where it was at in the first place ;)

Attached to today's post is a rough-draft bash script that attempts to grab a process's idle time. It won't work in all instances, although I've tried to capture as many of those instances as possible. The one big gotcha in this whole mess is that you can't take the output of ps and directly retrieve the idle time for a process from the listing, even if you do your own formatting (I wrote this on Solaris 10 and looked at SUSE Linux 9, but found no love :) Instead, I found that I needed to run ps, extract the pty associated with the process from that (if it existed - which is an exception the script catches) and then use either who or w to retrieve the idle time associate with the pty.

See what I mean? Shouldn't it be a little bit less of a hassle than that?

Okay. I'll admit, if it was, I wouldn't be having half the fun I'm having now trying to script it all out for myself ;) So far, what I've put together works fairly well, although I'm not 100% certain that it's bullet-proof so I would recommend that you leave the "business end" of the code commented out (The stuff that performs unforgivable actions, like killing ;). I have a hard time reproducing it, but I can swear that this code will (every once in a good while) determine that a process that hasn't been idle at all (which removes a column from the "w -s" output) has been idle too long. I'm still working on that part and welcome any suggestions regarding the script, how to make it better, why I'm doing everything the wrong (and/or hard) way when I don't need to and any other constructive criticism :)

The script runs very simply, and you only need to supply it with a PID. You can, optionally supply a username as a second argument:

host # ./rip 17787

if you just run it with no arguments, you'll get a usage screen, which may or may not help ;)

host # ./rip
Usage: ./rip PID [user]
User defaults to the value
of $LOGNAME if not specified


and the following is a sample of the output you might get on a specific run. Here, I've written a command line while loop from a pipe to barbarically hammer out multiple instances at a time ;)

host # time ps -ef|grep "[b]ash"|awk '{print $2}'|while read x;do ./rip $x;done
PID 2664 is not attached to a p/tty!
-----------------------------------
PID 10700 is either non-existent, not owned by "root" or not attached to a p/tty!
-----------------------------------
PID 10855 is OK - Not Idle At All - Remove this message!
-----------------------------------
PID 23217 is OK - Not Idle At All - Remove this message!
-----------------------------------
PID 14730 is either non-existent, not owned by "root" or not attached to a p/tty!
-----------------------------------


Here's another example. This time you'll see what you get if you try to run the script specifying a user other than the user that owns the processes or, in this case, a completely bogus user. This "test" in the script really isn't necessary and I only included it as feeble attempt at damage control. Feel free to remove it if you like:

host # time ps -ef|grep "[b]ash"|awk '{print $2}'|while read x;do ./rip $x joeUser;done
ps: unknown user joeUser
PID 2664 is either non-existent, not owned by "joeUser" or not attached to a p/tty!
-----------------------------------
ps: unknown user joeUser
PID 23633 is either non-existent, not owned by "joeUser" or not attached to a p/tty!
-----------------------------------
ps: unknown user joeUser
PID 10700 is either non-existent, not owned by "joeUser" or not attached to a p/tty!
-----------------------------------
ps: unknown user joeUser
PID 10855 is either non-existent, not owned by "joeUser" or not attached to a p/tty!
-----------------------------------
ps: unknown user joeUser
PID 14730 is either non-existent, not owned by "joeUser" or not attached to a p/tty!
-----------------------------------


I hope you find some good use for this script!

NOTE: Please keep in mind the caveat noted above regarding the sometimes-false-positive I believe this script returns under certain circumstances when it decides a non-idle process (with nothing displayed in the idle column from "w -s" output) has been idle too long! It may never happen again and I may have been seeing spots. Just want to keep you in a "safe" mindset, just in case I'm not completely insane ;)

Cheers,


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

#
# rip - Kill any processes that we know have been idle for more than 45 minutes
#
# 2009 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

if [ $# -lt 1 ]
then
echo "Usage: $0 PID [user]"
echo "User defaults to the value"
echo "of \$LOGNAME if not specified"
exit 1
fi

PID=$1
ISITAPID=$(echo $PID | grep [A-z])

if [ ! -z $ISITAPID ]
then
echo "PID $1 contains non-numeric characters!"
echo "-----------------------------------"
exit 2
fi

PID="$1"
USER=${2:-$LOGNAME}

PIDTTY=$(/usr/bin/ps -fu $USER -o pid,tty |/usr/bin/grep -w $PID|/usr/bin/grep -v grep)

#echo DEBUG::::: PIDTTY $PIDTTY

if [ -z "$PIDTTY" ]
then
echo "PID $PID is either non-existent, not owned by \"$USER\" or not attached to a p/tty!"
echo "-----------------------------------"
exit 3
else
TTYNUMBER=$(echo "$PIDTTY"|/usr/bin/sed '/TT/d'|/usr/bin/awk -F"/" '{print $2}')
fi

if [ -z "$TTYNUMBER" ]
then
echo "PID $PID is not attached to a p/tty!"
echo "-----------------------------------"
exit 4
fi

#echo DEBUG::::: W $(w -s|/usr/bin/sed 1d|/usr/bin//awk '{if ( $2 == '"$TTYNUMBER"' ) print $0}')

TIME=$(w -s|/usr/bin/sed 1d|/usr/bin/awk '{if ( $2 == '"$TTYNUMBER"' ) print $3}')

ISITANUMBER=$(echo $TIME | grep [A-z])
if [ ! -z $ISITANUMBER ]
then
unset TIME
fi

LONGTIME=$(echo $TIME | grep [A-z])

#echo DEBUG::::: LONGTIME $LONGTIME TIME $TIME

if [ -z "$LONGTIME" -a -z "$TIME" ]
then
echo "PID $PID is OK - Not Idle At All $TIME - Remove this message!"
elif [ ! -z $LONGTIME ]
then
echo "PID $PID is ancient - Idle for $TIME... Killing $PID"
# DO_WHAT_YOU_HAVE_TO_DO_TO_THE_PID_HERE
else
TIMEIDLE=$(echo $TIME|grep -v "[:]")
# echo DEBUG::::: TIME $TIME
if [ -z $TIMEIDLE ]
then
echo "PID $PID has been idle way too long - $LONGTIME $TIME so far... Killing $PID"
# DO_WHAT_YOU_HAVE_TO_DO_TO_THE_PID_HERE
elif [ $TIMEIDLE -gt 45 ]
then
echo "PID $PID has been idle too long - $TIMEIDLE minutes so far... Killing $PID"
# DO_WHAT_YOU_HAVE_TO_DO_TO_THE_PID_HERE
else
echo "PID $PID is OK - Only idle for $TIME minute(s) - Remove this message!"
fi
fi
echo "-----------------------------------"


, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.