A set of ebooks on the history of psychology, buddhism and various other similar topics. Written by George Boeree, a professor at Shippensburg University, these is a great collection.

Python is easy to learn, and more powerful than Bash. I wasn’t supposed to tell you this–it’s supposed to be a secret. Anything more than a few lines of Bash could be done better in Python. Python is often just as portable as Bash too. Off the top of my head, I can’t think of any *NIX operating systems, that don’t include Python. Even IRIX has Python installed. If you can write a function in Bash, or even piece together a few commands into a script and make it executable, then you can learn Python.

In fact, at my own company I'm developing a pretty extensive automated test suite (written in Python of course, and using nose) that verifies all the steps we go through whenever we deploy a server or a network device. It's very satisfying to see those dots and have a total of N passed tests and 0 failed tests, with N increasing daily. Automated tests for sysadmin tasks is an area little explored, so there's lots of potential for cool stuff to happen. If you're doing something similar and have ideas to share, please leave a comment.

This is a follow-up to my previous post on writing automated tests for sysadmin-related checklists. That post seems to have struck a chord, judging by the comments it generated.

“The increased density of the blades allows us to add a significantly higher number of processors into a confined physical footprint while still managing within tight power consumption and cooling requirements,” Adam Shand, the acting CTO of Weta Digital, told VFX World. That kind of density creates a cooling challenge. “One of our most important decisions was to invest in water-cooled racks from Rittal, which allows us to precisely control the amount of cooling that a specific rack requires,” said Shand. “The tangible benefit is that we can manage our energy costs at a very granular level. Instead of the traditional and less responsive method of eliminating hot spots around cooling, these racks self-regulate the amount of cooling they require in real time, adjusting the waterflow and fan speeds as required.” Shand says the New Zealand climate allows Weta Digital to use free cooling for a large portion of the year.

Weta Digital has completed a new multimillion dollar datacentre in Miramar that it says will strengthen its position as a world leader in special effects for film and television. The company is the first in the world to use "double density" blade serves made by Hewlett- Packard, that combine two powerful servers in a single pizza-box sized enclosure, saving space and energy. The 1280 blades will be used to produce effects for films that include Avatar, TinTin and The Lovely Bones. Acting chief technology officer Adam Shand says one of the most important decisions building the datacentre was to invest in an efficient, self- regulating water-cooling system supplied by German firm Rittal.

Mac OS X makes extensive use of a variety of event notification mechanisms, as described here. One such mechanism is documented in the notify(3) man pages, and it is these types of libnotify notifications that are the focus of this post. Notifications can be posted and observed in a global namespace across all processes (with optional access control), and in general allow a process to respond to events that occur outside the purview of the process.

One simple example would be the shutdown or restart event. When the user clicks Restart, loginwindow posts a notification called com.apple.loginwindow.restartinitiated. Various other processes who have registered to receive this notification will now know that a restart is coming (soon!). A lot of stuff begins to happen to prepare the machine for a reboot. Getting that event notification from userspace out in a timely fashion to all the various custodians who need to do work is key to the whole operation.

I see two primary use cases for such a notification system: 1) You are authoring software that creates events that others (perhaps even other parts of your own software, running in a different context) need to respond to, or 2) you wish to respond to a notification for a certain event that you know is being posted by someone else.

The first case is succinctly demonstrated by the example in the man page for notifyutil. To expand on it just a bit, and more clearly demonstrate the common case of there being some indeterminate time delta between registering for a notification and receiving it, consider this example:

Open a Terminal window. Execute:

date; notifyutil -t -1 some.pants ; say 'zomg'

Open another Terminal window, but keep the first one in view. In your mind, count to ‘d’, and then execute:

notifyutil -p some.pants

In the first window, you see that a notification for some.pants was received roughly ‘d’ seconds after the registration occured, and if your audio volume is up, you hear that Mac OS X does not know how to pronounce ‘zomg’.

The second case of responding to notifications generated by others is less obvious, since in normal operation, all of these notifications are invisible to the user. You have to know the name of a notification to register for it. The fun begins when you remember to ask the question “how do I find out what notifications are being posted?”. The answer might be obvious to anyone who read the man page! notifyd is our friend, buddy. As all this is open source, we could confirm our suspicions that notifyd probably logs to ASL, and sure enough, in notify_proc.c, we see :

log_message(ASL_LEVEL_DEBUG, "__notify_server_post %s", name);

Ok! So all we need to do is ask ASL to show us debug messages for notifyd.

sudo syslog -c notifyd -d

At this point, the messages are theoretically being sent. However, the default data store (i.e. that which you can examine with the Console app) still isn’t listening, so we also need:

sudo syslog -c syslogd -d

Next you should make a new log database query using Console:

Now you can just sit back and watch the notifications roll in! For example, if I choose Restart from the Apple menu:

__notify_server_post com.apple.loginwindow.likelyShutdown
__notify_server_post com.apple.loginwindow.restartinitiated

and then a few seconds later, after I click Cancel:

__notify_server_post com.apple.loginwindow.logoutcancelled

At this point, you should be armed with enough information to find out what events are flying around, and then attach an action of your choosing to events of interest. Have fun!

Check your battery’s health with System Profiler
Mac Tip #354, 03 September 2008

· A new Mac laptop, such as a MacBook Pro or a MacBook, can run for maybe 3 or 4 hours on battery. After a while though that fully charged battery will last for only 2 hours, then 1, until eventually it just dies.

That’s the way of all rechargeable batteries — they have a certain number of ‘cycles’ in them. As the months go by the battery’s capacity gradually declines.

Back in Battery check, Mac Tip #260/13-Sept-2006, I explained how to use the button on the battery itself to check the charge. System Profiler gives you more detailed information: the number of charging cycles, the make of the battery, the capacity and so on.

Open up Applications > Utilities > System Profiler. A window appears showing information about your Mac.

Click on Power, under Hardware in the list of Contents on the left of the System Profiler window. Battery and other Power Settings information is displayed in the right-hand pane.

The Model Information section shows the battery’s serial number, manufacturer and other data about its manufacture.

The Charge Information and Health Information sections are the ones to monitor.

A charge cycle is when the battery is run flat and then recharged to full. If you run the battery partly down and recharge it, that counts as part of one cycle.

The battery’s capacity to store a charge is measure in Milliamps Per Hour (mAh).

· Click the screenshots for larger versions.

In the first screenshot I can see that my brand new battery has been discharged and recharged 3 times (Cycle count: 3). It’s fully charged and has 5417 mAh capacity remaining. Its Full charge capacity is 5454 mAh. The Condition of the battery is listed as: Good, under Health Information.

The 3rd party battery (ie, not from Apple) in the second screenshot has been discharged and recharged 49 times (Cycle count: 49). It’s fully charged but has only 1673 mAh capacity remaining. Its Full charge capacity is only 1680 mAh. The Condition of the battery is listed as: Check battery, under Health Information.

This battery seems to have a problem. It hasn’t been used very much but its capacity is very low. I suspect a problem with it and have contacted the supplier for support.

If you use a Mac laptop, take a look at System Profiler to give it a health check.

Update: I’m using Mac OS X 10.5.4, Leopard. Older versions of the Operating System may have less Power information, or maybe none at all.

In this article, we’ll take a look at using Leopard’s launchd and rsync to develop a automated data backup methodology. In addition to rsync and launchd, we will also integrate Bernhard Baehr’s SleepWatcher utility.

In our environment, we use NetApp filers to host our network shares and are using CIFS network shares. However, with NetApp and Mac OS X, those file shares could be NFS, just as well. This will also work with your AFP shares.

Components

• launchd

  launchd is Apple’s (not-so-new) system startup program. launchd is more powerful than older programs such as init and cron, since it can not only start and stop programs but can also be triggered by other events, such as availability of network shares etc, which we will use in this setup.

• rsync

  Most people reading this article will already know what rsync is. rsync is a program that behaves in much the same way that rcp does, but has many more options and uses the rsync remote-update protocol to greatly speed up file transfers when the destination file is being updated. In this setup, rsync does the heavy lifting of actually performing the backups.

• SleepWatcher

  is a niftly command line tool (daemon) developed by Bernhard Baehr for Mac OS X that monitors sleep, wakeup and idleness of a Mac. It can be used to execute a Unix command when the Mac or the display of the Mac goes to sleep mode or wakes up or after a given time without user interaction. We will be using SleepWatcher to do just that - to automatically mount and unmount network shares.

1. SleepWatcher
   As mentioned above, SleepWatcher monitors the sleep and wake states of a Mac (among other things). We’ll be using both of the SleepWatcher’s components, the SleepWatcher command line tool and the SleepWatcher StartupItem. As part of the tool’s functionality, SleepWatcher employs two hidden files, .wakeup and .sleep in the user’s home directory to execute user specific commands when a Mac wakes up from sleep or goes to sleep, as the case may be.

   First, we need a wakeup script to automatically mount our network shares when the laptop/Mac wakes up from sleep.

   /Users/username/.wakeup
   

   #!/bin/bash
   
   filer="bitbucket.example.com";
   test "$SHELLOPTS" && shopt -s xpg_echo
   
   function checkWorkNetwork() {
     # sleep for a bit, wait for network to show up
     sleep 30
     count=$(/sbin/ping -c 3 -o $filer | grep "received" | awk -F',' '{ print $2 }' | awk '{ print $1 }')
     if [ $count -eq 0 ]; then
       # 100% failed - try one more time after sleeping for 30 secs
       sleep 30
       count=$(/sbin/ping -c 3 -o $filer | grep "received" | awk -F',' '{ print $2 }' | awk '{ print $1 }')
       if [ $count -eq 0 ]; then
         /usr/bin/logger -t SleepWatcher:wakeup "filer not reachable...Exiting".
         exit;
       fi
     else
       /usr/bin/logger -t SleepWatcher:wakeup "filer is reachable...Continuing".
     fi
   }
   
   function doMounts {
     vol=$1;
     mnt_cnt=$(/sbin/mount | grep -ic "/Volumes/$vol")
     if [ $mnt_cnt -le 0 ]; then
       /usr/bin/osascript <<EOT
       tell application "Finder" to mount volume "cifs://$filer/$vol"
   EOT
     fi
   }
   
   checkWorkNetwork
   doMounts Backups
   
   exit 0
   
   

   In the above wakeup script, first we check to see if we are on our work network using the function
   checkWorkNetwork . and mount the network resources. This function can easily be modified to adapt to a home network. At home, if we are on a wireless network, a function like the following may be used.

   function checknMountHomeNetwork {
    # sleep for a bit, wait for network to show up
    sleep 60
    homelan=$(/usr/sbin/system_profiler SPAirPortDataType | awk -F': ' '/Current Wireless Network/{print $2}')
   
    if [ "$homelan" == "JewelThief-Fake" ]; then
           /usr/bin/osascript <<EOT
           tell application "Finder" to mount volume "cifs://192.168.1.3/Documents"
   EOT
           /usr/bin/logger -t SleepWatcher:wakeup "Mounted Home Volumes...Exiting"
           exit 0
    else
           /usr/bin/logger -t SleepWatcher:wakeup "Cannot ascertain home network...Exiting"
    fi
   }
   
   

   Next, we need a sleep script to trigger the stoppage of any running rsync backups and unmount the network shares.

   The sleep script needs to execute and finish within 15 seconds. After this timeout, the system forces the sleep mode and your script may not finish properly.

   /Users/username/.sleep
   

   #!/bin/bash
   
   test "$SHELLOPTS" && shopt -s xpg_echo
   
   function doUmounts {
           /usr/bin/logger -t SleepWatcher:sleep "Going to sleep...unmounting network volumes"
           /usr/bin/osascript -e 'tell application "Finder" to eject (every disk whose local volume is false)'
   }
   
   function kill_rsync {
           /usr/bin/logger -t SleepWatcher:sleep "Stopping rsync"
           pid=`ps auxw | grep rsync | grep -v grep | awk '{print $2}'`
           while [ ! -z "$pid" ]; do
                   echo -n "."
                   for pid in $pid
                   do
                           kill -9 $pid
                   done
                   sleep 3
                   pid=`ps auxw | grep rsync | grep -v grep | awk '{print $2}'`
           done
   }
   
   kill_rsync
   doUmounts
   
   exit 0
   
   

   In the above sleep script, we first kill off any running rsync jobs using the kill_rsync function and unmount the network shares using the doUmounts function.

2. Now that we have the wakeup and sleep scripts, our network shares will be automatically mounted and unmounted. We will now need a way to detect these network shares transitions and periodically launch our backup scripts. For this, we need a rsync backup script and a launchd hook to manage the automatic launch of this script.

   Let’s first take a look at the rsync script itself. In our example, the script is installed in /usr/local/.

3. /usr/local/laptop_rsync.sh
   

   #!/bin/sh
   
   PROG=$0
   SRC="$HOME/Documents/"
   DST="/Volumes/Backups/$USER/"
   CMD="/usr/bin/rsync -q  --numeric-ids -E  -axzS --exclude-from=$HOME/.rsync_excludes.txt $SRC $DST"
   
   if [ ! -r $SRC ]; then
    /usr/bin/logger -t $PROG "Source $SRC not readable - Cannot start the sync process"
    exit;
   fi
   
   if [ ! -w $DST ]; then
    /usr/bin/logger -t $PROG "Destination $DST not writeable - Cannot start the sync process"
    exit;
   fi
   
   /usr/bin/logger -t $PROG "Start rsync"
   $CMD
   /usr/bin/logger -t $PROG "End rsync"
   
   exit 0
   
   

   In this script, we are using rsync to synchronize the Mac user’s Documents/ directory to a network share. We are also using a exclude list here called .ib_backup_excludes.txt to prevent any unwanted/personal files and directories from being synchronized to the network share.

   $HOME/.rsync_excludes.txt
/tmp/*
/Network/*
/cores/*
*/.Trash
/afs/*
/automount/*
/private/tmp/*
/private/var/run/*
/private/var/spool/postfix/*
/private/var/vm/*
/Previous Systems.localized
.Spotlight-*/
._*
Personal
personal


   Most importantly, if we create a directory called Personal or personal within our Documents/ directory and place all of our personal files, they will not be synchronized to our network files share.

4. Lastly, we need way to automatically run this rsync script (and not run it when the network shares have disappeared). Here’s where Apple’s launchd comes in. We will give control of the above laptop_rsync.sh script to launchd by creating a launchd agent file. From the launchd.plist man page, the agent files can be deployed in several locations, depending on whether this is done by a regular user or by an administrator. In our example, I am using the administrator location and installing the launchd agent file in /Library/LaunchAgents/.

   The rsync launchd agent file is /Library/LaunchAgents/name.rajeev.laptop_rsync.plist

   <?xml version="1.0" encoding="UTF-8"?>
   <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
   <plist version="1.0">
   <dict>
           <key>KeepAlive</key>
           <dict>
                   <key>PathState</key>
                   <dict>
                           <key>/Volumes/Documents/</key>
                           <true/>
                   </dict>
           </dict>
           <key>Label</key>
           <string>name.rajeev.laptop_rsync</string>
           <key>Program</key>
           <string>/usr/local/laptop_rsync.sh</string>
           <key>StartInterval</key>
           <integer>3600</integer>
           <key>ThrottleInterval</key>
           <integer>3600</integer>
           <key>WorkingDirectory</key>
           <string>/usr/local/</string>
   </dict>
   </plist>
   
   

   In the above plist agent file, note that we are using the key PathState. launchd monitors for this path specified by this key to become available and launches the script once the path becomes available. In our example, once this path becomes available (via the wakeup script which is in turn launched by SleepWatcher), the rsync backup job is kicked off. Also note that we are using a ThrottleInterval to prevent launchd from starting the rsync script too fast.

   This agent file can itself be loaded upon restart by using the $HOME/.launchd.conf or /etc/launchd.conf. The command to load and unload this agent into launchd is below:

   • launchd load /Library/LaunchAgents/name.rajeev.laptop_rsync.plist. This will load this agent into launchd
   • launchd unload /Library/LaunchAgents/name.rajeev.laptop_rsync.plist. This will unload this agent into launchd

Evernote Evernote Genre: Productivity Album Price: Free Release Date: 29 May 2008

Great Idea, Great App  I've been heavily using Evernote for the last few days and I've been very impressed. Having the ability to access text notes, voice notes and pictures seamlessly between my Mac, my iPhone and my browser is a godsend. I've actually ditched the builtin notes program on the iPhone and in Mail.app in favour of Evernote. It's not perfect yet and there are some rough edges but so far they are all minor and I'm very pleased. Great work! © Evernote 2008

How to use Bluetooth to connect a Wiimote to a PC and use it as a remote control. Cool!

How on earth did I miss this? Anyway, hilarious ...

It’s “hack some shit up” Friday. Everybody ready?

Have a Time Capsule or Mac OS X Server configured to provide network time machine backups? Of course you do! Let’s examine the client / server interaction.

The Time Machine preference pane provides a list of available backup volumes. This list includes local volumes as well as network volumes. Mac OS X makes extensive use of multicast dns for service discovery, so let’s check there first. Get yourself a copy of Bonjour Browser, then fire it up and look under ‘local’ for things that seem interesting. Ooo there’s one called “adisk”, let’s check in there…

From http://www.dns-sd.org/ServiceTypes.html we see:

adisk          Automatic Disk Discovery
                Bob Bradley <bradley at apple.com>
                Defined TXT keys: sys, dkX

We could read a bit more about service discovery via multicast dns to learn that:

DNS-SD uses DNS TXT records to store arbitrary name/value pairs
conveying additional information about the named service. Each
name/value pair is encoded as its own constituent string within the
DNS TXT record, in the form "name=value".
...
The intention of DNS-SD TXT records is to convey a small amount of
useful additional information about a service. Ideally it SHOULD NOT
be necessary for a client to retrieve this additional information
before it can usefully establish a connection to the service. For a
well-designed TCP-based application protocol, it should be possible,
knowing only the host name and port number, to open a connection
to that listening process, and then perform version- or feature-
negotiation to determine the capabilities of the service instance.
For example, when connecting to an AppleShare server over TCP, the
client enters into a protocol exchange with the server to determine
which version of the AppleShare protocol the server implements, and
which optional features or capabilities (if any) are available.

Ok, so there’s a record there, and along with the standard parts of the record, there are two additional key / value pairs in the text record portion. The meaning of the keys and values is probably only known for sure by Time Machine and friends, but certainly some of that data looks familiar.

First there’s the IPv4 address… that’s probably provided automatically when the record is published, based on the machine’s current IP address. IPv6 address follows, again we needn’t worry about that. The port number is listed at 9, which is ‘discard’, so this is probably not used. What port *is* used, then? Well, we know that Time Machine over the network supports either AFP or SMB, so it’s going to be one of those two. Seems like AFP would be most common, just since it is the APPLE file protocol and whatnot…

Then we have the sys and dk0 items… well I dunno what waMA means (something something mac address?), but what follows certainly looks like a MAC address… and sure enough, it matches the MAC address on my Time Capsule.

Note also that the second key in the shown record is dk0, whereas the spec says the defined TXT keys are “sys, dkX”. The “X” here suggests that there might be more than one of these keys, and they are numbered starting from 0 and counting up. Let’s pick appart the dk0 entry.

dk0=adVF=0xa1,adVN=backupz,adVU=AF9AC8F1-BCF5-3E63-9EBD-CD171CF5061B

• adVF: no idea
• adVN: hmm… another “adV” prefix. air disk volume? air disk volume name?
• adVU: air disk volume UUID! (I think)

Ok, now let’s have some fun. Fire up file sharing. Click the little + button to create a new share point. Select a folder / volume that has some eh… free space on it The name of the folder / volume as it appears in the list of share points is the AFP share point name, so we should try to use that for ‘adVN’.

Next, use diskutil to find the UUID for the volume on which your new share point resides. The argument after “info” is the path to the volume. “/” for the boot volume, “/Volumes/whatever” for something other than the boot volume.

sudo diskutil info / | grep "UUID"

Finally, find your system’s MAC address (the one that corresponds to your primary network interface).

netstat -rn | grep default | awk '{print $6}' | xargs ifconfig | grep ether

Now we should have enough information to try to create a record. Looking at the man page for dns-sd, we see the basic usage for registering a record is:

dns-sd -R name type domain port [key=value ...]

Let’s try it! We still don’t know what adVF is, so we’ll just use 0xa1 like the Time Capsule does and see what happens. In the command below, replace 00:11:22:33:44:55 with your MAC address, “Backupz” with your new share point name, and “AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE” with the UUID of the volume on which your share point resides.

sudo dns-sd -R "Back it on up" "_adisk._tcp." "local" "9" \
        "sys=waMA=00:11:22:33:44:55" \
        "dk0=adVF=0xa1,adVN=Backupz,adVU=AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE"

You should see something similar to

Registering Service Back it on up._adisk._tcp..local port 9 TXT ...

Leave that window open. Open a new terminal window.

dns-sd -B _adisk._tcp

You should see your service instance listed. cntrl-c. Get the details:

dns-sd -L "Back it on up" _adisk._tcp local

Looks good. Now go to another mac on your local network. Open the Time Machine prefpane. Click “Choose Backup Disk…”.

Cool!

Limited testing has shown that once you pick a network Time Machine disk in this fashion, the multicast dns service advertisement record is no longer needed. Apparently the Time Machine client caches the information about the location of the network disk. Also, probably a good idea to make sure that your network time machine volume is not guest-accessible…

Here’s a little script to ease the process…

#!/bin/sh
# Instructions:
# 1. Use the Sharing preference pane to configure an AFP share point that will
# be your network backup volume. Connect from another mac using Finder -->
# Connect To Server. Make sure you can log in. Note the name of the AFP volume.
# 2. Use diskutil info to get the UUID from the volume on which your AFP
# sharepoint resides.
# 3. Customize the values below.

# Customize here
servicename="Back it on up"           # the name shown in parens in the TM browser
macaddy="00:11:22:33:44:55"           # MAC address of the AFP server
volname="backupz"                             # AFP sharepoint name
voluuid="6E132F57-8292-4A73-9B37-5C0F82928E85" # volume UUID 

# 4. Make sure this script is executable (chmod +x), then run it. You will be
# prompted for your password (must be admin).
# 5. Go to your other mac, use the Time Machine prefpane to select your shared
# AFP volume. The other mac needs to be on the same local network.
# 6. After selecting the volume, you can cntrl-c this script to stop it.
# 7. Enjoy your network time machine backups!

# no touchy
servicetype="_adisk._tcp."
domain="local"
port="9"
adVF="0xa1"                   # magic?

# we need at least two key / value pairs, like such:
# sys=waMA=00:1F:5B:34:BC:41
# something something Mac Address, I guess...
# dk0=adVF=0x0a1,adVN=TM Test,adVU=AF9AC8F1-BCF5-3E63-9EBD-CD171CF5061B
# dkn iterates starting with dk0, then dk1... these are backup volumes
# adVF is ... something. I used trial and error to find that 0xa1 works
# adVN is volume name
# adVU is volume UUID, which you can get from diskutil info.

# dns-sd -R name type domain port [key=value ...]
sudo dns-sd -R "$servicename" "$servicetype" "$domain" "$port" \
                "sys=waMA=$macaddy" \
                "dk0=adVF=$adVF,adVN=$volname,adVU=$voluuid"

Have fun