Nothing But Words

If you need to make use of an external program from within a PHP script, then this essay is for you. My example script is for managing an sftp connection (using OpenSSH), but the principals can be applied to any interaction that requires communication between your script and an external process.

One of the Penn Medical School’s business partners recently stopped allowing ftp connections to their servers for retrieving data files. They required us to switch to sftp (secure ftp). Those providing services for transferring sensitive data files over the internet have been steadily moving from ftp to sftp over the past couple of years, and from what I can see, the pace is accelerating. This poses a programming challenge if you have scripts that automate your ftp needs, as they’ll need to be re-written for sftp. This is not a trivial undertaking, especially if you’re programming in PHP. You can’t just swap out your PHP ftp function calls with sftp equivalents. Actually, you can, but you probably don’t want to, as you would have to upgrade to PHP 5 (adoption of which has been very slow across the PHP community) and you would have to install the PECL/ssh2 library, which – as noted on php.net – currently has no stable version.

So we had to roll our own sftp solution, which required using PHP’s program execution functions. The php.net documentation is good on this topic, but much of it is fully comprehensible only if you already know what you’re doing (this isn’t a criticism – it’s a documentation site after all, not a tutorial site). This annotated sample script will help you get started if you’re new to PHP’s program execution functions.

#!/usr/local/bin/php
<?php

$keyPath = 'path/to/your/ssh_key';
$login = 'your_username';
$server = 'your_sftp_server';
$connectionString = "Connecting to $server...\n";

$childPipes = array(
    0 => array("pipe", "r"), // stdin is a pipe that the child will read from
    1 => array("pipe", "w"), // stdout is a pipe that the child will write to
    2 => array("pipe", "w"), // stderr is a pipe that the child will write to
);

# turning off password authentication will avoid getting a password prompt if
# the key fails for any reason
$connection = proc_open(
    "sftp -oPasswordAuthentication=no -oIdentityFile={$keyPath} {$login}@{$server}",
    $childPipes, $parentPipes);

if ($connection === FALSE) {
    print "Cannot connect to $server.\n";
    exit;
}

PHP’s proc_open is a fork by another name. The $childPipes array is for setting up the communication channels from the child process perspective, and proc_open will set $parentPipes to a corresponding set of communication channels from the parent process perspective. Looking at the definition of $childPipes, the logic may seem backwards at first, but it’s not. For example, the parent process will write to the child’s stdin (element 0), which means the child process is reading that channel.

In the user contributed notes on the php.net proc_open page, most folks write out stderr to a file. But for our sftp script we need to see what’s coming through on stderr, so we’re not directing it to a file.

For establishing the connection, we turn off password authentication, which means we won’t get a password prompt if the key authentication fails. This is important, since the script cannot see or respond to such a prompt (the prompt goes directly to the terminal, so you can’t see it on stdin or stdout; you could see it if you want to do TTY buffering, but let’s not go there…).

# The "connecting..." message is written to stderr. Make sure there's nothing
# besides that in stderr before continuing.
$error = readError($parentPipes, TRUE);
sleep(3);
$error .= readError($parentPipes);

if ($error != $connectionString) {
    fclose($parentPipes[0]);
    fclose($parentPipes[1]);
    fclose($parentPipes[2]);
    $closeStatus = proc_close($connection);
    print $error;
    print "proc_close return value: $closeStatus\n";
    exit;
}

I don’t know if this is typical, but the sftp server we’re connecting to returns the “connecting…” welcome message on stderr (we’re reading stderr with a custom function named readError, which we’ll get to below). Having this message on stderr is problematic, since it’s not really an error message. An actual connection error, such as having a bad key, will come through on stderr after the “connecting…” message. This means we first look for the “connecting…” string (the TRUE argument to readError turns blocking on, so we’ll wait for it to appear – more on this below in the readError function), and then we have no choice but to sleep for a few seconds, to see if anything else comes through on stderr. And finally, to see if there was anything in stderr besides the “connecting…” message, we have no choice but to analyze the string . This is an ugly solution, but dealing with stderr is difficult, since you never know when an error may or may not appear.

If we detect an error, we close the pipes before closing the connection. This is important for avoiding the possibility of a deadlock.

# gets us past the first "sftp>" prompt
$output = readOut($parentPipes);

After logging in, we’ll get an “sftp>” prompt on stdout. We’ll read from stdout to get past this prompt, using the custom function readOut (which is defined below).

# Get the directory listing and print it
writeIn($parentPipes, "ls -l");
$output .= readOut($parentPipes);
$error = readError($parentPipes);

if (strlen($error)) {
    fclose($parentPipes[0]);
    fclose($parentPipes[1]);
    fclose($parentPipes[2]);
    $closeStatus = proc_close($connection);
    print $error;
    print "proc_close return value: $closeStatus\n";
    exit;
}

print $output;

# close the sftp connection
writeIn($parentPipes, "quit");
fclose($parentPipes[0]);
fclose($parentPipes[1]);
fclose($parentPipes[2]);
$closeStatus = proc_close($connection);

if ($closeStatus != 0) {
    print "proc_close return value: $closeStatus\n";
}

This code just demonstrates getting a directory listing (using the custom function writeIn), printing it, and then closing the connection. You can use this as a template for any sftp commands you want to run.

function readOut($pipes, $end = 'sftp> ', $length = 1024) {
    stream_set_blocking($pipes[1], FALSE);

    while (!feof($pipes[1])) {
        $buffer = fgets($pipes[1], $length);
        $returnValue .= $buffer;

        if (substr_count($buffer, $end) > 0) {
            $pipes[1] = "" ;
            break;
        }
    }

    return $returnValue;
}

readOut loops over the stdout pipe until it sees an “sftp>” prompt, which is how we know that the server has finished writing to stdout. Note that we’ve turned off stream_set_blocking. This lets us define our own controls for reading from the stdout stream. In this case, we want readOut to return when the server has finished responding to a command. The best marker for that is the appearance of the “sftp>” after it finishes processing a command, so we set the while loop to break when it sees the prompt.

function readError($pipes, $blocking = FALSE, $length = 1024) {
    stream_set_blocking($pipes[2], $blocking);

    while (!feof($pipes[2])) {
        $buffer = fgets($pipes[2], $length);
        $returnValue .= $buffer;

        if ((!strlen($buffer) && $blocking === FALSE)
          || ($blocking === TRUE && substr_count($buffer, "\n") > 0)) {
            $pipes[2] = "" ;
            break;
        }
    }

    return $returnValue;
}

function writeIn($pipes, $string) {
    fwrite($pipes[0], $string . "\n");
}
?>

Reading from stderr is more complicated than reading from stdout, because 1. there is no equivalent to the “sftp>” prompt to let us know when the server is done writing to stdout, and 2. at any given time, there may or may not be an error. For most places in the script, we solve this problem by:

1. Calling readError after calling readOut. This is based on the supposition – which has proved reliable – that the server will finish writing to stderr by the time it has finished writing to stdout.
2. Setting stream_set_blocking to false. If we set it to true, the script would wait indefinitely for something to appear on stderr, and most of the time there will be nothing there.

The one situation when this approach doesn’t work is when we first log in, since the server writes to stderr before writing to stdout (as described above, it sends that “connecting…” message on stderr). In this case we turn blocking on, since we know the message is coming.

So far this script has been used with only one sftp server, so you may need to make some adjustments to make it work with your server (particularly with how it reads stderr when logging in). Also, I’d be interested in hearing from anyone who has a more elegant solution to handling the initial connection.

Stepping back from the specifics of sftp, the key thing to take away from this is that you will need to acquire a detailed knowledge of the behaviors of your external process so that your script can interact with it reliably. In particular, you need to test your handling of all the different kinds of errors the external process might throw at your script.

Rating:

4248 Spruce St
Philadelphia, PA 19104
(215) 222-6800

As the only source of Pho within walking distance of my office in University City, this greasy spoon gets a free extra half star in its rating. In fact, I think it’s the only pho in the city west of the Schuylkill River (after moving here two years ago I was dumbfounded when I learned that it’s pronounced “skoo-gull” – go figure). What’s visible in the restaurant from a patron’s perspective hovers just on the sticky edge of sanitary. It’s a small restaurant, seating about 25, and I’m told that until recently it had a tattered “Grand Opening” banner festooned across its front wall for about 2 years. I went with a couple co-workers who were already familiar the place, and as we placed our order, the waitress sounded so profoundly bored I was worried she was going to stroll out into the traffic after leaving our table, just to put some excitement in her day.

Having said all that, the pho actually wasn’t bad. Fortunately, the numerous drops of oil floating on the surface of the pho (not something you normally see) didn’t seem to affect the flavor. The main thing they got right was the meat – it was of a decent quality and it wasn’t overcooked or undercooked. My only complaint with the meat was that they didn’t have lean brisket on the menu – if you wanted brisket, you had to go with the fat brisket . The noodles and the extras (sprouts, basil, etc.) were all fine. But they failed to inspire where it counts the most: the broth. The broth is the hardest thing to get right with pho, and it’s the most important thing to get right. The broth wasn’t bad, but it was bland, lacking the complex flavor that defines a good bowl of pho.

For me the main attraction of Pho Cafe Saigon is that I can go there for lunch without disrupting my work day. But if you’re not limited to the University City area, then I’d say head down Washington Ave to Pho 75.

Here are the scans of Maria’s dad that I’ve been promising. Going through these photos was a lot of fun – it was hard picking just a handful to post here.

Eidan is three months old now, and he’s on the verge of outgrowing his 3-6 month outfits. We’re also seeing his personality starting to emerge. Most of the time when he’s being held, he wants to be upright and facing away from you – he wants to see what’s going on around him. And he loves to be outside – sometimes we can park him in his stroller under a tree in the yard and he’ll be happy to just look at the trees, listen to the birds, etc. for as long as half an hour. He’s very different from Kai in this respect – when Kai was a baby he’d observe for a few minutes, but then he’d want to move on to an activity.

Over the past week Eidan has become quite a talker. All babies make cooing noises, as Kai did, but this is different – he will often engage in a discourse for several minutes, waving his right arm forcefully to emphasize key points. His favorite time of day is dinner – as we’re all sitting around the table talking, he will jump right into the conversation. Of course we have no idea what he’s saying, but judging by his demeanor, he’s obviously sharing some very poignant and humorous insights.

Aside from his animated conversation, overall he has a more mellow personality than Kai, which makes Maria and I very lucky. Most days we barely have time to get through our daily routines (work, dinner, clean the kids, wash the dishes, etc.), so the fact that we can stick Eidan in his neglect-o-matic swing for 20 minutes or so when we’re in pinch makes a big difference.

Here’s the first of my scans from Tony’s scrapbook. This is an interview with him and Michiko, published in the Atlanta Constitution, October 11, 1960. Oh the things you could say then that you can’t say now.

I’ve split the scan in half so it’ll fit in my blog layout – here’s the unaltered copy.

As the semester goes on, my Java class is getting harder, and it’s eating up what little time I had left for the blog. But Maria’s mom is coming to visit for a couple weeks – she’ll help out with the kids, so hopefully that will give me some time to catch up on blogging.

I have more photos to share – the first couple are from Maria’s trip to Denver with the boys, for Tony’s Bon Voyage Barbecue. The picture of the bottle of the liquor next to his photo has a box in the middle containing his ashes. Guests could share a drink with him and sign the book. The rest are Kai and Eidan pictures (my friends Jay and Pauline were here last weekend from California – they’re the ones in the pumpkin carving photos).

Maria brought back some great photos of Tony and Michiko from their traveling road show days. I’ll try to get some of them scanned soon.

And I’m turning comments back on – let’s see if the spammers have gone away.