Flaky Palm

FlakyPalm Background

While the GNU textutils were applications in use for years, some people were critical of our use of those applications. They were not developed to handle exceptional conditions. Desktop utility programs are written assuming a local data storage device. Any failure of the disk would have greater impact to the overall operating system with catastrophic effects not limited to the application in question.

Our next candidate, after FlakyC, was wireless handhelds. It was a platform to which failures, especially transient failures, would be more common. We used the Palm device emulator as our basis. It was easier to modify the code for the Palm device emulator than to devise a scheme to insert our exception injection engine into an actual handheld device. The other advantage to using the emulator is that we can verify that it is a correctly running system.

We tested seven email clients for the Palm device. We tested each mail client using a POP mail server. A connection to the POP mail server was made, and one email message was downloaded from the server to the client. The email client was configured to leave the message on the server.

The results were that wireless handheld applications were no better at exception handling than desktop utility applications. We categorized the results in a similar way as our FlakyC results for when the applications handled the exception correctly (HC), or handled it incorrectly (HI). However, unlike FlakyPOSIX, which had many silent failures (S), the FlakyPalm system experienced applications that would become stuck in an infinite loop (L). On a Palm device when an application gets stuck in an infinite loop it is necessary to kill the application by performing a soft reboot of the device.

The close on a Palm device is unlike the close under C/C++. The close operation can return an exception, but it is designed to forcefully close the connection. All client applications used the forceful close, which technically exempted them from handling the exception.

Test Harness

FlakyPalm is a modified version of the Palm emulator that allows developers to test how Palm wireless applications handle an unreliable network connection.

Example

You will need the modified Palm emulator executable (zip), a Palm ROM (available from Palm, registration required), and a Palm application that connects to a network. As an example, we will be using a Palm finger application.

Unzip the Palm emulator into any directory. One of the unzipped files will be behavior_control.dat. Open behavior_control.dat. On the line with "Open," there will be two zeroes. Change the second zero to a two. This will cause the first open operation to be failed. On the line with the "Close," change the second zero to a two. This will cause your second close operation to fail.

Double-click Emulator.exe. If a dialog box appears, click New Session and click OK. On the next dialog box, click the drop-down list next to "ROM file," choose "Other...," and select the .rom file you downloaded from the PalmOS web site. When the Palm emulator starts, drag the finger.prc file that was included in the zip file to the Palm emulator. This will install the finger application. Click the Applications button on the Palm emulator and click the PalmFinger icon.

When the Finger application loads, click the Finger button. It should look like the successful run shown in Figure 2. Click the Finger button again. You should see an error dialog appear as in the screenshot in Figure 3.

FIGURE 1. Example Behavior Control File

FIGURE 2. Success run of the Finger Application.

FIGURE 3. Finger response to a failure of the Open operation.

This is an example of a failure that is handled correctly. The application recognized the failure of the Open operation and alerted the user.

Hit the OK button, and then click the Finger button again. This time the application will appear to run normally and you will see the result of the finger command. However, a close failure occurred, and the application did not notify the user. This is an example of faulty error handling. Now close the application by right-clicking on the emulator and select Exit. Choose No when prompted to save the current session.

The error_log.dat file contains the network operations that were called and shows which ones were failed. You can see where the open and the close errors occurred (they are designated by a "Failed Open" and "Failed Close").

FIGURE 4. The Error Log File

The Behavior Control File

The behavior_control.dat file is used to selectively fail network operations. Each line in the file has a network operation followed by two numbers. The first number represents continuous failure, with 0 meaning that continous failure is off and 1 meaning that continuous failure is on. The second number represents a single failure or when to start continuous failure, depending on the first number. If the first number is 0, the second number represents the number of operations to complete successfully until failing the operation once. If the first number is 1, the second number represents the number of operations to complete succesfully until failing it continously.

For example, Open 0 5 means that the 5th open will fail and Open 1 5 means that every open after the 4th will fail. When both numbers are set to zero, the associated operation will never fail. When both numbers are set to one, the associated operation will always fail.

What to Look For

The Palm emulator can simulate how your Palm application will respond to different types of network failures. After running the emulator, an error_log.dat file will be generated in the C:\ directory. The log shows which operations were called and which operations were failed.

A variety of popular applications were tested, and the FlakyPalm emulator showed that many of these applications do not properly handle certain failures. For example, most applications failed to correctly detect close and send failures.