[ Introduction : Exception Handling : The FlakyIO Framework : FlakyPOSIX : FlakyPalm : FlakyNet : FlakyDisk ]
Flaky Net
Michael W. Bigrigg, Copyright
2004-2007
The goal of FlakyNET is to produce a tool that will inject exceptions into the network connection communication between client and server in order to identify cases where a program does not properly handle server failures. The information can be used to make more robust programs that can handle errors gracefully.
Below is an example of the improvements that can be made after seeing the results of the FlakyNet fault injection tool. The first screen shot is an actual result of an exception being inserted from a database connection.
The application, programmed in PHP, connected to a mySQL database to insert information. An exception was generated during the inserting of the data, though the initial connection to the database and previous operations on the database worked successfully. Too often, it is only the connection to the database that is checked for errors, and subsequent operations are assumed to work.
FlakyNet Background
In an attempt to build an exception injection system using COTS parts, we created the FlakyNet system that would sit between the client and the server as a proxy for a web server. The original FlakyC system could mimic a transient failure, but we wanted to do one without having access to the source code. In addition, we wanted to transparently interpose a generic framework piece that could sit between two machines.
While our goals were a simple system that could be general enough to fit any application, as we progressed along the path, we inadvertently transformed our exception injection concept into a fault injection reality when attempting to insert exceptions into the HTTP protocol, the standard communication protocol between Internet clients and servers. The HTTP protocol did not allow for an exception to be passed from server to client and we were forced to perturbate the network connection to cause an exception to be raised. The exception given was one based on the disconnection of a network.
We ended up developing a fault injection system and not an exception injection system. The engine itself must raise an exception not just cause events to occur that would result in an error, which is what we ended up building in the case of the FlakyNet system. This led us to an understanding that any system we build with exception injection must have explicit exceptions either as part of the language or part of the application protocol.
HTTP Protocol
The HTTP protocol seems to provide for error reporting, as we are all used to the 404 error message that identifies that a page cannot be found. This may be due to an error in specifying the location or due to modification of a web space.
FIGURE 3. 404 Not Found Error Message
When you receive that message, it is not because of any coordination of information that is passed from the server to the client. The server is providing that page of information just as it would any other HTML file.
FIGURE 4. 404 Not Found Error Message Source
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<HTML><HEAD>
<TITLE>404 Not Found</TITLE>
</HEAD><BODY>
<H1>Not Found</H1>
The requested URL /nothere.html was not found on this server.<P>
<HR>
<ADDRESS>Apache/1.3.31 Server at www.cs.cmu.edu Port 80</ADDRESS>
</BODY></HTML>
Another alternative is when a machine is down and otherwise unavailable. Again, this is a fault and not an exception. The message displayed is not generated by the server, but by the client web application.
Example
Our example FlakyNet exception injection tool was written using Java. It is an application program as it needs to accept a connection from a client and then turn around and form a connection with a server.
All parameters specified on the command line.
java -cp (directory) FlakyNET (hostname) (local port) (remote port) (bytes to fail) (which socket) (logfile optional)
-cp (directory) sets the java classpath to the directory path that is specified by (directory). (directory) should be the path to where the FlakyNET class files are stored.
(hostname) can be either an IP address or a DNS entry of the computer where the server is located.
(local port) is the port number that FlakyNET will wait on listening for any clients trying to connect.
(remote port) is the port number that FlakyNET will connect to and pass all information sent by the client to this port on the host.
(bytes to fail) is the number of bytes FlakyNET will pass from the server to the client or vice versa before failing the connection.
(which socket) must be either FromServer or FromClient. These variables allow FlakyNET to monitor the traffic that is sent from either the Server or the Client. When the amount of data that is sent, from the selected connection, is equal to the number of bytes to fail, FlakyNET will fault the connection.
(logfile) this variable is optional. If a name is supplied here then FlakyNET will create a file by that name and write the server/client logs to this file. If nothing is supplied then the default is to print the log files to the screen.
FIGURE 6. A Running Example
C:\FLAKYNET> java -cp C:\FLAKYNET FlakyNET localhost 3307 3306 1000 FromClient
{Tue Oct 08 17:14:18 EDT 2002:FlakyNET} FlakyNET system started.{}
{Tue Oct 08 17:14:19 EDT 2002:FlakyNET} Fault Injector Started
The above starts FlakyNET and has it connect to the server running on localhost at port 3306. FlakyNET then waits on port 3307 for a client to connect and then will pass 1000 bytes of information from the client to the server before failing the connection. And since there is no specified logfile, the log is writen to the screen
The log file writes out all data that is passed by FlakyNET between the server and the client. Each entry has a header composed of the Date and Time as welll as whether the client or server sent the information. After the header the data is printed out as ASCII text. Data that can't be convert to proper ASCII text may appear as unknown characters.
{Thu Sep 26 13:38:45 EDT 2002:CLIENT} show databases{}
{Thu Sep 26 13:38:45 EDT 2002:SERVER}Database@Finndirmysqltes{}
{Thu Sep 26 13:38:56 EDT 2002:CLIENT} dir{}
{Thu Sep 26 13:38:56 EDT 2002:SERVER} {}
The following is an example that will show the capabilities of the FlakyNet fault injection system. It consists of a PHP/Apache web client connecting to a MySQL server. We will show the difference in behaviors between a PHP program that does handle network failures and one that does not.
Download Apache HTTP Server v1.3.26 install program here and to download the pgp key please right click and choose save target as from here.
To install just double click on the msi install package and when prompted for server information please fill in the following values. For network domain: localhost. For server name: localhost. And for Administrator's email address please fill in your own email address. It is up to you whether to run the service for all users or only for your windows username. After installing apache go to the webpage for http://localhost and you should see the apache webpage.
Click here to download PHP version 4.0.4pl1 for windows. After downloading the zip file, make a new directory c:\php, and then unzip the php zip file to this directory.
To configure PHP with the apache server you will need to do the following.
Go to c:\php and copy the php4ts.dll file to the c:\winnt\system32 folder.
Go into the start menu and find the apache http server under programs and then select configure the apache httpd.conf configuration file. This should bring up a text editor with the apache httpd.conf file already loaded. We will have to make some minor changes to this file to enable apache to use PHP files.
The first change happens in the LoadModule section of the httpd.conf file. Add tis line to the end of the LoadModule list:
LoadModule php4_module c:/php/sapi/php4apache.dll
Make sure not to put a # infront of the LoadModule.
In the next section, AddModule, we will have to add this line at the end of the AddModule list:
Further down in the httpd.conf file you will find a AddType section, to this we will add two lines. At the end of the AddType list add,
AddType application/x-httpd-php .php4
AddType application/x-httpd-php .php
Save the new httpd.conf file and return to the c:\php folder
The last change we will have to make is to copy the php.ini-dist to the c:\winnt folder. You will also have to change its name from php.ini-dist to php.ini. Now all that is left to do is restart the apache server from the start menu.
In your C:/Program Files/Apache Group/Apache/htdocs" folder creat a new text file and insert the following text into it:
Save the file as test.php and then go to http://localhost/test.php. If your page displays Hello World then php has been configured correctly. If not consult the install.txt file in your c:/php folder.
Click here to download MySQL version 3.23.52. Unzip this file to somewhere on your computer and run the setup file.
After installing the setup program open a command prompt and go to c:\mysql\bin and type:
This will configure the defaults for the mysql server.
This will log you into mysql as root. Here we will need to setup a password for the root account and also create a new user so the demonstration files can have access to the database server. To set the root password you will need to type:
mysql> SET PASSWORD FOR root=PASSWORD('xxxxxxxxx');
Where xxxxxxxxx is your new password.
Now you will need to create a new user for demonstration purposes. For this you will need to type:
mysql>GRANT ALL PRIVILEGES ON *.* TO NEWUSER@"%"
->IDENTIFIED BY 'PASSWORD' WITH GRANT OPTION;
If you are running the mysql server on the same computer as the apache server and php then please substitute localhost for "%". It is important that your remember the new username and password that you create. The demonstration program will ask for it often so it can access the mysql server. You should quit from mysql by typing exit.
After installing this software please test it to verify that all software is working.
In the below zip file are Html and PHP files used to show how the FlakyNET fault injection tool works and what it can accomplish when used. Please download this file and unzip it to the directory your Apache HTTP Server will serve files. The default is "C:/Program Files/Apache Group/Apache/htdocs" for windows. Please run the creat-db.html file first to create and populate the demonstration database and table.
Output of the demonstation files
FlakyNET settings and log output. It looks identical for both the original and improved functions.
What the original function produces when FlakyNET injects network faults while trying to register users.
With the restults of faulting the original function we were able to see that the insert function's error handling capabilities were poor. And we were able to improve them in an updated version as seen above.
