Atomic Spin :: Elements of Software

We recently got a question from one of our customers on writing and running test suites for multi-platform applications (the case in question involves Java). The thread of discussion follows. First, we have Andy’s original question. Next, Karlin and Dave respond…

From: "Andy Brooks" 
Date: Apr 5, 2007 2:08 PM GMT-04:00
Subject: Multi-Platform Automated Testing

Mike,

I have a question that you guys at AO might have an answer to or at least be interested in. Maybe you could put it on Atomic Spin…

We have a Java application and have just decided to ‘test’ it on platforms other than Windows. For our internal unit tests everything works fine. For our external acceptance tests they fail when we check for particular error messages. In the example below the text “(The system cannot find the file specified)” is specific to Windows; on Linux and Solaris the message is different. On Solaris we have also found that the floating point processing yields different results (although this is easily fixed with deltas). In both these instances the application is ‘working’, but the tests aren’t.

What have you guys done in similar situations?

There appear to be options.

1. Detect the platform and run the applicable test.
2. Try to make a platform independent test.
3. Have different test suites for platform specific tests.

Then there is the issue of the build machine(s). If you don’t have build machines for each platform how do you know you are good?

1
2
3
4
5
6
7
8
9
10
11
12
13

// Example
public void testNonExistentInputFile()
{
        File inputFile = new File("blah.xml");
        String[] args = {inputFile.getName(), RTB_OUTPUT};
        CliView.main(args);
        assertEquals(
                inputFile.getAbsoluteFile() +
                        " (The system cannot find the file specified)" +
                        Utils.NL,
                testErrStream.toString());

}

From: "Karlin Fox" 
Date: Apr 11, 2007 11:57 PM GMT-04:00
Subject: Re: Multi-Platform Automated Testing

I’ve never dealt with this directly but I’ve got an idea. How about using Properties files to specify a regex to match the error message? Using a regex allows the match to be variably specific depending on how cross-platform the message is, while still hopefully encouraging a strict match when available. Using a Properties file decouples the regex from the acceptance test suite compilation or distribution, which means shorter turn around for adding new platforms or messages, or for tweaking.

You’d have to decide how to paramaterize based on platform. I’d imagine that the most troublesome messages to match are those that share nothing in common on each platform. Regex’s can’t help you with that problem so you could allow a platform-specific regex scheme, probably just using a part of the os.name property as part of the Properties keys.

If regex’s are too complex, just use Strings and indexOf() as a simpler scheme that still allows multi-platform specificity and decoupled expectation data.

1
2
3
4
5

assertPlatformSpecificMatch(
        testErrStream.toString(),
        inputFile.getAbsoluteFile(),
        PlatformMatchers.getRegexp("missingfile.message"),
        Utils.NL);

1
2
3

assertPlatformSpecificMatch(
        "missingfile.message",
        testErrStream.toString());

Properties and regex’s are built in but as I write this my idea already seems overwrought… Anyone got a simpler idea?

-Karlin

From: "Dave Crosby" 
Date: Apr 12, 2007 12:29 AM GMT-04:00
Subject: Re: Multi-Platform Automated Testing

I tend to write my tests and test helpers such that platform-specific variances get abstracted into methods that can, when needed, be defined in an object that gets instantiated based on platform. Something like errorHelper.seeAndDismissFileNotFoundError(). You have an interface called ErrorHelper, you implement LinuxErrorHelper, SolarisErrorHelper, etc. Though you may spend some time nailing down the magic for each platform, any complications you encounter can be met with custom code without complicating the actual tests.

A more concrete example…

The new DCI system tests:

1. build the rubygem
2. install the rubygem in a temp gem home
3. execute the dci server process
4. (run tests from the outside)
5. kill the server

All steps (except 4) are significantly different from Windows to Unix. Setting environment vars and formatting the command line are dissimilar; the means of capturing and reviewing stdout, stderr and logs are a little different, the act of running the program itself is different. On Windows, the process is synchronous with the command line (you have to use a Thread to run it from within a test, then kill the thread to stop it). On Unix, it runs as a daemon with a “start” command, it leaves a PID file, then you run a “stop” command to end it. (No extra Thread required.)

(The point is, the differences don’t fall into a tight pattern, they’re a little higher up the abstraction tree.)

To solve it, the test driver uses an “app runner” object, which has methods like “start_server”, “stop_server”, “output”, “exit_status”. Two different implementations are available: WindowsAppRunner and UnixAppRunner (they derive from AppRunner which is an abstract class, because some of the operations were algorithmically similar on the inside). At test-run-time, RUBY_PLATFORM is used to instantiate the correct runner.

What’s important to remember is that platform-specific differences are a) few and b) unlikely to fall into a neat pattern. (Eg, piloting a file chooser in Windows might be a bit different than on Linux, or OS X with that crazy aqua stuff)... so much that the problem becomes bigger than string translation and file path separators.)

Given that there are probably only a handful of specific instances within a given app, writing alternate objects to solve each case will not give you too much of a headache.

That said, if the observed differences for a given application do end up falling into a pattern, such as lots of strings that seem to vary from platform to platform merely in their content, Karlin’s Properties file approach would be very useful.