Hands-On Testing and Analysis

Introducing JetTest

For a recent project, the DeepStorage team was tasked with testing a storage system using Microsoft Exchange Server Jetstress. Jetstress is a heavy-weight benchmark tool that leverages Microsoft Exchange’s JET database engine, which has historically been taxing on storage systems.

Among the problems we regularly encounter while testing, three are especially prescient. The first is that tests are time-consuming, generally on the order of hours per run. Jetstress, in particular, can experience setup and cleanup times that more than double the amount of time needed to run its tests. The second is that running test instances simultaneously on multiple machines, as we are wont to do, generally requires configuring and initiating each instance manually. Moreover, virtual machines have limited access to their host systems’ clocks. Together, this makes coordinating the start times of tests involving multiple virtual machines difficult if not impossible. Ideally, we’d want all of the involved test machines to start within at most a few seconds of each other.

Today, we’re happy to announce the release of JetTest, an open-source, Python-based tool to facilitate the coordinated execution of Jetstress testing over a LAN.

JetTest and Jetstress

Suppose we want to know how our storage system behaves with various Jetstress loads. Traditionally, this requires manually configuring Jetstress on each of our test machines via the GUI, running the test, and then coming back several hours later to configure the next test. With JetTest, we can configure a series of tests to be run in sequence, start the run, and come back the next day to check the results for all of them.

We create a collection of identical test virtual machines and connect them to a common file share. Each machine runs a lightweight software agent (part of NetJobs, below) that listens for commands from a control center machine. Next, we generate a template configuration file using the Jetstress GUI.

JetTest takes a  configuration file specifying the location of the file share, the Jetstress configuration template, and the test machines, as well as a list of mailbox and IOPS values to use for each test. It then automatically creates Jetstress configurations for each test machine, runs the test, and, when complete, moves on to the next.

JetEdit and NetJobs

JetTest makes use of two additional tools (included) we built to facilitate different parts of the process: JetEdit and NetJobs. JetEdit is a simple command-line tool for modifying the values of specific keys in Jetstress XML configuration files.

NetJobs is something we’ve been working on for a while and handles most of the heavy lifting. It consists of a pair of software agent and control center programs that allows us to coordinate the execution of CLI-based tests on remote machines. This helps to combat the problem of coordinating tests by eliminating the need for the test machines to rely on their system clocks. The software agent on each machine listens for incoming connections from the control center and executes tests on its behalf.

A general-purpose version of NetJobs is currently in active development, which you can check out here.