SPEC OSG SPECmail2009 Benchmark
Workload Characterization for SPECmail_Ent2009 Metric

Mike Abbott, Yun-seng Chao

December 2008

 

Summary

This document summarizes the studies on mail server workload collected from multiple university and corporate sources, using a variety of IMAP4 clients. The analyzed workloads consist of both SMTP and IMAP4 requests. Each request is described by parameters which fully characterize its behavior. The proposed models, which are obtained by analyzing these parameters, are able to reproduce the behavior of the mail server workloads.

Document Organization

The report is organized as follows. We start with a description of the measurements and of the parameters considered in our studies. We then present the models characterizing the mail server workloads and we briefly describe how to use these models.

 

SPECmail2009 Additions/Changes

Much of the document discusses the workload changes between new SPECmail2009 and the original SPECmail2008 benchmark workload. Many of the internal distributions were updated with complete message and folder profiles provided by Apple, Inc in 2008. Most of this data replaces the original message and mailbox composition distributions. The SMTP traffic levels have been incorporated into the recipient and message size distributions.

One workload addition not discussed in this document is the ability to test using encrypted TCP connections. The reason lies in where this encryption incurs its cost. The e-mail clients issue commands according to user or programatic directives, regardless of the network connection's encryption mode. Empirical data shows both SUT and e-mail clients require extra computing and/or memory resources if encryption exists. Therefore, the benchmark's Secure metric influences the number of concurrent network sessions and interarrival times but not the actual command sequences. The two SPECmail2009 metrics show the effects of encrypted network connections on the SUT.

Measurements and Parameters

The measurements analyzed in our studies come from different sources. The measurements related to SMTP and IMAP4 have been provided by four companies and by two universities.? The collected sessions were divided into five IMAP4 and two SMTP groups.? The sessions within each group form the basis for all of the parameters that define the Enterprise User Profile, emulated by the SPECmail2009 benchmark.

 

Mailbox and Message Structures

The IMAP4 protocol allows email clients to create and maintain any number of folders and subfolders, in addition to the standard Inbox folder used in the SPECmail2001 POP3 user profile.? The IMAP4 command set also allows email clients to ask the server to describe these structures.? This information is independent of the delivery or retrieval protocols and so is treated outside of specific protocol and/or server context.

Multipurpose Internet Mail Extension (MIME) Profile

MIME is an internet attachment scheme, defined as a formal standard by RFCs 1521, 1522, and 1523.? The Sun and Apple data sets provided detailed information about mailbox and message structure.? Thus they form the basis for the following probability distribution tables used in the benchmark.?

The initial processing of all message sizes distinguished between single part sizes and multipart sizes.? The IMAP4 benchmark prioritizes individual MIME part size over the global message size distribution.

Single Part messages (Sun: 76% of total, Apple: 47% of total)

1. Use “Content-type: text/plain” or no content-type at all in message headers
2. Use subpart content size distribution

 

Multipart Message (Sun: 24% of total, Apple: 53% of total)

1. Use “Content Type: multipart/mixed; boundary=”xxxxxxxxx-counter” or “Content Type: multipart/alternative; boundary=”xxxxxxxxx-counter” in message headers
2. Use distributions for message part width and depth to help establish the set of multipart message bodies.
3. Categorize MIME messages to fall into one of these pre-defined multipart buckets.
4. Use subpart content size distribution to define the sub-part sizes in the fixed pool of pre-defined multipart messages.

Below are the distributions used in constructing messages in compliant with the MIME standard.

 

 

The following tables show the distribution of the number of MIME parts at the top level (without regard to nesting). It reflects the count of multipart/mixed parts immediately “attached” to the main message. It does not reflect any counting of multipart/alternative parts (i.e. text/plain and text/html, alternative formats of the same text). Nor does it reflect the MIME attachment depths (“attachments” to “attachments” or forwarded messages).

 

 

 

 

The next tables show the distribution of the nested MIME Part Levels that occur within a given message from the sample of MIME parts. It generally reflects messages or attachments which are forwarded multiple times, each time adding another depth level to the resulting message.

 

 

 

The following tables show the distribution of primary MIME Content Type (not including subtype) of all the parts in the entire sample.

 

 

After Sun's values were reviewed, a former employee noted that the Unix company that provided MIME distributions tended to use more text messages. Other companies have more and larger MIME parts that have richer, non-textual, content such as word processor documents, presentations, spreadsheets, web pages, calendar events, images, audio, and both rich and simple alternate MIME structures. The major effect of this shift is a tendency to increase the overall message sizes, and decreasing the Text content type in favor of the other categories.

However, increased Alternate structures does not eliminate the Text portion's counts. It just increases the other content types counters. Also, the IMAP server is not required to interpret the actual MIME parts content. It must extract the MIME part(s) and send the content, as is, to the IMAP4 client, which performs the interpretation. Therefore, the shift in Content Type distribution affects the benchmark's MIME structure of the message delivered to the SUT. The SUT still must deconstruct these MIME structures, but not the actual content.

 

Messages Per Folder

The following tables show the distribution of messages in folders at the first five levels.

 

Here is the same data from Apple bucketed such that each contains roughly five percentage points. These are the actual values used in the benchmark.