FAQ Table of Contents at 22 April, '08

About the Benchmark

1. Why did SPEC choose a server side Java workload as a basis for the SPECpower_ssj2008 benchmark?
   • We chose the workload that best met the following requirements:
     • Scalable, Multi-Threaded
     • Portable across a wide range of operating environments
     • Minimal Cost to Run ($$ and time)
2. Are the results of SPECpower_ssj2008 comparable to any other benchmarks?
   • No, they are not. Several substantive changes have been made that make the "ssj2008" workload(s) unique and different from all other benchmarks.
3. What is a "compliant" run of SPECpower_ssj2008?
   • A compliant benchmark run meets all the requirements of the SPECpower_ssj2008 run rules for a valid run that could be submitted to SPEC for publication on the SPEC web site.
   • In addition to the run and report rules, several validation and tolerance checks are built-in to the benchmark.
   • If you intend to submit results to SPEC and/or publicly use the SPECpower metric "overall ssj_ops/watt", the run must be compliant.
4. What is "FDR"?
   • FDR is an acronym for "Full Disclosure Report".
   • The FDR is the document that represents a full benchmark run
5. Can I "turn off" compliance checking? My results are only non-compliant because I do not have a temperature sensor.
   • No. This is to prevent a Full Disclosure report being used that is or might be non-compliant.
6. What is a "graduated workload"?
   • A graduated workload operates at several levels, usually stepping up or down by some regular value.
7. Why did SPEC choose to implement a graduated workload?
   • Server power consumption has been shown to be significantly different at low load levels versus high load levels, and server loads vary widely from hour to hour and day to day. Since no definitive surveys exist, and all data centers are different, SPEC has chosen to provide graduated loads across a set of load levels to portray performance and power consumption in a reasonable range.
8. How did SPEC determine the increments of the load levels?
   • Load levels are graduated by increments of 10%. This choice was a compromise between too few load levels and too many. Increments of 5% would require 20 levels and double the run-time. Five would be too few.
9. Can I run other increments?
   • Yes. For non-compliant runs, you may run as few or as many as your needs dictate.
   • The number of load levels is set by an entry in the SPECpower_ssj.props file.
10. Is there an upper limit to the number of load levels I might run?
    • We believe that hundreds of load levels are possible but this capability has not been tested. 50 levels (2% increments) have been run successfully.
11. Can I run just one or two load levels?
    • Yes. The number of load levels is set by an entry in the SPECpower_ssj.props file.
12. What is "active idle"?
    • Idle is, generally speaking, the state when the system is not running any applications or performing any operating system management tasks. CPU utiliztion is zero.
      • We could label this state "OS idle".
    • The duration of idle states can vary from fractions of seconds to minutes. Modern operating systems run many asynchronous background tasks and therefore most servers are never totally idle for long periods.
    • "active idle" is a SPEC defined state where an application is running and no transactions are incoming or in process; the system is ready to quickly respond to any incoming transactions.
13. How is active idle measured?
    • Active idle is "the eleventh load level" of the ssj_2008 graduated workload.
    • Performance is zero and power is measured at what is usually the lowest rate of consumption.
    • This load is treated the same as all others in computing the SPECpower_ssj2008 overall ssj ops / watt.
14. How long is the active idle period?
    • The duration of the active idle load level in the case of compliant runs, is the same as all other load levels; 240 seconds.
    • In the SPECpower_ssj2008 Run and Reporting Rules document, active idle is documented in section 2.3.2.1.
15. Can I acquire and run this benchmark in my own facility?
    • Yes. See "Acquiring the Benchmark" below.
16. Is there a "Users Guide" that walks me through set up, run and results interpretation?
    • Yes, that document is the "SPECpower_ssj2008 Users Guide".
17. Can I get documentation without buying the benchmark?
    • Yes, the Users Guide is available on the SPEC web site
      • www.spec.org/power_ssj2008
18. What are the key documents I should read before I get started?
    1. SPECpower Methodology
    2. SPECpower_ssj2008 Users Guide
    3. The set of Design Documents
    4. FAQ (this document)
    5. Run and Reporting Rules
       • if submitting results to SPEC or,
       • if making public comparisons using SPECpower benchmark results
19. What operating system environments are supported?
    • SPECpower_ssj2008 software elements have been successfully tested in several common operating environments (see "Software Requirements" below).
      • Find a list of tested environments at www.spec.org/specpower
    • CCS requires Linux, Solaris or Windows operating environment (see below).
20. In what what language is the benchmark written?
    • Much of the code is written in Java.
    • PTDaemon (Power Temperature Daemon) code is C++
21. Why is PTDaemon in C++ while all the other code is Java?
    • The PTD code must manage and then read and write to serial ports and therefore uses low level constructs not generally available in Java.

Hardware Requirements

1. What is the minimum hardware required?
   • At minimum, you will need two network connected systems and one power analyzer/meter.
   • One is the SUT or System Under Test, the other the CCS or Control Collect System.
     • The CCS system can be a notebook computer (minimum 512KB memory).
   • For compliant runs, a power analyzer and a temperature sensor accepted by SPEC are required, plus full adherence to the benchmark run rules.
2. How much computer equipment do I need to run the SPECpower Benchmark?
   • The minimum equipment required is two computers and one power analyzer.
     • A temperature sensor is also required for compliant results.
   • Computer One: The SUT (System Under Test)
     • the system for which you wish to measure power and test performance.
     • As with most benchmarks, this machine runs the "workload"
   • Computer Two: A "CCS" (Control, Collect System) which collects and logs the performance and power data.
     • The two systems must be networked via LAN (TCP/IP communication is used).
     • the power analyzer data transfer cable is attached (typically RS232, USB or GP-IB)
   • Power Analyzer: A device that meets the current 'specifications' and with data logging capability (see "Power Analyzers" below)
3. How much does a power analyzer cost?
   • A quality power analyzer can cost more than $2,000(USD).
   • See Power Analyzers below.
4. Can I run without a power analyzer?
   • Yes, for non-compliant runs. The PTDaemon (Power Temperature Daemon) has a device type "dummy" for these occasions.
   • The absence of power data makes computing a power-performance metric impossible.
5. Can I run without a temperature sensor?
   • Yes, for non-compliant runs. The PTDaemon (Power Temperature Daemon) has a device type "dummy" for these occasions.
   • For more information, see the FAQ section on Temperature Sensors.

Software Requirements

1. What Operating Systems are supported?
   • On the SUT side, the workload will run on any OS for which a JVM is available.
   • On the CCS side, the CCS software and the PTDaemon has been tested on SuSE Linux, Microsoft Windows XP and Server 2003 (32 and 64 bit), and Sun Solaris.
     • The power and temperature measurement devices you choose may or may not have drivers for your intended operating environment. Best to check.
2. What other software will I need?
   1. A Java virtual machine (JVM).
      • The workload and the CCS are written in Java to enable the highest degree of portability across software operating environments.
3. How long does it take the run the benchmark?
   • The run time of the 'workload' is approximately one and one-quarter hours (~70 minutes) with default settings. Here's how it breaks down
     • There are 10 load levels an idle measurement period and three calibration levels that each run for 240 seconds (4 minutes).
     • Between each load level there is a 10 second period of zero activity
     • Each load level has a ramp-up and a ramp-down period, each 30 seconds.
     • (240+30+30+10)* 14 = ~75 minutes

Results Publication

1. Am I required to submit benchmark results to SPEC?
   • If the results are for your use only, then No.
   • If you intend to use the results publicly, then it is (currently) required that they follow the run and reporting rules and then be reviewed by SPEC prior to that public use.
     • This requirement is a provision of the license agreement.
2. If I am not a SPEC member or associate, can I submit benchmark results for publication by SPEC?
   1. Yes, subject to the review requirements and a nominal publication fee.
   • If your company or organization is a member of SPEC you may submit, have your results reviewed, and potentially published on the SPEC web site.
3. Are results published on the SPEC web site audited?
   • Results are "peer reviewed" prior to posting on the SPEC web site. The full veracity and accuracy of the results is ultimately the responsibility of the organization that tested and submitted the results.

Acquiring the Benchmark

1. How much does it cost?
   • The cost of the benchmark and the order form are on the web at: http://www.spec.org/order.html
2. How do I order the benchmark?
   • You will find an order form at: http://www.spec.org/order.html

Contacting SPEC

1. I have a question or problem that isn't covered in the the documentation.
   • Send your query via e-mail to: support@spec.org with "SPECpower" in the subject line.

Running the SPECpower_ssj2008 Benchmark

Set Up and Starting

1. What do I have to change just to get started?
   • See the document "Quick Start Guide"
2. CCS terminates with error such as: Can't find "SCmain".
   • This is more likely can't find the JVM. Check the path to "java.exe"
3. What if the PTDaemon can't connect to the power analyzer?
   • Check COM port and the analyzer type passed to the power daemon (ptd_xxxx.exe)
   • Assure the analyzer is connected via serial or USB cable and the analyzer is powered on.
4. Waiting for connection - forever
   • Check the IP address of the SUT in the ccs.props file.
     • The CCS system and the SUT system should be able to 'ping' each other.
   • Check the IP address and TCP/IP port number in the ccs.props file and assure it is the same port number that is passed to the PTDaemon (runpower.bat).
     • In most cases the CCS software and the PTDaemon will be running on the same system. There should be no changes made to the ccs.props file in this case.
   • Kill or terminate all the SPECpower software and re-start.
5. Command window immediately closes (Windows) when you execute the batch file (by double clicking on the file name)
   • If you execute the run.bat by double clicking on the file name , a command window will appear then close upon program termination. Instead open a command window first (using open_cmd_window.bat for example) then enter runxxx.bat. The window will stay open and allow examination of the activity.

Collection System

1. Can I use a laptop or desktop system for the Control Collection System?
   • Yes; with either type of system, on Windows OS, make sure to:
     1. set your Power Scheme to "Maximum Performance" and,
     2. screen saver to "none" and,
     3. set "Power Options" (plugged in) to System Standby "Never"
2. What are the minimum system requirements?
   • Most modern machines are suitable
   • Minimum processor speed of ~1GHz, minimum 512MB memory,
   • Ethernet network capability with TCP/IP installed
   • ~10MB free disk space for code and log files
   • No special graphics requirements

Power Analyzers

1. Is a power analyzer required to run the benchmark?
   • For compliant runs, yes.
   • For any other runs, the PTDaemon (Power Temperature Daemon) has a device type "dummy" for these occasions.
2. What is the difference between “tested” and “accepted” power measurement devices?
   • Tested devices run successfully with the PTDdaemon though not all are accepted for compliant benchmark runs.
3. What power analyzers have been tested with the SPEC Power Benchmark?
   • A list of successfully tested and compliant power measuring devices are listed on the SPEC web site at: http://www.spec.org/power_ssj2008/docs/device-list.html
   • A list of all power measuring devices that have been successfully tested with the PTDaemon can be seen by running the PTDaemon with no input parameters. Not all tested devices are compliant.
4. Do I have to configure the power measurement device manually?
   • This depends on the device brand and/or model. Some devices need to be configured manually via the front panel; for example communication settings or current and voltage ranges.
   • For more details on device setup, see the Hardware Setup Guide at: http://www.spec.org/power_ssj2008/docs/SPECpower_ssj2008-Hardware_Setup_Guide.pdf
5. Help! My power analyzer only supports RS-232 for data logging, I have no serial ports; what do I do?
   • There are devices that convert RS-232 to USB, that cost $10-14 (US$).
     • Several brands of serial to USB converters have been found to work well.
     • With Windows, install the software per manufacturer instructions, then use the device manager to see what "COM" port number was assigned.
6. Help again! My power analyzer did not come with an RS-232 cable. Any clues?
   • One combination that has worked with some devices is a standard "Modem" cable with the addition of a "NULL MODEM" adapter. Each device brand or model can be different, some experimentation might be required.
   • The above combination can also be plugged into a RS-232 to USB converter.
7. The PTDaemon software can't connect to my power analyzer - what is wrong?
   • First, assure the analyzer is communicating on the data cable with the host machine. The best way to verify this is to first use the vendor supplied software to read the analyzer. Most problems can be resolved with this method, problems such as cable not connected, wrong cable, incorrect device settings and so on.
   • If the vendor supplied software shows the correct readings, assure that the following are correct:
     • the IP address and port number in the ccs.props file.
     • the analyzer type used when the PTDaemon is started (and the port #).
     • If your problems persist, send e-mail to SPECpower support at this e-mail address: support@spec.org with a subject of SPECpower.
8. Why does my power analyzer display unexpected watts readings?
   • This is likely to be traced back to how the analyzer is connected or the device settings.
   • More details on device setup can be found in the Hardware Setup Guide at: http://www.spec.org/power_ssj2008/docs/SPECpower_ssj2008-Hardware_Setup_Guide.pdf
   • Instructions will also be found in the power analyzer documentation or on the vendor web site. You may also contact the device vendor for further technical support.
9. Will my power analyzer work with the SPECpower PTDaemon even though my device does not appear on the accepted devices list shown on the SPEC web site (http://www.spec.org/power_ssj2008/docs/device-list.html)?
   • Run the PTDaemon program with no parameters to see a list of tested devices and their device codes. The data communications protocol has been implemented for these devices and properly set up, do function with the PTDaemon.
   • As of version 1.2, there are approximately 20 devices that successfully transfer data. Note that data transfer capability does not imply that the device is accepted for compliant measurements.
   • If your device is not on the tested device list, it is unlikely that PTDaemon will successfully read data.
   • Communication protocols are quite different among the various manufacturers and models of power analyzers.
10. Will the SPECpower PTDaemon (Power Temperature Daemon) connect to and read my power analyzer even though my device is not on the tested list?
    • It is unlikely that PTDaemon will successfully read a meter not on the "tested" list.
    • See the responses to the FAQ just above.
11. How can I get my power analyzer included in the SPEC accepted devices list?
    • The process for getting a new measurement device accepted for use with SPECpower_ssj2008 is described briefly in section 2.13.5 of the run rules. Basically, documentation must show that the device has sufficient accuracy, code must be written to enable SPEC's PTDaemon software to read measured values from the device, and then testing must be performed to show that the device does meet SPEC's requirements in actual usage. Even if the device doesn't meet the SPEC requirements, it may still be possible to add software support such that the device could be used for internal (non-publishable) measurements.
12. No data cable came with my analyzer, what now?
    • Each brand/model power analyzer may require a different cable.
    • Data cable information can be found in the Hardware Setup Guide at: http://www.spec.org/power_ssj2008/docs/SPECpower_ssj2008-Hardware_Setup_Guide.pdf
13. Why do I see -1 (or -2) in the watts column in the ssj.nnnn.ccs-log.csv log file?
    • This indicates an error acquiring data.
    • -1 indicates the data is unreadable
    • -2 indicates a communication or timing error
14. What is wrong when I see very very large numbers recorded for watts (I have a Yokogawa WT210 Power Analyzer)?
    • Your analyzer is in the "wrong mode".
    • Reset your analyzer to factory default settings by following these steps:
    • Reset Analyzer to initial values:
      1. turn the analyzer off,
      2. press and hold enter key (on front panel), while powering on the analyzer, holding the enter key down until errors display (which are normal).
      3. done
    • Next it is best to assure the RS-232 (serial) communications are set up properly .
15. Yokogawa WT210 Power Analyzer - Front Panel Setup
    • Press the “local” key
      • in display window “C” use the Up/down arrow keys to select 488.2; Press enter
      • In display window “A” use the Up/down arrow keys to select “hand 0”; Press enter
      • In display window “B” use the Up/down arrow keys to select “For 0”; Press enter
      • In display window “C” use the Up/down arrow keys to select “b 9600”; Press enter
      • In display window “C” use the Up/down arrow keys to select “CR”. Press enter

Temperature Sensors

1. How do I run without a temperature sensor?
   • The PTDaemon (Power Temperature Daemon) has a sensor type of "dummy".
     • this is the best temporary fix if you will add or use a sensor in other runs.
   • You may also remove the temperature sensor data source from the ccs.props file.

1. Help! My Digi Watchport/H meter worked fine last week, this week I get this set of errors from the power daemon.
   • Serial Port Create File failed with error 2, serial port open failed with error 0, Failed to connect to meter
   • When I look at the Ports with the device manager, I see this message in the box on the "general" tab: Windows cannot identify this hardware because it does not have a valid hardware identification number. (Code 9)FYI, Digi Watchport driver version 4.2 for Windows XP
   • Answer:
   • Unplug the Watchport USB connection, count to 10, plug it back in.
   • This problem sometimes occurs after the system has been re-booted.
   • At this time (27 Aug '07), we are not sure the root cause.

SPECpower_ssj2008 Results Reports

1. Where are SPECpower_ssj2008 results cataloged on the SPEC web site?
   • Results disclosures are at this URL: http://www.spec.org/power_ssj2008/results/
2. How often are new results posted?
   • New disclosures are added approximately every two weeks.
3. Can I "turn off" compliance checking? My results are only non-compliant because I do not have a temperature sensor.
   • No. This is to prevent a Full Disclosure report being used that is or might be non-compliant. 1 On the SPECpower report, if I compute my own performance to power ratio, why do I sometimes get different results for some load levels? I've seen a difference of up to two points.
   • This is due to the higher precision of the measured results used to compute the ratios that appear in the report. This is an artifact of rounding. The ratios in the reports are correct.