| |
 |
|
Date |
:
Feb 1st, 2001 |
| Category |
:
Editorials |
| Manufacturer |
:
N/A |
| Author |
:
Jin-Wei Tioh |
|
Intel's IOMeter is an extremely flexible and powerful program. WinBench 99, detailed earlier, uses access patterns created by real applications. While this pattern accurately represents typical usage, since the access patterns are
fixed, manufacturers can tune their drives' firmware to better suit this pattern, and hence, score better on the WinBench 99 tests. On the other hand, IOMeter's access patterns are synthetic (ie. completely
user-defined), thus hindering a manufacturer's efforts to "cheat" on the benchmarks.
However, since the access patterns are user-defined, the most raised and pressing question would be : "What represents a good access pattern?" Add this to the fact that IOMeter has a setting for the # of Outstanding I/Os, which varies the load it places on a given target (in this case, the drive to be benchmarked) with its access patterns, and the question becomes : "What represents a good access pattern and load?".
What is a fair access pattern? What we're trying to do here is to assess the performance of a hard disk in both workstation and file server scenarios. Thankfully, IOMeter already comes bundled with a file server access pattern, which is what
we'll use. However, a workstation's disk access pattern is helluva lot more tricky. After all, there are "normal" and "high-end" workstation users, each with different expectations and requirements. It should be safe to say that in a "normal" user's workstation, random access typifies the usage pattern, while the pre-dominant I/O operations are reads rather than writes. Why? Excluding users who deal with high-capacity data, typical users spend are loading programs (eg. the operating system, office and internet applications) more than they are creating/modifying data (which are comparatively minute to programs). Thus, the conclusion that there are more reads than writes.
BlueSmoke has standardized on these access patterns :
| Transfer
Size Request |
%
Access Specification |
%
Random Distribution |
%
Read Distribution |
| File
Server Access Pattern |
| 0.5
KB |
10 |
100 |
80 |
| 1
KB |
5 |
100 |
80 |
| 2
KB |
5 |
100 |
80 |
| 4
KB |
60 |
100 |
80 |
| 8
KB |
2 |
100 |
80 |
| 16
KB |
4 |
100 |
80 |
| 32
KB |
4 |
100 |
80 |
| 64
KB |
10 |
100 |
80 |
| Workstation
Access Pattern |
| 8
KB |
100 |
80 |
80 |
| Database
Access Pattern |
| 8
KB |
100 |
100 |
67 |
| Video
Workstation Access Pattern |
| 8
KB |
100 |
0 |
0 |
|
|
The "Database" pattern was created to be similar to IOMeter's default access pattern, which is described as representing "a typical database workload". The only discrepancy is that the default block size is 2k, whereas
we have chosen to standardize on an 8k block size to somewhat standardize all the access patterns. This "Database" pattern has an even higher bias towards random access than the "Workstation" pattern.
The "Video Workstation" pattern was created to represent a typical non-linear video workstation, whereby a video stream is digitized (if not already so) and "captured", ie. stored to the hard disk. Having logged quite some time of non-linear digital video editing, I know that the disk access are almost always sequential in nature (most people defrag their free space before capturing video), and the pre-dominant I/O operations are writes. Thus, one can gauge a drive's suitability for video capturing.
In an attempt to answer the load question, we fired up Win2K's perfmon.exe, and launched a couple of typical operations. Launching FreeCell incurred 14.9 I/Os. Windows calculator caused 4.6 I/Os and typical applications, such as Microsoft Word, Microsoft Excel and Lotus 1-2-3 caused spikes between 32 and 64 I/Os. Defragmenting was by far the worst offender, causing spikes as high as 119.5 I/Os. So, what test parameters should be used? A depth of 1 represents an extremely
linear load, which does not represent any kind of real access. A depth of 4, representing very rudimentary operations such as launching the windows calculator, is still too light. So, a compromise was reached with a depth of 16 to represent a "Light" load. For the "Moderate" and "Heavy" loads, I settled for the "Light"
mutliplied by a factor of 4. The four access patterns mentioned earlier are all subjected to the following loads :
| Linear |
1
Outstanding I/O |
| Light |
16
Outstanding I/Os |
| Moderate |
64
Outstanding I/Os |
| Heavy |
256
Outstanding I/Os |
|
|
Those aren't the only variables that exist in an IOMeter trial. An IOMeter trial runs indefinitely until the user manually halts it unless otherwise specified. We've standardized on a 60 second "ramp-up" time whereby the access pattern are run without affecting the final scores. This is to eliminate any anomalies, such as a swap file access, that may occur within the first stage of testing. For each of our 16 tests (4 access patterns, each under 4
separate loads), we'll use a 10 minute timer.
Any partition on the test drive is deleted, so as to have a fully unpartitioned disk. BlueSmoke is using the beta version of IOMeter, build 1999.10.20 to be exact. We chose to go with the beta rather than the release version because the beta version has extra facilities to run multiple trials. Combined with the fact that reboots between trials are unnecessary (unlike WinBench 99),
we've written a batch file that runs all 16 tests in a neat and unattended
fashion. If you visit the IOMeter
page, you'll notice that there is a release version,
build 1998.10.08. Don't worry, as results from the beta
version (1999.10.20) are comparable to the current release
version. If you're still doubtful, you can download the beta
version directly from Intel here.
IOMeter has a multitude of test results, which include Total I/Os Per
Second, Total MBs Per Second, CPU
Utilization, CPU Effectiveness and Average I/O Response
Time.
The Total I/Os Per Second is the most important result, and will be the only IOMeter result directly presented in articles (there will be provisions for readers to access a drive's full IOMeter profile). It is basically the average number of I/O requests completed in a second, whereby a request involves either a block read or block write - from 0.5k to 64k in the File Server pattern, and 8k in the Workstation/Database/Video Workstation patterns.
The Total MBs Per Second is more or less an extrapolation of the Total I/Os Per Second. In the Workstation/Database/Video Workstation patterns, this figure is the Total I/Os Per Second, multiplied by the block size (8k) per I/O, and divided by 1024.
The CPU Utilization is simply the raw percentage of CPU cycles utilized in completing the test drive's I/O requests.
The CPU Effectiveness is the Total I/Os Per Second divided by the CPU Utilization. This results in the efficiency of drive in using CPU cycles, ie. the number of I/O requests per % CPU utilization.
Lastly, at the "Linear" load, the Average I/O Response
Time, which is measured milliseconds, is also another way of expressing the Total I/Os Per Second. You can calculate the Total I/Os Per Second by dividing 1000 milliseconds (1 full second) by the Average I/O Response Time.
Well, this
concludes our storage testbed disclosure. We already have a
ATA 7200rpm roundup underway, but we're trying to obtain
evaluation units from Maxtor, Western Digital, Fujitsu and
IBM. The same goes for the value category 5400rpm hard
drives. Stay tuned!
|
|
