Whenever you read a PC review or a component review, benchmark results typically accompany it. Such results are most often in the form of numbers, such as a score or a frames-per-second total. Sometimes they're relative, in that the result is posted as a multiple of some reference system or number.
Because benchmark scores mostly appear in product reviews, that's how people frequently view them--simply as a way of comparing products. But benchmarking is useful for more than just figuring out what graphics card to buy.
- Benchmark software can help to stress-test new systems (particularly important for PCs you assemble yourself).
- Benchmarks can test for sudden performance issues. If your system has apparently slowed down in some way, you can use before-and-after benchmarks to confirm your suspicions.
- You can run some benchmarks to check whether certain tweaks you've implemented on a system actually speed things up or not.
There's more to benchmarking than simply installing the benchmark software, firing it up, and running it at whatever default settings exist. I'll walk through the reasons you may want to benchmark in more detail, and then I'll discuss how to properly run a benchmark.
Before diving into the whys and wherefores of benchmarking, let's start with my Golden Rule.
My Golden Rule of Benchmarking--
It's a simple rule, and it applies to most users who aren't professional product reviewers: Run benchmarks that give you information about what you want to do with your PC. Anything else is irrelevant.
If you're primarily a PC game player, you probably care about how well games run on your system. Secondarily, you might care about 3DMark scores. In contrast, a Cinebench score or a Photoshop-filter performance test won't be as important to you.
Of course, most people use PCs for more than one purpose. Even so, you probably use your PC for one purpose more than you do for others, so it's worthwhile to focus on how well your system performs in that arena.
By the way, there is a corollary to My Rule: If your PC runs games very well, it will run just about anything well.
You'll find some narrow exceptions to that rule. For example, if you want to use an architectural CAD application, a system that runs games smoothly will likely do fairly well with that CAD app--but it would probably perform even better with a professional graphics card. At the same time, if you use a pro graphics card, game performance will likely decrease a bit.
By the way, there is a corollary to My Rule: If your PC runs games very well, it will run just about anything well.
You'll find some narrow exceptions to that rule. For example, if you want to use an architectural CAD application, a system that runs games smoothly will likely do fairly well with that CAD app--but it would probably perform even better with a professional graphics card. At the same time, if you use a pro graphics card, game performance will likely decrease a bit.
Also, it's true that a great gaming system will run office applications very well, but if you're using office apps exclusively, you probably don't need a high-performance graphics card--and certainly not two of them.
Finally, keep in mind a general exception to Loyd's Golden Rule of Benchmarking: If you're a PC-performance geek, you'll be driven to dissect the performance of every aspect of your PC.
Most people don't care if the primary hard drive is a little slower than a newer model. PC-performance geeks are obsessed with knowing just how much slower their current drives are in comparison with new models. It's just the way their brains work--I should know.
Stress-Testing Your System
When you build a system, you naturally want to test it, to make sure everything works well. You can use software designed to test a system, but those programs don't always do a good job of verifying system performance. In the past I've run 3DMark 2006, which allowed you to loop the test ad infinitum. To heat things up a bit, I'd also run the multicore version of Mersenne Prime95 simultaneously, using the benchmark mode of Prime95 that consumes lots of memory bandwidth.
Whenever I burned in a system this way, I ran both applications simultaneously for 2 to 4 hours.
It's my belief that system burn-in isn't as necessary today as it once was. One of the reasons to burn in a system was to check for early component failure: If all the components survived the burn-in process, they'd likely run for years without issues. Today, components are generally more reliable. If you've built your own system, though, you'll probably want to run a few stress tests to confirm that it's stable at a high performance level before you get started with any serious gaming or other taxing tasks.
Whenever I burned in a system this way, I ran both applications simultaneously for 2 to 4 hours.
It's my belief that system burn-in isn't as necessary today as it once was. One of the reasons to burn in a system was to check for early component failure: If all the components survived the burn-in process, they'd likely run for years without issues. Today, components are generally more reliable. If you've built your own system, though, you'll probably want to run a few stress tests to confirm that it's stable at a high performance level before you get started with any serious gaming or other taxing tasks.
Benchmarks for Troubleshooting
If your system starts to feel sluggish, try running a few performance tests. Whether the game frame rates seem to be slower than before, or the hard-drive performance appears to be declining, or you just get the feeling that the PC has become less responsive, running performance tests can possibly help you spot significant problems.
Once, my PC's game performance suddenly slowed to a crawl. It turned out that my system's cooling fans, including the GPU and CPU coolers, had become clogged with dust. That resulted in high enough heat levels that both components throttled back significantly.
If you want to use benchmarks as problem-solving tools, first you need to run some tests while your system is healthy, so that you have a baseline for comparison.
If you want to use benchmarks as problem-solving tools, first you need to run some tests while your system is healthy, so that you have a baseline for comparison.
System Tweaking
If you overclock your system, or if you just like to play around with different graphics settings, running benchmarks can give you some idea of the impact of a tweak. If you really want to verify the positive or negative effect of a setting change, though, you need to be patient, and change only one setting at a time before rerunning a test.
Remember the Golden Rule of Benchmarking--use tests that are applicable to what you want to do with your system. For example, you may find that altering some memory timings could have a huge impact in a synthetic memory benchmark, but video-rendering time remains the same.
Now that you've looked at the whys of benchmarking, it's time to dive into how to run a benchmark effectively.
Remember the Golden Rule of Benchmarking--use tests that are applicable to what you want to do with your system. For example, you may find that altering some memory timings could have a huge impact in a synthetic memory benchmark, but video-rendering time remains the same.
Now that you've looked at the whys of benchmarking, it's time to dive into how to run a benchmark effectively.
How to Benchmark
Remember a few rules of thumb:
- Prep for benchmarking (including cleaning out the system).
- Let the system settle down for a bit.
- Understand the benchmark.
- Check for repeatable results.
- Read the results carefully.
- Don't draw conclusions that are too broad.
- Document, document, document!
Prep for Benchmarking
Before you fire up a benchmarking app and start running performance tests, you need to prepare the PC. In a perfect world, you want repeatable results, but the architecture of modern operating systems sometimes conflicts with that laudable goal.Ideally, you want a clean system for running performance tests. But Windows runs a lot of stuff in the background. You should leave normal Windows services running; usually they don't have a major impact on benchmark results (unless something like a scheduled defrag or backup is occurring). However, third-party software often installs services and tray applications that use up CPU cycles, ping the network, or occasionally write to storage.
You can review your machine's background items by running msconfig.exe: Open the Start menu, type Run, and then type msconfig in the Run box.
In the example shown here, I'd uncheck Microsoft Office, Adobe Updater, all the Logitech crud, the Acronis backup helper apps, and other similar stuff. If this were a new PC on which I'd want to run baseline performance tests, I'd probably uninstall any antivirus or security programs as well (remembering to reinstall them when the machine was back in actual use).
One other major headache is making sure that all needed updates--including Windows patches--are installed. After you install the latest Windows fixes, it's best to disable the Windows Update service. (You'll find the control panel link to change Windows Update settings under the 'System and Security' heading in Windows 7.)
Make sure all your drivers are up-to-date as well--particularly the driver for your graphics card, but also the audio driver.
Once you have a clean, updated system, you're almost ready to start.
Let the System Settle
After you first boot the system, Windows does a lot of stuff--loading services and drivers, defragging the hard drive, reordering applications on the drive, and more. It's a good idea to let Windows settle for a bit.
The amount of time varies depending on the system's age. A brand-new PC with a just-installed Windows configuration may take a few hours to settle down. Whereas an old PC just needs a few minutes of rest after boot-up. Admittedly, the differences will be minor-- maybe a percent or two. But letting the system idle for 15 minutes or so makes a bigger difference; running a benchmark on a just-booted system will practically guarantee variable results.
Understand the Benchmark
When you install a test, you might be interested in just firing it up and running it at the default settings. But unless you understand the context and goals of the test, the numbers it produces won't mean much to you.Take particular care if you're running game benchmarks. Some games will reset their detail settings if you install a new graphics card--or even install an updated driver. If you run the test with different detail settings, it's impossible to get a valid comparison.
If the benchmark has documentation, read that carefully. You may discover some inherent issues that give one type of component an advantage in the benchmark that isn't always reflected in the real world. One example of this is the 'PPU Enabled' setting in 3DMark Vantage. (PPU stands for "physics processing unit.") Leaving PPU enabled boosts scores on Nvidia-based cards only. Is that fair? Maybe it reflects what you'd see in a game that uses Nvidia's proprietary PhysX physics engine, but not so much for most games.
Knowing the ins and outs of your performance tests is vital to ensuring repeatable results. Which brings us to our next topic...
Check for Repeatable Results
In trying out a new benchmark, run the test several times on the first go-around. Can you get results within a 2 to 4 percent window? If the differences exceed 5 percent, either something is wrong with the benchmark, or your system isn't quite settled or prepped properly.
Or, perhaps the benchmark settings have changed. Altering default settings can bite you if you're not careful. For example, when I run the Unigine Heaven benchmark, I change the default settings, but the test resets itself every time it runs. So I have to take special care to ensure that I reenable the settings.
Benchmarks that allow you to create batch files with custom command-line settings are terrific, because you can always be sure that the settings are what you want, even if the benchmark resets itself upon exit.
The flip side of the coin is that if you consistently get results within 3 percent of one another, that's good enough. People tend to agonize over a sub-1 percent difference in benchmark results. Even if that sub-1 percent number is repeatable, it's generally irrelevant in terms of real-world performance.
Remember, if you want repeatable results, make sure that all things are equal, except the one thing you're changing. For instance, don't test different hard drives in different systems--always use the same system for testing the same type of component. That gives you a frame of reference for your results. If you're changing multiple items of hardware at the same time, it's hard to figure out exactly which one was responsible for any change in performance.
Or, perhaps the benchmark settings have changed. Altering default settings can bite you if you're not careful. For example, when I run the Unigine Heaven benchmark, I change the default settings, but the test resets itself every time it runs. So I have to take special care to ensure that I reenable the settings.
Benchmarks that allow you to create batch files with custom command-line settings are terrific, because you can always be sure that the settings are what you want, even if the benchmark resets itself upon exit.
The flip side of the coin is that if you consistently get results within 3 percent of one another, that's good enough. People tend to agonize over a sub-1 percent difference in benchmark results. Even if that sub-1 percent number is repeatable, it's generally irrelevant in terms of real-world performance.
Remember, if you want repeatable results, make sure that all things are equal, except the one thing you're changing. For instance, don't test different hard drives in different systems--always use the same system for testing the same type of component. That gives you a frame of reference for your results. If you're changing multiple items of hardware at the same time, it's hard to figure out exactly which one was responsible for any change in performance.
Read the Results Carefully
Some performance tests take special care to present results in a format that's easy to understand. Others seem to go out of their way to confuse you.
For example, here's a results presentation from a popular game benchmark, FRAPS:
Average FPS: 39
Minimum FPS: 13
Maximum FPS: 79
It's easy to look at this result and write down "79." Whoa, that new graphics card really is fast. But if you've been recording the average frame rate, such a quick glance will mean you've just skewed the results unfairly--and completely unintentionally. So read the results with care.
For example, here's a results presentation from a popular game benchmark, FRAPS:
Average FPS: 39
Minimum FPS: 13
Maximum FPS: 79
It's easy to look at this result and write down "79." Whoa, that new graphics card really is fast. But if you've been recording the average frame rate, such a quick glance will mean you've just skewed the results unfairly--and completely unintentionally. So read the results with care.
Don't Draw Overly Broad Conclusions
Let's say you run a popular CPU benchmark--something like Prime95--and you find out that your machine's score is relatively low compared with those of other processors. This depresses you somehow, because your system will now be forever inadequate in all applications.
Actually, that may not be true. For example, your PC may run some games just fine. Or, if you have ample DRAM installed, your system's performance in photo-editing applications will be pretty good.
It's easy to draw a sweeping conclusion from a couple of tests. Resist the urge and remember: Run the benchmarks that matter for your own system usage pattern.
Document, Document, Document!
You've just installed more memory that's also faster than your previous kind. Your spiffy new PC now has 4GB of DDR3 running at 1600MHz, up from 2GB at 1066MHz. You run several performance tests to check for any improvement. After looking at the numbers, you think the numbers are better.
You did record your earlier results, right?
I keep master spreadsheets of most of my performance tests, usually by component type. I can fire up my graphics-card spreadsheet and look at results going back several years. Such information is really useful, because we want to believe that newer hardware runs applications faster, but that isn't always true. Unless you document past results, any actual gain is hard to determine.
You did record your earlier results, right?
I keep master spreadsheets of most of my performance tests, usually by component type. I can fire up my graphics-card spreadsheet and look at results going back several years. Such information is really useful, because we want to believe that newer hardware runs applications faster, but that isn't always true. Unless you document past results, any actual gain is hard to determine.
Go Forth and Benchmark
Now that you understand some of the basics of performance testing, download a few benchmarks and give it a try. Whether you want to compare different components, troubleshoot a performance issue, or see if your system tweaks have had any effect, benchmarking will help you better understand your PC and how it performs.
No comments:
Post a Comment