(Credit:Zain bin Awais/PCMag/Shutterstock)
Most of the software you install and use visibly does what you expect. Video editors let you tune up your videos. Games respond to your button-mashing. Chat programs connect with your friends. But when you install the protection of an antivirus utility or security suite, there’s no immediate way to know how well it works. Unless you get a notification that the utility foiled an attack, you don’t really know what it’s doing. That’s where PCMag can help. When evaluating these products for review, we put their claims to the test in many ways. Each review reports the results of our tests and provides hands-on experience with the product. This article will fill you in on just how these tests work.
Of course, not every test is appropriate for every product. Many antivirus utilities include protection against phishing, for example, but some don't. Many suites include parental controls, but others omit them. Some nominally standalone antivirus products add firewall protection as a bonus. Whatever features a given product offers, we put them to the test.
We Test Antivirus Protection Using Actual Malware
Every full-powered antivirus tool includes two core features. The on-demand scanner hunts down and removes existing malware infections, while the real-time monitor blocks new attacks. There’s only one way to be sure these protective features work: to hit them with real-world malware. We use virtual machines for this testing, so there’s no risk of spreading any missed infections.
We Collect Real-World Malware
Each spring, when most security vendors have finished their annual update cycle, we gather a new set of malware samples for this test. We start with a feed of the latest malware-hosting URLs, download many thousands of samples, and winnow them down to a manageable number.
We analyze each sample using a suite of hand-coded tools. Some of the samples detect when they're running in a virtual machine and refrain from malicious activity; we simply don't use those. We look for a variety of types and for samples that modify the file system and the Registry.
During this process, we check the identifying hash value of each sample against a database maintained by Google subsidiary VirusTotal. This database scans files using about 70 antivirus engines and reports how many (and which) of them identified a file as malicious. If fewer than half the engines classify a sample as malicious, we discard it. Once we’ve pared the collection down to something reasonable, we record exactly what system changes each sample makes.
We Challenge Each Antivirus to Defend Against Malware
To test a product's malware-blocking capabilities, we start by opening our samples. In some products, real-time protection kicks in immediately, wiping out known malware. If necessary to trigger real-time protection, we single-click each sample, or copy the collection to a new folder, or download the samples from cloud storage—whatever it takes. We note which samples the antivirus eliminates on sight.
Next, we launch each surviving sample and watch how the antivirus handles it. We record the total percentage detected, regardless of when the detection happened.
Detection of a malware attack is necessary but not sufficient; the antivirus must also actively prevent the attack. A small, hand-coded program checks the system to determine whether the malware has made any Registry changes or installed any of its files. For executable files, it also checks whether any of those processes are running. As soon as the measurement is complete, we shut down the virtual machine.
If a product prevents the installation of all executable traces by a malware sample, it earns eight, nine, or 10 points, depending on how well it prevents cluttering the system with non-executable traces. Detecting malware but failing to prevent the installation of executable components earns half credit (five points). Finally, if we find one or more malware processes running despite the antivirus's protection attempts, that's worth only three points. The average of all these scores becomes the product's final malware protection score.
We Evaluate Web-Level Malware Protection
The best time to frustrate a malware attack is before it ever reaches your computer. Many antivirus products integrate with your browsers and steer them away from known malware-hosting pages. A few operate below the browser level, with no extension required. If protection doesn't kick in at that level, there's always an opportunity to wipe out the malware payload during or immediately after download.
While our basic malware protection test uses the same set of samples for months, the malware-hosting URLs used to test Web-based protection change each time. We get those links from a feed of the very latest malicious URLs detected by the London-based MRG-Effitas, and typically use URLs that are no more than a few days old.
Using a small purpose-built utility, we go down the list, launching each URL in turn. We discard any of them that don't point to a malware download and any that return error messages. For the rest, we note whether the antivirus prevents access to the URL, wipes out the download, or sits idly by. After recording the result, the utility jumps to the next URL in the list that isn't in the same domain. We skip any files larger than 10MB or that have already appeared in the same test. We keep at it until we've collected data for 100 verified malware-hosting URLs.
The score in this test is simply the percentage of URLs for which the antivirus prevented malware from being downloaded, whether by blocking access to the URL or by deleting the downloaded file. Perfect 100% scores are not uncommon, and most tested antivirus tools manage 90% or better protection.
We Test Using Actual Phishing Sites
Why go to the effort of creating an elaborate data-stealing Trojan when you can just trick people into giving up their secrets? That's the mindset of malefactors who create and manage phishing websites. These fraudulent sites mimic banks and other sensitive sites. If you enter your login credentials, you've just given away the keys to the kingdom. And phishing is platform-independent; it works on any operating system that supports web browsing.
These fake websites are typically blocklisted shortly after creation, so for testing, we use only the most recent phishing URLs. We gather these from phishing-oriented websites, aiming for an equal split between verified phishing pages and those reported as fraud but not yet verified. The best security programs detect these new fakes using real-time analysis. Those relying on simple-minded blacklists typically earn lower scores.
We use four virtual machines for this test, one protected by the product under testing, and one each using the phishing protection built into Chrome, Edge, and Firefox. A small utility program launches each URL in the four browsers. If any of the four returns an error message, we discard that URL. If the resulting page doesn't actively imitate another site or attempt to capture username and password data, we discard it. For the rest, we note whether each product detected the fraudulent page.
Unlike the web-level protection test, antiphishing scores vary widely. Some products achieve 100% detection, while others can’t even outscore the built-in protection in the three browsers.
We’ve Simplified Antispam Testing
These days, email accounts for most consumers have spam vacuumed out of them by the email provider or by a utility running on the email server. In fact, the average consumer’s need for spam filtering is slim to none. Austrian test lab AV-Comparatives tested antispam functionality back in 2016, finding that even Microsoft Outlook alone blocked almost 90% of spam, and most suites did better—some of them much better. But the lab hasn’t published another such test. According to one of the principals there, “end consumers use hosted mail solutions like Gmail, so no spam filter is needed anymore.”
Years ago, we ran our own antispam tests using a real-world account that we carefully tuned to receive both plenty of spam and plenty of valid email. To test a spam filter, we would download thousands of messages and manually check whether any spam slipped into the inbox or, worse, valid mail got marked as spam. This test took more time and effort than any of our other hands-on tests. Expending maximal effort on a feature of minimal importance no longer makes sense.
There are still important points to report about a suite's spam filter. What email clients does it support? Can you use it with an unsupported client? Is it limited to POP3 email accounts, or does it also handle IMAP, Exchange, or even web-based email? At present, we carefully consider each suite's antispam capabilities, but we no longer suffer through downloading and analyzing thousands of emails
Security Suite Performance Doesn’t Require Testing
When your security suite is busily watching for malware attacks, defending against network intrusions, preventing your browser from visiting dangerous websites, and so on, it uses your system's CPU and other resources to do its job. Many, many years ago, security suites deservedly earned a reputation for sucking up system resources so much that they’d affect your activities on the computer. If users turn off protection because it’s a performance drag, that’s no kind of protection.
Over the years, security companies steadily tuned up their products to eliminate any noticeable slowdown. Until recently, we ran simple tests to measure each suite’s impact on system performance. We’ve dropped that test because almost all recent suites pass with minimal performance impact. In some cases, the tests run faster after installing the suite.
We Test Firewall Protection Several Ways
The typical personal firewall has two jobs: protecting the computer from outside attack and ensuring that programs don't misuse the network connection. In the past, we used a physical computer that used the router's DMZ port to effectively get a direct connection to the internet. More recently, routers and ISPs have made that connection more difficult. In any case, a computer that's connected through a router (as almost all are) is effectively invisible to the internet at large, making our port-scan tests and other web-based tests less relevant. The built-in Windows firewall handles stealthing all ports, so we've dropped this test.
We Evaluate the Firewall’s Program Control
Program control in the earliest personal firewalls was painfully interactive. Every time a new program tried to access the network, the firewall displayed a prompt asking the user whether to allow or block access. This approach isn't very effective since the user generally has no idea what action is correct. Most will just allow everything. Others will click Block every time until they break some important program; after that, they allow everything.
It's becoming increasingly common for suite firewalls to include this level of program control without making it the default. For those products, we turn them on before testing. Also, many firewalls come preconfigured with access permissions for known programs. For our hands-on check of program control functionality, we use an app guaranteed to be unknown—a tiny in-house-coded browser utility not found anywhere else.
At the other end of the spectrum, the best firewalls automatically configure network permissions for known-good programs, eliminate known-bad programs, and step up surveillance of unknown programs. Only if an unknown program attempts a suspicious connection does the firewall kick in to stop it. That’s not something we can test, since we don’t write zero-day malware, but we can observe this functionality during malware protection testing.
We Check Protection Against Exploits
Software isn't and can't be perfect, so the bad guys work hard to find security holes in popular operating systems, browsers, and applications. They devise exploits to compromise system security using any vulnerabilities they find. Naturally, the maker of the exploited product issues a security patch as soon as possible, but until you apply that patch, you're vulnerable.
The smartest firewalls intercept these exploit attacks at the network level, so they never even reach your computer. Even for those that don't scan at the network level, in many cases, the antivirus component wipes out the exploit's malware payload. We use the CORE Impact penetration tool to hit each test system with about 30 recent exploits and record how well the security product fends them off.
We Probe the Firewall’s Defenses
Finally, we run a sanity check to see whether a malware coder could easily disable security protection. We look for an on/off switch in the Registry and test whether it can be used to turn off protection. We attempt to terminate security processes using the Task Manager and third-party task-killer utilities. And we check whether it's possible to stop or disable the product's essential Windows services.
We Verify Parental Control Features
At present, PCMag doesn’t rate or recommend standalone parental control and monitoring services. Rather, we suggest using the free screen time and other parental features built into modern operating systems. But when a security suite includes parental control as one of its components, we still put it through its paces to be sure it does what it promises.
The typical parental control utility keeps kids away from unsavory sites, monitors their internet usage, and lets parents determine when and for how long the kids are allowed to use the internet each day. Other features range from limiting chat contacts to patrolling Facebook posts for risky topics.
We always perform a sanity check to make sure the content filter correctly blocks inappropriate websites. As it turns out, finding porn sites for testing is a snap. Just about any URL composed of a size adjective and the name of a normally covered body part is already a porn site. Very few products fail this test.
We use a tiny in-house browser utility to verify that content filtering is browser-independent. We issue a three-word network command (no, we're not publishing it here) that disables some simple-minded content filters. And we check whether we can defeat the filter by using a secure anonymizing proxy website.
Imposing time limits on children's computer or internet use is only effective if the kids can't interfere with the timer. We verify that the time-scheduling feature works, then try foiling it by resetting the system date and time. The best products don't rely on the system clock for their date and time.
After that, it's simply a matter of testing the program's claimed features. If it promises the ability to block the use of specific programs, we engage that feature and try to break it by moving, copying, or renaming the program. If it says it strips out bad words from email or instant messaging, we add a random word like “fnord” to the block list and verify that it doesn't get sent. If it claims it can limit instant messaging contacts, we set up a conversation between two of our accounts and then ban one of them. Whatever control or monitoring power the program promises, we do our best to put it to the test.
We Release Real World Ransomware, Safely
Modern antivirus programs defend against all types of malware, from simple PUAs (potentially unwanted applications) to pernicious ransomware attacks. Ransomware stands out as perhaps the worst threat because, if your antivirus software misses it, you can't recover by removing it later. Your files remain encrypted and inaccessible even if you remove the offending malware app.
This being the case, many antivirus programs add layers of protection specifically designed to defend against ransomware. One typical defense involves detailed monitoring for ransomware behavior. Another requires authorization for any program making changes to important files. Either way, we do our best to test the ransomware component.
To start such a test, we disable all other real-time antivirus protection layers. This helps simulate a situation in which a zero-day ransomware attack bypasses those defenses. We isolate the test virtual machine from the internet. And, one by one, we release a dozen or more real-world ransomware samples.
When the antivirus detects and blocks a ransomware attack, we double-check that it was successful. Sometimes files may get encrypted before behavioral detection kicks in. And with the systems that defend files in sensitive locations, ransomware may encrypt files outside those locations. We also note when the ransomware presents its ransom note, even if it didn't succeed in encrypting any files.
We Monitor and Interpret Antivirus Lab Tests
We don't have the resources to run the kind of exhaustive antivirus tests performed by independent labs around the world, so we pay close attention to their findings. We follow five labs that regularly release scored test results, which we use to inform our reviews.
AV-Test’s Three-Way Evaluation
Based in Magdeburg, Germany, the AV-Test Institute continuously puts antivirus programs through a variety of tests. The one we focus on is a three-part test that awards up to 6 points in each of three categories: Protection, Performance, and Usability. To achieve certification, a product must earn a total of 10 points and have no zeros in any category. That’s a fairly low bar. The very best products take home a perfect 18 points in this test.
To test protection, the researchers expose each product to AV-Test's reference set of over 100,000 samples and to several thousand extremely widespread samples. Products get credit for preventing infestation at any stage, whether by blocking access to the malware-hosting URL, detecting malware using signatures, or preventing it from running based on its behavior. The best products often reach 100% success in this test.
Performance is important—if the antivirus noticeably slows system performance, some users will turn it off. AV-Test's researchers measure the difference in time required to perform more than a dozen common system actions with and without the security product present. Among these actions are downloading files from the internet, copying files both locally and across the network, and running common programs. Averaging multiple runs can identify the impact of each product.
The Usability test isn't necessarily what you'd think. It has nothing to do with ease of use or user interface design. Rather, it measures the usability problems that occur when an antivirus program erroneously flags a legitimate program or website as malicious or suspicious. Researchers actively install and run an ever-changing collection of popular programs, noting any odd behavior by the antivirus. A separate scan-only test checks to make sure the antivirus doesn't identify any of over 600,000 legitimate files as malware.
Multiple Tests From AV-Comparatives
We track results from three of the many tests regularly released by AV-Comparatives, which is based in Austria and works closely with the University of Innsbruck. Security tools that pass a test receive Standard certification; those that fail are designated as merely Tested. If a program exceeds the minimum requirements, it can earn Advanced or Advanced+ certification.
This lab’s file-detection test is a simple, static test that runs each antivirus against about 100,000 malware samples. A parallel false-positives test ensures accuracy—too many false positives can knock a program’s rating down by one or even two ranks. Much like the AV-Test, the performance test measures any impact on system performance. We consider the dynamic whole-product test to be the most significant. This test aims to simulate as closely as possible an actual user's experience, allowing all components of the security product to participate in fighting the malware.
SE Labs Replays Actual Malware Attacks
Where AV-Test and AV-Comparatives typically test a couple of dozen products, SE Labs generally reports on no more than 10. That's in large part because of the nature of this lab's test. Researchers capture real-world malware-hosting websites and use a replay technique so that each product encounters precisely the same drive-by download or other Web-based attacks. It's extremely realistic but arduous.
A program that totally blocks one of these attacks earns three points. If it sprang into action after the attack began but managed to remove all executable traces, that's worth two points. And if it merely terminated the attack without full cleanup, it still gets one point. In the unfortunate event that the malware runs unchecked on the test system, the product under test loses five points. Because of this last condition, some products (notably early versions of Windows Defender) have received scores below zero.
In a separate test, the researchers evaluate how well each product avoids erroneously identifying valid software as malicious, weighing results based on each valid program's prevalence and the impact of a false-positive identification. They combine the results of these two tests and certify products as successful at one of five levels: AAA, AA, A, B, or C.
Tests by MRG-Effitas Are Pass/Fail
As noted above, we use a feed of samples supplied by MRG-Effitas in our hands-on malicious URL blocking test. We also follow quarterly results from this lab’s ongoing tests. The 360 Assessment & Certification test simulates real-world protection against current malware, much like the dynamic real-world test used by AV-Comparatives. A product that completely prevents any infestation by the sample set receives Level 1 certification. Level 2 certification means that at least some of the malware samples planted files or other traces on the test system, but these traces were eliminated later. Any product that doesn’t achieve one of these levels simply fails.
AVLab Reports Malware Defense Percentages
Poland-based AVLab Cybersecurity Foundation is the latest addition to our lab collection. This lab challenges antivirus apps with a collection of in-the-wild malware and reports the percentage caught immediately and after launch. Since it's a success either way, we report on the total percentage.
We Calculate an Aggregate Testing Score
Coming up with an overall summary of lab results isn't easy, since the labs don't all test the same collection of programs, and they all use different scoring systems. We've devised an algorithm that normalizes each lab's scores to a value from 0 to 10. Our aggregate lab results chart reports a weighted average of these scores, along with the number of lab tests. The best possible result is a perfect 10 score based on results from all five labs.
What About VPNs?
You may have noted that this list of testing methods doesn't cover virtual private networks, or VPNs. Evaluating a VPN is very different from evaluating any other part of a security suite, so we have a separate article on how we test VPN services.
About Our Expert
When the IBM PC was new, I served as the president of the San Francisco PC User Group for three years. That’s how I met PCMag’s editorial team, who brought me on board in 1986. In the years since that fateful meeting, I’ve become PCMag’s expert on security, privacy, and identity protection, putting antivirus tools, security suites, and all kinds of security software through their paces.
Before my current security gig, I supplied PCMag readers with tips and solutions on using popular applications, operating systems, and programming languages in my "User to User" and "Ask Neil" columns, which began in 1990 and ran for almost 20 years. Along the way, I wrote more than 40 utility articles, as well as Delphi Programming for Dummies and six other books covering DOS, Windows, and programming. I also reviewed thousands of products of all kinds, ranging from early Sierra Online adventure games to AOL’s precursor Q-Link.
In the early 2000s, I turned my focus to security and the growing antivirus industry. After years of working with antivirus, I’m known throughout the security industry as an expert on evaluating antivirus tools. I serve as an advisory board member for the Anti-Malware Testing Standards Organization (AMTSO), an international nonprofit group dedicated to coordinating and improving testing of anti-malware solutions.
- Think Incognito Mode Protects You? Here's the Terrifying Truth About Browser Fingerprinting
- Proton Mail Review: Complete Inbox Security Made Simple
- The First Antivirus Built for Family IT Support Misses the Mark on Protection
- This Free Tool Reveals Every Creepy Secret ChatGPT Knows About You
- Your Online Shopping Cart Is Spying on You (And Selling the Evidence)
- More from Neil J. Rubenking
