TestMyAV – an independent next-gen testing vendor?

There is a new cat in town – TestMyAV. Instead of my words, let’s introduce TestMyAV by the words from Tom Wright, from http://www.channelweb.co.uk/crn-uk/news/3001315/msp-launches-av-testing-website-to-stamp-out-misleading-reports.

“Security MSP Cognition has launched an anti-virus (AV) testing website to fight against the “misleading” testing culture in the cybersecurity industry.

Cognition – a Cylance and Palo Alto Networks partner – launched the testmyav.com website to give resellers and end users the resources to test AV products themselves instead of relying on tests sponsored or commissioned by vendors.”

After a short signup process, you can access the whole article in case you are interested.

Here is a new independent website to advise people doing independent tests. We do believe to test products yourself is as important as listening to professionals who do these tests for years. There is no one-size fits all. Everyone has different priorities, budget, risk tolerance, and so on. So there you go, test for yourself. But be sure to use independent test methodologies.

None of the current AV Testing vendor methodologies are perfect. And they will never be perfect.

But let’s discuss the independence of TestMyAV.

Introducing malware- by bypassing key AV protections

The TestMyAV recommendation is to:

When you do these tests, beware that whenever you introduce the malware not directly from it’s original malicious source (maliciuous URL, spam folder, drive-by-exploit website) into the test system, the test is not real-world anymore. Downloading malware from TestMyAV will bypass any URL protection. URL protections are mostly reactive, but in real tests, it blocks a lot of malware easily. If you bypass these protections in your test, you favor those vendors who don’t have URL protection, and those who have URL protection will score lower.

Mutation tools

“Evade antivirus detection by modifying and crafting malware yourself, just like the malware authors do.” Actually, we believe this kind of test is interesting, and mimics real-life. Is this independent? Well, find out.

The hash modifier tool should change the results only if the product to be tested heavily relies on hash-based detection. Not very interesting.

Mpress, VMprotect and Hyperion. These packers pack the original file in a way that traditional signature-based scanning will be bypassed. Bad guys use packers for this reason. Should an AV block a packed malware? Sure, it should. But why VMProtect or Mpress or Hyperion? Well, if you have seen any “Unbelievable tour” demo from a next-gen vendor, you know why. They use VMProtect in their test to demonstrate that they detect 100% of the VMProtect packed malware, while the competition does not. But let’s step back and look at the big picture. What if this next-gen vendor detects EVERY (both legit and malicious) file packed with VMProtect or MPress? Is it still an independent test? There can be multiple reasons why all packed files are detected as malicious. Either because all packed files are malicious, or because the vendor believes in high false positive ratio in order to have higher true positive detection. Which is fine. But please do mention during the demos that EVERY VMProtect packed file will be detected by this next-gen vendor. Let me leave a screenshot from this “signature-less” next-gen vendor product here. We have no idea what this code does, but it is an interesting coincidence to find these strings both in the code of the next-gen product, and in an “independent” test methodology.

You don’t believe us that next-gen vendor will detect all packed file as malicious? Test yourself! Another coincidence is that TestMyAV CEO is the CEO of Cognition, and Cognition is a Gold reseller of this next-gen product.

If you favor higher true positive ratio even in the case of higher false-positive: Go ahead, change the default policy of your AV. Change heuristics level from low to high. If there is a check-box “detect all packed files as malicious”, just enable it. You don’t need machine-learning or anything nextgen for these kind of things.

Introduce multiple malware to the test system at the same time – because that is how real world works*

TestMyAV recommends the following test procedure to test malware.

The only time we have seen this test methodology was the same next-gen vendor, when they were performing their “Unbelievable tour” demos. There is one reason why they recommend this kind of test methodology.

Because there is no environment where this is realistic. Executing tens or hundreds of malware at the same time never happens in real life. This is still unclear whether this influences the test in favor of the next-gen vendor or not. But this is not real-life test.

Antivirus Features Recommendations

TestMyAV is kind enough to give us important features of a good AV.

These are very solid and good recommendations. We have created a useful spreadsheet to show how “independent” this recommendation is.

Also, we don’t see why Script control, Powershell control and Macro control is in three lines instead of one, but that is just us …

Test yourself and TestMyAV

Actually, the very idea of test yourself started with this next-gen vendor first. We hope any independent testing methodology would not go as far as to copy sentences from a vendor site.

Actually, there are many many copy-paste similarities in both methodology. We don’t want to cover all here. But if you don’t believe us, test yourself! (Hint: check the malware-safety <->
Safely Handling Malware topics, the Mutating samples, and so on). The similarities of the two methodology is so close that in a university one of the students would have been punished for plagiarism …

A final word on sponsored tests

Whenever you read about sponsored tests, you have to keep in mind:

1. If a vendor sponsors a test and fails being the best, they have the right to bury the test and never allow to publish it. This is the Nr. 1. reason all sponsored test is won by the sponsor.
2. Non-sponsored tests are like the Olympics. Everyone has the same advantage. Sponsored tests are like special Olympics, where:
3. The sponsor decides when to do the test. Clearly, they will sponsor the test just after intensive R&D finished, when the new beta is out and it is just about to beat the competition.
4. The sponsor decides the competitors he wants to compete with.
5. The sponsor decides the type of competition they want to participate in. If they are good at running and swimming, but suck at ski jumping, well, running and swimming will be in the report.
6. Which concludes that at the time of the test, among the tested participants, looking at the competitions, the sponsor is the best.
7. Whether this is interesting in your case when you buy a new endpoint protection product, or not, is your decision.

Conclusion

Test yourself is important part of your vendor selection process. But when you test yourself, the test can be done in millions of ways, and only one is optimal for you. Be sure not to be influenced by methodologies which are not independent.

PS: We have applied to join TestMyAV, although we received mixed messages whether testing companies are welcomed or not. We want to join to measure the quality of their test feed. We are waiting to be approved, so far 1 week passed without any feedback.

* sarcasm

Update 2017. April 17. : Quote from https://arstechnica.com/information-technology/2017/04/the-mystery-of-the-malware-that-wasnt/

For its part, Cylance denied the screenshot actually showed Protect’s code. “It’s a hex view of a sample packed with MPRESS and VMprotect, it looks like,” a Cylance spokesperson said in response to that allegation. “It’s a sample from TestMyAV, I believe. It’s malware, not Cylance.”

To prove we are telling the truth, we created this YouTube video.

Update: Cylance posted a new blog-post about this topic. There is a lot of mumbo-jumbo going around in the article, but let me summarize:
1. They acknowledge this is their code, and they use it for detection
2. This is their generic, “non-signature” based detection for VMprotect and MPRESS.
3. So, they are acknowledging that they detect EVERY Vmprotect/MPRESS packed files as malicious via purpose-built generic detection, yet they recommend people to test with Vmprotect/MPRESS packed files.
4. Any conclusion on the ethics of this approach is left as an exercise to the reader.