(Nov 1, 2019 05:33 AM)Wikipedia Wrote:  VirusTotal submissions get scanned by literally thousands of YARA rules setup by members of VirusTotal Intelligence, a very expensive enterprise offering by Google. You do not need to be an AV vendor to purchase and use this feature, but many AV vendors get it for free nowadays in exchange for providing a version of their engine for use on the platform.

The detection results on VirusTotal are NOT reliable for the real-world. AV vendors do NOT have to provide the same engine as found in their commercial software and more often than not, the engines being put onto VirusTotal are lacking a lot of features that the commercial software will possess as well as being tweaked to have a lower false positive or higher false positive detection ratio.

All of the scanning services like VirusTotal which do not actually share the sample with the AV vendors and do not rely on VirusTotal internally are even more unreliable and all detection ratios should be taken with an even bigger grain of salt.

Any dynamic tests which are performed in a virtual environment without ensuring the use of all default protection layers are also unreliable. AV software tends to watch out for where a file originated from (e.g. whether it came from an online source like a web browser or whether it was already on the machine locally) and this can affect anything from how heuristic analysis treats the file or how dynamic analysis treats the file. It varies because it's vendor-dependent and there is no "rule book" on how each vendor handles cases like these, so there's no single "generic formula" for accurately testing AV software with cases like these, until you reverse-engineer the entire thing or have insider knowledge for the vendor in question.

Furthermore, any dynamic tests which are performed without an active internet connection are unreliable because of cloud reputation scanning, which often are very good at flagging unknown applications as suspicious. The results from a cloud reputation scan may also dictate whether a sample is emulated before being allowed to properly run on the environment or how it is treated with heuristic memory scanning and other forms of dynamic analysis as well, and once again, it's all vendor-dependent with no "generic formula" for accurately testing these cases across all AV software.

That being said, any dynamic tests which are performed with an active internet connection also mess things up for you, because most vendors are not so great on privacy and will upload that unknown program to the cloud so it can be processed by an automated sandbox system. Vendors like Avira use sandbox projects like Cuckoo on the cloud, Fortinet have their own FortiSandbox appliances and several vendors are using Hybrid-Analysis. The results are often processed automatically and when there is an issue, a human analyst will get involved and check the logs or start manually analyzing the sample. This is really bad for you, because detection can be caused by dynamic markers stored on the cloud in sandbox reports as well (Fortinet can do this - detect unknown malware which has been seen in the sandbox before and was considered suspicious based on log output).

I can continue to go on and on with this but I'm probably talking to a brick wall.