There are differnent methods ofc.
One is to loop through all processes running, and compare byte pattern, so you can identify malware by a specific bytearray/string.
The other way would to use heuristic detection methods.
Check if the file is saved in suspicous locations like AppData, it uses reflection and maybe also has a startupkey. Then you can tell with a pretty good chance that this is a keylogger/trojan whatever.
One is to loop through all processes running, and compare byte pattern, so you can identify malware by a specific bytearray/string.
The other way would to use heuristic detection methods.
Check if the file is saved in suspicous locations like AppData, it uses reflection and maybe also has a startupkey. Then you can tell with a pretty good chance that this is a keylogger/trojan whatever.