Security teams are experimenting with visual representations of malware and network traffic, but the real test is whether these models can generalize beyond the lab.