Detecting Polymorphic Buffer Overflow Exploits with a Static Analysis Approach

Remote exploit attacks are the most serious threats in network security area. Polymorphism is a kind of code-modifying technique used to evade detection. A novel approach using static analysis methods is proposed to discover the polymorphic exploit codes hiding in network data flows. The idea of abstract execution is firstly adopted to construct control flow graph, then both symbolic execution and taint analysis are used to detect exploit payloads, at last predefined length of NOOP instruction sequence is recognized to help detection. Experimental results show that the approach is capable of correctly distinguishing the exploit codes from regular network flows.