Integrity-Based Kernel Malware Detection

Authors

Feng ZhuFollow

Document Type

Dissertation

Degree

Doctor of Philosophy (PhD)

Major/Program

Computer Science

First Advisor's Name

Jinpeng Wei

First Advisor's Committee Title

Major Professor

Second Advisor's Name

Bogdan Carbunar

Second Advisor's Committee Title

Committee Member

Third Advisor's Name

Xudong He

Third Advisor's Committee Title

Committee Member

Fourth Advisor's Name

Gang Quan

Fourth Advisor's Committee Title

Committee Member

Fifth Advisor's Name

Geoffrey Smith

Fifth Advisor's Committee Title

Committee Member

Keywords

Integrity modeling, Invariant, Kernel Queue, Malware detection, System Security

Date of Defense

6-5-2014

Abstract

Kernel-level malware is one of the most dangerous threats to the security of users on the Internet, so there is an urgent need for its detection. The most popular detection approach is misuse-based detection. However, it cannot catch up with today's advanced malware that increasingly apply polymorphism and obfuscation. In this thesis, we present our integrity-based detection for kernel-level malware, which does not rely on the specific features of malware.

We have developed an integrity analysis system that can derive and monitor integrity properties for commodity operating systems kernels. In our system, we focus on two classes of integrity properties: data invariants and integrity of Kernel Queue (KQ) requests.

We adopt static analysis for data invariant detection and overcome several technical challenges: field-sensitivity, array-sensitivity, and pointer analysis. We identify data invariants that are critical to system runtime integrity from Linux kernel 2.4.32 and Windows Research Kernel (WRK) with very low false positive rate and very low false negative rate. We then develop an Invariant Monitor to guard these data invariants against real-world malware. In our experiment, we are able to use Invariant Monitor to detect ten real-world Linux rootkits and nine real-world Windows malware and one synthetic Windows malware.

We leverage static and dynamic analysis of kernel and device drivers to learn the legitimate KQ requests. Based on the learned KQ requests, we build KQguard to protect KQs. At runtime, KQguard rejects all the unknown KQ requests that cannot be validated. We apply KQguard on WRK and Linux kernel, and extensive experimental evaluation shows that KQguard is efficient (up to 5.6% overhead) and effective (capable of achieving zero false positives against representative benign workloads after appropriate training and very low false negatives against 125 real-world malware and nine synthetic attacks).

In our system, Invariant Monitor and KQguard cooperate together to protect data invariants and KQs in the target kernel. By monitoring these integrity properties, we can detect malware by its violation of these integrity properties during execution.

Identifier

FI14110701

Recommended Citation

Zhu, Feng, "Integrity-Based Kernel Malware Detection" (2014). FIU Electronic Theses and Dissertations. 1572.
https://digitalcommons.fiu.edu/etd/1572