[LKM] Loadable Kernel Modules Detection: -> rootkit detection overview

Bruce F Lucca lucca at Buffalo.com
Mon Apr 9 12:36:22 EDT 2001

[LKM] Loadable Kernel Modules Detection: rootkit detection overview
Date: Sat, 7 Apr 2001 16:06:45 +0200
From: "Michael 'fr0sty' Kummer" <frost at PACKETST0RM.NET>
Subject: Detecting LKM rootkits 


The purpose of this paper is cover LKM basics, detecting "Trojaned" LKM's
and figuring out which LKM is installed on your machine.


What is a LKM? Loadable Kernel Modules (LKM) are files that contain
dynamically loadable kernel components. LKM's are normally used to load
device drivers and other hardware drivers. LKM's can be found on Linux,
Solaris and BSD (Open, Free and Net). 

This paper will focus on Linux.

Linux comes with various tools to assist the system administrator in
loading, listing and unloading of the kernel modules. Before we cover the
tools, lets look at some important files and directories associated with
Kernel Modules. 

The first directory we want to look at is /lib/modules/"kernel_version". 

Lets look and see what files we can find under /lib/modules."Kernel_version":


Table 1. /lib/modules listing

Table 1 shows us both directories and files related to LKM's. The only ones
listed that are not directories are modules.dep, modules.isapnpmap,
modules.pcimap and mosules.usbmap. Those are actual files that list modules
within the system (combined). Lets take a quick look at the net directory 

(This is not a complete listing, due to space):

Table 2. net directory

There are a couple of important bits of information we can gather from this
table. The first bit is, modules are listed as .o. 


Because they are object files that contain the actual module itself. The
second bit is, the listing is not complete. I did not have room to put all
of the modules.

The second important directory/file we want to look at it
/etc/conf.modules. Conf.modules is the configuration file that allows the
system administrator to specify a variety of parameters that control the
loading of the modules. The conf.modules file typically looks like this:

Table 3. conf.modules file

The conf.modules file allows the administrator to assign alias to commonly
used modules. 

This file is not required. A system administrator can create his/her own
configuration file. A system administrator can do that by using the
modprobe -C command.

Since I mentioned modprobe earlier, lets briefly take a look at modprobe
and the other tools available in Linux that load and unload modules.

The first tool is modprobe. Modprobe can load single and multiple modules.
It uses the modules.dep file to look up dependencies. Depmod creates a
"Makefile" like dependency file. This file is called modules.dep.  

The last three commands I will cover are lsmod, insmod and rmmod. lsmod
provides the administrator a listing of all modules currently loaded in the
kernel. This list can also be found at /proc/modules. 

This is helpful if you want to figure out what modules are currently
running. You will see a little later that lsmod is not always a good tool
to use for the detection of rootkit LKM's. insmod loads modules (seems

Some of the "trojaned" LKM code I have seen uses insmod -f to load the
"trojaned" module. rmmod is normally used to remove modules from the
kernel. Normally rmmod will NOT remove any "trojaned" LKM. 

We will discuss details of that a little bit later. 

Hopefully, this section provided everybody the basics of Loadable Kernel

If this was not detailed enough see <http://www.kernel.org/LDP/>.

LKM rootkits and kstat

Recently, there has been a lot of press about adore. Well, the worm adore
not the LKM adore. So I decided to look into the LKM adore (everyone else
has looked at the worm). BTW, if want to download adore go to

Adore is a Linux LKM rootkit. 

It is easy to install and only requires a few minor adjustments when
configuring. Adore can be installed with a default configuration or the
user can make changes to some of the code. 

The readme file recommends that the user change the settings for ELITE_CMD
and for HIDDEN_PORT. 

When you run ./configure, adore asks you for a password. This is for the
backdoor port (hence, change the HIDDEN_PORT) If you want more information
on the install you will have to download it. 

I ran the default installation, which consisted of running./configure and
make. After running make, you will see two files (ava, startadore). In
order for adore too execute you have to run startadore. Once you run
startadore the user can then run ava. Table 4 is the output from ./ava.

Usage: ./ava {h,u,r,R,i,v,U} [file, PID or dummy (for U)]

h hide file
u unhide file
r execute as root
R remove PID forever
U uninstall adore
i make PID invisible
v make PID visible

Table 4. Ava output

Table 4 shows us the options that adore provides to us. I will not cover
each switch and what it does only because it does not help us detect this
rootkit. Now that we have covered the basics of adore (LKM) lets look at
how we detect adore and other rootkits.

Many rootkits hide processes, directories, files and even connections. But
many of them do so by modifying the source code of binaries such as ps, df,
netstat, top and lsof. There are a couple of ways to detect these types of
rootkits (i.e. t0rn):

1) md5 checksums

2) Compiling these binaries from a known good source (i.e. cd-rom, disk).

3) Some rootkits have a default port that an administrator can focus in on.

4) Running a program such as chkrootkit

Many of the techniques used to detect rootkits like t0rn are not effective
against LKM rootkits (chkrootkit can detect some of them). Since LKM
rootkits access the kernel, they can hide processes, connections,
directories and files without modifying the binaries. 

MD5 checksums become useless because there are no files being modified.
This means the checksums will not change. 

Checking ports MIGHT be an option BUT that only tells you that you have
been "rooted."

Well, how can I detect these monsters? Easy. 

There is a program I highly recommend to everyone concerned with LKM
rootkits. This program is called KSTAT. You can find it at
<http://s0ftpj.org/en/site.html>. KSTAT works by checking the memory
(/dev/kmem) for information about the host(including LKMs). 

Want to see what kstat looks like? Good. Here is the output for kstat:

Usage: kstat [-i iff] [-P] [-p pid] [-M] [-m addr] [-s]

-i    iff may be specified as 'all' or as name (e.g. eth0) displays info
about the queried interface

-P    displays all processes

-p    pid is the process id of the queried task

-M    displays the kernel's LKMs' linked list

-m    addr is the hex address of the queried module displays info about the
module to be found at addr

-s    displays info about the system calls' table

As you can see, kstat provides a person with many options for detecting
these rootkits. Lets go through some of the options and from these options
we will learn how to detect the following LKM rootkits:

1) knark

2) adore

3) rkit

BTW, if there's any rootkits I have omitted please let me know and I will
update the paper. : ) 
["Michael 'fr0sty' Kummer" <frost at PACKETST0RM.NET>]

Kstat -s seems to be the best way to detect LKM rootkits. The other options
are really great, but -s works all the time. 

Here is an output from kstat -s:

SysCall                         Address
sys_exit                        0xc0117ce4
sys_fork                        0xc0108ebc
sys_read                        0xc012604c
sys_write                       0xc0126110
sys_open                        0xc0125c10
sys_close                       0xc0125d60
sys_waitpid                     0xc0117ff8
sys_creat                       0xc0125ca4
sys_link                        0xc012de60
sys_unlink                      0xc012dc90
sys_execve                      0xc0108f18
sys_chdir                       0xc01254a0
sys_time                        0xc01184b4
sys_mknod                       0xc012d77c
sys_chmod                       0xc01256e4

Table 5. kstat -s

Table 5 is not a complete output from kstat -s, but it does give us an idea
as too what the output looks like. Remember -s provides us with a picture
of the sys_call_table. Keep this information fresh as we will revisit this
again later.

Kstat -P is another switch that is very effective. Kstat -P shows us all of
the processes running at that time. This includes the processes hidden by
an LKM rootkit.

Table 6 is an example of kstat -P:

1     0      0   0     init
2     1      0   0     kflushd
3     1      0   0     kupdate
4     1      0   0     kpiod
5     1      0   0     kswapd
6     1      0   0     mdrecoveryd
241   1      1   0     portmap
256   1      0   0     lockd
257  256     0   0     rpciod
266   1      0   0     rpc.statd
280   1      0   0     apmd
331   1      0   0     syslogd

Table 6. kstat -P

When I first ran this program I wasn't sure kstat could do what it said
but.it proved my doubts WRONG. Here is what I did to verify this switch. I
used ava -i 241(portmap). I then ran kstat -P. 

As you can see from table 6,  portmap(bold) shows up. I then ran
ps-ef(Table 7)and could not find it. Lsof also did not show it. 

Table 7 is the output from ps -ef:

root         1     0  0 Mar30 ?        00:00:06 init [3]
root         2     1  0 Mar30 ?        00:00:00 [kflushd]
root         3     1  0 Mar30 ?        00:00:00 [kupdate]
root         4     1  0 Mar30 ?        00:00:00 [kpiod]
root         5     1  0 Mar30 ?        00:00:00 [kswapd]
root         6     1  0 Mar30 ?        00:00:00 [mdrecoveryd]
root       256     1  0 Mar30 ?        00:00:00 [lockd]
root       257   256  0 Mar30 ?        00:00:00 [rpciod]
root       266     1  0 Mar30 ?        00:00:00 rpc.statd
root       280     1  0 Mar30 ?        00:00:00 /usr/sbin/apmd -p 10 -w 5
-W -s
root       331     1  0 Mar30 ?        00:00:00 syslogd -m 0
root       340     1  0 Mar30 ?        00:00:00 klogd
nobody     354     1  0 Mar30 ?        00:00:00 identd -e -o
nobody     357   354  0 Mar30 ?        00:00:00 identd -e -o
nobody     359   357  0 Mar30 ?        00:00:00 identd -e -o
nobody     360   357  0 Mar30 ?        00:00:00 identd -e -o
nobody     361   357  0 Mar30 ?        00:00:00 identd -e -o
daemon     372     1  0 Mar30 ?        00:00:00 /usr/sbin/atd
root       386     1  0 Mar30 ?        00:00:00 crond
root       404     1  0 Mar30 ?        00:00:00 inetd
root       418     1  0 Mar30 ?        00:00:00 lpd
root       462     1  0 Mar30 ?        00:00:00 sendmail: accepting
root       477     1  0 Mar30 ?        00:00:00 gpm -t ps/2
root       491     1  0 Mar30 ?        00:00:01 httpd
xfs        531     1  0 Mar30 ?        00:00:00 xfs -droppriv -daemon -port -1
root       571     1  0 Mar30 tty1     00:00:00 login -- root
root       572     1  0 Mar30 tty2     00:00:00 /sbin/mingetty tty2
root       573     1  0 Mar30 tty3     00:00:00 /sbin/mingetty tty3
root       574     1  0 Mar30 tty4     00:00:00 /sbin/mingetty tty4
root       575     1  0 Mar30 tty5     00:00:00 /sbin/mingetty tty5
root       576     1  0 Mar30 tty6     00:00:00 /sbin/mingetty tty6
nobody    4290   491  0 Apr01 ?        00:00:00 httpd
nobody    4291   491  0 Apr01 ?        00:00:00 httpd
nobody    4292   491  0 Apr01 ?        00:00:00 httpd
nobody    4293   491  0 Apr01 ?        00:00:00 httpd
nobody    4294   491  0 Apr01 ?        00:00:00 httpd
nobody    4295   491  0 Apr01 ?        00:00:00 httpd
nobody    4298   491  0 Apr01 ?        00:00:00 httpd
nobody    4299   491  0 Apr01 ?        00:00:00 httpd
root      8073   571  0 Apr02 tty1     00:00:00 -bash
root     10659  8073  0 11:24 tty1     00:00:00 ps -ef

Table 7. ps -ef output

There are two other switches we will go over, they are kstat -p and kstat
-M. First, lets look at kstat -p. kstat -p gives us more information about
a process. In order to run kstat -p you will need to provide a process id.
For example: kstat -p 241. This checks process 241 (portmap) and provide an
output. Sometimes it can provide us with more information about a LKM

TABLE 8 is the output from kstat -p 241:

Name:     portmap
State:    S (sleeping)
Pid:      241
Ppid:     1 (init)
Uid:      1     1     1     1
Gid:      0     0     0     0
Crucial Capabilities Check
Open Files
0     CHAR        /dev/null
1     CHAR        /dev/null
2     CHAR        /dev/null
7     FIFO        ///
8     FIFO        ///
21    CHAR        /dev/null

Table 8. kstat -p output

Finally, lets look at kstat -M. Under normal conditions an administrator
can perform lsmod or more /proc/modules and find out what modules he/she
are running. When a machine has been "rooted" you can't trust lsmod for
accurate information. Kstat -M will catch many of the basic LKM rootkits
that I have tested. 

Kstat -M list all of the modules loaded. I have seen it where kstat -M did
list anything for knark, but.nothings perfect. The output from kstat -M is
similar to lsmod.

Detecting knark, adore and other LKM rootkits

So far we have covered a lot of material about LKM's, rootkits and kstat.
Now we are going to put all of that together and learn how to detect some
of the LKM rootkits available today.

The first LKM rootkit we want to look at is knark. Knark is probably the
best-known LKM rootkit and one of the best written as well.

Detecting it with kstat is fairly straight up as well. Remember Table 5?
Well in order to detect a knark LKM rootkit you will need to run kstat -s.

Once run, you need to look for the following:

sys_fork         0xc284652c WARNING! Should be at 0xc0108c88
sys_read         0xc2846868 WARNING! Should be at 0xc012699c
sys_execve       0xc2846bb8 WARNING! Should be at 0xc0108ce4
sys_kill         0xc28465d4 WARNING! Should be at 0xc01106b4
sys_ioctl        0xc2846640 WARNING! Should be at 0xc012ff78
sys_settimeofday 0xc2846a8c WARNING! Should be at 0xc0118364
sys_clone        0xc2846580 WARNING! Should be at 0xc0108ca4

Table 9. knark detection

Lets take a closer look at this table and see what all of this mess means
to us. First, we see that there are seven (7) sys_call_table entries that
have warnings. 

Lets look at sys_settimeofday, here we see that currently it is making it's
home in memory at 0xc2846a8c. kstat -s tells us that this is wrong and it
should be at 0xc0118364. When knark  is installed it changes the
sys_call_table and the memory locations where you can find: sys_fork,
sys_read, sys_execve, sys_kill, sys_ioctl, sys_settimeofday and sys_clone. 

This is how you could detect knark. Knowing which sys_call_table entries
have been changed could help you identify the actual rootkit itself. Knark
changes seven(7) total. After running kstat -s, the next step would be to
run ps-ef along with kstat -P and compare the two. If the are any
differences in the two you can then take the correct action. 

Keep in mind that memory locations can change from box to box. As I said
earlier, detecting LKM rootkits with kstat is quite simple.

Lets look at adore. Table 10 will show us what adore changes:

sys_fork       0xc4051428 WARNING! Should be at 0xc0108c88
sys_write      0xc4051590 WARNING! Should be at 0xc01269b8
sys_close      0xc405163c WARNING! Should be at 0xc01264a4
sys_kill       0xc40514d0 WARNING! Should be at 0xc011060c
sys_mkdir      0xc405172c WARNING! Should be at 0xc012e540
sys_clone      0xc405147c WARNING! Should be at 0xc0108ca4
sys_getdents   0xc40512a4 WARNING! Should be at 0xc013022c

Table 10. Adore detection

Adore changes seven (7) entries as well (just like knark). Knark and adore
only change three (3) of the same sys_call_table entries.

They are sys_fork, sys_kill, sys_clone. 

This is a good thing because it allows us the ability to detect each one
separately. Again, once this has been done, the administrator can then run
kstat -P and ps-ef and compare the two to figure out what processes are
running and hiding.

The last rootkit we will look at is rkit. This rootkit does not hide itself
as well as the other two do. As a matter of fact it only changes sys_setuid
and can be found using kstat -M.


LKM rootkits can make a system administrator's life a nightmare. They are
hard to detect, but using tools like kstat and understanding what the
rootkit changes can make our life easier. Since tools like kstat are
available, it would help systems administrators if they took a "picture" of
the sys_call_table after a fresh install and any upgrades. This will help
in identifying the good from the bad.  

Although I have not had the time, one could write a shell | perl script to
automate this process and check weekly, daily, every 10 minutes or whatever.




LKM rootkits


Toby Miller is a GIAC Certified Analyst and MCP. He is currently working
towards his CISSP and RHCE. Toby has contributed to 2 books, written papers
for SANS, Securityfocus, performs Risk Assessments and runs a lab for a
living. For entertainment Toby likes to analyze exploit signatures on his
home network. Toby can be reached at <tmiller at va.prestige.net>.

More information about the nflug mailing list