Protecting linux servers from malware with ClamAV and rkhunter

Table of contents

clamscan [+eicar test file, limit scan dirs + problematic device files, systemd freshclam service & manual freshclam] => error msgs when forgetting to use pkgmgr flag

clamdscan [+rerun speed improvement, fdpass, multiscan, separate apt pkg & service] => takes muhc more resources, run limited or at night as it may impact service quality;; limit to user files you really need to scan

clamdtop (clamd only)

rkhunter

cronjob scan script w/ notify

Installing ClamAV

The first step in virus protection on linux is typically running clamav, as it is the most widely used antivirus software for the platform. On debian-like systems, it can be installed from the main package repositories:

sudo apt install clamav

Since clamav relies on a database of virus definitions for scanning, you will need to keep this information updated over time. YOu can manually update the virus definition database using the freshclam command:

sudo freshclam

For a more robust setup, this should run regularly. The debian package ships with a systemd service called clamav-freshclam that will take care of regularly updating the database in the background:

sudo systemctl enable clamav-freshclam.service
sudo systemctl start clamav-freshclam.service

To verify that the commands worked, look at the service status:

systemctl status clamav-freshclam.service

It should look like this:

Scanning for viruses with ClamAV

Scanning files or directories for viruses is done using the clamscan command:

clamscan .

The default output is quite verbose:

Scans will typically use the -r flag to scan directories recursively. To reduce the output down to the list of files that are infected, you can add the -i flag:

clamscan -r -i .

This command only prints the files that are suspected of being malicious, one per line, followed by the scan summary at the end.

If you want to verify that your clamscan is working properly, you can run it against the EICAR test file:

wget https://secure.eicar.org/eicar.com
clamscan eicar.com

The output should report the file:

/home/demo/eicar.com: Win.Test.EICAR_HDB-1 FOUND

----------- SCAN SUMMARY -----------
Known viruses: 8693895
Engine version: 1.0.3
Scanned directories: 0
Scanned files: 1
Infected files: 1
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 13.305 sec (0 m 13 s)
Start Date: 2024:06:04 14:47:54
End Date:  2024:06:04 14:48:07

For a more robust test, place the file within a directory you would like to scan and verify that clamav correctly finds and reports it.


As a last note, full system scans of the root directory / are discouraged, as clamav may cause problems when scanning special system or device files in /dev, /proc etc. You should be careful when scanning these directories (or avoid them altogether):

  • /dev
  • /proc
  • /sys
  • /run
  • /tmp
  • /var/run
  • /var/lock
  • /var/tmp
  • /mnt
  • /media
  • /lost+found

These directories are typically safe to scan:

  • /home
  • /usr
  • /var (excluding /var/run, /var/lock, /var/tmp)
  • /etc
  • /opt
  • /srv

Improving scan performance with clamd

The clamscan utility uses only a single thread for virus scanning. For improved performance, you can use the clamav-daemon with the clamdscan utility. Install it with

sudo apt install clamav-daemon clamdscan
sudo systemctl enable clamav-daemon.service
sudo systemctl start clamav-daemon.service

The clamdscan command (note the 'd') behaves very similar to the clamscan command, except that it passes files to the local clamav-daemon service for scanning:

clamdscan --fdpass -m -i .

The output of the command is very similar to clamscan. Note that the -r flag is not required (clamdscan is recursive by default). Additionally, the -m flag allows multiscan mode (using multiple threads to scan several files at once).

The --fdpass flag is important if you are scanning files not owned by the user clamav-daemon is running as (clamav by default on debian). Unless you are scanning files that are readable by that user, the --fdpass flag will open the file as the user you are currently logged in as, and pass the file descriptor to the clamav-daemon for scanning, thus circumventing the file permission issue. If you forget to use the flag when needed, you will see errors like this:

/home/demo/.bash_history: Access denied. ERROR
/home/demo/.local/share: File path check failure: Permission denied. ERROR

There is no harm in passing the --fdpass flag, even when not necessary, so you can always include it in your clamdscan commands.

Since it can be difficult to judge the current process of clamav-daemon, it ships with the clamdtop command. This can be used to get a quick overview of running scans:

The interface is quite minimal, showing information about thread and hardware usage, and what commands are currently being processed.

Finding rootkits with rkhunter

Finding viruses is one part of malware defense, but what if the virus is found too late? If system files were already modified, removing the virus alone will not undo that damage. The rkhunter command is used to find more subtle changes that may be caused by virus infection (or at least things that are suspicious). Install it with

sudo apt install rkhunter

To check the entire system for suspicious files, run

sudo rkhunter --check --pkgmgr DPKG

This will print a long list of files as they are being checked, prompting you to confirm after each category, and ends with a summary of the scan:

System checks summary
=====================

File properties checks...
   Files checked: 144
   Suspect files: 1

Rootkit checks...
   Rootkits checked : 476
   Possible rootkits: 2

Applications checks...
   All checks skipped

The system checks took: 1 minute and 5 seconds

All results have been written to the log file: /var/log/rkhunter.log

One or more warnings have been found while checking the system.
Please check the log file (/var/log/rkhunter.log)

Be sure to include the --pkgmgr DPKG flag (adjust if your distribution does not use apt/dpkg)! If you forget to add it on debian, you will get false positives like these:

Warning: The command '/usr/bin/lwp-request' has been replaced by a script: /usr/bin/lwp-request: Perl script text executable
Warning: Hidden directory found: /etc/.java

To automate rkhunter scans a little more, you can use to --sk flag to skip manual keypress confirmation during scanning, and --rwo to only print warnings without verbose information about successful scans.

sudo rkhunter --check --sk --rwo --pkgmgr DPKG

Setting up cronjobs to automate scanning

Before setting up a cronjob, think about two things:

  • What files do you need to scan? ClamAV (and especially clamd) can be quite taxing on the CPU and disk it is using, which may negatively impact production systems on the server. Files that are being changed at runtime should be scanned regularly, but if you can avoid some larger chunks like backup directories or log files, you may consider not including them.
  • When do scans run? Scans can use a lot of hardware resources, so running them during peak hours may cause issues for your users. Consider running scans outside of business hours, or at times with less user traffic.

To run both clamdscan and rkhunter from a script, you can use a clever design feature: if they find a threat to report, they will exit with a non-zero exit code; if they find nothing the exit code will be 0. An automated script may look like this:

scan.sh

#!/bin/bash

SERVER_NAME=$(hostname)
DISCORD_WEBHOOK="https://discord.com/api/webhooks/<your webhook url>"

# helper function to send discord webhooks
function discord_webhook() {
   local MESSAGE=$1
   local JSON_PAYLOAD=$(jq -n --arg username "$SERVER_NAME" --arg content "$MESSAGE" \
                       '{username: $username, content: $content}')
   curl -X POST -H "Content-Type: application/json" -d "$JSON_PAYLOAD" $DISCORD_WEBHOOK
}

# rkhunter rootkit scan
RKHUNTER_OUTPUT=$(sudo rkhunter -c --rwo --sk --pkgmgr DPKG)
if [ $? -ne 0 ]; then
   discord_webhook "$(printf "%s\n%s" "rkhunter found problems:" "$RKHUNTER_OUTPUT")"
fi

# clamav virus scan
CLAMAV_OUTPUT=$(sudo clamdscan -i /home /usr /var /etc /opt /srv)
if [ $? -ne 0 ]; then
   discord_webhook "clamscan found problems:\n$CLAMAV_OUTPUT"
fi

This script simply runs both scans and sends a notification webhook to a discord channel if anything is found. You can easily adjust this to trigger a webhook to a company slack/mattermost server or send an email message instead.

The webhook includes the output of the command that found a threat, to minimize the time needed to route the issue to the correct person and debug the cause.

To use it, you should move it somewhere accessible, like /usr/local/bin:

sudo mv scan.sh /usr/local/bin/avscan.sh
sudo chmod +x /usr/local/bin/avscan.sh

For most servers, running scans once per night will be sufficient and cause minimal impact for other services on the system. Edit your cronjobs with

crontab -e

And add a job to run the antivirus scanning script once per night, at 1am:

0 1 * * * /usr/local/bin/avscan.sh

More articles

Creating an effective disaster recovery plan for IT infrastructure

Know what to do when things go wrong ahead of time

Dealing with pagination queries in SQL

The difficult choice behind chunking results into simple pages

Understanding postgres query plans

Making sense of postgres explanations

Modern linux networking basics

Getting started with systemd-networkd, NetworkManager and the iproute2 suite

Understanding how RAID 5 works

Striking a balance between fault tolerance, speed and usable disk space

A practical guide to filesystems

Picking the best option to format your next drive