Work Together For The Benefit Of All ManKind…

I see sloppy Administrators do this again and again…

They might update the Linux-Xen enabled Kernel on Dom0, but often DomU keeps the same for different reasons.

Running a (para) virtual environment, the freedom of running different Linux distributions, is often a goal. If one keeps a single architect environment stack, like Ubuntu Hardy Dom0 and DomU’s or CentOS 5.x Dom0 and DomU, keeping kernels in DomU up to date is low hassle.

The hassle starts to arise when you deploy mixed environments, like running Ubuntu Hardy as Dom0 and CentOS 5.x as DomU, or vice versa. You could setup CentOS or Ubuntu to use each others Kernel packages, though that seemed a bit overkill for my setup. Having a Debian Etch DomU on a Ubuntu Hardy Dom0 is fixable with pointing Etch to grab the Kernel from Hardy via an apt-repo.

PyGrub solves some hassles, so I recommend reading up on that and verifying that CVE-2007-4993 is not affecting you.

But for the cases where I have a bit hassle, and I dont want to use PyGrub, I wrote a small bash script to update the Linux Kernels.
Get the script here, and update/change/modify or learn from it, before you use it.
It Powers down the DomU if it is booted, and mounts the Logical Volume of the DomU, before it copies the kernel modules to the DomU filesystem. Runs depmod and unmounts the filesystem. Then it gives you the small change you need to update your xen-domU.cfg with (I dont use pygrub).

BTW: This paper has a nice walk through from Xen DomU to Xen Dom0 bypassing SELinux http://invisiblethingslab.com/pub/xenfb-adventures-10.pdf. Recommended read

Now go and update some Kernels!

BTW: Upgrading might not be enough…

After upgrading (up2date, yum, apt, …) my Linux systems, I check with lsof to see if any processes needs a restart…

Why?

Because, running processes might still be using old libraries and binaries etc, and would need a restart to use the new ones…

So… You might be vulnerable, even if you do install security updates regularly…

On older versions of lsof, I used to issue: lsof +L1|grep DEL
This does not seem to be sufficient on newer versions of lsof… Might be a bug?

After searching the web for information for an easier or better way of doing this, I found little… I even did not find any good info on the way I am used of doing it… If you have a smarter way of checking this, I would love to hear from you…

Here are some references to what I found:
* A bugzilla thread on redhat.com. It also has a script for redhat based systems.
* Debian/Ubuntu based systems comes with debian-goodies… apt-get install debian-goodies and then you can use checkrestart. Which checks for programs that needs restart

To manually check, here are some commands you can issue, depending on your version of lsof.

# lsof -n +L | grep -w DEL | egrep -v ” (/dev|/SYSV|/tmp)”
# lsof -n | grep “path inode=”
# lsof -n +L1 | egrep -w “txt|mem” | grep -v ” /SYSV”
# lsof -n +L | grep -w DEL | egrep -v ” (/dev|/SYSV|/tmp) ”

Hope you make this check a habit after updating your servers…

You might think that you are safe, but something might be lurking on your system…

There are several ways to protect you from being 0wned by 5kr1p7 kiddies or more 1337 crackers. But still, your system might get 0wned, and you might not know it…

First, you should have a system that can update it self with new security patches/fixes. Red Hat/CentOS, Debian/Ubuntu and so on (Most modern systems today), have this functionality. Then you should use it! This is probably the best way to prevent unauthorized access to you systems.

Then you should probably know a thing or two on how to configure your system to be as secure as you need it to be…

If you have a system with lots of user accounts, may it be ssh, ftp, mail etc, then it might just be time, before someone hijacks an account or two…
Say if someone got login to your server as a normal user, they can misuse your system, generating very little noise, and you might not know about it.

Anyways, back to the point of this posting, checking your systems for things you might not know about…

In my basic toolkit, I use chkrootkit, rkhunter (You could also read here), lynis and unix-privesc-check. I also use ClamAV (clamscan) to scan the file system for suspicious files. I also have some one-liners (baked into a bash script) that extracts some interesting things based on system processes and the file system.

For more advanced “Host based Intrusion Detection”, I recommend that you look at OSSEC. You could also look at Aide and tools alike. RPM based distros like Red Hat, Fedora and CentOS can to an extent use the rpm command to verify installed packages and their signatures.

I will not go into details on how to use any of the tools that I mentioned. If you care, you should pursue the links, and even test the tools.

I will give a brief overview (Copy and paste from their websites), so you know a bit what they are all about:

chkrootkit: chkrootkit is a tool to locally check for signs of a rootkit.

rkhunter: Rootkit Hunter (RKH) is an easy-to-use tool which checks computers running UNIX (clones) for the presence of rootkits and other unwanted tools.

lynis: Lynis is an auditing tool for Unix (specialists). It scans the system and available software, to detect security issues. Beside security related information it will also scan for general system information, installed packages and configuration mistakes. This software aims in assisting automated auditing, software patch management, vulnerability and malware scanning of Unix based systems. It can be run without prior installation, so inclusion on read only storage is no problem (USB stick, cd/dvd).

unix-privesc-check: Unix-privesc-checker is a single bash script that runs on Unix systems. It tries to find misconfigurations that could allow local unprivileged users to escalate privileges to other users or to access local apps (e.g. databases).

ClamAV: Clam AntiVirus is an open source (GPL) anti-virus toolkit for UNIX, designed especially for e-mail scanning on mail gateways. It provides a number of utilities including a flexible and scalable multi-threaded daemon, a command line scanner and advanced tool for automatic database updates.

OSSEC: OSSEC is a scalable, multi-platform, open source Host-based Intrusion Detection System (HIDS). It has a powerful correlation and analysis engine, integrating log analysis, file integrity checking, Windows registry monitoring, centralized policy enforcement, rootkit detection, real-time alerting and active response. It runs on most operating systems, including Linux, OpenBSD, FreeBSD, MacOS, Solaris and Windows.

Aide: Advanced Intrusion Detection Environment. AIDE is an intrusion detection system that detects changes to files on the local system. It creates a database from the regular expression rules that it finds from the config file. Once this database is initialized it can be used to verify the integrity of the files. It has several message digest algorithms (md5, sha1, rmd160, tiger, haval, etc.) that are used to check the integrity of the file. More algorithms can be added with relative ease. All of the usual file attributes can also be checked for inconsistencies.

If you find any thing suspicious (Like a rootkit), you should probably go into “forensic mode”, as you cant trust your system and the installed binaries. Read more here about computer forensics.

If you still do not trust your system, you can Snort your network traffic, or better yet, have a full blown Sguil installation in front of your network/servers. If you get even more paranoid, you should probably shut down the system, and go fishing….

Any suggestions on other useful tools are welcome!

I have been running CentOS 5 with Xen virtualization since around mid April (I started to play with Redhat 5 beta and its Xen in Desember 2006). CentOS 5 ships with Xen-3.0.3 and with RedHat’s virt-manager, which is a very simple but OK tool to manage Xen guests/domUs. The only 1337 about virt-manager at this point, in my opinion, is the vnc integration.

A quick way to get up and running (if you are connected to internet), is to set up dhcpd on Dom0 and export an kickstart.cfg file by httpd etc. That is, if you dont have a dhcp on you network. If you do have a dhcp-server on your network, you should drop the dhcp part.

# yum install dhcp httpd
# cp /root/anaconda-ks.cfg /var/www/html/xen.cfg
# vi /var/www/html/xen.cfg

Change the permision of the file and remove etc. cdrom (install media) and %packages that you dont need. Driveorder should be xvda –driveorder=xvda.
Also add url –url http://your-centos-mirror.something.com/centos/5/os/{ARCH}. http://www.centos.org/modules/tinycontent/index.php?id=13 has a list of mirrors

# vi /etc/dhcpd.conf # setup you own network:

ddns-update-style interim;
ignore client-updates;
subnet 10.10.10.0 netmask 255.255.255.0 {
# — default gateway
option routers 10.10.10.1;
option subnet-mask 255.255.255.0;
option domain-name “v.gamelinux.org”;
option domain-name-servers 10.10.10.2;
# we want control over DomU’s adress
host node1 {
next-server node1.gamelinux.org;
hardware ethernet 00:16:3e:3e:c2:fd,;
fixed-address 10.10.10.101;
}
host node2 {
next-server node2.gamelinux.org;
hardware ethernet 00:16:3e:4a:6f:87;
fixed-address 10.10.10.102;
}
}

# service dhcpd start
# service httpd start

For some reason I like to have my DomU’s on a logical volume. I do not recommend this yet, cuz I lack testing, but here is how I end up doing it:

# lvcreate -L 10G -n LogVolN1 VolGroup00
# lvcreate -L 10G -n LogVolN2 VolGroup00

My setup, needs me to tail /var/log/messages to get the MAC address of the nodes (when you start the node installation), then edit the dhcp.conf and restart dhcpd. You could just setup a dhcp-pool.

Then you just fire up virt-manager, enter system name (node1 etc), choose your flavor of virtualization, and point to a centos mirror (http://mirror.hh.se/centos/5.0/os/i386/ ect.). The kickstart URL, should be the ip/host of you Dom0 (http://dom0.v.gamelinux.org/xen1.cfg etc.). Then choose normal disk partition : and use the lvm we just created (/dev/VolGroup00/LogVolN1). Choose an amount of memory and cpus, and the start the installation.

How much cpu and memory you give your guests are all up to you!