c0d3 :: j0rg3

A collection of tips, tricks and snips. A proud Blosxom weblog. All code. No cruft.

Sun, 13 Jul 2014

Simple Protection with iptables, ipset and Blacklists

Seems I’ve always just a few more things going on than I can comfortably handle. One of those is an innocent little server holding the beginnings of a new project.

If you expose a server to the Internet, very quickly your ports are getting scanned and tested. If you’ve an SSH server, there are going to be attempts to login as ‘root’ which is why it is ubiquitously advised that you disable root login. Also why many advise against allowing passwords at all.

We could talk for days about improvements; it’s usually not difficult to introduce some form of two-factor authentication (2FA) for sensitive points of entry such as SSH. You can install monitoring software like Logwatch which can summarize important points from your logs, such as: who has logged via SSH, how many times root was used, etc.

DenyHosts and Fail2ban are very great ways to secure things, according to your needs.

DenyHosts works primarily with SSH and asks very little from you in way of configuration, especially if you’re using a package manager to install a version that is configured for the distribution on which you’re working. If you’re installing from source you may need to find where are your SSH logs (e.g., /var/log/secure, /var/log/auth.log). It’s extremely easy to set up DenyHosts to synchronize so that you’re automatically blocking widely-known offenders whether or not they’re after your server.

In contrast, Fail2ban is going to take more work to get set up. However, it is extremely configurable and works with any log file you point it toward which means that it can watch anything (e.g., FTP, web traffic, mail traffic). You define your own jails which means you can ban problematic IP addresses according to preference. Ban bad HTTP attempts from HTTP only or stick their noses in the virtual corner and don’t accept any traffic from them until they’ve served their time-out by completely disallowing their traffic. You can even use Fail2ban to scan its own logs, so repeating offenders can be locked out for longer.

Today we’re going to assume that you’ve a new server that shouldn’t be seeing any traffic except from you and any others involved in the project. In that case, you probably want to block traffic pretty aggressively. If you’ve physical access to the server (or the ability to work with staff at the datacenter) then it’s better to err in the direction of accidentally blocking good guys than trying to be overly fault-tolerant.

The server we’re working on today is a Debian Wheezy system. It has become a common misconception that Ubuntu and Debian are, intents and purposes, interchangeable. They’re similar in many respects and Ubuntu is great preparation for using Debian but they are not the same. The differences, I think, won’t matter for this exercise but I am unsure because this was written using Wheezy.

Several minutes after bringing my new server online, I started seeing noise in the logs. I was still getting set up and really didn’t want to stop and take protective measures but there’s no point in securing a server after its been compromised. The default Fail2ban configuration was too forgiving for my use. It was scanning for 10 minutes and banning for 10 minutes. Since only a few people should be accessing this server, there’s no reason for anyone to be trying a different password every 15 minutes (for hours).

I found a ‘close-enough’ script and modified it. Here, we’ll deal with a simplified version.

First, lets create a name for these ne’er-do-wells in iptables:
iptables -N bad_traffic

For this one, we’ll use Perl. We’ll look at our Apache log files to find people sniffing ‘round and we’ll block their traffic. Specifically, we’re going to check Apache’s ‘error.log’ for the phrases ‘File does not exist’ and ‘client denied by server configuration’ and block people causing those errors. This would be excessive for servers intended to serve the general populace. For a personal project, it works just fine as a ‘DO NOT DISTURB’ sign.


#!/usr/bin/env perl
use strict;
use POSIX qw(strftime);

my $log = ($ARGV[0] ? $ARGV[0] : "/var/log/apache2/error.log");
my $chain = ($ARGV[1] ? $ARGV[1] : "bad_traffic");

my @bad = `grep -iE 'File does not exist|client denied by server configuration' $log |cut -f8 -d" " | sed 's/]//' | sort -u`;
my @ablk = `/sbin/iptables -S $chain|grep DROP|awk '{print $4}'|cut -d"/" -f1`;

foreach my $ip (@bad) {
if (!grep $_ eq $ip, @ablk) {
chomp $ip;
`/sbin/iptables -A $chain -s $ip -j DROP`;
print strftime("%b %d %T",localtime(time))." badht: blocked bad HTTP traffic from: $ip\n";
}
}

That gives us some great, utterly unforgiving, blockage. Looking at the IP addresses attempting to pry, I noticed that most of them were on at least one of the popular block-lists.

So let’s make use of some of those block-lists! I found a program intended to apply those lists locally but, of course, it didn’t work for me. Here’s a similar program; this one will use ipset for managing the block-list though only minor changes would be needed to use iptables as above:

#!/bin/bash
IP_TMP=ip.tmp
IP_BLACKLIST_TMP=ip-blacklist.tmp

IP_BLACKLIST=ip-blacklist.conf

WIZ_LISTS="chinese nigerian russian lacnic exploited-servers"

BLACKLISTS=(
"http://danger.rulez.sk/projects/bruteforceblocker/blist.php" # BruteForceBlocker IP List
"http://rules.emergingthreats.net/blockrules/compromised-ips.txt" # Emerging Threats - Compromised IPs
"http://www.spamhaus.org/drop/drop.txt" # Spamhaus Don't Route Or Peer List (DROP)
"http://www.spamhaus.org/drop/edrop.txt" # Spamhaus Don't Route Or Peer List (DROP) Extended
"http://cinsscore.com/list/ci-badguys.txt" # C.I. Army Malicious IP List
"http://www.openbl.org/lists/base.txt" # OpenBL.org 90 day List
"http://www.autoshun.org/files/shunlist.csv" # Autoshun Shun List
"http://lists.blocklist.de/lists/all.txt" # blocklist.de attackers
)

for address in "${BLACKLISTS[@]}"
do
echo -e "\nFetching $address\n"
curl "$address" >> $IP_TMP
done

for list in $WIZ_LISTS
do
wget "http://www.wizcrafts.net/$list-iptables-blocklist.html" -O - >> $IP_TMP
done

wget 'http://wget-mirrors.uceprotect.net/rbldnsd-all/dnsbl-3.uceprotect.net.gz' -O - | gunzip | tee -a $IP_TMP

grep -o '^[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}[/][0-9]\{1,3\}' $IP_TMP | tee -a $IP_BLACKLIST_TMP
grep -o '^[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}[^/]' $IP_TMP | tee -a $IP_BLACKLIST_TMP

sed -i 's/\t//g' $IP_BLACKLIST_TMP
sort -u $IP_BLACKLIST_TMP | tee $IP_BLACKLIST

rm $IP_TMP
rm $IP_BLACKLIST_TMP

wc -l $IP_BLACKLIST

if hash ipset 2>/dev/null
then
ipset flush bloxlist
while IFS= read -r ip
do
ipset add bloxlist $ip
done < $IP_BLACKLIST
else
echo -e '\nipset not found\n'
echo -e "\nYour bloxlist file is: $IP_BLACKLIST\n"
fi


Download here:
    bad_traffic.pl
    bloxlist.sh


Tags: , , , , , , , , , ,
Permalink: 20140713.simple.protection.with.iptables.ipset.and.blacklilsts

Mon, 17 Feb 2014

Installing INN’s Project Largo in a Docker containter

Prereqruisites: Docker, Git, SSHFS.

Today we’re going to look at using Docker to create a WordPress installation with the Project Largo parent theme and a child theme stub for us to play with.

Hart Hoover has established an image for getting a WordPress installation up and running using Docker. For whatever reason, it didn’t work for me out-of-box but we’re going to use his work to get started.

Let’s make a place to work and move into that directory:
cd ~
mkdir project.largo.wordpress.docker
cd project.largo.wordpress.docker

We’ll clone the Docker/Wordpress project. For me, it couldn’t untar the latest WordPress. So we’ll download it outside the container, untar it and modify the Dockerfile to simply pull in a copy:
git clone https://github.com/hhoover/docker-wordpress.git
cd docker-wordpress/
ME=$(whoami)
wget http://wordpress.org/latest.tar.gz
tar xvf latest.tar.gz
sed -i 's/ADD http:\/\/wordpress.org\/latest.tar.gz \/wordpress.tar.gz/ADD \.\/wordpress \/wordpress/' Dockerfile
sed -i '/RUN tar xvzf \/wordpress\.tar\.gz/d' Dockerfile

Then, build the project which may take some time.
sudo docker build -t $ME/wordpress .

If you’ve not the images ready for Docker, the process should begin with something like:
Step 0 : FROM boxcar/raring
Pulling repository boxcar/raring
32737f8072d0: Downloading [> ] 2.228 MB/149.7 MB 12m29s

And end something like:
Step 20 : CMD ["/bin/bash", "/start.sh"]
---> Running in db53e215e2fc
---> 3f3f6489c700
Successfully built 3f3f6489c700

Once the project is built, we will start it and forward ports from the container to the host system, so that the Docker container’s site can be accessed through port 8000 of the host system. So, if you want to see it from the computer that you’ve installed it on, you could go to ‘HTTP://127.0.0.1:8000’. Alternatively, if your host system is already running a webserver, we could use SSHFS to mount the container’s files within the web-space of the host system.

In this example, however, we’ll just forward the ports and mount the project locally (using SSHFS) so we can easily edit the files perhaps using a graphical IDE such as NetBeans or Eclipse.

Okay, time to start our Docker image and find its IP address (so we can mount its files):
DID=$(docker run -p 8000:80 -d $ME/wordpress)
DIP=$(docker inspect $DID | grep IPAddress | cut -d '"' -f 4)
docker logs $DID| grep 'ssh user password:' --color

Copy the SSH password and we will make a local directory to access the WordPress installation of our containter.
cd ~
mkdir largo.mount.from.docker.container
sshfs user@$DIP:/var/www $HOME/largo.mount.from.docker.container
cd largo.mount.from.docker.container
PROJECT=$(pwd -P)

Now, we can visit the WordPress installation and finish setting up. From the host machine, it should be ‘HTTP://127.0.0.1:8000’. There you can configure Title, Username, Password, et cet. and finish installing WordPress.

Now, let’s get us some Largo! Since this is a test project, we’ll sacrifice security to make things easy. Our Docker WordPress site isn’t ready for us to easily install the Largo parent theme, so we’ll make the web directory writable by everybody. Generally, this is not a practice I would condone. It’s okay while we’re experimenting but permissions are very important on live systems!

Lastly, we’ll download and install Largo and the Largo child theme stub.
ssh user@$DIP 'sudo chmod -R 777 /var/www'
wget https://github.com/INN/Largo/archive/master.zip -O $PROJECT/wp-content/themes/largo.zip
unzip $PROJECT/wp-content/themes/largo.zip -d $PROJECT/wp-content/themes/
mv $PROJECT/wp-content/themes/Largo-master $PROJECT/wp-content/themes/largo
wget http://largoproject.wpengine.netdna-cdn.com/wp-content/uploads/2012/08/largo-child.zip -O $PROJECT/wp-content/themes/largo-child.zip
unzip $PROJECT/wp-content/themes/largo-child.zip -d $PROJECT/wp-content/themes
rm -rf $PROJECT/wp-content/themes/__MACOSX/

We are now ready to customize our Project Largo child theme!


Tags: , , , , , ,
Permalink: 20140217.project.largo.docker

Wed, 15 May 2013

Git: an untracked mess?

There may be times when you find your Git repository burdened with scads of untracked files left aside while twiddling, testing bug patches, or what-have-youse.

For the especially scatter-brained among us, these things can go unchecked until a day when the useful bits of a git status scroll off the screen due to utterly unimportant stuff. Well, hopefully unimportant.

But we’d better not just cleave away everything that we haven’t checked in. You wonder:
What if there’s something important in one of those files?

You are so right!

Let’s fix this!

Firstly, we want a solution that’s reproducible. Only want to invent this wheel once, right?

Let’s begin with the play-by-play:

Git, we want a list of what isn’t tracked: git ls-files -o --exclude-standard -z

We’ll back these files up in our home directory (~), using CPIO but we don’t want a poorly-named directory or finding anything will become its own obstacle. So we’ll take use the current date (date +%Y-%m-%d), directory (pwd) and branch we’re using (git branch) and we’ll twist all of it into a meaningful, but appropriate, directory name using sed. git ls-files -o --exclude-standard -z | cpio -pmdu ~/untracked-git-backup-`date +%Y-%m-%d`.`pwd | sed 's,^\(.*/\)\?\([^/]*\),\2,'`.`git branch | grep "*" | sed "s/* //"`/

Then Tell Git to remove the untracked files and directories: git clean -d -f

Ahhhh… Much better. Is there anything left out? Perhaps. What if we decide that moving these files away was a mistake? The kind of mistake that breaks something. If we realize right away, it’s easily-enough undone. But what if we break something and don’t notice for a week or two? It’d probably be best if we had an automated script to put things back the way they were. Let’s do that.

Simple enough. We’ll just take the opposite commands and echo them into a script to be used in case of emergency.

Create the restore script (restore.sh), to excuse faulty memory: echo "(cd ~/untracked-git-backup-`date +%Y-%m-%d`.`pwd | sed 's,^\(.*/\)\?\([^/]*\),\2,'`.`git branch | grep "*" | sed "s/* //"`/; find . -type f \( ! -iname 'restore.sh' \) | cpio -pdm `pwd`)" > ~/untracked-git-backup-`date +%Y-%m-%d`.`pwd | sed 's,^\(.*/\)\?\([^/]*\),\2,'`.`git branch | grep "*" | sed "s/* //"`/restore.sh

Make the restore script executable: chmod u+x ~/untracked-git-backup-`date +%Y-%m-%d`.`pwd | sed 's,^\(.*/\)\?\([^/]*\),\2,'`.`git branch | grep "*" | sed "s/* //"`/restore.sh

Lastly, the magic, compressed into one line that will stop if any command does not report success: a='untracked-git-backup-'`date +%Y-%m-%d`.`pwd | sed 's,^\(.*/\)\?\([^/]*\),\2,'`.`git branch | grep "*" | sed "s/* //"`; git ls-files -o --exclude-standard -z | cpio -pmdu ~/$a/ && git clean -d -f && echo "(cd ~/$a/; find . -type f \( ! -iname 'restore.sh' \) | cpio -pdm `pwd`)" > ~/$a/restore.sh && chmod +x ~/$a/restore.sh; unset a


Tags: , , , ,
Permalink: 20130515.git.untracked.mess