Serial console support in Grub2

This is another one of those “just so I don’t forget” posts. I’m working towards migrating from IPA in a KVM-based virtual machine running CentOS 6 to one running CentOS 7 and the way of making serial consoles work in 7 is different than it was in 6 because CentOS 7 (and thus RHEL7) use Grub2.

Instead of editing /boot/grub.cfg, you need to edit /etc/defaults/grub and add:

GRUB_TERMINAL="serial"
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200"

The above will enable Grub output to the serial console. Next, change the GRUB_CMDLINE_LINUX command to the following so you can use “virsh console” to connect to the guest:

GRUB_CMDLINE_LINUX="rhgb quiet console=tty0 console=ttyS0,115200"

Once you have saved the file, you need to recreate the grub.cfg file with grub2-mkconfig:

# grub2-mkconfig -o /boot/grub2/grub.cfg

Now, reboot the system and you should be able to use “virsh console [VMNAME]” to have a serial console to the guest.

Getting started with firewalld

I’m mostly writing this for my own reference as I spent a bunch of time figuring this out while I was on holidays with some serious oVirt misadventures and didn’t document any of what I did, so since I had to reinstall CentOS 7, I’m stuck doing this all over again.

Effectively I’m migrating from CentOS 6 to CentOS 7 and trying to take advantage of the new way of doing things. I could easily switch things to use /etc/sysconfig/iptables (that I can handle pretty much in my sleep) but it feels a bit like cheating, so I want to figure out firewalld which is to me a totally alien way of handling my firewall rules.

Suffice it to say that there may be better ways of doing this (and, to you the reader, probably far better tutorials), but this is how I went about it. This is on a CentOS 7.1 system.

First thing is to make sure firewalld is running and will run. It should be the default but never hurts to check:

[root@thor dhcp]# systemctl status firewalld
firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)
   Active: active (running) since Sat 2015-08-15 18:21:30 MDT; 18min ago
 Main PID: 745 (firewalld)
...

Ok, it’s active and enabled (in particular pay attention to the enabled bit that is bolded above) so on a reboot it will come up. firewalld has this concept of zones, so you can set certain ethernet interfaces to be assigned to a particular zone. The default zone is public, but as this is an internal DNS/DHCP/web server for my LAN, I want to set it to the internal zone:

[root@thor dhcp]# firewall-cmd --get-zones
block dmz drop external home internal public trusted work
[root@thor dhcp]# firewall-cmd --set-default-zone=internal
success

I also want to make sure that the one ethernet interface on this machine, enp2s0 (so long eth0!) is assigned the same zone. I believe this would happen by default, but it doesn’t hurt to be explicit:

[root@thor dhcp]# firewall-cmd --zone=internal --change-interface=enp2s0 --permanent
success

Now we can see how this zone is configured:

[root@thor dhcp]# firewall-cmd --zone=internal --list-all
internal (default, active)
  interfaces: enp2s0
  sources:
  services: dhcpv6-client ipp-client mdns samba-client ssh
  ports:
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:

Not a lot there right now and it’s definitely setup to be a workstation, not a server. You can see the available list of services that come pre-defined (they are XML files in /usr/lib/firewalld/services) by using:

[root@thor dhcp]# firewall-cmd --get-services

If you want to examine any of the listed services, just look at the XML file. For instance, /usr/lib/firewalld/services/dns.xml looks like:

<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>DNS</short>
  <description>The Domain Name System (DNS) is used to provide and request host and domain names. Enable this option, if you plan to provide a domain name service (e.g. with bind).</description>
  <port protocol="tcp" port="53"/>
  <port protocol="udp" port="53"/>
</service>

Pretty straightforward. When adding new services or rules, firewalld makes them temporary, in that they will persist until a reboot or service restart. You need to use the “–permanent” option to, well, make them permanent. Since this server is going to do DNS/DHCP/HTTP/HTTPS we need to do:

[root@thor dhcp]# firewall-cmd --permanent --zone=internal --add-service=http
success
[root@thor dhcp]# firewall-cmd --permanent --zone=internal --add-service=https
success
[root@thor dhcp]# firewall-cmd --permanent --zone=internal --add-service=dns
success
[root@thor dhcp]# firewall-cmd --permanent --zone=internal --add-service=dhcp
success
[root@thor dhcp]# firewall-cmd --reload
success
[root@thor dhcp]# firewall-cmd --zone=internal --list-all
internal (default, active)
  interfaces: enp2s0
  sources:
  services: dhcp dhcpv6-client dns http https ipp-client mdns samba-client ssh
  ports:
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:

In the above we added four services: HTTP, HTTPS, DNS, and DHCP. We then reloaded the rules (so as to apply them) and then listed the internal zone where we can now see our new services are enabled.

If you want to enable a service for which no default service definition exists, you can create one in /etc/firewalld/services/ as an XML file (copy a similar one from /usr/lib/firewalld/services/ and adjust for your service). If you’re using SELinux, be sure to run restorecon -Rv /etc/firewalld and make sure the XML file has 0640 permissions and is owned root:root.

Instead of making a new service, however, you can simply add ports to the configuration which might be easier. For instance, if I wanted to expose apcupsd but didn’t want to make a service I might do:

[root@thor dhcp]# firewall-cmd --zone=internal --add-port=3551/udp --permanent
success
[root@thor dhcp]# firewall-cmd --zone=internal --add-port=3551/tcp --permanent
success
[root@thor dhcp]# firewall-cmd --reload
success
[root@thor dhcp]# firewall-cmd --zone=internal --list-all
internal (default, active)
  interfaces: enp2s0
  sources:
  services: dhcp dhcpv6-client dns http https ipp-client mdns samba-client ssh
  ports: 3551/udp 3551/tcp
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:

For further reading, I recommend:

In brief, managing the firewall is a little bit annoying to do this way compared to the old way of “vim /etc/sysconfig/iptables; service iptables restart” but I can understand why it’s done this way now. It’s quite modular and adaptable and allows you to make temporary changes to the firewall easily without having to know iptables commands. There are a lot of commands, and since firewalls tend not to be things you tinker with often, they may not be overly memorable (thus deciding to write this after my second time of doing it in as many weeks as I didn’t remember a darn thing).

Tell me your secrets

It was a long day; my daughter got back from youth camp this afternoon, I got home from being in Raleigh at 2am, and we’re all beat. So we had some fun with the cat.

And now I just need to share it so I never forget. =)

Blocking ad networks with named

I’ve meant to do this for ages, so on my first day of my “staycation”, despite vowing to myself that I wouldn’t look at a computer screen this week (hey, it’s not actually the technical start of my week off is it?), I fiddled this morning with BIND to try and avoid seeing ads on my devices. While AdBlock works great on my browsers, that doesn’t transfer well to mobile devices and apps with built-in advertising, etc.

Unless you’re running your own BIND DNS server at home, you won’t be able to do this. If you have a home network with named running (my local network does) and unless you restrict all outbound DNS and allow DNS lookups only from your named server (which I do, it forces all of the machines on the system to use my DNS server which is configured to only ask OpenDNS for DNS info), this also won’t really work for you (at least not in the way that I’ve done it).

So this assumes some knowledge of BIND and networking. This is not so much a tutorial on how to configure BIND as it is some quick tips and shared info on what I did this morning.

First you need to setup a master zone. Mine looks like this:

zone "rpz.linsec.ca" {
        type master;
        file "master/rpz.linsec.ca.zone";
};

NOTE: You may also need the following in your options section, but I’m not 100% sure as it was there before:

    response-policy {
        zone "rpz.linsec.ca";
    };

This makes anything defined in this zone to be considered authoritative, just like the DNS settings I have for my local network. As an aside, you can use this to block entire domains (like youtube or facebook if you have kids at home staring at screens all day…).

I then wrote a script which pulls data from MVPS Hosts. Their data is meant to be put into a hosts file, but that means it would only work on a single machine and I’m trying to solve a multi-machine/mobile issue, not just a single computer. The script takes my rpz.linsec.ca.zone file and mashes in data from MVPS Hosts and to create a new file that we will use:

#!/bin/sh
IFS=$'\n'

input=$(mktemp /tmp/mvps.hosts.XXXXXX)
output=$(mktemp /tmp/rpz.linsec.ca.zone.XXXXXX)
source="/etc/named/rpz.linsec.ca.zone"
serial=$(grep serial ${source} | awk '{print $1}')
n_serial="$(date +%Y%m%d)01"

curl -s http://winhelp2002.mvps.org/hosts.txt >${input}

dos2unix -o ${input} >/dev/null 2>&1

lines=$(wc -l ${input} | awk '{print $1}')

if [ ${lines} -lt 10000 ]; then
    exit 1
fi

for line in $(cat ${source}); do
    if [ "${line}" == ";START ADHOSTS" ]; then
        break
    else
    	echo ${line} >>${output}
   fi
done

echo "" >>${output}
echo ";START ADHOSTS" >>${output}
for hostname in $(cat ${input} | egrep -v '^#' | awk '{print $2}'); do
    if [ "${hostname}" != "localhost" ]; then
        echo "${hostname}    IN    CNAME    ." >>${output}
    fi
done
echo ";END ADHOSTS" >>${output}

perl -pi -e "s/${serial}/${n_serial}/g" ${output}

rm -f ${input}
cp -f ${output} ${source}
rm -f ${output}

Note that you need dos2unix installed. Everything else is fairly standard. The MVPS Hosts file seems to be updated monthly, so this something you could possibly add to a monthly cronjob or just run manually every once in a while. So far it seems to work pretty good over here. I had initially thought about writing something in python, but bash is just so much faster (for me).

Also, if you put things in your zone file before the “;START ADHOSTS” line they’ll be retained, so if you do want to block specific domains (you may want to block iadsdk.apple.com and qwapi.com if you don’t want to see iOS iAd ads) you still can, and take advantage of the MVPS Hosts list (if someone has a better list, I would love to see it).

I hope this helps someone else out. Comments for improvement are welcome, this was a pretty quick-and-dirty script that, I’ll admit, does a few things oddly.

Custom Email Notifications in GitLab

I started playing around with GitLab last month in order to get to know it a better and, while I like it well enough, the one thing that drove me nuts was the email that it sent out alerting of changes. My old git setup used the wonderful git-notifier script to send out emails and I much prefer the format it used than the format GitLab uses. Unfortunately, at that time, without ponying up for the enterprise edition it didn’t look feasible to change without some serious work that I didn’t have the time or effort to invest.

Yesterday I was looking at the latest version (7.6.2) and noticed the community edition support for custom hooks. After upgrading, I fiddled with it and git-notifier to try to make the two work well together. With a little elbow-grease (git-notifier works well with straight git repos or gitolite) I got it to work, although it is a bit of a nuisance because, with regular git or gitolite, you can get some information from the repo exposed via the calling scripts and environment that does not seem to be present in GitLab.

If you follow the instructions on the custom hooks document referenced above, you’ll end up with something along the lines of /var/opt/gitlab/git-data/repositories// .git/custom_hooks (in my case it is /srv/git-data/repositories// .git/custom_hooks). In this directory (which must be owned git:git, including all its contents) lives a post-receive script which looks like:

#!/bin/sh
repo_name="mygroup/myrepo"

base_dir="/srv/git-data/repositories"
git_host="gitlab.myhost.com"
send_from="mailer@myhost.com"
send_to="commits@myhost.com"

pushd ${base_dir}/${repo_name}.git >/dev/null 2>&1

/srv/git-hooks/git-notifier $@ --link="http://${git_host}/${repo_name}/commit/%s" --sender="${send_from}"  \
  --mailinglist="${send_to}" --repouri="ssh://git@${git_host}:${repo_name}.git" --emailprefix="[git/%r]"

popd >/dev/null 2>&1

I have git-notifier in a directory called /srv/git-hooks and it’s owned root:root and mode 0755. This will tell git-notifier to send an email to the $send_to address, from the $send_from address, and defines a few things like the repository itself and the host (all things that would normally be exposed via the environment in a git or gitolite setup but are lacking with GitLab). But this can be used as a template and the only thing you should have to change is the value of $repo_name (everything else can be the same unless you need to define them differently per-repo or per-group).

The downside to this is that you need shell access to set it up, which may prove troublesome for larger installations or shared environments. For a personal or work environment this is probably an ok requirement. Make sure that you disable the “Emails on push” service for the repository in GitLab or you’ll get both the stock GitLab commitdiff email and git-notifier’s email.

I’m extremely grateful for those who contributed this support to GitLab as it means I spent a lot less time dorking around with this than I would have had I done it all myself, and while it was a bit of a nuisance to setup, it works quite well and I’m back to getting my old style of email notifications which are much more useful (for one thing, GitLab seems to have an upper size limit and if that is exceeded it sends no mail at all, whereas git-notifier will send you a list of changed files without the actual diff… a much more useful and meaningful email than sending nothing at all (if you look at git-notifier’s changelog you’ll see that was contributed by me in version 0.3-18, almost 2.5 years ago… that’s how long I’ve been using git-notifier)).

I wish I could contribute some sane code back to git-notifier to support GitLab, but without GitLab exposing things like the repository name or committer name to the environment I don’t think it would be possible unless I’ve missed something non-obvious.

SSL Certificate Verification failure with fink’s Python 2.7.9

Python 2.7.9 was released nearly a month ago and with it came some SSL-related changes (it backported the Python 3.4 ssl module and does HTTPS certificate validation using the system’s certificate store). The latter can cause some problems with home-grown CA’s, however. On Mac OS X, the CA certificate store is in the Keychain Access application which isn’t exposed to commandline tools like Python. This will cause HTTPS certificate validation to fail because Python doesn’t know anything about the CA certificate used to sign the certificate being used by a HTTPS server.

If you’re using the system OpenSSL, supposedly you can export the CA’s of interest to the /System/Library/OpenSSL/cert.pem file (untested). I use fink and fink’s OpenSSL does not seem to use this directory. Instead it uses /sw/etc/ssl/ and if you install fink’s ca-bundle package you will have a stock /sw/etc/ssl/certs/ca-bundle.crt file which presumably works with some applications. This file can be replaced with an updated CA bundle containing the CA certificate that is used to sign the service(s) you want to connect to.

However, replacing that file is not enough. If you upgrade to Python 2.7.9 in fink and make that change, you will still see this annoying error:

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

when attempting to connect to a site using a certificate signed by a non-stock CA. Note that prior to 2.7.9, Python did not do this CA validation so you would not see this error until upgrading to 2.7.9.

The fix is quite simple. Put your new ca-bundle.crt file in place as noted above, and then, as root, symlink this file to /sw/etc/ssl/cert.pem:

# cd /sw/etc/ssl
# ln -s certs/ca-bundle.crt cert.pem

Now when using Python 2.7.9 (on a fink-using system) you will be able to connect to those sites and avoid the “certificate verify failed” error noted above.