Red Hat

Heartbleed

I’ve refrained from posting or saying anything about Heartbleed all week because I didn’t want to add to any sensationalism and hype, and I’ve also been too busy actually dealing with it (as opposed to simply talking about it or running around with hands waving in the air like a mad man). Now that the dust has settled a bit, I just want to link to some sites that I think are good to keep handy as we see this play out. I don’t need to talk about the flaw itself as all you need to do is google “heartbleed” and you’ll get all the info you want; certainly more than I can provide here (although you will have to distill the sensational from the facts).

So, the sites:

  • Heartbleed Bug Health Report; they’re keeping it up to date, but it’s essentially a “top 1000 still-vulnerable sites” list
  • Mashable’s Heartbleed Hit List which has a list of some of the bigger sites/services that were (or were not) affected and whether they still are; when I looked this morning it was last updated as of last night so presumably they’re keeping it fairly up to date
  • DigitalTrends Mobile app list which has a list of vulnerable/not-vulnerable mobile apps
  • The Heartbleed site which is being kept up to date with regards to linking to various advisories

Some of these sites (and the apps that use such sites) have been fixed this week. There is speculation that this has been known for a while which means the “window of opportunity” may be much bigger than was initially thought. Some of the numbers being tossed around are pretty gross exaggerations though (one I saw was “66% of the internet vulnerable!”) so you have to take things with a grain of salt. The best advice is to look at the sites you use and if they have fixed the flaw (and were previously vulnerable) and recommend you doing something (like changing your password), strongly consider doing as they suggest — PROVIDED THEY HAVE ALREADY FIXED THE FLAW! Sorry for the caps but I talked to some people yesterday who had rushed to change their password and when I asked them if the site in question was fixed already, they gave me a blank stare.

It does you NO good to change your password to a site that is STILL vulnerable. You will only have to change it again.

Anyways, look at the sites noted above, breathe, and keep in mind that changing passwords occasionally is a good thing. Maybe now is the time to start using something like LastPass, 1Password, KeePass, or something similar and having it generate pure random nonsense for a password, knowing that you can use this tool/service to remember it for you (although, arguably, this whole situation makes me quite happy that I use 1Password (an app on my computer) instead of a service.

My last point on this is that people need to upgrade if they’re using an affected version of OpenSSL. If you are, and your operating system provides it (which is the case with Red Hat Enterprise Linux and Fedora, among many others) then you really should be updating to the packages provided. It’s not a question of whether you should or shouldn’t — you should. Period. This has been a crazy week and a lot of crazy things have happened and this is a really really bad thing IF you’re affected. So if you are (as in you’re running Red Hat Enterprise Linux 6.5 or a current Fedora, etc.) then you really need to update ASAP. And then you need to assess your next steps (changing passwords on vulnerable (and now fixed) services, revoking and reissuing certificates if you feel it necessary, etc.).

Anyways, that’s all I have to say about Heartbleed. It will be interesting to see what the next few weeks will be like as we continue to get a bigger picture of what’s happened here, how, and to whom. And to see what damage has been done, and who responded appropriately and when. For instance, if there were a site or service I was using and as of today (being Saturday, and this thing exploded on Monday) it was still NOT fixed, I don’t think I would be using that site/service anymore. To put it into perspective, Red Hat had updates out late Monday for Red Hat Enterprise Linux 6.5 and the other affected products early Tuesday morning (my time). Everything was available to customers in under 24hrs. It’s not hard to install — “yum update” and reboot (to make sure everything is covered). So for a site to be still affected by this now? There’s really no excuse as far as I’m concerned.

Finally, just to note that I did get some minor press coverage (so this is more vanity than useful), LinuxInsider reported on Heartbleed and my name is noted, although my answers to the questions must have been less than exciting as there wasn’t too much noted there other than where Red Hat customers could go for more info. =)

And to finish off, the obligatory xkcd:

I’m FedUp with Fedora!

Sorry, couldn’t resist. =)

So normally I do my updates from one version of Fedora to the next using yum, in particular the Upgrading Fedora using yum guide. Usually it works pretty good. I didn’t really have much good experience with PreUpgrade the few times I tried it, so I wanted to give FedUp a try.

In my Parallels Fedora 17 VM it worked amazingly well. So decided to try it on my laptop, which is also running Fedora 17. I think it makes sense to do a little bit of house-keeping before running it though, and the FedUp page doesn’t mention any of this (perhaps it’s no longer needed?). Anyways, a few steps:

* yum install rpmconf; rpmconf -a (review any .rpmnew/.rpmsave files, merge changes as required)
* find /etc /var -name '*?.rpm?*' (find any other old .rpmnew/.rpmsave files)
* yum install yum-utils; package-cleanup --leaves (review and remove any unused packages, not all will be removable)
* package-cleanup --orphans (find and remove any orphan packages no longer in the repositories)

Now you can run FedUp:

* yum install fedup
* fedup-cli --network 18 --debuglog /root/fedupdebug.log

If this completes without error (check the log), you can reboot. At grub you’ll see a “System Upgrade” entry. When that’s done, it’ll reboot into Fedora 18.

The wiki page talks about upgrading GRUB2 since you’ll be booting from Fedora 17′s GRUB2. If you’ve got a BIOS-based system, you can use the Updating GRUB2 configuration on BIOS systems instructions. For those using UEFI, instructions are on the same page.

You may also want to run package-cleanup --orphans after you do the upgrade as well, just to get rid of any other leftovers. The only issue I discovered so far with the upgrade is that Google Chrome didn’t work out-of-the-box. However, doing a yum remove google-chrome-stable; yum install google-chrome got that sorted out (although it did install the unstable version; the stable version had issues with missing libraries and wouldn’t load).

All in all, upgrading from Fedora 17 to 18 went a heck of a lot smoother than a fresh install did. I also got to see what the new GDM/GNOME looks like (quite nice, actually, although I think I’ll give MATE a try on the laptop as well because, while GNOME3 is pretty, I definitely preferred GNOME2).

Good job, Fedora-folks! Now I just have to upgrade my main workstation, but I think I’m going to play on the laptop for a bit before taking that step. Just in case I find any other gotchya’s.

Systemd article on wiki

In light of my recent (mis-)adventures with Fedora 18 installation in VMware Fusion and the need to figure out some systemd stuff, I’ve started writing a hints/tip sheet type entry on my wiki: Systemd. Far from complete yet, but I’m going to use it to document some hints, tricks, light-bulb-moments, and comparisons to SysV-init tools (chkconfig, service). I’m no stranger to alternative boot/service management systems but it indeed makes me chuckle when people were complaining about Annvix using runit to manage the init system. I definitely appreciate the work gone into Systemd (having done a similar thing myself for Annvix although obviously not anywhere near the scale/scope of Systemd!), so do want to learn it as opposed to using old scripts as a crutch-interface to the new stuff. So as I likely stumble across various bits of useful info, I’ll be adding it to that wiki article if anyone else is interested in checking it out (or perhaps giving me tips… like how the heck do I actually get gettys to run in graphical.target!! I want a working CTRL-ALT-F2/F3 to login at the console please!!).

Upgrading and Installing Fedora 18 in VMware Fusion

I was thoroughly excited about Fedora 18 until I actually went to install it. I use Fedora 17 on a laptop and a desktop, but I also have copies in virtual machines that I use for testing — these virtual machines run in VMware Fusion on my Mac Pro. So never having used Fedora 18 during any of the betas or anything, first thing I did was try to install it in VMware.

A few times.

First I have to say that I’m not a big fan of the new Anaconda UI. One of the things I appreciated about Anaconda in the past is that it was easy to use, but didn’t prevent me from doing “power things”, like partitioning disks the way I wanted to partition them. This seems to no longer be the case, and Anaconda does a lot of hand-holding. It has been a habit to go through the list of packages to be installed (select a group, then go through and weed out the stuff I don’t want), so things I don’t want don’t even touch the disk. Doesn’t seem like I can do this anymore either. I don’t agree with some reviewers who seem to think the UI change is to make it look/feel more mobile or “touchscreen-like”, but I do feel it’s dumbed down quite a bit which makes me unhappy. I lived with it, until trying to change the partition layout killed Anaconda. Twice. After that I decided to let it do what it wanted since it was just a VM and I didn’t care enough to fight it.

So Fedora gets installed and lo! A nice blue screen where I can do absolutely nothing. No icons, no text, no login box, nothing. Afterwards I read this has to do with having 3D graphics enabled in VMware (but at that point I had already reinstalled to use LXDE rather than GNOME, so I’ve not tried anything regarding that, however I did find this blog posting about Fedora 18 in VMware Fusion 5/VMware Workstation 9 which explains things a bit).

On my home network I run IPA on Red Hat Enterprise Linux 6 to handle my Kerberos/LDAP auth duties. This means when I install an operating system, no matter what it is, I don’t create local users. So when I installed Fedora 18, I only have the root user and the rest come from IPA. A few problems with that:

  • Setting up IPA in Anaconda does not work
  • Post-install, logging in as root at the GUI is not permitted
  • I have a grub error on tty2, not a login (and nothing on tty3 for that matter)
  • I have to reboot and edit grub to add systemd.unit=multi-user.target to boot into the console to get a tty (see Systemd: Boot Kernel Command Line)
  • After I install freeipa-client, I can enroll in the domain
  • SSSD is not enabled when you install it! Once again I have to reboot to the console
  • Enable SSSD and reboot, only to find that once again I cannot login as a user in the GUI (that I could login as on the console)
  • Head-scratching moment when I realize I know next to nothing about systemd and can only conclude that perhaps NetworkManager starts upon login in the graphical boot and not at system start, which prevents SSSD from talking to IPA to confirm my login
  • Looking at /etc/sysconfig/network-scripts/ifup-eth0 tells me that perhaps that is not the case
  • Big WTF moment and go to bed

Currently I am unhappy with Fedora 18. I suppose that on bare metal it probably works better. I also suppose that with a real user account it works better. At this point I don’t want to do yet another reinstall because then I have to remove the host from IPA and re-enroll, which is annoying (and I really don’t want to revisit Anaconda either — I’ll get to do that when I create the 64bit Fedora 18 VM later).

Ok, after a few days I tried a few other things. The first was getting rid of NetworkManager thinking that perhaps it would only start when I login (which I can’t, if there is no network, because my login is IPA-based). So I had to do:

# systemctl disable NetworkManager.service
# systemctl enable network.service

The most annoying thing, however, is that when I boot into graphical mode, I get no gettys. I’m pretty sure I told systemd that I want three of them (I have three symlinks in /etc/systemd/system/getty.target.wants: getty@tty1.service through getty@tty3.service; sadly when I boot into graphical mode and switch to TTY2, I get a grub error message and nothing on TTY3… what the heck?)

Another thing I did was disable SELinux. In multi-user.target, I can login as my IPA-user no problem. With LXDM, however, I never could (assuming, perhaps incorrectly, that network initialization was delayed). But upon further inspection I see a lot of:

lxdm-binary: pam_selinux(lxdm:session): Setting key creation contect guest_u:guest_r:oddjob_mkhomedir_t:s0 failed: Permission denied

I have zero interest and zero time in making SELinux work with LXDM and since this is a virtual machine, I don’t care if SELinux is disabled.

Lo and behold, I can now log into LXDM. In light of the SELinux thing, I’m thinking that NetworkManager is not to blame (silly me, I made two changes then rebooted — it could be either one, but I strongly suspect SELinux). Maybe it would have worked better with GNOME (will try that on the next VM I install).

On another side note, since I use VMWare Fusion for the VM, for the tools install (probably the same for the latest VMware Workstation as well, etc.), you may want to check out this thread. In particular, seems like linux/version.h is missing and makes the tools angry. The fix is easy though:

# cd /usr/src/kernels/[current_version]/include
# cp generated/uapi/linux/version.h linux/

Then you can compile the tools. IIRC, you’ll also not want to use the i686 PAE kernel (doesn’t seem to work on older versions, I suspect the same may be true with F18). Easy enough to yum remove the PAE kernels and install the regular kernel. YMMV.

Finally, the next day I installed Fedora 18 64-bit in VMWare Fusion. With the lessons learned above, it was a lot less painful. I still am not a big fan of Anaconda… I like the look of it, but the practical “working” bits of it definitely need some work. I don’t like that you can pick only one group of packages. What if I want both GNOME desktop and a working web server? Doesn’t seem like it can be done. I also like picking through individual packages and sorely miss that — makes for having to install a bunch of stuff later. Also not sure why vim isn’t installed by default… I think this is the first time ever I’ve seen “vim: command not found”.

Also, despite getting the VMware tools installed, if you want GDM to present you with anything, you need to disable 3D graphics. I have it enabled on my 32-bit VM (which is using LXDM, works fine), but on the 64-bit VM (using GDM), even with vmware-tools compiled and running, I needed to disable 3D graphic acceleration. I suppose there is a bug in the driver and/or X there. Not a big deal to disable since I’ve done “yum groupinstall ‘MATE Desktop’” (what need do I have for 3D graphics? BTW, MATE is sweet!) and now GDM loads.

IPA still would not work to configure from Anaconda. Definitely a bug to be filed there.

Anyways, time to end this. This entry has been in the works for about two weeks (sadly, I’ve not had time to finish it or get my Fedora VM’s running until yesterday, and finished up the 64-bit install this morning). So far the install of Fedora 18, in VMware, has been a bit of a PITA. Next order of business is to see how the upgrade works… I’ve got a Fedora 17 install in Parallels to upgrade to make sure all my work-related bits work, before attempting to upgrade on bare metal. I’m quite certain that a lot of this can be attributed to the virtual machines and it’ll work better on bare metal. I’m also happy that those won’t be fresh installs, so I don’t have to deal with Anaconda.

Replaced my Kerberos+LDAP setup with FreeIPA

So I’ve been having to deal with some IPA-related bugs in the past little bit, which of course got me thinking that I had no idea what IPA did or how to use it (thankfully I wasn’t responsible for fixing the bugs!). But as I had to deal with this issues to some degree, I got to figure out what FreeIPA was and what it did. In short, FreeIPA rocks. As many of you know, I’ve written quite a few articles and blog posts about using Kerberos or OpenLDAP for authentication. It’s no secret that I make heavy use of Linux at home, but also of the Mac, so for me any solution needs to deal with both in a semi-reasonable way. I could do Kerberos auth on OS X easily enough, but never did have luck with LDAP. On Linux, it’s a piece of cake.

I’ve been using Kerberos and LDAP at home for years, largely because I have to do a lot of testing of things in virtual machines, so when a new version of something comes out (new Fedora, new major version of RHEL, etc.), I spin up a new VM and install it. Using Kerberos and LDAP make the setup a breeze, and if I change my password, I’m not changing it on 20-odd systems/virtual machines.

I’m happy to say that FreeIPA exceeded my expectations, despite a bit of a rocky start (due to my not reading enough of the docs, annoyingly enough). I’ve now got it in place, it’s doing Kerberos+LDAP on the Linux clients and also on the Macs! I have, for the first time ever (not counting OS X server 10.4 or something, and using OpenDirectory), gotten to login to an OS X system with network credentials. I’ve also made use of the DogTag CA and had my internal mediawiki instance (which used mod_auth_kerb for SSO authentication) also use HTTPS now with mod_nss and my shiny new IPA CA.

There’s a bunch more about FreeIPA than what I’ve done so far. I’ve just scratched the surface (and even that, not entirely as I’ve still got a dozen or so systems/vms to switch from the old Kerberos+LDAP setup to using FreeIPA), but I’m looking forward to playing with the other things like a hopefully much easier kerberized NFSv4, storing sudo configs in the directory, auto-mounted home directories (don’t care too much about that for the workstations but that will be sweet for the virtual machines), and so on. FreeIPA has a really really nice package that takes care of all this stuff and I’m kinda kicking myself that I didn’t play with it sooner.

And, because of my really odd love for this sort of thing, I’ve written an article on my wiki: Using FreeIPA for User Authentication which goes into the whole setup and enrolment. A lot of it is covered in the upstream docs, but the upstream docs only got to OS X 10.4, so my 10.7/10.8 setup required a bit more futzing and research. While I’m “officially” calling this article done, as I spend time over the Christmas holidays playing around with it, I will no doubt be adding more info as I discover it.

Fedora 15 upgrade

So I upgraded my Fedora 14 workstation to Fedora 15 last night using the yum update method (I’ve used preupgrade a few times and it’s worked on some and botched on others (mostly due to not enough space on /boot)). Since with other distros I’ve either used apt to do a dist-ugprade or the urpmi equivalent, this is somewhat my preferred upgrade path. I’ve done it before and it worked amazingly well, so I did it again last night using these great instructions: Upgrading Fedora using yum.

The only gotchya is that due to the replacement of init by systemd, when it came time to reboot, halt/reboot/etc were unable to send the correct signals to something that would shut the machine down, so I had to do a hard reboot (which never plays nice with my RAID arrays, but upon reboot there was no RAID re-sync which is either cool or scary, I’m not yet sure which). So that was a bit nerve-wracking. Otherwise it was just a lengthy process with yum telling me I had 2850 packages to deal with (including installing and removing). Instructions are good and clear. Highly recommended if you’re even moderately technically inclined.

Now I get a good look at GNOME3, which doesn’t work in my Fedora 15 vm’s (well, it works, but it looks a lot like GNOME2 due to the “poor” video support in a vm). I’m not sure what the big deal is… it’s a little wonky and takes some getting used to. I dislike that conky doesn’t show up on the desktop, but so far that’s my only real complaint. I had icons for Komodo and CrashPlan on the desktop that are no longer visible, so had to use alacarte (“yum install alacarte; alacarte”) to create new icons to go into the GNOME menu system. Then I could add them to my favourites and was off and running. It was about 1am when I finished so I haven’t had too much time to play with it yet (although I also installed LXDE to give it a go as well, in case I didn’t like GNOME3). So far I don’t mind it though.

Everything else seemed to work out of the box other than my apache configuration file. I have a few includes in /etc/httpd/conf/vhosts.d/*.conf and they weren’t loading, so I think the handling of virtual hosts has changed because once I removed the default virtualhost definition (““) that I had defined, the virtual hosts worked again.

All in all, I’m pleased. I’ve played with F15 in my vm’s since it came out (but mostly for testing security issues, etc.) so this is the first workstation with “stuff” that I’ve upgraded. So one work vm and one laptop to go and then F14 is history. Good job on this release, Fedora Folks!