Programming

Custom Email Notifications in GitLab

I started playing around with GitLab last month in order to get to know it a better and, while I like it well enough, the one thing that drove me nuts was the email that it sent out alerting of changes. My old git setup used the wonderful git-notifier script to send out emails and I much prefer the format it used than the format GitLab uses. Unfortunately, at that time, without ponying up for the enterprise edition it didn’t look feasible to change without some serious work that I didn’t have the time or effort to invest.

Yesterday I was looking at the latest version (7.6.2) and noticed the community edition support for custom hooks. After upgrading, I fiddled with it and git-notifier to try to make the two work well together. With a little elbow-grease (git-notifier works well with straight git repos or gitolite) I got it to work, although it is a bit of a nuisance because, with regular git or gitolite, you can get some information from the repo exposed via the calling scripts and environment that does not seem to be present in GitLab.

If you follow the instructions on the custom hooks document referenced above, you’ll end up with something along the lines of /var/opt/gitlab/git-data/repositories// .git/custom_hooks (in my case it is /srv/git-data/repositories// .git/custom_hooks). In this directory (which must be owned git:git, including all its contents) lives a post-receive script which looks like:

#!/bin/sh
repo_name="mygroup/myrepo"

base_dir="/srv/git-data/repositories"
git_host="gitlab.myhost.com"
send_from="mailer@myhost.com"
send_to="commits@myhost.com"

pushd ${base_dir}/${repo_name}.git >/dev/null 2>&1

/srv/git-hooks/git-notifier $@ --link="http://${git_host}/${repo_name}/commit/%s" --sender="${send_from}"  \
  --mailinglist="${send_to}" --repouri="ssh://git@${git_host}:${repo_name}.git" --emailprefix="[git/%r]"

popd >/dev/null 2>&1

I have git-notifier in a directory called /srv/git-hooks and it’s owned root:root and mode 0755. This will tell git-notifier to send an email to the $send_to address, from the $send_from address, and defines a few things like the repository itself and the host (all things that would normally be exposed via the environment in a git or gitolite setup but are lacking with GitLab). But this can be used as a template and the only thing you should have to change is the value of $repo_name (everything else can be the same unless you need to define them differently per-repo or per-group).

The downside to this is that you need shell access to set it up, which may prove troublesome for larger installations or shared environments. For a personal or work environment this is probably an ok requirement. Make sure that you disable the “Emails on push” service for the repository in GitLab or you’ll get both the stock GitLab commitdiff email and git-notifier’s email.

I’m extremely grateful for those who contributed this support to GitLab as it means I spent a lot less time dorking around with this than I would have had I done it all myself, and while it was a bit of a nuisance to setup, it works quite well and I’m back to getting my old style of email notifications which are much more useful (for one thing, GitLab seems to have an upper size limit and if that is exceeded it sends no mail at all, whereas git-notifier will send you a list of changed files without the actual diff… a much more useful and meaningful email than sending nothing at all (if you look at git-notifier’s changelog you’ll see that was contributed by me in version 0.3-18, almost 2.5 years ago… that’s how long I’ve been using git-notifier)).

I wish I could contribute some sane code back to git-notifier to support GitLab, but without GitLab exposing things like the repository name or committer name to the environment I don’t think it would be possible unless I’ve missed something non-obvious.

SSL Certificate Verification failure with fink’s Python 2.7.9

Python 2.7.9 was released nearly a month ago and with it came some SSL-related changes (it backported the Python 3.4 ssl module and does HTTPS certificate validation using the system’s certificate store). The latter can cause some problems with home-grown CA’s, however. On Mac OS X, the CA certificate store is in the Keychain Access application which isn’t exposed to commandline tools like Python. This will cause HTTPS certificate validation to fail because Python doesn’t know anything about the CA certificate used to sign the certificate being used by a HTTPS server.

If you’re using the system OpenSSL, supposedly you can export the CA’s of interest to the /System/Library/OpenSSL/cert.pem file (untested). I use fink and fink’s OpenSSL does not seem to use this directory. Instead it uses /sw/etc/ssl/ and if you install fink’s ca-bundle package you will have a stock /sw/etc/ssl/certs/ca-bundle.crt file which presumably works with some applications. This file can be replaced with an updated CA bundle containing the CA certificate that is used to sign the service(s) you want to connect to.

However, replacing that file is not enough. If you upgrade to Python 2.7.9 in fink and make that change, you will still see this annoying error:

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

when attempting to connect to a site using a certificate signed by a non-stock CA. Note that prior to 2.7.9, Python did not do this CA validation so you would not see this error until upgrading to 2.7.9.

The fix is quite simple. Put your new ca-bundle.crt file in place as noted above, and then, as root, symlink this file to /sw/etc/ssl/cert.pem:

# cd /sw/etc/ssl
# ln -s certs/ca-bundle.crt cert.pem

Now when using Python 2.7.9 (on a fink-using system) you will be able to connect to those sites and avoid the “certificate verify failed” error noted above.

Merry Christmas 2014!

I just wanted to wish everyone a Merry Christmas and Happy New Year, from my family to yours. My wife found the most awesome card for my teammates at work and it is just too good not to share with everyone — for those who are programmers or into IT, this is perhaps one of the most fitting cards for our industry. =)

God bless you all and my prayer for each and every person reading this is that this year has been good, but that next year will be even better!

atleasttheywork

Git patch workflow

Probably most people who use git know about patch management and whatnot, so I’m writing this largely for myself as I keep forgetting (mostly because I don’t have to do it very often).

Some code that I’m working on has a development branch which I’m working on while other fixes are being made to the master branch. This is great, except for the fact that I need to apply any changes made to master to the development branch as well (we’ll call it “feat/X”). And since I’m the only one working on feat/X, but others add fixes or necessary changes to master, I need to apply those changes. Some changes will apply clean, others need to be modified due to one part that is receiving a major overhaul.

There is probably a better way to do this and if so, please do let me know in the comments. Even though I use git almost daily, I am by no means a power user and would love more “insider tips”.

So I have my working copy, and I need it to know about both branches. By default, if you clone the repository you’ll just be on the master branch. So in this case I would have to git checkout feat/X. Once I do that, I will see:

% git branch
* feat/X
  master

So do a git checkout master to switch back to the master branch. From here we can make a series of patches that represent code changes made to master after feat/X was branched:

% git format-patch feat/X

The first time you do this, all the patches will be new. But as you move along, you’ll continue to see these patches show up so it’s useful to keep them kicking around so you know which are new. You’ll see patches in the format “0001-The-commit-text.patch” and “0002-Other-commit-text.patch”.

I move the patches out of the way (I have horrid file management, so I just mv *patch ~/). Then git checkout feat/X to switch to the feat/X branch. Now we apply the patches one-by-one:

% git am <0001-The-commit-text.patch

If the patch is successful, you can do the same to the next. If not, however, you need to fiddle with things a bit. You'll see an error like this:

% git am <~/0001-The-commit-text.patch
Applying: The commit text
error: patch failed: something/else.py:559
error: something/else.py: patch does not apply
Patch failed at 0001 The commit text
The copy of the patch that failed is found in:
   /home/vdanen/my-git-repo/.git/rebase-apply/patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

You can run git am --abort which will revert the commit attempt, to allow you to manually apply the patch and commit. You can also use git am --skip to do the same. I suspect if you fed git something like git am <~/*patch these commands would mean different things (abort the session versus skip the one patch) but if you're doing one patch at a time they both do the same thing.

You can also check the log output with git log master..HEAD --stat which will show that our branch contains the update, along with the appropriate author information (might be you, might be me, might be someone else). I tend to skip the log bit and just apply one patch at a time. Rinse and repeat.

Finally, if you don't move your patches like I did, you can use git clean to remove the patches from your working directory. Use '-n' first though, so it runs in dry-run mode.

Like I indicated earlier, there are probably simpler ways to do this, but this works for me. I got all of this from Ry's Git Tutorial on Patch Workflows. Excellent tutorial.

Git commits on xkcd

I’ve had no time to do any coding lately, which makes me sad, but when I did have time and was doing lots of commits in short periods of time, I definitely felt like doing this. Many many times.

Review of O’Reilly’s Learning Python, 5th Edition

I’ve been programming in Python for a few years now (I can pretty much mark the beginning because of starting at Red Hat). Since picking it up, I’ve fallen in love with the language and have a few books on the subject. One of the most indispensable books I have is Learning Python 3rd Edition published by O’Reilly. Recently I received a copy of the 5th Edition for review.

learning-python

My first reaction was that it was no mere book, but a tome. The 3rd Edition was no slouch, weighing in at 700 pages, but the 5th Edition is a hefty 1540 pages, over twice the size! This edition was released in June 2013 (ISBN 978-1-449-35573-9); the previous edition that I owned (the 3rd) was published in October 2007. This new edition was updated to cover Python versions 3.3 and 2.7 (the 4th Edition covered 2.6 and 3.0/3.1, so I imagine it too had a hefty size increase over the 3rd Edition).

Before I go further with the review I have to note that Learning Python 3rd Edition is nearly always on my desk. I referenced that book all the time. With the number of alternatives out there (searching on the internet or looking at other books that I have, such as Programming Python, the Python Cookbook, and Python in a Nutshell), this is the one book I used all the time. It is worn, crinkled, dirty, and probably sticky in a few spots as well. However, I’m not one to read books like this from front to back — I use them as reference material for areas that interest me or I need help with (or I need to brush up on).

The first section of the book, “Getting Started” goes into the basics of Python: what it is, what it’s used for, how to use it, why you would use it, and so forth. The 5th Edition expands on this, particular in regards to version 3.3 and its new options in Windows. This is all the really basic stuff, explained quite well and great for those interested in getting into Python without a lot of knowledge of the language. Those more experienced with Python will likely skip this section for the most part, but there are some good bits in here.

The second section of the book, “Types and Operations”, gets to the meat of writing code. This is the section that talks about Python object types (lists, dictionaries, tuples), how Python handles numbers, dynamic typing, manipulating and using strings, and handling file operations. One thing I noticed immediately is that the section dealing with numbers is greatly expanded from the 3rd Edition and this is largely due to the changes of how these are handled in Python 2.x vs 3.x. It is in this chapter that you begin to see why the book is so hefty — instead of focusing on just one major version of the language, it provides the necessary information for both 2.x and 3.x and the differences between the two. Because Python 2.x is still so widely used, it would have been impossible to ignore it unless they decided to write two books, one for each major version. The section on handling strings has likewise been expanded, enhanced, and re-organized with quite a bit of extra content. Again, quite a bit of this is due to the coverage of both Python 2.x and 3.x.

The third section of the book, “Statements and Syntax”, gets into the fundamentals of handling your code: typical statements (if/elif/else, variable assignments, loop handling, creating functions, namespaces, module handling, and exception handling). Here again a lot of content is devoted to the differences between 2.x and 3.x, and even between different versions of 2.x. There is a lot of content here — chapters devoted to topics about looping, if statements, iteration and comprehension, the 2.x print statement versus the 3.x print function, and even how to generate documentation for your code using PyDoc. For those learning Python, and even for those who choose to use this book as a reference, this section will be greatly used.

Perhaps one of the most significantly changed sections between the two editions of Learning Python (again, noting that I’ve never looked at the 4th Edition so I don’t know how significant a change it is between the 4th and 5th editions), is what was “Functions” in the 3rd edition is now “Functions and Generators” in the 5th Edition. There is a significant overhaul between the two: where before “Scopes and Arguments” were a single chapter, we now have two chapters, one devoted to each topic; the 33 page chapter has turned into 67 pages spanning two chapters. It also goes into great detail about generator functions and expressions (something I have yet to fully explore). Suffice it to say, there is a good 45 pages of new content here that will prove interesting to read (list comprehension, generators, etc.).

The fifth section, “Modules and Packages” likewise contains greatly expanded content covering the use of modules, how to create them, why (and how) you should use them, the differences between importing and reloading modules, full coverage of Python 3.3 namespace packages, problems you can encounter with module use, and so forth. It talks about the features common to both major versions of Python, and has sections that are more specific, such as how byte code is handled (.pyc in versions prior to 3.2 and the __pycache__ directory in 3.2+) and namespace packages that were introduced in 3.3 (which is also shows the differences between regular packages and namespace packages). This once again shows the level of detail the book provides for users of any version of Python.

Section six is all about Object-Oriented Programming and classes, and gives great detail and examples on topics such as polymorphism, classes and subclasses, object handling, operator overloading, and more.

Section seven goes into the how’s and why’s of exceptions and how to use them to write good error-handling in your code. This is the last section that appears in both editions, although (as with every other section), this one is greatly expanded.

The new section in the 5th Edition, compared to the 3rd, is the “Advanced Topics” section and its five chapters. This section goes into unicode and byte strings — an area that has seen a lot of changes between Python 2.x and 3.x, as well as managed attributes, decorators for functions and classes, metaclasses and the differences between them.

Obviously I’ve not yet read the entire book and chances are I never will. Most of these programming books I never do read end-to-end, but I do use them as functional references and to learn new things from (pretty much the entire “Advanced Topics” section is new to me), as well as to brush up on older things. From the parts I’ve read, and the comparisons I’ve made to the 3rd edition, this is most definitely a worthwhile “upgrade”. One thing I appreciate as well is the consistent back-and-forth regarding Python 2.x versus 3.x. As a Python coder who has never yet touched Python 3.x, this book will help me understand the subtle, and sometimes not-so-subtle, nuances between both major versions — when the time comes (aka: when the time is available). As such, this book is valuable to me now (writing code in Python 2.x) and will be valuable to me in the future.

I certainly appreciate the expanded content, as it’s already twigged a few things in my mind about the code I’ve written so far, and has given me some new ideas on how to handle certain issues or improve performance/handling of my code. This is both good and dangerous! Having said that, this is most definitely a book worth getting for any Python programmer who either is new to the language or is a veteran of the language — the first will appreciate the no-nonsense easy-to-understand approach to introducing both the basics and some quite advanced topics, while the latter will appreciate it as a reference book or to expand on their understanding of certain topics. For myself, I’m somewhere in the middle of the two and this book is introducing me to a lot of stuff I do not yet know, and is a very handy reference for the stuff that I do know (and need reminders of every once in a while).

I’ve highly recommended the 3rd edition of this book to anyone asking about good books on Python, and the 5th Edition is no different. I highly recommend this book to anyone who wants more than just a passing understanding of Python.

And as a replacement for my well-worn and well-loved 3rd Edition, I know this book will see much use from me.