The first is to setup some global options, some of which are nice for folks coming from Subversion. Having a global ignore file is useful. Mine has the following contents:

*~
*.orig
*.rej
*.swp
.#*
*.o
.DS_Store

Then adjust some global git options:

$ git config --global core.excludesfile ~/.gitignore
$ git config --global alias.st status
$ git config --global alias.ci commit
$ git config --global alias.co checkout
$ git config --global alias.br branch
$ git config --global user.name "Your Name"
$ git config --global user.email you@example.com
$ git config --global core.editor "vim"
$ git config --global color.branch auto
$ git config --global color.diff auto
$ git config --global color.interactive auto
$ git config --global color.status auto

The last few allow for colorized output, which I like (makes things like git status easier to read).

I also found out that I had screwed up the remote origin when setting up a new repository, and didn’t want to re-do everything, so found this useful one-liner:

$ git remote rm origin

Git n00bs like me will appreciate the above. =) (Note to self, express git urls as ssh://git.remote.com/path/to/repo.git rather than ssh://git.remote.com:/path/to/repo.git!)

Finally, I found an excellent resource called Create a new Git Remote Repository from some local files (or local git repository). Very accurate, very clear, and very easy to follow. Essentially I was taking a 4GB set of documents and wanted to turn it into a remote repository so that I could push/pull from my laptop and using this article, I was able to do so easily.

I’ve pulled out AIDE+gpg and rsec from the Annvix tools subversion repository and they are now on github:

AIDE+gpg on github
rsec on github

The next step, maybe for around Christmas, is to turn these into Fedora and/or EPEL packages so that I can (finally?) actually be a Fedora contributor beyond just filing security bugs. I may be the only one made happy by that, but I think it would be cool. =)

First, the location of AIDE+gpg in the subversion repo was “tools/AIDE+gpg” and it had one sub-directory, “trunk”. I never did end up using tags or branches or anything there. So the entire step from start to finish to do it was as follows:

git init AIDE
cd AIDE
echo "vdanen = Vincent Danen <[my email]>" >authors.txt
git svn init -T trunk svn+ssh://[repo]/tools/AIDE+gpg --no-metadata
git svn fetch -A authors.txt
git init --bare ../tmp
cd ../tmp
git symbolic-ref HEAD refs/heads/trunk
cd ../AIDE
git remote add bare ../tmp
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
cd ../tmp
git branch -a
git branch -m trunk master
mv tmp AIDE+gpg.git

Presto. Done. Apparently I should now put this repo somewhere public (like github). I will figure that part out after lunch.

The problem is that all of my version control repos are in subversion, and I hate losing history. On some, I went through a painful CVS->SVN migration when I first started using subversion, and I was pleasantly surprised that git makes it quite a bit easier. I found this blog posting that helped me (for the most part… if you don’t use tags and branches, etc. you want to pay attention when doing some of the steps… took me a bit to figure that out). Most notably, in steps four and five when changing the “trunk” to “master”; it assumes you have the standard trunk/, tags/, branches/ layout (which I do in some repos, and not in others). If you use that convention, it works fine. If not, you can run into problems like I did.

The posting indicates to use:

git svn clone [SVN repo URL] --no-metadata -A authors-transform.txt \
   --stdlayout ~/temp
...
git init --bare ~/new-bare.git
cd ~/new-bare.git
git symbolic-ref HEAD refs/heads/trunk
cd ~/temp
git remote add bare ~/new-bare.git
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
cd ~/new-bare.git
git branch -m trunk master

But this didn’t work for me, as –stdlayout isn’t so standard in my case (no tags/trunk/branches, so the top-level is the “trunk”). Instead I had to do:

git svn clone [svn repo] -A authors-transform.txt ~/tmp/git
cd ~/tmp/git
git init --bare ~/git/scripts.git
git remote add bare ~/git/scripts.git
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
cd ~/git/scripts.git
git branch -m git-svn master

In this case, the only branch is the “git-svn” branch, so we want to turn “git-svn” into “master” (rather than trying to hunt down some non-existant branch called “trunk”). There might have been a better way to do this, but I’m a n00b so forgive me. All the explanation for the above is in that blog post I mentioned before (I’m mostly noting this as undoubtably I’ll bump my head against this again).

Some other useful links I found were this git guide and a piece on setting up gitweb on Fedora (works on RHEL also). Gitweb was essential, as I’m used to using viewvc with my subversion repos.

So will I use git for all my repos? Probably not. There are some old ones that don’t need to be converted because it’s all legacy code, and there are some others that I’ve built up with too much automation. I would like to try to extract some pieces of existing subversion repos into git, however. The AIDE+gpg scripts are one; they’re in the Annvix tools repo, and I’d like to try to break it out into it’s own git repo with history… not sure if this is possible but I’ll poke around and see what I can come up with. There are a few other Annvix tools that I’d do the same with (the rsec tool for one). Since Annvix isn’t in development anymore, I’d like to “untie” those tools from it and offer them as stand-alone things (probably on github or something).

So that’s my Saturday adventures for this weekend. =)

But the problem I’m having now, and maybe someone on the lazyweb can help me with this, is that it is absolutely freaking slow. Querying the database is fast enough when you’re searching for something, but doing the imports and especially handling the updates is a process of hours. For instance, to have it manage the updates for Fedora 14 (the SRPMs), I’m looking at 1311 package updates and 353 new packages being added. It’s been running since last night (I don’t know the specific time it started, but it has to be well over 10hrs ago) and it’s still in the “remove old package records” stage. That seems highly excessive to me.

But the database is big:

rqs 0.6 ($Id: rqs 425 2011-01-21 23:37:19Z vdanen@ANNVIX.CA $)

Database statistics:

   Database  => User: rq, Host: localhost, Database: rqs
   Data size => 9.96 GB

   Tag records  : 5                Package records : 21629
   File records : 9554598          Source records  : 76031
   Ctags records: 56300434         Requires records: 100345

And what it is doing is first removing all of the old records (packages, files, ctags, requires, source files) before it can add in the new. Of course, it does this one at a time:

if self.type == 'source':
  tables = ('packages', 'sources', 'files', 'ctags', 'buildreqs')
for rnum in to_remove:
  r_count = r_count + 1
  for table in tables:
    query  = "DELETE FROM %s WHERE p_record = %d" % (table, rnum)
    result = self.db.do_query(query)

I think the problem is I’ve reached my limit of understanding of efficient MySQL as I’ve never dealt with data sets this huge. What would be the best way to handle this? Would using temporary tables work best? Would having each rpm repo on its own table work better? That would logistically be more work as I’d have to create SQL tables on the fly (and while I would like this thing to be portable between db types at some point, it currently isn’t — but this would have an impact when I want to make it portable). The problem then is if I want to query and look for ‘sshd.c’ for instance, I’d have to look across multiple tables (such as ‘rqs_f14_source’ and ‘rqs_rhel6_source’ instead of just ‘source’), but my concern is that might increase the query time (querying right now is pretty fast, and I don’t want to negatively impact querying times, but this import/update time now is ridiculous).

But maybe that is the answer? Multiple tables, grouped by repo (or tag, as rq calls them) as opposed to giant tables now? I mean, looking at the number of rows in the ctags table (56,300,434), this is a pretty huge dataset. Or can you remove from multiple tables at the same time (the p_record column is identical across tables… same name, same data).

Any hints to speed this sucker up?

I’ve been at it for about 2 weeks now with what little time I have and the original suite consists of two programs: rqp and rqs. These two programs were a result of a rewrite of the “srpm” PHP program that Stew Benedict wrote years ago for the Mandriva secteam (and which has proven invaluable of the years). It handled only src.rpm files, so I rewrote and extended it to handle binary and source rpm files (rqp and rqs respectively). It was a quick-n-dirty extension that was used for a few years on the Mandriva secteam and worked really well (and probably is still working well for them).

I’ve gotten rqp converted to Python and it’s working quite well. Because rqp and rqs share a *lot* of functions, I wanted to write rqp.py and rqs.py and have a shared module between them (rq.py) and this is where I’m having issues. Does anyone know how to have a module get access to an object from the calling program? For instance, rqp uses the optparse module to nicely handle the commandline arguments, but a lot of the modules use options.foo to look up arguments that were passed. The functions in rq.py (the module) can’t seem to get the value of options, and while I could put the optparse stuff in the module, it would leave the actual rqp and rqs programs really really small (and I’d have to either put in two functions because the two programs use different syntax). So I’ve got in rqp.py:

import rq

which works ok, but if rq.db_connect() requires options.database (for instance), it can’t get it. So in rq.py, I tried doing a circular import or whatever it’s called by doing:

import options from rqp

But this doesn’t want to work either. The output looks quite sad:

% ./rqp.py -d
rqp 0.2 ($Id: rqp 275 2009-03-07 22:34:15Z vdanen $)

Traceback (most recent call last):
  File "./rqp.py", line 548, in 
    import rq
  File "/Users/vdanen/svn/scripts/rq/trunk/rq.py", line 11, in 
    from rqp import options
ImportError: cannot import name options

There has to be a way to make the options variable (and one or two others I think) accessible to the module, but I haven’t been able to find a way to do it. Does anyone know how this can be accomplished?

Here’s the link: If programming languages were religions….

_arguments:comparguments:303: invalid argument: ARG

Looks like the reason is that the subversion developers changed “arg” to “ARG”. So when I googled I found this helpful post:

ZSH Tab Completion Fix for Subversion 1.5 which basically points to this new _subversion functions file to be plopped into the zsh functions directory (/usr/share/zsh/4.3.4/functions; change the version to suit of course — the directory may also be slightly different on Linux vs OS X).

Now tab completion works properly and life is good. =)