Documentation of mercurial setup of grml

1. Preface

Mercurial is a fast, lightweight and distributed Source Control Management system designed for efficient handling of very large distributed projects.

For grml we use a mixture between a subversion-like working practise and the kernel-like working practise.

We use /data/repos on the vserver as the directory containing the mercurial repositories. The mercurial setup/server is running inside a dedicated vserver. /data is a xfs partition made available through LVM and mounted into the vserver so we can use ACL features which vserver's ufs does not provide.

If you are interested in the "Server setup" of grml read on, if you are not interested in the server setup skip the following section and start at section "Working with mercurial".

2. Server setup

2.1. Setup of hgweb

% cp /usr/share/doc/mercurial/examples/hgwebdir.cgi /data/repos/

% cat > /data/repos/hgweb.config << EOF
[paths]
grml-etc-core/ =  /data/repos/grml-etc-core/
udev/ =  /data/repos/udev/
hg-doc/ =  /data/repos/hg-doc/

[web]
style=gitweb
allowbz2 = yes
allowgz = yes
allowzip = yes

motd = (c) grml-team 2003++<br />
       Repositories maintained by <a href="http://grml.org/team/#mika">Michael Prokop</a>.<br />
       <a href="http://grml.org/mercurial/">Documentation about the setup is available online.</a>
EOF

2.2. Apache2 setup for hgweb

# cat /etc/apache2/sites-available/default
NameVirtualHost *:80

<VirtualHost *:80>
        ServerAdmin repos@grml.org.invalid
        ServerName repos.grml.org

DocumentRoot /data/deb

<Directory />
        Options FollowSymLinks
        AllowOverride None
</Directory>

<Directory "/data">
        AllowOverride None
        Options Indexes FollowSymLinks MultiViews
        Order allow,deny
        allow from all
        # RedirectMatch ^/$ /deb/
</Directory>

<Directory "/data/repos">
        AllowOverride None
        Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
        AddHandler cgi-script .cgi
        Order allow,deny
        Allow from all
</Directory>

RedirectMatch ^/web/$ /hgwebdir.cgi
ScriptAlias   /hgwebdir.cgi   /data/repos/hgwebdir.cgi

ErrorLog /var/log/apache2/error.log

LogLevel warn

        CustomLog /var/log/apache2/access.log combined
        ServerSignature On
</VirtualHost>

<VirtualHost *:80>
        ServerName hg.grml.org
        ServerAdmin repos@grml.org.invalid

<Directory "/">
   Options ExecCGI
   AddHandler cgi-script .cgi
</Directory>

RewriteEngine on
RewriteRule (.*) /data/repos/hgwebdir.cgi$1

LogLevel warn

        ErrorLog /var/log/apache2/error.log
        CustomLog /var/log/apache2/access.log combined
        ServerSignature On
</VirtualHost>

Now repositories can be accessed via http://hg.grml.org/ - for example http://hg.grml.org/hg-doc to access the repository named hg-doc.

2.3. Activate FastCGI with Apache2 setup for hgwebdir

If you have hg.grml.org allready running it's a one minute job:

Install libapache2-mod-fastcgi and python-flup
Copy hgwebdir.cgi to hgwebdir.fcgi

Change the line in hgwebdir.fcgi from

wsgicgi.launch(wsgiapplication(make_web_app))

to

from flup.server.fcgi import WSGIServer
WSGIServer(wsgiapplication(make_web_app)).run()

This 2 lines uses the threaded fcgi version. You could also use the fork fcgi version with
```
flup.server.fcgi_fork import WSGIServer
```
To make hg.grml.org using the fcgi version you should change your apache config form something like this
```
RewriteRule (.*) /data/repos/hgwebdir.cgi$1
```
```
to
```
```
RewriteRule (.*) /data/repos/hgwebdir.fcgi$1
```
Thats it!

2.4. User setup

2.4.1. Accounts for core developers

grml core developers have a ssh account without any restrictions.

2.4.2. Accounts for contributors

Contributors are allowed to work on some selected packages (having write access therefore of course) but not on all repositories.

# grep $USER /etc/passwd
$USER:*:1002:1002:$DEMOUSER,,,:/home/$USER:/bin/zsh
      ^
Important: disable password so user can run hg-ssh but does not get a shell!

hg-ssh is a wrapper for ssh access to a limited set of mercurial repos, to be used in ~/.ssh/authorized_keys and can be found at /usr/share/doc/mercurial/examples/hg-ssh (so install hg-ssh in $PATH):

# cat /home/$USER/.ssh/authorized_keys
command="cd /data/repos/ && hg-ssh hg-doc",no-port-forwarding,no-X11-forwarding,no-agent-forwarding ssh-rsa [key...]

2.4.3. Permission handling

Set up a dummy user, we don't really use UNIX permissions but POSIX ACLs instead:

# cd /data/repos
# adduser dummy --system
# addgroup dummy --system
# addgroup dummy dummy
# chown -R dummy:dummy .

Make sure people from group repos have access anytime:

# cat /data/repos/default_acl_schema
# file: a
# owner: mika
# group: mika
user::rwx
user:www-data:r-x
group::r-x
group:www-data:r-x
group:repos:rwx
mask::rwx
other::r-x
default:user::rwx
default:user:www-data:r-x
default:group::r-x
default:group:www-data:r-x
default:group:repos:rwx
default:mask::rwx
default:other::r-x
# setfacl -R --set-file /data/repos/default_acl_schema /data/repos

Team setup - user(s) of group repos-udev should have access to their package:

# addgroup repos-udev
# addgroup tklauser repos-udev
# setfacl -R -m d:g:repos-udev:rwx,g:repos-udev:rwx udev

Example for setting up a private repository where other users should not have access to:

# cd /data/repos
# mkdir private
# chmod 750 private
# chown -R mika.mika private
# setfacl -R --remove-all private

Important: to avoid conflicts with permissions do not work on the server via ssh in the repositories in /data/repos/ themselve but clone them to your $HOME directory instead or download them to your own box! (This seems to be a bug in the permission handling of mercurial. We are investigating on this issue…)

3. Working with mercurial

Tip	Keep your changesets small, sync and push often!

First of all read the quick start howto.

3.1. Mercurial and Debian packages

3.1.1. Working with your own packages

In this case the mercurial repository on grml.org itself is interesting for you. You push all your changes immediatly to the repository.

3.1.2. For changes in other packages

You create a local repository (hg clone …), make your changes and inform the (grml) maintainer who is responsible for this package about your changes. Preferred is the mercurial nativ way to communicate changesets (push/pull/clone). Look at the different ways to communicate changes (see "How to create a patch" for more details).

3.1.3. Building the debian package

We assume you know dpkg-buildpackage -rfakeroot already.

If you use mercurial you have a single directory named .hg in the root directory of your repository (unlike svn, which has .svn directories in every single directory of your repository). To avoid shipping the .hg directory we wrote hg-buildpackage. hg-buildpackage is a simple wrapper around dpkg-buildpackage. To use it make sure you have grml-mercurial-utils installed (it's available from grml-repos, if you are using grml you have it on your box already but please make sure you are using a recent version).

3.1.4. Working with non-native Debian package

Variante 1) only directory debian/ is under revision control

% mv package package-old
% unp newupstream-package.tar.gz
% cp -a package-old/.hg package/
% cp -a package-old/debian package/
% cd package
<adjust debian packaging..>
% hg ci -m "notice about changes"
% hg push
% hg-buildpackage -rfakeroot ...

Variante 2) directory debian/ including upstream sources are under revision control:

% mv package package-old
% unp newupstream-package.tar.gz
% cp -a package-old/.hg package/
% cp -a package-old/debian package/
% cd package
% hg addremove
% hg ci -m 'new upstream version'
% hg tag <upstream version'
<adjust debian packaging..>
% hg ci -m "notice about changes"
% hg tag <debian version>
% hg push
% hg-buildpackage -rfakeroot ...

Variante 3) upstream tracking repository + versioned mercurial patch queue

% mv package package-old
% unp newupstream-package.tar.gz
% cp -a package-old/.hg package/
% cd package
% hg addremove
% hg ci -m 'new upstream version'
% hg tag <upstream version>
% hg push
% hg qpush debian.patch
<adjust debian packaging..>
% hg qci -m 'new upstream version'
% hg-buildpackage -mqd -rfakeroot
% cd .hg/patches
% hg tag <debian version>
% hg push

3.1.5. Get notifications when pushing

If you want to be notified as soon as changesets are pushed to the repository activate the incoming hook in .hg/hgrc of the repository:

[hooks]
incoming.notify = commithook

whereas commithook is a script inside $PATH:

% cat /usr/bin/commithook
#!/bin/sh
if ! [ -x /usr/bin/mail ] ; then
   exit 0
fi

SUBJECT=`hg log -r $HG_NODE | grep "^summary:" | cut -b 14-`
SUBJECT="$SUBJECT - http://hg.grml.org/$(basename `hg root`)/rev/$HG_NODE"
RECIPIENT="$(hg debugconfig | grep web.author | sed 's/web.author=//'), repos@grml.org.invalid"

if [ -n "$RECIPIENT" ] ; then
   hg log -vr $HG_NODE | mail -s "commit: $SUBJECT" "$RECIPIENT"
fi

Tip	Also take a look at CommitHook in mercurial-wiki.

3.1.6. Get notification when commiting

If you want to be notified as soon as a commit happens use the commit hook via .hg/hgrc of the repository:

[hooks]
commit = commithook

whereas commithook is a script inside $PATH:

#!/bin/sh
SUBJECT=`hg log -r $HG_NODE | grep "^summary:" | cut -b 14-`
hg log -vpr $HG_NODE | mail -s "commit: $SUBJECT" commit-list@grml.org.invalid

Tip	Also take a look at CommitHook in mercurial-wiki.

3.2. Configuration of mercurial

Configure the global setup of mercurial via /etc/mercurial/hgrc, see man 5 hgrc for details.

Important: please adjust the default name for commits (could be any config file: ~/.hgrc or repository .hg/hgrc):

[ui]
username = Michael Prokop <mika@grml.org>

Put all your personal configuration stuff into the file $HOME/.hgrc. Example configuration file from grml:

# Filename:      $HOME/.hgrc
# Purpose:       configuration file for mercurial
# Authors:       grml-team (grml.org), (c) Michael Prokop <mika@grml.org>
# Bug-Reports:   see http://grml.org/bugs/
# License:       This file is licensed under the GPL v2.
# Latest change: Mon Okt 23 00:06:34 CEST 2006 [mika]
################################################################################

# See 'man 5 hgrc' and http://www.selenic.com/mercurial/hgrc.5.html
# for more details about possibilities for configuration of mercurial.

[ui]
username = Michael Prokop <mika@grml.org>
# debug = true
# verbose = true
# merge = hgmergevim

# useful for patchbomb extension (e.g.: 'hg email -t grml@localhost tip')
[email]
from = grml User <grml@localhost>
method = /usr/sbin/sendmail

# Extension stuff, see /etc/mercurial/hgrc.d/hgext.rc
# and http://www.selenic.com/mercurial/wiki/index.cgi/ExtensionHowto

[extensions]

# Hooks to control commit access to parts of a tree.
# acl=/usr/share/python-support/mercurial/hgext/acl.py

# Update Bugzilla bugs when changesets mention them (> 0.9-1).
# bugzilla = /home/grml/mercurial-snapshot/hgext/bugzilla.py

# Graph amount of code changed per author over time (> 0.9-1).
# churn = /home/grml/mercurial-snapshot/contrib/churn.py
# churn =

# Extension for using an external program to diff repository (or
# selected files). Available in 0.9.1.
# extdiff=/usr/share/python-support/mercurial/hgext/extdiff.py
hgext.extdiff=

# Convenience wrapper for pulling and merging.
# fetch =

# Extension that provides commands to help working with trees
# composed of many Mercurial repositories. See
# http://www.terminus.org/hg/hgforest
# forest =

# Extension for signing and checking signatures.
# gpg=/usr/share/python-support/mercurial/hgext/gpg.py
# gpg=

# Extension for binary searching in O(log2(n)) for the changeset
# introducing a (mis)feature, see
# http://www.selenic.com/mercurial/wiki/index.cgi/UsingBisect
# hbisect=/usr/share/python-support/mercurial/hgext/hbisect.py

# Graphical gitk-like repository browser, invoked with hg view.
# hgk=/usr/share/python-support/mercurial/hgext/hgk.py

# Mercurial Queue management extension - see
# http://www.selenic.com/mercurial/wiki/index.cgi/MqExtension
# mq=/usr/share/python-support/mercurial/hgext/mq.py

# Template-driven email notifications, see
# http://www.selenic.com/mercurial/wiki/index.cgi/NotifyExtension
# notify=/usr/share/python-support/mercurial/hgext/notify.py
# hgext.notify =

# Extension providing the hg email command for sending a collection of
# Mercurial changesets as a series of patch emails.
# patchbomb=/usr/share/python-support/mercurial/hgext/patchbomb.py

# Cherry-picking, rebasing and changeset rewriting - see
# http://www.selenic.com/mercurial/wiki/index.cgi/TransplantExtension
# transplant =

# Extension for line ending conversion filters for the Windows platform.
# win32text=/usr/share/python-support/mercurial/hgext/win32text.py

[extdiff]
# DirDiff script for Vim: http://www.vim.org/scripts/script.php?script_id=102
# wget http://www.vim.org/scripts/download_script.php?src_id=5306 -O ~/.vim/plugin/DirDiff.vim
# Notice: opts.* works only in Mercurial >0.9.1, use hgvimdiff as wrapper therefore
cmd.vimdiff=/usr/bin/hgvimdiff
# cmd.vimdiff=/usr/bin/vim.basic
# opts.vimdiff=-f '+next' '+execute "DirDiff" argv(0) argv(1)'

# vim: ft=config

3.3. Set up a new repository

% cd /data/repos/hg-doc
% hg init .
% cat > .hg/hgrc << EOF
[paths]
default = ssh://mika@repos.grml.org//data/repos/hg-doc

[email]
from = repos@grml.org.invalid

[web]
name = hg-docu
description = documentation of hg
author = Michael Prokop
EOF

% hg add
% hg ci -m "* initial checkin"

3.4. Checkout of a repository

For developers with account:

% hg clone ssh://repos.grml.org//data/repos/hg-doc

Anonymous (without account, password,…):

% hg clone http://hg.grml.org/hg-doc

3.5. How to create a patch

If you do not have the repository yet download it via:

% hg clone http://hg.grml.org/$PACKAGENAME

If you already have the repository please make sure you are using the current version:

% hg pull
% hg update

Now apply your changes (with the editor of you choice). Then commit your changes local via running:

% hg commit -m "short information regarding your changes"

Finally export the patch:

% hg export tip > hg-doc_newfeature.patch

and send the hg-doc_newfeature.patch via mail to the maintainer of $PACKAGENAME and make sure you send a copy (CC:) to <repos@grml.org.invalid>.

Notice: using the patchbomb extension you can send the patch via mail even faster:

% hg email -t repos@grml.org.invalid tip

3.6. Working with mq

Example session:

% hg qinit -c
% hq qimport /raid/Grml/kernel/patches.2.6.18/[1-5]*
% hg qpush -a
% hg qcommit -m "initial version for 2.6.18-grml"
% hg qpop 4310_reiser4-for-2.6.18.patch
% hg qdel 4310_reiser4-for-2.6.18.patch
% hg qpush -a
% hg qcommit -m "removed 4310_reiser4-for-2.6.18.patch"
% hg qcommit -m "qrefresh for 5002_linux-2.6.17-commandline.patch"
% hg qpop 5002_linux-2.6.17-commandline.patch
% cd .hg/patches/
% hg rename 5002_linux-2.6.17-commandline.patch 5002_linux-2.6.18-commandline.patch
% cd ../..
% hg qdel 5002_linux-2.6.17-commandline.patch
% echo 5002_linux-2.6.18-commandline.patch >> .hg/patches/series
% hg qpush 5002_linux-2.6.18-commandline.patch
% hg qrefresh
% hg qcommit -m "renamed 5002_linux-2.6.17-commandline.patch into 5002_linux-2.6.18-commandline.patch"
% hg qapp

Create an all-in-one patch:

% hg -v qapp
35b24375c791bd4a6ab3f6266ed4c86e7db1c116:1000_2.6.18.1.patch
dbfdc94b23e11a79557e2380113783ab40e2ea58:2500_via-irq-quirk-revert.patch
a73fe2a1ba6744119d7c46689fbd647d29284068:4005_sky2-v1.9.patch
55ebc8d22f4bc98c5030f0630b3009fcbd4daaeb:4010_r8169-8168.patch
d1dfcd3b8952db725b5ac920e1ef0741ba0a9873:4105_dm-bbr.patch
771f8dfaca97ac92349e7949a438e5221a435afd:4110_promise-pdc2037x.patch
152b0976f9d18424139612df4f996b6b34ab89ac:4150_iteraid.patch
e0646b22c4e6accbf11e7e69f43826b101b73f0f:4300_squashfs-3.1.patch
16d9f8a538ccbdbff23353cf88bf24a6e3329b85:4400_speakup-20060814.patch
d7c39ee5a2b1b4367f5d9b2b88f1de2260202be9:5000_grml-version.patch
a900842d6ffee405cad60f94c6b5e291f513e452:5001_grml_logo.patch
6d96a8f6a4e73e9a0a1b41cc53c6708801ab882d:5002_linux-2.6.18-commandline.patch
% hg -v qapp | head -1
35b24375c791bd4a6ab3f6266ed4c86e7db1c116:1000_2.6.18.1.patch
% hg diff -r 35b24375c791bd4a6ab3f6266ed4c86e7db1c116 > all-in-one
hg diff -r 35b24375c791bd4a6ab3f6266ed4c86e7db1c116 > all-in-one  5,31s user 0,30s system 99% cpu 5,638 total

4. Tips and tricks for working

4.1. Merging

If you want to use vimdiff for merging make sure you have grml-mercurial-utils installed (it's available from grml-repos, if you are using grml you have it on your box already) and put the following line into your ~/.hgrc:

[ui]
merge = hgmergevim

You can also define your own command for merging. Very useful is the DirDiff script for Vim. Add the following lines to your ~/.hgrc:

[extdiff]
# wget http://www.vim.org/scripts/download_script.php?src_id=5306 -O ~/.vim/plugin/DirDiff.vim
# Notice: opts.* works only in Mercurial >0.9.1, use hgvimdiff as wrapper therefore
cmd.vimdiff=/usr/bin/hgvimdiff
# cmd.vimdiff=/usr/bin/vim.basic
# opts.vimdiff=-f '+next' '+execute "DirDiff" argv(0) argv(1)'

Now you can run for example hg vimdiff -r2 -r4 for running Vim in DirDiff mode for displaying the changes between revision 2 and 4.

4.2. Repository stuff

If you do not want to type the target everytime you push to the central adjust the repository config (.hg/hgrc):

[paths]
default-push = ssh://repos.grml.org//data/repos/package

If you dont want to write the source everytime you make a pull adjust the repository config (.hg/hgrc):

[paths]
default = /home/grml/work/projects/package

If you want to be able to use hg clone ssh://repos.grml.org/hg/hg-doc for cloning the repository instead of ssh://repos.grml.org//data/repos/hg-doc just create a symlink on your box:

% ln -s /data/repos ~/hg

4.3. Recursive mercurial commands

If you have multiple repositories inside a single directory you should be aware of hgr for running mercurial commands recursive.

For example run:

% cd ~/grml/hg && hgr status

to get the status of all the repositories inside ~/grml/hg.

Run:

% cd ~/grml/hg && hgr in

to show all new changesets found in source directories of all the repositories.

Notice: hgr is part of the Debian package grml-mercurial-utils. Mare sure you have it installed (it's available from grml-repos, if you are using grml you have it on your box already but please make sure you are using a recent version).

4.4. Set up zsh completion for mercurial

grml 0.9 will provide zsh completion for mercurial out-of-the-box. Read on to know what has been done to provide this feature.

Create a directory /etc/zsh/site-functions, adjust $FPATH via /etc/zsh/zshrc and copy file to appropriate location:

# mkdir /etc/zsh/site-functions
# [ -d /etc/zsh/site-functions ] && export FPATH=/etc/zsh/site-functions:$FPATH
# zcat /usr/share/doc/mercurial/examples/zsh_completion.gz > /etc/zsh/site-functions/_hg

4.5. Install current mercurial snapshot in $HOME

Make sure you have python2.4-dev, gcc and libc6-dev on your system, then run:

% wget http://www.selenic.com/mercurial/mercurial-snapshot.tar.gz
% unp mercurial-snapshot.tar.gz
% cd mercurial-snapshot
% make install-home-bin

and finally make sure you have the following PATH set:

% export PYTHONPATH=${HOME}/lib/python
% export PATH=${HOME}/bin:$PATH

Tip	Take a look at the zsh functions gethgclone (get latest hg version via hg itself) and gethgsnap (get latest mercurial-snapshot.tar.gz) in /etc/skel/.zshrc (becoming $HOME/.zshrc) on grml.

4.6. Install current mercurial version

Upgrading mercurial might be interesting for you if you are runing hgweb so you can use some brand new features. It is very simple and easy to keep it up2date.

Make sure you have python2.4-dev, gcc and libc6-dev on your system, then run:

% cd /data
% hg clone http://selenic.com/repo/hg mercurial
% cd mercurial
% make local

Then insert/adjust sys.path.insert(0, "/data/mercurial") in hgwebdir.cgi. That's it!

4.7. Import patches from mails

Just run hg import -. If you are using mutt[-ng] the following snippet might be interesting for your ~/.mutt[ng]rc:

macro pager ,i "|hg import --cwd hg/repos/hg-doc -" "hg import hg/repos/hg-doc"

4.8. Static URLs for files via hgweb

Using mercurial version >0.9.1 provides very handy, static URLs. Examples:

4.9. Serve your own repositories for LAN access

You want to share you local repositories temporary so other people (for example in your LAN) can access it as well?

% hg serve

That's it. You can even serve your repositories via hgweb, just run:

% cat > hgwebconfig << EOF
[paths]
demorepos = /tmp/demorepos

[web]
style=gitweb
% hg serve -d --webdir-conf hgwebconfig

4.10. Speed up ssh access to repository

Since OpenSSH 4.0 a feature called ControlMaster is available. Activate ControlMaster through your ssh config:

% cat >> ~/.ssh/config << EOF
Host *
ControlMaster auto
ControlPath ~/.ssh/master-%r@%h:%p
EOF

Make sure you have an open connection to the server, then commands like hg push will happen much faster.

4.11. Avoid typing of ssh key passphrase

You can either use ssh-agent:

% eval $(ssh-agent) && ssh-add

or use keychain instead:

% cat ~/.zlogin
if [[ -x =keychain ]] ; then
  eval $(keychain --nocolor --quiet --agents ssh --eval ~/.ssh/id_rsa)
fi

4.12. Tips for debugging mercurial

Problems with mercurial you can't explain at all, even though you read the docs? :-)

Take a look at mercurial's global options:

-v --verbose         enable additional output
   --debug           enable debugging output
   --debugger        start debugger

Using strace might help as well, run something like:

% strace -f -eopen hg status

A hardcore variant is using pydb, an enhanced Python command-line debugger:

% pydb -X =hg status

5. Todo - Checkout

find a way for hg log —style=changelog to provide release based verbose, full changelogs [based on tags?]
extend hg-buildpackage
/usr/share/python-support/mercurial/hgext/hbisect.py
/usr/share/doc/mercurial/examples/vim/patchreview.vim.gz