Converting from SVN to Mercurial

From ADempiere
Jump to: navigation, search
This Wiki is read-only for reference purposes to avoid broken links.

-- Historical Document -- --Tony

Note

This document contains notes on the process of converting our SVN repositories to Mercurial.

In order to simplify the explanation, this document assumes you are using a linux operating system and all VCS repositories on your local system, are located under the /opt/repos folder. It is recommended to change the ownership of this folder to your own user. At this stage, I will not cover the conversion of the SVN /contributions folder, however it will be possible to include this in the future.

Most popular conversion tools:

hgsvn - http://pypi.python.org/pypi/hgsvn/

A set of scripts to work locally on Subversion checkouts, using Mercurial.


hgsubversion - http://mercurial.selenic.com/wiki/HgSubversion

An extension for Mercurial that allows using Mercurial as a Subversion client


ConvertExtension - http://mercurial.selenic.com/wiki/ConvertExtension

The convert extension converts repositories from other SCMs (or even Mercurial itself) into Mercurial repositories, with options for filtering and renaming. It's also useful to filter Mercurial repositories to get subsets of an existing one (with the help of the --filemap option).


Of the available conversion tools, the ConvertExtension offered the best flexibility in dealing with the conversion of a non-standard SVN repository. The ConvertExtension is included by default in the standard Mercurial distribution.


With all the conversion tools tried so far, trying to convert our large SVN repository (>2.3GB) over the web, is almost impossible to achieve. Because of this it is necessary to first create a local mirror of the repository. This process still involves downloading the entire repository, but the process is easily resumed if network timeouts occur.

Creating a local SVN mirror:

Create a new SVN repository.

svnadmin create /opt/repos/adsvn

Create required pre-revprop-change hook.

cat <<'EOF' > /opt/repos/adsvn/hooks/pre-revprop-change
#!/bin/sh
USER="$3"
if [ "$USER" = "svnsync" ]; then exit 0; fi
echo "Only the svnsync user can change revprops" >&2
exit 1
EOF

Make the file executable.

chmod +x /opt/repos/adsvn/hooks/pre-revprop-change

Initialize the SVN mirror

svnsync init --username svnsync file:///opt/repos/adsvn \
        http://adempiere.svn.sourceforge.net/svnroot/adempiere

Start to sync the local mirror

Warning: this step will take a long time > 2.3GB of data to download

svnsync sync file:///opt/repos/adsvn

If the sync times out or is accidentally aborted just re-issue:

svnsync sync file:///opt/repos/adsvn

If you cannot resume the sync because the repository is locked, clear it using the following:

svn pdel --revprop -r 0 svn:sync-lock file:///opt/repos/adsvn

ADempiere SVN Repository Layout

The ADempiere SVN repository has the following layout (exluding the contributions folder, see note above):


branches/ branch1/ project files ----> Standard SVN branch, project files copied from revision xxxx
branch2/ project files ----> Standard SVN branch, project files uploaded
branch3/ other files Standard SVN branch, other files uploaded
branch4/ sub-folder1/ project files ----> Sub-folder containing SVN branch, project files copied from revision yyyy
sub-folder2/ project files ----> Sub-folder containing SVN branch, project files uploaded
sub-folder3/ other files ----> Sub-folder containing SVN branch, other files uploaded
....
release/ project files ----> New trunk, project files copied from revision zzzz
trunk/ project files ----> Old trunk, now SVN branch located at the repository root
tags/ tag1/ project files ----> Standard SVN tag
tag2/ project files ----> Standard SVN tag
....


Issues and Solutions

When attempting to convert this repository to Mercurial, there are a number of issues to solve.

1) The main SVN project folder, normally 'trunk', is now called 'release'.
This easily fixed using the following option:
--config convert.svn.trunk = release

2) The Mercurial 'default' branch is still being set to the SVN 'trunk' folder. It should be set to 'release'.
This has been submitted as a bug to the Mecurial bug tracker system.

3) The SVN 'trunk' folder is a branch not under the 'branches' folder.

4) Branches with sub-folders as in 'branch4' above, are treated as one branch, even though the intention is that each sub-folder is an individual branch. This also means that even though 'sub-folder1' contains project files that were copied from a particular revision of trunk, this is ignored by the convert tool and the files are addded. This increases the size of the converted repository.

5) Branches which contain a complete set of project files that have been uploaded to the SVN repository, as in 'branch2' above, result in an increased size of both the SVN and the converted Mercurial repositories. This isssue is not easily solved, but rather a note for future reference. The overall size of an individual repository is important with Distributed Version Control Systems, as developers need to clone a copy of it initially. Projects which are not really part of the main project, should be in their own repository.


At this stage issues 2, 3 and 4 above can only be solved by customizations to the standard Mercurial Convert Extension. I have created a patch for issue 2 and used some code from one of the Mercurial developers to address issues 3 and 4. This patch creates a new option which enables specifying the branches to be included in a text file. This allows us to include the SVN 'trunk' folder as a branch. Also, in the case of 'branch4' we can now specify the individual branches that it contains:

 /branches/branch4/sub-folder1
 /branches/branch4/sub-folder2
 /branches/branch4/sub-folder3

Because the convert tool now sees the sub-folders as individual branches, the convert tool will now read the SVN 'copy from' directive in 'sub-folder1' and treat the files as copies from a particular SVN revision.

Patching the Convert Extension

All the changes occur in one file:

/path_to_Mercurial_installation/hgext/convert/subversion.py

A patched version of this file, compatible with mercurial 1.4.2 and 1.4.3 is available at: http://bitbucket.org/tspc/hg-crew-tspc-patches/downloads/subversion.py.zip

Backup your current subversion.py and replace with the downloaded subversion.py

There is also a patch queue for those who want to apply the patches to their own mercurial project: http://bitbucket.org/tspc/hg-crew-tspc-patches/

Using the Convert Extension

To enable the Mercurial convert extension edit the .hgrc configuration file, which should be in your home folder, as shown below:

[extensions]
hgext.convert=

Create a text file in /opt/repos/branchfile.txt containing a list of the branches you wish to copy:

 trunk
 branches/3E_WebServices
 branches/adempiere321
 branches/adempiere341
 branches/adempiere343
 branches/banym
 branches/experimental
 branches/fitnesseIntegration
 branches/fix_fifo_costing
 branches/globalqss/adempiere355a
 branches/globalqss/LCO_343
 branches/globalqss/LCO_355
 branches/globalqss/patches_354
 branches/hengsin
 branches/ilogic
 branches/ivanceras
 branches/libero
 branches/liberoHR
 branches/libero_costingvariance
 branches/libero_payroll_342
 branches/metas/mvcForms
 # branches/mysql         ==> Excluded this branch as it adds 500MB to the size of the repository
 branches/osgi/approach1
 branches/osgi/approach2
 branches/pae
 branches/patches_321
 branches/patches_340
 branches/patches_342
 branches/patches_352
 branches/qa
 branches/visible_fields

Create a text file, /opt/repos/branchmap.txt, containing a list of branches to be renamed:

 adempiere355a gqss_adempiere355a
 adempiere356 gqss_adempiere356
 LCO_343 gqss_LCO_343
 LCO_355 gqss_LCO_355
 patches_354 gqss_patches_354
 mvcForms metas_mvcForms
 approach1 osgi_approach1
 approach2 osgi_approach2

Update your local svn mirror:

 svnsync sync file:///opt/repos/adsvn

Run the conversion:

 hg convert --datesort --config convert.svn.trunk=release --config convert.svn.branchfile=branchfile.txt \
            --branchmap branchmap.txt /opt/repos/adsvn /opt/repos/adhg

The conversion should take around 10 minutes and your mercurial repository will be in /opt/repos/adhg. The size of the converted repository is currently around 1GB. To keep the converted repository up to date, just repeat the svnsyc and hg convert commands above. Do not modify the converted repository directly, to experiment with it just clone a copy.

 hg clone /opt/repos/adhg /opt/repos/adhg_test

This will create a copy of the repository and automatically check-out the 'default' branch. If you don't want to check-out a working copy, use the hg clone -U option.


to be continued ..........