Converting from SVN to Mercurial
-- Historical Document -- --Tony
This document contains notes on the process of converting our SVN repositories to Mercurial.
In order to simplify the explanation, this document assumes you are using a linux operating system and all VCS repositories on your local system, are located under the /opt/repos folder. It is recommended to change the ownership of this folder to your own user. At this stage, I will not cover the conversion of the SVN /contributions folder, however it will be possible to include this in the future.
Most popular conversion tools:
A set of scripts to work locally on Subversion checkouts, using Mercurial.
hgsubversion - http://mercurial.selenic.com/wiki/HgSubversion
An extension for Mercurial that allows using Mercurial as a Subversion client
ConvertExtension - http://mercurial.selenic.com/wiki/ConvertExtension
The convert extension converts repositories from other SCMs (or even Mercurial itself) into Mercurial repositories, with options for filtering and renaming. It's also useful to filter Mercurial repositories to get subsets of an existing one (with the help of the --filemap option).
Of the available conversion tools, the ConvertExtension offered the best flexibility in dealing with the conversion of a non-standard SVN repository. The ConvertExtension is included by default in the standard Mercurial distribution.
With all the conversion tools tried so far, trying to convert our large SVN repository (>2.3GB) over the web, is almost impossible to achieve. Because of this it is necessary to first create a local mirror of the repository. This process still involves downloading the entire repository, but the process is easily resumed if network timeouts occur.
Creating a local SVN mirror:
Create a new SVN repository.
svnadmin create /opt/repos/adsvn
Create required pre-revprop-change hook.
cat <<'EOF' > /opt/repos/adsvn/hooks/pre-revprop-change #!/bin/sh USER="$3" if [ "$USER" = "svnsync" ]; then exit 0; fi echo "Only the svnsync user can change revprops" >&2 exit 1 EOF
Make the file executable.
chmod +x /opt/repos/adsvn/hooks/pre-revprop-change
Initialize the SVN mirror
svnsync init --username svnsync file:///opt/repos/adsvn \ http://adempiere.svn.sourceforge.net/svnroot/adempiere
Start to sync the local mirror
Warning: this step will take a long time > 2.3GB of data to download
svnsync sync file:///opt/repos/adsvn
If the sync times out or is accidentally aborted just re-issue:
svnsync sync file:///opt/repos/adsvn
If you cannot resume the sync because the repository is locked, clear it using the following:
svn pdel --revprop -r 0 svn:sync-lock file:///opt/repos/adsvn
ADempiere SVN Repository Layout
The ADempiere SVN repository has the following layout (exluding the contributions folder, see note above):
|branches/||branch1/||project files||---->||Standard SVN branch, project files copied from revision xxxx|
|branch2/||project files||---->||Standard SVN branch, project files uploaded|
|branch3/||other files||Standard SVN branch, other files uploaded|
|branch4/||sub-folder1/||project files||---->||Sub-folder containing SVN branch, project files copied from revision yyyy|
|sub-folder2/||project files||---->||Sub-folder containing SVN branch, project files uploaded|
|sub-folder3/||other files||---->||Sub-folder containing SVN branch, other files uploaded|
|release/||project files||---->||New trunk, project files copied from revision zzzz|
|trunk/||project files||---->||Old trunk, now SVN branch located at the repository root|
|tags/||tag1/||project files||---->||Standard SVN tag|
|tag2/||project files||---->||Standard SVN tag|
Issues and Solutions
When attempting to convert this repository to Mercurial, there are a number of issues to solve.
1) The main SVN project folder, normally 'trunk', is now called 'release'.
This easily fixed using the following option:
--config convert.svn.trunk = release
2) The Mercurial 'default' branch is still being set to the SVN 'trunk' folder. It should be set to 'release'.
This has been submitted as a bug to the Mecurial bug tracker system.
3) The SVN 'trunk' folder is a branch not under the 'branches' folder.
4) Branches with sub-folders as in 'branch4' above, are treated as one branch, even though the intention is that each sub-folder is an individual branch. This also means that even though 'sub-folder1' contains project files that were copied from a particular revision of trunk, this is ignored by the convert tool and the files are addded. This increases the size of the converted repository.
5) Branches which contain a complete set of project files that have been uploaded to the SVN repository, as in 'branch2' above, result in an increased size of both the SVN and the converted Mercurial repositories. This isssue is not easily solved, but rather a note for future reference. The overall size of an individual repository is important with Distributed Version Control Systems, as developers need to clone a copy of it initially. Projects which are not really part of the main project, should be in their own repository.
At this stage issues 2, 3 and 4 above can only be solved by customizations to the standard Mercurial Convert Extension. I have created a patch for issue 2 and used some code from one of the Mercurial developers to address issues 3 and 4. This patch creates a new option which enables specifying the branches to be included in a text file. This allows us to include the SVN 'trunk' folder as a branch. Also, in the case of 'branch4' we can now specify the individual branches that it contains:
/branches/branch4/sub-folder1 /branches/branch4/sub-folder2 /branches/branch4/sub-folder3
Because the convert tool now sees the sub-folders as individual branches, the convert tool will now read the SVN 'copy from' directive in 'sub-folder1' and treat the files as copies from a particular SVN revision.
Patching the Convert Extension
All the changes occur in one file:
A patched version of this file, compatible with mercurial 1.4.2 and 1.4.3 is available at: http://bitbucket.org/tspc/hg-crew-tspc-patches/downloads/subversion.py.zip
Backup your current subversion.py and replace with the downloaded subversion.py
There is also a patch queue for those who want to apply the patches to their own mercurial project: http://bitbucket.org/tspc/hg-crew-tspc-patches/
Using the Convert Extension
To enable the Mercurial convert extension edit the .hgrc configuration file, which should be in your home folder, as shown below:
Create a text file in /opt/repos/branchfile.txt containing a list of the branches you wish to copy:
trunk branches/3E_WebServices branches/adempiere321 branches/adempiere341 branches/adempiere343 branches/banym branches/experimental branches/fitnesseIntegration branches/fix_fifo_costing branches/globalqss/adempiere355a branches/globalqss/LCO_343 branches/globalqss/LCO_355 branches/globalqss/patches_354 branches/hengsin branches/ilogic branches/ivanceras branches/libero branches/liberoHR branches/libero_costingvariance branches/libero_payroll_342 branches/metas/mvcForms # branches/mysql ==> Excluded this branch as it adds 500MB to the size of the repository branches/osgi/approach1 branches/osgi/approach2 branches/pae branches/patches_321 branches/patches_340 branches/patches_342 branches/patches_352 branches/qa branches/visible_fields
Create a text file, /opt/repos/branchmap.txt, containing a list of branches to be renamed:
adempiere355a gqss_adempiere355a adempiere356 gqss_adempiere356 LCO_343 gqss_LCO_343 LCO_355 gqss_LCO_355 patches_354 gqss_patches_354 mvcForms metas_mvcForms approach1 osgi_approach1 approach2 osgi_approach2
Update your local svn mirror:
svnsync sync file:///opt/repos/adsvn
Run the conversion:
hg convert --datesort --config convert.svn.trunk=release --config convert.svn.branchfile=branchfile.txt \ --branchmap branchmap.txt /opt/repos/adsvn /opt/repos/adhg
The conversion should take around 10 minutes and your mercurial repository will be in /opt/repos/adhg. The size of the converted repository is currently around 1GB. To keep the converted repository up to date, just repeat the svnsyc and hg convert commands above. Do not modify the converted repository directly, to experiment with it just clone a copy.
hg clone /opt/repos/adhg /opt/repos/adhg_test
This will create a copy of the repository and automatically check-out the 'default' branch. If you don't want to check-out a working copy, use the hg clone -U option.
to be continued ..........