Quantcast

SVN -> Git *but* with special changes

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

SVN -> Git *but* with special changes

Abscissa
I have a couple big projects in SVN that I'd like to convert to Git. Being in SVN, they've operated under a couple assumptions that are not true under Git. These assumptions are:

1. Directories can exist even if there's nothing in them (just like any filesystem).

2. Keeping binary files in version control isn't a big deal because the whole repo doesn't get copied to everyone's system or use up people's GitHub storage space.

The SVN repos have been relying on those, but both are false under Git, so I need to do a "modified" conversion, rather than just a straight one.

So, how can I convert an SVN repo to Git, and have the conversion add dummy files to empty directories and exclude specific files? (Also, there are tags and branches to be converted too, in the SVN-standard "tags" and "branches" directories.)
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SVN -> Git *but* with special changes

Stephen Bash
----- Original Message -----

> From: "Abscissa" <[hidden email]>
> To: [hidden email]
> Sent: Wednesday, September 28, 2011 12:37:29 PM
> Subject: SVN -> Git *but* with special changes
>
> I have a couple big projects in SVN that I'd like to convert to Git.
> Being in SVN, they've operated under a couple assumptions that are
> not true under Git. These assumptions are:
>
> 1. Directories can exist even if there's nothing in them (just like
> any filesystem).

What requires the empty directories to exist?  The build system?  Can you just let them go away in Git and fix it downstream in the user's working copy?
 
> 2. Keeping binary files in version control isn't a big deal because
> the whole repo doesn't get copied to everyone's system or use up
> people's GitHub storage space.

I'd eliminate these on the SVN side before converting to Git.  If you have svnadmin access to the SVN repo svnadmin dump, svndumpfilter, svnadmin load is a pretty easy process.

> The SVN repos have been relying on those, but both are false under
> Git, so I need to do a "modified" conversion, rather than just a
> straight one.
>
> So, how can I convert an SVN repo to Git, and have the conversion add
> dummy files to empty directories and exclude specific files? (Also,
> there are tags and branches to be converted too, in the SVN-standard
> "tags" and "branches" directories.)

To do the actual conversion, svn-fe and git fast-import are by far the quickest way to get the data into Git.  Filtering into tags and branches is a bit of a trick though [1].  git-svn has (IMO) a good branching UI, but can be very slow for large repositories.

[1] http://thread.gmane.org/gmane.comp.version-control.git/158940/focus=159151 : note this thread is almost a year out of date now, and even I know much better ways to go about this now -- but the scripts are not written.

HTH,
Stephen
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SVN -> Git *but* with special changes

Matthieu Moy-2
Stephen Bash <[hidden email]> writes:

>> 2. Keeping binary files in version control isn't a big deal because
>> the whole repo doesn't get copied to everyone's system or use up
>> people's GitHub storage space.
>
> I'd eliminate these on the SVN side before converting to Git. If you
> have svnadmin access to the SVN repo svnadmin dump, svndumpfilter,
> svnadmin load is a pretty easy process.

Otherwise, you can do it on the Git side with:

  git filter-branch --tree-filter 'rm -f some-large-blob'

--
Matthieu Moy
http://www-verimag.imag.fr/~moy/
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SVN -> Git *but* with special changes

Abscissa
In reply to this post by Stephen Bash
>> 1. Directories can exist even if there's nothing in them (just like
>> any filesystem).
>
>What requires the empty directories to exist?  The build system?

Yes.

>Can you just let them go away in Git and fix it downstream in the user's working copy?

That won't work retroactively for already-existing revisions. I'd rather not break the build system for older revisions just for such a goofy reason as empty dirs being deemed unnecessary.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SVN -> Git *but* with special changes

Jeff King
In reply to this post by Abscissa
On Wed, Sep 28, 2011 at 09:37:29AM -0700, Abscissa wrote:

> I have a couple big projects in SVN that I'd like to convert to Git. Being in
> SVN, they've operated under a couple assumptions that are not true under
> Git. These assumptions are:
>
> 1. Directories can exist even if there's nothing in them (just like any
> filesystem).

The usual technique is to put an empty .gitignore file into the empty
directory, which will make sure it always exists.  If you import with
git-svn, it supports "--preserve-empty-dirs", which will do this for you
automatically.

> 2. Keeping binary files in version control isn't a big deal because the
> whole repo doesn't get copied to everyone's system or use up people's GitHub
> storage space.

git-svn --ignore-paths will handle this for you.

Side note on the GitHub thing: all of the forks will share objects, so
it's not much as space as you might think. And all of the paid plans
count repos, not bytes. Of course, the painful part is probably cloning
the big objects to everybody's workstation. :)

> (Also, there are tags and branches to be converted too, in the
> SVN-standard "tags" and "branches" directories.)

git svn --stdlayout ?

-Peff
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SVN -> Git *but* with special changes

Jeff King
In reply to this post by Matthieu Moy-2
On Wed, Sep 28, 2011 at 07:44:24PM +0200, Matthieu Moy wrote:

> > I'd eliminate these on the SVN side before converting to Git. If you
> > have svnadmin access to the SVN repo svnadmin dump, svndumpfilter,
> > svnadmin load is a pretty easy process.
>
> Otherwise, you can do it on the Git side with:
>
>   git filter-branch --tree-filter 'rm -f some-large-blob'

It's much more efficient to do:

    git filter-branch --index-filter \
      'git rm --cached --ignore-unmatch some-large-blob'

which avoids checking out the large files over and over[1], just to
delete them. Of course, it's even more efficient not to import them from
svn in the first place. :)

-Peff

[1] This is straight from the filter-branch manpage, btw.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SVN -> Git *but* with special changes

Abscissa
In reply to this post by Stephen Bash
I got busy with other things and haven't had a chance to get back to this until now. I very much appreciate the help but I'm still unclear on this part:

>git-svn has (IMO) a good branching UI, but can be very slow for large repositories.
>
>[1] http://thread.gmane.org/gmane.comp.version-control.git/158940/focus=159151 : note this thread is almost a year out of date now, and even I know much better ways to go about this now -- but the scripts are not written.

I don't really understand that. So...how do I convert from svn to git while keeping branches and tags intact? If it's automated, I don't care about speed: I can just start it and move on to other things while it's working.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SVN -> Git *but* with special changes

Abscissa
In reply to this post by Jeff King
>If you import with
>git-svn, it supports "--preserve-empty-dirs", which will do this for you
>automatically.

I came across this, and it seems to mostly work: http://john.albin.net/git/git-svn-migrate

Except it fails to do the "--preserve-empty-dirs". It does say that it passes any options it doesn't recognize directly to git-svn, so I gave it that, but just got:

- Cloning repository...
Unknown option: preserve-empty-dirs

Now, I realize this isn't a forum for that particular "git-svn-migrate" tool, and I don't expect anyone to go digging though it on my account, so I'm willing to try to add it into the script if I can just find out the proper way to use "--preserve-empty-dirs".

However, I suspect that script may not be my problem at all: If I do this:

git svn help | grep preserve

I get absolutely nothing. There doesn't seem to be a "--preserve-empty-dirs".
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SVN -> Git *but* with special changes

Jeff King
On Sat, Jan 07, 2012 at 09:03:51PM -0800, Abscissa wrote:

> However, I suspect that script may not be my problem at all: If I do this:
>
> git svn help | grep preserve
>
> I get absolutely nothing. There doesn't seem to be a
> "--preserve-empty-dirs".

What version of git are you using? That option was added in git 1.7.7.

-Peff
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SVN -> Git *but* with special changes

Abscissa
Ok, I see. It's reporting 1.7.0, so that would explain it. One "sudo apt-get upgrade git" and...erm...well, it seems to be upgrading my whole damn computer, but I'll see how it all works out. Thanks!
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SVN -> Git *but* with special changes

Abscissa
Well that's strange, it finished "upgrading", but now git is still just reporting 1.7.0.4, which is *exactly* the same version it said before. The git-svn package should already be up-to-date because I just installed it  today. So I don't know what's up with that.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SVN -> Git *but* with special changes

Carlos Martín Nieto
On Sat, Jan 07, 2012 at 09:25:27PM -0800, Abscissa wrote:
> Well that's strange, it finished "upgrading", but now git is still just
> reporting 1.7.0.4, which is *exactly* the same version it said before. The
> git-svn package should already be up-to-date because I just installed it
> today. So I don't know what's up with that.

Nothing odd about that. apt-get upgrade means "upgrade my system". If
you want to get a newer version of package X, you do apt-get install X
and it will install the latest version of that package.

Your OS seems ancient, you might be better off installing from source.

   cmn

signature.asc (501 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SVN -> Git *but* with special changes

Andreas Schwab-3
Carlos Martín Nieto <[hidden email]> writes:

> On Sat, Jan 07, 2012 at 09:25:27PM -0800, Abscissa wrote:
>> Well that's strange, it finished "upgrading", but now git is still just
>> reporting 1.7.0.4, which is *exactly* the same version it said before. The
>> git-svn package should already be up-to-date because I just installed it
>> today. So I don't know what's up with that.
>
> Nothing odd about that. apt-get upgrade means "upgrade my system". If
> you want to get a newer version of package X, you do apt-get install X
> and it will install the latest version of that package.

If apt-get upgrade doesn't get you a newer version then apt-get install
won't help you either.  Both use the same installation source.

Andreas.

--
Andreas Schwab, [hidden email]
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SVN -> Git *but* with special changes

Thomas Hochstein
In reply to this post by Abscissa
Abscissa schrieb:

> Ok, I see. It's reporting 1.7.0, so that would explain it. One "sudo apt-get
> upgrade git" and...erm...well, it seems to be upgrading my whole damn
> computer, but I'll see how it all works out. Thanks!

"apt-get upgrade" will upgrade _all_ packages and doesn't take a
parameter:
| upgrade
|    upgrade is used to install the newest versions of all packages
|    currently installed on the system from the sources enumerated in
|    /etc/apt/sources.list.

-thh

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SVN -> Git *but* with special changes

Thomas Hochstein
In reply to this post by Abscissa
Abscissa wrote:

> Well that's strange, it finished "upgrading", but now git is still just
> reporting 1.7.0.4, which is *exactly* the same version it said before.

What kind of distribution do you use?

"apt-get" sounds like Debian or Ubuntu, but those all have at least
git 1.7.1:

| Debian versions:
| stable      1:1.7.2.5-3
| stable-bpo  1:1.7.7.3-1~bpo60+2
| testing     1:1.7.7.3-1
| unstable    1:1.7.8.2-1
| exp         1:1.7.8~rc3-1

| Ubuntu versions:
| Precise Pangolin  1:1.7.7.3-1
| Oneiric Ocelot    1:1.7.5.4-1
| Natty Narwhal     1:1.7.4.1-3
| Maverick Meerkat  1:1.7.1-1.1ubuntu0.1

-thh

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SVN -> Git *but* with special changes

Adam Borowski
In reply to this post by Andreas Schwab-3
On Sun, Jan 08, 2012 at 11:47:52AM +0100, Andreas Schwab wrote:

> Carlos Martín Nieto <[hidden email]> writes:
> > On Sat, Jan 07, 2012 at 09:25:27PM -0800, Abscissa wrote:
> >> Well that's strange, it finished "upgrading", but now git is still just
> >> reporting 1.7.0.4, which is *exactly* the same version it said before. The
> >> git-svn package should already be up-to-date because I just installed it
> >> today. So I don't know what's up with that.
> >
> > Nothing odd about that. apt-get upgrade means "upgrade my system". If
> > you want to get a newer version of package X, you do apt-get install X
> > and it will install the latest version of that package.
>
> If apt-get upgrade doesn't get you a newer version then apt-get install
> won't help you either.
No, this is true only if none of packages involved uses a new library
(including new sonames).  "apt-get upgrade" is forbidden to add or remove
packages, and thus will skip upgrades that need a new dependency.
The message at the end will mention "## not upgraded" though.

"apt-get install X" or "apt-get dist-upgrade" have no such restrictions.

--
1KB // Yo momma uses IPv4!

signature.asc (501 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SVN -> Git *but* with special changes

Abscissa
I see. That's strange, I've been using "apt-get upgrade xxx" for at least a couple years, I can't believe I never knew that! Thanks for the detailed info, everyone (and for the patience!)

In answer to someone's question, yea, it's Ubuntu (Kubuntu 10.04, and yea, I know that's old, but it's not my primary system and I haven't had a chance yet to upgrade it and get everything set back up again). I also tried it on a Debian 6 Live/Persistent system and got the same results...apparently for the same reason.

I did manage to get 1.7.8 installed (on both systems) by downloading and installing the source (it was kind of a pain figuring out the names of some of the dependent packages, but I managed to get it.)

So hopefully this should all work now...
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SVN -> Git *but* with special changes

Abscissa
Damn. That 'git-svn-migrate' tool is just giving me errors now. It just gets through little over 10% of the revisions, and gives:

-----------------------------
Failed to strip path 'bin/lang/.gitignore' ((?-xism:^trunk(/|$)))

- Converting svn:ignore properties into a .gitignore file...
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions
log --no-color --no-decorate --first-parent --pretty=medium HEAD: command returned error: 128
-----------------------------

And then it keeps going and gives:

-----------------------------
To /var/git/Goldie.git
 ! [rejected]        master -> trunk (non-fast-forward)
error: failed to push some refs to '/var/git/Goldie.git'
To prevent you from losing history, non-fast-forward updates were rejected
-----------------------------

I guess I'll have to try some other approach. :/
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SVN -> Git *but* with special changes

Michael Haggerty-2
In reply to this post by Abscissa
On 01/08/2012 11:28 PM, Abscissa wrote:
> In answer to someone's question, yea, it's Ubuntu (Kubuntu 10.04, and yea, I
> know that's old, but it's not my primary system and I haven't had a chance
> yet to upgrade it and get everything set back up again). I also tried it on
> a Debian 6 Live/Persistent system and got the same results...apparently for
> the same reason.

For Ubuntu users: there is a git PPA [1] which usually has quite
up-to-date git packages for recent Ubuntu releases.  If you configure
your system to use this PPA (instructions are on the page) then you can
stay up-to-date with minimal effort.  Currently they have version
1.7.8.2 available for hardy, lucid, maverick, natty, and oneiric and
older versions for some other Ubuntu releases.

Caveat: I am not affiliated with the PPA and cannot vouch for its integrity.

Michael

[1] https://launchpad.net/~git-core/+archive/ppa

--
Michael Haggerty
[hidden email]
http://softwareswirl.blogspot.com/
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SVN -> Git *but* with special changes

Abscissa
In reply to this post by Abscissa
I'm trying to do it manually with git-svn, but I'm getting the same problem. I have two different SVN repos I want to do (turns out neither have branches, I had thought they did, but one of them does have tags). Doing this:

git svn clone {the url to the SVN repo} --prefix=svn/ --preserve-empty-dirs --authors-file={already prepared authors file} --trunk=trunk --tags=tags

It gets partway through, and then gives me:

Failed to strip path '{some path}/.gitignore' ((?-xism:^trunk(/|$)))

Where {some path} is an empty dir in the trunk of one repo, and a completely non-existent path in the other. (In both cases I'm looking at the revision git-svn had gotten to when bailing, plus the revision before and the revision after).

I don't have a clue what that error message is trying to tell me or what is going wrong. :(

If it helps, these are the repos I'm trying to convert:

http://svn.dsource.org/projects/semitwist   (git svn clone fails at r46 (out of 242) on 'src/nonFatalAssertTest/.gitignore')
http://svn.dsource.org/projects/goldie   (git svn clone fails at r85 (out of 557) on 'bin/lang/.gitignore')

(Yes, they do have a non-standard top-level "downloads" directory, but that's just how that host does file downloads, and I *don't* need to preserve it)
12
Loading...