[PATCH 0/5] Fix 'url.*.insteadOf' for submodule URLs

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

[PATCH 0/5] Fix 'url.*.insteadOf' for submodule URLs

Johan Herland
As suggested in a thread some time ago, one could redefine the URL used to
fetch submodules by adding a 'url.*.insteadOf' rule prior to the first
invocation of 'git submodule update'.

However, this does not work with current Git, because the super-repo config
(which is home to the 'url.*.insteadOf' rule) is not consulted by the 'git
clone' that is invoked by 'git submodule update'.

These patches fix this issue by making 'git submodule' explicitly rewrite
the submodule URL according to the super-repo config, prior to calling 'git
clone'.

In order for the 'git submodule' shell script to properly rewrite URLs, it
must gain access to the URL rewriting functionality in remote.c. To expose
the URL rewriting functionality to 'git submodule' (and others, if needed),
a new option ('rewrite-url') has been added to 'git config'.

The patch series is based on master. Whether or not this should be
considered for v1.6.0 is to be decided by Junio. (My personal opinion is
that although we're late in the release cycle, the patches are fairly
straightforward, and should not pose a great risk of regressions.)

Johan Herland (5):
  Add testcase for 'git submodule' with url.*.insteadOf set in the
    super-repo
  Teach 'git config' to rewrite URLs according to current
    url.*.insteadOf rules
  Add selftest for new option '--rewrite-url' to 'git config'
  Add documentation on the new --rewrite-url option to 'git config'
  Teach 'git submodule' to rewrite submodule URLs according to
    super-repo's rules

 Documentation/git-config.txt |   10 ++++++++++
 builtin-config.c             |   23 ++++++++++++++++++++++-
 git-submodule.sh             |    6 ++++++
 t/t1300-repo-config.sh       |   14 ++++++++++++++
 t/t7400-submodule-basic.sh   |   11 +++++++++++
 5 files changed, 63 insertions(+), 1 deletions(-)


Have fun!

...Johan
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

[PATCH 1/5] Add testcase for 'git submodule' with url.*.insteadOf set in the super-repo

Johan Herland
Currently, setting url.*.insteadOf in the super-repo in order to rewrite
submodule URLs, don't work. When cloning/fetching the submodule, the
super-repo config is never consulted, and thus the url.*.insteadOf rule
is never seen. This adds a testcase that confirms the current behaviour.

Signed-off-by: Johan Herland <[hidden email]>
---
 t/t7400-submodule-basic.sh |   11 +++++++++++
 1 files changed, 11 insertions(+), 0 deletions(-)

diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index cbc0c34..bafc46c 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -209,4 +209,15 @@ test_expect_success 'update --init' '
 
 '
 
+test_expect_failure 'update --init with url.*.insteadOf' '
+
+ rm -rf init &&
+ git config -f .gitmodules submodule.example.url "http://example.com/init2"
&&
+ git config --remove-section submodule.example
+ git config "url.$(pwd)/.insteadOf" "http://example.com/" &&
+ git submodule update --init init &&
+ test -d init/.git
+
+'
+
 test_done
--
1.6.0.rc1.34.g0fe8c


--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

[PATCH 2/5] Teach 'git config' to rewrite URLs according to current url.*.insteadOf rules

Johan Herland
In reply to this post by Johan Herland
This patch adds the --rewrite-url option to 'git config'. The option takes
exactly one argument; a URL that is to be rewritten according to the longest
matching url.*.insteadOf rule in the current config. The resulting URL is
printed on stdout, and 0 is returned.

The rationale for this patch is to enable access to Git's URL rewriting
functionality from shell scripts.

The URL rewriting functionality is implemented by piggybacking on the
existing URL rewriting code in remote.c.

Signed-off-by: Johan Herland <[hidden email]>
---
 builtin-config.c |   23 ++++++++++++++++++++++-
 1 files changed, 22 insertions(+), 1 deletions(-)

diff --git a/builtin-config.c b/builtin-config.c
index 91fdc49..977e6cb 100644
--- a/builtin-config.c
+++ b/builtin-config.c
@@ -1,9 +1,10 @@
 #include "builtin.h"
 #include "cache.h"
 #include "color.h"
+#include "remote.h"
 
 static const char git_config_set_usage[] =
-"git config [ --global | --system | [ -f | --file ] config-file ] [ --bool | --int | --bool-or-int ] [ -z | --null ]
[--get | --get-all | --get-regexp | --replace-all | --add | --unset | --unset-all] name [value [value_regex]] | --rename-section old_name new_name | --remove-section name | --list | --get-color
var [default] | --get-colorbool name [stdout-is-tty]";
+"git config [ --global | --system | [ -f | --file ] config-file ] [ --bool | --int | --bool-or-int ] [ -z | --null ]
[--get | --get-all | --get-regexp | --replace-all | --add | --unset | --unset-all] name [value [value_regex]] | --rename-section old_name new_name | --remove-section name | --list | --get-color
var [default] | --get-colorbool name [stdout-is-tty] | --rewrite-url url";
 
 static char *key;
 static regex_t *key_regexp;
@@ -281,6 +282,24 @@ static int get_colorbool(int argc, const char **argv)
  }
 }
 
+static int rewrite_url(int argc, const char **argv)
+{
+ /*
+ * git config --rewrite_url <url>
+ *
+ * returns <url> after rewriting it according to url.*.insteadOf rules.
+ */
+
+ if (argc > 1)
+ usage(git_config_set_usage);
+
+ struct remote *remote = remote_get(argv[0]);
+ if (remote->url_nr != 1)
+ die("Expected exactly one URL from remote_get()!");
+ printf("%s\n", remote->url[0]);
+ return 0;
+}
+
 int cmd_config(int argc, const char **argv, const char *prefix)
 {
  int nongit;
@@ -362,6 +381,8 @@ int cmd_config(int argc, const char **argv, const char *prefix)
  return get_color(argc-2, argv+2);
  } else if (!strcmp(argv[1], "--get-colorbool")) {
  return get_colorbool(argc-2, argv+2);
+ } else if (!strcmp(argv[1], "--rewrite-url")) {
+ return rewrite_url(argc-2, argv+2);
  } else
  break;
  argc--;
--
1.6.0.rc1.34.g0fe8c


--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

[PATCH 3/5] Add selftest for new option '--rewrite-url' to 'git config'

Johan Herland
In reply to this post by Johan Herland
Simple check that URL rewriting functionality works as expected.

Signed-off-by: Johan Herland <[hidden email]>
---
 t/t1300-repo-config.sh |   14 ++++++++++++++
 1 files changed, 14 insertions(+), 0 deletions(-)

diff --git a/t/t1300-repo-config.sh b/t/t1300-repo-config.sh
index 64567fb..dcd04c2 100755
--- a/t/t1300-repo-config.sh
+++ b/t/t1300-repo-config.sh
@@ -741,4 +741,18 @@ test_expect_success 'symlinked configuration' '
 
 '
 
+test_expect_success '--rewrite-url' '
+
+ git config url.ssh://example.com/foo/.insteadOf http://example.com/ &&
+ git config url.../.insteadOf git://example.com/ &&
+ a=$(git config --rewrite-url http://example.com/baz/xyzzy.git) &&
+ test "zssh://example.com/foo/baz/xyzzy.git" = "z$a" &&
+ b=$(git config --rewrite-url git://example.com/baz/xyzzy.git) &&
+ test "z../baz/xyzzy.git" = "z$b" &&
+ c=$(git config --rewrite-url ssh://example.com/foo/baz/xyzzy.git) &&
+ test "zssh://example.com/foo/baz/xyzzy.git" = "z$c" &&
+ d=$(git config --rewrite-url rsync://example.com/baz/xyzzy.git) &&
+ test "zrsync://example.com/baz/xyzzy.git" = "z$d"
+'
+
 test_done
--
1.6.0.rc1.34.g0fe8c


--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

[PATCH 4/5] Add documentation on the new --rewrite-url option to 'git config'

Johan Herland
In reply to this post by Johan Herland
Signed-off-by: Johan Herland <[hidden email]>
---
 Documentation/git-config.txt |   10 ++++++++++
 1 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/Documentation/git-config.txt b/Documentation/git-config.txt
index 28e1861..154c80b 100644
--- a/Documentation/git-config.txt
+++ b/Documentation/git-config.txt
@@ -22,6 +22,7 @@ SYNOPSIS
 'git config' [<file-option>] [-z|--null] -l | --list
 'git config' [<file-option>] --get-color name [default]
 'git config' [<file-option>] --get-colorbool name [stdout-is-tty]
+'git config' [<file-option>] --rewrite-url url
 
 DESCRIPTION
 -----------
@@ -157,6 +158,15 @@ See also <<FILES>>.
  output.  The optional `default` parameter is used instead, if
  there is no color configured for `name`.
 
+--rewrite-url url::
+
+ Rewrite `url` according to the longest matching URL rewriting rule
+ (see documentation on `url.<base>.insteadOf` for more information
+ on URL rewriting rules), and output the resulting URL to the
+ standard output. If there is no matching URL rewriting rule, the
+ original `url` is printed on the standard output. In either case,
+ the exit code is 0.
+
 [[FILES]]
 FILES
 -----
--
1.6.0.rc1.34.g0fe8c


--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

[PATCH 5/5] Teach 'git submodule' to rewrite submodule URLs according to super-repo's rules

Johan Herland
In reply to this post by Johan Herland
When a 'url.*.insteadOf' rule in the superrepo matches a submodule URL, we
should rewrite the submodule URL accordingly, so that we don't request the
submodule from location that is inaccessible (or unwanted by the user for
some other reason).

Signed-off-by: Johan Herland <[hidden email]>
---
 git-submodule.sh           |    6 ++++++
 t/t7400-submodule-basic.sh |    2 +-
 2 files changed, 7 insertions(+), 1 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index b40f876..ea2aa3b 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -96,6 +96,12 @@ module_clone()
  test -e "$path" &&
  die "A file already exist at path '$path'"
 
+ # The user may have added url.*.insteadOf rules intending to rewrite
+ # submodule URLs. We must explicitly do this rewrite before calling
+ # 'git clone', since 'git clone' will not consult the super-repo
+ # config and thus never see any url.*.insteadOf rules placed therein.
+ url=$(git config --rewrite-url "$url")
+
  git-clone -n "$url" "$path" ||
  die "Clone of '$url' into submodule path '$path' failed"
 }
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index bafc46c..9c56de2 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -209,7 +209,7 @@ test_expect_success 'update --init' '
 
 '
 
-test_expect_failure 'update --init with url.*.insteadOf' '
+test_expect_success 'update --init with url.*.insteadOf' '
 
  rm -rf init &&
  git config -f .gitmodules submodule.example.url "http://example.com/init2" &&
--
1.6.0.rc1.34.g0fe8c

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

[PATCH 1/5 v2] Add testcase for 'git submodule' with url.*.insteadOf set in the super-repo

Johan Herland
In reply to this post by Johan Herland
Currently, setting url.*.insteadOf in the super-repo in order to rewrite
submodule URLs, don't work. When cloning/fetching the submodule, the
super-repo config is never consulted, and thus the url.*.insteadOf rule
is never seen. This adds a testcase that confirms the current behaviour.

Signed-off-by: Johan Herland <[hidden email]>
---
 t/t7400-submodule-basic.sh |   11 +++++++++++
 1 files changed, 11 insertions(+), 0 deletions(-)


The previous patch was whitespace-damaged. Sorry. Trying again.


...Johan


diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index cbc0c34..bafc46c 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -209,4 +209,15 @@ test_expect_success 'update --init' '
 
 '
 
+test_expect_failure 'update --init with url.*.insteadOf' '
+
+ rm -rf init &&
+ git config -f .gitmodules submodule.example.url "http://example.com/init2" &&
+ git config --remove-section submodule.example
+ git config "url.$(pwd)/.insteadOf" "http://example.com/" &&
+ git submodule update --init init &&
+ test -d init/.git
+
+'
+
 test_done
--
1.6.0.rc1.34.g0fe8c


--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 0/5] Fix 'url.*.insteadOf' for submodule URLs

Johannes Schindelin
In reply to this post by Johan Herland
Hi,

On Mon, 4 Aug 2008, Johan Herland wrote:

> As suggested in a thread some time ago, one could redefine the URL used
> to fetch submodules by adding a 'url.*.insteadOf' rule prior to the
> first invocation of 'git submodule update'.

If I suggested it, but forgot the "--global" flag to "git config", I
apologize.

Ciao,
Dscho

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 0/5] Fix 'url.*.insteadOf' for submodule URLs

Johan Herland
On Monday 04 August 2008, Johannes Schindelin wrote:
> On Mon, 4 Aug 2008, Johan Herland wrote:
> > As suggested in a thread some time ago, one could redefine the URL used
> > to fetch submodules by adding a 'url.*.insteadOf' rule prior to the
> > first invocation of 'git submodule update'.
>
> If I suggested it, but forgot the "--global" flag to "git config", I
> apologize.

Does this mean that you don't agree with the rationale for this patch? I.e.
that submodule URLs should not be rewritten according to the rules in the
super-repo (but instead require such rules to be set in the user's global
config)?

There are (at least) two reasons for why I think this should work without
having to use '--global':

1. Consistency: Other git commands in the supermodule does _not_ require the
URL rewriting rule to reside in the global config. Why should 'git
submodule' be different.

2. I believe there are valid use cases for adding URL rewriting rules to the
repo config instead of the global config. You may want to check out Fred's
version of project X (including submodules), without making your other
clones of project X start cloning/fetching from Fred.


Puzzled,
...Johan


--
Johan Herland, <[hidden email]>
www.herland.net
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 0/5] Fix 'url.*.insteadOf' for submodule URLs

Junio C Hamano
In reply to this post by Johannes Schindelin
Johannes Schindelin <[hidden email]> writes:

> On Mon, 4 Aug 2008, Johan Herland wrote:
>
>> As suggested in a thread some time ago, one could redefine the URL used
>> to fetch submodules by adding a 'url.*.insteadOf' rule prior to the
>> first invocation of 'git submodule update'.
>
> If I suggested it, but forgot the "--global" flag to "git config", I
> apologize.

That's a nice way to say that the patchset is unnecessary ;-).
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 0/5] Fix 'url.*.insteadOf' for submodule URLs

Junio C Hamano
In reply to this post by Johan Herland
Johan Herland <[hidden email]> writes:

> 1. Consistency: Other git commands in the supermodule does _not_ require the
> URL rewriting rule to reside in the global config. Why should 'git
> submodule' be different.

When it comes to "submodules", I do not think such consistency argument
makes much sense.  "git submodule" command crosses module boundary, normal
commands don't.  They are naturally different and they should be.

Your kind of consistency means breaking the separation between module
boundary, doesn't it?

Having said that...

> 2. I believe there are valid use cases for adding URL rewriting rules to the
> repo config instead of the global config. You may want to check out Fred's
> version of project X (including submodules), without making your other
> clones of project X start cloning/fetching from Fred.

I think you are referring to the example given in an earlier thread to
peek what your neighbor did between you two, without affecting other
people.

Personally I think it is partly showing the shortcoming of the current
"git submodule" that minimally supports the workflow to follow what the
canonical repository does, and partly showing that it is an abuse of that
interface to rewrite config file to temporarily switch to peek somewhere
else in such a workflow.

Let's step back and think what we would do if there is no submodule
involved.  That is, you usually follow origin, but you temporarily want to
peek at what Fred did.  How would you do this?

        $ git fetch $fred $branch_fred_wants_you_to_review
        $ git checkout FETCH_HEAD ;# this detaches HEAD.

And you take a look around.  Perhaps you like the change and decide to
merge that to your branch.  Perhaps you create your own branch on top of
that state, build a few fix-up commits, and give the result back to Fred.

Shouldn't peeking what Fred did in the whole submodule hierarchy be
essentially the same thing?  That is,

        $ git submodule for-each-submodule sh -c '
                git fetch "$fred/$1" $branch_fred_wants_you_to_review &&
                git checkout FETCH_HEAD
        ' -

where "for-each-submodule" would iterate over the submodules in the
current superproject that you are interested in (that is, you actually
have corresponding repositories there), and runs any given command with
the path to the submodule in that directory.

Hmm?

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 0/5] Fix 'url.*.insteadOf' for submodule URLs

Johan Herland
On Monday 04 August 2008, Junio C Hamano wrote:

> Johan Herland <[hidden email]> writes:
> > 1. Consistency: Other git commands in the supermodule does _not_
> > require the URL rewriting rule to reside in the global config. Why
> > should 'git submodule' be different.
>
> When it comes to "submodules", I do not think such consistency argument
> makes much sense.  "git submodule" command crosses module boundary,
> normal commands don't.  They are naturally different and they should be.
>
> Your kind of consistency means breaking the separation between module
> boundary, doesn't it?

To a certain degree, yes, but (1) only for this specific config rule
(url.*.insteadOf), and (2) only at the specific point in time when the
submodule is cloned. This means that after the submodule is cloned, it is
indeed not affected by the super-repo's config. E.g. you can change the
origin URL of the submodule back to the original (un-rewritten) URL in the
submodule's config, and the super-repo will not try to rewrite it at again.

I believe that the module boundary should not be crossed in general. But
this patch only "crosses" that boundary before it really exists (i.e.
before the submodule has been cloned into the super-repo). The event of
introducing a submodule into a super-repo, is AFAICS the _only_ event where
I would consider to cross the module boundary (and then only for a good
reason).

> Having said that...
>
> > 2. I believe there are valid use cases for adding URL rewriting rules
> > to the repo config instead of the global config. You may want to check
> > out Fred's version of project X (including submodules), without making
> > your other clones of project X start cloning/fetching from Fred.
>
> I think you are referring to the example given in an earlier thread to
> peek what your neighbor did between you two, without affecting other
> people.

Correct.

> Personally I think it is partly showing the shortcoming of the current
> "git submodule" that minimally supports the workflow to follow what the
> canonical repository does, and partly showing that it is an abuse of that
> interface to rewrite config file to temporarily switch to peek somewhere
> else in such a workflow.

Indeed. However, the development of submodules usability seems so slow that
even though this use case is an ugly workaround, it's one I thought we'd
have to live with for some time... [1]

> Let's step back and think what we would do if there is no submodule
> involved.  That is, you usually follow origin, but you temporarily want
> to peek at what Fred did.  How would you do this?
>
> $ git fetch $fred $branch_fred_wants_you_to_review
> $ git checkout FETCH_HEAD ;# this detaches HEAD.
>
> And you take a look around.  Perhaps you like the change and decide to
> merge that to your branch.  Perhaps you create your own branch on top of
> that state, build a few fix-up commits, and give the result back to Fred.
>
> Shouldn't peeking what Fred did in the whole submodule hierarchy be
> essentially the same thing?  That is,
>
> $ git submodule for-each-submodule sh -c '
> git fetch "$fred/$1" $branch_fred_wants_you_to_review &&
> git checkout FETCH_HEAD
> ' -
>
> where "for-each-submodule" would iterate over the submodules in the
> current superproject that you are interested in (that is, you actually
> have corresponding repositories there), and runs any given command with
> the path to the submodule in that directory.
>
> Hmm?

Yes, having such functionality in 'git submodule' would be wonderful.
However, implementations of such functionality have been slow in coming,
and apparently hard to implement without being workflow-agnostic (if I
remember correctly).

In light of improvements to 'git submodule', feel free to disregard my patch
series (although I still find 'git config --rewrite-url' useful for
dry-testing my 'url.*.insteadOf' rules...).


Thanks for providing constructive feedback.


Have fun!

...Johan


[1]: Maybe 'git submodule' would improve more quickly if we ate our own
dogfood, i.e. if we included submodules (e.g. gitk and git-gui) in git.git?

--
Johan Herland, <[hidden email]>
www.herland.net
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html