[PATCH 1/6] Make test-lib.sh emit valid TAP format

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

[PATCH 1/6] Make test-lib.sh emit valid TAP format

Ævar Arnfjörð Bjarmason
TAP, the the Test Anything Protocol, is a simple text-based interface
between testing modules in a test harness. test-lib.sh's output was
already very close to being valid TAP. This change brings it all the
way there.

The advantage of using TAP is that any program that reads the format
(a "test harness") can run the tests. The most popular of these is the
prove(1) utility that comes with Perl. It can run tests in parallel,
display colored output, format the output to console, file, HTML etc.,
and much more.

On my quad Xeon server running the test suite with `make test` takes
21 minutes. Running it with `prove -j 15 ./t[0-9]*.sh` takes just over
5 minutes.

With parallel tests the whole test suite doesn't have to stall because
tests like t3404-rebase-interactive.sh take a long time.

Signed-off-by: Ævar Arnfjörð Bjarmason <[hidden email]>
---
 t/README      |   49 +++++++++++++++++++++++++++++++++----------------
 t/test-lib.sh |   30 ++++++++++++++++++------------
 2 files changed, 51 insertions(+), 28 deletions(-)

diff --git a/t/README b/t/README
index dcd3ebb..fc4bb04 100644
--- a/t/README
+++ b/t/README
@@ -18,25 +18,42 @@ The easiest way to run tests is to say "make".  This runs all
 the tests.
 
     *** t0000-basic.sh ***
-    *   ok 1: .git/objects should be empty after git-init in an empty repo.
-    *   ok 2: .git/objects should have 256 subdirectories.
-    *   ok 3: git-update-index without --add should fail adding.
+    ok 1 - .git/objects should be empty after git init in an empty repo.
+    ok 2 - .git/objects should have 3 subdirectories.
+    ok 3 - success is reported like this
     ...
-    *   ok 23: no diff after checkout and git-update-index --refresh.
-    * passed all 23 test(s)
-    *** t0100-environment-names.sh ***
-    *   ok 1: using old names should issue warnings.
-    *   ok 2: using old names but having new names should not issue warnings.
-    ...
-
-Or you can run each test individually from command line, like
-this:
-
-    $ sh ./t3001-ls-files-killed.sh
-    *   ok 1: git-update-index --add to add various paths.
-    *   ok 2: git-ls-files -k to show killed files.
-    *   ok 3: validate git-ls-files -k output.
-    * passed all 3 test(s)
+    ok 43 - very long name in the index handled sanely
+    # fixed 1 known breakage(s)
+    # still have 1 known breakage(s)
+    # passed all remaining 42 test(s)
+    1..43
+    *** t0001-init.sh ***
+    ok 1 - plain
+    ok 2 - plain with GIT_WORK_TREE
+    ok 3 - plain bare
+
+Since the tests all output TAP (see http://testanything.org) they can
+be run with any TAP harness. Here's an example of paralell testing
+powered by a recent version of prove(1):
+        
+    $ prove --timer --jobs 15 ./t[0-9]*.sh
+    [19:17:33] ./t0005-signals.sh ................................... ok       36 ms
+    [19:17:33] ./t0022-crlf-rename.sh ............................... ok       69 ms
+    [19:17:33] ./t0024-crlf-archive.sh .............................. ok      154 ms
+    [19:17:33] ./t0004-unwritable.sh ................................ ok      289 ms
+    [19:17:33] ./t0002-gitfile.sh ................................... ok      480 ms
+    ===(     102;0  25/?  6/?  5/?  16/?  1/?  4/?  2/?  1/?  3/?  1... )===
+    
+You can also run each test individually from command line, like this:
+
+    $ sh ./t3010-ls-files-killed-modified.sh
+    ok 1 - git update-index --add to add various paths.
+    ok 2 - git ls-files -k to show killed files.
+    ok 3 - validate git ls-files -k output.
+    ok 4 - git ls-files -m to show modified files.
+    ok 5 - validate git ls-files -m output.
+    # passed all 5 test(s)
+    1..5
 
 You can pass --verbose (or -v), --debug (or -d), and --immediate
 (or -i) command line argument to the test, or by setting GIT_TEST_OPTS
diff --git a/t/test-lib.sh b/t/test-lib.sh
index acce3d0..282314c 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -161,7 +161,7 @@ if test -n "$color"; then
  *) test -n "$quiet" && return;;
  esac
  shift
- printf "* %s" "$*"
+ printf "%s" "$*"
  tput sgr0
  echo
  )
@@ -170,7 +170,7 @@ else
  say_color() {
  test -z "$1" && test -n "$quiet" && return
  shift
- echo "* $*"
+ echo "$*"
  }
 fi
 
@@ -340,25 +340,25 @@ test_have_prereq () {
 
 test_ok_ () {
  test_success=$(($test_success + 1))
- say_color "" "  ok $test_count: $@"
+ say_color "" "ok $test_count - $@"
 }
 
 test_failure_ () {
  test_failure=$(($test_failure + 1))
- say_color error "FAIL $test_count: $1"
+ say_color error "not ok - $test_count $1"
  shift
- echo "$@" | sed -e 's/^/ /'
+ echo "$@" | sed -e 's/^/# /'
  test "$immediate" = "" || { GIT_EXIT_OK=t; exit 1; }
 }
 
 test_known_broken_ok_ () {
  test_fixed=$(($test_fixed+1))
- say_color "" "  FIXED $test_count: $@"
+ say_color "" "ok $test_count - $@ # TODO known breakage"
 }
 
 test_known_broken_failure_ () {
  test_broken=$(($test_broken+1))
- say_color skip "  still broken $test_count: $@"
+ say_color skip "not ok $test_count - $@ # TODO known breakage"
 }
 
 test_debug () {
@@ -390,7 +390,7 @@ test_skip () {
  case "$to_skip" in
  t)
  say_color skip >&3 "skipping test: $@"
- say_color skip "skip $test_count: $1"
+ say_color skip "ok $test_count: # skip $1"
  : true
  ;;
  *)
@@ -601,18 +601,22 @@ test_done () {
 
  if test "$test_fixed" != 0
  then
- say_color pass "fixed $test_fixed known breakage(s)"
+ say_color pass "# fixed $test_fixed known breakage(s)"
  fi
  if test "$test_broken" != 0
  then
- say_color error "still have $test_broken known breakage(s)"
+ say_color error "# still have $test_broken known breakage(s)"
  msg="remaining $(($test_count-$test_broken)) test(s)"
  else
  msg="$test_count test(s)"
  fi
  case "$test_failure" in
  0)
- say_color pass "passed all $msg"
+ # Maybe print SKIP message
+ [ -z "$skip_all" ] || skip_all=" # SKIP $skip_all"
+
+ say_color pass "# passed all $msg"
+ say "1..$test_count$skip_all"
 
  test -d "$remove_trash" &&
  cd "$(dirname "$remove_trash")" &&
@@ -621,7 +625,9 @@ test_done () {
  exit 0 ;;
 
  *)
- say_color error "failed $test_failure among $msg"
+ say_color error "# failed $test_failure among $msg"
+ say "1..$test_count"
+
  exit 1 ;;
 
  esac
--
1.7.1.dirty

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

[PATCH 2/6] Move t6000lib.sh to lib-*

Ævar Arnfjörð Bjarmason
The naming of this test library conflicted with the recommendation in
t/README's "Naming Tests" section. Now it's possible to naïvely test
t[0-9]*.sh with prove(1) without running into errors.

Signed-off-by: Ævar Arnfjörð Bjarmason <[hidden email]>
---
 t/{t6000lib.sh => lib-t6000.sh} |    0
 t/t6002-rev-list-bisect.sh      |    2 +-
 t/t6003-rev-list-topo-order.sh  |    2 +-
 t/t6101-rev-parse-parents.sh    |    2 +-
 4 files changed, 3 insertions(+), 3 deletions(-)
 rename t/{t6000lib.sh => lib-t6000.sh} (100%)

diff --git a/t/t6000lib.sh b/t/lib-t6000.sh
similarity index 100%
rename from t/t6000lib.sh
rename to t/lib-t6000.sh
diff --git a/t/t6002-rev-list-bisect.sh b/t/t6002-rev-list-bisect.sh
index b4e8fba..fb07536 100755
--- a/t/t6002-rev-list-bisect.sh
+++ b/t/t6002-rev-list-bisect.sh
@@ -5,7 +5,7 @@
 test_description='Tests git rev-list --bisect functionality'
 
 . ./test-lib.sh
-. "$TEST_DIRECTORY"/t6000lib.sh # t6xxx specific functions
+. "$TEST_DIRECTORY"/lib-t6000.sh # t6xxx specific functions
 
 # usage: test_bisection max-diff bisect-option head ^prune...
 #
diff --git a/t/t6003-rev-list-topo-order.sh b/t/t6003-rev-list-topo-order.sh
index 2c73f2d..e4c52b0 100755
--- a/t/t6003-rev-list-topo-order.sh
+++ b/t/t6003-rev-list-topo-order.sh
@@ -6,7 +6,7 @@
 test_description='Tests git rev-list --topo-order functionality'
 
 . ./test-lib.sh
-. "$TEST_DIRECTORY"/t6000lib.sh # t6xxx specific functions
+. "$TEST_DIRECTORY"/lib-t6000.sh # t6xxx specific functions
 
 list_duplicates()
 {
diff --git a/t/t6101-rev-parse-parents.sh b/t/t6101-rev-parse-parents.sh
index f105fab..e673c25 100755
--- a/t/t6101-rev-parse-parents.sh
+++ b/t/t6101-rev-parse-parents.sh
@@ -6,7 +6,7 @@
 test_description='Test git rev-parse with different parent options'
 
 . ./test-lib.sh
-. "$TEST_DIRECTORY"/t6000lib.sh # t6xxx specific functions
+. "$TEST_DIRECTORY"/lib-t6000.sh # t6xxx specific functions
 
 date >path0
 git update-index --add path0
--
1.7.1.dirty

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

[PATCH 3/6] Skip tests in a way that makes sense under TAP

Ævar Arnfjörð Bjarmason
In reply to this post by Ævar Arnfjörð Bjarmason
SKIP messages are now part of the TAP plan. A TAP harness now knows
why a particular test was skipped and can report that information.

Signed-off-by: Ævar Arnfjörð Bjarmason <[hidden email]>
---
 t/lib-git-svn.sh                       |    4 ++--
 t/lib-httpd.sh                         |    3 +--
 t/t1304-default-acl.sh                 |    9 ++++++---
 t/t1509-root-worktree.sh               |    6 +++---
 t/t2007-checkout-symlink.sh            |    2 +-
 t/t3300-funny-names.sh                 |    2 +-
 t/t3302-notes-index-expensive.sh       |    2 +-
 t/t3600-rm.sh                          |    2 +-
 t/t3701-add-interactive.sh             |    4 ++--
 t/t3902-quoted.sh                      |    2 +-
 t/t4004-diff-rename-symlink.sh         |    2 +-
 t/t4011-diff-symlink.sh                |    2 +-
 t/t4016-diff-quote.sh                  |    2 +-
 t/t4023-diff-rename-typechange.sh      |    2 +-
 t/t4114-apply-typechange.sh            |    2 +-
 t/t4115-apply-symlink.sh               |    2 +-
 t/t4122-apply-symlink-inside.sh        |    2 +-
 t/t5302-pack-index.sh                  |    2 +-
 t/t5503-tagfollow.sh                   |    2 +-
 t/t5522-pull-symlink.sh                |    2 +-
 t/t5540-http-push.sh                   |    2 +-
 t/t5541-http-push.sh                   |    2 +-
 t/t5550-http-fetch.sh                  |    2 +-
 t/t5551-http-fetch.sh                  |    2 +-
 t/t5561-http-backend.sh                |    2 +-
 t/t5705-clone-2gb.sh                   |    2 +-
 t/t6035-merge-dir-to-symlink.sh        |    2 +-
 t/t7004-tag.sh                         |    2 +-
 t/t7006-pager.sh                       |    2 +-
 t/t7800-difftool.sh                    |    2 +-
 t/t9001-send-email.sh                  |    4 ++--
 t/t9100-git-svn-basic.sh               |    2 +-
 t/t9119-git-svn-info.sh                |    2 +-
 t/t9129-git-svn-i18n-commitencoding.sh |    2 +-
 t/t9143-git-svn-gc.sh                  |    2 +-
 t/t9200-git-cvsexportcommit.sh         |    4 ++--
 t/t9400-git-cvsserver-server.sh        |    6 +++---
 t/t9401-git-cvsserver-crlf.sh          |    6 +++---
 t/t9600-cvsimport.sh                   |    2 +-
 t/t9700-perl-git.sh                    |    4 ++--
 40 files changed, 56 insertions(+), 54 deletions(-)

diff --git a/t/lib-git-svn.sh b/t/lib-git-svn.sh
index 0f7f35c..344785d 100644
--- a/t/lib-git-svn.sh
+++ b/t/lib-git-svn.sh
@@ -5,11 +5,11 @@ git_svn_id=git""-svn-id
 
 if test -n "$NO_SVN_TESTS"
 then
- say 'skipping git svn tests, NO_SVN_TESTS defined'
+ skip_all='skipping git svn tests, NO_SVN_TESTS defined'
  test_done
 fi
 if ! test_have_prereq PERL; then
- say 'skipping git svn tests, perl not available'
+ skip_all='skipping git svn tests, perl not available'
  test_done
 fi
 
diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh
index da4b8d5..a0944d6 100644
--- a/t/lib-httpd.sh
+++ b/t/lib-httpd.sh
@@ -5,8 +5,7 @@
 
 if test -z "$GIT_TEST_HTTPD"
 then
- say "skipping test, network testing disabled by default"
- say "(define GIT_TEST_HTTPD to enable)"
+ skip_all="Network testing disabled (define GIT_TEST_HTTPD to enable)"
  test_done
 fi
 
diff --git a/t/t1304-default-acl.sh b/t/t1304-default-acl.sh
index 055ad00..97ab02a 100755
--- a/t/t1304-default-acl.sh
+++ b/t/t1304-default-acl.sh
@@ -15,9 +15,12 @@ umask 077
 # is a good candidate: exists on all unices, and it has permission
 # anyway, so we don't create a security hole running the testsuite.
 
-if ! setfacl -m u:root:rwx .; then
-    say "Skipping ACL tests: unable to use setfacl"
-    test_done
+setfacl_out="$(setfacl -m u:root:rwx . 2>&1)"
+setfacl_ret=$?
+
+if [ $setfacl_ret != 0 ]; then
+ skip_all="Skipping ACL tests: unable to use setfacl (output: '$setfacl_out'; return code: '$setfacl_ret')"
+ test_done
 fi
 
 check_perms_and_acl () {
diff --git a/t/t1509-root-worktree.sh b/t/t1509-root-worktree.sh
index 5322a3b..7f60fd0 100755
--- a/t/t1509-root-worktree.sh
+++ b/t/t1509-root-worktree.sh
@@ -99,17 +99,17 @@ test_foobar_foobar() {
 }
 
 if ! test_have_prereq POSIXPERM || ! [ -w / ]; then
- say "Dangerous test skipped. Read this test if you want to execute it"
+ skip_all="Dangerous test skipped. Read this test if you want to execute it"
  test_done
 fi
 
 if [ "$IKNOWWHATIAMDOING" != "YES" ]; then
- say "You must set env var IKNOWWHATIAMDOING=YES in order to run this test"
+ skip_all="You must set env var IKNOWWHATIAMDOING=YES in order to run this test"
  test_done
 fi
 
 if [ "$UID" = 0 ]; then
- say "No you can't run this with root"
+ skip_all="No you can't run this with root"
  test_done
 fi
 
diff --git a/t/t2007-checkout-symlink.sh b/t/t2007-checkout-symlink.sh
index 20f3343..616ca2a 100755
--- a/t/t2007-checkout-symlink.sh
+++ b/t/t2007-checkout-symlink.sh
@@ -8,7 +8,7 @@ test_description='git checkout to switch between branches with symlink<->dir'
 
 if ! test_have_prereq SYMLINKS
 then
- say "symbolic links not supported - skipping tests"
+ skip_all="symbolic links not supported - skipping tests"
  test_done
 fi
 
diff --git a/t/t3300-funny-names.sh b/t/t3300-funny-names.sh
index db46d53..a99e4d8 100755
--- a/t/t3300-funny-names.sh
+++ b/t/t3300-funny-names.sh
@@ -26,7 +26,7 @@ echo 'Foo Bar Baz' >"$p2"
 
 test -f "$p1" && cmp "$p0" "$p1" || {
  # since FAT/NTFS does not allow tabs in filenames, skip this test
- say 'Your filesystem does not allow tabs in filenames, test skipped.'
+ skip_all='Your filesystem does not allow tabs in filenames, test skipped.'
  test_done
 }
 
diff --git a/t/t3302-notes-index-expensive.sh b/t/t3302-notes-index-expensive.sh
index ee84fc4..361a10a 100755
--- a/t/t3302-notes-index-expensive.sh
+++ b/t/t3302-notes-index-expensive.sh
@@ -8,7 +8,7 @@ test_description='Test commit notes index (expensive!)'
 . ./test-lib.sh
 
 test -z "$GIT_NOTES_TIMING_TESTS" && {
- say Skipping timing tests
+ skip_all="Skipping timing tests"
  test_done
  exit
 }
diff --git a/t/t3600-rm.sh b/t/t3600-rm.sh
index 0aaf0ad..b514cbb 100755
--- a/t/t3600-rm.sh
+++ b/t/t3600-rm.sh
@@ -39,7 +39,7 @@ if test -f test-file
 then
  test_set_prereq RO_DIR
 else
- say 'skipping removal failure test (perhaps running as root?)'
+ skip_all='skipping removal failure test (perhaps running as root?)'
 fi
 chmod 775 .
 rm -f test-file
diff --git a/t/t3701-add-interactive.sh b/t/t3701-add-interactive.sh
index b6eba6a..7ad8465 100755
--- a/t/t3701-add-interactive.sh
+++ b/t/t3701-add-interactive.sh
@@ -4,7 +4,7 @@ test_description='add -i basic tests'
 . ./test-lib.sh
 
 if ! test_have_prereq PERL; then
- say 'skipping git add -i tests, perl not available'
+ skip_all='skipping git add -i tests, perl not available'
  test_done
 fi
 
@@ -154,7 +154,7 @@ rm -f .gitignore
 
 if test "$(git config --bool core.filemode)" = false
 then
- say 'skipping filemode tests (filesystem does not properly support modes)'
+ say '# skipping filemode tests (filesystem does not properly support modes)'
 else
  test_set_prereq FILEMODE
 fi
diff --git a/t/t3902-quoted.sh b/t/t3902-quoted.sh
index 29103f6..147e634 100755
--- a/t/t3902-quoted.sh
+++ b/t/t3902-quoted.sh
@@ -17,7 +17,7 @@ DQ='"'
 echo foo 2>/dev/null > "Name and an${HT}HT"
 test -f "Name and an${HT}HT" || {
  # since FAT/NTFS does not allow tabs in filenames, skip this test
- say 'Your filesystem does not allow tabs in filenames, test skipped.'
+ skip_all='Your filesystem does not allow tabs in filenames, test skipped.'
  test_done
 }
 
diff --git a/t/t4004-diff-rename-symlink.sh b/t/t4004-diff-rename-symlink.sh
index a4da119..1a09e8d 100755
--- a/t/t4004-diff-rename-symlink.sh
+++ b/t/t4004-diff-rename-symlink.sh
@@ -14,7 +14,7 @@ by an edit for them.
 
 if ! test_have_prereq SYMLINKS
 then
- say 'Symbolic links not supported, skipping tests.'
+ skip_all='Symbolic links not supported, skipping tests.'
  test_done
 fi
 
diff --git a/t/t4011-diff-symlink.sh b/t/t4011-diff-symlink.sh
index d7e327c..2b9a22d 100755
--- a/t/t4011-diff-symlink.sh
+++ b/t/t4011-diff-symlink.sh
@@ -11,7 +11,7 @@ test_description='Test diff of symlinks.
 
 if ! test_have_prereq SYMLINKS
 then
- say 'Symbolic links not supported, skipping tests.'
+ skip_all='Symbolic links not supported, skipping tests.'
  test_done
 fi
 
diff --git a/t/t4016-diff-quote.sh b/t/t4016-diff-quote.sh
index 55eb5f8..34e5144 100755
--- a/t/t4016-diff-quote.sh
+++ b/t/t4016-diff-quote.sh
@@ -14,7 +14,7 @@ P2='pathname with SP'
 P3='pathname
 with LF'
 : 2>/dev/null >"$P1" && test -f "$P1" && rm -f "$P1" || {
- say 'Your filesystem does not allow tabs in filenames, test skipped.'
+ skip_all='Your filesystem does not allow tabs in filenames, test skipped.'
  test_done
 }
 
diff --git a/t/t4023-diff-rename-typechange.sh b/t/t4023-diff-rename-typechange.sh
index 9bdf659..40a95a1 100755
--- a/t/t4023-diff-rename-typechange.sh
+++ b/t/t4023-diff-rename-typechange.sh
@@ -6,7 +6,7 @@ test_description='typechange rename detection'
 
 if ! test_have_prereq SYMLINKS
 then
- say 'Symbolic links not supported, skipping tests.'
+ skip_all='Symbolic links not supported, skipping tests.'
  test_done
 fi
 
diff --git a/t/t4114-apply-typechange.sh b/t/t4114-apply-typechange.sh
index 99ec13d..164d58c 100755
--- a/t/t4114-apply-typechange.sh
+++ b/t/t4114-apply-typechange.sh
@@ -11,7 +11,7 @@ test_description='git apply should not get confused with type changes.
 
 if ! test_have_prereq SYMLINKS
 then
- say 'Symbolic links not supported, skipping tests.'
+ skip_all='Symbolic links not supported, skipping tests.'
  test_done
 fi
 
diff --git a/t/t4115-apply-symlink.sh b/t/t4115-apply-symlink.sh
index b852e58..aff4348 100755
--- a/t/t4115-apply-symlink.sh
+++ b/t/t4115-apply-symlink.sh
@@ -11,7 +11,7 @@ test_description='git apply symlinks and partial files
 
 if ! test_have_prereq SYMLINKS
 then
- say 'Symbolic links not supported, skipping tests.'
+ skip_all='Symbolic links not supported, skipping tests.'
  test_done
 fi
 
diff --git a/t/t4122-apply-symlink-inside.sh b/t/t4122-apply-symlink-inside.sh
index 0d3c1d5..923fcab 100755
--- a/t/t4122-apply-symlink-inside.sh
+++ b/t/t4122-apply-symlink-inside.sh
@@ -5,7 +5,7 @@ test_description='apply to deeper directory without getting fooled with symlink'
 
 if ! test_have_prereq SYMLINKS
 then
- say 'Symbolic links not supported, skipping tests.'
+ skip_all='Symbolic links not supported, skipping tests.'
  test_done
 fi
 
diff --git a/t/t5302-pack-index.sh b/t/t5302-pack-index.sh
index 4360e77..fb3a270 100755
--- a/t/t5302-pack-index.sh
+++ b/t/t5302-pack-index.sh
@@ -74,7 +74,7 @@ if msg=$(git verify-pack -v "test-3-${pack3}.pack" 2>&1) ||
 then
  test_set_prereq OFF64_T
 else
- say "skipping tests concerning 64-bit offsets"
+ say "# skipping tests concerning 64-bit offsets"
 fi
 
 test_expect_success OFF64_T \
diff --git a/t/t5503-tagfollow.sh b/t/t5503-tagfollow.sh
index d5db75d..bab1a53 100755
--- a/t/t5503-tagfollow.sh
+++ b/t/t5503-tagfollow.sh
@@ -6,7 +6,7 @@ test_description='test automatic tag following'
 
 case $(uname -s) in
 *MINGW*)
- say "GIT_DEBUG_SEND_PACK not supported - skipping tests"
+ skip_all="GIT_DEBUG_SEND_PACK not supported - skipping tests"
  test_done
 esac
 
diff --git a/t/t5522-pull-symlink.sh b/t/t5522-pull-symlink.sh
index 7206817..298200f 100755
--- a/t/t5522-pull-symlink.sh
+++ b/t/t5522-pull-symlink.sh
@@ -6,7 +6,7 @@ test_description='pulling from symlinked subdir'
 
 if ! test_have_prereq SYMLINKS
 then
- say 'Symbolic links not supported, skipping tests.'
+ skip_all='Symbolic links not supported, skipping tests.'
  test_done
 fi
 
diff --git a/t/t5540-http-push.sh b/t/t5540-http-push.sh
index 37fe875..a266ca5 100755
--- a/t/t5540-http-push.sh
+++ b/t/t5540-http-push.sh
@@ -11,7 +11,7 @@ This test runs various sanity checks on http-push.'
 
 if git http-push > /dev/null 2>&1 || [ $? -eq 128 ]
 then
- say "skipping test, USE_CURL_MULTI is not defined"
+ skip_all="skipping test, USE_CURL_MULTI is not defined"
  test_done
 fi
 
diff --git a/t/t5541-http-push.sh b/t/t5541-http-push.sh
index 795dc2b..a5e9be6 100755
--- a/t/t5541-http-push.sh
+++ b/t/t5541-http-push.sh
@@ -7,7 +7,7 @@ test_description='test smart pushing over http via http-backend'
 . ./test-lib.sh
 
 if test -n "$NO_CURL"; then
- say 'skipping test, git built without http support'
+ skip_all='skipping test, git built without http support'
  test_done
 fi
 
diff --git a/t/t5550-http-fetch.sh b/t/t5550-http-fetch.sh
index 8cfce96..10681fb 100755
--- a/t/t5550-http-fetch.sh
+++ b/t/t5550-http-fetch.sh
@@ -4,7 +4,7 @@ test_description='test dumb fetching over http via static file'
 . ./test-lib.sh
 
 if test -n "$NO_CURL"; then
- say 'skipping test, git built without http support'
+ skip_all='skipping test, git built without http support'
  test_done
 fi
 
diff --git a/t/t5551-http-fetch.sh b/t/t5551-http-fetch.sh
index 7faa31a..fd19121 100755
--- a/t/t5551-http-fetch.sh
+++ b/t/t5551-http-fetch.sh
@@ -4,7 +4,7 @@ test_description='test smart fetching over http via http-backend'
 . ./test-lib.sh
 
 if test -n "$NO_CURL"; then
- say 'skipping test, git built without http support'
+ skip_all='skipping test, git built without http support'
  test_done
 fi
 
diff --git a/t/t5561-http-backend.sh b/t/t5561-http-backend.sh
index 8c6d0b2..b5d7fbc 100755
--- a/t/t5561-http-backend.sh
+++ b/t/t5561-http-backend.sh
@@ -4,7 +4,7 @@ test_description='test git-http-backend'
 . ./test-lib.sh
 
 if test -n "$NO_CURL"; then
- say 'skipping test, git built without http support'
+ skip_all='skipping test, git built without http support'
  test_done
 fi
 
diff --git a/t/t5705-clone-2gb.sh b/t/t5705-clone-2gb.sh
index adfaae8..23fe8fa 100755
--- a/t/t5705-clone-2gb.sh
+++ b/t/t5705-clone-2gb.sh
@@ -4,7 +4,7 @@ test_description='Test cloning a repository larger than 2 gigabyte'
 . ./test-lib.sh
 
 test -z "$GIT_TEST_CLONE_2GB" &&
-say "Skipping expensive 2GB clone test; enable it with GIT_TEST_CLONE_2GB=t" &&
+skip_all="Skipping expensive 2GB clone test; enable it with GIT_TEST_CLONE_2GB=t" &&
 test_done &&
 exit
 
diff --git a/t/t6035-merge-dir-to-symlink.sh b/t/t6035-merge-dir-to-symlink.sh
index 3202e1d..cd3190c 100755
--- a/t/t6035-merge-dir-to-symlink.sh
+++ b/t/t6035-merge-dir-to-symlink.sh
@@ -5,7 +5,7 @@ test_description='merging when a directory was replaced with a symlink'
 
 if ! test_have_prereq SYMLINKS
 then
- say 'Symbolic links not supported, skipping tests.'
+ skip_all='Symbolic links not supported, skipping tests.'
  test_done
 fi
 
diff --git a/t/t7004-tag.sh b/t/t7004-tag.sh
index 73dbc43..ac943f5 100755
--- a/t/t7004-tag.sh
+++ b/t/t7004-tag.sh
@@ -583,7 +583,7 @@ test_expect_success \
 # subsequent tests require gpg; check if it is available
 gpg --version >/dev/null 2>/dev/null
 if [ $? -eq 127 ]; then
- say "gpg not found - skipping tag signing and verification tests"
+ say "# gpg not found - skipping tag signing and verification tests"
 else
  # As said here: http://www.gnupg.org/documentation/faqs.html#q6.19
  # the gpg version 1.0.6 didn't parse trust packets correctly, so for
diff --git a/t/t7006-pager.sh b/t/t7006-pager.sh
index d9202d5..e2a44a5 100755
--- a/t/t7006-pager.sh
+++ b/t/t7006-pager.sh
@@ -29,7 +29,7 @@ then
  }
  test_set_prereq TTY
 else
- say no usable terminal, so skipping some tests
+ say "# no usable terminal, so skipping some tests"
 fi
 
 unset GIT_PAGER GIT_PAGER_IN_USE
diff --git a/t/t7800-difftool.sh b/t/t7800-difftool.sh
index 1de83ef..196827e 100755
--- a/t/t7800-difftool.sh
+++ b/t/t7800-difftool.sh
@@ -11,7 +11,7 @@ Testing basic diff tool invocation
 . ./test-lib.sh
 
 if ! test_have_prereq PERL; then
- say 'skipping difftool tests, perl not available'
+ skip_all='skipping difftool tests, perl not available'
  test_done
 fi
 
diff --git a/t/t9001-send-email.sh b/t/t9001-send-email.sh
index 640b3d2..ddc3d8d 100755
--- a/t/t9001-send-email.sh
+++ b/t/t9001-send-email.sh
@@ -4,7 +4,7 @@ test_description='git send-email'
 . ./test-lib.sh
 
 if ! test_have_prereq PERL; then
- say 'skipping git send-email tests, perl not available'
+ skip_all='skipping git send-email tests, perl not available'
  test_done
 fi
 
@@ -58,7 +58,7 @@ test_no_confirm () {
 # Exit immediately to prevent hang if a no-confirm test fails
 check_no_confirm () {
  test -f no_confirm_okay || {
- say 'No confirm test failed; skipping remaining tests to prevent hanging'
+ skip_all='confirm test failed; skipping remaining tests to prevent hanging'
  test_done
  }
 }
diff --git a/t/t9100-git-svn-basic.sh b/t/t9100-git-svn-basic.sh
index 570e035..13766ab 100755
--- a/t/t9100-git-svn-basic.sh
+++ b/t/t9100-git-svn-basic.sh
@@ -15,7 +15,7 @@ case "$GIT_SVN_LC_ALL" in
  test_set_prereq UTF8
  ;;
 *)
- say "UTF-8 locale not set, some tests skipped ($GIT_SVN_LC_ALL)"
+ say "# UTF-8 locale not set, some tests skipped ($GIT_SVN_LC_ALL)"
  ;;
 esac
 
diff --git a/t/t9119-git-svn-info.sh b/t/t9119-git-svn-info.sh
index a9a558d..5fb94fb 100755
--- a/t/t9119-git-svn-info.sh
+++ b/t/t9119-git-svn-info.sh
@@ -13,7 +13,7 @@ case $v in
 1.[456].*)
  ;;
 *)
- say "skipping svn-info test (SVN version: $v not supported)"
+ skip_all="skipping svn-info test (SVN version: $v not supported)"
  test_done
  ;;
 esac
diff --git a/t/t9129-git-svn-i18n-commitencoding.sh b/t/t9129-git-svn-i18n-commitencoding.sh
index b9224bd..6831fcc 100755
--- a/t/t9129-git-svn-i18n-commitencoding.sh
+++ b/t/t9129-git-svn-i18n-commitencoding.sh
@@ -72,7 +72,7 @@ done
 if locale -a |grep -q en_US.utf8; then
  test_set_prereq UTF8
 else
- say "UTF-8 locale not available, test skipped"
+ say "# UTF-8 locale not available, test skipped"
 fi
 
 test_expect_success UTF8 'ISO-8859-1 should match UTF-8 in svn' '
diff --git a/t/t9143-git-svn-gc.sh b/t/t9143-git-svn-gc.sh
index 99f69c6..337ea59 100755
--- a/t/t9143-git-svn-gc.sh
+++ b/t/t9143-git-svn-gc.sh
@@ -43,7 +43,7 @@ then
  gunzip .git/svn/refs/remotes/git-svn/unhandled.log.gz
  '
 else
- say "Perl Compress::Zlib unavailable, skipping gunzip test"
+ say "# Perl Compress::Zlib unavailable, skipping gunzip test"
 fi
 
 test_expect_success 'git svn gc does not change unhandled.log files' '
diff --git a/t/t9200-git-cvsexportcommit.sh b/t/t9200-git-cvsexportcommit.sh
index fc3795d..fd268b7 100755
--- a/t/t9200-git-cvsexportcommit.sh
+++ b/t/t9200-git-cvsexportcommit.sh
@@ -7,14 +7,14 @@ test_description='Test export of commits to CVS'
 . ./test-lib.sh
 
 if ! test_have_prereq PERL; then
- say 'skipping git cvsexportcommit tests, perl not available'
+ skip_all='skipping git cvsexportcommit tests, perl not available'
  test_done
 fi
 
 cvs >/dev/null 2>&1
 if test $? -ne 1
 then
-    say 'skipping git cvsexportcommit tests, cvs not found'
+    skip_all='skipping git cvsexportcommit tests, cvs not found'
     test_done
 fi
 
diff --git a/t/t9400-git-cvsserver-server.sh b/t/t9400-git-cvsserver-server.sh
index daef2d6..dda3e4c 100755
--- a/t/t9400-git-cvsserver-server.sh
+++ b/t/t9400-git-cvsserver-server.sh
@@ -11,17 +11,17 @@ cvs CLI client via git-cvsserver server'
 . ./test-lib.sh
 
 if ! test_have_prereq PERL; then
- say 'skipping git cvsserver tests, perl not available'
+ skip_all='skipping git cvsserver tests, perl not available'
  test_done
 fi
 cvs >/dev/null 2>&1
 if test $? -ne 1
 then
-    say 'skipping git-cvsserver tests, cvs not found'
+    skip_all='skipping git-cvsserver tests, cvs not found'
     test_done
 fi
 "$PERL_PATH" -e 'use DBI; use DBD::SQLite' >/dev/null 2>&1 || {
-    say 'skipping git-cvsserver tests, Perl SQLite interface unavailable'
+    skip_all='skipping git-cvsserver tests, Perl SQLite interface unavailable'
     test_done
 }
 
diff --git a/t/t9401-git-cvsserver-crlf.sh b/t/t9401-git-cvsserver-crlf.sh
index ed7b513..925bd0f 100755
--- a/t/t9401-git-cvsserver-crlf.sh
+++ b/t/t9401-git-cvsserver-crlf.sh
@@ -41,16 +41,16 @@ not_present() {
 cvs >/dev/null 2>&1
 if test $? -ne 1
 then
-    say 'skipping git-cvsserver tests, cvs not found'
+    skip_all='skipping git-cvsserver tests, cvs not found'
     test_done
 fi
 if ! test_have_prereq PERL
 then
-    say 'skipping git-cvsserver tests, perl not available'
+    skip_all='skipping git-cvsserver tests, perl not available'
     test_done
 fi
 "$PERL_PATH" -e 'use DBI; use DBD::SQLite' >/dev/null 2>&1 || {
-    say 'skipping git-cvsserver tests, Perl SQLite interface unavailable'
+    skip_all='skipping git-cvsserver tests, Perl SQLite interface unavailable'
     test_done
 }
 
diff --git a/t/t9600-cvsimport.sh b/t/t9600-cvsimport.sh
index b572ce3..2eff9cd 100755
--- a/t/t9600-cvsimport.sh
+++ b/t/t9600-cvsimport.sh
@@ -4,7 +4,7 @@ test_description='git cvsimport basic tests'
 . ./lib-cvs.sh
 
 if ! test_have_prereq PERL; then
- say 'skipping git cvsimport tests, perl not available'
+ skip_all='skipping git cvsimport tests, perl not available'
  test_done
 fi
 
diff --git a/t/t9700-perl-git.sh b/t/t9700-perl-git.sh
index 8686086..3999bf7 100755
--- a/t/t9700-perl-git.sh
+++ b/t/t9700-perl-git.sh
@@ -7,12 +7,12 @@ test_description='perl interface (Git.pm)'
 . ./test-lib.sh
 
 if ! test_have_prereq PERL; then
- say 'skipping perl interface tests, perl not available'
+ skip_all='skipping perl interface tests, perl not available'
  test_done
 fi
 
 "$PERL_PATH" -MTest::More -e 0 2>/dev/null || {
- say "Perl Test::More unavailable, skipping test"
+ skip_all="Perl Test::More unavailable, skipping test"
  test_done
 }
 
--
1.7.1.dirty

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

[PATCH 4/6] Turn setup code in t2007-checkout-symlink.sh into a test

Ævar Arnfjörð Bjarmason
In reply to this post by Ævar Arnfjörð Bjarmason
Previously the test would print to stdout which interfered with the
TAP output. Now this scaffolding code is just a normal test.

Signed-off-by: Ævar Arnfjörð Bjarmason <[hidden email]>
---
 t/t2007-checkout-symlink.sh |    6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/t/t2007-checkout-symlink.sh b/t/t2007-checkout-symlink.sh
index 616ca2a..05cc8fd 100755
--- a/t/t2007-checkout-symlink.sh
+++ b/t/t2007-checkout-symlink.sh
@@ -44,8 +44,10 @@ test_expect_success 'switch from symlink to dir' '
 
 '
 
-rm -fr frotz xyzzy nitfol &&
-git checkout -f master || exit
+test_expect_success 'Remove temporary directories & switch to master' '
+ rm -fr frotz xyzzy nitfol &&
+ git checkout -f master
+'
 
 test_expect_success 'switch from dir to symlink' '
 
--
1.7.1.dirty

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

[PATCH 5/6] We use TAP so the Perl test can run without scaffolding

Ævar Arnfjörð Bjarmason
In reply to this post by Ævar Arnfjörð Bjarmason
This removes the test_external and test_external_without_stderr
functions added by Lea Wiemann in
fb32c410087e68d650b31f68e66b3d9cbcce4a56. Nothing else used them, and
now that we're using TAP they shouldn't be necessary.

Signed-off-by: Ævar Arnfjörð Bjarmason <[hidden email]>
---
 t/t9700-perl-git.sh |    7 ++---
 t/t9700/test.pl     |    9 +++++++
 t/test-lib.sh       |   59 ---------------------------------------------------
 3 files changed, 12 insertions(+), 63 deletions(-)

diff --git a/t/t9700-perl-git.sh b/t/t9700-perl-git.sh
index 3999bf7..6c22dbc 100755
--- a/t/t9700-perl-git.sh
+++ b/t/t9700-perl-git.sh
@@ -46,8 +46,7 @@ test_expect_success \
      git config --add test.int 2k
      '
 
-test_external_without_stderr \
-    'Perl API' \
-    "$PERL_PATH" "$TEST_DIRECTORY"/t9700/test.pl
+"$PERL_PATH" "$TEST_DIRECTORY"/t9700/test.pl
 
-test_done
+# The Perl test finalizes the plan, so don't call test_done() here.
+GIT_EXIT_OK=t
diff --git a/t/t9700/test.pl b/t/t9700/test.pl
index 666722d..c1ac913 100755
--- a/t/t9700/test.pl
+++ b/t/t9700/test.pl
@@ -7,6 +7,13 @@ use strict;
 
 use Test::More qw(no_plan);
 
+BEGIN {
+ # t9700-perl-git.sh kicks off our testing, so we have to go from
+ # there.
+ $Test::Builder::Test->{Curr_Test} = 1;
+ $Test::Builder::Test->{No_Ending} = 1;
+}
+
 use Cwd;
 use File::Basename;
 
@@ -105,3 +112,5 @@ my $last_commit = $r2->command_oneline(qw(rev-parse --verify HEAD));
 like($last_commit, qr/^[0-9a-fA-F]{40}$/, 'rev-parse returned hash');
 my $dir_commit = $r2->command_oneline('log', '-n1', '--pretty=format:%H', '.');
 isnt($last_commit, $dir_commit, 'log . does not show last commit');
+
+printf "1..%d\n", $Test::Builder::Test->{Curr_Test};
diff --git a/t/test-lib.sh b/t/test-lib.sh
index 282314c..b3c58f2 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -453,65 +453,6 @@ test_expect_code () {
  echo >&3 ""
 }
 
-# test_external runs external test scripts that provide continuous
-# test output about their progress, and succeeds/fails on
-# zero/non-zero exit code.  It outputs the test output on stdout even
-# in non-verbose mode, and announces the external script with "* run
-# <n>: ..." before running it.  When providing relative paths, keep in
-# mind that all scripts run in "trash directory".
-# Usage: test_external description command arguments...
-# Example: test_external 'Perl API' perl ../path/to/test.pl
-test_external () {
- test "$#" = 4 && { prereq=$1; shift; } || prereq=
- test "$#" = 3 ||
- error >&5 "bug in the test script: not 3 or 4 parameters to test_external"
- descr="$1"
- shift
- if ! test_skip "$descr" "$@"
- then
- # Announce the script to reduce confusion about the
- # test output that follows.
- say_color "" " run $test_count: $descr ($*)"
- # Run command; redirect its stderr to &4 as in
- # test_run_, but keep its stdout on our stdout even in
- # non-verbose mode.
- "$@" 2>&4
- if [ "$?" = 0 ]
- then
- test_ok_ "$descr"
- else
- test_failure_ "$descr" "$@"
- fi
- fi
-}
-
-# Like test_external, but in addition tests that the command generated
-# no output on stderr.
-test_external_without_stderr () {
- # The temporary file has no (and must have no) security
- # implications.
- tmp="$TMPDIR"; if [ -z "$tmp" ]; then tmp=/tmp; fi
- stderr="$tmp/git-external-stderr.$$.tmp"
- test_external "$@" 4> "$stderr"
- [ -f "$stderr" ] || error "Internal error: $stderr disappeared."
- descr="no stderr: $1"
- shift
- say >&3 "expecting no stderr from previous command"
- if [ ! -s "$stderr" ]; then
- rm "$stderr"
- test_ok_ "$descr"
- else
- if [ "$verbose" = t ]; then
- output=`echo; echo Stderr is:; cat "$stderr"`
- else
- output=
- fi
- # rm first in case test_failure exits.
- rm "$stderr"
- test_failure_ "$descr" "$@" "$output"
- fi
-}
-
 # This is not among top-level (test_expect_success | test_expect_failure)
 # but is a prefix that can be used in the test script, like:
 #
--
1.7.1.dirty

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

[PATCH 6/6] Add test_harness make target for testing with prove(1)

Ævar Arnfjörð Bjarmason
In reply to this post by Ævar Arnfjörð Bjarmason
The --jobs option I'm supplying to prove(1) is not supported in all
prove versions, some older ones (before Perl 5.10.1) don't have
it. Anyone that's hacking Git probably has 5.10.1, and you can always
use the old `make test` for non-parallel tests.

Signed-off-by: Ævar Arnfjörð Bjarmason <[hidden email]>
---
 Makefile   |    3 +++
 t/Makefile |    3 +++
 2 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/Makefile b/Makefile
index 910f471..c72c3d4 100644
--- a/Makefile
+++ b/Makefile
@@ -1912,6 +1912,9 @@ export NO_SVN_TESTS
 test: all
  $(MAKE) -C t/ all
 
+test_harness: all
+ $(MAKE) -C t/ test_harness
+
 test-ctype$X: ctype.o
 
 test-date$X: date.o ctype.o
diff --git a/t/Makefile b/t/Makefile
index 25c559b..ef876e4 100644
--- a/t/Makefile
+++ b/t/Makefile
@@ -19,6 +19,9 @@ TSVN = $(wildcard t91[0-9][0-9]-*.sh)
 all: pre-clean
  $(MAKE) aggregate-results-and-cleanup
 
+test_harness:
+ prove --jobs 9 ./t[0-9]*.sh
+
 $(T):
  @echo "*** $@ ***"; GIT_CONFIG=.git/config '$(SHELL_PATH_SQ)' $@ $(GIT_TEST_OPTS)
 
--
1.7.1.dirty

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 1/6] Make test-lib.sh emit valid TAP format

Sverre Rabbelier-2
In reply to this post by Ævar Arnfjörð Bjarmason
Heya,

On Fri, May 7, 2010 at 21:37, Ævar Arnfjörð Bjarmason <[hidden email]> wrote:
> On my quad Xeon server running the test suite with `make test` takes
> 21 minutes. Running it with `prove -j 15 ./t[0-9]*.sh` takes just over
> 5 minutes.

That's not a fair comparison, how much does 'make -j 15 test' take you?

--
Cheers,

Sverre Rabbelier
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 1/6] Make test-lib.sh emit valid TAP format

Jakub Narębski
In reply to this post by Ævar Arnfjörð Bjarmason
Ævar Arnfjörð Bjarmason  <[hidden email]> writes:

> TAP, the the Test Anything Protocol, is a simple text-based interface
> between testing modules in a test harness. test-lib.sh's output was
> already very close to being valid TAP. This change brings it all the
> way there.
>
> The advantage of using TAP is that any program that reads the format
> (a "test harness") can run the tests. The most popular of these is the
> prove(1) utility that comes with Perl. It can run tests in parallel,
> display colored output, format the output to console, file, HTML etc.,
> and much more.
>
> On my quad Xeon server running the test suite with `make test` takes
> 21 minutes. Running it with `prove -j 15 ./t[0-9]*.sh` takes just over
> 5 minutes.
>
> With parallel tests the whole test suite doesn't have to stall because
> tests like t3404-rebase-interactive.sh take a long time.

I would have thought that it would be better for git test suite to
enable TAP output with --tap switch.  On the other hand changing
output to TAP, replacing old format, would be less code to maintain.


I am not sure if testing with 'prove' and TAP output is compatibile
with all current git test suite options, i.e. --debug, --tee,
--verbose and --immediate, and with GIT_SKIP_TESTS environmental
variable.  Also valuable way of checking where the error occurs in the
test, namely 'sh -x ./tXXXX-test.sh' would not work, I think, with
'prove'.

Note also that having Perl (and 'prove') installed is not requirement
for git runnig, and I think it should not be requirement for git
development.


P.S. I think that such series would be better with the cover letter.
--
Jakub Narebski
Poland
ShadeHawk on #git
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 6/6] Add test_harness make target for testing with prove(1)

Jakub Narębski
In reply to this post by Ævar Arnfjörð Bjarmason
Ævar Arnfjörð Bjarmason  <[hidden email]> writes:

> The --jobs option I'm supplying to prove(1) is not supported in all
> prove versions, some older ones (before Perl 5.10.1) don't have
> it. Anyone that's hacking Git probably has 5.10.1, and you can always
> use the old `make test` for non-parallel tests.

I am hacking Git, and I don't have perl >= 5.10.1.  On the other hand
I can alway run "make test" or individual tests instead of
"make test_harness".

A better solution would be to check Perl version, or even better of
"prove" supports '--jobs' option, and selectively enable parallel
running of test harness in Makefile.
--
Jakub Narebski
Poland
ShadeHawk on #git
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 2/6] Move t6000lib.sh to lib-*

Jakub Narębski
In reply to this post by Ævar Arnfjörð Bjarmason
Ævar Arnfjörð Bjarmason  <[hidden email]> writes:

> The naming of this test library conflicted with the recommendation in
> t/README's "Naming Tests" section. Now it's possible to naïvely test
> t[0-9]*.sh with prove(1) without running into errors.
>
> Signed-off-by: Ævar Arnfjörð Bjarmason <[hidden email]>
> ---
>  t/{t6000lib.sh => lib-t6000.sh} |    0
>  t/t6002-rev-list-bisect.sh      |    2 +-
>  t/t6003-rev-list-topo-order.sh  |    2 +-
>  t/t6101-rev-parse-parents.sh    |    2 +-
>  4 files changed, 3 insertions(+), 3 deletions(-)
>  rename t/{t6000lib.sh => lib-t6000.sh} (100%)

That's a good change anyway, even if the rest of series is not
accepted, or at least not accepted in current version.

--
Jakub Narebski
Poland
ShadeHawk on #git
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 4/6] Turn setup code in t2007-checkout-symlink.sh into a test

Jakub Narębski
In reply to this post by Ævar Arnfjörð Bjarmason
Ævar Arnfjörð Bjarmason  <[hidden email]> writes:

> Previously the test would print to stdout which interfered with the
> TAP output. Now this scaffolding code is just a normal test.
>
> Signed-off-by: Ævar Arnfjörð Bjarmason <[hidden email]>

That's a good change anyway, independent on the TAP-ification of
output of git test suite.

> ---
>  t/t2007-checkout-symlink.sh |    6 ++++--
>  1 files changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/t/t2007-checkout-symlink.sh b/t/t2007-checkout-symlink.sh
> index 616ca2a..05cc8fd 100755
> --- a/t/t2007-checkout-symlink.sh
> +++ b/t/t2007-checkout-symlink.sh
> @@ -44,8 +44,10 @@ test_expect_success 'switch from symlink to dir' '
>  
>  '
>  
> -rm -fr frotz xyzzy nitfol &&
> -git checkout -f master || exit
> +test_expect_success 'Remove temporary directories & switch to master' '
> + rm -fr frotz xyzzy nitfol &&
> + git checkout -f master
> +'
>  
>  test_expect_success 'switch from dir to symlink' '
>  
> --
> 1.7.1.dirty
>

--
Jakub Narebski
Poland
ShadeHawk on #git
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 1/6] Make test-lib.sh emit valid TAP format

Ævar Arnfjörð Bjarmason
In reply to this post by Sverre Rabbelier-2
On Fri, May 7, 2010 at 19:50, Sverre Rabbelier <[hidden email]> wrote:
> On Fri, May 7, 2010 at 21:37, Ævar Arnfjörð Bjarmason <[hidden email]> wrote:
>> On my quad Xeon server running the test suite with `make test` takes
>> 21 minutes. Running it with `prove -j 15 ./t[0-9]*.sh` takes just over
>> 5 minutes.
>
> That's not a fair comparison, how much does 'make -j 15 test' take you?

Indeed it isn't. I embarrassingly hadn't tested running `make test`
with -j. The time it takes prove and `make test` to complete tests
with -j 15 is the same (within a margin of error), although prove's
output is much better.

I'll submit an improved patch that doesn't have a confusing commit message.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 1/6] Make test-lib.sh emit valid TAP format

Ævar Arnfjörð Bjarmason
In reply to this post by Jakub Narębski
On Fri, May 7, 2010 at 20:17, Jakub Narebski <[hidden email]> wrote:

> Ævar Arnfjörð Bjarmason  <[hidden email]> writes:
>
>> TAP, the the Test Anything Protocol, is a simple text-based interface
>> between testing modules in a test harness. test-lib.sh's output was
>> already very close to being valid TAP. This change brings it all the
>> way there.
>>
>> The advantage of using TAP is that any program that reads the format
>> (a "test harness") can run the tests. The most popular of these is the
>> prove(1) utility that comes with Perl. It can run tests in parallel,
>> display colored output, format the output to console, file, HTML etc.,
>> and much more.
>>
>> On my quad Xeon server running the test suite with `make test` takes
>> 21 minutes. Running it with `prove -j 15 ./t[0-9]*.sh` takes just over
>> 5 minutes.
>>
>> With parallel tests the whole test suite doesn't have to stall because
>> tests like t3404-rebase-interactive.sh take a long time.
>
> I would have thought that it would be better for git test suite to
> enable TAP output with --tap switch.  On the other hand changing
> output to TAP, replacing old format, would be less code to maintain.

That was my thinking as well. TAP is human readable like the old
format, the only difference is that TAP is standardized and can be
read by a wide variety of existing programs. For comparison:

Before the patch:

    $ ./t0005-signals.sh
    *   ok 1: sigchain works
    * passed all 1 test(s)

After:

    $ ./t0005-signals.sh
    ok 1 - sigchain works
    # passed all 1 test(s)
    1..1

Or alternatively:

    $ prove ./t0005-signals.sh
    ./t0005-signals.sh .. ok
    All tests successful.
    Files=1, Tests=1,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.01
cusr  0.02 csys =  0.06 CPU)
    Result: PASS

> I am not sure if testing with 'prove' and TAP output is compatibile
> with all current git test suite options, i.e. --debug, --tee,
> --verbose and --immediate, and with GIT_SKIP_TESTS environmental
> variable.

If you pass things like the --verbose output through prove it'll
work. This is because TAP specifies that any unknown output can be
ignored (i.e. lines that don't match /^(?:not)? ok/).

> Also valuable way of checking where the error occurs in the test,
> namely 'sh -x ./tXXXX-test.sh' would not work, I think, with
> 'prove'.

This doesn't break any method of currently running tests (e.g. running
them manually). You don't *have* to run prove, you just can if you
want to.

> Note also that having Perl (and 'prove') installed is not requirement
> for git runnig, and I think it should not be requirement for git
> development.

It should not be. Using fancy TAP formatting tools (whether it's prove
or something else) is purely optional.

> P.S. I think that such series would be better with the cover letter.

I'll make sure to submit one in the future if I send a large patch
series again, sorry.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

[PATCH 1/6 v2] Make test-lib.sh emit valid TAP format

Ævar Arnfjörð Bjarmason
In reply to this post by Ævar Arnfjörð Bjarmason
TAP, the the Test Anything Protocol, is a simple text-based interface
between testing modules in a test harness. test-lib.sh's output was
already very close to being valid TAP. This change brings it all the
way there. Before:

   $ ./t0005-signals.sh
   *   ok 1: sigchain works
   * passed all 1 test(s)

And after:

   $ ./t0005-signals.sh
   ok 1 - sigchain works
   # passed all 1 test(s)
   1..1

The advantage of using TAP is that any program that reads the format
(a "test harness") can run the tests. The most popular of these is the
prove(1) utility that comes with Perl. It can run tests in parallel,
display colored output, format the output to console, file, HTML etc.,
and much more. An example:

   $ prove ./t0005-signals.sh
   ./t0005-signals.sh .. ok
   All tests successful.
   Files=1, Tests=1,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.01 cusr  0.02 csys =  0.06 CPU)
   Result: PASS

prove(1) gives you human readable output without being too
verbose. Running the test suite in parallel with `make test -j15`
produces a flood of text. Running them with `prove -j 15 ./t[0-9]*.sh`
makes it easy to follow what's going on.

Signed-off-by: Ævar Arnfjörð Bjarmason <[hidden email]>
---

On Fri, May 7, 2010 at 20:25, Ævar Arnfjörð Bjarmason <[hidden email]> wrote:
> I'll submit an improved patch that doesn't have a confusing commit message

Here's that patch. It now also has examples of the test output
before/after and a demonstration of prove(1) running on a test.

 t/README      |   49 +++++++++++++++++++++++++++++++++----------------
 t/test-lib.sh |   30 ++++++++++++++++++------------
 2 files changed, 51 insertions(+), 28 deletions(-)

diff --git a/t/README b/t/README
index dcd3ebb..fc4bb04 100644
--- a/t/README
+++ b/t/README
@@ -18,25 +18,42 @@ The easiest way to run tests is to say "make".  This runs all
 the tests.
 
     *** t0000-basic.sh ***
-    *   ok 1: .git/objects should be empty after git-init in an empty repo.
-    *   ok 2: .git/objects should have 256 subdirectories.
-    *   ok 3: git-update-index without --add should fail adding.
+    ok 1 - .git/objects should be empty after git init in an empty repo.
+    ok 2 - .git/objects should have 3 subdirectories.
+    ok 3 - success is reported like this
     ...
-    *   ok 23: no diff after checkout and git-update-index --refresh.
-    * passed all 23 test(s)
-    *** t0100-environment-names.sh ***
-    *   ok 1: using old names should issue warnings.
-    *   ok 2: using old names but having new names should not issue warnings.
-    ...
-
-Or you can run each test individually from command line, like
-this:
-
-    $ sh ./t3001-ls-files-killed.sh
-    *   ok 1: git-update-index --add to add various paths.
-    *   ok 2: git-ls-files -k to show killed files.
-    *   ok 3: validate git-ls-files -k output.
-    * passed all 3 test(s)
+    ok 43 - very long name in the index handled sanely
+    # fixed 1 known breakage(s)
+    # still have 1 known breakage(s)
+    # passed all remaining 42 test(s)
+    1..43
+    *** t0001-init.sh ***
+    ok 1 - plain
+    ok 2 - plain with GIT_WORK_TREE
+    ok 3 - plain bare
+
+Since the tests all output TAP (see http://testanything.org) they can
+be run with any TAP harness. Here's an example of paralell testing
+powered by a recent version of prove(1):
+        
+    $ prove --timer --jobs 15 ./t[0-9]*.sh
+    [19:17:33] ./t0005-signals.sh ................................... ok       36 ms
+    [19:17:33] ./t0022-crlf-rename.sh ............................... ok       69 ms
+    [19:17:33] ./t0024-crlf-archive.sh .............................. ok      154 ms
+    [19:17:33] ./t0004-unwritable.sh ................................ ok      289 ms
+    [19:17:33] ./t0002-gitfile.sh ................................... ok      480 ms
+    ===(     102;0  25/?  6/?  5/?  16/?  1/?  4/?  2/?  1/?  3/?  1... )===
+    
+You can also run each test individually from command line, like this:
+
+    $ sh ./t3010-ls-files-killed-modified.sh
+    ok 1 - git update-index --add to add various paths.
+    ok 2 - git ls-files -k to show killed files.
+    ok 3 - validate git ls-files -k output.
+    ok 4 - git ls-files -m to show modified files.
+    ok 5 - validate git ls-files -m output.
+    # passed all 5 test(s)
+    1..5
 
 You can pass --verbose (or -v), --debug (or -d), and --immediate
 (or -i) command line argument to the test, or by setting GIT_TEST_OPTS
diff --git a/t/test-lib.sh b/t/test-lib.sh
index acce3d0..282314c 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -161,7 +161,7 @@ if test -n "$color"; then
  *) test -n "$quiet" && return;;
  esac
  shift
- printf "* %s" "$*"
+ printf "%s" "$*"
  tput sgr0
  echo
  )
@@ -170,7 +170,7 @@ else
  say_color() {
  test -z "$1" && test -n "$quiet" && return
  shift
- echo "* $*"
+ echo "$*"
  }
 fi
 
@@ -340,25 +340,25 @@ test_have_prereq () {
 
 test_ok_ () {
  test_success=$(($test_success + 1))
- say_color "" "  ok $test_count: $@"
+ say_color "" "ok $test_count - $@"
 }
 
 test_failure_ () {
  test_failure=$(($test_failure + 1))
- say_color error "FAIL $test_count: $1"
+ say_color error "not ok - $test_count $1"
  shift
- echo "$@" | sed -e 's/^/ /'
+ echo "$@" | sed -e 's/^/# /'
  test "$immediate" = "" || { GIT_EXIT_OK=t; exit 1; }
 }
 
 test_known_broken_ok_ () {
  test_fixed=$(($test_fixed+1))
- say_color "" "  FIXED $test_count: $@"
+ say_color "" "ok $test_count - $@ # TODO known breakage"
 }
 
 test_known_broken_failure_ () {
  test_broken=$(($test_broken+1))
- say_color skip "  still broken $test_count: $@"
+ say_color skip "not ok $test_count - $@ # TODO known breakage"
 }
 
 test_debug () {
@@ -390,7 +390,7 @@ test_skip () {
  case "$to_skip" in
  t)
  say_color skip >&3 "skipping test: $@"
- say_color skip "skip $test_count: $1"
+ say_color skip "ok $test_count: # skip $1"
  : true
  ;;
  *)
@@ -601,18 +601,22 @@ test_done () {
 
  if test "$test_fixed" != 0
  then
- say_color pass "fixed $test_fixed known breakage(s)"
+ say_color pass "# fixed $test_fixed known breakage(s)"
  fi
  if test "$test_broken" != 0
  then
- say_color error "still have $test_broken known breakage(s)"
+ say_color error "# still have $test_broken known breakage(s)"
  msg="remaining $(($test_count-$test_broken)) test(s)"
  else
  msg="$test_count test(s)"
  fi
  case "$test_failure" in
  0)
- say_color pass "passed all $msg"
+ # Maybe print SKIP message
+ [ -z "$skip_all" ] || skip_all=" # SKIP $skip_all"
+
+ say_color pass "# passed all $msg"
+ say "1..$test_count$skip_all"
 
  test -d "$remove_trash" &&
  cd "$(dirname "$remove_trash")" &&
@@ -621,7 +625,9 @@ test_done () {
  exit 0 ;;
 
  *)
- say_color error "failed $test_failure among $msg"
+ say_color error "# failed $test_failure among $msg"
+ say "1..$test_count"
+
  exit 1 ;;
 
  esac
--
1.7.1.dirty

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 6/6] Add test_harness make target for testing with prove(1)

Junio C Hamano
In reply to this post by Jakub Narębski
Jakub Narebski <[hidden email]> writes:

> Ævar Arnfjörð Bjarmason  <[hidden email]> writes:
>
>> The --jobs option I'm supplying to prove(1) is not supported in all
>> prove versions, some older ones (before Perl 5.10.1) don't have
>> it. Anyone that's hacking Git probably has 5.10.1, and you can always
>> use the old `make test` for non-parallel tests.
>
> I am hacking Git, and I don't have perl >= 5.10.1.  On the other hand
> I can alway run "make test" or individual tests instead of
> "make test_harness".
>
> A better solution would be to check Perl version, or even better of
> "prove" supports '--jobs' option, and selectively enable parallel
> running of test harness in Makefile.

Besides, "make -j15 test" from the toplevel already runs the tests in
parallel.  I don't see much point in this change.

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 6/6] Add test_harness make target for testing with prove(1)

Tor Arntsen
In reply to this post by Ævar Arnfjörð Bjarmason
On Fri, May 7, 2010 at 21:37, Ævar Arnfjörð Bjarmason <[hidden email]> wrote:
> The --jobs option I'm supplying to prove(1) is not supported in all
> prove versions, some older ones (before Perl 5.10.1) don't have
> it. Anyone that's hacking Git probably has 5.10.1, [...]

Not on the platforms I'm testing Git on. (The non-Linux ones.) It's
troublesome enough that parts of Git itself requires 5.8 or later..

>and you can always use the old `make test` for non-parallel tests.

If tests still can be run.. ok then.

-Tor
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 1/6 v2] Make test-lib.sh emit valid TAP format

Ævar Arnfjörð Bjarmason
In reply to this post by Ævar Arnfjörð Bjarmason
On Fri, May 7, 2010 at 20:59, Ævar Arnfjörð Bjarmason <[hidden email]> wrote:
> Here's that patch. It now also has examples of the test output
> before/after and a demonstration of prove(1) running on a test.

Is there something I can do to make this more acceptable for
inclusion, or is it OK as-is?
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html