[PATCH v1 0/3] travis-ci: run Git bisect on failed tests
Locked
[PATCH v1 0/3] travis-ci: run Git bisect on failed tests
 
|
From: Lars Schneider <[hidden email]>
This patch series run Git bisect on failing tests on Travis CI. Example output on Travis CI: https://travis-ci.org/larsxschneider/git/jobs/132049871 Plaintext example output: https://s3.amazonaws.com/archive.travis-ci.org/jobs/132049871/log.txt Please scroll all the way down to see the bisect output. Cheers, Lars Lars Schneider (3): travis-ci: move "after_failure" code to dedicated file in /ci travis-ci: disable verbose test output travis-ci: run Git bisect on failed tests .travis.yml | 14 ++-------- ci/test-report.sh | 84 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 86 insertions(+), 12 deletions(-) create mode 100755 ci/test-report.sh -- 2.5.1 -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [hidden email] More majordomo info at http://vger.kernel.org/majordomo-info.html |
[PATCH v1 1/3] travis-ci: move "after_failure" code to dedicated file in /ci
|
|
From: Lars Schneider <[hidden email]>
Move the code and adjust it to the Git shell script coding guidelines. Signed-off-by: Lars Schneider <[hidden email]> --- .travis.yml | 12 +----------- ci/test-report.sh | 19 +++++++++++++++++++ 2 files changed, 20 insertions(+), 11 deletions(-) create mode 100755 ci/test-report.sh diff --git a/.travis.yml b/.travis.yml index adab5b8..a93ecb3 100644 --- a/.travis.yml +++ b/.travis.yml @@ -101,17 +101,7 @@ script: make --quiet test after_failure: - > : '<-- Click here to see detailed test output! '; - for TEST_EXIT in t/test-results/*.exit; - do - if [ "$(cat "$TEST_EXIT")" != "0" ]; - then - TEST_OUT="${TEST_EXIT%exit}out"; - echo "------------------------------------------------------------------------"; - echo "$(tput setaf 1)${TEST_OUT}...$(tput sgr0)"; - echo "------------------------------------------------------------------------"; - cat "${TEST_OUT}"; - fi; - done; + ./ci/test-report.sh notifications: email: false diff --git a/ci/test-report.sh b/ci/test-report.sh new file mode 100755 index 0000000..d08a999 --- /dev/null +++ b/ci/test-report.sh @@ -0,0 +1,19 @@ +#!/bin/sh +# +# Print test results +# +for TEST_EXIT in t/test-results/*.exit +do + if test "$(cat "$TEST_EXIT")" != "0" + then + TEST="${TEST_EXIT%.exit}" + TEST_OUT="${TEST}.out" + echo "------------------------------------------------------------------------" + echo " $(tput setaf 1)${TEST} Output$(tput sgr0)" + echo "------------------------------------------------------------------------" + cat "$TEST_OUT" + echo "" + echo "" + fi +done + -- 2.5.1 -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [hidden email] More majordomo info at http://vger.kernel.org/majordomo-info.html |
[PATCH v1 2/3] travis-ci: disable verbose test output
|
|
In reply to this post by larsxschneider
From: Lars Schneider <[hidden email]>
The verbose output clutters the Travis CI webview and is not really useful since test debugging usually happens on a local machine. Signed-off-by: Lars Schneider <[hidden email]> --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index a93ecb3..81d2027 100644 --- a/.travis.yml +++ b/.travis.yml @@ -29,7 +29,7 @@ env: - LINUX_GIT_LFS_VERSION="1.2.0" - DEFAULT_TEST_TARGET=prove - GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save" - - GIT_TEST_OPTS="--verbose --tee" + - GIT_TEST_OPTS="--tee" - GIT_TEST_CLONE_2GB=YesPlease # t9810 occasionally fails on Travis CI OS X # t9816 occasionally fails with "TAP out of sequence errors" on Travis CI OS X -- 2.5.1 -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [hidden email] More majordomo info at http://vger.kernel.org/majordomo-info.html |
[PATCH v1 3/3] travis-ci: run Git bisect on failed tests
|
|
In reply to this post by larsxschneider
From: Lars Schneider <[hidden email]>
Junio usually pushes many commits at once to the public "pu"/"next"/ "master" branches. If a test fails then it is not obvious what commit caused the failure. Therefore we run Git bisect with the merge base between the failing rev and its more stable branch ("next" for "pu", "master" for "next", and "maint" for "master") as good ref to find the offending commit. This is only enabled on "github.com/git/git" because there we can assume that all relevant branches are up to date. Signed-off-by: Lars Schneider <[hidden email]> --- .travis.yml | 2 +- ci/test-report.sh | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 66 insertions(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 81d2027..922807b 100644 --- a/.travis.yml +++ b/.travis.yml @@ -101,7 +101,7 @@ script: make --quiet test after_failure: - > : '<-- Click here to see detailed test output! '; - ./ci/test-report.sh + ./ci/test-report.sh $TRAVIS_REPO_SLUG $TRAVIS_BRANCH; notifications: email: false diff --git a/ci/test-report.sh b/ci/test-report.sh index d08a999..8f7adad 100755 --- a/ci/test-report.sh +++ b/ci/test-report.sh @@ -1,5 +1,12 @@ #!/bin/sh # +# Print test results and run Git bisect on failed tests. +# +REPO_ORG_NAME=$1 +CURRENT_BRANCH_NAME=$2 + + +# # Print test results # for TEST_EXIT in t/test-results/*.exit @@ -17,3 +24,61 @@ do fi done + +# +# Run Git bisect +# +run_bisect () { + TEST_SCRIPT=$1 + BAD_REV=$2 + GOOD_RV=$3 + TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null) + cat > "$TMPDIR/bisect-run.sh" <<EOF + +EOF + chmod +x "$TMPDIR/bisect-run.sh" + git bisect start $BAD_REV $GOOD_RV + git bisect run "$TMPDIR/bisect-run.sh" + if test -e ./t/$TEST_SCRIPT.sh && make --jobs=2 >/dev/null 2>&1 + then + cd t && ./$TEST_SCRIPT.sh >/dev/null 2>&1 + else + # If the test file does not exist or the build fails then tell + # Git bisect to skip the commit. + exit 125 + fi + git bisect reset >/dev/null 2>&1 +} + +case "$CURRENT_BRANCH_NAME" in + master) STABLE_BRANCH="maint";; + next) STABLE_BRANCH="master";; + pu) STABLE_BRANCH="next";; +esac + +if test "$REPO_ORG_NAME" = "git/git" && test -n $STABLE_BRANCH +then + BAD_REV=$(git rev-parse HEAD) + + # Travis CI clones are shallow. It is possible that the last good revision + # was not fetched, yet. Therefore we need to fetch all commits on the + # stable branch. + git config remote.origin.fetch "+refs/heads/$STABLE_BRANCH:refs/remotes/origin/$STABLE_BRANCH" + git fetch --unshallow --quiet + LAST_GOOD_REV=$(git merge-base $BAD_REV "remotes/origin/$STABLE_BRANCH") + + for TEST_EXIT in t/test-results/*.exit + do + if test "$(cat "$TEST_EXIT")" != "0" + then + TEST="${TEST_EXIT%.exit}" + TEST_SCRIPT=${TEST#t/test-results/} + echo "------------------------------------------------------------------------" + echo " $(tput setaf 1)${TEST} Bisect$(tput sgr0)" + echo "------------------------------------------------------------------------" + run_bisect $TEST_SCRIPT $BAD_REV $LAST_GOOD_REV + echo "" + echo "" + fi + done +fi -- 2.5.1 -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [hidden email] More majordomo info at http://vger.kernel.org/majordomo-info.html |
Re: [PATCH v1 3/3] travis-ci: run Git bisect on failed tests
|
|
On Sun, May 22, 2016 at 1:00 PM, <[hidden email]> wrote:
[...] > +# > +# Run Git bisect > +# > +run_bisect () { > + TEST_SCRIPT=$1 > + BAD_REV=$2 > + GOOD_RV=$3 > + TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null) > + cat > "$TMPDIR/bisect-run.sh" <<EOF > + > +EOF > + chmod +x "$TMPDIR/bisect-run.sh" > + git bisect start $BAD_REV $GOOD_RV > + git bisect run "$TMPDIR/bisect-run.sh" > + if test -e ./t/$TEST_SCRIPT.sh && make --jobs=2 >/dev/null 2>&1 > + then > + cd t && ./$TEST_SCRIPT.sh >/dev/null 2>&1 > + else > + # If the test file does not exist or the build fails then tell > + # Git bisect to skip the commit. > + exit 125 > + fi Shouldn't all the above "if ... fi" be in the here document creating "$TMPDIR/bisect-run.sh"? > + git bisect reset >/dev/null 2>&1 > +} -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [hidden email] More majordomo info at http://vger.kernel.org/majordomo-info.html |
Re: [PATCH v1 3/3] travis-ci: run Git bisect on failed tests
|
|
> On 22 May 2016, at 17:35, Christian Couder <[hidden email]> wrote: > > On Sun, May 22, 2016 at 1:00 PM, <[hidden email]> wrote: > > [...] > >> +# >> +# Run Git bisect >> +# >> +run_bisect () { >> + TEST_SCRIPT=$1 >> + BAD_REV=$2 >> + GOOD_RV=$3 >> + TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null) >> + cat > "$TMPDIR/bisect-run.sh" <<EOF >> + >> +EOF >> + chmod +x "$TMPDIR/bisect-run.sh" >> + git bisect start $BAD_REV $GOOD_RV >> + git bisect run "$TMPDIR/bisect-run.sh" >> + if test -e ./t/$TEST_SCRIPT.sh && make --jobs=2 >/dev/null 2>&1 >> + then >> + cd t && ./$TEST_SCRIPT.sh >/dev/null 2>&1 >> + else >> + # If the test file does not exist or the build fails then tell >> + # Git bisect to skip the commit. >> + exit 125 >> + fi > > Shouldn't all the above "if ... fi" be in the here document creating > "$TMPDIR/bisect-run.sh"? Thank you, Lars > >> + git bisect reset >/dev/null 2>&1 >> +} -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [hidden email] More majordomo info at http://vger.kernel.org/majordomo-info.html |
Re: [PATCH v1 3/3] travis-ci: run Git bisect on failed tests
|
|
In reply to this post by larsxschneider
Hey Lars,
On Sun, May 22, 2016 at 4:30 PM, <[hidden email]> wrote: > From: Lars Schneider <[hidden email]> > > Junio usually pushes many commits at once to the public "pu"/"next"/ > "master" branches. If a test fails then it is not obvious what commit > caused the failure. Therefore we run Git bisect with the merge base > between the failing rev and its more stable branch ("next" for "pu", > "master" for "next", and "maint" for "master") as good ref to find the > offending commit. This is only enabled on "github.com/git/git" because > there we can assume that all relevant branches are up to date. > > Signed-off-by: Lars Schneider <[hidden email]> > --- > .travis.yml | 2 +- > ci/test-report.sh | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 66 insertions(+), 1 deletion(-) > > diff --git a/.travis.yml b/.travis.yml > index 81d2027..922807b 100644 > --- a/.travis.yml > +++ b/.travis.yml > @@ -101,7 +101,7 @@ script: make --quiet test > after_failure: > - > > : '<-- Click here to see detailed test output! '; > - ./ci/test-report.sh > + ./ci/test-report.sh $TRAVIS_REPO_SLUG $TRAVIS_BRANCH; > > notifications: > email: false > diff --git a/ci/test-report.sh b/ci/test-report.sh > index d08a999..8f7adad 100755 > --- a/ci/test-report.sh > +++ b/ci/test-report.sh > @@ -1,5 +1,12 @@ > #!/bin/sh > # > +# Print test results and run Git bisect on failed tests. > +# > +REPO_ORG_NAME=$1 > +CURRENT_BRANCH_NAME=$2 > + > + > +# > # Print test results > # > for TEST_EXIT in t/test-results/*.exit > @@ -17,3 +24,61 @@ do > fi > done > > + > +# > +# Run Git bisect > +# > +run_bisect () { > + TEST_SCRIPT=$1 > + BAD_REV=$2 > + GOOD_RV=$3 > + TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null) > + cat > "$TMPDIR/bisect-run.sh" <<EOF If you are doing a re-roll, then you could probably fix the style issue. ' cat >"$TMPDIR..." ' > + > +EOF > + chmod +x "$TMPDIR/bisect-run.sh" > + git bisect start $BAD_REV $GOOD_RV > + git bisect run "$TMPDIR/bisect-run.sh" > + if test -e ./t/$TEST_SCRIPT.sh && make --jobs=2 >/dev/null 2>&1 > + then > + cd t && ./$TEST_SCRIPT.sh >/dev/null 2>&1 > + else > + # If the test file does not exist or the build fails then tell > + # Git bisect to skip the commit. > + exit 125 > + fi > + git bisect reset >/dev/null 2>&1 > +} > + > +case "$CURRENT_BRANCH_NAME" in > + master) STABLE_BRANCH="maint";; > + next) STABLE_BRANCH="master";; > + pu) STABLE_BRANCH="next";; > +esac > + > +if test "$REPO_ORG_NAME" = "git/git" && test -n $STABLE_BRANCH > +then > + BAD_REV=$(git rev-parse HEAD) > + > + # Travis CI clones are shallow. It is possible that the last good revision > + # was not fetched, yet. Therefore we need to fetch all commits on the > + # stable branch. > + git config remote.origin.fetch "+refs/heads/$STABLE_BRANCH:refs/remotes/origin/$STABLE_BRANCH" > + git fetch --unshallow --quiet > + LAST_GOOD_REV=$(git merge-base $BAD_REV "remotes/origin/$STABLE_BRANCH") > + > + for TEST_EXIT in t/test-results/*.exit > + do > + if test "$(cat "$TEST_EXIT")" != "0" > + then > + TEST="${TEST_EXIT%.exit}" > + TEST_SCRIPT=${TEST#t/test-results/} > + echo "------------------------------------------------------------------------" > + echo " $(tput setaf 1)${TEST} Bisect$(tput sgr0)" > + echo "------------------------------------------------------------------------" > + run_bisect $TEST_SCRIPT $BAD_REV $LAST_GOOD_REV > + echo "" > + echo "" > + fi > + done > +fi Regards, Pranit Bauva -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [hidden email] More majordomo info at http://vger.kernel.org/majordomo-info.html |
Re: [PATCH v1 3/3] travis-ci: run Git bisect on failed tests
|
|
In reply to this post by larsxschneider
[hidden email] writes:
> Junio usually pushes many commits at once to the public "pu"/"next"/ > "master" branches. If a test fails then it is not obvious what commit > caused the failure. Therefore we run Git bisect with the merge base > between the failing rev and its more stable branch ("next" for "pu", > "master" for "next", and "maint" for "master") as good ref to find the > offending commit. This is only enabled on "github.com/git/git" because > there we can assume that all relevant branches are up to date. Just FYI, 'git log next..pu' is not a very interesting range unless you also use the "--no-merges" option. Is there a good way to tell the CI that, instead of testing 'pu' (or a specific branch in general), test all new commits that appear on the first-parent chain between 'master'..'pu'? It would be ideal if CI can do this every time 'master' and 'pu' gets updated. - Find all commits that appear on the first-parent chain between 'master..pu'. - For each of them: - Test it. This tests the merge result and can catch breakages introduced by mismerging. - Also test its second parent if it is a merge. This tests the tip of each topic branch in isolation. CI gets a bonus point if it can remember that the tip of the topic did not move and the commit already passed the test to skip it ;-) - Optionally, for those topics that failed the "tip of the topic branch" test, "bisect master..$topic" to see where it breaks. The thing is, "bisect" works only when you have a _single_ cause of breakage, but because 'pu' is a pile of unsorted raw material yet to be sifted into dirt and gem, once you see a failure at the tip of 'pu', mechanical 'bisect run' may not be a very useful tool, as you do not know if there is only one breakage you are looking for. Because the unit of integration we use is a topic branch (not an individual commit on a topic branch), "this topic is broken", and "this topic itself may be good, but merging it breaks 'pu'" are far more interesting and relevant piece of information than "this commit is what broke 'pu'". -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [hidden email] More majordomo info at http://vger.kernel.org/majordomo-info.html |
Re: [PATCH v1 3/3] travis-ci: run Git bisect on failed tests
|
|
In reply to this post by pranitbauva1997
> On 22 May 2016, at 19:21, Pranit Bauva <[hidden email]> wrote: > > Hey Lars, > > On Sun, May 22, 2016 at 4:30 PM, <[hidden email]> wrote: >> From: Lars Schneider <[hidden email]> >> ... >> + >> +# >> +# Run Git bisect >> +# >> +run_bisect () { >> + TEST_SCRIPT=$1 >> + BAD_REV=$2 >> + GOOD_RV=$3 >> + TMPDIR=$(mktemp -d -t "ci-report-bisect-XXXXXX" 2>/dev/null) >> + cat > "$TMPDIR/bisect-run.sh" <<EOF > > If you are doing a re-roll, then you could probably fix the style issue. > ' cat >"$TMPDIR..." ' Good eye :-) I will fix this, too. Thank you, Lars -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [hidden email] More majordomo info at http://vger.kernel.org/majordomo-info.html |
|
|
In reply to this post by larsxschneider
On Sun, May 22, 2016 at 01:00:55PM +0200, [hidden email] wrote:
> From: Lars Schneider <[hidden email]> > > The verbose output clutters the Travis CI webview and is not really > useful since test debugging usually happens on a local machine. I have not really been using the Travis CI results, so perhaps my opinion does not count. But in other systems, I have found that the more verbose the CI output, the better, simply because you will inevitably be faced with a test that breaks on CI and not your local machine, and you will have no way to get more details. I don't know if Travis provides a better way to hide the output in the non-failing cases. -Peff -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [hidden email] More majordomo info at http://vger.kernel.org/majordomo-info.html |
«
Return to git
|
1 view|%1 views
| Free forum by Nabble | Edit this page |