Quantcast

Failing tests on GitHub

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Failing tests on GitHub

Jeff Allen-2
A while ago our GitHub repo
https://github.com/jythontools/jython/commits/master sprouted continuous
integration tests, thanks (I think) to Darjus. However, these tests
always fail.

This seems to me to remove their primary value, which is as a tripwire
to tell us when something has broken, or when a PR would break it.
That's particularly valuable where tests run on a different platform
from the developer's regression test.

The tests failing at GitHub (over the various tools and test
conditions), that don't fail on the (= my Windows) desktop, are:

test_socket test_sort test_ssl

I'm tweaking the expected failures in regrtest to get it to run cleanly
again under ant on the desktop, as we encourage users to run that. But
perhaps the target should be to have it run cleanly at GitHub too,
adding things that don't to the "list of shame".

Is it sensible to add as expected failures, those things that routinely
fail in the GitHub tests? Or do they fail for a reason we could
immediately fix by configuration there?

Jeff

--
Jeff Allen


------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Jython-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Failing tests on GitHub

Adam Burke-2
If test_socket and test_ssl fail due to environmental reasons - eg can't reliably obtain a non-clashing socket on the build server - it's probably worth having a mechanism to distinguish and disable them in that environment.

Test_sort seems like it should pass everywhere though?

Adam

> 在 2017年3月13日,下午5:59,Jeff Allen <[hidden email]> 写道:
>
> A while ago our GitHub repo
> https://github.com/jythontools/jython/commits/master sprouted continuous
> integration tests, thanks (I think) to Darjus. However, these tests
> always fail.
>
> This seems to me to remove their primary value, which is as a tripwire
> to tell us when something has broken, or when a PR would break it.
> That's particularly valuable where tests run on a different platform
> from the developer's regression test.
>
> The tests failing at GitHub (over the various tools and test
> conditions), that don't fail on the (= my Windows) desktop, are:
>
> test_socket test_sort test_ssl
>
> I'm tweaking the expected failures in regrtest to get it to run cleanly
> again under ant on the desktop, as we encourage users to run that. But
> perhaps the target should be to have it run cleanly at GitHub too,
> adding things that don't to the "list of shame".
>
> Is it sensible to add as expected failures, those things that routinely
> fail in the GitHub tests? Or do they fail for a reason we could
> immediately fix by configuration there?
>
> Jeff
>
> --
> Jeff Allen
>
>
> ------------------------------------------------------------------------------
> Announcing the Oxford Dictionaries API! The API offers world-renowned
> dictionary content that is easy and intuitive to access. Sign up for an
> account today to start using our lexical data to power your apps and
> projects. Get started today and enter our developer competition.
> http://sdm.link/oxford
> _______________________________________________
> Jython-dev mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/jython-dev

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Jython-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Failing tests on GitHub

Jeff Allen-2
Yes, it was misleading to put test_sort in the same list. It fails for
Java 8, everywhere AFAIK (http://bugs.jython.org/issue2399), best dealt
with in regrtest somehow I suppose.

With test_socket test_sort test_ssl we could exclude them from the
custom ant regrtest-*, but it would be nicer if the environment allowed
them to run. (Not testing SSL seems a bit shoddy.)

Jeff Allen

On 13/03/2017 20:07, Adam Burke wrote:

> If test_socket and test_ssl fail due to environmental reasons - eg can't reliably obtain a non-clashing socket on the build server - it's probably worth having a mechanism to distinguish and disable them in that environment.
>
> Test_sort seems like it should pass everywhere though?
>
> Adam
>
>> 在 2017年3月13日,下午5:59,Jeff Allen <[hidden email]> 写道:
>>
>> A while ago our GitHub repo
>> https://github.com/jythontools/jython/commits/master sprouted continuous
>> integration tests, thanks (I think) to Darjus. However, these tests
>> always fail.
>>
>> This seems to me to remove their primary value, which is as a tripwire
>> to tell us when something has broken, or when a PR would break it.
>> That's particularly valuable where tests run on a different platform
>> from the developer's regression test.
>>
>> The tests failing at GitHub (over the various tools and test
>> conditions), that don't fail on the (= my Windows) desktop, are:
>>
>> test_socket test_sort test_ssl
>>
>> I'm tweaking the expected failures in regrtest to get it to run cleanly
>> again under ant on the desktop, as we encourage users to run that. But
>> perhaps the target should be to have it run cleanly at GitHub too,
>> adding things that don't to the "list of shame".
>>
>> Is it sensible to add as expected failures, those things that routinely
>> fail in the GitHub tests? Or do they fail for a reason we could
>> immediately fix by configuration there?
>>
>> Jeff
>>
>> --
>> Jeff Allen
>>
>>
>> ------------------------------------------------------------------------------
>> Announcing the Oxford Dictionaries API! The API offers world-renowned
>> dictionary content that is easy and intuitive to access. Sign up for an
>> account today to start using our lexical data to power your apps and
>> projects. Get started today and enter our developer competition.
>> http://sdm.link/oxford
>> _______________________________________________
>> Jython-dev mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/jython-dev


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Jython-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Failing tests on GitHub

Jeff Allen-2
I have managed to achieve two green bots. So we have a canary in some
parts of the mine.

All three Circle bots appear to run exactly the same JVM. Presumably the
identical configuration is a mistake. (I was suspicious of this, so I
added a tell to regrtest.py.) Yet one out of three tests passes by chance:

CircleCI 0: (fail test_gc)
      [exec] == 2.7.1rc1 (, Mar 20 2017, 00:16:00)
      [exec] == [OpenJDK 64-Bit Server VM (Oracle Corporation)]
      [exec] == platform: java1.7.0_95
      [exec] == encodings: stdin=UTF-8, stdout=UTF-8, FS=None
      [exec] == locale: default=('en_US', 'UTF-8'), actual=(None, None)
CircleCI 1: (pass)
      [exec] == 2.7.1rc1 (, Mar 20 2017, 00:16:00)
      [exec] == [OpenJDK 64-Bit Server VM (Oracle Corporation)]
      [exec] == platform: java1.7.0_95
      [exec] == encodings: stdin=UTF-8, stdout=UTF-8, FS=None
      [exec] == locale: default=('en_US', 'UTF-8'), actual=(None, None)
CircleCI 2: (fail test_gc)
      [exec] == 2.7.1rc1 (, Mar 20 2017, 00:17:20)
      [exec] == [OpenJDK 64-Bit Server VM (Oracle Corporation)]
      [exec] == platform: java1.7.0_95
      [exec] == encodings: stdin=UTF-8, stdout=UTF-8, FS=None
      [exec] == locale: default=('en_US', 'UTF-8'), actual=(None, None)

Travis is configured to work with 3 genuinely different JVMs:

Travis 1: (pass)
      [exec] == 2.7.1rc1 (, Mar 20 2017, 00:17:49)
      [exec] == [Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)]
      [exec] == platform: java1.7.0_80
      [exec] == encodings: stdin=UTF-8, stdout=UTF-8, FS=None
      [exec] == locale: default=('en_US', 'UTF-8'), actual=(None, None)
Travis 2: (fail test_builtin with machine stack dump)
      [exec] == 2.7.1rc1 (, Mar 20 2017, 00:17:48)
      [exec] == [OpenJDK 64-Bit Server VM (Oracle Corporation)]
      [exec] == platform: java1.7.0_121
      [exec] == encodings: stdin=UTF-8, stdout=UTF-8, FS=None
      [exec] == locale: default=('en_US', 'UTF-8'), actual=(None, None)
Travis 3: (fail test_sort)
      [exec] == 2.7.1rc1 (, Mar 20 2017, 00:18:59)
      [exec] == [Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)]
      [exec] == platform: java1.8.0_92
      [exec] == encodings: stdin=UTF-8, stdout=UTF-8, FS=None
      [exec] == locale: default=('en_US', 'UTF-8'), actual=(None, None)

I left test_sort enabled because a fix seemed imminent. However, if I
were wholly consistent with what I preach here, I'd have made it an
expected failure in regrtest.py.

Jeff

Jeff Allen



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Jython-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Failing tests on GitHub

Jeff Allen-2
Update: all three Travis tests green now -- a great joint effort! Not
sure what fixed the openjdk7 core dump.

https://travis-ci.org/jythontools/jython/builds/213754331

I assume someone has disabled the Circle CI tests to work on them?

Jeff Allen

On 20/03/2017 07:31, Jeff Allen wrote:

> I have managed to achieve two green bots. So we have a canary in some
> parts of the mine.
>
> All three Circle bots appear to run exactly the same JVM. Presumably the
> identical configuration is a mistake. (I was suspicious of this, so I
> added a tell to regrtest.py.) Yet one out of three tests passes by chance:
>
> CircleCI 0: (fail test_gc)
>        [exec] == 2.7.1rc1 (, Mar 20 2017, 00:16:00)
>        [exec] == [OpenJDK 64-Bit Server VM (Oracle Corporation)]
>        [exec] == platform: java1.7.0_95
>        [exec] == encodings: stdin=UTF-8, stdout=UTF-8, FS=None
>        [exec] == locale: default=('en_US', 'UTF-8'), actual=(None, None)
> CircleCI 1: (pass)
>        [exec] == 2.7.1rc1 (, Mar 20 2017, 00:16:00)
>        [exec] == [OpenJDK 64-Bit Server VM (Oracle Corporation)]
>        [exec] == platform: java1.7.0_95
>        [exec] == encodings: stdin=UTF-8, stdout=UTF-8, FS=None
>        [exec] == locale: default=('en_US', 'UTF-8'), actual=(None, None)
> CircleCI 2: (fail test_gc)
>        [exec] == 2.7.1rc1 (, Mar 20 2017, 00:17:20)
>        [exec] == [OpenJDK 64-Bit Server VM (Oracle Corporation)]
>        [exec] == platform: java1.7.0_95
>        [exec] == encodings: stdin=UTF-8, stdout=UTF-8, FS=None
>        [exec] == locale: default=('en_US', 'UTF-8'), actual=(None, None)
>
> Travis is configured to work with 3 genuinely different JVMs:
>
> Travis 1: (pass)
>        [exec] == 2.7.1rc1 (, Mar 20 2017, 00:17:49)
>        [exec] == [Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)]
>        [exec] == platform: java1.7.0_80
>        [exec] == encodings: stdin=UTF-8, stdout=UTF-8, FS=None
>        [exec] == locale: default=('en_US', 'UTF-8'), actual=(None, None)
> Travis 2: (fail test_builtin with machine stack dump)
>        [exec] == 2.7.1rc1 (, Mar 20 2017, 00:17:48)
>        [exec] == [OpenJDK 64-Bit Server VM (Oracle Corporation)]
>        [exec] == platform: java1.7.0_121
>        [exec] == encodings: stdin=UTF-8, stdout=UTF-8, FS=None
>        [exec] == locale: default=('en_US', 'UTF-8'), actual=(None, None)
> Travis 3: (fail test_sort)
>        [exec] == 2.7.1rc1 (, Mar 20 2017, 00:18:59)
>        [exec] == [Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)]
>        [exec] == platform: java1.8.0_92
>        [exec] == encodings: stdin=UTF-8, stdout=UTF-8, FS=None
>        [exec] == locale: default=('en_US', 'UTF-8'), actual=(None, None)
>
> I left test_sort enabled because a fix seemed imminent. However, if I
> were wholly consistent with what I preach here, I'd have made it an
> expected failure in regrtest.py.
>
> Jeff
>
> Jeff Allen
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Jython-dev mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/jython-dev
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Jython-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-dev
Loading...