Explore integrating django-docker-box in some way?

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Explore integrating django-docker-box in some way?

Tom Forbes

Hello all,

I’ve been working on a docker-compose based alternative to django-box (imaginatively named django-docker-box) over the last month and it finally appears to be mostly complete.

For reference the tool is just a Dockerfile and a docker-compose definition that is able to run a complete test matrix of every supported Python and DB version. It’s as simple as docker-compose run sqlite. You can see a full test run (excluding oracle) here: https://travis-ci.com/orf/django-docker-box/builds/90167436

Florian suggested I create a thread here to gather feedback and discuss any potential future directions for the project, so here goes:

Firstly I’d like to know if there is any support for moving this under the Django project itself, maybe even as a replacement for django-box? I think the setup is pretty quick compared to django-box and is more flexible in terms of database version support as well as working with Oracle. I’d also really like some help improving Oracle support if anyone has the time!

Secondly is there any support for integrating this with our current Jenkins setup? I think it would be pretty neat to have parity between what runs on the CI and what we can run locally and have any improvements shared between both. Perhaps a full matrix run (which right now is 66 different environments) is out of the question but a smaller subset could be good?

Thirdly, and this is a bit wild, but what about using this to reduce the burden of running Jenkins by running the tests on a managed CI service like Travis CI? We would likely still need Jenkins due to issues with Oracle and running tests on Windows (unless https://github.com/django/django/pull/10259 works with Docker!), but we could offload some of the environments onto a third party service. Travis gives large OS projects like Django increased concurrency limits on their accounts so we could end up with pretty speedy test runs. Also with docker-compose switching between CI services (including Jenkins) would be very simple.

The repo is here: https://github.com/orf/django-docker-box.

Any feedback on these points or the project itself would be greatly appreciated,

Tom



--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFNZOJNFvp5ke_SzZtto2JEy%3DUsUSWFHByf-sA_ahUW9Vb_Brg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Explore integrating django-docker-box in some way?

Josh Smeaton
I'm sorry I haven't had the time to review or contribute yet, but I think it'll be a very useful project - especially for new contributors that might have a little docker experience. The current vagrant solution is heavy, does not work properly on windows and some linuxes, and isn't that easy to maintain or deploy. I'd be in favour of adding the docker files directly to django/django to minimise setup burden (DJANGO_PATH), and improving the contributing docs to show users how to test using docker.

One of the hardest things I found at sprints was getting development environments setup to effectively contribute - even using the docker-box project which I understand quite well. Anything we can do to improve that situation will be very beneficial.

I have fewer opinions about the official CI story, hopefully some of the infrastructure team can comment more on that. I think that replacing the ansible roles with a docker setup can have some definite improvements and open up CI tasks to a larger pool of people (anyone that can edit docker files), but it'd come with maintaining the host that runs docker (cleaning up images, dealing with disk space issues, etc).


On Monday, 5 November 2018 01:20:03 UTC+11, Tom Forbes wrote:

Hello all,

I’ve been working on a docker-compose based alternative to django-box (imaginatively named django-docker-box) over the last month and it finally appears to be mostly complete.

For reference the tool is just a Dockerfile and a docker-compose definition that is able to run a complete test matrix of every supported Python and DB version. It’s as simple as docker-compose run sqlite. You can see a full test run (excluding oracle) here: <a href="https://travis-ci.com/orf/django-docker-box/builds/90167436" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ftravis-ci.com%2Forf%2Fdjango-docker-box%2Fbuilds%2F90167436\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDOVycg-Gw0hn6KygDO3ZZcGoJpA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ftravis-ci.com%2Forf%2Fdjango-docker-box%2Fbuilds%2F90167436\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDOVycg-Gw0hn6KygDO3ZZcGoJpA&#39;;return true;">https://travis-ci.com/orf/django-docker-box/builds/90167436

Florian suggested I create a thread here to gather feedback and discuss any potential future directions for the project, so here goes:

Firstly I’d like to know if there is any support for moving this under the Django project itself, maybe even as a replacement for django-box? I think the setup is pretty quick compared to django-box and is more flexible in terms of database version support as well as working with Oracle. I’d also really like some help improving Oracle support if anyone has the time!

Secondly is there any support for integrating this with our current Jenkins setup? I think it would be pretty neat to have parity between what runs on the CI and what we can run locally and have any improvements shared between both. Perhaps a full matrix run (which right now is 66 different environments) is out of the question but a smaller subset could be good?

Thirdly, and this is a bit wild, but what about using this to reduce the burden of running Jenkins by running the tests on a managed CI service like Travis CI? We would likely still need Jenkins due to issues with Oracle and running tests on Windows (unless <a href="https://github.com/django/django/pull/10259" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango%2Fpull%2F10259\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFxI_a8zxeI8dScBv2L3pPh1TaNiA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango%2Fpull%2F10259\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFxI_a8zxeI8dScBv2L3pPh1TaNiA&#39;;return true;">https://github.com/django/django/pull/10259 works with Docker!), but we could offload some of the environments onto a third party service. Travis gives large OS projects like Django increased concurrency limits on their accounts so we could end up with pretty speedy test runs. Also with docker-compose switching between CI services (including Jenkins) would be very simple.

The repo is here: <a href="https://github.com/orf/django-docker-box" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Forf%2Fdjango-docker-box\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG8d-9Yk3zsBlVe79B14SMSJif77A&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Forf%2Fdjango-docker-box\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG8d-9Yk3zsBlVe79B14SMSJif77A&#39;;return true;">https://github.com/orf/django-docker-box.

Any feedback on these points or the project itself would be greatly appreciated,

Tom



--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Explore integrating django-docker-box in some way?

Tom Forbes

Thank you for the reply Josh. I didn’t anticipate any suggestions for including this inside core but off the back of your suggestion I’ve made a ticket here: https://code.djangoproject.com/ticket/30010.

I don’t think it should be complex at all to include this inside Django - it’s four or five new files at most. Hopefully this should improve the experience at sprints, however the current Dockerfile weighs in at 650+mb so the problem may switch from ‘it is hard to set up an environment’ to ‘it is hard to download one’!




On 5 November 2018 at 23:02:30, Josh Smeaton ([hidden email]) wrote:

I'm sorry I haven't had the time to review or contribute yet, but I think it'll be a very useful project - especially for new contributors that might have a little docker experience. The current vagrant solution is heavy, does not work properly on windows and some linuxes, and isn't that easy to maintain or deploy. I'd be in favour of adding the docker files directly to django/django to minimise setup burden (DJANGO_PATH), and improving the contributing docs to show users how to test using docker.

One of the hardest things I found at sprints was getting development environments setup to effectively contribute - even using the docker-box project which I understand quite well. Anything we can do to improve that situation will be very beneficial.

I have fewer opinions about the official CI story, hopefully some of the infrastructure team can comment more on that. I think that replacing the ansible roles with a docker setup can have some definite improvements and open up CI tasks to a larger pool of people (anyone that can edit docker files), but it'd come with maintaining the host that runs docker (cleaning up images, dealing with disk space issues, etc).


On Monday, 5 November 2018 01:20:03 UTC+11, Tom Forbes wrote:

Hello all,

I’ve been working on a docker-compose based alternative to django-box (imaginatively named django-docker-box) over the last month and it finally appears to be mostly complete.

For reference the tool is just a Dockerfile and a docker-compose definition that is able to run a complete test matrix of every supported Python and DB version. It’s as simple as docker-compose run sqlite. You can see a full test run (excluding oracle) here: <a href="https://travis-ci.com/orf/django-docker-box/builds/90167436" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ftravis-ci.com%2Forf%2Fdjango-docker-box%2Fbuilds%2F90167436\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDOVycg-Gw0hn6KygDO3ZZcGoJpA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ftravis-ci.com%2Forf%2Fdjango-docker-box%2Fbuilds%2F90167436\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDOVycg-Gw0hn6KygDO3ZZcGoJpA&#39;;return true;"> https://travis-ci.com/orf/django-docker-box/builds/90167436

Florian suggested I create a thread here to gather feedback and discuss any potential future directions for the project, so here goes:

Firstly I’d like to know if there is any support for moving this under the Django project itself, maybe even as a replacement for django-box? I think the setup is pretty quick compared to django-box and is more flexible in terms of database version support as well as working with Oracle. I’d also really like some help improving Oracle support if anyone has the time!

Secondly is there any support for integrating this with our current Jenkins setup? I think it would be pretty neat to have parity between what runs on the CI and what we can run locally and have any improvements shared between both. Perhaps a full matrix run (which right now is 66 different environments) is out of the question but a smaller subset could be good?

Thirdly, and this is a bit wild, but what about using this to reduce the burden of running Jenkins by running the tests on a managed CI service like Travis CI? We would likely still need Jenkins due to issues with Oracle and running tests on Windows (unless <a href="https://github.com/django/django/pull/10259" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango%2Fpull%2F10259\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFxI_a8zxeI8dScBv2L3pPh1TaNiA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango%2Fpull%2F10259\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFxI_a8zxeI8dScBv2L3pPh1TaNiA&#39;;return true;"> https://github.com/django/django/pull/10259 works with Docker!), but we could offload some of the environments onto a third party service. Travis gives large OS projects like Django increased concurrency limits on their accounts so we could end up with pretty speedy test runs. Also with docker-compose switching between CI services (including Jenkins) would be very simple.

The repo is here: <a href="https://github.com/orf/django-docker-box" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Forf%2Fdjango-docker-box\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG8d-9Yk3zsBlVe79B14SMSJif77A&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Forf%2Fdjango-docker-box\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG8d-9Yk3zsBlVe79B14SMSJif77A&#39;;return true;"> https://github.com/orf/django-docker-box.

Any feedback on these points or the project itself would be greatly appreciated,

Tom



--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFNZOJODt98cvOyaKzxOB3tUuMSUPW_5-D481Cj43BDw%2BdhYiA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Explore integrating django-docker-box in some way?

Josh Smeaton
Size of the image could definitely be a concern, especially at sprints where wifi speeds aren't always optimal. The django-box image is significantly larger though so it'd still be a net win. There are also optimisations that can be made to the image for reducing size over time, so I'd fully expect it to come down. I've spent a little bit of time trying to optimise a $work$ python docker file, I'll provide what I've got as an issue to possibly look at.

I see that the ticket has been accepted and I think that's a great step forward. I'd also like to hear from the infrastructure team what their thoughts on using docker over customised build environments would be.

Florian, Tim, Markus .. any thoughts? (Apologies, I've missed some, this list of names is from memory).

On Wednesday, 5 December 2018 10:39:16 UTC+11, Tom Forbes wrote:

Thank you for the reply Josh. I didn’t anticipate any suggestions for including this inside core but off the back of your suggestion I’ve made a ticket here: <a href="https://code.djangoproject.com/ticket/30010" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F30010\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGZYd9PksA8SDZQxBDl898efxSsZg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F30010\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGZYd9PksA8SDZQxBDl898efxSsZg&#39;;return true;">https://code.djangoproject.com/ticket/30010.

I don’t think it should be complex at all to include this inside Django - it’s four or five new files at most. Hopefully this should improve the experience at sprints, however the current Dockerfile weighs in at 650+mb so the problem may switch from ‘it is hard to set up an environment’ to ‘it is hard to download one’!




On 5 November 2018 at 23:02:30, Josh Smeaton (<a href="javascript:" target="_blank" gdf-obfuscated-mailto="vf2Cfb29BgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">josh.s...@...) wrote:

I'm sorry I haven't had the time to review or contribute yet, but I think it'll be a very useful project - especially for new contributors that might have a little docker experience. The current vagrant solution is heavy, does not work properly on windows and some linuxes, and isn't that easy to maintain or deploy. I'd be in favour of adding the docker files directly to django/django to minimise setup burden (DJANGO_PATH), and improving the contributing docs to show users how to test using docker.

One of the hardest things I found at sprints was getting development environments setup to effectively contribute - even using the docker-box project which I understand quite well. Anything we can do to improve that situation will be very beneficial.

I have fewer opinions about the official CI story, hopefully some of the infrastructure team can comment more on that. I think that replacing the ansible roles with a docker setup can have some definite improvements and open up CI tasks to a larger pool of people (anyone that can edit docker files), but it'd come with maintaining the host that runs docker (cleaning up images, dealing with disk space issues, etc).


On Monday, 5 November 2018 01:20:03 UTC+11, Tom Forbes wrote:

Hello all,

I’ve been working on a docker-compose based alternative to django-box (imaginatively named django-docker-box) over the last month and it finally appears to be mostly complete.

For reference the tool is just a Dockerfile and a docker-compose definition that is able to run a complete test matrix of every supported Python and DB version. It’s as simple as docker-compose run sqlite. You can see a full test run (excluding oracle) here: <a href="https://travis-ci.com/orf/django-docker-box/builds/90167436" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ftravis-ci.com%2Forf%2Fdjango-docker-box%2Fbuilds%2F90167436\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDOVycg-Gw0hn6KygDO3ZZcGoJpA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ftravis-ci.com%2Forf%2Fdjango-docker-box%2Fbuilds%2F90167436\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDOVycg-Gw0hn6KygDO3ZZcGoJpA&#39;;return true;"> https://travis-ci.com/orf/django-docker-box/builds/90167436

Florian suggested I create a thread here to gather feedback and discuss any potential future directions for the project, so here goes:

Firstly I’d like to know if there is any support for moving this under the Django project itself, maybe even as a replacement for django-box? I think the setup is pretty quick compared to django-box and is more flexible in terms of database version support as well as working with Oracle. I’d also really like some help improving Oracle support if anyone has the time!

Secondly is there any support for integrating this with our current Jenkins setup? I think it would be pretty neat to have parity between what runs on the CI and what we can run locally and have any improvements shared between both. Perhaps a full matrix run (which right now is 66 different environments) is out of the question but a smaller subset could be good?

Thirdly, and this is a bit wild, but what about using this to reduce the burden of running Jenkins by running the tests on a managed CI service like Travis CI? We would likely still need Jenkins due to issues with Oracle and running tests on Windows (unless <a href="https://github.com/django/django/pull/10259" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango%2Fpull%2F10259\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFxI_a8zxeI8dScBv2L3pPh1TaNiA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango%2Fpull%2F10259\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFxI_a8zxeI8dScBv2L3pPh1TaNiA&#39;;return true;"> https://github.com/django/django/pull/10259 works with Docker!), but we could offload some of the environments onto a third party service. Travis gives large OS projects like Django increased concurrency limits on their accounts so we could end up with pretty speedy test runs. Also with docker-compose switching between CI services (including Jenkins) would be very simple.

The repo is here: <a href="https://github.com/orf/django-docker-box" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Forf%2Fdjango-docker-box\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG8d-9Yk3zsBlVe79B14SMSJif77A&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Forf%2Fdjango-docker-box\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG8d-9Yk3zsBlVe79B14SMSJif77A&#39;;return true;"> https://github.com/orf/django-docker-box.

Any feedback on these points or the project itself would be greatly appreciated,

Tom



--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="vf2Cfb29BgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-develop...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="vf2Cfb29BgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-d...@googlegroups.com.
Visit this group at <a href="https://groups.google.com/group/django-developers" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;">https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;"> https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Explore integrating django-docker-box in some way?

Tom Forbes

To have this completely working at sprints without having everyone building their own local images we would need to have the Jenkins server use docker in some capacity. This would also require an official django account on Docker hub.

The pattern I’m using right now is that on every build we pull the django-ci:latest image (from my personal account). Docker uses this image as a cache automatically (preventing rebuilds). On any successful master build we push the new image to docker-hub, so subsequent builds can utilise it.

Then anyone wanting to speed up their bootstrapping can do docker-compose pull and automatically have the latest image available for running right away. We can make this smaller, for sure, but we can also suggest people download this beforehand (i.e at their hotel).

I don’t know how feasible this is but it’s also very easy to run a caching docker mirror (docker run -p 5000:5000 registry). Organizers could run this at large events and configuring docker to use a local mirror on the network is a one-line change for atendees.




On 4 December 2018 at 23:52:42, Josh Smeaton ([hidden email]) wrote:

Size of the image could definitely be a concern, especially at sprints where wifi speeds aren't always optimal. The django-box image is significantly larger though so it'd still be a net win. There are also optimisations that can be made to the image for reducing size over time, so I'd fully expect it to come down. I've spent a little bit of time trying to optimise a $work$ python docker file, I'll provide what I've got as an issue to possibly look at.

I see that the ticket has been accepted and I think that's a great step forward. I'd also like to hear from the infrastructure team what their thoughts on using docker over customised build environments would be.

Florian, Tim, Markus .. any thoughts? (Apologies, I've missed some, this list of names is from memory).

On Wednesday, 5 December 2018 10:39:16 UTC+11, Tom Forbes wrote:

Thank you for the reply Josh. I didn’t anticipate any suggestions for including this inside core but off the back of your suggestion I’ve made a ticket here: <a href="https://code.djangoproject.com/ticket/30010" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F30010\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGZYd9PksA8SDZQxBDl898efxSsZg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F30010\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGZYd9PksA8SDZQxBDl898efxSsZg&#39;;return true;"> https://code.djangoproject.com/ticket/30010.

I don’t think it should be complex at all to include this inside Django - it’s four or five new files at most. Hopefully this should improve the experience at sprints, however the current Dockerfile weighs in at 650+mb so the problem may switch from ‘it is hard to set up an environment’ to ‘it is hard to download one’!




On 5 November 2018 at 23:02:30, Josh Smeaton (<a href="" target="_blank" rel="nofollow" onmousedown="this.href=&#39;&#39;;return true;" onclick="this.href=&#39;&#39;;return true;">josh.s...@...) wrote:

I'm sorry I haven't had the time to review or contribute yet, but I think it'll be a very useful project - especially for new contributors that might have a little docker experience. The current vagrant solution is heavy, does not work properly on windows and some linuxes, and isn't that easy to maintain or deploy. I'd be in favour of adding the docker files directly to django/django to minimise setup burden (DJANGO_PATH), and improving the contributing docs to show users how to test using docker.

One of the hardest things I found at sprints was getting development environments setup to effectively contribute - even using the docker-box project which I understand quite well. Anything we can do to improve that situation will be very beneficial.

I have fewer opinions about the official CI story, hopefully some of the infrastructure team can comment more on that. I think that replacing the ansible roles with a docker setup can have some definite improvements and open up CI tasks to a larger pool of people (anyone that can edit docker files), but it'd come with maintaining the host that runs docker (cleaning up images, dealing with disk space issues, etc).


On Monday, 5 November 2018 01:20:03 UTC+11, Tom Forbes wrote:

Hello all,

I’ve been working on a docker-compose based alternative to django-box (imaginatively named django-docker-box) over the last month and it finally appears to be mostly complete.

For reference the tool is just a Dockerfile and a docker-compose definition that is able to run a complete test matrix of every supported Python and DB version. It’s as simple as docker-compose run sqlite. You can see a full test run (excluding oracle) here: <a href="https://travis-ci.com/orf/django-docker-box/builds/90167436" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ftravis-ci.com%2Forf%2Fdjango-docker-box%2Fbuilds%2F90167436\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDOVycg-Gw0hn6KygDO3ZZcGoJpA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ftravis-ci.com%2Forf%2Fdjango-docker-box%2Fbuilds%2F90167436\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDOVycg-Gw0hn6KygDO3ZZcGoJpA&#39;;return true;"> https://travis-ci.com/orf/django-docker-box/builds/90167436

Florian suggested I create a thread here to gather feedback and discuss any potential future directions for the project, so here goes:

Firstly I’d like to know if there is any support for moving this under the Django project itself, maybe even as a replacement for django-box? I think the setup is pretty quick compared to django-box and is more flexible in terms of database version support as well as working with Oracle. I’d also really like some help improving Oracle support if anyone has the time!

Secondly is there any support for integrating this with our current Jenkins setup? I think it would be pretty neat to have parity between what runs on the CI and what we can run locally and have any improvements shared between both. Perhaps a full matrix run (which right now is 66 different environments) is out of the question but a smaller subset could be good?

Thirdly, and this is a bit wild, but what about using this to reduce the burden of running Jenkins by running the tests on a managed CI service like Travis CI? We would likely still need Jenkins due to issues with Oracle and running tests on Windows (unless <a href="https://github.com/django/django/pull/10259" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango%2Fpull%2F10259\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFxI_a8zxeI8dScBv2L3pPh1TaNiA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango%2Fpull%2F10259\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFxI_a8zxeI8dScBv2L3pPh1TaNiA&#39;;return true;"> https://github.com/django/django/pull/10259 works with Docker!), but we could offload some of the environments onto a third party service. Travis gives large OS projects like Django increased concurrency limits on their accounts so we could end up with pretty speedy test runs. Also with docker-compose switching between CI services (including Jenkins) would be very simple.

The repo is here: <a href="https://github.com/orf/django-docker-box" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Forf%2Fdjango-docker-box\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG8d-9Yk3zsBlVe79B14SMSJif77A&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Forf%2Fdjango-docker-box\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG8d-9Yk3zsBlVe79B14SMSJif77A&#39;;return true;"> https://github.com/orf/django-docker-box.

Any feedback on these points or the project itself would be greatly appreciated,

Tom



--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="" target="_blank" rel="nofollow" onmousedown="this.href=&#39;&#39;;return true;" onclick="this.href=&#39;&#39;;return true;">django-develop...@googlegroups.com.
To post to this group, send email to <a href="" target="_blank" rel="nofollow" onmousedown="this.href=&#39;&#39;;return true;" onclick="this.href=&#39;&#39;;return true;">django-d...@googlegroups.com.
Visit this group at <a href="https://groups.google.com/group/django-developers" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;"> https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;"> https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFNZOJNuTGmmwFqCQwM77a15bOho2DeSiyVm0acL1Yy7ytj%3DLg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Explore integrating django-docker-box in some way?

Bruno A.
Just a note that as a newish contributor to Django, this django-docker-box is fantastic, it makes things much easier to setup.

Thank you to everyone involved!

On Wednesday, 5 December 2018 00:02:27 UTC, Tom Forbes wrote:

To have this completely working at sprints without having everyone building their own local images we would need to have the Jenkins server use docker in some capacity. This would also require an official django account on Docker hub.

The pattern I’m using right now is that on every build we pull the django-ci:latest image (from my personal account). Docker uses this image as a cache automatically (preventing rebuilds). On any successful master build we push the new image to docker-hub, so subsequent builds can utilise it.

Then anyone wanting to speed up their bootstrapping can do docker-compose pull and automatically have the latest image available for running right away. We can make this smaller, for sure, but we can also suggest people download this beforehand (i.e at their hotel).

I don’t know how feasible this is but it’s also very easy to run a caching docker mirror (docker run -p 5000:5000 registry). Organizers could run this at large events and configuring docker to use a local mirror on the network is a one-line change for atendees.




On 4 December 2018 at 23:52:42, Josh Smeaton (<a href="javascript:" target="_blank" gdf-obfuscated-mailto="OZMkWQG_BgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">josh....@...) wrote:

Size of the image could definitely be a concern, especially at sprints where wifi speeds aren't always optimal. The django-box image is significantly larger though so it'd still be a net win. There are also optimisations that can be made to the image for reducing size over time, so I'd fully expect it to come down. I've spent a little bit of time trying to optimise a $work$ python docker file, I'll provide what I've got as an issue to possibly look at.

I see that the ticket has been accepted and I think that's a great step forward. I'd also like to hear from the infrastructure team what their thoughts on using docker over customised build environments would be.

Florian, Tim, Markus .. any thoughts? (Apologies, I've missed some, this list of names is from memory).

On Wednesday, 5 December 2018 10:39:16 UTC+11, Tom Forbes wrote:

Thank you for the reply Josh. I didn’t anticipate any suggestions for including this inside core but off the back of your suggestion I’ve made a ticket here: <a href="https://code.djangoproject.com/ticket/30010" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F30010\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGZYd9PksA8SDZQxBDl898efxSsZg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F30010\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGZYd9PksA8SDZQxBDl898efxSsZg&#39;;return true;"> https://code.djangoproject.com/ticket/30010.

I don’t think it should be complex at all to include this inside Django - it’s four or five new files at most. Hopefully this should improve the experience at sprints, however the current Dockerfile weighs in at 650+mb so the problem may switch from ‘it is hard to set up an environment’ to ‘it is hard to download one’!




On 5 November 2018 at 23:02:30, Josh Smeaton ([hidden email]) wrote:

I'm sorry I haven't had the time to review or contribute yet, but I think it'll be a very useful project - especially for new contributors that might have a little docker experience. The current vagrant solution is heavy, does not work properly on windows and some linuxes, and isn't that easy to maintain or deploy. I'd be in favour of adding the docker files directly to django/django to minimise setup burden (DJANGO_PATH), and improving the contributing docs to show users how to test using docker.

One of the hardest things I found at sprints was getting development environments setup to effectively contribute - even using the docker-box project which I understand quite well. Anything we can do to improve that situation will be very beneficial.

I have fewer opinions about the official CI story, hopefully some of the infrastructure team can comment more on that. I think that replacing the ansible roles with a docker setup can have some definite improvements and open up CI tasks to a larger pool of people (anyone that can edit docker files), but it'd come with maintaining the host that runs docker (cleaning up images, dealing with disk space issues, etc).


On Monday, 5 November 2018 01:20:03 UTC+11, Tom Forbes wrote:

Hello all,

I’ve been working on a docker-compose based alternative to django-box (imaginatively named django-docker-box) over the last month and it finally appears to be mostly complete.

For reference the tool is just a Dockerfile and a docker-compose definition that is able to run a complete test matrix of every supported Python and DB version. It’s as simple as docker-compose run sqlite. You can see a full test run (excluding oracle) here: <a href="https://travis-ci.com/orf/django-docker-box/builds/90167436" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ftravis-ci.com%2Forf%2Fdjango-docker-box%2Fbuilds%2F90167436\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDOVycg-Gw0hn6KygDO3ZZcGoJpA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ftravis-ci.com%2Forf%2Fdjango-docker-box%2Fbuilds%2F90167436\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDOVycg-Gw0hn6KygDO3ZZcGoJpA&#39;;return true;"> https://travis-ci.com/orf/django-docker-box/builds/90167436

Florian suggested I create a thread here to gather feedback and discuss any potential future directions for the project, so here goes:

Firstly I’d like to know if there is any support for moving this under the Django project itself, maybe even as a replacement for django-box? I think the setup is pretty quick compared to django-box and is more flexible in terms of database version support as well as working with Oracle. I’d also really like some help improving Oracle support if anyone has the time!

Secondly is there any support for integrating this with our current Jenkins setup? I think it would be pretty neat to have parity between what runs on the CI and what we can run locally and have any improvements shared between both. Perhaps a full matrix run (which right now is 66 different environments) is out of the question but a smaller subset could be good?

Thirdly, and this is a bit wild, but what about using this to reduce the burden of running Jenkins by running the tests on a managed CI service like Travis CI? We would likely still need Jenkins due to issues with Oracle and running tests on Windows (unless <a href="https://github.com/django/django/pull/10259" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango%2Fpull%2F10259\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFxI_a8zxeI8dScBv2L3pPh1TaNiA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango%2Fpull%2F10259\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFxI_a8zxeI8dScBv2L3pPh1TaNiA&#39;;return true;"> https://github.com/django/django/pull/10259 works with Docker!), but we could offload some of the environments onto a third party service. Travis gives large OS projects like Django increased concurrency limits on their accounts so we could end up with pretty speedy test runs. Also with docker-compose switching between CI services (including Jenkins) would be very simple.

The repo is here: <a href="https://github.com/orf/django-docker-box" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Forf%2Fdjango-docker-box\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG8d-9Yk3zsBlVe79B14SMSJif77A&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Forf%2Fdjango-docker-box\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG8d-9Yk3zsBlVe79B14SMSJif77A&#39;;return true;"> https://github.com/orf/django-docker-box.

Any feedback on these points or the project itself would be greatly appreciated,

Tom



--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to [hidden email].
Visit this group at <a href="https://groups.google.com/group/django-developers" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;"> https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;"> https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="OZMkWQG_BgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-d...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="OZMkWQG_BgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-d...@googlegroups.com.
Visit this group at <a href="https://groups.google.com/group/django-developers" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;">https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;"> https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/6e7b2e3c-483a-43e9-94d5-bc772b1f02c6%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Explore integrating django-docker-box in some way?

Josh Smeaton
+1

On that note, how do people feel about archiving/deprecating https://github.com/django/django-box and updating the readme to point to the docker box project?

The last official version it supports is Django 2.1, but it was already on life support at that time, since half of the people who tried to use it couldn't.

On Friday, 4 October 2019 21:40:02 UTC+10, Bruno Alla wrote:
Just a note that as a newish contributor to Django, this django-docker-box is fantastic, it makes things much easier to setup.

Thank you to everyone involved!

On Wednesday, 5 December 2018 00:02:27 UTC, Tom Forbes wrote:

To have this completely working at sprints without having everyone building their own local images we would need to have the Jenkins server use docker in some capacity. This would also require an official django account on Docker hub.

The pattern I’m using right now is that on every build we pull the django-ci:latest image (from my personal account). Docker uses this image as a cache automatically (preventing rebuilds). On any successful master build we push the new image to docker-hub, so subsequent builds can utilise it.

Then anyone wanting to speed up their bootstrapping can do docker-compose pull and automatically have the latest image available for running right away. We can make this smaller, for sure, but we can also suggest people download this beforehand (i.e at their hotel).

I don’t know how feasible this is but it’s also very easy to run a caching docker mirror (docker run -p 5000:5000 registry). Organizers could run this at large events and configuring docker to use a local mirror on the network is a one-line change for atendees.




On 4 December 2018 at 23:52:42, Josh Smeaton ([hidden email]) wrote:

Size of the image could definitely be a concern, especially at sprints where wifi speeds aren't always optimal. The django-box image is significantly larger though so it'd still be a net win. There are also optimisations that can be made to the image for reducing size over time, so I'd fully expect it to come down. I've spent a little bit of time trying to optimise a $work$ python docker file, I'll provide what I've got as an issue to possibly look at.

I see that the ticket has been accepted and I think that's a great step forward. I'd also like to hear from the infrastructure team what their thoughts on using docker over customised build environments would be.

Florian, Tim, Markus .. any thoughts? (Apologies, I've missed some, this list of names is from memory).

On Wednesday, 5 December 2018 10:39:16 UTC+11, Tom Forbes wrote:

Thank you for the reply Josh. I didn’t anticipate any suggestions for including this inside core but off the back of your suggestion I’ve made a ticket here: <a href="https://code.djangoproject.com/ticket/30010" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F30010\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGZYd9PksA8SDZQxBDl898efxSsZg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F30010\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGZYd9PksA8SDZQxBDl898efxSsZg&#39;;return true;"> https://code.djangoproject.com/ticket/30010.

I don’t think it should be complex at all to include this inside Django - it’s four or five new files at most. Hopefully this should improve the experience at sprints, however the current Dockerfile weighs in at 650+mb so the problem may switch from ‘it is hard to set up an environment’ to ‘it is hard to download one’!




On 5 November 2018 at 23:02:30, Josh Smeaton ([hidden email]) wrote:

I'm sorry I haven't had the time to review or contribute yet, but I think it'll be a very useful project - especially for new contributors that might have a little docker experience. The current vagrant solution is heavy, does not work properly on windows and some linuxes, and isn't that easy to maintain or deploy. I'd be in favour of adding the docker files directly to django/django to minimise setup burden (DJANGO_PATH), and improving the contributing docs to show users how to test using docker.

One of the hardest things I found at sprints was getting development environments setup to effectively contribute - even using the docker-box project which I understand quite well. Anything we can do to improve that situation will be very beneficial.

I have fewer opinions about the official CI story, hopefully some of the infrastructure team can comment more on that. I think that replacing the ansible roles with a docker setup can have some definite improvements and open up CI tasks to a larger pool of people (anyone that can edit docker files), but it'd come with maintaining the host that runs docker (cleaning up images, dealing with disk space issues, etc).


On Monday, 5 November 2018 01:20:03 UTC+11, Tom Forbes wrote:

Hello all,

I’ve been working on a docker-compose based alternative to django-box (imaginatively named django-docker-box) over the last month and it finally appears to be mostly complete.

For reference the tool is just a Dockerfile and a docker-compose definition that is able to run a complete test matrix of every supported Python and DB version. It’s as simple as docker-compose run sqlite. You can see a full test run (excluding oracle) here: <a href="https://travis-ci.com/orf/django-docker-box/builds/90167436" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ftravis-ci.com%2Forf%2Fdjango-docker-box%2Fbuilds%2F90167436\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDOVycg-Gw0hn6KygDO3ZZcGoJpA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ftravis-ci.com%2Forf%2Fdjango-docker-box%2Fbuilds%2F90167436\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDOVycg-Gw0hn6KygDO3ZZcGoJpA&#39;;return true;"> https://travis-ci.com/orf/django-docker-box/builds/90167436

Florian suggested I create a thread here to gather feedback and discuss any potential future directions for the project, so here goes:

Firstly I’d like to know if there is any support for moving this under the Django project itself, maybe even as a replacement for django-box? I think the setup is pretty quick compared to django-box and is more flexible in terms of database version support as well as working with Oracle. I’d also really like some help improving Oracle support if anyone has the time!

Secondly is there any support for integrating this with our current Jenkins setup? I think it would be pretty neat to have parity between what runs on the CI and what we can run locally and have any improvements shared between both. Perhaps a full matrix run (which right now is 66 different environments) is out of the question but a smaller subset could be good?

Thirdly, and this is a bit wild, but what about using this to reduce the burden of running Jenkins by running the tests on a managed CI service like Travis CI? We would likely still need Jenkins due to issues with Oracle and running tests on Windows (unless <a href="https://github.com/django/django/pull/10259" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango%2Fpull%2F10259\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFxI_a8zxeI8dScBv2L3pPh1TaNiA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango%2Fpull%2F10259\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFxI_a8zxeI8dScBv2L3pPh1TaNiA&#39;;return true;"> https://github.com/django/django/pull/10259 works with Docker!), but we could offload some of the environments onto a third party service. Travis gives large OS projects like Django increased concurrency limits on their accounts so we could end up with pretty speedy test runs. Also with docker-compose switching between CI services (including Jenkins) would be very simple.

The repo is here: <a href="https://github.com/orf/django-docker-box" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Forf%2Fdjango-docker-box\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG8d-9Yk3zsBlVe79B14SMSJif77A&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Forf%2Fdjango-docker-box\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG8d-9Yk3zsBlVe79B14SMSJif77A&#39;;return true;"> https://github.com/orf/django-docker-box.

Any feedback on these points or the project itself would be greatly appreciated,

Tom



--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to [hidden email].
Visit this group at <a href="https://groups.google.com/group/django-developers" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;"> https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;"> https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at <a href="https://groups.google.com/group/django-developers" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;">https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;"> https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/ca0af91f-f78e-4b04-afd3-203740baa160%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Explore integrating django-docker-box in some way?

Carlton Gibson-3
Hi Bruno. Thanks for your post! I've come to take silence as positive feedback, but actual comments go further than you'd think. :) Tom was the driver here, so credit to him, and those who helped/tested. 

Hi Josh. Yes, it's probably time. (I imagine "use a VM running docker" is now likely the easiest route if you need to use a VM...) 
Thank you for your work maintaining the django-box project! Prior to Tom's work on django-docker-box, I was using the VM extensively, so much valued. 🎩

Kind Regards,

Carlton


On Friday, 4 October 2019 13:55:17 UTC+2, Josh Smeaton wrote:
+1

On that note, how do people feel about archiving/deprecating <a href="https://github.com/django/django-box" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango-box\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFEoT9z7W3njm97MrKe_XqMr_nnaA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango-box\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFEoT9z7W3njm97MrKe_XqMr_nnaA&#39;;return true;">https://github.com/django/django-box and updating the readme to point to the docker box project?

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/e351e72c-060c-4233-9461-028dd7624823%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Explore integrating django-docker-box in some way?

Tom Forbes
In reply to this post by Bruno A.
Thank you for the kind words Bruno! I'm glad it's helped you, if you have any suggestions for improvements then please open an issue on the repo or post a message here, I know it's not perfect. I would have replied earlier but this message didn't get delivered to me.

I'm biased, but I'd be +1 on archiving the old django-box. It's served us well, but unless someone is willing to spend some time updating it then it's going to confuse new users.

On Friday, 4 October 2019 12:40:02 UTC+1, Bruno Alla wrote:
Just a note that as a newish contributor to Django, this django-docker-box is fantastic, it makes things much easier to setup.

Thank you to everyone involved!

On Wednesday, 5 December 2018 00:02:27 UTC, Tom Forbes wrote:

To have this completely working at sprints without having everyone building their own local images we would need to have the Jenkins server use docker in some capacity. This would also require an official django account on Docker hub.

The pattern I’m using right now is that on every build we pull the django-ci:latest image (from my personal account). Docker uses this image as a cache automatically (preventing rebuilds). On any successful master build we push the new image to docker-hub, so subsequent builds can utilise it.

Then anyone wanting to speed up their bootstrapping can do docker-compose pull and automatically have the latest image available for running right away. We can make this smaller, for sure, but we can also suggest people download this beforehand (i.e at their hotel).

I don’t know how feasible this is but it’s also very easy to run a caching docker mirror (docker run -p 5000:5000 registry). Organizers could run this at large events and configuring docker to use a local mirror on the network is a one-line change for atendees.




On 4 December 2018 at 23:52:42, Josh Smeaton ([hidden email]) wrote:

Size of the image could definitely be a concern, especially at sprints where wifi speeds aren't always optimal. The django-box image is significantly larger though so it'd still be a net win. There are also optimisations that can be made to the image for reducing size over time, so I'd fully expect it to come down. I've spent a little bit of time trying to optimise a $work$ python docker file, I'll provide what I've got as an issue to possibly look at.

I see that the ticket has been accepted and I think that's a great step forward. I'd also like to hear from the infrastructure team what their thoughts on using docker over customised build environments would be.

Florian, Tim, Markus .. any thoughts? (Apologies, I've missed some, this list of names is from memory).

On Wednesday, 5 December 2018 10:39:16 UTC+11, Tom Forbes wrote:

Thank you for the reply Josh. I didn’t anticipate any suggestions for including this inside core but off the back of your suggestion I’ve made a ticket here: <a href="https://code.djangoproject.com/ticket/30010" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F30010\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGZYd9PksA8SDZQxBDl898efxSsZg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F30010\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGZYd9PksA8SDZQxBDl898efxSsZg&#39;;return true;"> https://code.djangoproject.com/ticket/30010.

I don’t think it should be complex at all to include this inside Django - it’s four or five new files at most. Hopefully this should improve the experience at sprints, however the current Dockerfile weighs in at 650+mb so the problem may switch from ‘it is hard to set up an environment’ to ‘it is hard to download one’!




On 5 November 2018 at 23:02:30, Josh Smeaton ([hidden email]) wrote:

I'm sorry I haven't had the time to review or contribute yet, but I think it'll be a very useful project - especially for new contributors that might have a little docker experience. The current vagrant solution is heavy, does not work properly on windows and some linuxes, and isn't that easy to maintain or deploy. I'd be in favour of adding the docker files directly to django/django to minimise setup burden (DJANGO_PATH), and improving the contributing docs to show users how to test using docker.

One of the hardest things I found at sprints was getting development environments setup to effectively contribute - even using the docker-box project which I understand quite well. Anything we can do to improve that situation will be very beneficial.

I have fewer opinions about the official CI story, hopefully some of the infrastructure team can comment more on that. I think that replacing the ansible roles with a docker setup can have some definite improvements and open up CI tasks to a larger pool of people (anyone that can edit docker files), but it'd come with maintaining the host that runs docker (cleaning up images, dealing with disk space issues, etc).


On Monday, 5 November 2018 01:20:03 UTC+11, Tom Forbes wrote:

Hello all,

I’ve been working on a docker-compose based alternative to django-box (imaginatively named django-docker-box) over the last month and it finally appears to be mostly complete.

For reference the tool is just a Dockerfile and a docker-compose definition that is able to run a complete test matrix of every supported Python and DB version. It’s as simple as docker-compose run sqlite. You can see a full test run (excluding oracle) here: <a href="https://travis-ci.com/orf/django-docker-box/builds/90167436" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ftravis-ci.com%2Forf%2Fdjango-docker-box%2Fbuilds%2F90167436\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDOVycg-Gw0hn6KygDO3ZZcGoJpA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ftravis-ci.com%2Forf%2Fdjango-docker-box%2Fbuilds%2F90167436\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDOVycg-Gw0hn6KygDO3ZZcGoJpA&#39;;return true;"> https://travis-ci.com/orf/django-docker-box/builds/90167436

Florian suggested I create a thread here to gather feedback and discuss any potential future directions for the project, so here goes:

Firstly I’d like to know if there is any support for moving this under the Django project itself, maybe even as a replacement for django-box? I think the setup is pretty quick compared to django-box and is more flexible in terms of database version support as well as working with Oracle. I’d also really like some help improving Oracle support if anyone has the time!

Secondly is there any support for integrating this with our current Jenkins setup? I think it would be pretty neat to have parity between what runs on the CI and what we can run locally and have any improvements shared between both. Perhaps a full matrix run (which right now is 66 different environments) is out of the question but a smaller subset could be good?

Thirdly, and this is a bit wild, but what about using this to reduce the burden of running Jenkins by running the tests on a managed CI service like Travis CI? We would likely still need Jenkins due to issues with Oracle and running tests on Windows (unless <a href="https://github.com/django/django/pull/10259" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango%2Fpull%2F10259\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFxI_a8zxeI8dScBv2L3pPh1TaNiA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango%2Fpull%2F10259\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFxI_a8zxeI8dScBv2L3pPh1TaNiA&#39;;return true;"> https://github.com/django/django/pull/10259 works with Docker!), but we could offload some of the environments onto a third party service. Travis gives large OS projects like Django increased concurrency limits on their accounts so we could end up with pretty speedy test runs. Also with docker-compose switching between CI services (including Jenkins) would be very simple.

The repo is here: <a href="https://github.com/orf/django-docker-box" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Forf%2Fdjango-docker-box\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG8d-9Yk3zsBlVe79B14SMSJif77A&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Forf%2Fdjango-docker-box\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG8d-9Yk3zsBlVe79B14SMSJif77A&#39;;return true;"> https://github.com/orf/django-docker-box.

Any feedback on these points or the project itself would be greatly appreciated,

Tom



--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to [hidden email].
Visit this group at <a href="https://groups.google.com/group/django-developers" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;"> https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;"> https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at <a href="https://groups.google.com/group/django-developers" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;">https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;"> https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/04b7db46-f6d4-4f60-9e28-4a4b41f84cad%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Explore integrating django-docker-box in some way?

Adam Johnson-2
+1 to archiving django-box

On Tue, 8 Oct 2019 at 11:01, Tom Forbes <[hidden email]> wrote:
Thank you for the kind words Bruno! I'm glad it's helped you, if you have any suggestions for improvements then please open an issue on the repo or post a message here, I know it's not perfect. I would have replied earlier but this message didn't get delivered to me.

I'm biased, but I'd be +1 on archiving the old django-box. It's served us well, but unless someone is willing to spend some time updating it then it's going to confuse new users.

On Friday, 4 October 2019 12:40:02 UTC+1, Bruno Alla wrote:
Just a note that as a newish contributor to Django, this django-docker-box is fantastic, it makes things much easier to setup.

Thank you to everyone involved!

On Wednesday, 5 December 2018 00:02:27 UTC, Tom Forbes wrote:

To have this completely working at sprints without having everyone building their own local images we would need to have the Jenkins server use docker in some capacity. This would also require an official django account on Docker hub.

The pattern I’m using right now is that on every build we pull the django-ci:latest image (from my personal account). Docker uses this image as a cache automatically (preventing rebuilds). On any successful master build we push the new image to docker-hub, so subsequent builds can utilise it.

Then anyone wanting to speed up their bootstrapping can do docker-compose pull and automatically have the latest image available for running right away. We can make this smaller, for sure, but we can also suggest people download this beforehand (i.e at their hotel).

I don’t know how feasible this is but it’s also very easy to run a caching docker mirror (docker run -p 5000:5000 registry). Organizers could run this at large events and configuring docker to use a local mirror on the network is a one-line change for atendees.




On 4 December 2018 at 23:52:42, Josh Smeaton ([hidden email]) wrote:

Size of the image could definitely be a concern, especially at sprints where wifi speeds aren't always optimal. The django-box image is significantly larger though so it'd still be a net win. There are also optimisations that can be made to the image for reducing size over time, so I'd fully expect it to come down. I've spent a little bit of time trying to optimise a $work$ python docker file, I'll provide what I've got as an issue to possibly look at.

I see that the ticket has been accepted and I think that's a great step forward. I'd also like to hear from the infrastructure team what their thoughts on using docker over customised build environments would be.

Florian, Tim, Markus .. any thoughts? (Apologies, I've missed some, this list of names is from memory).

On Wednesday, 5 December 2018 10:39:16 UTC+11, Tom Forbes wrote:

Thank you for the reply Josh. I didn’t anticipate any suggestions for including this inside core but off the back of your suggestion I’ve made a ticket here: https://code.djangoproject.com/ticket/30010.

I don’t think it should be complex at all to include this inside Django - it’s four or five new files at most. Hopefully this should improve the experience at sprints, however the current Dockerfile weighs in at 650+mb so the problem may switch from ‘it is hard to set up an environment’ to ‘it is hard to download one’!




On 5 November 2018 at 23:02:30, Josh Smeaton ([hidden email]) wrote:

I'm sorry I haven't had the time to review or contribute yet, but I think it'll be a very useful project - especially for new contributors that might have a little docker experience. The current vagrant solution is heavy, does not work properly on windows and some linuxes, and isn't that easy to maintain or deploy. I'd be in favour of adding the docker files directly to django/django to minimise setup burden (DJANGO_PATH), and improving the contributing docs to show users how to test using docker.

One of the hardest things I found at sprints was getting development environments setup to effectively contribute - even using the docker-box project which I understand quite well. Anything we can do to improve that situation will be very beneficial.

I have fewer opinions about the official CI story, hopefully some of the infrastructure team can comment more on that. I think that replacing the ansible roles with a docker setup can have some definite improvements and open up CI tasks to a larger pool of people (anyone that can edit docker files), but it'd come with maintaining the host that runs docker (cleaning up images, dealing with disk space issues, etc).


On Monday, 5 November 2018 01:20:03 UTC+11, Tom Forbes wrote:

Hello all,

I’ve been working on a docker-compose based alternative to django-box (imaginatively named django-docker-box) over the last month and it finally appears to be mostly complete.

For reference the tool is just a Dockerfile and a docker-compose definition that is able to run a complete test matrix of every supported Python and DB version. It’s as simple as docker-compose run sqlite. You can see a full test run (excluding oracle) here: https://travis-ci.com/orf/django-docker-box/builds/90167436

Florian suggested I create a thread here to gather feedback and discuss any potential future directions for the project, so here goes:

Firstly I’d like to know if there is any support for moving this under the Django project itself, maybe even as a replacement for django-box? I think the setup is pretty quick compared to django-box and is more flexible in terms of database version support as well as working with Oracle. I’d also really like some help improving Oracle support if anyone has the time!

Secondly is there any support for integrating this with our current Jenkins setup? I think it would be pretty neat to have parity between what runs on the CI and what we can run locally and have any improvements shared between both. Perhaps a full matrix run (which right now is 66 different environments) is out of the question but a smaller subset could be good?

Thirdly, and this is a bit wild, but what about using this to reduce the burden of running Jenkins by running the tests on a managed CI service like Travis CI? We would likely still need Jenkins due to issues with Oracle and running tests on Windows (unless https://github.com/django/django/pull/10259 works with Docker!), but we could offload some of the environments onto a third party service. Travis gives large OS projects like Django increased concurrency limits on their accounts so we could end up with pretty speedy test runs. Also with docker-compose switching between CI services (including Jenkins) would be very simple.

The repo is here: https://github.com/orf/django-docker-box.

Any feedback on these points or the project itself would be greatly appreciated,

Tom



--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/04b7db46-f6d4-4f60-9e28-4a4b41f84cad%40googlegroups.com.


--
Adam

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMyDDM1Hc3uLda5XfVzRU%3DN18SyTOCcc2f7UkR5jomW%2B9jFakQ%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Explore integrating django-docker-box in some way?

Josh Smeaton
I was going to archive the repo, but it seems I don't have the necessary permissions. Carlton, do you?

On Tuesday, 8 October 2019 21:45:22 UTC+11, Adam Johnson wrote:
+1 to archiving django-box

On Tue, 8 Oct 2019 at 11:01, Tom Forbes <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="h1jQvyYuDAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">t...@...> wrote:
Thank you for the kind words Bruno! I'm glad it's helped you, if you have any suggestions for improvements then please open an issue on the repo or post a message here, I know it's not perfect. I would have replied earlier but this message didn't get delivered to me.

I'm biased, but I'd be +1 on archiving the old django-box. It's served us well, but unless someone is willing to spend some time updating it then it's going to confuse new users.

On Friday, 4 October 2019 12:40:02 UTC+1, Bruno Alla wrote:
Just a note that as a newish contributor to Django, this django-docker-box is fantastic, it makes things much easier to setup.

Thank you to everyone involved!

On Wednesday, 5 December 2018 00:02:27 UTC, Tom Forbes wrote:

To have this completely working at sprints without having everyone building their own local images we would need to have the Jenkins server use docker in some capacity. This would also require an official django account on Docker hub.

The pattern I’m using right now is that on every build we pull the django-ci:latest image (from my personal account). Docker uses this image as a cache automatically (preventing rebuilds). On any successful master build we push the new image to docker-hub, so subsequent builds can utilise it.

Then anyone wanting to speed up their bootstrapping can do docker-compose pull and automatically have the latest image available for running right away. We can make this smaller, for sure, but we can also suggest people download this beforehand (i.e at their hotel).

I don’t know how feasible this is but it’s also very easy to run a caching docker mirror (docker run -p 5000:5000 registry). Organizers could run this at large events and configuring docker to use a local mirror on the network is a one-line change for atendees.




On 4 December 2018 at 23:52:42, Josh Smeaton ([hidden email]) wrote:

Size of the image could definitely be a concern, especially at sprints where wifi speeds aren't always optimal. The django-box image is significantly larger though so it'd still be a net win. There are also optimisations that can be made to the image for reducing size over time, so I'd fully expect it to come down. I've spent a little bit of time trying to optimise a $work$ python docker file, I'll provide what I've got as an issue to possibly look at.

I see that the ticket has been accepted and I think that's a great step forward. I'd also like to hear from the infrastructure team what their thoughts on using docker over customised build environments would be.

Florian, Tim, Markus .. any thoughts? (Apologies, I've missed some, this list of names is from memory).

On Wednesday, 5 December 2018 10:39:16 UTC+11, Tom Forbes wrote:

Thank you for the reply Josh. I didn’t anticipate any suggestions for including this inside core but off the back of your suggestion I’ve made a ticket here: <a href="https://code.djangoproject.com/ticket/30010" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F30010\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGZYd9PksA8SDZQxBDl898efxSsZg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fcode.djangoproject.com%2Fticket%2F30010\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGZYd9PksA8SDZQxBDl898efxSsZg&#39;;return true;"> https://code.djangoproject.com/ticket/30010.

I don’t think it should be complex at all to include this inside Django - it’s four or five new files at most. Hopefully this should improve the experience at sprints, however the current Dockerfile weighs in at 650+mb so the problem may switch from ‘it is hard to set up an environment’ to ‘it is hard to download one’!




On 5 November 2018 at 23:02:30, Josh Smeaton ([hidden email]) wrote:

I'm sorry I haven't had the time to review or contribute yet, but I think it'll be a very useful project - especially for new contributors that might have a little docker experience. The current vagrant solution is heavy, does not work properly on windows and some linuxes, and isn't that easy to maintain or deploy. I'd be in favour of adding the docker files directly to django/django to minimise setup burden (DJANGO_PATH), and improving the contributing docs to show users how to test using docker.

One of the hardest things I found at sprints was getting development environments setup to effectively contribute - even using the docker-box project which I understand quite well. Anything we can do to improve that situation will be very beneficial.

I have fewer opinions about the official CI story, hopefully some of the infrastructure team can comment more on that. I think that replacing the ansible roles with a docker setup can have some definite improvements and open up CI tasks to a larger pool of people (anyone that can edit docker files), but it'd come with maintaining the host that runs docker (cleaning up images, dealing with disk space issues, etc).


On Monday, 5 November 2018 01:20:03 UTC+11, Tom Forbes wrote:

Hello all,

I’ve been working on a docker-compose based alternative to django-box (imaginatively named django-docker-box) over the last month and it finally appears to be mostly complete.

For reference the tool is just a Dockerfile and a docker-compose definition that is able to run a complete test matrix of every supported Python and DB version. It’s as simple as docker-compose run sqlite. You can see a full test run (excluding oracle) here: <a href="https://travis-ci.com/orf/django-docker-box/builds/90167436" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ftravis-ci.com%2Forf%2Fdjango-docker-box%2Fbuilds%2F90167436\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDOVycg-Gw0hn6KygDO3ZZcGoJpA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ftravis-ci.com%2Forf%2Fdjango-docker-box%2Fbuilds%2F90167436\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEDOVycg-Gw0hn6KygDO3ZZcGoJpA&#39;;return true;"> https://travis-ci.com/orf/django-docker-box/builds/90167436

Florian suggested I create a thread here to gather feedback and discuss any potential future directions for the project, so here goes:

Firstly I’d like to know if there is any support for moving this under the Django project itself, maybe even as a replacement for django-box? I think the setup is pretty quick compared to django-box and is more flexible in terms of database version support as well as working with Oracle. I’d also really like some help improving Oracle support if anyone has the time!

Secondly is there any support for integrating this with our current Jenkins setup? I think it would be pretty neat to have parity between what runs on the CI and what we can run locally and have any improvements shared between both. Perhaps a full matrix run (which right now is 66 different environments) is out of the question but a smaller subset could be good?

Thirdly, and this is a bit wild, but what about using this to reduce the burden of running Jenkins by running the tests on a managed CI service like Travis CI? We would likely still need Jenkins due to issues with Oracle and running tests on Windows (unless <a href="https://github.com/django/django/pull/10259" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango%2Fpull%2F10259\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFxI_a8zxeI8dScBv2L3pPh1TaNiA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fdjango%2Fdjango%2Fpull%2F10259\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFxI_a8zxeI8dScBv2L3pPh1TaNiA&#39;;return true;"> https://github.com/django/django/pull/10259 works with Docker!), but we could offload some of the environments onto a third party service. Travis gives large OS projects like Django increased concurrency limits on their accounts so we could end up with pretty speedy test runs. Also with docker-compose switching between CI services (including Jenkins) would be very simple.

The repo is here: <a href="https://github.com/orf/django-docker-box" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Forf%2Fdjango-docker-box\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG8d-9Yk3zsBlVe79B14SMSJif77A&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Forf%2Fdjango-docker-box\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG8d-9Yk3zsBlVe79B14SMSJif77A&#39;;return true;"> https://github.com/orf/django-docker-box.

Any feedback on these points or the project itself would be greatly appreciated,

Tom



--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to [hidden email].
Visit this group at <a href="https://groups.google.com/group/django-developers" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;"> https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;"> https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at <a href="https://groups.google.com/group/django-developers" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/django-developers&#39;;return true;">https://groups.google.com/group/django-developers.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;"> https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="h1jQvyYuDAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">django-d...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/django-developers/04b7db46-f6d4-4f60-9e28-4a4b41f84cad%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/django-developers/04b7db46-f6d4-4f60-9e28-4a4b41f84cad%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/django-developers/04b7db46-f6d4-4f60-9e28-4a4b41f84cad%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/django-developers/04b7db46-f6d4-4f60-9e28-4a4b41f84cad%40googlegroups.com.


--
Adam

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/90f70b0e-9082-489c-ae0a-98341c917a48%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Explore integrating django-docker-box in some way?

Nick Sarbicki
Just out of interest, Tom, have you tried using Alpine as the base? Or is there a reason to avoid it?

I usually find once you introduce all the dependencies for django it doesn't make a huge difference but it might shave some of the weight off if we're worried about image size.


- Nick


On Tue, Oct 8, 2019 at 11:52 AM Josh Smeaton <[hidden email]> wrote:
I was going to archive the repo, but it seems I don't have the necessary permissions. Carlton, do you?

On Tuesday, 8 October 2019 21:45:22 UTC+11, Adam Johnson wrote:
+1 to archiving django-box

On Tue, 8 Oct 2019 at 11:01, Tom Forbes <[hidden email]> wrote:
Thank you for the kind words Bruno! I'm glad it's helped you, if you have any suggestions for improvements then please open an issue on the repo or post a message here, I know it's not perfect. I would have replied earlier but this message didn't get delivered to me.

I'm biased, but I'd be +1 on archiving the old django-box. It's served us well, but unless someone is willing to spend some time updating it then it's going to confuse new users.

On Friday, 4 October 2019 12:40:02 UTC+1, Bruno Alla wrote:
Just a note that as a newish contributor to Django, this django-docker-box is fantastic, it makes things much easier to setup.

Thank you to everyone involved!

On Wednesday, 5 December 2018 00:02:27 UTC, Tom Forbes wrote:

To have this completely working at sprints without having everyone building their own local images we would need to have the Jenkins server use docker in some capacity. This would also require an official django account on Docker hub.

The pattern I’m using right now is that on every build we pull the django-ci:latest image (from my personal account). Docker uses this image as a cache automatically (preventing rebuilds). On any successful master build we push the new image to docker-hub, so subsequent builds can utilise it.

Then anyone wanting to speed up their bootstrapping can do docker-compose pull and automatically have the latest image available for running right away. We can make this smaller, for sure, but we can also suggest people download this beforehand (i.e at their hotel).

I don’t know how feasible this is but it’s also very easy to run a caching docker mirror (docker run -p 5000:5000 registry). Organizers could run this at large events and configuring docker to use a local mirror on the network is a one-line change for atendees.




On 4 December 2018 at 23:52:42, Josh Smeaton ([hidden email]) wrote:

Size of the image could definitely be a concern, especially at sprints where wifi speeds aren't always optimal. The django-box image is significantly larger though so it'd still be a net win. There are also optimisations that can be made to the image for reducing size over time, so I'd fully expect it to come down. I've spent a little bit of time trying to optimise a $work$ python docker file, I'll provide what I've got as an issue to possibly look at.

I see that the ticket has been accepted and I think that's a great step forward. I'd also like to hear from the infrastructure team what their thoughts on using docker over customised build environments would be.

Florian, Tim, Markus .. any thoughts? (Apologies, I've missed some, this list of names is from memory).

On Wednesday, 5 December 2018 10:39:16 UTC+11, Tom Forbes wrote:

Thank you for the reply Josh. I didn’t anticipate any suggestions for including this inside core but off the back of your suggestion I’ve made a ticket here: https://code.djangoproject.com/ticket/30010.

I don’t think it should be complex at all to include this inside Django - it’s four or five new files at most. Hopefully this should improve the experience at sprints, however the current Dockerfile weighs in at 650+mb so the problem may switch from ‘it is hard to set up an environment’ to ‘it is hard to download one’!




On 5 November 2018 at 23:02:30, Josh Smeaton ([hidden email]) wrote:

I'm sorry I haven't had the time to review or contribute yet, but I think it'll be a very useful project - especially for new contributors that might have a little docker experience. The current vagrant solution is heavy, does not work properly on windows and some linuxes, and isn't that easy to maintain or deploy. I'd be in favour of adding the docker files directly to django/django to minimise setup burden (DJANGO_PATH), and improving the contributing docs to show users how to test using docker.

One of the hardest things I found at sprints was getting development environments setup to effectively contribute - even using the docker-box project which I understand quite well. Anything we can do to improve that situation will be very beneficial.

I have fewer opinions about the official CI story, hopefully some of the infrastructure team can comment more on that. I think that replacing the ansible roles with a docker setup can have some definite improvements and open up CI tasks to a larger pool of people (anyone that can edit docker files), but it'd come with maintaining the host that runs docker (cleaning up images, dealing with disk space issues, etc).


On Monday, 5 November 2018 01:20:03 UTC+11, Tom Forbes wrote:

Hello all,

I’ve been working on a docker-compose based alternative to django-box (imaginatively named django-docker-box) over the last month and it finally appears to be mostly complete.

For reference the tool is just a Dockerfile and a docker-compose definition that is able to run a complete test matrix of every supported Python and DB version. It’s as simple as docker-compose run sqlite. You can see a full test run (excluding oracle) here: https://travis-ci.com/orf/django-docker-box/builds/90167436

Florian suggested I create a thread here to gather feedback and discuss any potential future directions for the project, so here goes:

Firstly I’d like to know if there is any support for moving this under the Django project itself, maybe even as a replacement for django-box? I think the setup is pretty quick compared to django-box and is more flexible in terms of database version support as well as working with Oracle. I’d also really like some help improving Oracle support if anyone has the time!

Secondly is there any support for integrating this with our current Jenkins setup? I think it would be pretty neat to have parity between what runs on the CI and what we can run locally and have any improvements shared between both. Perhaps a full matrix run (which right now is 66 different environments) is out of the question but a smaller subset could be good?

Thirdly, and this is a bit wild, but what about using this to reduce the burden of running Jenkins by running the tests on a managed CI service like Travis CI? We would likely still need Jenkins due to issues with Oracle and running tests on Windows (unless https://github.com/django/django/pull/10259 works with Docker!), but we could offload some of the environments onto a third party service. Travis gives large OS projects like Django increased concurrency limits on their accounts so we could end up with pretty speedy test runs. Also with docker-compose switching between CI services (including Jenkins) would be very simple.

The repo is here: https://github.com/orf/django-docker-box.

Any feedback on these points or the project itself would be greatly appreciated,

Tom



--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/04b7db46-f6d4-4f60-9e28-4a4b41f84cad%40googlegroups.com.


--
Adam

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/90f70b0e-9082-489c-ae0a-98341c917a48%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAGuvt93zYniq5uL7me-Pa4D%3DnEJb0aox0PcfheVMqOFFFWRwzQ%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Explore integrating django-docker-box in some way?

Carlton Gibson-3
In reply to this post by Josh Smeaton
I’ll have a look.. 👀 

On Tue, 8 Oct 2019 at 12:52, Josh Smeaton <[hidden email]> wrote:
I was going to archive the repo, but it seems I don't have the necessary permissions. Carlton, do you?

On Tuesday, 8 October 2019 21:45:22 UTC+11, Adam Johnson wrote:
+1 to archiving django-box

On Tue, 8 Oct 2019 at 11:01, Tom Forbes <[hidden email]> wrote:
Thank you for the kind words Bruno! I'm glad it's helped you, if you have any suggestions for improvements then please open an issue on the repo or post a message here, I know it's not perfect. I would have replied earlier but this message didn't get delivered to me.

I'm biased, but I'd be +1 on archiving the old django-box. It's served us well, but unless someone is willing to spend some time updating it then it's going to confuse new users.

On Friday, 4 October 2019 12:40:02 UTC+1, Bruno Alla wrote:
Just a note that as a newish contributor to Django, this django-docker-box is fantastic, it makes things much easier to setup.

Thank you to everyone involved!

On Wednesday, 5 December 2018 00:02:27 UTC, Tom Forbes wrote:

To have this completely working at sprints without having everyone building their own local images we would need to have the Jenkins server use docker in some capacity. This would also require an official django account on Docker hub.

The pattern I’m using right now is that on every build we pull the django-ci:latest image (from my personal account). Docker uses this image as a cache automatically (preventing rebuilds). On any successful master build we push the new image to docker-hub, so subsequent builds can utilise it.

Then anyone wanting to speed up their bootstrapping can do docker-compose pull and automatically have the latest image available for running right away. We can make this smaller, for sure, but we can also suggest people download this beforehand (i.e at their hotel).

I don’t know how feasible this is but it’s also very easy to run a caching docker mirror (docker run -p 5000:5000 registry). Organizers could run this at large events and configuring docker to use a local mirror on the network is a one-line change for atendees.




On 4 December 2018 at 23:52:42, Josh Smeaton ([hidden email]) wrote:

Size of the image could definitely be a concern, especially at sprints where wifi speeds aren't always optimal. The django-box image is significantly larger though so it'd still be a net win. There are also optimisations that can be made to the image for reducing size over time, so I'd fully expect it to come down. I've spent a little bit of time trying to optimise a $work$ python docker file, I'll provide what I've got as an issue to possibly look at.

I see that the ticket has been accepted and I think that's a great step forward. I'd also like to hear from the infrastructure team what their thoughts on using docker over customised build environments would be.

Florian, Tim, Markus .. any thoughts? (Apologies, I've missed some, this list of names is from memory).

On Wednesday, 5 December 2018 10:39:16 UTC+11, Tom Forbes wrote:

Thank you for the reply Josh. I didn’t anticipate any suggestions for including this inside core but off the back of your suggestion I’ve made a ticket here: https://code.djangoproject.com/ticket/30010.

I don’t think it should be complex at all to include this inside Django - it’s four or five new files at most. Hopefully this should improve the experience at sprints, however the current Dockerfile weighs in at 650+mb so the problem may switch from ‘it is hard to set up an environment’ to ‘it is hard to download one’!




On 5 November 2018 at 23:02:30, Josh Smeaton ([hidden email]) wrote:

I'm sorry I haven't had the time to review or contribute yet, but I think it'll be a very useful project - especially for new contributors that might have a little docker experience. The current vagrant solution is heavy, does not work properly on windows and some linuxes, and isn't that easy to maintain or deploy. I'd be in favour of adding the docker files directly to django/django to minimise setup burden (DJANGO_PATH), and improving the contributing docs to show users how to test using docker.

One of the hardest things I found at sprints was getting development environments setup to effectively contribute - even using the docker-box project which I understand quite well. Anything we can do to improve that situation will be very beneficial.

I have fewer opinions about the official CI story, hopefully some of the infrastructure team can comment more on that. I think that replacing the ansible roles with a docker setup can have some definite improvements and open up CI tasks to a larger pool of people (anyone that can edit docker files), but it'd come with maintaining the host that runs docker (cleaning up images, dealing with disk space issues, etc).


On Monday, 5 November 2018 01:20:03 UTC+11, Tom Forbes wrote:

Hello all,

I’ve been working on a docker-compose based alternative to django-box (imaginatively named django-docker-box) over the last month and it finally appears to be mostly complete.

For reference the tool is just a Dockerfile and a docker-compose definition that is able to run a complete test matrix of every supported Python and DB version. It’s as simple as docker-compose run sqlite. You can see a full test run (excluding oracle) here: https://travis-ci.com/orf/django-docker-box/builds/90167436

Florian suggested I create a thread here to gather feedback and discuss any potential future directions for the project, so here goes:

Firstly I’d like to know if there is any support for moving this under the Django project itself, maybe even as a replacement for django-box? I think the setup is pretty quick compared to django-box and is more flexible in terms of database version support as well as working with Oracle. I’d also really like some help improving Oracle support if anyone has the time!

Secondly is there any support for integrating this with our current Jenkins setup? I think it would be pretty neat to have parity between what runs on the CI and what we can run locally and have any improvements shared between both. Perhaps a full matrix run (which right now is 66 different environments) is out of the question but a smaller subset could be good?

Thirdly, and this is a bit wild, but what about using this to reduce the burden of running Jenkins by running the tests on a managed CI service like Travis CI? We would likely still need Jenkins due to issues with Oracle and running tests on Windows (unless https://github.com/django/django/pull/10259 works with Docker!), but we could offload some of the environments onto a third party service. Travis gives large OS projects like Django increased concurrency limits on their accounts so we could end up with pretty speedy test runs. Also with docker-compose switching between CI services (including Jenkins) would be very simple.

The repo is here: https://github.com/orf/django-docker-box.

Any feedback on these points or the project itself would be greatly appreciated,

Tom



--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/04b7db46-f6d4-4f60-9e28-4a4b41f84cad%40googlegroups.com.


--
Adam

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/90f70b0e-9082-489c-ae0a-98341c917a48%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAJwKpySD5xVeCPnkRumJmtztN3tYT4%2B1uEG4GumVqiCSZVEUMA%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Explore integrating django-docker-box in some way?

Tom Forbes
In reply to this post by Nick Sarbicki
I did experiment with it. The main problem is that the image we use requires a bunch of system libraries (see https://github.com/django/django-docker-box/blob/master/Dockerfile#L4). I think a few where missing our outdated in Alpine, but if that's changed then I'm not against trying it again.

Tom

On 8 Oct 2019, at 11:58, Nick Sarbicki <[hidden email]> wrote:

Just out of interest, Tom, have you tried using Alpine as the base? Or is there a reason to avoid it?

I usually find once you introduce all the dependencies for django it doesn't make a huge difference but it might shave some of the weight off if we're worried about image size.


- Nick


On Tue, Oct 8, 2019 at 11:52 AM Josh Smeaton <[hidden email]> wrote:
I was going to archive the repo, but it seems I don't have the necessary permissions. Carlton, do you?

On Tuesday, 8 October 2019 21:45:22 UTC+11, Adam Johnson wrote:
+1 to archiving django-box

On Tue, 8 Oct 2019 at 11:01, Tom Forbes <[hidden email]> wrote:
Thank you for the kind words Bruno! I'm glad it's helped you, if you have any suggestions for improvements then please open an issue on the repo or post a message here, I know it's not perfect. I would have replied earlier but this message didn't get delivered to me.

I'm biased, but I'd be +1 on archiving the old django-box. It's served us well, but unless someone is willing to spend some time updating it then it's going to confuse new users.

On Friday, 4 October 2019 12:40:02 UTC+1, Bruno Alla wrote:
Just a note that as a newish contributor to Django, this django-docker-box is fantastic, it makes things much easier to setup.

Thank you to everyone involved!

On Wednesday, 5 December 2018 00:02:27 UTC, Tom Forbes wrote:

To have this completely working at sprints without having everyone building their own local images we would need to have the Jenkins server use docker in some capacity. This would also require an official django account on Docker hub.

The pattern I’m using right now is that on every build we pull the django-ci:latest image (from my personal account). Docker uses this image as a cache automatically (preventing rebuilds). On any successful master build we push the new image to docker-hub, so subsequent builds can utilise it.

Then anyone wanting to speed up their bootstrapping can do docker-compose pull and automatically have the latest image available for running right away. We can make this smaller, for sure, but we can also suggest people download this beforehand (i.e at their hotel).

I don’t know how feasible this is but it’s also very easy to run a caching docker mirror (docker run -p 5000:5000 registry). Organizers could run this at large events and configuring docker to use a local mirror on the network is a one-line change for atendees.





On 4 December 2018 at 23:52:42, Josh Smeaton ([hidden email]) wrote:

Size of the image could definitely be a concern, especially at sprints where wifi speeds aren't always optimal. The django-box image is significantly larger though so it'd still be a net win. There are also optimisations that can be made to the image for reducing size over time, so I'd fully expect it to come down. I've spent a little bit of time trying to optimise a $work$ python docker file, I'll provide what I've got as an issue to possibly look at.

I see that the ticket has been accepted and I think that's a great step forward. I'd also like to hear from the infrastructure team what their thoughts on using docker over customised build environments would be.

Florian, Tim, Markus .. any thoughts? (Apologies, I've missed some, this list of names is from memory).

On Wednesday, 5 December 2018 10:39:16 UTC+11, Tom Forbes wrote:

Thank you for the reply Josh. I didn’t anticipate any suggestions for including this inside core but off the back of your suggestion I’ve made a ticket here: https://code.djangoproject.com/ticket/30010.

I don’t think it should be complex at all to include this inside Django - it’s four or five new files at most. Hopefully this should improve the experience at sprints, however the current Dockerfile weighs in at 650+mb so the problem may switch from ‘it is hard to set up an environment’ to ‘it is hard to download one’!




On 5 November 2018 at 23:02:30, Josh Smeaton ([hidden email]) wrote:

I'm sorry I haven't had the time to review or contribute yet, but I think it'll be a very useful project - especially for new contributors that might have a little docker experience. The current vagrant solution is heavy, does not work properly on windows and some linuxes, and isn't that easy to maintain or deploy. I'd be in favour of adding the docker files directly to django/django to minimise setup burden (DJANGO_PATH), and improving the contributing docs to show users how to test using docker.

One of the hardest things I found at sprints was getting development environments setup to effectively contribute - even using the docker-box project which I understand quite well. Anything we can do to improve that situation will be very beneficial.

I have fewer opinions about the official CI story, hopefully some of the infrastructure team can comment more on that. I think that replacing the ansible roles with a docker setup can have some definite improvements and open up CI tasks to a larger pool of people (anyone that can edit docker files), but it'd come with maintaining the host that runs docker (cleaning up images, dealing with disk space issues, etc).


On Monday, 5 November 2018 01:20:03 UTC+11, Tom Forbes wrote:

Hello all,

I’ve been working on a docker-compose based alternative to django-box (imaginatively named django-docker-box) over the last month and it finally appears to be mostly complete.

For reference the tool is just a Dockerfile and a docker-compose definition that is able to run a complete test matrix of every supported Python and DB version. It’s as simple as docker-compose run sqlite. You can see a full test run (excluding oracle) here: https://travis-ci.com/orf/django-docker-box/builds/90167436

Florian suggested I create a thread here to gather feedback and discuss any potential future directions for the project, so here goes:

Firstly I’d like to know if there is any support for moving this under the Django project itself, maybe even as a replacement for django-box? I think the setup is pretty quick compared to django-box and is more flexible in terms of database version support as well as working with Oracle. I’d also really like some help improving Oracle support if anyone has the time!

Secondly is there any support for integrating this with our current Jenkins setup? I think it would be pretty neat to have parity between what runs on the CI and what we can run locally and have any improvements shared between both. Perhaps a full matrix run (which right now is 66 different environments) is out of the question but a smaller subset could be good?

Thirdly, and this is a bit wild, but what about using this to reduce the burden of running Jenkins by running the tests on a managed CI service like Travis CI? We would likely still need Jenkins due to issues with Oracle and running tests on Windows (unless https://github.com/django/django/pull/10259 works with Docker!), but we could offload some of the environments onto a third party service. Travis gives large OS projects like Django increased concurrency limits on their accounts so we could end up with pretty speedy test runs. Also with docker-compose switching between CI services (including Jenkins) would be very simple.

The repo is here: https://github.com/orf/django-docker-box.

Any feedback on these points or the project itself would be greatly appreciated,

Tom



--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/04b7db46-f6d4-4f60-9e28-4a4b41f84cad%40googlegroups.com.


--
Adam

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/90f70b0e-9082-489c-ae0a-98341c917a48%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAGuvt93zYniq5uL7me-Pa4D%3DnEJb0aox0PcfheVMqOFFFWRwzQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/1BCF38D3-B3A8-4C25-9AF9-A2EA2E8A1B2E%40tomforb.es.

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Explore integrating django-docker-box in some way?

Nick Sarbicki
That makes sense.

Do we have a list of the versions required? I'm happy to try it out and see if it makes a difference.

- Nick


On Tue, Oct 8, 2019 at 12:05 PM Tom Forbes <[hidden email]> wrote:
I did experiment with it. The main problem is that the image we use requires a bunch of system libraries (see https://github.com/django/django-docker-box/blob/master/Dockerfile#L4). I think a few where missing our outdated in Alpine, but if that's changed then I'm not against trying it again.

Tom

On 8 Oct 2019, at 11:58, Nick Sarbicki <[hidden email]> wrote:

Just out of interest, Tom, have you tried using Alpine as the base? Or is there a reason to avoid it?

I usually find once you introduce all the dependencies for django it doesn't make a huge difference but it might shave some of the weight off if we're worried about image size.


- Nick


On Tue, Oct 8, 2019 at 11:52 AM Josh Smeaton <[hidden email]> wrote:
I was going to archive the repo, but it seems I don't have the necessary permissions. Carlton, do you?

On Tuesday, 8 October 2019 21:45:22 UTC+11, Adam Johnson wrote:
+1 to archiving django-box

On Tue, 8 Oct 2019 at 11:01, Tom Forbes <[hidden email]> wrote:
Thank you for the kind words Bruno! I'm glad it's helped you, if you have any suggestions for improvements then please open an issue on the repo or post a message here, I know it's not perfect. I would have replied earlier but this message didn't get delivered to me.

I'm biased, but I'd be +1 on archiving the old django-box. It's served us well, but unless someone is willing to spend some time updating it then it's going to confuse new users.

On Friday, 4 October 2019 12:40:02 UTC+1, Bruno Alla wrote:
Just a note that as a newish contributor to Django, this django-docker-box is fantastic, it makes things much easier to setup.

Thank you to everyone involved!

On Wednesday, 5 December 2018 00:02:27 UTC, Tom Forbes wrote:

To have this completely working at sprints without having everyone building their own local images we would need to have the Jenkins server use docker in some capacity. This would also require an official django account on Docker hub.

The pattern I’m using right now is that on every build we pull the django-ci:latest image (from my personal account). Docker uses this image as a cache automatically (preventing rebuilds). On any successful master build we push the new image to docker-hub, so subsequent builds can utilise it.

Then anyone wanting to speed up their bootstrapping can do docker-compose pull and automatically have the latest image available for running right away. We can make this smaller, for sure, but we can also suggest people download this beforehand (i.e at their hotel).

I don’t know how feasible this is but it’s also very easy to run a caching docker mirror (docker run -p 5000:5000 registry). Organizers could run this at large events and configuring docker to use a local mirror on the network is a one-line change for atendees.





On 4 December 2018 at 23:52:42, Josh Smeaton ([hidden email]) wrote:

Size of the image could definitely be a concern, especially at sprints where wifi speeds aren't always optimal. The django-box image is significantly larger though so it'd still be a net win. There are also optimisations that can be made to the image for reducing size over time, so I'd fully expect it to come down. I've spent a little bit of time trying to optimise a $work$ python docker file, I'll provide what I've got as an issue to possibly look at.

I see that the ticket has been accepted and I think that's a great step forward. I'd also like to hear from the infrastructure team what their thoughts on using docker over customised build environments would be.

Florian, Tim, Markus .. any thoughts? (Apologies, I've missed some, this list of names is from memory).

On Wednesday, 5 December 2018 10:39:16 UTC+11, Tom Forbes wrote:

Thank you for the reply Josh. I didn’t anticipate any suggestions for including this inside core but off the back of your suggestion I’ve made a ticket here: https://code.djangoproject.com/ticket/30010.

I don’t think it should be complex at all to include this inside Django - it’s four or five new files at most. Hopefully this should improve the experience at sprints, however the current Dockerfile weighs in at 650+mb so the problem may switch from ‘it is hard to set up an environment’ to ‘it is hard to download one’!




On 5 November 2018 at 23:02:30, Josh Smeaton ([hidden email]) wrote:

I'm sorry I haven't had the time to review or contribute yet, but I think it'll be a very useful project - especially for new contributors that might have a little docker experience. The current vagrant solution is heavy, does not work properly on windows and some linuxes, and isn't that easy to maintain or deploy. I'd be in favour of adding the docker files directly to django/django to minimise setup burden (DJANGO_PATH), and improving the contributing docs to show users how to test using docker.

One of the hardest things I found at sprints was getting development environments setup to effectively contribute - even using the docker-box project which I understand quite well. Anything we can do to improve that situation will be very beneficial.

I have fewer opinions about the official CI story, hopefully some of the infrastructure team can comment more on that. I think that replacing the ansible roles with a docker setup can have some definite improvements and open up CI tasks to a larger pool of people (anyone that can edit docker files), but it'd come with maintaining the host that runs docker (cleaning up images, dealing with disk space issues, etc).


On Monday, 5 November 2018 01:20:03 UTC+11, Tom Forbes wrote:

Hello all,

I’ve been working on a docker-compose based alternative to django-box (imaginatively named django-docker-box) over the last month and it finally appears to be mostly complete.

For reference the tool is just a Dockerfile and a docker-compose definition that is able to run a complete test matrix of every supported Python and DB version. It’s as simple as docker-compose run sqlite. You can see a full test run (excluding oracle) here: https://travis-ci.com/orf/django-docker-box/builds/90167436

Florian suggested I create a thread here to gather feedback and discuss any potential future directions for the project, so here goes:

Firstly I’d like to know if there is any support for moving this under the Django project itself, maybe even as a replacement for django-box? I think the setup is pretty quick compared to django-box and is more flexible in terms of database version support as well as working with Oracle. I’d also really like some help improving Oracle support if anyone has the time!

Secondly is there any support for integrating this with our current Jenkins setup? I think it would be pretty neat to have parity between what runs on the CI and what we can run locally and have any improvements shared between both. Perhaps a full matrix run (which right now is 66 different environments) is out of the question but a smaller subset could be good?

Thirdly, and this is a bit wild, but what about using this to reduce the burden of running Jenkins by running the tests on a managed CI service like Travis CI? We would likely still need Jenkins due to issues with Oracle and running tests on Windows (unless https://github.com/django/django/pull/10259 works with Docker!), but we could offload some of the environments onto a third party service. Travis gives large OS projects like Django increased concurrency limits on their accounts so we could end up with pretty speedy test runs. Also with docker-compose switching between CI services (including Jenkins) would be very simple.

The repo is here: https://github.com/orf/django-docker-box.

Any feedback on these points or the project itself would be greatly appreciated,

Tom



--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/04b7db46-f6d4-4f60-9e28-4a4b41f84cad%40googlegroups.com.


--
Adam

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/90f70b0e-9082-489c-ae0a-98341c917a48%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAGuvt93zYniq5uL7me-Pa4D%3DnEJb0aox0PcfheVMqOFFFWRwzQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/1BCF38D3-B3A8-4C25-9AF9-A2EA2E8A1B2E%40tomforb.es.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAGuvt92p%3DOOm165MVEUF4Jac6Z%3Daa0D8siXON1p6zz5mUVrVbA%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Explore integrating django-docker-box in some way?

Jani Tiainen
In reply to this post by Tom Forbes
IIRC correctly I had problems with alpine and it's GIS libraries when preparing for my workshop on Djangocon EU 2019.

Most of the required libs weren't available in stable version. Haven't checked if situation has improved since.


ti 8. lokak. 2019 klo 14.05 Tom Forbes <[hidden email]> kirjoitti:
I did experiment with it. The main problem is that the image we use requires a bunch of system libraries (see https://github.com/django/django-docker-box/blob/master/Dockerfile#L4). I think a few where missing our outdated in Alpine, but if that's changed then I'm not against trying it again.

Tom

On 8 Oct 2019, at 11:58, Nick Sarbicki <[hidden email]> wrote:

Just out of interest, Tom, have you tried using Alpine as the base? Or is there a reason to avoid it?

I usually find once you introduce all the dependencies for django it doesn't make a huge difference but it might shave some of the weight off if we're worried about image size.


- Nick


On Tue, Oct 8, 2019 at 11:52 AM Josh Smeaton <[hidden email]> wrote:
I was going to archive the repo, but it seems I don't have the necessary permissions. Carlton, do you?

On Tuesday, 8 October 2019 21:45:22 UTC+11, Adam Johnson wrote:
+1 to archiving django-box

On Tue, 8 Oct 2019 at 11:01, Tom Forbes <[hidden email]> wrote:
Thank you for the kind words Bruno! I'm glad it's helped you, if you have any suggestions for improvements then please open an issue on the repo or post a message here, I know it's not perfect. I would have replied earlier but this message didn't get delivered to me.

I'm biased, but I'd be +1 on archiving the old django-box. It's served us well, but unless someone is willing to spend some time updating it then it's going to confuse new users.

On Friday, 4 October 2019 12:40:02 UTC+1, Bruno Alla wrote:
Just a note that as a newish contributor to Django, this django-docker-box is fantastic, it makes things much easier to setup.

Thank you to everyone involved!

On Wednesday, 5 December 2018 00:02:27 UTC, Tom Forbes wrote:

To have this completely working at sprints without having everyone building their own local images we would need to have the Jenkins server use docker in some capacity. This would also require an official django account on Docker hub.

The pattern I’m using right now is that on every build we pull the django-ci:latest image (from my personal account). Docker uses this image as a cache automatically (preventing rebuilds). On any successful master build we push the new image to docker-hub, so subsequent builds can utilise it.

Then anyone wanting to speed up their bootstrapping can do docker-compose pull and automatically have the latest image available for running right away. We can make this smaller, for sure, but we can also suggest people download this beforehand (i.e at their hotel).

I don’t know how feasible this is but it’s also very easy to run a caching docker mirror (docker run -p 5000:5000 registry). Organizers could run this at large events and configuring docker to use a local mirror on the network is a one-line change for atendees.





On 4 December 2018 at 23:52:42, Josh Smeaton ([hidden email]) wrote:

Size of the image could definitely be a concern, especially at sprints where wifi speeds aren't always optimal. The django-box image is significantly larger though so it'd still be a net win. There are also optimisations that can be made to the image for reducing size over time, so I'd fully expect it to come down. I've spent a little bit of time trying to optimise a $work$ python docker file, I'll provide what I've got as an issue to possibly look at.

I see that the ticket has been accepted and I think that's a great step forward. I'd also like to hear from the infrastructure team what their thoughts on using docker over customised build environments would be.

Florian, Tim, Markus .. any thoughts? (Apologies, I've missed some, this list of names is from memory).

On Wednesday, 5 December 2018 10:39:16 UTC+11, Tom Forbes wrote:

Thank you for the reply Josh. I didn’t anticipate any suggestions for including this inside core but off the back of your suggestion I’ve made a ticket here: https://code.djangoproject.com/ticket/30010.

I don’t think it should be complex at all to include this inside Django - it’s four or five new files at most. Hopefully this should improve the experience at sprints, however the current Dockerfile weighs in at 650+mb so the problem may switch from ‘it is hard to set up an environment’ to ‘it is hard to download one’!




On 5 November 2018 at 23:02:30, Josh Smeaton ([hidden email]) wrote:

I'm sorry I haven't had the time to review or contribute yet, but I think it'll be a very useful project - especially for new contributors that might have a little docker experience. The current vagrant solution is heavy, does not work properly on windows and some linuxes, and isn't that easy to maintain or deploy. I'd be in favour of adding the docker files directly to django/django to minimise setup burden (DJANGO_PATH), and improving the contributing docs to show users how to test using docker.

One of the hardest things I found at sprints was getting development environments setup to effectively contribute - even using the docker-box project which I understand quite well. Anything we can do to improve that situation will be very beneficial.

I have fewer opinions about the official CI story, hopefully some of the infrastructure team can comment more on that. I think that replacing the ansible roles with a docker setup can have some definite improvements and open up CI tasks to a larger pool of people (anyone that can edit docker files), but it'd come with maintaining the host that runs docker (cleaning up images, dealing with disk space issues, etc).


On Monday, 5 November 2018 01:20:03 UTC+11, Tom Forbes wrote:

Hello all,

I’ve been working on a docker-compose based alternative to django-box (imaginatively named django-docker-box) over the last month and it finally appears to be mostly complete.

For reference the tool is just a Dockerfile and a docker-compose definition that is able to run a complete test matrix of every supported Python and DB version. It’s as simple as docker-compose run sqlite. You can see a full test run (excluding oracle) here: https://travis-ci.com/orf/django-docker-box/builds/90167436

Florian suggested I create a thread here to gather feedback and discuss any potential future directions for the project, so here goes:

Firstly I’d like to know if there is any support for moving this under the Django project itself, maybe even as a replacement for django-box? I think the setup is pretty quick compared to django-box and is more flexible in terms of database version support as well as working with Oracle. I’d also really like some help improving Oracle support if anyone has the time!

Secondly is there any support for integrating this with our current Jenkins setup? I think it would be pretty neat to have parity between what runs on the CI and what we can run locally and have any improvements shared between both. Perhaps a full matrix run (which right now is 66 different environments) is out of the question but a smaller subset could be good?

Thirdly, and this is a bit wild, but what about using this to reduce the burden of running Jenkins by running the tests on a managed CI service like Travis CI? We would likely still need Jenkins due to issues with Oracle and running tests on Windows (unless https://github.com/django/django/pull/10259 works with Docker!), but we could offload some of the environments onto a third party service. Travis gives large OS projects like Django increased concurrency limits on their accounts so we could end up with pretty speedy test runs. Also with docker-compose switching between CI services (including Jenkins) would be very simple.

The repo is here: https://github.com/orf/django-docker-box.

Any feedback on these points or the project itself would be greatly appreciated,

Tom



--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/04b7db46-f6d4-4f60-9e28-4a4b41f84cad%40googlegroups.com.


--
Adam

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/90f70b0e-9082-489c-ae0a-98341c917a48%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAGuvt93zYniq5uL7me-Pa4D%3DnEJb0aox0PcfheVMqOFFFWRwzQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/1BCF38D3-B3A8-4C25-9AF9-A2EA2E8A1B2E%40tomforb.es.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAHn91odMZcbf98eMjmsw2AbndKKP8QBLHYJa9F76GF2nVAT9yg%40mail.gmail.com.