I'm trying to set up a distributed ipyparallel / ipython cluster. I'd like to run the controller locally on my laptop with engines running remotely on Amazon ec2 (without resorting to Starcluster).
I've read the latest documentation on setting up ssh ipython with local / remote hosts and tried the following:
-Generated 'parallel' profile using "ipython create profile --parallel..."
-set SSHEngineSetLauncher.engines dictionary to my ec2 instance address and engine count, as well as set the SSHEngineLaunch.user name appropriately.
-created an alias to launch ipengine via ssh on the ec2 instance and set the SSHEngineSetLauncher.engine_cmd accordingly.
-In ipcontroller_config.py, set HubFactory.ip to '*' as well as in ipengine_config.py.
-Set up password-less ssh login to my ec2 instance
Launching ipcluster from my laptop with this configuration and setting log-level to debug, I see that my local ipython profile is successfully transferred to my ec2 instance (along with the necessary .json security files) and that the ipengine command is successfully executed remotely. However, this results in a "Registration timeout”, as the ip address in the transferred ipcontroller_engine.json file is set to the local (as opposed to public) IP address of my laptop.
When I manually set HubFactory.ip in my ipcluster_config.py to the public IP of my laptop and re-launch ipcluster, I get a ZeroMQ error "Can't assign requested address.”
My question is, what’s the proper way to set the controller IP / location such so that remote engines (on AWS ec2, different LAN) can connect?
On Tue, Feb 21, 2017 at 8:37 PM, jeromeku <[hidden email]> wrote:
To connect engines to a controller over ssh you need the engines to be able to SSH to the controller (or at least its LAN), not just the other way around. That’s probably going to be trickiest bit if you want your controller on your laptop. Technically, the connection could be done with an ssh reverse tunnel, but IPython doesn’t have an option to do this for you automatically.
In general, when engines connect over ssh, you need this config:
It would probably be easiest, if it is acceptable, to run the controller on the EC2 server as well. There would be a lot fewer tunnelled connections this way. Plus, the SSHProxy launchers can work quite a bit better than plain SSH for multiple engines on a single remote host:
IPython-User mailing list
|Free forum by Nabble||Edit this page|