• PopulatePythonVirtualEnvironment

Use Case: Populate Python Virtual Environment

NOTE: Email thread regarding this use case can be found starting here.

In our deployment scenario, we are pushing a django application up to a generic cloud server.

The servers come up pretty vanilla on one particular provider; we can't set up boot images.

We need to put up a Python virtual environment, activate it, then install a bunch of Python stuff into it.

This case just handles setting up the Python virtual environment and installing Python requirements into it.

We assume that all of the system level prerequisites have already been handled by the InstallSystemLevelUtility use case.

The Challenge

Since "run()" gets a whole new shell each time, you can't carry-over the environment and end up having to re-establishing the virtual environment each time. I'm not a Capistrano user, only having played with it for a little while a few months ago, but it appears that Capistrano partially handles this with "cap shell."

Proposed Syntax

• What I want is something like:

   1 runner=BatchRunner()
   2 runner.addCommand("cd my-new-virtualenv")
   3 runner.addCommand("bin/activate")
   4 runner.addCommand("easy_install some_project_into_my_virtual_env")
   5 runner.addCommand("easy_install some_other_project_into_my_virtual_env")
   6 runner.run()

• Or, even better:

   1 shell_commands = """
   2 cd my-new-virtualenv
   3 bin/activate
   4 easy_install some_project_into_my_virtual_env
   5 easy_install some_other_project_into_my_virtual_env
   6 """
   7 runner=BatchRunner()
   8 runner.addCommand("some_pre_command")   # I can add one to the beginning
   9 runner.addCommands(shell_commands)              # then add my batch commands
  10 runner.addCommand("some_post_command")  # I can add one to the end if I want
  11 runner.run()

• Or, even:

   1 batchCommandsRunner=BatchRunner(shell_commands)
   2 runner=BatchRunner()
   3 runner.addCommand("some_pre_command")   # I can add one to the beginning
   4 runner.addCommands(batchCommandsRunner) # then add my batch commands
   5 runner.addCommand("some_post_command")  # I can add one to the end if I want
   6 runner.run()

That way all the commands can be run, in sequence, but without causing multiple connections or having to re-establish the environment. The third syntax allows easy combinations of pre-configured BatchRunners, even add-on module type ones.

Alternative Proposal 1

Installing into virtual environments is a common use case. It might be useful to simplify matters by providing an operation to construct a virtualenv and make is a context manager. A usage example might be:

   1 with virtualenv('/path/to/env', no-site-packages=True, unzip-setuptools=True) as myenv:
   2     run("easy_install sqlalchemy")
   3     run("easy_install my_package")

or more explicitly...

   1 myenv = virtualenv('/path/to/env', no-site-packages=True, unzip-setuptools=True)
   2 with myenv as myenv:
   3     run("easy_install sqlalchemy")
   4     run("easy_install my_package")

   1 myenv = virtualenv('/path/to/env', no-site-packages=True, unzip-setuptools=True)
   2 run("easy_install sqlalchemy", virtenv=myenv)
   3 run("easy_install my_package")

This might be implemented by allowing standard operations (run, sudo etc.) to accept a karg virtenv. The virtualenv context manager would simply call the enclosed operations with the virtenv arg set. Each operation would need to preempt the user-supplied command by calling virtenv/bin/activate OR rewrite the command to reference the virtual environment /bin dir explicitly.

A simple, if inelegant method

I use this:

   1 def env(cmd, *a, **kw):
   2     run("cd $(project_root); source env/bin/activate; %s" % cmd, *a, **kw)

It's basically a drop-in replacement for run(), except that it activates the virtualenv first. For example:

   1 env("pip freeze -r requirements.txt frozen.txt", fail='abort')

I'm not sure why it's a problem to re-activate the env a bunch of times in a row, as long as you don't have to manually write the code to do it every time. The "open a shell and run multiple commands" idea seems much more important for avoiding a lot of ssh logins, each of which has a significant time delay. ssh's ControlMaster option is worth looking into for this.