Writing Workers in Python

Python has become one of the most popular languages for web software and scientific or mathematical computing. By offloading tasks to IronWorker, computations and requests can be run in parallel using the power of the cloud. This article will get you started writing Python workers, but you should be familiar with the basics of IronWorker.

Table of Contents

Quick Start

Get the CLI

We've created a command line interface to the IronWorker service that makes working with the service a lot easier and more convenient. It does, however, require you to have Ruby 1.9+ installed and to install the iron_worker_ng gem. Once Ruby 1.9+ is installed, you can just the following command to get the gem:

Command Line

$ gem install iron_worker_ng

Get the Python Client Library

You can download the Python client library, iron_worker_python, from Github—note that you'll need the ironcorepython library installed, too. Users of pip or easyinstall can simply use pip install iron-worker and `easyinstall iron-worker`.

Create Your Configuration File

The Python library uses a configuration file or environment variables set that tell it what your credentials are. We have some pretty good documentation about how this works, but for simplicity's sake, just save the following as iron.json in the root of your project:

iron.json

{
  "project_id": "INSERT YOUR PROJECT ID HERE",
  "token": "INSERT YOUR TOKEN HERE"
}

You should insert your project ID and token into that iron.json file. Then, assuming you're running the commands from within the folder, the library will pick up your credentials and use them automatically.

Write Your Python Worker

hello_worker.py

print "Hello from Python"

Create a .worker File

Worker files are a simple way to define your worker and its dependencies. Save the following in a file called hello.worker

hello.worker

# set the runtime language. Python workers use "python"
runtime "python"
# exec is the file that will be executed:
exec "hello_worker.py"

You could include gems and other files in there too. You can read more about .worker files here.

Upload the Worker

Command Line

$ iron_worker upload hello

That command will read your .worker file, create your worker code package and upload it to IronWorker. Head over to hud.iron.io, click the Worker link on your projects list, then click the Tasks tab. You should see your new worker listed there with zero runs. Click on it to show the task list which will be empty, but not for long.

Queue Up Tasks for Your Worker

enqueue.py

from iron_worker import *

worker = IronWorker()
response = worker.queue(code_name="hello")

You can now queue up a task by calling python enqueue.py from your shell.

Another way is to use CLI:

Command Line

$ iron_worker queue hello

Now look at the task list in HUD and you should see your task show up and go from "queued" to "running" to "completed".

Now that we know it works, let’s queue up a bunch of tasks from code.

Note: Once you upload a code package, you can queue as many tasks as you'd like against it. You only need to re-upload the code package when your code changes.

Deep Dive

Payload Example

Retrieving the payload in Python is the same as it is on any other language. Retrieve the -payload argument passed to the script, load that file, and parse it as JSON.

In your worker:

import sys, json

payload_file = None
payload = None

for i in range(len(sys.argv)):
    if sys.argv[i] == "-payload" and (i + 1) < len(sys.argv):
        payload_file = sys.argv[i + 1]
        with open(payload_file,'r') as f:
            payload = json.loads(f.read())
        break

Ensuring your script exits with the right exit code

It is important in some cases to declare a explicit exit code to give our systems a indication if your worker has completed sucessfully or failed. this also prevents instances where your worker may just hang or wait.
In your worker:

Python: exit(1)
sys.exit(1)

Environment

The Python environment that the workers run in on IronWorker is as follows:

Python Version Version 2.7.2

Installed Modules

python-lxml
numpy
scipy
pymongo
gevent
PIL

You can just use import {MODULE_NAME} to use these modules in your workers.

Note: While it is possible to use these modules without bundling them, we highly recommend that you include modules your code is reliant upon in the code package whenever possible. Most of these modules are included in the environment because they are binary modules, making it impossible to supply them at runtime. The ones that are not binary modules are some of the more popular modules, which we include to allow users to try things out and test things with minimal setup and pain. We cannot guarantee which version of the module will be available, and we may update them without warning. Reliance on these modules may cause some unexpected conflicts in your code.