IronWorkers in Python

This approach uses our depreciated workflow. Please see https://github.com/iron-io/dockerworker/tree/master/[ython for the current process.

Python has become one of the most popular languages for web software and scientific or mathematical computing. By offloading tasks to IronWorker, computations and requests can be run in parallel using the power of the cloud. This article will get you started writing Python workers, but you should be familiar with the basics of IronWorker.

Quick Start

Get the CLI

We’ve created a command line interface to the IronWorker service that makes working with the service a lot easier and more convenient. It does, however, require you to have Ruby 1.9+ installed and to install the iron_worker_ng gem. Once Ruby 1.9+ is installed, you can just run the following command to get the gem:

$ gem install iron_worker_ng

Get the Python Client Library

You can download the Python client library, iron_worker_python, from Github—note that you’ll need the iron_core_python library installed, too. Users of pip or easy_install can simply use pip install iron-worker and easy_install iron-worker.

Create Your Configuration File

The Python library uses a configuration file or environment variables set that tell it what your credentials are. We have some pretty good documentation about how this works, but for simplicity’s sake, just save the following as iron.json in the root of your project:

{
  "project_id": "INSERT YOUR PROJECT ID HERE",
  "token": "INSERT YOUR TOKEN HERE"
}

You should insert your project ID and token into that iron.json file. Then, assuming you’re running the commands from within the folder, the library will pick up your credentials and use them automatically.

Write Your Python Worker

print "Hello from Python"

Create a .worker File

Worker files are a simple way to define your worker and its dependencies. Save the following in a file called hello.worker

# set the runtime language. Python workers use "python"
runtime "python"
# exec is the file that will be executed:
exec "hello_worker.py"

You could include python packages and build a worker remotely if one of its libraries depends on some binary extensions. You can read more about remote build here.

runtime "python"
exec "hello_worker.py"

pip 'twilio'
pip 'mock', '1.0.1'

remote

You also could include python packages from requirements.txt. The format of the requirements.txt should be a standard format.

runtime "python"
exec "hello_worker.py"

requirements 'requirements.txt'

remote

You could include other files in there too. You can read more about .worker files here.

Upload the Worker

$ iron_worker upload hello

That command will read your .worker file, create your worker code package and upload it to IronWorker. Head over to hud-e.iron.io, click the Worker link on your projects list, then click the Tasks tab. You should see your new worker listed there with zero runs. Click on it to show the task list which will be empty, but not for long.

Queue Up Tasks for Your Worker

from iron_worker import *

worker = IronWorker()
response = worker.queue(code_name="hello")

You can now queue up a task by calling python enqueue.py from your shell.

Another way is to use CLI:

$ iron_worker queue hello

Now look at the task list in HUD and you should see your task show up and go from “queued” to “running” to “completed”.

Now that we know it works, let’s queue up a bunch of tasks from code.

Deep Dive

Payload Example

Retrieving the payload in Python is the same as it is on any other language. Retrieve the -payload argument passed to the script, load that file, and parse it as JSON.

In your worker:

import sys, json

payload_file = None
payload = None

for i in range(len(sys.argv)):
    if sys.argv[i] == "-payload" and (i + 1) < len(sys.argv):
        payload_file = sys.argv[i + 1]
        with open(payload_file,'r') as f:
            payload = json.loads(f.read())
        break

Ensuring your script exits with the right exit code

It is important in some cases to declare a explicit exit code to give our systems a indication if your worker has completed sucessfully or failed. this also prevents instances where your worker may just hang or wait. In your worker:

Python: exit(1)
sys.exit(1)

Environment

The Python environment that the workers run in on IronWorker is as follows:

You can just use import {MODULE_NAME} to use these modules in your workers.