API

In general, Ocypod attempts to return meaningful HTTP status codes to indicate whether an operation was successful or not.

As well as the status codes detailed per endpoint, Ocypod will also return the following 5xx codes:

Queue endpoints

Used for interacting with queues, i.e. creating new queues, updating queue settings, creating jobs, and fetching jobs.

GET /queue

Get all known queues as a JSON list.

Returns

Example

$ curl localhost:8023/queue
["my-queue", "another_queue", "queue3"]

GET /queue/{queue_name}

Get a queue's settings.

Returns

Example

$ curl localhost:8023/queue/example
{"timeout":"10m",
 "heartbeat_timeout":"5m",
 "expires_after":"5m",
 "retries":5,
 "retry_delays":["10s","30s","5m"]}

GET /queue/{queue_name}/size

Get number of jobs currently queued for a given queue name.

Returns

Example

$ curl localhost:8023/queue/example/size
3

PUT /queue/{queue_name}

Create a new queue, or update an existing queue with given settings.

Request

The request body must contain JSON of the form:

{"timeout": <duration>,
 "heartbeat_timeout": <duration>,
 "expires_after": <duration>,
 "retries": <integer>,
 "retry_delays": [<duration>[, <duration>...]]}

where <duration> is a human readable string of the form "30s", "5m", "1w2d7h", etc.

Any of the fields may be omitted (e.g. {} is the minimum accepted input), in which case server-wide default values will be used.

Set timeout, heartbeat_timeout, or expires_after to "0s" to disable timeout/expiry.

Set retries to 0 to disable retries.

Returns

Example

$ curl -i -H 'content-type: application/json' -XPUT -d '{"timeout": "10m"}' localhost:8023/queue/example
HTTP/1.1 201 Created
location: /queue/example
content-length: 0
date: Fri, 16 Nov 2018 17:47:55 GMT

DELETE /queue/{queue_name}

Delete an existing queue, and any jobs still queued on it. Any running or ended jobs won't be affected.

If any running jobs created on this queue have retries remaining, they'll instead remain in their failed/timed out state, and be eligible for expiry if they can't be requeued due to their original queue no longer existing.

Returns

Example

$ curl -i -XDELETE localhost:8023/queue/example
HTTP/1.1 204 Queue deleted
date: Tue, 20 Nov 2018 18:07:47 GMT

GET /queue/{queue_name}/job

Get the next job to work from the given queue, if any, as a JSON job payload.

This JSON payload is of the form:

{"id": <integer>, "input": <any JSON>}

where id is a unique job ID that's autogenerated by Ocypod when a job is created, and input is any JSON provided by the client that created the job.

When a client gets a job in this way, the job is marked as running, and is removed from the queue.

Returns

Example

$ curl -i localhost:8023/queue/example/job
HTTP/1.1 200 OK
content-length: 58
content-type: application/json
date: Tue, 20 Nov 2018 18:52:56 GMT

{"id":77,"input":{"some_key": [1, 2, 3]}}

POST /queue/{queue_name}/job

Create a new job on given queue with given JSON.

Request

The request body must contain JSON of the form:

{"input": <any JSON>,
 "tags": <list of strings>,
 "timeout": <duration>,
 "heartbeat_timeout": <duration>,
 "expires_after": <duration>,
 "retries": <integer>,
 "retry_delays": <list of durations>}

All fields are optional (i.e. {} is the minimum valid job that can be created).

input represents the job's main payload, and is what clients that accept the job will use to work on. This will typically contain input data to process, and can be any simple or complex JSON value. Defaults to null if not specified.

tags are an optional list of strings to attach to this job. These can be used to look up jobs by tag using the /tag endpoints. Uses might include attaching a username to jobs, or attaching a batch ID to a number of related jobs created by the same process. Defaults to [] if not specified.

timeout is the maximum amount of time the job can run before it's marked as timed out. Default is to use the queue's setting.

heartbeat_timeout is the maximum amount of time between heartbeats before the job is marked as timed out. Default is to use the queue's setting.

expires_after is the amount of time the job and its metadata will remain in Ocypod after the job completes, fails, or times out (with no retries remaining). Default is to use the queue's setting.

retries is the number of times to re-queue this job on failure or time out. Default is to use the queue's setting.

retry_delays is a list of durations specifying the minimum time to wait before re-queuing a job on failure. This can be used to specify linear/exponential/any backoff intervals when retrying jobs. Default is to use the queue's setting.

If the number of retries exceeds the length of this list, then the last value will always be used, e.g. with retries: 4 and retry_delays: ["10s", "1m", "5m"], the job won't be retried for at least 10 seconds after the 1st failure, for 1 minute after the 2nd failure, for 5 minutes after the 3rd failure, and for 5 minutes on the 4th failure.

Returns

201 - job successfully created, response contains ID of new job, and location of job in location header 400 - invalid queue name or invalid job creation JSON given 404 - queue with given name not found

GET /queue/{queue_name}/job_ids

Get a list of job IDs by status for all jobs originally created in the given queue.

Returns

JSON response of the form:

{<status>: [<job ID>, ...], ...}

Example

$ curl -i localhost:8023/queue/example/job_ids
{"completed":[1,2],"cancelled":[],"timed_out":[],"queued":[4,5],"running":[3],"failed":[]}

Job endpoints

Endpoints for interacting with jobs in any state.

GET /job/{job_id}[?fields=<comma separated list of fields>]

Get metadata about given job as a JSON object.

Optionally accepts a comma separated list of fields to fetch, otherwise will get all fields by default.

Response JSON is an object with each field being a key, i.e.

 {<field1>: <value>, <field2>: <value>, ... }

Returns

Example

$ curl 'localhost:8023/job/77?fields=id,status,started_at,ended_at'
{"id":77,
 "status":"running",
 "started_at":"2018-11-20T18:52:42.700853Z",
 "ended_at":null}

PATCH /job/{job_id}

Clients can use this to modify a job's status, and/or its output field.

This endpoint is generally used to mark a job as completed/failed/cancelled, to write progress/error information to output, and to write final results to output.

Request

The request must be JSON of the form:

{"status": ("completed"|"failed"|"cancelled"),
 "output": <any JSON>}

All fields are optional, only fields that are present will cause any changes.

Response

Example

Mark a job as completed, and write its results:

$ curl -i -XPATCH -H 'content-type: application/json' -d '{"status": "completed", "output": {"result": 321, "progress": 100.0, "warnings": 3}}' localhost:8023/job/23
HTTP/1.1 204 OK
date: Wed, 21 Nov 2018 11:13:34 GMT

DELETE /job/{job_id}

Delete a job from Ocypod, regardless of its state.

Any jobs which have ended and have a non-zero expires_after field will eventually be cleaned up once they've ended, so this endpoint is only needed if this is something you want manual control over.

While you can delete jobs that are currently running, you'll need to make sure that clients/workers gracefully handle this case.

For more graceful job removal, you can set a job's status to cancelled, and allow clients to handle this.

Response

Example

$ curl -XDELETE localhost:8023/job/123
HTTP/1.1 204 OK
date: Wed, 21 Nov 2018 11:13:34 GMT

PUT /job/{job_id}/heartbeat

Used by clients that are working on a job to send a heartbeat for it. This is lets the server know that the job is still actively being worked on.

This is only useful for jobs with a non-zero heartbeat_timeout set, and can only be used on jobs that are running.

Response

Example

$ curl -XPUT localhost:8023/job/23/heartbeat
HTTP/1.1 204 OK
date: Wed, 21 Nov 2018 11:13:34 GMT

GET /job/{job_id}/output

Get a job's output JSON field.

Response

200 - JSON contents of the job's output field 404 - job with given ID does not exist

Example

$ curl localhost:8023/job/123/output
{"a": 1, "b": [2, 3, 4]}

PUT /job/{job_id}/output

Set a job's output field to given JSON. This has the same effect as updating using the PATCH /job/{job_id} endpoint and setting the output field in the request JSON, but is often more convenient if only updating this field.

Response

Example

$ curl -i -XPUT -H 'content-type: application/json' -d '[1,2,3]' localhost:8023/job/12/output
HTTP/1.1 204 OK
date: Wed, 21 Nov 2018 11:13:34 GMT

Tag endpoints

Information about jobs that were created with tags can be retreived here. Tags are typically used as a way of grouping multiple jobs in some way.

Use cases include e.g.:

GET /tag/{tag_name}

Get a JSON list of job IDs with given tag. If no jobs have the requested tag, or if the tag doesn't exist, then an empty list will be returned.

Response

Example

$ curl localhost:8023/tag/user3
[17, 19, 225]

Information endpoints

These provide information about the Ocypod system as a whole.

GET /info

TODO

GET /info/version

Get the version string for the currently running Ocypod server.

Response

Example

$ curl localhost:8023/info/version
"0.1.2"

Healthcheck endpoints

Provides a way of checking the health of the Ocypod server. Returns JSON containing the health of the system, and optionally a message about any error detected.

GET /health

Get JSON indicating the health of the Ocypod server. Currently this checks whether it can successfully connect to and communicate with Redis.

Returns JSON of the form:

{"status": ("healthy"|"unhealthy"), "error": <error message>}

The error field is optionally present if they status is "unhealthy", the "status" field is always present.

Response

Example

$ curl localhost:8023/health
{"status": "healthy"}