Patterns for REST API bulk operations

Recently at work we’ve gone through a few iterations of design for bulk operations.

Our motivations for batching are:

  1. Increase execution performance. We’d like to avoid looping through operations in a higher tier, executing each one as a separate HTTP request. One other way to address this would be simultaneous parallel requests.

  2. Decrease resource usage. We want to push related operations down to lower layers. Usually there is some natural UX tying the operations together, so naturally they can coalesce into single database statements.

I’ve seen the solution implemented in a number of ways, but not in a way that I felt was clear for the caller to invoke as well as easy for the service to implement.

Object Schema

For the below examples, we’ll work on operations for a feature I’ll just call ‘objects’.

For each object, we may be tracking a number of attributes. It’ll be stored in the database and communicated in JSON via a number of attribute keys.

Column Type
id integer
name string
color string
created_at date
updated_at date

Bulk Actions Overview

So let’s jump into the API for bulk actions:

Verb URL Request Payload Response Payload Action
GET /objects/[?query] { {‘id’:1, ‘name’:, ‘color’:}, {‘id’:2, ‘name’:, ‘color’:}, … } Fetch a number of objects matching the query
POST /objects/ { {‘name’:, ‘color’:}, {‘name’:, ‘color’:}, … } { {‘id’:1, ‘name’:, ‘color’:}, {‘id’:2, ‘name’:, ‘color’:}, … } Create a number of objects
PUT /objects/[?query] {‘name’:, ‘color’:} Update a number of objects, common attributes
PUT /objects/[?query] { {‘id’:1, ‘name’:, ‘color’:}, {‘id’:2, ‘name’:, ‘color’:}, … } Update a number of objects, unique attributes
DELETE /objects/[?query] Delete a number of objects

Let’s get in to details.

Bulk Read and Query (Selection Criteria)

The bulk read is a pattern commonly defined for REST APIs and very straightforward.

The [?query] portion in the URL above contains query params which filter the set of objects to apply the bulk action to.

For instance, for our object we can specify:

  • common attributes: /objects/?color=blue
  • array selections: /objects/?id=1&id=2
  • pagination selections: /objects/?limit=10&offset=20

It’s easy to see that this gets dropped right into the WHERE clause in a SQL backend. Omitting a query by default should imply all objects.

This same pattern will be used for bulk update and bulk delete actions.

Bulk Create

Note that the response is at least as big as the request. This may not be ideal, so I’ll propose 2 options.

  1. Your API should return an id list if your REST client is an installed application or JS in the browser which can make use of the id’s in the response.

  2. Your API should omit a response body if it subsequently will be calling the bulk read. This is the case where your presentation will have sorting, filtering, and pagination applied to all objects. For instance, you would probably present an alert confirming the creation, followed by a bulk read to update the object list.

Bulk Update 1: Common Attributes

The use case here is when you have a web form which allows you to perform some action on a list of objects. Usually the user has selected items from the list via checkboxes, and then chosen some action from a top-level menu which will be performed on all those items.

Using the example above, we may update a number of objects to be color:green. The following would be appropriate:

PUT /objects/?id=1&id=5&id=6

{ 'color':'green'}

In most cases this will translate into a single UPDATE query on a database.

Bulk Update 2: Unique Attributes

One example use case here is an admin screen presenting a 2d matrix of checkboxes. In this case you have a set of attributes to apply a list of objects in any combination.

Implementation is nothing more than just looping over the objects in the body and executing the same logic as the singular PUT /objects/<id>.

PUT /objects/

{ 'id':1, 'color':'green' }, { 'id':3, 'color':'blue' }

Bulk Update: Further Notes

You’ll notice that there is an overloaded controller here in doing PUT /objects/.

The form for Bulk Update 1: Common Attributes has the advantage of allowing for easily affecting all objects, even those whose id is unknown by the caller. Second, it matches a lot of common style of web interfacing where a list of items has checkbox selection, and then a single common action is performed.

The form for Bulk Update 2: Unique Attributes has the advantage of more flexibility in terms of which attributes each object is updating, and is a natural extension of the singular case. Unfortunately it doesn’t allow for the all case. It also translates to the same number of queries as there are objects to be updated, which isn’t ideal for performance.

My recommendation is to implement only the Bulk Update 1: Common Attributes unless you absolutely need more flexibility.

Bulk Deletes

The delete case here is very straightforward. Note that calling DELETE /objects/ will delete every object. If that’s dangerous, it’s better to implement soft deletion of objects, which would be a call to PUT /objects/ setting a deleted flag to true. It feels a little funny to me to use soft deletion with the DELETE verb since the object doesn’t go away, but that’s just personal preference.

Summary

My hope is that this is a useful guide for those who need to implement real-world REST APIs and make a REST-based service perform acceptably.

Feel free to drop me a line if you have any particular cases where the above suggestions break down. I’m still learning.

Reference

Tags: REST code