A Practical Guide to Building Bidirectional Pagination in AWS DynamoDB

Pagination is the most common requirement for any API built. Usually, in the query parameter of the API, we will pass the required page number and the number of results to be present in the response.

But, in DynamoDB, we won’t be able to jump to a specific page. That’s a design approach taken to maintain the same millisecond response time from the database, even if it has millions of records.

So, how does Pagination work in DynamoDB? We can pass the number of items required in a response using Limit. And then we can request the next page and so on.

Unidirectional Pagination

Let us create a DynamoDB Table to store Invoice details. For now, we can keep the table simple with Invoice Number as the partition key and Invoice Date as the sort key. And, let’s create a GSI with partition key as entityType, which will be set to “invoice”, and sort key as invoice date.

Now, let’s use GSI to get invoices.

This query returns the first 10 invoices based on the ascending order of the invoice date. And, an interesting part in the response is the “Last Evaluated Key”. This key gives the primary key of the last element which was returned in the result. It is like an address, which you can use to fetch the next page.

Now, let’s use this last evaluated key and get the next page.

If you observe the response of the second page, it is from the 11th invoice to the 20th invoice and has the last evaluated key value in the response for the next page.

Great, now we have achieved Unidirectional pagination, which allows us to go in the forward direction to get data in small chunks.

Bidirectional Pagination

We know how to utilise the limit and Exclusive Start key parameter to achieve pagination. Is it possible to go to the previous page?

The client can store the Last Evaluated Key on its side and get the previous page content. That looks simple, doesn’t it? But, what if the number of pages is more, and it’s complicated to store the keys for each page and maintain them? Instead, let’s see how to handle it on the API side.

We have the last evaluated key for a given page, and it is passed as the exclusive start key. DynamoDB goes in ascending order of sort key to get the data for the next page. So the simple approach would be to ask DynamoDB to get the items from a specific item in descending order, so that we would be able to get the previous page data.

We are able to retrieve the previous page, but the results are not sorted in ascending order. We can just reverse the response list to get the data in the desired format.

If we observe carefully, the page 2 data doesn’t have the 20th invoice in the response and also has the 10th invoice, which is supposed to be in 1st first. So, we can have a simple update done to get the response in the expected format.

Great, now we have achieved bidirectional pagination.

Complete Code:

import boto3

dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table("invoice")

def get_invoices(pageSize=10, after=None, before=None):
    """Get invoices with pagination support."""
    query_params = {
        "IndexName": "entityType-invoiceDate-index",
        "KeyConditionExpression": "entityType = :entityType",
        "ExpressionAttributeValues": {
            ":entityType": "invoice",
        },
        "Limit": pageSize,
    }

    # if before is provided, we need to reduce the limit by 1 to account for the item that will be used as ExclusiveStartKey
    scanIndexForward = True

    if before:
        query_params["Limit"] = pageSize - 1
        scanIndexForward = False
        query_params["ExclusiveStartKey"] = before

    if after:
        query_params["ExclusiveStartKey"] = after

    query_params["ScanIndexForward"] = scanIndexForward

    response = table.query(**query_params)
    items = response.get("Items", [])
    last_evaluated_key = response.get("LastEvaluatedKey", None)

    # if before is provided, we need to get the complete item which is mentioned in ExclusiveStartKey
    if before:
        get_param = {
            "Key": {
                "invoiceDate": before["invoiceDate"],
                "invoiceNumber": before["invoiceNumber"],
            }
        }
        get_response = table.get_item(**get_param)
        item = get_response.get("Item", None)
        if item:
            items.reverse()
            items.append(item)

    # set the next cursor value to return in response   
    if scanIndexForward:
        next_cursor = last_evaluated_key if last_evaluated_key else None
        previous_cursor = after
    else:
        next_cursor =  {
            "invoiceNumber": items[-1]["invoiceNumber"],
            "invoiceDate": items[-1]["invoiceDate"],
            "entityType": "invoice"
        }
        previous_cursor = last_evaluated_key if last_evaluated_key else None       

    return {
        "items": items,
        "after":  next_cursor,
        "before": previous_cursor,
    }

Get 1st Page:

Getting 2nd Page:

Let’s get the previous page — 1st page — using the before key.

Magic and Logic behind fetching the previous page

When going to the previous page, DynamoDB doesn’t give us a direct way. To achieve this, we have hacked a workaround.

1. We fetch pageSize – 1 records to skip the overlapping item between pages.
2. We fetch that skipped item separately (using get_item).
3. We reverse the list (since the query runs backwards) and add the skipped item back, giving us a proper previous page.

Key Terminology of DynamoDB Query.

Limit: While querying, we can pass a limit parameter, which decides the maximum number of items to be present in the response. If we don’t pass a limit, DynamoDB will try to get as many items till the total response size reaches 1MB in size.

Last Evaluated Key: This gives the primary key of the last element which is present in the response for that given page.

Exclusive Start Key: We can ask DynamoDB to start getting items from a specific record instead of getting them from the beginning.

Scan Index forward: While getting the items from a specific item, if scan index forward is set to true, then DynamoDB fetches the item in ascending order based on sort key and in reverse when it is set to false.

Conclusion:

By using ExclusiveStartKey and playing with the sort direction, we can build clean, reliable B*idirectional pagination* on top of DynamoDB.

It’s fast and works well, though DynamoDB doesn’t support „go to page 10“ style jumps (by design, to stay super fast). But for „next“ and „previous“ style UIs, this pattern works great.

The only thing to watch out for is if new items get added or removed, in between pages might shift a bit. But for most apps, this is perfectly fine.