Playbook
Search
⌃K

CRUD APIs on DynamoDB

Guidelines to quickly create CRUD actions on DynamoDB items from API Gateway

Overview & motivations

When we perform basic CRUD actions against DynamoDB elements, using a API Gateway + Lambda with a Node.js wrapper to DynamoDB becomes an overkill. AWS has a specific integration pattern for such use cases, using only API Gateway as a direct proxy to DynamoDB.
API Gateway can indeed execute commands directly on DynamoDB (like Query, Scan, PutItem, UpdateItem) the same way that it does execute Lambda functions. However this requires some integration work to ensure that the request and response both get translated from an HTTP to a DynamoDB world and reverse.
Although it is possible to script the entire deployment of a set of API actions (using the AWS CLI or Terraform / CloudFormation templates), we prefer to stick to a manual way of doing things so that we all understand the elements at play here. This is the goal of these guidelines. Please note that it's best to have a previous understanding of how DynamoDB works (by having toyed with Lambdas for example), to grasp the full power offered by this kind of integrations.
Note that many of the below elements can be found in a simpler setting in this blog article from AWS.

Elements required and setup

The key elements involved in the setup of a CRUD API to DynamoDB are the following:
  • An API endpoint (the API as per AWS language)
  • Within this API endpoint, a set of resources, than can be nested and/or parameterised (e.g. /insurers, /insurers/active, /insurers/{id})
  • For each resource, a set of HTTP methods (e.g. GET /insurers, POST /insurers)
  • For each HTTP method, an AWS service integration consisting of
    • a Method Request (typically requires no configuration but the addition of an API key if required)
    • an Integration Request, responsible to translate the application/json content of an HTTP request into a DynamoDB-readable command by means of a Mapping Template
    • an Integration Response, performing the reverse task of mapping a DynamoDB command output to an HTTP response
    • a Method Response, typically set at a status 200 response
  • If relevant, an API key for the API (to be associated to an usage plan, typically TangleJinx)
  • A deployment stage for the API (typically called default in our case)
The key part of the setup happens in the initial creation of the integration, where it is required to select that the HTTP method proxies an AWS service, that is, DynamoDB. There, the following inputs are required:
  • AWS Region: eu-central-1
  • AWS Service: DynamoDB
  • AWS Subdomain: leave blank
  • HTTP method: set to POST, regardless of the API action. This is because all requests to DynamoDB actually require POST to execute
  • Action Type: use action name
  • Action: either Scan, Query, PutItem, GetItem, UpdateItem etc (any valid DynamoDB command)
  • Execution role: choose the arn:aws:iam::ACCOUNT_ID:role/lambda_full_access role (we've set it up so that it can accept API Gateway as a Trusted Entity)
  • Content Handling: Passthrough
When this is all setup, it remains the configure the actual integration templates for the request and response. Typical templates are provided in the below section.

General notes

  • You need to enable CORS if the call is made by a browser (Ardor) but you don't need it if the call if made from a server (Yago)
  • You can easily test the API by clicking on the little lightning icon top left of the method execution panel.
  • It is always required to deploy the API to be able to test it with any new configuration. It is after deployment, by clicking on "Stages > Default" that one gets to see the actual invoke url by which the API can be called.

Integration template examples

Note that these integration templates rely on some AWS-proprietary templating language which takes a bit of time to get used to. That's why we have this library - don't be ashamed to just copy-paste these examples even if you don't understand the full syntax of this peculiar language.
Note that we stick to our success-message-data pattern like in all other APIs.

General notes

  • Each API Gateway request has a uuid accessible in the $context.requestId variable, which is very handy for creating new objects' uuid.
  • S stands for String, N for Number, L for List, B for Boolean, and M for Map.
  • When writing in the templating language, note that quotes will be printed as is in the request/response, meaning for example that when parsing back strings received in the response, it is still necessary to enclose them in quotes so that we get a rightly formatted JSON.
  • Note that as per the JSON specifications, it is necessary to use double quotes and not single quotes.
  • If you want to return an object after POSTing to DynamoDB, you need to use UpdateItem with the RETURN_ALL attribute. Using PutItem will prevent any item from being returned.

GET all from table - Scan action

Integration request

Integration request
{
"TableName": "unpaids"
}
Always specify the table name by hard-coding it, never let it be defined in the HTTP request parameters. We don't want anyone fooling around by being capable of scanning all our tables!

Integration response

Integration response
#set($inputRoot = $input.path('$'))
{
"success": true,
"message": "unpaids successfully retrieved",
"data": [
#foreach($elem in $inputRoot.Items) {
"uuid": "$elem.uuid.S",
"user_uuid": "$elem.user_uuid.S",
"user_full_name": "$elem.user_full_name.S",
"contract_reference": "$elem.contract_reference.S",
"insurer": "$elem.insurer.S",
"policy_type": "$elem.policy_type.S",
"amount_due": $elem.amount_due.N,
"due_at": $elem.due_at.N,
"last_reminder_at": $elem.last_reminder_at.N,
"reminders_count": $elem.reminders_count.N,
"reminders_history": $elem.reminders_history.L,
"status": "$elem.status.S"
}#if($foreach.hasNext),#end
#end
]
}

Returning nested objects

Note that when a record's field is an object, one can return its values in the integration response by chaining the .M field (i.e., the Map) to get the subfields, as shown in the example below.
Nested integration response
#set($inputRoot = $input.path('$'))
{
"success": true,
"message": "unpaids successfully retrieved",
"data": [
#foreach($elem in $inputRoot.Items) {
"uuid": "$elem.uuid.S",
"user_uuid": "$elem.user_uuid.S",
"user_full_name": "$elem.user_full_name.S",
"contract_reference": "$elem.contract_reference.S",
"insurer": "$elem.insurer.S",
"policy_type": "$elem.policy_type.S",
"amount_due": $elem.amount_due.N,
"due_at": $elem.due_at.N,
"last_reminder_at": $elem.last_reminder_at.N,
"reminders_count": $elem.reminders_count.N,
"reminders_history": $elem.reminders_history.L,
"status": "$elem.status.S"
}#if($foreach.hasNext),#end
#end
]
}

GET element by uuid - GetItem action

This method requires setting up a dynamic /{uuid} API resource.

Integration request

Integration request
{
"TableName": "unpaids",
"Key": {
"uuid": {
"S": "$input.params('uuid')"
}
}
}

Integration response

Integration response
{
"success": true,
"message": "unpaid successfully retrieved",
"data": {
"uuid": "$input.path('$.Item.uuid.S')",
"user_uuid": "$input.path('$.Item.user_uuid.S')",
"user_full_name": "$input.path('$.Item.user_full_name.S')",
"contract_reference": "$input.path('$.Item.contract_reference.S')",
"insurer": "$input.path('$.Item.insurer.S')",
"policy_type": "$input.path('$.Item.policy_type.S')",
"amount_due": $input.path('$.Item.amount_due.N'),
"due_at": $input.path('$.Item.due_at.N'),
"last_reminder_at": $input.path('$.Item.last_reminder_at.N'),
"reminders_count": $input.path('$.Item.reminders_count.N'),
"reminders_history": $input.path('$.Item.reminders_history.L'),
"status": "$input.path('$.Item.status.S')"
}
}

GET filtered elements - Scan action

This method allows queries with params of the formparam1=foo&param2=bar

Integration request

Integration request
#if($input.params('status')=='' && $input.params('user_uuid')=='')
{
"TableName": "unpaids"
}
#{else}
#if($input.params('status')!='' && $input.params('user_uuid')!='')
{
"TableName": "unpaids",
"ExpressionAttributeNames": {
"#s":"status"
},
"FilterExpression": "#s = :s AND user_uuid = :u",
"ExpressionAttributeValues": {
":s": {
"S": "$input.params('status')"
},
":u": {
"S": "$input.params('user_uuid')"
}
}
}
#{else}
#if($input.params('status')!='')
{
"TableName": "unpaids",
"ExpressionAttributeNames": {
"#s":"status"
},
"FilterExpression": "#s = :s",
"ExpressionAttributeValues": {
":s": {
"S": "$input.params('status')"
}
}
}
#{else}
{
"TableName": "unpaids",
"FilterExpression": "user_uuid = :u",
"ExpressionAttributeValues": {
":u": {
"S": "$input.params('user_uuid')"
}
}
}
#end
#end
#end

Integration response

Integration response
#set($inputRoot = $input.path('$'))
{
"success": true,
"message": "unpaids successfully retrieved",
"data": [
#foreach($elem in $inputRoot.Items) {
"uuid": "$elem.uuid.S",
"user_uuid": "$elem.user_uuid.S",
"user_full_name": "$elem.user_full_name.S",
"contract_reference": "$elem.contract_reference.S",
"insurer": "$elem.insurer.S",
"policy_type": "$elem.policy_type.S",
"amount_due": $elem.amount_due.N,
"due_at": $elem.due_at.N,
"last_reminder_at": $elem.last_reminder_at.N,
"reminders_count": $elem.reminders_count.N,
"reminders_history": $elem.reminders_history.L,
"status": "$elem.status.S"
}#if($foreach.hasNext),#end
#end
]
}

POST create element - UpdateItem action

Integration request

Note how we use the requestId from the context, as outlined above.
Note that we need to specify ALL_NEW as ReturnValues in order to receive the full updated object in the integration response.
Note that the ExpressionAttributeNames is used to replace variable names which are reserved keywords from AWS with another one that are not.
Integration request
{
"TableName": "unpaids",
"Key": {
"uuid": {
"S": "unpaid_$context.requestId"
}
},
"ExpressionAttributeNames": {
"#s":"status"
},
"UpdateExpression": "set user_uuid = :u, #s = :s, user_full_name = :n, contract_reference = :c, insurer = :i, policy_type = :p, amount_due = :a, due_at = :d, last_reminder_at = :l, reminders_count = :r, reminders_history = :h",
"ExpressionAttributeValues": {
":u": {"S": "$input.path('$.user_uuid')"},
":s": {"S": "waiting_for_payment"},
":n": {"S": "$input.path('$.user_full_name')"},
":c": {"S": "$input.path('$.contract_reference')"},
":i": {"S": "$input.path('$.insurer')"},
":p": {"S": "$input.path('$.policy_type')"},
":a": {"N": "$input.path('$.amount_due')"},
":d": {"N": "$input.path('$.due_at')"},
":l": {"N": "0"},
":r": {"N": "0"},
":h": {"L": []}
},
"ReturnValues": "ALL_NEW"
}

Integration response

Integration response
#set($attributes = $input.path('$').Attributes)
{
"success": true,
"message": "unpaid successfully created",
"data": {
"uuid": "$attributes.uuid.S",
"user_uuid": "$attributes.user_uuid.S",
"user_full_name": "$attributes.user_full_name.S",
"contract_reference": "$attributes.contract_reference.S",
"insurer": "$attributes.insurer.S",
"policy_type": "$attributes.policy_type.S",
"amount_due": $attributes.amount_due.N,
"due_at": $attributes.due_at.N,
"last_reminder_at": $attributes.last_reminder_at.N,
"reminders_count": $attributes.reminders_count.N,
"reminders_history": $attributes.reminders_history.L,
"status": "$attributes.status.S",
}
}

PUT update element - UpdateItem action

Integration request

Note that we need to specify ALL_NEW as ReturnValues in order to receive the full updated object in the integration response.
Integration request
{
"TableName": "unpaids",
"Key": {
"uuid": {
"S": "$input.path('$.uuid')"
}
},
"ExpressionAttributeNames": {
"#s":"status"
},
"UpdateExpression": "set user_uuid = :u, #s = :s, user_full_name = :n, contract_reference = :c, insurer = :i, policy_type = :p, amount_due = :a, due_at = :d",
"ExpressionAttributeValues": {
":u": {"S": "$input.path('$.user_uuid')"},
":s": {"S": "$input.path('$.status')"},
":n": {"S": "$input.path('$.user_full_name')"},
":c": {"S": "$input.path('$.contract_reference')"},
":i": {"S": "$input.path('$.insurer')"},
":p": {"S": "$input.path('$.policy_type')"},
":a": {"N": "$input.path('$.amount_due')"},
":d": {"N": "$input.path('$.due_at')"}
},
"ReturnValues": "ALL_NEW"
}

Integration response

Integration response
#set($attributes = $input.path('$').Attributes)
{
"success": true,
"message": "unpaid successfully updated",
"data": {
"uuid": "$attributes.uuid.S",
"user_uuid": "$attributes.user_uuid.S",
"user_full_name": "$attributes.user_full_name.S",
"contract_reference": "$attributes.contract_reference.S",
"insurer": "$attributes.insurer.S",
"policy_type": "$attributes.policy_type.S",
"amount_due": $attributes.amount_due.N,
"due_at": $attributes.due_at.N,
"last_reminder_at": $attributes.last_reminder_at.N,
"reminders_count": $attributes.reminders_count.N,
"reminders_history": $attributes.reminders_history.L,
"status": "$attributes.status.S",
}
}

Default integration response

Instead of manually putting all the elements you want to retrieve in an integration response, you can dynamically get all the elements.
#set($attributes = $input.path('$').Attributes)
{
"success": true,
"message": "xxx créé avec succès",
"data": {
#foreach ($mapEntry in $attributes.entrySet())
#set ($value=$mapEntry.value.entrySet().iterator().next().value)
#set ($valueType=$mapEntry.value.entrySet().iterator().next().key)
#if($valueType=="N" || $valueType=="BOOL")
"$mapEntry.key":$value
#elseif($valueType=="S" || $valueType=="B")
"$mapEntry.key":"$value"
#end
#if($foreach.hasNext),#end
#end
}
}

Validations and JSON schemas

TODO
INFRA & DEVOPS GUIDES - Previous
API Gateway patterns
Next - INFRA & DEVOPS GUIDES
SSH management
Last modified 1yr ago