CRUD APIs on DynamoDB
Guidelines to quickly create CRUD actions on DynamoDB items from API Gateway
When we perform basic CRUD actions against DynamoDB elements, using a API Gateway + Lambda with a Node.js wrapper to DynamoDB becomes an overkill. AWS has a specific integration pattern for such use cases, using only API Gateway as a direct proxy to DynamoDB.
API Gateway can indeed execute commands directly on DynamoDB (like
Query
, Scan
, PutItem
, UpdateItem
) the same way that it does execute Lambda functions. However this requires some integration work to ensure that the request and response both get translated from an HTTP to a DynamoDB world and reverse.Although it is possible to script the entire deployment of a set of API actions (using the AWS CLI or Terraform / CloudFormation templates), we prefer to stick to a manual way of doing things so that we all understand the elements at play here. This is the goal of these guidelines. Please note that it's best to have a previous understanding of how DynamoDB works (by having toyed with Lambdas for example), to grasp the full power offered by this kind of integrations.
Note that many of the below elements can be found in a simpler setting in this blog article from AWS.
The key elements involved in the setup of a CRUD API to DynamoDB are the following:
- An API endpoint (the API as per AWS language)
- Within this API endpoint, a set of resources, than can be nested and/or parameterised (e.g.
/insurers
,/insurers/active
,/insurers/{id}
) - For each resource, a set of HTTP methods (e.g.
GET /insurers
,POST /insurers
) - For each HTTP method, an AWS service integration consisting of
- a Method Request (typically requires no configuration but the addition of an API key if required)
- an Integration Request, responsible to translate the
application/json
content of an HTTP request into a DynamoDB-readable command by means of a Mapping Template - an Integration Response, performing the reverse task of mapping a DynamoDB command output to an HTTP response
- a Method Response, typically set at a
status 200
response
- If relevant, an API key for the API (to be associated to an usage plan, typically
TangleJinx
) - A deployment stage for the API (typically called
default
in our case)
The key part of the setup happens in the initial creation of the integration, where it is required to select that the HTTP method proxies an AWS service, that is, DynamoDB. There, the following inputs are required:
- AWS Region:
eu-central-1
- AWS Service: DynamoDB
- AWS Subdomain: leave blank
- HTTP method: set to
POST
, regardless of the API action. This is because all requests to DynamoDB actually requirePOST
to execute - Action Type: use action name
- Action: either
Scan
,Query
,PutItem
,GetItem
,UpdateItem
etc (any valid DynamoDB command) - Execution role: choose the
arn:aws:iam::ACCOUNT_ID:role/lambda_full_access role
(we've set it up so that it can accept API Gateway as a Trusted Entity) - Content Handling: Passthrough
When this is all setup, it remains the configure the actual integration templates for the request and response. Typical templates are provided in the below section.
- You need to enable CORS if the call is made by a browser (Ardor) but you don't need it if the call if made from a server (Yago)
- You can easily test the API by clicking on the little lightning icon top left of the method execution panel.
- It is always required to deploy the API to be able to test it with any new configuration. It is after deployment, by clicking on "Stages > Default" that one gets to see the actual
invoke url
by which the API can be called.
Note that these integration templates rely on some AWS-proprietary templating language which takes a bit of time to get used to. That's why we have this library - don't be ashamed to just copy-paste these examples even if you don't understand the full syntax of this peculiar language.
Note that we stick to our
success-message-data
pattern like in all other APIs.- Each API Gateway request has a uuid accessible in the
$context.requestId
variable, which is very handy for creating new objects' uuid. S
stands for String,N
for Number,L
for List,B
for Boolean, andM
for Map.- When writing in the templating language, note that quotes will be printed as is in the request/response, meaning for example that when parsing back strings received in the response, it is still necessary to enclose them in quotes so that we get a rightly formatted JSON.
- Note that as per the JSON specifications, it is necessary to use double quotes and not single quotes.
- If you want to return an object after
POST
ing to DynamoDB, you need to useUpdateItem
with theRETURN_ALL
attribute. UsingPutItem
will prevent any item from being returned.
Integration request
{
"TableName": "unpaids"
}
Always specify the table name by hard-coding it, never let it be defined in the HTTP request parameters. We don't want anyone fooling around by being capable of scanning all our tables!
Integration response
#set($inputRoot = $input.path('$'))
{
"success": true,
"message": "unpaids successfully retrieved",
"data": [
#foreach($elem in $inputRoot.Items) {
"uuid": "$elem.uuid.S",
"user_uuid": "$elem.user_uuid.S",
"user_full_name": "$elem.user_full_name.S",
"contract_reference": "$elem.contract_reference.S",
"insurer": "$elem.insurer.S",
"policy_type": "$elem.policy_type.S",
"amount_due": $elem.amount_due.N,
"due_at": $elem.due_at.N,
"last_reminder_at": $elem.last_reminder_at.N,
"reminders_count": $elem.reminders_count.N,
"reminders_history": $elem.reminders_history.L,
"status": "$elem.status.S"
}#if($foreach.hasNext),#end
#end
]
}
Note that when a record's field is an object, one can return its values in the integration response by chaining the
.M
field (i.e., the Map) to get the subfields, as shown in the example below.Nested integration response
#set($inputRoot = $input.path('$'))
{
"success": true,
"message": "unpaids successfully retrieved",
"data": [
#foreach($elem in $inputRoot.Items) {
"uuid": "$elem.uuid.S",
"user_uuid": "$elem.user_uuid.S",
"user_full_name": "$elem.user_full_name.S",
"contract_reference": "$elem.contract_reference.S",
"insurer": "$elem.insurer.S",
"policy_type": "$elem.policy_type.S",
"amount_due": $elem.amount_due.N,
"due_at": $elem.due_at.N,
"last_reminder_at": $elem.last_reminder_at.N,
"reminders_count": $elem.reminders_count.N,
"reminders_history": $elem.reminders_history.L,
"status": "$elem.status.S"
}#if($foreach.hasNext),#end
#end
]
}
This method requires setting up a dynamic
/{uuid}
API resource.Integration request
{
"TableName": "unpaids",
"Key": {
"uuid": {
"S": "$input.params('uuid')"
}
}
}
Integration response
{
"success": true,
"message": "unpaid successfully retrieved",
"data": {
"uuid": "$input.path('$.Item.uuid.S')",
"user_uuid": "$input.path('$.Item.user_uuid.S')",
"user_full_name": "$input.path('$.Item.user_full_name.S')",
"contract_reference": "$input.path('$.Item.contract_reference.S')",
"insurer": "$input.path('$.Item.insurer.S')",
"policy_type": "$input.path('$.Item.policy_type.S')",
"amount_due": $input.path('$.Item.amount_due.N'),
"due_at": $input.path('$.Item.due_at.N'),
"last_reminder_at": $input.path('$.Item.last_reminder_at.N'),
"reminders_count": $input.path('$.Item.reminders_count.N'),
"reminders_history": $input.path('$.Item.reminders_history.L'),
"status": "$input.path('$.Item.status.S')"
}
}
This method allows queries with params of the form
param1=foo¶m2=bar
Integration request
#if($input.params('status')=='' && $input.params('user_uuid')=='')
{
"TableName": "unpaids"
}
#{else}
#if($input.params('status')!='' && $input.params('user_uuid')!='')
{
"TableName": "unpaids",
"ExpressionAttributeNames": {
"#s":"status"
},
"FilterExpression": "#s = :s AND user_uuid = :u",
"ExpressionAttributeValues": {
":s": {
"S": "$input.params('status')"
},
":u": {
"S": "$input.params('user_uuid')"
}
}
}
#{else}
#if($input.params('status')!='')
{
"TableName": "unpaids",
"ExpressionAttributeNames": {
"#s":"status"
},
"FilterExpression": "#s = :s",
"ExpressionAttributeValues": {
":s": {
"S": "$input.params('status')"
}
}
}
#{else}
{
"TableName": "unpaids",
"FilterExpression": "user_uuid = :u",
"ExpressionAttributeValues": {
":u": {
"S": "$input.params('user_uuid')"
}
}
}
#end
#end
#end
Integration response
#set($inputRoot = $input.path('$'))
{
"success": true,
"message": "unpaids successfully retrieved",
"data": [
#foreach($elem in $inputRoot.Items) {
"uuid": "$elem.uuid.S",
"user_uuid": "$elem.user_uuid.S",
"user_full_name": "$elem.user_full_name.S",
"contract_reference": "$elem.contract_reference.S",
"insurer": "$elem.insurer.S",
"policy_type": "$elem.policy_type.S",
"amount_due": $elem.amount_due.N,
"due_at": $elem.due_at.N,
"last_reminder_at": $elem.last_reminder_at.N,
"reminders_count": $elem.reminders_count.N,
"reminders_history": $elem.reminders_history.L,
"status": "$elem.status.S"
}#if($foreach.hasNext),#end
#end
]
}
Note how we use the
requestId
from the context, as outlined above.Note that we need to specify
ALL_NEW
as ReturnValues
in order to receive the full updated object in the integration response.Note that the
ExpressionAttributeNames
is used to replace variable names which are reserved keywords from AWS with another one that are not.Integration request
{
"TableName": "unpaids",
"Key": {
"uuid": {
"S": "unpaid_$context.requestId"
}
},
"ExpressionAttributeNames": {
"#s":"status"
},
"UpdateExpression": "set user_uuid = :u, #s = :s, user_full_name = :n, contract_reference = :c, insurer = :i, policy_type = :p, amount_due = :a, due_at = :d, last_reminder_at = :l, reminders_count = :r, reminders_history = :h",
"ExpressionAttributeValues": {
":u": {"S": "$input.path('$.user_uuid')"},
":s": {"S": "waiting_for_payment"},
":n": {"S": "$input.path('$.user_full_name')"},
":c": {"S": "$input.path('$.contract_reference')"},
":i": {"S": "$input.path('$.insurer')"},
":p": {"S": "$input.path('$.policy_type')"},
":a": {"N": "$input.path('$.amount_due')"},
":d": {"N": "$input.path('$.due_at')"},
":l": {"N": "0"},
":r": {"N": "0"},
":h": {"L": []}
},
"ReturnValues": "ALL_NEW"
}
Integration response
#set($attributes = $input.path('$').Attributes)
{
"success": true,
"message": "unpaid successfully created",
"data": {
"uuid": "$attributes.uuid.S",
"user_uuid": "$attributes.user_uuid.S",
"user_full_name": "$attributes.user_full_name.S",
"contract_reference": "$attributes.contract_reference.S",
"insurer": "$attributes.insurer.S",
"policy_type": "$attributes.policy_type.S",
"amount_due": $attributes.amount_due.N,
"due_at": $attributes.due_at.N,
"last_reminder_at": $attributes.last_reminder_at.N,
"reminders_count": $attributes.reminders_count.N,
"reminders_history": $attributes.reminders_history.L,
"status": "$attributes.status.S",
}
}
Note that we need to specify
ALL_NEW
as ReturnValues
in order to receive the full updated object in the integration response.Integration request
{
"TableName": "unpaids",
"Key": {
"uuid": {
"S": "$input.path('$.uuid')"
}
},
"ExpressionAttributeNames": {
"#s":"status"
},
"UpdateExpression": "set user_uuid = :u, #s = :s, user_full_name = :n, contract_reference = :c, insurer = :i, policy_type = :p, amount_due = :a, due_at = :d",
"ExpressionAttributeValues": {
":u": {"S": "$input.path('$.user_uuid')"},
":s": {"S": "$input.path('$.status')"},
":n": {"S": "$input.path('$.user_full_name')"},
":c": {"S": "$input.path('$.contract_reference')"},
":i": {"S": "$input.path('$.insurer')"},
":p": {"S": "$input.path('$.policy_type')"},
":a": {"N": "$input.path('$.amount_due')"},
":d": {"N": "$input.path('$.due_at')"}
},
"ReturnValues": "ALL_NEW"
}
Integration response
#set($attributes = $input.path('$').Attributes)
{
"success": true,
"message": "unpaid successfully updated",
"data": {
"uuid": "$attributes.uuid.S",
"user_uuid": "$attributes.user_uuid.S",
"user_full_name": "$attributes.user_full_name.S",
"contract_reference": "$attributes.contract_reference.S",
"insurer": "$attributes.insurer.S",
"policy_type": "$attributes.policy_type.S",
"amount_due": $attributes.amount_due.N,
"due_at": $attributes.due_at.N,
"last_reminder_at": $attributes.last_reminder_at.N,
"reminders_count": $attributes.reminders_count.N,
"reminders_history": $attributes.reminders_history.L,
"status": "$attributes.status.S",
}
}
Instead of manually putting all the elements you want to retrieve in an integration response, you can dynamically get all the elements.
#set($attributes = $input.path('$').Attributes)
{
"success": true,
"message": "xxx créé avec succès",
"data": {
#foreach ($mapEntry in $attributes.entrySet())
#set ($value=$mapEntry.value.entrySet().iterator().next().value)
#set ($valueType=$mapEntry.value.entrySet().iterator().next().key)
#if($valueType=="N" || $valueType=="BOOL")
"$mapEntry.key":$value
#elseif($valueType=="S" || $valueType=="B")
"$mapEntry.key":"$value"
#end
#if($foreach.hasNext),#end
#end
}
}
TODO