Making Extremely Fast AWS Elastic Cache Redis with Lua

Posted Jan 11, 2025

By Roman Tsypuk 7 min read

AWS ElastiCache for Redis offers a managed, high-performance, in-memory data store that is widely adopted for caching, session management, real-time analytics, and more. To further enhance performance, Redis includes support for Lua scripting, enabling developers to execute complex operations atomically and reduce the overhead of multiple network calls.

This combination of AWS ElastiCache for Redis and Lua scripting provides a powerful framework for optimizing workloads. By leveraging Lua, developers can move compute logic closer to the data, minimizing latency and improving throughput for critical operations. In this guide, we’ll explore how to use Lua scripting in AWS ElastiCache for Redis

Elastic Cache and Redis versions

Amazon ElastiCache is a fully managed, in-memory data store and cache service that supports both Redis OSS (Open Source Software) and Valkey-compatible engines. As of January 2025, ElastiCache offers support for the following Redis OSS versions:

Redis OSS 7.1: This version includes performance enhancements, such as extended enhanced I/O threads and improved memory access patterns, enabling workloads to achieve higher throughput and lower latencies.
Redis OSS 5.0.6: Includes features like Streams, improved memory efficiency, and better eviction policies.
Redis OSS 4.0.10: Provides enhancements in memory management and introduces modules for extending Redis functionalities.

It’s important to note that earlier versions, such as Redis OSS 3.x and 2.x, have reached End of Life (EOL) and are no longer supported.

Additionally, AWS has introduced Valkey, a Redis-compatible engine that offers enhanced performance and features. Valkey is designed to be compatible with Redis APIs, ensuring that your existing applications can work seamlessly with it.

So prior adding Lua check you redis version and lua interpreter version:

  
redis-cli -h domain.cache.amazonaws.com -p 6379
domain.cache.amazonaws.com:6379> info
# Server
redis_version:5.0.6
redis_git_sha1:0
redis_git_dirty:0
redis_build_id:0
redis_mode:cluster
os:Amazon ElastiCache
arch_bits:64

Elastic common use cases

Redis (Remote Dictionary Server) is a highly versatile, in-memory data store often used for its performance, simplicity, and flexibility. Below are common use cases where Redis excels:

Caching (TTL)
Session Management
Real-Time Analytics
Pub/Sub Messaging
Leaderboards and Ranking Systems
Distributed Locks
Queue Management
Configuration and Feature Flags
Gaming and Real-Time Applications

Where Lua is used?

Lua is a lightweight, high-performance scripting language widely used in various fields due to its simplicity, extensibility, and ease of embedding in applications.

Here are some prominent examples of where Lua is used:

Game engines (Unity, World of Warcraft, Angry Birds, MineCraft)
Embedded Systems (Cisco devices automation and scripting, smart-TV, IP-boxes, IoT)
Web (Kong API Gateway, Nginx scripting)
DB (Redis, Tarantool)
Networking (Wireshark plugins, Snort IDS)
Video (VLC plugins)

Lua has a small footprint (less than 30K lines of code of core, GC and virtual Machine code) with high performance make it ideal for resource-constrained environments.

How Redis executed Lua

Redis executes Lua scripts using its built-in Lua interpreter, which is based on the Lua 5.1 engine. This mechanism allows atomic execution of scripts and provides several benefits for transactional or complex operations in Redis.

Here’s how Redis executes Lua scripts step by step:

A Lua script is sent to Redis using the EVAL or EVALSHA command.
Redis passes the Lua script to its built-in Lua interpreter.
The entire Lua script is executed as a single atomic operation. No other commands can run during the script execution. This guarantees consistency, similar to a transaction.

Data Locality

Besides transactional guarantee, Redis executes all Lua script code on a interpreter at data node.

As a results there is no overhead for sending multiple requests for each command of script and have any network overhead.

Such scripts are executed fast and have direct access to Data.

Mutation Scripts

As a best practice only mutation operations (scripts) are placed under Lua script which guarantees the transactional execution. The read calls are follow eventual consistancy and do not require Lua scripting.

Here are examples of Lua scripts that perform complex logic of both TTL and SET for unique user records modification sessions:

Locking

  
-- Lua script for managing a set with TTL for values
-- Parameters:
-- KEYS[1] - the key representing the id
-- ARGV[1] - the conv_message_id to add to the set
-- Get the current timestamp from Redis server in seconds
local currentTimestamp = redis.call('TIME')[1]

-- Define the TTL (300 seconds)
local ttl = 300

-- Calculate the expiration timestamp for the given conv_message_id
local expirationTimestamp = tonumber(currentTimestamp) + ttl

-- Add the conv_message_id to the sorted set with its expiration timestamp as the score
redis.call('ZADD', KEYS[1], expirationTimestamp, ARGV[1])

-- Remove all values from the set where the TTL (score) is less than the current timestamp
redis.call('ZREMRANGEBYSCORE', KEYS[1], '-inf', currentTimestamp)

-- Retrieve all elements from the set after cleanup
local elements = redis.call('ZRANGE', KEYS[1], 0, -1)

-- Return all elements
return elements

Lock release:

  
local id = KEYS[1]
local userID = ARGV[1]

-- Get the current timestamp from Redis server in seconds
local currentTime = redis.call('TIME')[1]

-- Remove all values from the set where the TTL (score) is less than the current timestamp
redis.call('ZREMRANGEBYSCORE', id, '-inf', currentTime)

-- Retrieve all remaining valid entries from the set
local entries = redis.call('ZRANGE', id, 0, -1)

-- Prepare the arguments for a single ZREM call
local zremArgs = {}
for _, entry in ipairs(entries) do
    if entry:match("^" .. userID .. ":(.-):preordered$") then
        table.insert(zremArgs, entry)
    end
end

-- Remove all preordered entries in a single ZREM call
if #zremArgs > 1 then
    redis.call('ZREM', id, unpack(zremArgs))
end

return true

Changing the state of Record:

  
local id =  KEYS[1]           -- The key for the ZSET
local userID = ARGV[1]          -- The user ID
local product = ARGV[2]          -- The conversation ID
local ttlOffset = tonumber(ARGV[3]) -- TTL offset in seconds

-- Get the current timestamp from Redis
local currentTime = tonumber(redis.call('TIME')[1])
local updatedTTL = currentTime + ttlOffset

-- Scan the ZSET for matching conversation ID
local elements = redis.call('ZRANGEBYSCORE', "UniqueSession:" ..id, '-inf', '+inf', 'WITHSCORES')

for i = 1, #elements, 2 do
    local entry = elements[i]
    local score = tonumber(elements[i + 1])

    if entry == userID .. ":" .. product .. ":preordered" then
        local updatedEntry = string.gsub(entry, ":preordered$", ":udpated")
        redis.call('ZREM', "UniqueSession:" ..id, entry)
        -- Add to ZSET with the new value and TTL
        redis.call('ZADD', "UniqueSession:" ..id, updatedTTL, updatedEntry)
        return updatedEntry
    end
end

Such Lua scripts are compacted and transferred to Redis Data Node based on the key distribution and are cached for future invocations.

As a result all commands in a single script are invoked locally on Redis Node given a big boost in performance and removing any additional network overhead for each command round trip.

With such technic Lua can add additional performance improvement to already existing Elastic Cache Redis mode.

Common Pitfalls

The script can be executed only on a Single Node. So if you are using Redis in cluster mode (not a single node mode), then Redis master node should make decision on which node the key is located and will submit script to be executed on that node, following Data locality principle.

So, key should not be generated inside the Lua script at runtime, it should be available at invocation moment, otherwise you will receive error:

  
(error) ERR Error running script (call to f_c9b97033a0d4103dab472899949f376aedcf9242): @user_script:11: @user_script: 11: Lua script attempted to access a non local key in a cluster node

Micro optimisations

Redis REM, ADD commands can be executed as a single command

  
local zaddArgs = {}
-- Fill zaddArgs with data, then invoke with a single command
redis.call('ZADD', keyID, unpack(zaddArgs))

Scripts can be cached on a Redis side, and be executed using EVALSHA command by a hashcode ID