Caching Strategies to Reduce API Latency

Most API performance problems trace back to the same bottleneck: your application asking for the same data repeatedly. A poorly cached backend sends thousands of identical database queries when a single lookup would suffice. A caching strategy that reduces latency can drop response times by 90%, turning a p99 transaction into sub-50ms snappiness.

The challenge isn't knowing that caching helps—it's knowing where to cache and how to keep your cache from lying to users. In this post, we'll walk through in-memory and distributed caching patterns, show you how to identify cacheable queries in traces, and explain the tradeoffs that separate a win from a regret.

Why APIs Slow Down Without Caching

Before diving into solutions, understand the cost. A typical endpoint that fetches a user profile, their recent orders, and a product list might hit the database 10–50 times. If each query takes 5–10ms and you serve 1,000 requests per second, you're burning 50–500 seconds of database time per second—a disaster.

The n-plus-one query problem is the classic culprit: you fetch a list of items, then loop and fetch details for each one. But repeated config lookups, session validation, and feature-flag checks compound the pain. Without caching, even a moderately complex request balloons in latency.

In-Memory Caching: The Simplest Win

The fastest cache is memory in your application process. No network round-trip, no serialization overhead.

How it works:

• Store frequently accessed data in a HashMap or similar structure.
• Check the cache before querying the database.
• Evict stale entries after a TTL (time-to-live) or when memory pressure rises.

Here's a basic example in Node.js:

const cache = new Map();

async function getUserWithCache(userId) {
  const cacheKey = `user:${userId}`;
  
  // Return from cache if present and not stale
  if (cache.has(cacheKey)) {
    const { data, expiresAt } = cache.get(cacheKey);
    if (Date.now() < expiresAt) {
      return data;
    }
    cache.delete(cacheKey);
  }
  
  // Fetch from database
  const user = await db.query('SELECT * FROM users WHERE id = $1', [userId]);
  
  // Store in cache with 5-minute TTL
  cache.set(cacheKey, { data: user, expiresAt: Date.now() + 5 * 60 * 1000 });
  return user;
}

Strengths:

• Instant lookups (microseconds).
• No external dependency.
• Works perfectly for small, stable datasets.

Drawbacks:

• Each process has its own copy—no sharing across replicas.
• Cache invalidation is manual and error-prone.
• Memory is finite; you can't cache everything.
• If your app crashes, the cache vanishes.

In-memory caching shines for immutable data (feature flags, config) and read-heavy, slow-change data (user profiles if updated infrequently).

Redis: Distributed Caching at Scale

When your app runs on 10 servers, in-memory caches become a problem: each replica has stale or incomplete data. Redis solves this by offering a single, shared, in-memory cache accessible from any process over the network.

How it works:

• Queries and writes go to Redis first; if the key exists, return it.
• On miss or expiry, fetch from the database and store the result in Redis.
• All replicas share the same cache.

Example in Java with Redis:

@Autowired
private RedisTemplate<String, User> redisTemplate;

public User getUserWithCache(long userId) {
  String cacheKey = "user:" + userId;
  
  // Try Redis first
  User cached = redisTemplate.opsForValue().get(cacheKey);
  if (cached != null) {
    return cached;
  }
  
  // Fetch from database
  User user = userRepository.findById(userId).orElse(null);
  if (user != null) {
    // Store in Redis with 5-minute expiry
    redisTemplate.opsForValue().set(cacheKey, user, Duration.ofMinutes(5));
  }
  
  return user;
}

Strengths:

• Shared across all app instances.
• Fast (single-digit milliseconds over the network).
• Built-in TTL and eviction policies.
• Persistence options available (RDB snapshots, AOF logs).

Drawbacks:

• Adds operational complexity: another service to deploy, monitor, and scale.
• Network latency (even if low) is slower than in-memory.
• Cache coherency is your responsibility.

Redis is the go-to for medium to large deployments and any multi-instance setup.

The Cache-Aside Pattern

The cache-aside (or "lazy-loading") pattern is the most common way to integrate caching into an application:

1. On a read request, check the cache.
2. If there, return it immediately.
3. If not (cache miss), fetch from the source (database), store in cache, and return.
4. On a write, update the source and invalidate the cache entry.

This pattern is simple and works well for most scenarios. The risk is stale reads: if a write happens between cache checks, the cache may hold obsolete data. TTLs mitigate this—set them short enough to be acceptable.

An alternative is the write-through pattern: always write to the cache then the database. This keeps cache and source in sync but adds latency to writes and complexity if the cache is unavailable.

import redis
import json
import time

cache = redis.Redis(host='localhost', port=6379)

def get_product(product_id):
  cache_key = f"product:{product_id}"
  
  # Check cache
  cached = cache.get(cache_key)
  if cached:
    return json.loads(cached)
  
  # Cache miss: fetch from database
  product = db.query("SELECT * FROM products WHERE id = %s", (product_id,))
  
  # Store for 10 minutes
  cache.setex(cache_key, 600, json.dumps(product))
  return product

def update_product(product_id, data):
  # Update database
  db.execute("UPDATE products SET ... WHERE id = %s", (product_id,))
  
  # Invalidate cache
  cache.delete(f"product:{product_id}")

Identifying Cacheable Queries with Tracing

The biggest mistake teams make is caching the wrong things. They cache database calls indiscriminately and wonder why memory grows without latency gains.

Use distributed tracing to find the actual bottlenecks. LightTrace traces each database query as a span within a transaction, showing you:

• Which queries run most often.
• Which queries take the longest.
• Which queries run repeatedly in a single request.

In the transaction waterfall, you'll spot patterns like "SELECT * FROM users" appearing 50 times in 100ms. That's your caching target. Slow database queries that don't repeat are a different problem (indexes, query rewrites, or schema changes—not caching).

By examining transaction p95 and p99 latency, you can measure the before-and-after impact of a caching change. Set the baseline, add caching, and see whether your p99 actually improves.

Common Caching Mistakes

1. Caching too aggressively. Cache every query and your memory explodes. Cache with purpose: high-frequency, low-change data only.

2. Forgetting cache invalidation. A cached value that never expires is a time bomb. Use TTLs religiously, and for critical data, implement explicit invalidation on updates.

3. Sharing cache across tenants. In a multi-tenant app, one tenant's "John Smith" must not appear in another tenant's session. Namespace your cache keys: user:tenant-123:john-smith.

4. Ignoring cache-miss behavior. If Redis goes down or is empty, does your app gracefully fall back to the database, or does it crash? Add timeouts and fallbacks.

5. Not measuring improvement. Cache changes are worthless if you don't verify impact. Use tracing and APM metrics (throughput, latency percentiles) to confirm gains.

Putting It Together

Here's a typical caching strategy for a typical application:

• Immutable, global data (feature flags, config): in-memory with a 10-minute TTL or process-startup refresh.
• User-specific data (profiles, preferences): Redis with a 5-minute TTL, namespace by user ID.
• Content (product catalogs, article metadata): Redis with a longer TTL (30 minutes to 1 hour), invalidate on publish.
• Transient data (session tokens, rate-limit counters): Redis with short TTLs (seconds to minutes).
• Don't cache: real-time data (inventory counts), sensitive data (payment methods), or data that changes per-request.

And always measure. Use distributed tracing to confirm that your cache reduces latency where it matters. A well-executed caching strategy is one of the highest-ROI optimizations a backend engineer can do.