Song (Amay) Zheng

Notes on Java concurrency and backend fundamentals

Notes I put together after a conversation that ran up and down the backend stack. Every Java block on this page compiles and runs on JDK 21, and the numbers quoted come from those runs.

See it running: beacon.amayz.dev is a small live app that uses all of this, a Find My at toy scale on Java 21, Cassandra and ZooKeeper. The source and the commit-by-commit walkthrough are at github.com/songz/beacon.

#A singleton, tested with threads

#The class

A singleton needs three things: a private constructor so nobody else can call new, a static field that holds the one instance, and a static accessor. The interesting part is the first call. If two threads both read null, both construct.

The version I reach for is the initialization-on-demand holder. The JVM initializes a class lazily, on first use, and guarantees that initialization runs exactly once under an internal lock. So the instance is built on the first call to getInstance(), with no lock on any later call.

#The test

The test releases 64 threads at the same instant and counts how many distinct instances come back. Two latches do the work: ready makes sure every thread is parked before the race starts, and go releases them together. Without a start gate the first thread often finishes before the last one starts, and a broken singleton passes by luck.

import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;

public class Solution {

    // Broken on purpose: check-then-act is not atomic.
    static class Naive {
        static final AtomicInteger built = new AtomicInteger();
        private static Naive instance;
        private Naive() { built.incrementAndGet(); sleep(5); }
        static Naive getInstance() {
            if (instance == null) instance = new Naive();
            return instance;
        }
    }

    // Thread-safe, lazy, lock-free: the holder idiom.
    static class Config {
        static final AtomicInteger built = new AtomicInteger();
        private Config() { built.incrementAndGet(); sleep(5); }
        private static class Holder { static final Config INSTANCE = new Config(); }
        static Config getInstance() { return Holder.INSTANCE; }
    }

    // Release n threads at once; return how many distinct instances came back.
    static int distinctInstances(Callable<Object> get, int n) throws Exception {
        ExecutorService pool = Executors.newFixedThreadPool(n);
        CountDownLatch ready = new CountDownLatch(n), go = new CountDownLatch(1);
        List<Future<Object>> futures = new ArrayList<>();
        for (int i = 0; i < n; i++) {
            futures.add(pool.submit(() -> { ready.countDown(); go.await(); return get.call(); }));
        }
        ready.await();          // all threads parked at the gate
        go.countDown();         // release them together
        Set<Object> seen = Collections.newSetFromMap(new IdentityHashMap<>());
        for (Future<Object> f : futures) seen.add(f.get());   // get() rethrows worker exceptions
        pool.shutdown();
        return seen.size();
    }

    public static void main(String[] args) throws Exception {
        System.out.println("naive:  " + distinctInstances(Naive::getInstance, 64) + " instances, built " + Naive.built.get());
        int ok = distinctInstances(Config::getInstance, 64);
        System.out.println("holder: " + ok + " instance, built " + Config.built.get());
        if (ok != 1 || Config.built.get() != 1) throw new AssertionError("singleton broken");
    }

    static void sleep(long ms) { try { Thread.sleep(ms); } catch (InterruptedException e) { Thread.currentThread().interrupt(); } }
}

On JDK 21 the naive version returned 63 distinct instances out of 64 threads. The holder returned 1.

A few details that matter. The sleep(5) stands in for real constructor work; without it the race window is nanoseconds and the broken version can pass. Membership is by identity, because a class that overrides equals could make two instances look like one. And Future.get() rethrows anything a worker threw, so a crashed thread fails the test instead of vanishing.

#Five ways to write it

Form Lazy Thread-safe Cost
Eager static final INSTANCE = new X() no yes, class init is locked by the JVM built at class load even if unused
static synchronized getInstance() yes yes a lock on every call, forever
Double-checked locking yes only with volatile the most code
Holder idiom yes yes, lock-free none worth naming
enum X { INSTANCE; } at enum init yes also survives serialization and reflection

Double-checked locking needs volatile because instance = new X() is three steps: allocate, run the constructor, publish the reference. Without volatile the reference can be published before the constructor finishes, and a second thread that skips the lock uses a half-built object. The volatile write creates a happens-before edge, so any thread that reads a non-null reference sees a fully constructed object.

public class Dcl {
    private static volatile Dcl instance;
    private Dcl() {}

    public static Dcl getInstance() {
        Dcl local = instance;               // one volatile read on the fast path
        if (local == null) {
            synchronized (Dcl.class) {
                local = instance;
                if (local == null) instance = local = new Dcl();
            }
        }
        return local;
    }

    public static void main(String[] args) {
        if (getInstance() != getInstance()) throw new AssertionError();
        System.out.println("dcl ok");
    }
}

A singleton can still be broken: reflection can call the private constructor, deserialization creates a new object unless readResolve() returns the instance, and each class loader gets its own copy. An enum handles the first two for free.

#Where the shape shows up in real services

The pattern matters less as a class and more as a rule: build the expensive, thread-safe thing once per process and share it.

#Threads

#Ways to start concurrent work

Four generations. Extend Thread and override run. Pass a Runnable to a Thread, usually as a lambda, which separates the task from the thread. Submit Runnable or Callable tasks to an ExecutorService; a Callable returns a value and can throw, and you get a Future back. Compose async steps with CompletableFuture. And since Java 21, virtual threads.

My default is an ExecutorService: a pool reuses threads instead of paying creation cost per task, bounds concurrency so a spike can't create ten thousand OS threads, and gives futures, cancellation and a clean shutdown. For I/O-bound work on Java 21, a virtual-thread-per-task executor, because blocking a virtual thread is cheap. Always start(), never run(): run() executes on the caller's thread.

import java.util.concurrent.*;

public class ThreadWays {
    public static void main(String[] args) throws Exception {
        Thread a = new Thread() { @Override public void run() { System.out.println("1 subclass"); } };
        Thread b = new Thread(() -> System.out.println("2 runnable"));
        a.start(); b.start(); a.join(); b.join();

        try (ExecutorService pool = Executors.newFixedThreadPool(4)) {     // Java 19+: close() shuts it down
            Future<Integer> f = pool.submit(() -> 6 * 7);                  // Callable<Integer>
            System.out.println("3 executor " + f.get());
        }

        String s = CompletableFuture.supplyAsync(() -> "report").thenApply(x -> x + " decrypted").join();
        System.out.println("4 " + s);

        Thread v = Thread.ofVirtual().start(() -> System.out.println("5 virtual"));
        v.join();
    }
}

#Sizing a pool

CPU-bound work: about the number of cores. I/O-bound work: more, roughly cores times (1 + wait time / compute time), or virtual threads and skip the math. A ThreadPoolExecutor with a bounded queue and CallerRunsPolicy gives natural backpressure; newFixedThreadPool's unbounded queue hides overload until memory runs out.

#Race conditions

#The problem and the fixes

A race condition is shared mutable state where at least one thread writes and the result depends on timing. count++ is the classic case: read, add, write, three steps, and two threads can both read 5 and both write 6.

Fixes, in the order I reach for them. Don't share: confine data to one thread, or make it immutable. Use a thread-safe building block: AtomicInteger for counters, ConcurrentHashMap with compute, merge or computeIfAbsent for compound map updates, BlockingQueue for handoff. Lock with synchronized or ReentrantLock around the whole critical section when several fields must change together. volatile gives visibility and ordering, not atomicity: right for a stop flag, wrong for a counter. And every check-then-act has to be one atomic step; if (!map.containsKey(k)) map.put(k, v) races even on a ConcurrentHashMap, putIfAbsent doesn't.

import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;

public class Race {
    static int unsafe = 0, locked = 0;
    static final Object LOCK = new Object();
    static final AtomicInteger atomic = new AtomicInteger();

    public static void main(String[] args) throws Exception {
        try (ExecutorService pool = Executors.newFixedThreadPool(8)) {
            for (int t = 0; t < 8; t++) pool.submit(() -> {
                for (int i = 0; i < 100_000; i++) {
                    unsafe++;                                  // lost updates
                    synchronized (LOCK) { locked++; }          // mutual exclusion + visibility
                    atomic.incrementAndGet();                  // lock-free compare-and-swap
                }
            });
        }
        System.out.println("unsafe " + unsafe + ", synchronized " + locked + ", atomic " + atomic.get() + " (expected 800000)");
        if (locked != 800_000 || atomic.get() != 800_000) throw new AssertionError();
    }
}

In one run, the unsafe counter came out at 796,277 of 800,000.

#The toolbox

#Deadlock

Four conditions hold together: mutual exclusion, hold-and-wait, no preemption, circular wait. Break the cycle by acquiring locks in one global order (by id, for example), or use tryLock with a timeout and back off. jstack <pid> prints "Found one Java-level deadlock" with the threads and locks involved.

#Constants and static initialization

A constant is static final: one per class, assigned exactly once. A primitive or String literal is a compile-time constant and gets inlined into callers. Static logic goes in a static initializer block, which runs once when the class is initialized. final only stops reassignment; List.of and List.copyOf also stop mutation.

import java.util.*;

public class Constants {
    static final int MAX_RETRIES = 3;                 // compile-time constant, inlined
    static final Map<String, Integer> PORTS = Map.of("https", 443, "http", 80);
    static final List<String> REGIONS;                // blank final, assigned in the block

    static {                                          // runs once, at class initialization
        List<String> r = new ArrayList<>();
        r.add("us-west-1");
        r.add("eu-central-1");
        REGIONS = List.copyOf(r);
    }

    public static void main(String[] args) {
        System.out.println(MAX_RETRIES + " " + PORTS.get("https") + " " + REGIONS);
        try { REGIONS.add("x"); throw new AssertionError("should be immutable"); }
        catch (UnsupportedOperationException expected) { System.out.println("immutable ok"); }
    }
}

A class initializes on first active use: creating an instance, calling a static method, or touching a non-constant static field. Superclass first, then static fields and blocks top to bottom. That laziness is exactly what the holder singleton relies on.

#The JVM

#Heap size

JVM flags on the command line: -Xms for the initial heap and -Xmx for the maximum. In production I set them equal so the heap never resizes under load. In a container, -XX:MaxRAMPercentage=75 lets the heap follow the pod's memory limit; the JVM has been container-aware since JDK 10. The rest of the memory goes to metaspace, thread stacks (-Xss each), direct buffers and the code cache. Add -XX:+HeapDumpOnOutOfMemoryError and GC logging with -Xlog:gc*.

public class HeapInfo {
    public static void main(String[] args) {
        Runtime rt = Runtime.getRuntime();
        long mb = 1024 * 1024;
        System.out.println("max heap (-Xmx)   " + rt.maxMemory() / mb + " MB");
        System.out.println("committed heap    " + rt.totalMemory() / mb + " MB");
        System.out.println("cpus              " + rt.availableProcessors());
    }
}

#Memory areas and GC

#The memory model in one paragraph

Without synchronization there's no guarantee one thread sees another's writes: the compiler and CPU reorder and cache. The rule is happens-before. An unlock happens-before a later lock of the same monitor; a volatile write happens-before a later read of it; Thread.start() happens-before anything in the started thread; everything in a thread happens-before another thread's join() on it returns; and final fields set in a constructor are visible to any thread that sees the object. Two accesses to the same field that aren't ordered this way, with at least one write, are a data race.

#Non-blocking IO and Netty

Blocking IO is a thread per connection: the thread sleeps while waiting on the socket, so 10,000 connections need 10,000 threads. Non-blocking IO puts sockets in non-blocking mode behind a Selector (epoll on Linux), so one thread watches thousands of channels and acts only when one is ready.

Netty builds on that. A boss event-loop group accepts connections and hands each channel to a worker event loop, one thread that owns that channel for life, so a channel's handlers never run concurrently and need no locks. Each channel has a pipeline of handlers (decoders, business logic, encoders), and data moves in pooled, reference-counted ByteBufs. The rule that follows: never block an event-loop thread. A slow database call on it stalls every connection that loop owns, so blocking work goes to a separate executor and the result is written back to the channel.

#Backend fundamentals

#Horizontal scaling

More machines behind a load balancer instead of a bigger machine. It works cleanly when the service is stateless and state lives in a shared store. The costs: state has to move out of the process or be partitioned, you inherit partial failure and consistency problems, and the data tier becomes the bottleneck. Vertical scaling is simpler but has a ceiling and a single point of failure.

#Sharding

A good shard key has high cardinality, spreads access evenly, and keeps the hot queries on one shard.

#Protecting an API from overload

Rate limits and quotas per key, answered with 429 and Retry-After. Pagination with a maximum page size. Request size limits and timeouts. Idempotency keys so retries are safe. Load shedding when queues fill. And on the client side, exponential backoff with jitter so retries don't synchronize.

For a slow downstream dependency: a timeout on every call, a circuit breaker so the slow path fails fast instead of holding threads, a bulkhead (a separate bounded pool per dependency), retries with backoff only on idempotent calls, and a fallback or cached response while the breaker is open.

#Rate-limiting algorithms

Across a fleet, the counters live in Redis with an atomic increment or a Lua script. A thread-safe token bucket in Java:

import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.LongSupplier;

public class TokenBucket {
    private final long capacity;
    private final double refillPerNano;
    private final LongSupplier clock;     // injected so tests control time
    private double tokens;
    private long lastRefill;

    TokenBucket(long capacity, double refillPerSec, LongSupplier nanoClock) {
        this.capacity = capacity;
        this.refillPerNano = refillPerSec / 1_000_000_000.0;
        this.clock = nanoClock;
        this.tokens = capacity;
        this.lastRefill = nanoClock.getAsLong();
    }

    // Lazy refill: no timer thread, tokens are computed from elapsed time.
    synchronized boolean tryAcquire() {
        long now = clock.getAsLong();
        tokens = Math.min(capacity, tokens + (now - lastRefill) * refillPerNano);
        lastRefill = now;
        if (tokens >= 1) { tokens -= 1; return true; }
        return false;
    }

    public static void main(String[] args) throws Exception {
        long[] now = {0};
        TokenBucket b = new TokenBucket(5, 2.0, () -> now[0]);  // burst 5, 2 per second
        int allowed = 0;
        for (int i = 0; i < 10; i++) if (b.tryAcquire()) allowed++;
        now[0] += TimeUnit.SECONDS.toNanos(1);
        int refilled = 0;
        for (int i = 0; i < 10; i++) if (b.tryAcquire()) refilled++;

        TokenBucket shared = new TokenBucket(100, 0, () -> 0L);
        AtomicInteger wins = new AtomicInteger();
        try (ExecutorService pool = Executors.newFixedThreadPool(16)) {
            for (int t = 0; t < 16; t++) pool.submit(() -> { for (int i = 0; i < 50; i++) if (shared.tryAcquire()) wins.incrementAndGet(); });
        }
        System.out.println("burst " + allowed + ", after 1s " + refilled + ", 16 threads won " + wins.get() + " of 100");
        if (allowed != 5 || refilled != 2 || wins.get() != 100) throw new AssertionError();
    }
}

#Hashing, encoding, encryption

TLS 1.3 shows all of it working together. An ECDHE key exchange agrees on a shared secret, the server proves its identity by signing with the private key behind its X.509 certificate, both sides derive session keys with HKDF, and the data is encrypted with AES-GCM or ChaCha20-Poly1305. Asymmetric crypto authenticates and agrees on a key; symmetric crypto encrypts the data because it's far faster.

#One file, many recipients: envelope encryption

To share one large file with ten services, each with its own key pair, encrypt the file once and only the key ten times:

  1. Generate a random 256-bit data key and encrypt the file with it (AES-256-GCM).
  2. For each service, encrypt ("wrap") the 32-byte data key with that service's public key (RSA-OAEP, or ECIES).
  3. Send one ciphertext plus ten wrapped keys. Each service unwraps its copy with its private key and decrypts the file.

This is hybrid encryption. AWS KMS calls it envelope encryption (GenerateDataKey), and TLS, PGP and S/MIME all work the same way.

#Cassandra

Leaderless: every node is equal, and any node can coordinate a request. Nodes sit on a token ring; the partition key is hashed with Murmur3 and the token decides which node owns it, with virtual nodes for balance. Data is replicated to RF nodes across racks and data centers, and nodes learn cluster state through gossip. Consistency is tunable per query (ONE, QUORUM, LOCAL_QUORUM, ALL); with R + W > RF, reads see the latest write. Writes go to a commit log and a memtable, flush to immutable SSTables, and compaction merges them, which is why writes are fast. Hinted handoff, read repair and Merkle-tree anti-entropy repair keep replicas converged.

It fits append-heavy, time-bounded data keyed by something opaque: a partition key on that key (or its hash), a clustering key on time, and a table TTL for retention.

#Kafka

Producers write records to topics. A topic is split into partitions, each an ordered, append-only log; ordering holds only within a partition, and the record key picks the partition. Brokers store partitions with a leader and follower replicas, and the in-sync replica set decides what's committed. Consumers read in consumer groups, one consumer per partition per group, so partitions are the unit of parallelism. Consumers track progress with committed offsets, and retention is by time or size rather than consumption, so data can be replayed.

#ZooKeeper

A distributed coordination service: a small, strongly consistent tree replicated with a consensus protocol (ZAB), used for leader election, configuration, membership and locks. Kafka used it to hold cluster metadata and elect the controller; KRaft replaced it with a built-in Raft quorum (production-ready in 3.3, and ZooKeeper mode is gone in 4.0). etcd, which backs Kubernetes, and Consul fill the same role.