Producer-Consumer with Coroutines

The producer-consumer pattern is one of the most fundamental concurrency patterns in software development, and implementing producer-consumer coroutines in Lua is remarkably straightforward. It solves the problem of coordinating between a component that generates data and another that processes it. Coroutines provide an elegant solution without the complexity of threads.

Understanding the Problem

In a typical producer-consumer scenario:

• The producer generates or fetches data
• The consumer processes that data
• Both run at different speeds, so you need a way to coordinate them

Traditional approaches use threads, but Lua offers a simpler alternative: coroutines. Unlike threads, coroutines are cooperative—they yield control explicitly, making coordination straightforward.

Lua coroutine basics

Before diving into the pattern, let’s review the core coroutine functions:

-- Create a coroutine
local co = coroutine.create(function()
    print("Inside coroutine")
end)

-- Check status
print(coroutine.status(co))  -- "suspended"

-- Resume (start) the coroutine
coroutine.resume(co)  -- Prints: Inside coroutine
print(coroutine.status(co))  -- "dead"

The key functions are:

• coroutine.create(f) — Creates a new coroutine from a function
• coroutine.resume(co [, ...]) — Starts or resumes a coroutine
• coroutine.yield([...]) , Suspends the coroutine, passing values back
• coroutine.status(co) , Returns “suspended”, “running”, or “dead”

The producer-consumer pattern

Here’s how coroutines solve the producer-consumer problem:

-- Producer: generates values
local function producer()
    return coroutine.create(function()
        local values = {"apple", "banana", "cherry", "date"}
        for _, v in ipairs(values) do
            coroutine.yield(v)  -- Send value to consumer
        end
    end)
end

-- Consumer: processes values
local function receive(prod)
    local status, value = coroutine.resume(prod)
    return value
end

-- Main consumer loop
local p = producer()
while true do
    local value = receive(p)
    if value == nil then break end
    print("Processing: " .. value)
end

How it works:

1. The consumer calls receive(), which resumes the producer
2. The producer runs, yields a value, and suspends
3. The consumer processes the value
4. Repeat until the producer yields nil

A practical example

Let’s build a more realistic example, a file processor:

-- Simulated file reader (producer)
local function fileProducer(filename)
    return coroutine.create(function()
        -- In real code, you'd read from a file
        local lines = {
            "First line of data",
            "Second line of data", 
            "Third line of data"
        }
        for _, line in ipairs(lines) do
            coroutine.yield(line)
        end
    end)
end

-- Data processor (filter)
local function lineProcessor(prod)
    return coroutine.create(function()
        while true do
            local status, line = coroutine.resume(prod)
            if not line then break end
            
            -- Transform: uppercase and add prefix
            local processed = "[PROCESSED] " .. string.upper(line)
            coroutine.yield(processed)
        end
    end)
end

-- Consumer: writes processed data
local function consumer(prod)
    while true do
        local status, data = coroutine.resume(prod)
        if not data then break end
        print(data)
    end
end

-- Run the pipeline
consumer(lineProcessor(fileProducer()))

When you run this pipeline, fileProducer generates raw lines one at a time, lineProcessor transforms each by uppercasing and prepending a label, and consumer prints the result. Notice how deeply nested the call is, consumer(lineProcessor(fileProducer())), which works because each function returns a coroutine that the next stage can resume. Lua’s coroutine.resume returns a boolean status as its first value, so checking if not line correctly catches both end-of-stream nils and actual errors in the coroutine.

Output:

Running the pipeline shown above prints three transformed lines, one per original input line. Each value moves through every stage before the producer generates the next one, keeping memory usage constant regardless of input size. The filter coroutine sits between the producer and consumer, calling coroutine.resume on the upstream coroutine and coroutine.yield to pass results downstream. This chaining pattern is the foundation for building multi-stage data processing pipelines that stay simple even as you add more transformation steps.

[PROCESSED] FIRST LINE OF DATA
[PROCESSED] SECOND LINE OF DATA
[PROCESSED] THIRD LINE OF DATA

Common Pitfalls

1. Forgetting to Resume

Calling coroutine.create only allocates a coroutine object and leaves it in the suspended state. Nothing happens until something, usually your consumer loop, calls coroutine.resume. A common mistake is writing a producer, creating it, and expecting data to flow automatically. Without an explicit resume call, the coroutine sits idle forever. The WRONG example below never calls resume, so the yield is never reached and no value ever emerges. The CORRECT version adds a single resume call that triggers execution and retrieves the yielded value.

-- WRONG: Producer never resumed
local p = coroutine.create(function()
    coroutine.yield("value")
end)
-- Producer stays suspended forever!

-- CORRECT
local p = coroutine.create(function()
    coroutine.yield("value")
end)
coroutine.resume(p)  -- Now it runs

2. Infinite loops without yield

When a coroutine runs, it takes over the calling thread of execution entirely, Lua won’t interrupt it. If you write a while true loop without a coroutine.yield inside, that coroutine runs indefinitely and your consumer never regains control. The entire program appears to hang. Always place a yield call inside any potentially unbounded loop so the consumer can process each value before requesting the next. In the example below, coroutine.yield(i) pauses the loop after each increment, allowing the caller to handle the number and decide whether to continue.

-- Producer that yields continuously
local function infiniteProducer()
    return coroutine.create(function()
        local i = 0
        while true do
            i = i + 1
            coroutine.yield(i)  -- Yield each value
        end
    end)
end

3. Status Confusion

A coroutine reports "suspended" in two distinct situations: right after creation, before anyone has called resume, and after a yield returns control to the caller. Beginners sometimes check the status and misinterpret "suspended" as meaning the coroutine has not been started, when in fact it may be mid-execution waiting for the next resume. The only way a coroutine leaves the suspended state is when its function body completes or errors out, at which point the status changes to "dead". The example below traces the status through each of these transitions, showing the three possible states a coroutine occupies during its lifecycle.

local co = coroutine.create(function()
    coroutine.yield("paused")
end)

print(coroutine.status(co))  -- "suspended" (before first resume)

coroutine.resume(co)
print(coroutine.status(co))  -- "suspended" (after yield)

coroutine.resume(co)
print(coroutine.status(co))  -- "dead" (function completed)

Why coroutines over threads?

Aspect	Coroutines	Threads
Control	Cooperative (explicit yield)	Preemptive (OS schedules)
Complexity	Simple	Requires locks, mutexes
Memory	Lightweight	Heavy (stack per thread)
Debugging	Easier	Harder (race conditions)

Coroutines are perfect when:

• You have clear control points where work can pause
• You don’t need true parallelism (single-core is fine)
• You want simple, predictable behavior

Advanced: multiple producers

In real applications, you often need multiple producers generating data concurrently. Here’s how to handle that:

-- Create multiple producers
local function numberProducer(start, count)
    return coroutine.create(function()
        for i = start, start + count - 1 do
            coroutine.yield(i)
        end
    end)
end

-- Multiplexer: combines multiple producers
local function multiReceive(producers)
    local function roundRobin()
        local i = 1
        while true do
            local prod = producers[i]
            local status, value = coroutine.resume(prod)
            
            if not value then
                -- This producer is done, remove it
                table.remove(producers, i)
                if #producers == 0 then return nil end
                i = i - 1
            else
                coroutine.yield(value)
            end
            
            i = i + 1
            if i > #producers then i = 1 end
        end
    end
    return coroutine.create(roundRobin)
end

-- Usage
local producers = {
    numberProducer(1, 5),
    numberProducer(100, 5),
    numberProducer(200, 5)
}
local combined = multiReceive(producers)

while true do
    local status, value = coroutine.resume(combined)
    if not value then break end
    print("Received: " .. value)
end

When to use this pattern

The producer-consumer pattern with coroutines is ideal for:

• IO-bound tasks: Reading files, making network requests, or processing database results
• Data pipelines: Transforming data through multiple stages
• Task queues: Distributing work across multiple workers
• Event handling: Processing events as they arrive

Summary

The producer-consumer pattern with Lua coroutines provides a clean way to coordinate data flow between components. Key takeaways:

1. Use coroutine.create() to make a function async
2. Use coroutine.yield() to pause and pass data
3. Use coroutine.resume() to continue execution
4. The consumer drives the flow by resuming the producer
5. Filters can transform data between producer and consumer

This pattern scales from simple examples to complex data processing pipelines, all without the complexity of traditional threading.

Next steps

• Continue the series with coroutine wrappers and iterators to learn how coroutine.wrap simplifies the patterns you’ve built here.
• Explore async patterns with coroutines for event-loop and async/await simulations built on the producer-consumer foundation.

See also

• Coroutine basics — Review the fundamental coroutine API before tackling advanced patterns
• Cooperative multitasking with coroutines — Scheduling multiple coroutines cooperatively
• Lua event systems guide — Event-driven architectures that pair well with producer-consumer pipelines