Producer-Consumer with Coroutines

The producer-consumer pattern is one of the most fundamental concurrency patterns in software development. It solves the problem of coordinating between a component that generates data and another that processes it. In Lua, coroutines provide an elegant solution for this pattern without the complexity of threads.

Understanding the Problem

In a typical producer-consumer scenario:

• The producer generates or fetches data
• The consumer processes that data
• Both run at different speeds, so you need a way to coordinate them

Traditional approaches use threads, but Lua offers a simpler alternative: coroutines. Unlike threads, coroutines are cooperative—they yield control explicitly, making coordination straightforward.

Lua Coroutine Basics

Before diving into the pattern, let’s review the core coroutine functions:

-- Create a coroutine
local co = coroutine.create(function()
    print("Inside coroutine")
end)

-- Check status
print(coroutine.status(co))  -- "suspended"

-- Resume (start) the coroutine
coroutine.resume(co)  -- Prints: Inside coroutine
print(coroutine.status(co))  -- "dead"

The key functions are:

• coroutine.create(f) — Creates a new coroutine from a function
• coroutine.resume(co [, ...]) — Starts or resumes a coroutine
• coroutine.yield([...]) — Suspends the coroutine, passing values back
• coroutine.status(co) — Returns “suspended”, “running”, or “dead”

The Producer-Consumer Pattern

Here’s how coroutines solve the producer-consumer problem:

-- Producer: generates values
local function producer()
    return coroutine.create(function()
        local values = {"apple", "banana", "cherry", "date"}
        for _, v in ipairs(values) do
            coroutine.yield(v)  -- Send value to consumer
        end
    end)
end

-- Consumer: processes values
local function receive(prod)
    local status, value = coroutine.resume(prod)
    return value
end

-- Main consumer loop
local p = producer()
while true do
    local value = receive(p)
    if value == nil then break end
    print("Processing: " .. value)
end

How it works:

1. The consumer calls receive(), which resumes the producer
2. The producer runs, yields a value, and suspends
3. The consumer processes the value
4. Repeat until the producer yields nil

A Practical Example

Let’s build a more realistic example—a file processor:

-- Simulated file reader (producer)
local function fileProducer(filename)
    return coroutine.create(function()
        -- In real code, you'd read from a file
        local lines = {
            "First line of data",
            "Second line of data", 
            "Third line of data"
        }
        for _, line in ipairs(lines) do
            coroutine.yield(line)
        end
    end)
end

-- Data processor (filter)
local function lineProcessor(prod)
    return coroutine.create(function()
        while true do
            local status, line = coroutine.resume(prod)
            if not line then break end
            
            -- Transform: uppercase and add prefix
            local processed = "[PROCESSED] " .. string.upper(line)
            coroutine.yield(processed)
        end
    end)
end

-- Consumer: writes processed data
local function consumer(prod)
    while true do
        local status, data = coroutine.resume(prod)
        if not data then break end
        print(data)
    end
end

-- Run the pipeline
consumer(lineProcessor(fileProducer()))

Output:

[PROCESSED] FIRST LINE OF DATA
[PROCESSED] SECOND LINE OF DATA
[PROCESSED] THIRD LINE OF DATA

Common Pitfalls

1. Forgetting to Resume

A suspended coroutine won’t run unless resumed. Always ensure your consumer resumes the producer:

-- WRONG: Producer never resumed
local p = coroutine.create(function()
    coroutine.yield("value")
end)
-- Producer stays suspended forever!

-- CORRECT
local p = coroutine.create(function()
    coroutine.yield("value")
end)
coroutine.resume(p)  -- Now it runs

2. Infinite Loops Without Yield

If your producer runs forever, make sure it yields periodically:

-- Producer that yields continuously
local function infiniteProducer()
    return coroutine.create(function()
        local i = 0
        while true do
            i = i + 1
            coroutine.yield(i)  -- Yield each value
        end
    end)
end

3. Status Confusion

Remember: a coroutine is “suspended” before its first resume and after yield:

local co = coroutine.create(function()
    coroutine.yield("paused")
end)

print(coroutine.status(co))  -- "suspended" (before first resume)

coroutine.resume(co)
print(coroutine.status(co))  -- "suspended" (after yield)

coroutine.resume(co)
print(coroutine.status(co))  -- "dead" (function completed)

Why Coroutines Over Threads?

Aspect	Coroutines	Threads
Control	Cooperative (explicit yield)	Preemptive (OS schedules)
Complexity	Simple	Requires locks, mutexes
Memory	Lightweight	Heavy (stack per thread)
Debugging	Easier	Harder (race conditions)

Coroutines are perfect when:

• You have clear control points where work can pause
• You don’t need true parallelism (single-core is fine)
• You want simple, predictable behavior

Advanced: Multiple Producers

In real applications, you often need multiple producers generating data concurrently. Here’s how to handle that:

-- Create multiple producers
local function numberProducer(start, count)
    return coroutine.create(function()
        for i = start, start + count - 1 do
            coroutine.yield(i)
        end
    end)
end

-- Multiplexer: combines multiple producers
local function multiReceive(producers)
    local function roundRobin()
        local i = 1
        while true do
            local prod = producers[i]
            local status, value = coroutine.resume(prod)
            
            if not value then
                -- This producer is done, remove it
                table.remove(producers, i)
                if #producers == 0 then return nil end
                i = i - 1
            else
                coroutine.yield(value)
            end
            
            i = i + 1
            if i > #producers then i = 1 end
        end
    end
    return coroutine.create(roundRobin)
end

-- Usage
local producers = {
    numberProducer(1, 5),
    numberProducer(100, 5),
    numberProducer(200, 5)
}
local combined = multiReceive(producers)

while true do
    local status, value = coroutine.resume(combined)
    if not value then break end
    print("Received: " .. value)
end

When to Use This Pattern

The producer-consumer pattern with coroutines is ideal for:

• IO-bound tasks: Reading files, making network requests, or processing database results
• Data pipelines: Transforming data through multiple stages
• Task queues: Distributing work across multiple workers
• Event handling: Processing events as they arrive

Summary

The producer-consumer pattern with Lua coroutines provides a clean way to coordinate data flow between components. Key takeaways:

1. Use coroutine.create() to make a function async
2. Use coroutine.yield() to pause and pass data
3. Use coroutine.resume() to continue execution
4. The consumer drives the flow by resuming the producer
5. Filters can transform data between producer and consumer

This pattern scales from simple examples to complex data processing pipelines, all without the complexity of traditional threading.

See Also

• Lua Coroutine Reference (coming soon)
• More Lua tutorials (coming soon)