The Role of Commands and Events in a Distributed System

As we build larger, more distributed systems, the way we communicate between services shapes our architecture. Understanding the distinction between commands and events helps in designing effective communication patterns. Microservices and distributed systems introduce layers of complexity that require careful consideration of how components interact.

It might not be immediately clear what is the difference between Commands and Events. Fundamentally both are messages that carry information through a system, they are data payloads being transmitted. They often utilize the same shared data models like Money or Address, require serialization for network transmission (JSON, Protobuf, AVRO), and need to support schema evolution with backward compatibility.

However, their semantic differences run much deeper than their technical similarities. While both commands and events help reduce coupling, they serve fundamentally different roles with distinct implications for system design.

Commands

Command invokes behavior in the system. It expresses an intent to perform a specific action. Commands are handled by a single consumer, though they can have multiple producers. The naming convention for commands uses verbs in the present tense, such as RequestRefund, ScheduleMeeting, DeactivateUser, or PayBill. The schema of a command is owned by the consumer, this means that the consumer defines the structure of the command to which senders must adhere. Commands can be validated and rejected by the consumer, they can also fail during execution.

syntax = "proto3";

package users.commands.v1;

message DeactivateUser {
  string user_id = 1;
  string reason = 2;
  string correlation_id = 3;
  string initiated_by = 4;
}

Commands can be synchronous, sent via REST API or gRPC, where the sender expects immediate feedback about success, failure, or at least acknowledgment that the command was received and is being processed. Whether a command handler synchronously returns a reply or just initiates an asynchronous process typically depends on the desired user experience and technical constraints. When immediate feedback is important or the operation is quick, the command handler can perform the task and return a result synchronously. When the operation is long-running, resource-intensive, or depends on external systems, the command handler may instead start an asynchronous process, immediately acknowledge the command to the sender.

An example of a synchronous command in a user-facing API is a GraphQL mutation input that maps to a command object and is passed to a command handler. The GraphQL resolver acts as the presentation layer that transforms and optionally validates the input, and the handler executes the business logic, which maintains separation of concerns aligned with CQRS and layered architecture principles.

An internal API can follow the same pattern with gRPC by treating the implementation of gRPC service interface as a thin presentation layer where request messages map directly to command types. The service receives a request, optionally validates it, and dispatches the command to a handler in the application layer, providing indirection between the transport interface and the business logic.

You might run into the fire-and-forget pattern, where a command is sent and the sender doesn’t wait for any acknowledgment or result. In practice, I’d model those interactions as events so the producer and consumers stay loosely coupled, and so delivery, retries, and fan-out can be handled by messaging infrastructure.

In systems following CQRS and DDD, commands and command handlers are associated with the application layer. Command handlers are functionally equivalent to application services.

Events

Events tell other parts of the system that something happened, they signal state change in the system. Events are broadcast and a single event can have multiple consumers across different boundaries. This lets other components react without tight coupling to the producer. Like an orchestra, where the conductor’s gesture broadcasts a cue and musicians respond in real time.

The naming convention for events uses the past tense, for example RefundIssued, MeetingScheduled, UserDeactivated, or BillPaid, which reinforces that an event records a fact that has already occurred rather than an intention to act. The schema of an event is owned by the producer, which means the producer defines the structure of the event to which consumers must adhere.

syntax = "proto3";

package users.events.v1;

message UserDeactivated {
  string user_id = 1;
  google.protobuf.Timestamp occurred_at = 2;
  string reason = 3;
  string correlation_id = 4;
  string deactivated_by = 5;
}

Producers can also consume own events. The Listen to Yourself pattern is especially useful in Event Sourcing scenarios where events are the source of truth and state is derived from those events. This approach removes the dual-write problem and eliminates the need for Outbox. While it can also benefit scalability and improves resiliency, it shifts the consistency mechanism from the database and introduces eventual consistency in the producer boundary.

Since events represent facts and are immutable, they cannot be rejected. Consumers may fail while processing events and should implement retry mechanisms and idempotent handling to ensure reliable processing.

Events belong in the domain layer, they are raised by domain entities as part of the domain model behavior signaling state changes and relevant business outcomes. Event handlers belong to the application layer, similar to command handlers.

Async Commands, Correlation, Message IDs

In message-based systems, you might encounter commands being sent through a message broker without the sender waiting for an immediate response. This pattern makes commands behave more like events in terms of transport mechanism, which sometimes blurs the distinction between commands and events, the key distinction remains in the message’s semantics and the clear intent behind it.

Pursuing this approach introduces complexity, as you now need a mechanism to correlate asynchronous requests and responses if the sender expects feedback. In traditional synchronous commands, responses arrive immediately on the same connection. With asynchronous messaging, responses come later, often delivered via a separate channel such as an event. Correlation IDs are used to link the original request with its eventual outcome.

Asynchronous messaging also brings the possibility of duplicate message delivery due to network issues, broker restarts, or retry logic. Both asynchronous commands and events typically require message IDs to enable idempotent handling, ensuring repeated messages don’t trigger unintended side effects.

messageID, _ := uuid.NewV7()
correlationID, _ := uuid.NewV7()

// Define headers (Kafka example)
headers := []kgo.RecordHeader{
    {Key: "message_id", Value: []byte(messageID.String())},
    {Key: "correlation_id", Value: []byte(correlationID.String())},
    {Key: "command_type", Value: []byte("DeactivateUser")},
}

Because both message ID and correlation ID are artifacts of the asynchronous messaging rather than features of the event/command model, they are usually included in the message metadata rather than the payload. Most transport mechanisms, such as Kafka, RabbitMQ, or Pub/Sub support message headers or metadata where you can include this information without polluting the actual command or event schema.

Integration of Commands and Events

Commands often trigger events, creating paired relationships that help model intent and outcomes in systems using CQRS or event-driven patterns. For instance, a DeactivateUser command leads to an actual state change, which the system then communicates by publishing a UserDeactivated event or similar.

Key Characteristics

Conclusion

A useful, if imperfect, heuristic is that commands tell a specific consumer to perform an action. Events, in contrast, broadcast that something has already happened and represent a non-negotiable fact. Even if a business process expects another part of the system to act on an event, that responsibility lies with the consumer, not the event publisher.

Ultimately, robust systems use a thoughtful mix of both patterns. Commands drive actions at the boundaries of your services, while events propagate the results of those actions throughout the system. Mastering this fundamental distinction is therefore critical for designing architectures that are resilient, scalable, and maintainable.

Conductor’s gesture initiates a musical phrase, serving as a broadcast signal to which appropriate sections attune and respond in real time. Photo by Arindam Mahanta on Unsplash.