Skip to main content

Kafka Topic Configuration

Kafka topics for pluggable workers are managed via YAML files in the Terraform repo. Backstage automatically generates these when you create a new worker, but you can also add or edit them manually.

Required Fields​

Each YAML file must include these two fields:

topic_name: backend.pluggable.myWorker.tasks
user_name: pa-myWorker-pluggable-worker

Drop this into pluggable-workers/<worker-name>.yaml. No other files need to be edited — Terraform automatically discovers all YAML files in pluggable-workers/.

Topic Naming​

All new worker topics follow the backend.pluggable.<name>.tasks pattern. Each worker's Kafka user is automatically granted ACL access to the backend.pluggable.* prefix, so any worker can call another worker's topic for deferred execution without additional Terraform changes.

For pre-existing topics that don't follow this pattern, you need to explicitly grant access using the topic_acls field.

Optional Fields​

The following fields can be added to override defaults or extend access.

environments​

Overrides the default topic settings for specific environments. By default, the topic is created in beb-01, beb-02, and beb-03 with these values:

environments:
  beb-01:
    partitions: 12
    replicas: 3
    retention_ms: "86400000"       # 24h
    cleanup_policy: "compact,delete"
  beb-02:
    partitions: 4
    replicas: 3
    retention_ms: "86400000"       # 24h
    cleanup_policy: "compact,delete"
  beb-03:
    partitions: 4
    replicas: 3
    retention_ms: "86400000"       # 24h
    cleanup_policy: "compact,delete"

To override, specify only the values you want to change:

environments:
  beb-01:
    partitions: 24
    replicas: 3
    retention_ms: "864000000"
note

The number of partitions should not be less than the number of worker instances you plan to run, so the load is distributed evenly. Ideally, choose a partition count that is divisible by your instance count.

topic_acls​

Grants the worker's Kafka user Read/Write/Describe/Create ACLs on additional topics. This is useful when your worker needs access to pre-existing topics that were created outside the pluggable-workers/ setup (e.g. legacy topics).

topic_acls:
  - backend.someOldPluggableWorker.tasks