Logstreem
Kafka

Kafka

A Connect sink for topics you already publish to.

The Logstreem sink connector reads topics and forwards records to ingest, mapping topic to dataset.

{
  "name": "logstreem-sink",
  "config": {
    "connector.class": "com.logstreem.kafka.LogstreemSinkConnector",
    "tasks.max": "3",
    "topics": "app-logs,audit-events",
    "logstreem.token": "${env:LOGSTREEM_TOKEN}",
    "logstreem.dataset.map": "app-logs:api-gateway-prod,audit-events:audit-trail",
    "logstreem.timestamp.field": "ts",
    "batch.size": "1000",
    "linger.ms": "2000",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false"
  }
}

Map topics to few datasets

logstreem.dataset.map lets you fan many topics into one dataset. Prefer that over a dataset per topic — see datasets for why.
PreviousSyslogNextElasticsearch bulk