Batching and buffering

How to size batches, when to flush, and what to do when ingest is unreachable.

One event per HTTP request works, and will be the thing that breaks first. Batching is the difference between 6,000 requests a minute and six.

Sensible targets

Setting	Start at	Why
Batch size	1,000 events	Well under the 10,000 cap, still one round trip
Flush interval	2 seconds	Bounded lag without a request per event
Max payload	4 MB compressed	Half the 10 MB cap, leaves headroom for a fat event
Compression	gzip, always	Structured logs compress 8–15x; it is free latency

Flush on whichever comes first

Node

const MAX_EVENTS = 1000;
const MAX_WAIT_MS = 2000;

let buffer: unknown[] = [];
let timer: NodeJS.Timeout | null = null;

export function log(event: unknown) {
  buffer.push(event);
  if (buffer.length >= MAX_EVENTS) return flush();
  timer ??= setTimeout(flush, MAX_WAIT_MS);
}

async function flush() {
  if (timer) { clearTimeout(timer); timer = null; }
  if (!buffer.length) return;

  const batch = buffer;
  buffer = [];

  try {
    await fetch("https://api.logstreem.com/v1/ingest/api-gateway-prod", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.LOGSTREEM_TOKEN}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify(batch),
    });
  } catch {
    // Bounded retry buffer: drop oldest rather than grow without limit.
    buffer = [...batch.slice(-5000), ...buffer];
  }
}

// Do not lose the tail on shutdown.
process.on("beforeExit", flush);

Bound your retry buffer

An unbounded in-memory buffer during an ingest outage is how a logging library takes down the service it was meant to observe. Cap it, drop the oldest, and count the drops.

Ordering does not matter

Events are placed by their _time, not by arrival order. You can retry a failed batch after later batches have landed and queries still read correctly — see events.

PreviousHTTP ingest NextOpenTelemetry