Datasets
Datasets
The container events land in — how to name them, how many to have, and when to split.
A dataset is where events live. It is the unit of retention, of access control, and of the ['name'] reference at the top of every LSQL query. Sending to a name that does not exist creates it — there is nothing to provision.
Naming
- Lowercase letters, digits, hyphens and underscores. 1–128 characters.
- The name is permanent. Renaming means creating a new dataset and re-pointing your senders.
- Convention that ages well:
<system>-<environment>, e.g.api-gateway-prod,k8s-events-staging.
How many should you have?
Fewer than you think. Logstreem has no indexes, so a large dataset is not slower to query than a small one — splitting buys you nothing on performance. Split for the three reasons that actually matter:
| Reason to split | Example |
|---|---|
| Different retention | Audit logs at 400 days, debug logs at 7 |
| Different access | A dataset a contractor may read, one they may not |
| Different residency | EU customer events pinned to Frankfurt |
Do not split per service
One dataset per microservice is the most common mistake. It makes every cross-service query a
join, and correlating a request across services is the main thing you came here to do. Keep them together and filter with where service == "checkout".Limits
| Limit | Value |
|---|---|
| Datasets per organisation | 1,000 |
| Fields per dataset | 4,096 |
| Retention | 1–400 days |
| Event size | 1 MB |