Observability

Your logs are a storage product now.

A log line feels free when a developer writes it. It becomes a product the moment a vendor charges to ingest it, index it, and keep it searchable. Datadog, Splunk, Elastic, and a bucket of archived objects are different invoices for the same habit: keeping more detail than anyone queries. The first cut is what you ingest. The second is how long it stays hot. A smaller archive helps only the copy you still retain.

Why do Datadog or Splunk log costs get so high?

You pay to take logs in and, separately, to keep them searchable. Indexing every line at full retention prices data nobody queries. High-cardinality fields multiply events. The contract rate is on your order form or the vendor's current pricing page. Cut ingest and hot retention before you negotiate the rate.

Teams say the same thing in public: the aggregator charges on volume, on events, and on retention, and someone has to draw a line. Production that ships debug logs, health checks, and a unique id on every field will index a firehose. Filtering by level, dropping noisy patterns, and sampling success while keeping errors are application changes. They reduce the bytes and the events before a vendor meter sees them. That is the highest-leverage work, and it does not require a new compressor.

Datadog publishes a pricing page that separates ingest from indexing and that prices indexing by retention. Splunk's price depends on the contract you signed. Elastic publishes a pricing page for its cloud products and is a different system again if you self-host. This page does not quote those rates. Rates move, and enterprise discounts make list price a rumor. Open the page or the order form, write the date, and attach it to the review. Then decide what you are willing to index.

A shorter hot window is not the same as deleting evidence. Many teams keep seven to thirty days searchable and archive the rest. The archive is a different product: object storage, a data lake, a flex or infrequent tier the vendor sells. Rehydration and archive search have their own fees on some platforms. Read that section of the pricing page before you promise an auditor you can query last year in the same UI. If you cannot, say so in the runbook.

Should I archive logs to S3 instead of indexing them?

Archive logs you must retain and rarely query. Keep a short hot window for the queries you actually run. Object storage is cheaper to hold and slower to ask questions of. You will pay to store the archive and to scan it when you do look. Compress the archive if the format has room. Measure that, do not assume it.

An S3 archive of logs is still an S3 bill. Capacity, requests, and any retrieval or transfer apply, which is the whole point of the S3 page on this site. Partition the archive so a question scans a day, not a year. Columnar formats and compression inside the file format reduce bytes scanned by engines such as Athena. Those are properties of the format you choose. They are not a universal ratio. A JSON dump left as one object per line will store and scan poorly no matter who you buy.

People on operations forums describe moving audit logs that are needed quarterly out of a hot index and into object storage, then pulling them back for the rare investigation. That pattern works when the rare investigation is allowed to be slower. It fails when the on-call expectation is still a sub-minute search over a year. Write the expectation down. If the expectation is hot search, you will pay for hot search. Compression does not turn an archive into an index.

Self-hosted Elasticsearch, Loki, or another stack trades license for disk, memory, and people. The disk is still bytes. Retention on a self-hosted cluster is a capacity plan. Compressing segments helps that disk. It does not remove the need to delete indexes you no longer query. Elastic's pricing page matters only if Elastic runs it for you. If you run it, your cloud disk pricing page matters instead.

Does compressing logs reduce what I pay to retain them?

It reduces the stored size of the copy you compress, if the logs are still compressible. Text logs often are. A vendor that already compresses at ingest may leave little room. Hot indexing is often priced on events or ingested volume before your archive pass. Shrink the archive you keep. Filter the stream you index.

Log text is repetitive. Host names, paths, and JSON keys repeat. General-purpose compressors do well on that, which is why logging pipelines already gzip batches. If your archive is those gzip objects, a second compressor has to beat gzip on that data to matter. Sometimes a different approach does. Sometimes it does not. The sample is a day of the archive as you store it, not a pretty log line pasted into a box. We will measure that sample if you send it. We will not promise a factor.

The hot path is a different meter. If the vendor charges on ingested gigabytes or indexed events, the win is fewer events and shorter retention, not a tighter archive of last year. Compressing what you already decided to index may not change an event price at all. Read the meter. If the line says events, stop sending events. If the line says stored gigabytes of archive, then size is the lever and compression is in play.

Small objects return as a problem. A gzip file per request will cost more in puts than it saves in capacity. Batch by time, compress the batch, and store one object. That advice is on the S3 page because it is an S3 fact. Logs happen to be how teams accidentally create billions of keys. The archive design should look like a warehouse, not like a trash can of envelopes.

How long should I keep logs?

Keep hot logs for the window you actually search during incidents. Keep archives for the window a contract, a regulator, or a real investigation requires. Everything else is habit. Write the two windows down, price them on today's rate card, and delete or sample outside them. A longer window is a purchase, not a default.

Security teams and product teams pull in opposite directions. One wants every authentication event for a year. One wants debug logs from this morning. Both can be right if they do not share a tier. Authentication events can be a small, long archive. Debug logs can be a large, short hot window, or absent in production. The expensive mistake is one pipeline that treats them as the same object with the same retention.

Sampling is honest when you label it. Keeping every error and a fraction of successes is a product choice. Pretending the sample is the population will mislead an incident. Document the sample rate next to the dashboard. Do not call sampling compression. Compression returns the same bytes. Sampling does not.

When the windows are set, price them. Hot retention times the vendor's index rate. Archive retention times the object-storage rate plus an occasional scan. Put your rates in the calculator if you want a yearly figure at a ratio you assume for the archive only. Leave the hot event price out of that calculator. It does not know how to price an event. If the archive is the project, send us a day of it.

Questions.

Should I delete debug logs in production?

If nobody queries them during an incident, yes. Keep info or warning and above in production, and let lower environments log more. That is the line most teams draw when the bill is real. It reduces ingest before any archive or compressor sees the data.

Is an S3 log archive a Datadog replacement?

It replaces long retention, not live debugging. You gain cheap bytes and lose interactive search unless you add a query engine and accept scan cost. Keep a hot window for on-call. Archive what you must hold. Read Datadog's current pricing page before you compare the two bills.

Will qzip cut my logging bill in half?

We will not say that. Half would be a measurement on your logs, and we have not seen them. Text logs can be compressible. Logs that are already compressed may not be. Send a sample of the archive or the ingest stream. We will measure that and tell you what came back.

Provider docs, not our price list.

Links checked Oct 21, 2026. Rates and rules on those pages can change.

Send us a sample.

We will measure it on your files. We will not quote a savings number we have not measured. Write hello@qzip.ai or join the waitlist.