Logs you can query at three in the morning

Structured logs are objects with named fields, not sentences with values embedded. The test is whether you can count them: how many 500s on this route, for this tenant, in the last ten minutes. A prose log line cannot answer that without a regular expression written in a hurry, and hurry is the condition under which you will be writing it.

Correlation is the other half. A request id generated at the edge and attached to every line, including lines emitted by jobs that request enqueued, turns a haystack into a single filtered view. Add the user or tenant id where privacy permits, and the deploy version, so "did this start with the release?" is a filter rather than a debate.

Volume is a real cost, in money and in noise. Sample the routine, one in a hundred successful requests is plenty for baselines, and keep everything for errors and for slow requests. And decide deliberately what never gets logged: tokens, passwords, full bodies, personal data. Logs are shipped to third parties, retained for months and read by people who have no business seeing that.