Supported log types

Python tracebacks

Detected by: traceback headers, Python file-frame lines, common exception classes, and dependency paths such as site-packages or virtualenv directories.

Preserved: command, final exception, non-dependency user frames, failing import or call line, wrapper exit status, and selected environment hints.

Folded: dependency frames, repeated warnings, startup chatter, and wrapper noise that does not change the exception.

Limit: if the traceback is truncated before the exception line, tokensift may only report that a Python traceback was detected.

See the Python traceback cleaner.

CUDA / PyTorch OOM

Detected by: explicit CUDA out-of-memory messages, torch OOM exceptions, allocation lines, allocated/reserved memory lines, NCCL/cuDNN hints, and GPU names.

Preserved: training command, model and batch-size hints, precision, user call site, OOM line, GPU total/free/allocated/reserved memory stats, and version/environment hints.

Folded: progress bars, wandb chatter, repeated NCCL info, worker/dataloader lines, and torch dependency internals.

Limit: tokensift can preserve the memory evidence, but it cannot know whether the fix should be batch reduction, checkpointing, allocator settings, or model changes without project context.

See the CUDA OOM log cleaner.

TypeScript, npm, Vite, and Next.js builds

Detected by: TS error codes, assignability errors, missing modules, Vite/Next/webpack/tsc output, and npm/pnpm/yarn failure lines.

Preserved: build command, source location, TS code, primary type error, expected/actual type hints, import traces, package-manager failure, and relevant tool versions.

Folded: asset output, cache output, successful compilation noise, node_modules chatter, and unrelated lint warnings.

Limit: a single TS error can depend on type definitions far from the logged file. Include the surrounding source or type declaration when the retained brief is not enough.

Docker builds

Detected by: docker build commands, BuildKit step markers, Dockerfile line references, failed solve messages, executor failures, cached layers, and buildkit output.

Preserved: build command, failing BuildKit step, Dockerfile context, failed RUN command, package-manager error, base image metadata, and exit code.

Folded: cached/export layers, metadata transfer lines, package download/install progress, npm warning noise, and successful layer output.

Limit: private registry, network, and credential failures often need surrounding config that should be reviewed and redacted manually before sharing.

See the Docker build log cleaner.

pytest failures

Detected by: pytest commands, FAILURES or ERRORS sections, short test summary info, captured stdout/stderr/log blocks, pytest file paths, and assertion lines.

Preserved: failing test node id, user frames, assertion expression, expected/actual evidence, fixture/setup/teardown clues, and the short summary. A line from a captured block survives only when it independently matches one of those retained evidence classes.

Folded: passing test progress, repeated warnings, captured logs that do not change the assertion, and unrelated test-run noise.

Limit: if fixture state or mocked data lives outside the log, include a short note or source excerpt after reviewing the brief.

JavaScript and E2E test failures

Detected by: npm/pnpm/yarn test commands, FAIL lines for spec/test files, expect(received) output, expected/received strings, and failed/passed summaries.

Preserved: failing spec path, source line, assertion text, expected/received values, and run summary.

Folded: worker/server/browser noise, download progress, duplicate run lines, and repeated status messages.

Limit: screenshot, trace, DOM, or network evidence is not represented unless the text log names it. Add a concise artifact note when visual state matters.

Generic terminal logs

Detected by: fatal, error, failed, denied, timeout, missing, conflict, segmentation fault, OOM, traceback, and similar high-signal terms when no stronger parser wins.

Preserved: command, nearby error context, selected environment lines, exit/status/errno clues, and the final relevant log tail.

Folded: debug/info chatter, heartbeat/polling/watching output, progress output, duplicate lines, and standalone timestamps.

Limit: generic parsing is intentionally conservative. It does not infer domain-specific root causes from unfamiliar log formats.

Package installs, permission errors, and timeouts

Handled through: Docker, TypeScript, Python, or generic parsing depending on the wrapper that produced the failure.

Preserved: failing package name, resolver error, denied path, missing environment variable, auth or registry failure, timeout text, exit code, and command when present.

Folded: successful downloads, retries that repeat the same message, progress bars, and status-only lines.

Limit: tokensift cannot safely decide whether a permission or auth failure is caused by local permissions, CI secrets, registry policy, or network policy. Add the minimal relevant context yourself.