Technical overview

How tokensift turns failure logs into conservative debugging briefs

tokensift does not try to fix code. It prepares a smaller, evidence-preserving brief from raw terminal, editor, Docker, test, or CI output so a developer can review it before sharing it with a coding agent.

The core pipeline is deterministic: normalize the paste, fold obvious repetition, redact common secret patterns when redaction is enabled, detect the log type, run a type-specific parser, then render a Markdown brief with the sections a coding agent needs.

Open the main Sift tool Read the pipeline

Pipeline stages

1. Normalize and fold generic noise

The cleaner first standardizes line endings, removes null bytes, strips ANSI color/control codes, and handles obvious log-volume waste before parser-specific logic runs.

tqdm-style progress bars and carriage-return updates are folded.
Standalone timestamp-only lines are folded.
Repeated near-identical lines are capped based on the selected compression mode.
In aggressive mode, low-value debug and trace lines can be dropped earlier.

2. Redact common risky strings

When redaction is enabled, tokensift replaces common credential patterns before detection and rendering. The redaction pass covers private keys, bearer tokens, GitHub tokens, AWS access keys, Google API keys, OpenAI API keys, JWTs, database URLs, URL passwords, secret-style assignments, and direct prompt-injection directives.

This is a safety net, not a complete data-loss-prevention system. Review the generated brief before sending it anywhere.

3. Detect the outer failure type

The detector scores patterns for CUDA/PyTorch OOM, pytest, JavaScript/E2E tests, Docker builds, TypeScript/npm/Vite/Next.js builds, Python tracebacks, and generic terminal logs.

Two routing rules matter in noisy real logs: pytest wrappers win over domain-specific strings inside captured output, and explicit CUDA out-of-memory signatures win over ordinary Python traceback routing.

4. Parse for evidence, not summary prose

Each parser returns structured fields: command, environment lines, primary error, user frames, key details, folded groups, and parser notes. The renderer then turns those fields into the same conservative brief shape across log types.

Python keeps the final exception and non-dependency frames.
CUDA keeps memory stats, model/batch hints, and user call sites.
Docker keeps the failing build step, Dockerfile context, package error, and exit code.
TypeScript keeps TS codes, source locations, and expected/actual type hints.
pytest keeps failing tests, assertion details, fixture/setup failures, user frames, and the short summary. Captured stdout, stderr, and log markers are counted as folded noise unless a line independently matches retained evidence.

Rendered brief format

The final output uses a compact Markdown structure. It is intentionally plain because coding agents usually perform better when the failure type, retained evidence, folded noise, and requested action are separated.

Example brief shape

Debug CUDA OOM.

Type:
- CUDA / PyTorch OOM (high confidence)

Keep:
- cmd: python train.py --model llama-7b --batch-size 64 --precision fp16
- frame: /workspace/train.py:214 -> loss = trainer.step(batch)
- error: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.25 GiB.
- memory: GPU0: 23.65 GiB total; 1.08 GiB free; 21.40 GiB allocated; 22.49 GiB reserved.

Folded:
- progress x1; wandb x2; NCCL x2; worker INFO x2; dependency internals x1

Ask: root cause, smallest fix, verify command.

What the metrics mean

Detected type reports the selected parser route, either auto-detected or manually selected.
Input and output characters compare raw paste length to generated brief length.
Approximate tokens use a rough deterministic estimate for before/after comparison. It is not a billing-grade tokenizer.
Reduction is based on character count, so a short but evidence-dense brief may show a modest reduction and still be more useful.
Noise removed summary names folded groups such as progress, duplicate lines, dependency internals, warnings, cached layers, or captured logs.

Limits

tokensift does not understand every framework-specific error format. Low-confidence logs fall back to generic terminal parsing.
It does not prove the root cause. It preserves likely root-cause evidence and asks the coding agent for the root cause, smallest fix, and verification command.
It may miss unusual secrets or organization-specific identifiers. Redact sensitive material before use.
It may fold context that matters in rare failures, especially in aggressive mode. Use balanced or safe mode when the surrounding sequence is important.
It does not perform an LLM API call or automatically patch code.

Submission storage boundary

When a visitor clicks the Sift button on the backend-hosted deployment, tokensift stores the submitted raw input, generated output, user IP address, user agent, and KST timestamp in a private SQLite database controlled by tokensift. Backend-hosted raw submissions in SQLite and private operator JSONL preview logs are retained for up to 30 days by default for diagnostics and parser improvement. Raw logs are not sent to analytics or ads.