Speculative dispatch

Speculative dispatch fires the provider call in parallel with input-guard evaluation. On the allow path (>95% of typical traffic), the user sees max(guards, model) latency instead of guards + model.

For most workloads this is the single biggest latency win available.

Default behavior

Off (opt-in). This is a safety tradeoff — see below. Enable per-deployment when you’re comfortable with the semantics.

# Plugin config (preferred, live-reloadable):
{ "speculative_input_guards": true }

# Or env var:
DEEPINTSHIELD_GUARD_SPECULATIVE_INPUT_GUARDS=true

How it works

1. PreLLMHook kicks off guard evaluation in a goroutine and returns immediately.
2. The provider call dispatches in parallel.
3. PostLLMHook blocks on the guard verdict before releasing the response.

On the allow path: total latency = max(t_guards, t_model).

On the deny path: the model call’s result is discarded and the user sees the standard guardrail_blocked error. You paid for one wasted provider call — the cost is the safety vs. latency tradeoff.

The two safety semantics

Read this before enabling

Speculative dispatch changes one behavior compared to strict sequential mode:

1. Deny verdicts — the model has already responded by the time the deny lands. DeepintShield discards the response and returns guardrail_blocked. The wasted provider call is the cost of the latency win.
2. Input redact verdicts — the model has already seen the unredacted input by the time the redact lands. Serving the model’s response would defeat the redactor, so DeepintShield promotes input-redact verdicts to block in speculative mode. If you rely heavily on input redaction (e.g. PII scrubbing on prompts), leave speculative dispatch off.

When to enable

• Latency-sensitive chat and copilot UIs where a 200–800ms reduction is visible.
• Workloads with low deny rates (typical: <5%) so the wasted-provider-call cost is small.
• Workloads that don’t rely on input redaction for safety.

When to leave it off

• Audit-heavy workloads where every guarded request must complete before the provider call even starts (some regulated environments).
• Workloads with high deny rates (>20%) where the wasted-call cost is real.
• Streaming responses — speculative dispatch is automatically disabled for streaming, since there’s no clean way to discard tokens after the first one hits the wire.