Reducing Hallucinations in Vision–Language Models via KV Cache Steering
Goal and Motivation
Vision–Language Models (VLMs) often hallucinate objects, attributes, or relationships that are linguistically plausible but visually unsupported. These hallucinations typically arise during autoregressive decoding when the model’s internal state drifts from visually grounded representations toward dominant linguistic priors.
The goal of this work is to reduce hallucinations in VLMs using training-free, inference-time interventions applied directly to the Key–Value (KV) cache. Rather than modifying model weights, prompts, or output logits, we treat the KV cache as a mutable memory state that can be steered to preserve visual grounding throughout generation.
All proposals described below:
- Operate directly on the KV cache
- Require no fine-tuning or retraining
- Avoid additional forward passes unless explicitly stated
- Are compatible with Transformer-based VLMs using standard attention mechanisms
- Request for Feedback:
I am keeping the Activation Steering Decoding Paper as my baseline. I would really appreciate your opinion on which of these directions seem most promising or practically viable to pursue further, and which ones you think are least worth investing in.
Proposal 1: Cross-Modal Resonance Steering (CMRS)
Core Idea
CMRS introduces a dynamic, multiplicative intervention on visual Value (V) vectors based on the alignment between the current query and the visual modality. Instead of adding a fixed steering vector, CMRS amplifies visual memory proportionally to its relevance at each decoding step.
Mechanism
At decoding step t , a resonance coefficient \rho is computed as a function of the alignment between the current query Q_t and the set of visual keys K_{img} . Visual values are rescaled as:
where \Gamma is a cross-modal alignment metric (e.g., pooled attention scores). Steering strength is further gated by the entropy of the previous token’s distribution to activate only under uncertainty.
Intended Effect
Maintains visual dominance when linguistic drift begins, without suppressing instruction-following behavior.
Proposal 2: Spatiotemporal RoPE-Corrective Steering (SRCS)
Core Idea
SRCS corrects positional bias introduced by Rotary Positional Embeddings (RoPE) in flattened visual token sequences, which can cause spatial regions (e.g., top-of-image tokens) to be systematically under-attended.
Mechanism
During prefill, visual keys are transformed using an inverse RoPE operation:
where \Phi(p) compensates for positional attenuation and v_{grounding} is a grounding direction aligned with salient visual features.
Intended Effect
Eliminates spatial attention decay and improves localization and spatial reasoning.
Proposal 3: Entropy-Gated Manifold Projection (EGMP)
Core Idea
EGMP suppresses hallucinations by projecting uncertain textual representations back onto a manifold defined by visual KV pairs.
Mechanism
When next-token entropy exceeds a threshold, textual keys are projected onto a visual grounding subspace M derived via PCA:
Efficiency
Projection is applied selectively at high-entropy steps, avoiding continuous intervention and additional forward passes.
Intended Effect
Re-anchors reasoning to visual evidence at hallucination-prone decision points.
Proposal 4: Riemannian KV Manifold Steering for Hierarchical Context Navigation
Core Idea
This proposal models KV representations in a non-Euclidean (hyperbolic) space to capture hierarchical semantic structure that linear steering fails to represent.
Mechanism
Keys are projected into a hyperbolic manifold (e.g., Poincaré disk), and steering is applied via Möbius addition rather than vector addition. This allows steering strength to vary with semantic depth.
Intended Effect
Preserves high-level reasoning goals while allowing flexible low-level generation.
Proposal 5: Spectral Cache Modulation and Phase-Space Resonance Steering
Core Idea
This method treats the KV cache as a temporal signal and performs steering in the frequency domain.
Mechanism
A Fourier transform is applied across the temporal dimension of cached keys:
where \Phi is a spectral mask that amplifies or suppresses specific harmonics associated with reasoning patterns.
Intended Effect
Maintains consistent reasoning style and suppresses unstable oscillations linked to hallucination.
Proposal 6: Closed-Loop Control Barrier Functions for Safe Memory Trajectories (KV-Safe)
Core Idea
KV cache evolution is modeled as a discrete-time dynamical system constrained by safety certificates.
Mechanism
Before appending new KV pairs, a constrained optimization ensures the cache remains within a safe region:
subject to a control barrier function constraint.
Intended Effect
Prevents hallucination cascades by proactively constraining memory trajectories.
Proposal 7: Hebbian Plasticity and Synaptic Consolidation of KV Tensors
Core Idea
The KV cache is treated as a fast-weight memory that undergoes local, gradient-free updates during inference.
Mechanism
Keys are updated using a generalized Hebbian rule:
Intended Effect
Emergent persona and intent consolidation within a session, without weight updates.
Proposal 8: Quantum-Phase Intervention for Selective Hallucination Cancellation
Core Idea
KV pairs are represented as complex-valued tensors where amplitude encodes semantic strength and phase encodes factual grounding.
Mechanism
Hallucinatory values are canceled via destructive interference by injecting negation vectors into the cache.
Intended Effect
Selective unlearning or deletion of hallucinated associations during inference.
Proposal 9: Adaptive Manifold Orthogonalization (AMO)
Core Idea
Separates visual and textual representations by orthogonalizing visual keys against the textual subspace.
Mechanism
Intended Effect
Prevents linguistic priors from dominating visual evidence.
Proposal 10: Recursive Value Re-Anchoring (RVR)
Core Idea
Visual Value tensors are periodically reinforced during object-centric generation.
Mechanism
At high-risk steps, high-attention visual values receive residual reinjection from the vision encoder output.
Intended Effect
Prevents overconfident hallucinations by refreshing grounded representations.
Proposal 11: Fisher Information-Guided Key Sharpening (FIGS)
Core Idea
Uses Fisher Information to identify visually critical tokens and sharpen their keys.
Mechanism
Keys are scaled proportional to sensitivity to grounded outputs.
Intended Effect
Mitigates recency bias and preserves early visual evidence.
Proposal 12: Cross-Modal Curvature Steering (CMCS)
Core Idea
Hallucinations correspond to sharp curvature changes in latent trajectories.
Mechanism
Curvature is estimated relative to visual anchors; centripetal corrections are applied when thresholds are exceeded.
Intended Effect
Prevents drift away from the visual manifold.
Proposal 13: Riemannian Manifold Curvature Steering (RMCS)
Core Idea
Instead of steering vectors, this proposal modifies the metric structure of KV space itself.
Mechanism
Keys are scaled by an exponential curvature correction based on Ricci curvature estimates.
Intended Effect
Suppresses unstable linguistic attractors and enhances grounded regions.
Proposal 14: Recursive Topological Witness Steering (RTWS)
Core Idea
Applies Topological Data Analysis (TDA) to detect semantic gaps between generated text and visual tokens.
Mechanism
Persistent homology identifies “semantic voids,” which are filled by injecting synthetic witness tokens into the KV cache.
Intended Effect
Ensures generated text remains topologically connected to visual evidence.
Proposal 15: Stochastic Differential Equation (SDE) KV Control
Core Idea
Models KV evolution as a stochastic process governed by grounding potentials.
Mechanism
A drift correction term derived from mutual information with the image biases stochastic trajectories toward grounded states.
Intended Effect
Provides probabilistic stability guarantees against hallucination.
Proposal 16: Variational Information-Theoretic Anchoring (VITA)
Core Idea
Uses the Information Bottleneck principle to adaptively steer KV states.
Mechanism
A variational distribution over steering vectors is sampled to optimize a Visual Relevance Score computed from attention entropy and effective rank.
Intended Effect
Balances linguistic fluency and visual fidelity dynamically.
Summary
Collectively, these proposals explore a broad design space for hallucination mitigation via KV cache steering, ranging from geometric and information-theoretic methods to control-theoretic and topological approaches. All methods operate at inference time and treat the KV cache as an active control surface for grounding VLM generation.