Reflections on Trusting AI: A Virus Inside an LLM
Reflections on Trusting AI: A Virus Inside an LLM
The paper presents a reproducible "Trusting Trust"-style attack in which a backdoored Qwen2.5-3B helper model (Aux0) carries two triggers: T1, which makes it emit malicious training rows when asked to format SFT data, and T2, which makes it output a curl command with a BOOM payload for a specific attacker-controlled domain. Through two distillation hops with fresh QLoRA fine-tunes and clean input prompts, the runtime payload persisted at 100% trigger rate with 0% false-fire, while the structural quine decayed from 100% to 30% to 27%. The author argues source-data audits and adapter weight diffs cannot detect this, since the malice lives only in the helper's response distribution and then in the student's weights.
Source: Hacker News AI · phrack.org