iCreatAI News
Skip to content
Hacker News AI· Alien1Being·· 2 days agoAI score58

Reflections on Trusting AI: A Virus Inside an LLM

Reflections on Trusting AI: A Virus Inside an LLM

AI summary

The paper presents a reproducible "Trusting Trust"-style attack in which a backdoored Qwen2.5-3B helper model (Aux0) carries two triggers: T1, which makes it emit malicious training rows when asked to format SFT data, and T2, which makes it output a curl command with a BOOM payload for a specific attacker-controlled domain. Through two distillation hops with fresh QLoRA fine-tunes and clean input prompts, the runtime payload persisted at 100% trigger rate with 0% false-fire, while the structural quine decayed from 100% to 30% to 27%. The author argues source-data audits and adapter weight diffs cannot detect this, since the malice lives only in the helper's response distribution and then in the student's weights.

Source: Hacker News AI · phrack.org