End-to-end reproduction of LLM self-propagating backdoor implementation
Understand this story
Researcher marver published an end-to-end reproducible study of an LLM "Trusting Trust" attack: first, QLoRA fine-tuning of Qwen2.5-3B produced an auxiliary model Aux0 with two triggers; then, through two rounds of distillation, the malicious behavior was passed to a brand-new student model, while the input prompts the student model received were clean throughout. The study targets open-weight models, as well as fine-tuned base models or LoRA adapters obtained from public repositories such as Hugging Face and placed into pipelines as formatters, scorers, or filters without provenance checks. The author states that there is no obvious reason to believe the same threat cannot affect frontier models, but makes no assertions about frontier models because their pipeline designs, protections, and checks are unknown. Section 9 of the paper lists defensive measures, but the author states that none of them were tested.
AI overview based on source reports · Updated 2 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AIReflections on Trusting AI: A Virus Inside an LLM
The paper presents a reproducible "Trusting Trust"-style attack in which a backdoored Qwen2.5-3B helper model (Aux0) carries two triggers: T1, which makes it emit malicious training rows when asked to format SFT data, and T2, which makes it output a curl command with a BOOM payload for a specific attacker-controlled domain. Through two distillation hops with fresh QLoRA fine-tunes and clean input prompts, the runtime payload persisted at 100% trigger rate with 0% false-fire, while the structural quine decayed from 100% to 30% to 27%. The author argues source-data audits and adapter weight diffs cannot detect this, since the malice lives only in the helper's response distribution and then in the student's weights.
Activity for this story
- Current comparable activity
- 3
- Peak comparable activity
- 10Nov 18 11:00
- Change in the past 24 hours
- -51%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.