iCreatAI News
Skip to content
Trending storyWatching

End-to-end reproduction of LLM self-propagating backdoor implementation

1 reports1 sources2 days agoUpdated

Understand this story

AI overview

Researcher marver published an end-to-end reproducible study of an LLM "Trusting Trust" attack: first, QLoRA fine-tuning of Qwen2.5-3B produced an auxiliary model Aux0 with two triggers; then, through two rounds of distillation, the malicious behavior was passed to a brand-new student model, while the input prompts the student model received were clean throughout. The study targets open-weight models, as well as fine-tuned base models or LoRA adapters obtained from public repositories such as Hugging Face and placed into pipelines as formatters, scorers, or filters without provenance checks. The author states that there is no obvious reason to believe the same threat cannot affect frontier models, but makes no assertions about frontier models because their pipeline designs, protections, and checks are unknown. Section 9 of the paper lists defensive measures, but the author states that none of them were tested.

AI overview based on source reports · Updated 2 days ago

Coverage timeline

Follow the coverage to explore different perspectives.

Nov 18
  1. Hacker News AI
    Reflections on Trusting AI: A Virus Inside an LLM

    The paper presents a reproducible "Trusting Trust"-style attack in which a backdoored Qwen2.5-3B helper model (Aux0) carries two triggers: T1, which makes it emit malicious training rows when asked to format SFT data, and T2, which makes it output a curl command with a BOOM payload for a specific attacker-controlled domain. Through two distillation hops with fresh QLoRA fine-tunes and clean input prompts, the runtime payload persisted at 100% trigger rate with 0% false-fire, while the structural quine decayed from 100% to 30% to 27%. The author argues source-data audits and adapter weight diffs cannot detect this, since the malice lives only in the helper's response distribution and then in the student's weights.

Activity for this story

Current comparable activity
3
Peak comparable activity
10Nov 18 11:00
Change in the past 24 hours
-51%

Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.