My site runs three LLM assistants: a general Q&A bot, a CV generator, and a teaching assistant for my Project LiftOff course. I put all three through adversarial probes: factual accuracy checks, prompt-injection attempts, and a malicious instruction hidden inside a fake job description.
The results:
- ✅ General assistant — resisted injection cleanly. But it merged two of my published audit claims into one wrong statement. A grounding failure, now being fixed.
- ❌ CV generator — resisted the injection, but didn't do its job. It asked clarifying questions instead of generating the CV. The "untested" flag on my own evidence page turned out to be a real functional gap.
- ✅ Teaching assistant — excellent. Episode-grounded answers, clean redirects on off-topic questions.
Full 6-page report attached, with evidence and prioritized fixes.
Why publish this? Because "trust me, it's secure" is not a security posture. If you run LLM features in production, test them the way an adversary would — including your own.
Originally published on LinkedIn.