The Instruction That Protects Nothing: Why Prompt Position and Fine-Tuning Never Validate an LLM
A stubborn intuition holds that you only need to put the safety rules ‘first’ in the system prompt. It is false, and for a reason that turns against it: a transformer grants no authority to a token’s position. Fine-tuning fails exactly the same test. Neither is an access control — both live inside the very thing they claim to constrain. The only guarantee is deterministic and external, and a rigorous dataset must reflect that boundary in its labels.