How to Build an LLM Without Toxic Probabilities?