Don't ask an LLM for a confidence score