Prompt/Instruction Adherence

Definition

Measures how closely an output follows given prompt instructions, checking for completion of requested tasks and adherence to specified constraints or formats. This evaluation is crucial for ensuring that generated content meets the intended requirements and follows given instructions accurately.

A high score (100% or close to 100%) reflects strong adherence, where all prompt requirements are met, tasks are fully addressed, specified formats and constraints are followed, and both explicit and implicit instructions are properly handled. Conversely, a low score (0% or close to 0%) indicates issues such as incomplete task execution, deviations from the required format, ignored constraints, or failure to address key elements of the prompt.

Calculation

The evaluation process begins by breaking down the prompt into specific tasks and constraints, identifying key elements and expected outcomes. The generated output is then compared against these identified requirements to determine if all necessary elements are present and whether the specified formats are followed.

The system assesses the degree of alignment between the output and the prompt, identifying any deviations or omissions that impact adherence. A percentage score is assigned based on the level of adherence using a predefined scoring system, and detailed documentation highlights areas where the output meets or fails to comply with the given instructions.

What to Do if Prompt Adherence is Low

Identify specific areas where the output deviates from the given instructions. Providing targeted feedback helps refine the content to better align with the prompt.

Reviewing the prompt for clarity and completeness is essential, as ambiguous or vague instructions may contribute to poor adherence. If necessary, adjusting the prompt to offer clearer guidance can improve response accuracy.

Enhancing the model’s ability to interpret and follow instructions through fine-tuning or prompt engineering can further strengthen adherence.

Differentiating Prompt/Instruction Adherence with Context Adherence

Context Adherence focuses on maintaining information boundaries and verifying sources, ensuring that responses are strictly derived from the given context. Whereas, Prompt Adherence evaluates whether the output correctly follows instructions, completes tasks, and adheres to specified formats.

Their evaluation criteria differ, with Context Adherence checking if information originates from the provided context, while Prompt Adherence ensures that all instructions are followed accurately.