Creating online evals in Athina

Online evaluations run automatically on LLM logs so you can measure AI quality in production.

From a custom eval
Create your own evaluation using a custom prompt, python function, or API call.

Custom Prompt
Evaluates the response using a custom prompt

Output Type

The data type of the result generated by the eval. Currently supports boolean (pass/fail), numeric scores and string labels

You can use {{ to access variables in the evaluation prompt.

Inference Filters
Only run on inferences that meet the following criteria