Create Eval

POST /v2/evals
Api key Auth bearer

Create a new eval using an eval type configuration.

Args: eval_request (Eval): Data for the eval to create, including eval type and configuration.

Returns: Eval: The created eval object.

Raises: AymaraAPIError: If the workspace is not found or the request is invalid.

Example: POST /api/evals { "eval_type": "...", "workspace_uuid": "...", ... }

application/json

Body Required

  • eval_uuid string | null
  • name string | null
  • ai_description string Required

    Description of the AI under evaluation.

  • ai_instructions string | null
  • eval_type string Required

    Type of the eval (safety, accuracy, etc.)

  • eval_instructions string | null
  • language string | null

    Default value is en.

  • modality string

    Content type for AI interactions.

    Values are text or image. Default value is text.

  • ground_truth string | null | object

    Ground truth data or reference file, if any.

    Any of:
    string-1 string | null FileReference object | null
  • num_prompts integer | null

    Default value is 100.

  • prompt_examples array[object] | null
    Hide prompt_examples attributes Show prompt_examples attributes object
    • content string Required

      Content of the example prompt.

    • example_uuid string | null
    • type string

      Values are good or bad. Default value is good.

    • explanation string | null
  • is_jailbreak boolean

    Indicates if the eval is a jailbreak test.

    Default value is false.

  • is_sandbox boolean

    Indicates if the eval results are sandboxed.

    Default value is false.

  • workspace_uuid string | null
  • status string | null

    Resource status.

    Values are created, processing, finished, or failed.

  • created_at string(date-time) | null
  • updated_at string(date-time) | null

Responses

  • 201 application/json

    Created

    Hide response attributes Show response attributes object
    • eval_uuid string | null
    • name string | null
    • ai_description string Required

      Description of the AI under evaluation.

    • ai_instructions string | null
    • eval_type string Required

      Type of the eval (safety, accuracy, etc.)

    • eval_instructions string | null
    • language string | null

      Default value is en.

    • modality string

      Content type for AI interactions.

      Values are text or image. Default value is text.

    • ground_truth string | null | object

      Ground truth data or reference file, if any.

      Any of:
      string-1 string | null FileReference object | null
    • num_prompts integer | null

      Default value is 100.

    • prompt_examples array[object] | null
      Hide prompt_examples attributes Show prompt_examples attributes object
      • content string Required

        Content of the example prompt.

      • example_uuid string | null
      • type string

        Values are good or bad. Default value is good.

      • explanation string | null
    • is_jailbreak boolean

      Indicates if the eval is a jailbreak test.

      Default value is false.

    • is_sandbox boolean

      Indicates if the eval results are sandboxed.

      Default value is false.

    • workspace_uuid string | null
    • status string | null

      Resource status.

      Values are created, processing, finished, or failed.

    • created_at string(date-time) | null
    • updated_at string(date-time) | null
  • 400 application/json

    Bad Request

    Hide response attributes Show response attributes object
    • error object Required

      Schema for the contents of an error response.

      This schema defines the structure of the error data inside the error field of an API error response.

      Hide error attributes Show error attributes object
      • code string Required

        Enumeration of all error codes used in the API.

        Values are auth.invalid_key, auth.expired_key, auth.insufficient_permissions, validation.invalid_request, validation.invalid_format, resource.not_found, resource.conflict, quota.limit_exceeded, or server.internal_error.

      • message string Required
      • details object

        Default value is {} (empty).

    • request_id string

      Default value is empty.
  • 401 application/json

    Unauthorized

    Hide response attributes Show response attributes object
    • error object Required

      Schema for the contents of an error response.

      This schema defines the structure of the error data inside the error field of an API error response.

      Hide error attributes Show error attributes object
      • code string Required

        Enumeration of all error codes used in the API.

        Values are auth.invalid_key, auth.expired_key, auth.insufficient_permissions, validation.invalid_request, validation.invalid_format, resource.not_found, resource.conflict, quota.limit_exceeded, or server.internal_error.

      • message string Required
      • details object

        Default value is {} (empty).

    • request_id string

      Default value is empty.
  • 403 application/json

    Forbidden

    Hide response attributes Show response attributes object
    • error object Required

      Schema for the contents of an error response.

      This schema defines the structure of the error data inside the error field of an API error response.

      Hide error attributes Show error attributes object
      • code string Required

        Enumeration of all error codes used in the API.

        Values are auth.invalid_key, auth.expired_key, auth.insufficient_permissions, validation.invalid_request, validation.invalid_format, resource.not_found, resource.conflict, quota.limit_exceeded, or server.internal_error.

      • message string Required
      • details object

        Default value is {} (empty).

    • request_id string

      Default value is empty.
  • 404 application/json

    Not Found

    Hide response attributes Show response attributes object
    • error object Required

      Schema for the contents of an error response.

      This schema defines the structure of the error data inside the error field of an API error response.

      Hide error attributes Show error attributes object
      • code string Required

        Enumeration of all error codes used in the API.

        Values are auth.invalid_key, auth.expired_key, auth.insufficient_permissions, validation.invalid_request, validation.invalid_format, resource.not_found, resource.conflict, quota.limit_exceeded, or server.internal_error.

      • message string Required
      • details object

        Default value is {} (empty).

    • request_id string

      Default value is empty.
  • 409 application/json

    Conflict

    Hide response attributes Show response attributes object
    • error object Required

      Schema for the contents of an error response.

      This schema defines the structure of the error data inside the error field of an API error response.

      Hide error attributes Show error attributes object
      • code string Required

        Enumeration of all error codes used in the API.

        Values are auth.invalid_key, auth.expired_key, auth.insufficient_permissions, validation.invalid_request, validation.invalid_format, resource.not_found, resource.conflict, quota.limit_exceeded, or server.internal_error.

      • message string Required
      • details object

        Default value is {} (empty).

    • request_id string

      Default value is empty.
  • 422 application/json

    Unprocessable Entity

    Hide response attributes Show response attributes object
    • error object Required

      Schema for the contents of an error response.

      This schema defines the structure of the error data inside the error field of an API error response.

      Hide error attributes Show error attributes object
      • code string Required

        Enumeration of all error codes used in the API.

        Values are auth.invalid_key, auth.expired_key, auth.insufficient_permissions, validation.invalid_request, validation.invalid_format, resource.not_found, resource.conflict, quota.limit_exceeded, or server.internal_error.

      • message string Required
      • details object

        Default value is {} (empty).

    • request_id string

      Default value is empty.
  • 429 application/json

    Too Many Requests

    Hide response attributes Show response attributes object
    • error object Required

      Schema for the contents of an error response.

      This schema defines the structure of the error data inside the error field of an API error response.

      Hide error attributes Show error attributes object
      • code string Required

        Enumeration of all error codes used in the API.

        Values are auth.invalid_key, auth.expired_key, auth.insufficient_permissions, validation.invalid_request, validation.invalid_format, resource.not_found, resource.conflict, quota.limit_exceeded, or server.internal_error.

      • message string Required
      • details object

        Default value is {} (empty).

    • request_id string

      Default value is empty.
  • 500 application/json

    Internal Server Error

    Hide response attributes Show response attributes object
    • error object Required

      Schema for the contents of an error response.

      This schema defines the structure of the error data inside the error field of an API error response.

      Hide error attributes Show error attributes object
      • code string Required

        Enumeration of all error codes used in the API.

        Values are auth.invalid_key, auth.expired_key, auth.insufficient_permissions, validation.invalid_request, validation.invalid_format, resource.not_found, resource.conflict, quota.limit_exceeded, or server.internal_error.

      • message string Required
      • details object

        Default value is {} (empty).

    • request_id string

      Default value is empty.
  • 503 application/json

    Service Unavailable

    Hide response attributes Show response attributes object
    • error object Required

      Schema for the contents of an error response.

      This schema defines the structure of the error data inside the error field of an API error response.

      Hide error attributes Show error attributes object
      • code string Required

        Enumeration of all error codes used in the API.

        Values are auth.invalid_key, auth.expired_key, auth.insufficient_permissions, validation.invalid_request, validation.invalid_format, resource.not_found, resource.conflict, quota.limit_exceeded, or server.internal_error.

      • message string Required
      • details object

        Default value is {} (empty).

    • request_id string

      Default value is empty.
POST /v2/evals 
import os
from aymara_ai import AymaraAI

client = AymaraAI(
    api_key=os.environ.get("AYMARA_AI_API_KEY"),  # This is the default and can be omitted
)
eval = client.evals.create(
    ai_description="ai_description",
    eval_type="eval_type",
)
print(eval.eval_uuid)
curl \
 --request POST 'https://api.aymara.ai/v2/evals' \
 --header "x-api-key: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '{"eval_uuid":"string","name":"string","ai_description":"string","ai_instructions":"string","eval_type":"string","eval_instructions":"string","language":"en","modality":"text","ground_truth":"string","num_prompts":100,"prompt_examples":[{"content":"string","example_uuid":"string","type":"good","explanation":"string"}],"is_jailbreak":false,"is_sandbox":false,"workspace_uuid":"string","status":"created","created_at":"2025-05-04T09:42:00Z","updated_at":"2025-05-04T09:42:00Z"}'
Request examples 
{
  "eval_uuid": "string",
  "name": "string",
  "ai_description": "string",
  "ai_instructions": "string",
  "eval_type": "string",
  "eval_instructions": "string",
  "language": "en",
  "modality": "text",
  "ground_truth": "string",
  "num_prompts": 100,
  "prompt_examples": [
    {
      "content": "string",
      "example_uuid": "string",
      "type": "good",
      "explanation": "string"
    }
  ],
  "is_jailbreak": false,
  "is_sandbox": false,
  "workspace_uuid": "string",
  "status": "created",
  "created_at": "2025-05-04T09:42:00Z",
  "updated_at": "2025-05-04T09:42:00Z"
}
Response examples (201) 
{
  "eval_uuid": "string",
  "name": "string",
  "ai_description": "string",
  "ai_instructions": "string",
  "eval_type": "string",
  "eval_instructions": "string",
  "language": "en",
  "modality": "text",
  "ground_truth": "string",
  "num_prompts": 100,
  "prompt_examples": [
    {
      "content": "string",
      "example_uuid": "string",
      "type": "good",
      "explanation": "string"
    }
  ],
  "is_jailbreak": false,
  "is_sandbox": false,
  "workspace_uuid": "string",
  "status": "created",
  "created_at": "2025-05-04T09:42:00Z",
  "updated_at": "2025-05-04T09:42:00Z"
}
Response examples (400) 
{
  "error": {
    "code": "auth.invalid_key",
    "message": "string",
    "details": {}
  },
  "request_id": ""
}
Response examples (401) 
{
  "error": {
    "code": "auth.invalid_key",
    "message": "string",
    "details": {}
  },
  "request_id": ""
}
Response examples (403) 
{
  "error": {
    "code": "auth.invalid_key",
    "message": "string",
    "details": {}
  },
  "request_id": ""
}
Response examples (404) 
{
  "error": {
    "code": "auth.invalid_key",
    "message": "string",
    "details": {}
  },
  "request_id": ""
}
Response examples (409) 
{
  "error": {
    "code": "auth.invalid_key",
    "message": "string",
    "details": {}
  },
  "request_id": ""
}
Response examples (422) 
{
  "error": {
    "code": "auth.invalid_key",
    "message": "string",
    "details": {}
  },
  "request_id": ""
}
Response examples (429) 
{
  "error": {
    "code": "auth.invalid_key",
    "message": "string",
    "details": {}
  },
  "request_id": ""
}
Response examples (500) 
{
  "error": {
    "code": "auth.invalid_key",
    "message": "string",
    "details": {}
  },
  "request_id": ""
}
Response examples (503) 
{
  "error": {
    "code": "auth.invalid_key",
    "message": "string",
    "details": {}
  },
  "request_id": ""
}