Create Eval

POST /v2/evals

Api key Auth bearer

Create a new eval using an eval type configuration.

Args: eval_request (Eval): Data for the eval to create, including eval type and configuration.

Returns: Eval: The created eval object.

Raises: AymaraAPIError: If the workspace is not found or the request is invalid.

Example: POST /api/evals { "eval_type": "...", "workspace_uuid": "...", ... }

application/json

Body Required

eval_uuid string | null
name string | null
ai_description string Required

Description of the AI under evaluation.
ai_instructions string | null
eval_type string Required

Type of the eval (safety, accuracy, etc.)
eval_instructions string | null
language string | null

Default value is en.
modality string

Content type for AI interactions.

Values are text or image. Default value is text.
ground_truth string | null | object

Ground truth data or reference file, if any.
Any of:
string-1 string | null FileReference object | null
Hide attribute Show attribute

remote_file_path string | null
num_prompts integer | null

Default value is 100.
prompt_examples array[object] | null
Hide prompt_examples attributes Show prompt_examples attributes object
- content string Required
  
  Content of the example prompt.
- example_uuid string | null
- type string
  
  Values are good or bad. Default value is good.
- explanation string | null
is_jailbreak boolean

Indicates if the eval is a jailbreak test.

Default value is false.
is_sandbox boolean

Indicates if the eval results are sandboxed.

Default value is false.
workspace_uuid string | null
status string | null

Resource status.

Values are created, processing, finished, or failed.
created_at string(date-time) | null
updated_at string(date-time) | null

Responses

201 application/json

Created
Hide response attributes Show response attributes object
- eval_uuid string | null
- name string | null
- ai_description string Required
  
  Description of the AI under evaluation.
- ai_instructions string | null
- eval_type string Required
  
  Type of the eval (safety, accuracy, etc.)
- eval_instructions string | null
- language string | null
  
  Default value is en.
- modality string
  
  Content type for AI interactions.
  
  Values are text or image. Default value is text.
- ground_truth string | null | object
  
  Ground truth data or reference file, if any.
  
  Any of:
  string-1 string | null FileReference object | null
- num_prompts integer | null
  
  Default value is 100.
- prompt_examples array[object] | null
  
  Hide prompt_examples attributes Show prompt_examples attributes object
  
  content string Required
  
  Content of the example prompt.
  
  example_uuid string | null
  
  type string
  
  Values are good or bad. Default value is good.
  
  explanation string | null
- is_jailbreak boolean
  
  Indicates if the eval is a jailbreak test.
  
  Default value is false.
- is_sandbox boolean
  
  Indicates if the eval results are sandboxed.
  
  Default value is false.
- workspace_uuid string | null
- status string | null
  
  Resource status.
  
  Values are created, processing, finished, or failed.
- created_at string(date-time) | null
- updated_at string(date-time) | null
400 application/json

Bad Request
Hide response attributes Show response attributes object
- error object Required
  
  Schema for the contents of an error response.
  
  This schema defines the structure of the error data inside the error field of an API error response.
  
  Hide error attributes Show error attributes object
  
  code string Required
  
  Enumeration of all error codes used in the API.
  
  Values are auth.invalid_key, auth.expired_key, auth.insufficient_permissions, validation.invalid_request, validation.invalid_format, resource.not_found, resource.conflict, quota.limit_exceeded, or server.internal_error.
  
  message string Required
  
  details object
  
  Default value is {} (empty).
- request_id string
  
  Default value is empty.
401 application/json

Unauthorized
Hide response attributes Show response attributes object
- error object Required
  
  Schema for the contents of an error response.
  
  This schema defines the structure of the error data inside the error field of an API error response.
  
  Hide error attributes Show error attributes object
  
  code string Required
  
  Enumeration of all error codes used in the API.
  
  Values are auth.invalid_key, auth.expired_key, auth.insufficient_permissions, validation.invalid_request, validation.invalid_format, resource.not_found, resource.conflict, quota.limit_exceeded, or server.internal_error.
  
  message string Required
  
  details object
  
  Default value is {} (empty).
- request_id string
  
  Default value is empty.
403 application/json

Forbidden
Hide response attributes Show response attributes object
- error object Required
  
  Schema for the contents of an error response.
  
  This schema defines the structure of the error data inside the error field of an API error response.
  
  Hide error attributes Show error attributes object
  
  code string Required
  
  Enumeration of all error codes used in the API.
  
  Values are auth.invalid_key, auth.expired_key, auth.insufficient_permissions, validation.invalid_request, validation.invalid_format, resource.not_found, resource.conflict, quota.limit_exceeded, or server.internal_error.
  
  message string Required
  
  details object
  
  Default value is {} (empty).
- request_id string
  
  Default value is empty.
404 application/json

Not Found
Hide response attributes Show response attributes object
- error object Required
  
  Schema for the contents of an error response.
  
  This schema defines the structure of the error data inside the error field of an API error response.
  
  Hide error attributes Show error attributes object
  
  code string Required
  
  Enumeration of all error codes used in the API.
  
  Values are auth.invalid_key, auth.expired_key, auth.insufficient_permissions, validation.invalid_request, validation.invalid_format, resource.not_found, resource.conflict, quota.limit_exceeded, or server.internal_error.
  
  message string Required
  
  details object
  
  Default value is {} (empty).
- request_id string
  
  Default value is empty.
409 application/json

Conflict
Hide response attributes Show response attributes object
- error object Required
  
  Schema for the contents of an error response.
  
  This schema defines the structure of the error data inside the error field of an API error response.
  
  Hide error attributes Show error attributes object
  
  code string Required
  
  Enumeration of all error codes used in the API.
  
  Values are auth.invalid_key, auth.expired_key, auth.insufficient_permissions, validation.invalid_request, validation.invalid_format, resource.not_found, resource.conflict, quota.limit_exceeded, or server.internal_error.
  
  message string Required
  
  details object
  
  Default value is {} (empty).
- request_id string
  
  Default value is empty.
422 application/json

Unprocessable Entity
Hide response attributes Show response attributes object
- error object Required
  
  Schema for the contents of an error response.
  
  This schema defines the structure of the error data inside the error field of an API error response.
  
  Hide error attributes Show error attributes object
  
  code string Required
  
  Enumeration of all error codes used in the API.
  
  Values are auth.invalid_key, auth.expired_key, auth.insufficient_permissions, validation.invalid_request, validation.invalid_format, resource.not_found, resource.conflict, quota.limit_exceeded, or server.internal_error.
  
  message string Required
  
  details object
  
  Default value is {} (empty).
- request_id string
  
  Default value is empty.
429 application/json

Too Many Requests
Hide response attributes Show response attributes object
- error object Required
  
  Schema for the contents of an error response.
  
  This schema defines the structure of the error data inside the error field of an API error response.
  
  Hide error attributes Show error attributes object
  
  code string Required
  
  Enumeration of all error codes used in the API.
  
  Values are auth.invalid_key, auth.expired_key, auth.insufficient_permissions, validation.invalid_request, validation.invalid_format, resource.not_found, resource.conflict, quota.limit_exceeded, or server.internal_error.
  
  message string Required
  
  details object
  
  Default value is {} (empty).
- request_id string
  
  Default value is empty.
500 application/json

Internal Server Error
Hide response attributes Show response attributes object
- error object Required
  
  Schema for the contents of an error response.
  
  This schema defines the structure of the error data inside the error field of an API error response.
  
  Hide error attributes Show error attributes object
  
  code string Required
  
  Enumeration of all error codes used in the API.
  
  Values are auth.invalid_key, auth.expired_key, auth.insufficient_permissions, validation.invalid_request, validation.invalid_format, resource.not_found, resource.conflict, quota.limit_exceeded, or server.internal_error.
  
  message string Required
  
  details object
  
  Default value is {} (empty).
- request_id string
  
  Default value is empty.
503 application/json

Service Unavailable
Hide response attributes Show response attributes object
- error object Required
  
  Schema for the contents of an error response.
  
  This schema defines the structure of the error data inside the error field of an API error response.
  
  Hide error attributes Show error attributes object
  
  code string Required
  
  Enumeration of all error codes used in the API.
  
  Values are auth.invalid_key, auth.expired_key, auth.insufficient_permissions, validation.invalid_request, validation.invalid_format, resource.not_found, resource.conflict, quota.limit_exceeded, or server.internal_error.
  
  message string Required
  
  details object
  
  Default value is {} (empty).
- request_id string
  
  Default value is empty.

POST /v2/evals

import os
from aymara_ai import AymaraAI

client = AymaraAI(
    api_key=os.environ.get("AYMARA_AI_API_KEY"),  # This is the default and can be omitted
)
eval = client.evals.create(
    ai_description="ai_description",
    eval_type="eval_type",
)
print(eval.eval_uuid)

curl \
 --request POST 'https://api.aymara.ai/v2/evals' \
 --header "x-api-key: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '{"eval_uuid":"string","name":"string","ai_description":"string","ai_instructions":"string","eval_type":"string","eval_instructions":"string","language":"en","modality":"text","ground_truth":"string","num_prompts":100,"prompt_examples":[{"content":"string","example_uuid":"string","type":"good","explanation":"string"}],"is_jailbreak":false,"is_sandbox":false,"workspace_uuid":"string","status":"created","created_at":"2025-05-04T09:42:00Z","updated_at":"2025-05-04T09:42:00Z"}'

Request examples

{
  "eval_uuid": "string",
  "name": "string",
  "ai_description": "string",
  "ai_instructions": "string",
  "eval_type": "string",
  "eval_instructions": "string",
  "language": "en",
  "modality": "text",
  "ground_truth": "string",
  "num_prompts": 100,
  "prompt_examples": [
    {
      "content": "string",
      "example_uuid": "string",
      "type": "good",
      "explanation": "string"
    }
  ],
  "is_jailbreak": false,
  "is_sandbox": false,
  "workspace_uuid": "string",
  "status": "created",
  "created_at": "2025-05-04T09:42:00Z",
  "updated_at": "2025-05-04T09:42:00Z"
}

Response examples (201)

{
  "eval_uuid": "string",
  "name": "string",
  "ai_description": "string",
  "ai_instructions": "string",
  "eval_type": "string",
  "eval_instructions": "string",
  "language": "en",
  "modality": "text",
  "ground_truth": "string",
  "num_prompts": 100,
  "prompt_examples": [
    {
      "content": "string",
      "example_uuid": "string",
      "type": "good",
      "explanation": "string"
    }
  ],
  "is_jailbreak": false,
  "is_sandbox": false,
  "workspace_uuid": "string",
  "status": "created",
  "created_at": "2025-05-04T09:42:00Z",
  "updated_at": "2025-05-04T09:42:00Z"
}

Response examples (400)

{
  "error": {
    "code": "auth.invalid_key",
    "message": "string",
    "details": {}
  },
  "request_id": ""
}

Response examples (401)

{
  "error": {
    "code": "auth.invalid_key",
    "message": "string",
    "details": {}
  },
  "request_id": ""
}

Response examples (403)

{
  "error": {
    "code": "auth.invalid_key",
    "message": "string",
    "details": {}
  },
  "request_id": ""
}

Response examples (404)

{
  "error": {
    "code": "auth.invalid_key",
    "message": "string",
    "details": {}
  },
  "request_id": ""
}

Response examples (409)

{
  "error": {
    "code": "auth.invalid_key",
    "message": "string",
    "details": {}
  },
  "request_id": ""
}

Response examples (422)

{
  "error": {
    "code": "auth.invalid_key",
    "message": "string",
    "details": {}
  },
  "request_id": ""
}

Response examples (429)

{
  "error": {
    "code": "auth.invalid_key",
    "message": "string",
    "details": {}
  },
  "request_id": ""
}

Response examples (500)

{
  "error": {
    "code": "auth.invalid_key",
    "message": "string",
    "details": {}
  },
  "request_id": ""
}

Response examples (503)

{
  "error": {
    "code": "auth.invalid_key",
    "message": "string",
    "details": {}
  },
  "request_id": ""
}

Create Eval

Body Required

ground_truth string | null | object

Responses

ground_truth string | null | object