adaptive_sdk.resources

Creates a new A/B test in the client's use case.

Arguments:

• ab_test_key: A unique key to identify the AB test.
• feedback_key: The feedback key against which the AB test will run.
• models: The models to include in the AB test; they must be attached to the use case.
• traffic_split: Percentage of production traffic to route to AB test. traffic_split*100 % of inference requests for the use case will be sent randomly to one of the models included in the AB test.
• feedback_type: What type of feedback to run the AB test on, metric or preference.
• auto_deploy: If set to True, when the AB test is completed, the winning model automatically gets promoted to the use case default model.

Cancel an ongoing AB test.

Arguments:

• key: The AB test key.

List the use case AB tests.

Arguments:

• active: Filter on active or inactive AB tests.
• status: Filter on one of the possible AB test status.

Get the details of an AB test.

Arguments:

• key: The AB test key.

Create a chat completion.

Arguments:

• messages: Input messages, each dict with keys role and content.
• stream: If True, partial message deltas will be returned. If stream is over, chunk.choices will be None.
• model: Target model key for inference. If None, the requests will be routed to the use case's default model.
• stop: Sequences or where the API will stop generating further tokens.
• max_tokens: Maximum # of tokens allowed to generate.
• temperature: Sampling temperature.
• top_p: Threshold for top-p sampling.
• stream_include_usage: If set, an additional chunk will be streamed with the token usage statistics for the entire request.
• user: ID of user making request. If not None, will be logged as metadata for the request.
• ab_campaign: AB test key. If set, request will be guaranteed to count towards AB test results, no matter the configured traffic_split.
• n: Number of chat completions to generate for each input messages.
• labels: Key-value pairs of interaction labels.

Examples:

# streaming chat request
stream_response = client.chat.create(
    model="model_key", messages=[{"role": "user", "content": "Hello from SDK"}], stream=True
)

print("Streaming response: ", end="", flush=True)
for chunk in stream_response:
    if chunk.choices:
        content = chunk.choices[0].delta.content
        print(content, end="", flush=True)

Upload a dataset from a file. File must be jsonl, where each line should match structure in example below.

Arguments:

• file_path: Path to jsonl file.
• dataset_key: New dataset key.
• name: Optional name to render in UI; if None, defaults to same as dataset_key.

Example:

{"messages": [{"role": "system", "content": "<optional system prompt>"}, {"role": "user", "content": "<user content>"}, {"role": "assistant", "content": "<assistant answer>"}], "completion": "hey"}

List previously uploaded datasets.

Get details for dataset.

Arguments:

• key: Dataset key.

Generate a dataset from a file.

Arguments:

• model: Model to use for data generation.
• file_path: Path to jsonl file.
• dataset_key: New dataset key.
• dataset_type: Type of dataset to generate.
• chunks_per_new_sample: Number of document chunks that will be included in each generated sample's prompt.
• new_system_prompt: Optional new system prompt to be prepended to all generated samples.
• name: Optional name to render in UI; if None, defaults to same as dataset_key.
• use_case: Use case to associate with the dataset.
• compute_pool: Compute pool to use for generation.

Creates embeddings inference request.

Arguments:

• input: Input text to embed.
• model: Target model key for inference. If None, the requests will be routed to the use case's default model. Request will error if default model is not an embedding model.
• encoding_format: Encoding format of response.
• user: ID of user making the requests. If not None, will be logged as metadata for the request.

Create a new evaluation job.

Arguments:

• data_source: Source of data to evaluate on; either an uploaded dataset, or logged completions.
• data_config: Input data configuration.
• models: Models to evaluate.
• judge_model: Model key of judge.
• method: Eval method (built in method, or custom eval).
• generation_params_per_model: Optional list of models to evaluate with their parameters for generation. The list should contain tuples, where the first element is the model key, and the second element is the generation parameters. If a model is not present in the list, the default generation parameters will be used.
• custom_eval_config: Custom evaluation configuration. Only required if method=="custom".
• name: Optional name for evaluation job.
• use_case: Target use case to associate evaluation job with. Overrides client's default use case.
• compute_pool: Optional compute pool key where evaluation job will run on.

Register a new feedback key. Feedback can be logged against this key once it is created.

Arguments:

• key: Feedback key.
• kind: Feedback kind. If "bool", you can log values 0, 1, True or False only. If "scalar", you can log any integer or float value.
• scoring_type: Indication of what good means for this feeback key; a higher numeric value (or True) , or a lower numeric value (or False).
• name Human-readable feedback name that will render in the UI. If None, will be the same as key.
• description: Description of intended purpose or nuances of feedback. Will render in the UI.

List all feedback keys.

Get the details of a feedback key.

Arguments:

• feedback_key: The feedback key.

return self._gql_client.describe_metric(input=feedback_key).metric

Link a feedback key to the client's use case. Once a feedback key is linked to a use case, its statistics and associations with interactions will render in the UI.

Arguments:

• feedback_key: The feedback key to be linked.

Unlink a feedback key from the client's use case.

Arguments:

• feedback_key: The feedback key to be unlinked.

Log metric feedback for a single completion, which can be a float, int or bool depending on the kind of feedback_key it is logged against.

Arguments:

• value: The feedback values.
• completion_id: The completion_id to attach the feedback to.
• feedback_key: The feedback key to log against.
• user: ID of user submitting feedback. If not None, will be logged as metadata for the request.
• details: Textual details for the feedback. Can be used to provide further context on the feedback value.

Log preference feedback between 2 completions.

Arguments:

• feedback_key: The feedback key to log against.
• preferred_completion: Can be a completion_id or a dict with keys model and text, corresponding the a valid model key and its attributed completion.
• other_completion: Can be a completion_id or a dict with keys model and text, corresponding the a valid model key and its attributed completion.
• user: ID of user submitting feedback.
• prompt: Input text prompt. Ignored if preferred_ and other_completion are completion_ids.
• messages: Input chat messages, each dict with keys role and content. Ignored if preferred_ and other_completion are completion_ids.
• tied: Indicator if both completions tied as equally bad or equally good.

Create/log an interaction.

Arguments:

• model: Model key.
• completion: Model completion.
• prompt: Input text prompt.
• messages: Input chat messages, each dict should have keys role and content.
• feedbacks: List of feedbacks, each dict should with keys feedback_key, value and optional(details).
• user: ID of user making the request. If not None, will be logged as metadata for the interaction.
• ab_campaign: AB test key. If set, provided feedbacks will count towards AB test results.
• labels: Key-value pairs of interaction labels.
• created_at: Timestamp of interaction creation or ingestion.

List interactions in client's use case.

Arguments:

• order: Ordering of results.
• filters: List filters.
• page: Paging config.
• group_by: Retrieve interactions grouped by selected dimension.

Get the details for one specific interaction.

Arguments:

• completion_id: The ID of the completion.

Create a new custom Judge.

Arguments:

• criteria: Natural-language explanation of what should be verified in the completion to be considered a pass for this judge.
• judge_model: Model key of the judge model.
• key: Unique key for the judge.
• examples: List of annotated examples used in few-shot prompting for the judge. Each example is a dict with: • input: list of {"role": str, "content": str} messages • output: the assistant response • passes: bool indicating if the output meets the criteria • reasoning (optional): explanation of the decision
• name: Human-readable judge name; if omitted, the key will be used.
• feedback_key: Optional feedback key this judge will write feedback to. If omitted, the judge key will be used.
• use_case: Explicit use-case key. Falls back to client.default_use_case when omitted.

Update an existing Judge version. Any field set to None will be left unchanged.

Delete a Judge. Returns True on success.

List all Judges for the given use case.

List all historical versions of a Judge key.

Retrieve a specific Judge by key (optionally specifying version).

Add model from the HuggingFace Model hub to Adaptive model registry. It will take several minutes for the model to be downloaded and converted to Adaptive format.

Arguments:

• hf_model_id: The ID of the selected model repo on HuggingFace Model Hub.
• output_model_key: The key that will identify the new model in Adaptive.
• hf_token: Your HuggingFace Token, needed to validate access to gated/restricted model.

Add proprietary external model to Adaptive model registry.

Arguments:

• name: Adaptive name for the new model.
• external_model_id: Should match the model id publicly shared by the model provider.
• api_key: API Key for authentication against external model provider.
• provider: External proprietary model provider.

List all models in Adaptive model registry.

Get the details for a model.

Arguments:

• model: Model key.

Attach a model to the client's use case.

Arguments:

• model: Model key.
• wait: If the model is not deployed already, attaching it to the use case will automatically deploy it. If True, this call blocks until model is Online.
• make_default: Make the model the use case's default on attachment.

Detach model from client's use case.

Arguments:

• model: Model key.

Update compute config of model.

Update config of model attached to client's use case.

Arguments:

• model: Model key.
• is_default: Change the selection of the model as default for the use case. True to promote to default, False to demote from default. If None, no changes are applied.
• attached: Whether model should be attached or detached to/from use case. If None, no changes are applied.
• desired_online: Turn model inference on or off for the client use case. This does not influence the global status of the model, it is use case-bounded. If None, no changes are applied.

Deploy a model, loading it to memory and making it ready for inference.

Arguments:

• model: Model key.
• wait: If True, call block until model is in Online state.

Terminate model, removing it from memory and making it unavailable to all use cases.

Arguments:

• model: Model key.
• force: If model is attached to several use cases, force must equal True in order for the model to be terminated.

Creates new role.

Arguments:

• key: Role key.
• permissions: List of permission identifiers such as use_case:read. You can list all possible permissions with client.permissions.list().
• name: Role name; if not provided, defaults to key.

Create a new training job.

Arguments:

• model: Model to train.
• data_source: Source of data to train on; either an uploaded dataset, or logged completions.
• data_config: Training data configuration.
• feedback_type: If training on metric training_objective usingcompletions that have both metric and preference feedback logged, you must specify which of those sets to train on.
• parameter_efficient: If True, training will be parameter-efficient, and the output model will be a lightweight adapter coupled to the selected backbone model.
• aligment_method: Alignment method selection.
• alignment_objective: Configuration for the training objective, determines the reward during training.
• alignment_params: Alignment method-specific hyperparameter configuration.
• base_training_params: Common training parameters.
• job_name: Human readable training job name.
• output_model_name: Name for resulting model.
• wait: If True, only returns when training is finished.
• use_case: Target use case to associate training job with.
• compute_pool: Compute pool where training job will run.

Cancel ongoing training job.

List all training jobs.

Get details for training job.

Create new use case.

Arguments:

• key: Use case key.
• name: Human-readable use case name which will be rendered in the UI. If not set, will be the same as key.
• description: Description of model which will be rendered in the UI.

List all use cases.

Get details for the client's use case.

Get details of current user.

List all users registered to Adaptive deployment.

Update team and role for user.

Arguments:

• email: User email.
• team: Key of team to which user will be added to.
• role: Assigned role

Creates a new A/B test in the client's use case.

Arguments:

• ab_test_key: A unique key to identify the AB test.
• feedback_key: The feedback key against which the AB test will run.
• models: The models to include in the AB test; they must be attached to the use case.
• traffic_split: Percentage of production traffic to route to AB test. traffic_split*100 % of inference requests for the use case will be sent randomly to one of the models included in the AB test.
• feedback_type: What type of feedback to run the AB test on, metric (direct) or preference (comparison).
• auto_deploy: If set to True, when the AB test is completed, the winning model automatically gets promoted to the use case default model.

Cancel an ongoing AB test.

Arguments:

• key: The AB test key.

List the use case AB tests.

Arguments:

• active: Filter on active or inactive AB tests.
• status: Filter on one of the possible AB test status.

Get the details of an AB test.

Arguments:

• key: The AB test key.

Create a chat completion.

Arguments:

• messages: Input messages, each dict with keys role and content.
• stream: If True, partial message deltas will be returned.
• model: Target model key for inference. If None, the requests will be routed to the use case's default model.
• stop: Sequences or where the API will stop generating further tokens.
• max_tokens: Maximum # of tokens allowed to generate.
• temperature: Sampling temperature.
• top_p: Threshold for top-p sampling.
• stream_include_usage: If set, an additional chunk will be streamed with the token usaage statistics for the entire request.
• user: ID of user making request. If not None, will be logged as metadata for the request.
• ab_campaign: AB test key. If set, request will be guaranteed to count towards AB test results, no matter the configured traffic_split.
• n: Number of chat completions to generate for each input messages.
• labels: Key-value pairs of interaction labels.

Examples:

# async streaming chat request
async def async_chat_stream():
    stream_response = aclient.chat.create(
        model="model_key", messages=[{"role": "user", "content": "Hello from SDK"}], stream=True
    )

    print("Async chat streaming response: ", end="", flush=True)
    async for chunk in await stream_response:
        if chunk.choices:
            content = chunk.choices[0].delta.content
            print(content, end="", flush=True)

Upload a dataset from a file. File must be jsonl, where each line should match structure in example below.

Arguments:

• file_path: Path to jsonl file.
• dataset_key: New dataset key.
• name: Optional name to render in UI; if None, defaults to same as dataset_key.

Example:

{"messages": [{"role": "system", "content": "<optional system prompt>"}, {"role": "user", "content": "<user content>"}, {"role": "assistant", "content": "<assistant answer>"}], "completion": "hey"}

List previously uploaded datasets.

Get details for dataset.

Arguments:

• key: Dataset key.

Generate a dataset from a file.

Arguments:

• model: Model to use for data generation.
• file_path: Path to jsonl file.
• dataset_key: New dataset key.
• dataset_type: Type of dataset to generate.
• chunks_per_new_sample: Number of document chunks that will be included in each generated sample's prompt.
• new_system_prompt: Optional new system prompt to be prepended to all generated samples.
• name: Optional name to render in UI; if None, defaults to same as dataset_key.
• use_case: Use case to associate with the dataset.
• compute_pool: Compute pool to use for generation.

Creates embeddings inference request.

Arguments:

• input: Input text to embed.
• model: Target model key for inference. If None, the requests will be routed to the use case's default model. Request will error if default model is not an embedding model.
• encoding_format: Encoding format of response.
• user: ID of user making the requests. If not None, will be logged as metadata for the request.

Create a new evaluation job.

Arguments:

• data_source: Source of data to evaluate on; either an uploaded dataset, or logged completions.
• data_config: Input data configuration.
• models: Models to evaluate.
• judge_model: Model key of judge.
• method: Eval method (built in method, or custom eval).
• generation_params_per_model: Optional list of models to evaluate with their parameters for generation. The list should contain tuples, where the first element is the model key, and the second element is the generation parameters. If a model is not present in the list, the default generation parameters will be used.
• custom_eval_config: Custom evaluation configuration. Only required if method=="custom".
• name: Optional name for evaluation job.
• use_case: Target use case to associate evaluation job with. Overrides client's default use case.
• compute_pool: Optional compute pool key where evaluation job will run on.

Register a new feedback key. Feedback can be logged against this key once it is created.

Arguments:

• key: Feedback key.
• kind: Feedback kind. If "bool", you can log values 0, 1, True or False only. If "scalar", you can log any integer or float value.
• scoring_type: Indication of what good means for this feeback key; a higher numeric value (or True) , or a lower numeric value (or False).
• name: Human-readable feedback name that will render in the UI. If None, will be the same as key.
• description: Description of intended purpose or nuances of feedback. Will render in the UI.

List all feedback keys.

Get the details of a feedback key.

Arguments:

• feedback_key: The feedback key.

Link a feedback key to the client's use case. Once a feedback key is linked to a use case, its statistics and associations with interactions will render in the UI.

Arguments:

• feedback_key: The feedback key to be linked.

Unlink a feedback key from the client's use case.

Arguments:

• feedback_key: The feedback key to be unlinked.

Log metric feedback for a single completion, which can be a float, int or bool depending on the kind of feedback_key it is logged against.

Arguments:

• value: The feedback values.
• completion_id: The completion_id to attach the feedback to.
• feedback_key: The feedback key to log against.
• user: ID of user submitting feedback. If not None, will be logged as metadata for the request.
• details: Textual details for the feedback. Can be used to provide further context on the feedback value.

Log preference feedback between 2 completions.

Arguments:

• feedback_key: The feedback key to log against.
• preferred_completion: Can be a completion_id or a dict with keys model and text, corresponding the a valid model key and its attributed completion.
• other_completion: Can be a completion_id or a dict with keys model and text, corresponding the a valid model key and its attributed completion.
• user: ID of user submitting feedback.
• prompt: Input text prompt. Ignored if preferred_ and other_completion are completion_ids.
• messages: Input chat messages, each dict with keys role and content. Ignored if preferred_ and other_completion are completion_ids.
• tied: Indicator if both completions tied as equally bad or equally good.

Create/log an interaction.

Arguments:

• model: Model key.
• completion: Model completion.
• prompt: Input text prompt.
• messages: Input chat messages, each dict should have keys role and content.
• feedbacks: List of feedbacks, each dict should with keys feedback_key, value and optional(details).
• user: ID of user making the request. If not None, will be logged as metadata for the interaction.
• ab_campaign: AB test key. If set, provided feedbacks will count towards AB test results.
• labels: Key-value pairs of interaction labels.
• created_at: Timestamp of interaction creation or ingestion.

List interactions in client's use case.

Arguments:

• order: Ordering of results.
• filters: List filters.
• page: Paging config.
• group_by: Retrieve interactions grouped by selected dimension.

Get the details for one specific interaction.

Arguments:

• completion_id: The ID of the completion.

Add model from the HuggingFace Model hub to Adaptive model registry. It will take several minutes for the model to be downloaded and converted to Adaptive format.

Arguments:

• hf_model_id: The ID of the selected model repo on HuggingFace Model Hub.
• output_model_key: The key that will identify the new model in Adaptive.
• hf_token: Your HuggingFace Token, needed to validate access to gated/restricted model.

Add proprietary external model to Adaptive model registry.

Arguments:

• name: Adaptive name for the new model.
• external_model_id: Should match the model id publicly shared by the model provider.
• api_key: API Key for authentication against external model provider.
• provider: External proprietary model provider.

List all models in Adaptive model registry.

Get the details for a model.

Arguments:

• model: Model key.

Attach a model to the client's use case.

Arguments:

• model: Model key.
• wait: If the model is not deployed already, attaching it to the use case will automatically deploy it. If True, this call blocks until model is Online.
• make_default: Make the model the use case's default on attachment.

Detach model from client's use case.

Arguments:

• model: Model key.

Update compute config of model.

Update config of model attached to client's use case.

Arguments:

• model: Model key.
• is_default: Change the selection of the model as default for the use case. True to promote to default, False to demote from default. If None, no changes are applied.
• attached: Whether model should be attached or detached to/from use case. If None, no changes are applied.
• desired_online: Turn model inference on or off for the client use case. This does not influence the global status of the model, it is use case-bounded. If None, no changes are applied.

Deploy a model, loading it to memory and making it ready for inference.

Arguments:

• model: Model key.
• wait: If True, call block until model is in Online state.

Terminate model, removing it from memory and making it unavailable to all use cases.

Arguments:

• model: Model key.
• force: If model is attached to several use cases, force must equal True in order for the model to be terminated.

Creates new role.

Arguments:

• key: Role key.
• permissions: List of permission identifiers such as use_case:read. You can list all possible permissions with client.permissions.list().
• name: Role name; if not provided, defaults to key.

Create a new training job.

Arguments:

• model: Model to train.
• data_source: Source of data to train on; either an uploaded dataset, or logged completions.
• data_config: Training data configuration.
• feedback_type: If training on metric training_objective usingcompletions that have both metric and preference feedback logged, you must specify which of those sets to train on.
• parameter_efficient: If True, training will be parameter-efficient, and the output model will be a lightweight adapter coupled to the selected backbone model.
• aligment_method: Alignment method selection.
• alignment_objective: Configuration for the training objective, determines the reward during training.
• alignment_params: Alignment method-specific hyperparameter configuration.
• base_training_params: Common training parameters.
• job_name: Human readable training job name.
• output_model_name: Name for resulting model.
• wait: If True, only returns when training is finished.
• use_case: Target use case to associate training job with.
• compute_pool: Compute pool where training job will run.

Cancel ongoing training job.

List all training jobs.

Get details for training job.

Create new use case.

Arguments:

• key: Use case key.
• name: Human-readable use case name which will be rendered in the UI. If not set, will be the same as key.
• description: Description of model which will be rendered in the UI.

List all use cases.

Get details for the client's use case.

Get details of current user.

List all users registered to Adaptive deployment.

Update team and role for user.

Arguments:

• email: User email.
• team: Key of team to which user will be added to.
• role: Assigned role