AIO Cookbook

AIO Cookbook

AIO helps you observe, evaluate, and improve AI applications and workflows. Use this cookbook to create projects, track traces and spans, manage prompts, build datasets, run experiments, and send traces or threads for review.

Install and Configure

Install the tracer package:

Bash

pip install acceldata-aio-tracer

Configure the SDK from Python:

Python

import acceldata_aio_tracer as aio

aio.configure(

url="YOUR_TENANT_URL/aio/api",

access_key="YOUR_ACCESS_KEY",

secret_key="YOUR_SECRET_KEY",

project_name="PROJECT_NAME",

)

Use the same values in the examples below:

Python

aio_url = "YOUR_TENANT_URL/aio/api"

access_key = "YOUR_ACCESS_KEY"

secret_key = "YOUR_SECRET_KEY"

project_name = "customer-support-aio"

Feature Map

Projects

Use projects to organize traces, prompts, datasets, experiments, and other AIO resources for a specific application, team, or environment.

Create a project first so all related activity is grouped in one place.

Python

import acceldata_aio_tracer as aio

project_name = "customer-support-aio"

client = aio.AcceldataTracer(

url=aio_url,

access_key=access_key,

secret_key=secret_key

)

client.rest_client.projects.create_project(name=project_name)

print(f"Project ready: {project_name}")

Traces and Spans

Use traces to track the full execution flow of a single AI request or workflow from start to finish.

Use spans to track individual operations inside a trace. Spans help you understand what happens at each step of a workflow, measure execution time, monitor model calls, and add metadata for debugging.

@track() is the simplest way to create traces and spans automatically. The first tracked function creates the trace, and nested tracked function calls become spans inside that trace.

Python

from acceldata_aio_tracer import track, update_current_span

import ollama

@track(type="llm", project_name="customer-support-aio")

def call_llm(prompt):

response = ollama.chat(

model="llama3.1",

messages=[{"role": "user", "content": prompt}],

)

update_current_span(

metadata={"model": "llama3.1"},

tags=["customer-support", "llm-call"],

)

return response["message"]["content"]

@track(project_name="customer-support-aio")

def generate_answer(question):

prompt = f"Answer politely and concisely: {question}"

return call_llm(prompt)

print(generate_answer("Do you ship internationally?"))

client = aio.AcceldataTracer(

url=aio_url,

access_key=access_key,

secret_key=secret_key,

project_name=project_name

)

question = "What is your return policy?"

trace = client.trace(

name="manual-customer-support-request",

input={"question": question},

)

span = trace.span(

name="ollama-call",

type="llm",

input={"question": question},

model="llama3.1"

)

response = ollama.chat(

model="llama3.1",

messages=[

{

"role": "user",

"content": f"Answer this customer support question: {question}",

}

],

)

answer = response["message"]["content"]

span.end(output={"answer": answer})

trace.end(output={"answer": answer})

client.flush()

Threads

Use threads to group related traces across a conversation or multi-step workflow. Threads help you track the full context of a user session instead of viewing each trace separately.

Pass thread_id through opik_args when calling a tracked function.

Python

from acceldata_aio_tracer import track

import ollama

import uuid

thread_id = str(uuid.uuid4())

@track(project_name="customer-support-aio")

def handle_chat_turn(question):

response = ollama.chat(

model="llama3.1",

messages=[

{

"role": "user",

"content": f"Answer this customer support question: {question}",

}

],

)

return response["message"]["content"]

handle_chat_turn(

"Can I return an opened item?",

opik_args={"trace": {"thread_id": thread_id}},

)

handle_chat_turn(

"How long does the refund take?",

opik_args={"trace": {"thread_id": thread_id}},

)

Prompts

Use prompts to create reusable and versioned prompt templates for your AI workflows.

Creating a prompt syncs it to AIO. Creating another prompt with the same name and updated content creates a new version, making it easier to track prompt changes over time.

Python

import acceldata_aio_tracer as aio

from acceldata_aio_tracer import Prompt, track

import ollama

aio.Prompt(

name="support-answer",

prompt="Answer clearly and politely: {{question}}",

project_name=project_name,

metadata={"owner": "support"},

tags=["production"],

)

client = aio.AcceldataTracer(

url=aio_url,

access_key=access_key,

secret_key=secret_key,

project_name=project_name

)

prompt = client.get_prompt(

name="support-answer",

project_name=project_name

)

@track(project_name=project_name)

def answer_with_prompt(question):

user_message = prompt.format(question=question)

response = ollama.chat(

model="llama3.1",

messages=[{"role": "user", "content": user_message}],

)

return response["message"]["content"]

print(answer_with_prompt("How do I reset my password?"))

Datasets

Use datasets to store evaluation examples and test cases for your AI applications.

Datasets help you compare prompt, model, or workflow changes before shipping to production.

Python

import acceldata_aio_tracer as aio

client = aio.AcceldataTracer(

url=aio_url,

access_key=access_key,

secret_key=secret_key,

project_name=project_name

)

dataset = client.create_dataset(name="Product QA Dataset")

dataset.insert(

[

{

"input": {"question": "What is your return policy?"},

"expected_model_output": {"output": "30-day return policy"},

},

{

"input": {"question": "Do you ship internationally?"},

"expected_model_output": {"output": "Yes, we ship internationally"},

},

]

)

print(f"Dataset '{dataset.name}' created")

Experiments

Use experiments to run your application against a dataset and evaluate the results.

Experiments help you compare prompt versions, model settings, and workflow changes before shipping to production. Add scoring functions to measure output quality and track results over time.

Python

import acceldata_aio_tracer as aio

from acceldata_aio_tracer import evaluate, track

from opik.evaluation.metrics.score_result import ScoreResult

import ollama

client = aio.AcceldataTracer(

url=aio_url,

access_key=access_key,

secret_key=secret_key,

project_name=project_name

)

dataset = client.get_or_create_dataset(name="Product QA Dataset")

@track(project_name="customer-support-aio")

def support_task(item):

question = item["input"]["question"]

response = ollama.chat(

model="llama3.1",

messages=[{"role": "user", "content": f"Answer this question: {question}"}],

)

return {"output": response["message"]["content"]}

def score(dataset_item, task_outputs):

expected = dataset_item["expected_model_output"]["output"].lower()

actual = task_outputs["output"].lower()

return ScoreResult(

name="match_score",

value=1.0 if expected in actual else 0.0,

)

results = evaluate(

dataset=dataset,

task=support_task,

scoring_functions=[score],

experiment_name="support-baseline",

)

print(results)

Annotation Queues

Use annotation queues to collect traces or threads for structured review, feedback, and follow-up workflows.

Annotation queues help with quality checks, expert review, evaluation feedback, and identifying cases that need further action.

Python

import acceldata_aio_tracer as aio

import ollama

client = aio.AcceldataTracer(

url=aio_url,

access_key=access_key,

secret_key=secret_key,

project_name=project_name

)

queue = client.create_traces_annotation_queue(

name="support-review-queue",

description="Customer support traces for review",

)

traces = client.search_traces(

project_name="customer-support-aio",

filter_string="feedback_scores.match_score < 1",

max_results=1,

)

queue.add_traces(traces)