AIO Cookbook

AIO Cookbook

AIO helps you observe, evaluate, and improve AI applications and workflows. Use this cookbook to create projects, track traces and spans, manage prompts, build datasets, run experiments, and send traces or threads for review.

Install and Configure

Install the tracer package:

pip install acceldata-aio-tracer

Configure the SDK from Python:

import acceldata_aio_tracer as aio
aio.configure(
url="YOUR_TENANT_URL/aio/api",
access_key="YOUR_ACCESS_KEY",
secret_key="YOUR_SECRET_KEY",
project_name="PROJECT_NAME",
)

Use the same values in the examples below:

aio_url = "YOUR_TENANT_URL/aio/api"
access_key = "YOUR_ACCESS_KEY"
secret_key = "YOUR_SECRET_KEY"
project_name = "customer-support-aio"

Feature Map

Projects

Use projects to organize traces, prompts, datasets, experiments, and other AIO resources for a specific application, team, or environment.

Create a project first so all related activity is grouped in one place.

import acceldata_aio_tracer as aio
project_name = "customer-support-aio"
client = aio.AcceldataTracer(
url=aio_url,
access_key=access_key,
secret_key=secret_key
)
client.rest_client.projects.create_project(name=project_name)
print(f"Project ready: {project_name}")

Traces and Spans

Use traces to track the full execution flow of a single AI request or workflow from start to finish.

Use spans to track individual operations inside a trace. Spans help you understand what happens at each step of a workflow, measure execution time, monitor model calls, and add metadata for debugging.

@track() is the simplest way to create traces and spans automatically. The first tracked function creates the trace, and nested tracked function calls become spans inside that trace.

from acceldata_aio_tracer import track, update_current_span
import ollama
@track(type="llm", project_name="customer-support-aio")
def call_llm(prompt):
response = ollama.chat(
model="llama3.1",
messages=[{"role": "user", "content": prompt}],
)
update_current_span(
metadata={"model": "llama3.1"},
tags=["customer-support", "llm-call"],
)
return response["message"]["content"]
@track(project_name="customer-support-aio")
def generate_answer(question):
prompt = f"Answer politely and concisely: {question}"
return call_llm(prompt)
print(generate_answer("Do you ship internationally?"))

import acceldata_aio_tracer as aio
import ollama
client = aio.AcceldataTracer(
url=aio_url,
access_key=access_key,
secret_key=secret_key,
project_name=project_name
)
question = "What is your return policy?"
trace = client.trace(
name="manual-customer-support-request",
input={"question": question},
)
span = trace.span(
name="ollama-call",
type="llm",
input={"question": question},
model="llama3.1"
)
response = ollama.chat(
model="llama3.1",
messages=[
{
"role": "user",
"content": f"Answer this customer support question: {question}",
}
],
)
answer = response["message"]["content"]
span.end(output={"answer": answer})
trace.end(output={"answer": answer})
client.flush()

Threads

Use threads to group related traces across a conversation or multi-step workflow. Threads help you track the full context of a user session instead of viewing each trace separately.

Pass thread_id through opik_args when calling a tracked function.

from acceldata_aio_tracer import track
import ollama
import uuid
thread_id = str(uuid.uuid4())
@track(project_name="customer-support-aio")
def handle_chat_turn(question):
response = ollama.chat(
model="llama3.1",
messages=[
{
"role": "user",
"content": f"Answer this customer support question: {question}",
}
],
)
return response["message"]["content"]
handle_chat_turn(
"Can I return an opened item?",
opik_args={"trace": {"thread_id": thread_id}},
)
handle_chat_turn(
"How long does the refund take?",
opik_args={"trace": {"thread_id": thread_id}},
)

Prompts

Use prompts to create reusable and versioned prompt templates for your AI workflows.

Creating a prompt syncs it to AIO. Creating another prompt with the same name and updated content creates a new version, making it easier to track prompt changes over time.

import acceldata_aio_tracer as aio
from acceldata_aio_tracer import Prompt, track
import ollama
aio.Prompt(
name="support-answer",
prompt="Answer clearly and politely: {{question}}",
project_name=project_name,
metadata={"owner": "support"},
tags=["production"],
)
client = aio.AcceldataTracer(
url=aio_url,
access_key=access_key,
secret_key=secret_key,
project_name=project_name
)
prompt = client.get_prompt(
name="support-answer",
project_name=project_name
)
@track(project_name=project_name)
def answer_with_prompt(question):
user_message = prompt.format(question=question)
response = ollama.chat(
model="llama3.1",
messages=[{"role": "user", "content": user_message}],
)
return response["message"]["content"]
print(answer_with_prompt("How do I reset my password?"))

Datasets

Use datasets to store evaluation examples and test cases for your AI applications.

Datasets help you compare prompt, model, or workflow changes before shipping to production.

import acceldata_aio_tracer as aio
client = aio.AcceldataTracer(
url=aio_url,
access_key=access_key,
secret_key=secret_key,
project_name=project_name
)
dataset = client.create_dataset(name="Product QA Dataset")
dataset.insert(
[
{
"input": {"question": "What is your return policy?"},
"expected_model_output": {"output": "30-day return policy"},
},
{
"input": {"question": "Do you ship internationally?"},
"expected_model_output": {"output": "Yes, we ship internationally"},
},
]
)
print(f"Dataset '{dataset.name}' created")

Experiments

Use experiments to run your application against a dataset and evaluate the results.

Experiments help you compare prompt versions, model settings, and workflow changes before shipping to production. Add scoring functions to measure output quality and track results over time.

import acceldata_aio_tracer as aio
from acceldata_aio_tracer import evaluate, track
from opik.evaluation.metrics.score_result import ScoreResult
import ollama
client = aio.AcceldataTracer(
url=aio_url,
access_key=access_key,
secret_key=secret_key,
project_name=project_name
)
dataset = client.get_or_create_dataset(name="Product QA Dataset")
@track(project_name="customer-support-aio")
def support_task(item):
question = item["input"]["question"]
response = ollama.chat(
model="llama3.1",
messages=[{"role": "user", "content": f"Answer this question: {question}"}],
)
return {"output": response["message"]["content"]}
def score(dataset_item, task_outputs):
expected = dataset_item["expected_model_output"]["output"].lower()
actual = task_outputs["output"].lower()
return ScoreResult(
name="match_score",
value=1.0 if expected in actual else 0.0,
)
results = evaluate(
dataset=dataset,
task=support_task,
scoring_functions=[score],
experiment_name="support-baseline",
)
print(results)

Annotation Queues

Use annotation queues to collect traces or threads for structured review, feedback, and follow-up workflows.

Annotation queues help with quality checks, expert review, evaluation feedback, and identifying cases that need further action.

import acceldata_aio_tracer as aio
import ollama
client = aio.AcceldataTracer(
url=aio_url,
access_key=access_key,
secret_key=secret_key,
project_name=project_name
)
queue = client.create_traces_annotation_queue(
name="support-review-queue",
description="Customer support traces for review",
)
traces = client.search_traces(
project_name="customer-support-aio",
filter_string="feedback_scores.match_score < 1",
max_results=1,
)
queue.add_traces(traces)