Telemetry

How Spineforge instruments agents, what it captures, and where data goes.

How instrumentation works

Spineforge is built on OpenLLMetry — an OpenTelemetry-based auto-instrumentation library for LLM frameworks. When spineforge.init() runs, it creates an OTel TracerProvider, registers a custom SpineforgeSpanProcessor, and activates OpenLLMetry instrumentors for whichever supported frameworks are installed.

No changes to your agent logic are needed. The instrumentors hook into framework internals (on_tool_start, on_tool_end, etc.) and fire automatically.

Runs and Actions

Spineforge groups telemetry into two levels:

• Run — one top-level agent invocation. Created by with spine.run(input=...) as run:. A Run has a run_id, a spine_id, a start timestamp, an end timestamp, a status (running / success / error), and the input/output text.
• Action — one LLM call or tool invocation inside a Run. An Action has an action_id, a run_id, a type (llm_call or tool_call), a name (model or tool name), duration, token counts, cost, and status.

Span classification

The SpineforgeSpanProcessor receives every completed OTel span and classifies it:

The Sink abstraction

All events flow through the Sink interface. Sinks are pluggable — you can add custom sinks without touching instrumentation code:

class Sink(ABC):
    def emit(self, event: dict) -> None: ...
    def flush(self) -> None: ...
    def shutdown(self) -> None: ...

Built-in sinks

• ConsoleSink — prints compact coloured one-liners to stdout. Active when SPINEFORGE_VERBOSE=true.
• FileSink — appends events as JSON lines to .spineforge/logs/actions.jsonl. Non-blocking: events go to a background thread queue. Never blocks the agent's hot path.
• APISink — POSTs batches to the Spineforge backend via the /telemetry/ingest endpoint. Authenticated with a scoped JWT. On any failure (non-2xx, connection error), falls back to FileSink automatically.

JSONL event format

Each line in actions.jsonl is one JSON object — either a run or action event:

// Run event — start
{"event": "run", "run_id": "...", "spine_id": "...", "status": "running", "input": "...", "started_at": "..."}

// Action event
{"event": "action", "action_id": "...", "run_id": "...", "type": "llm_call",
 "name": "gpt-4o", "duration_ms": 612, "status": "success",
 "tokens": {"prompt": 48, "completion": 12, "total": 60},
 "cost_usd": 0.00034800, "rate_card_id": "rc_gpt4o_2025q1"}

// Run event — end
{"event": "run", "run_id": "...", "status": "success", "ended_at": "...", "output": "..."}

The @track_tool decorator

For agents using the raw OpenAI SDK, tool functions are not auto-captured by any instrumentor. Add @track_tool to instrument them:

from spineforge import track_tool

@track_tool
def search_web(query: str) -> str:
    return requests.get("https://api.search.io", params={"q": query}).text

The decorator emits a tool_call action spanning the function's execution time. LangChain-based agents don't need this — the LangChain instrumentor fires on Tool.run() automatically.

Never in the data path

Agents call LLM and tool providers directly. Spineforge observes via the OTel span pipeline — asynchronously, after the fact. If Spineforge is unreachable, agents keep running and the FileSink buffers events locally.

Next steps

• Sinks — configure, extend, or disable sinks
• Metrics Reference — how cost and latency are calculated from telemetry data
• SDK Reference — full API for Spine, Run, and track_tool