Graph Observability

Two parallel observability surfaces let operators see what a graph run is doing: GraphHooks for in-process control-flow reactions and OpenTelemetry spans for distributed-trace correlation. Both fire at the same lifecycle boundaries and are decoupled — use one, the other, or both.

Why Observe

A graph run can fan out into parallel agent nodes, suspend for human-in-the-loop input, retry under flakiness, and run for minutes before producing a final answer. Without observability:

• Operators cannot tell which node a long run is stuck on.
• SREs cannot correlate a graph span with a downstream incident.
• Auditors cannot reconstruct the order of approvals across a resume cycle.

The two surfaces together cover the spectrum:

• GraphHooks → in-process reactions (metrics, audit log writes, side-channel UI updates).
• OpenTelemetry spans → external trace correlation (Tempo, Honeycomb, Datadog, Jaeger), span-attribute querying, latency analysis.

Surface 1: GraphHooks

GraphHooks is an async callback ABC. Subclass and override only the methods you care about; unimplemented ones default to no-ops.

from typing import override
 
from augments.adk.graphs.hooks import GraphHooks
 
 
class AuditHooks(GraphHooks[Any]):
    @override
    async def on_node_interrupt(self, context, state, node_id, interrupt):
        audit_log.write({
            "graph_id": state.thread_id,
            "node_id": node_id,
            "interrupt_kind": interrupt.kind,
        })

Attach by passing the instance to Runner.arun_graph:

result = await Runner.arun_graph(graph, "go", hooks=[AuditHooks()])

Lifecycle Callbacks

Error Tolerance

Hook exceptions are logged but never abort the run — observability should not break orchestration. A hook that wants to halt execution must do so via a different channel (e.g. mutating context, raising inside the orchestration code itself).

Checkpointer Integration

The Checkpointer protocol extends HookProvider; checkpointers subscribe to on_node_end / on_graph_end and persist state. The BSP loop never calls save() directly — it just fires hooks. This keeps persistence pluggable: any object implementing HookProvider can be passed in the same hooks=[...] list as a plain GraphHooks subclass, and the registry dispatches both.

Surface 2: OpenTelemetry Spans

When an OTel tracer is installed, the BSP loop opens a three-level span tree on every graph run:

graph.<id>                              [bracket: full run]
├── graph.superstep.0                   [bracket: one BSP superstep]
│   ├── graph.node.<a>                  [bracket: one node attempt]
│   └── graph.node.<b>
├── graph.superstep.1
│   └── graph.node.<c>                  [status=interrupted on suspend]
└── graph.superstep.2  (resume)
    └── graph.node.<c>  (resumed)

Parallel siblings inside a superstep open as siblings under their parent superstep span — the BSP structure is visible in traces.

Installing the Tracer

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    ConsoleSpanExporter,
    SimpleSpanProcessor,
)
 
from augments.adk.tracing import set_tracer
from augments.adk.tracing.otel import OTelTracer
 
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
set_tracer(OTelTracer(provider=provider, service_name="my-graphs"))

For production exporters (OTLP, Jaeger, Honeycomb), swap ConsoleSpanExporter for the appropriate SpanExporter from opentelemetry-exporter-*.

Span Names

Attribute Reference

All graph-tracing attributes live under the augments.graph.* namespace.

Graph-run span (graph.<id>):

Superstep span (graph.superstep.<n>):

Per-node span (graph.node.<name>):

When a node raises a non-InterruptException, the span's OTel status is also set to ERROR with the exception type and message recorded as a span event (exception.type, exception.message).

Cost-Conservative Defaults

No span is emitted unless a tracer is explicitly installed via set_tracer(...). The default NoOpTracer returns a NoOpSpan for every factory call, and NoOpSpan.start() / NoOpSpan.finish() are empty — the disabled path is zero-overhead.

Custom Tracers

Building a backend that isn't OTel? Implement the Tracer protocol:

from augments.adk.tracing import Span, Tracer
from augments.adk.types.tracing import (
    AgentSpanData,
    CustomSpanData,
    FunctionSpanData,
    GenerationSpanData,
    GuardrailSpanData,
    HandoffSpanData,
    ResponseSpanData,
)
 
 
class MyTracer:
    def custom_span(self, data: CustomSpanData) -> Span[CustomSpanData]:
        # Inspect data.data["type"] to recognise graph-tracing spans:
        #   "graph"           — the run-level root
        #   "graph_superstep" — a BSP superstep boundary
        #   "graph_node"      — one node attempt
        if data.data.get("type") == "graph":
            self.on_graph_start(data.data["graph_id"])
        return Span(data)
 
    # ... agent_span / function_span / etc. ...

Graph-tracing spans route through custom_span so custom tracers don't need a graph-specific Tracer-protocol extension — the inner data["type"] discriminator carries the kind.

Worked Example

A runnable demo lives at examples/graphs/observability.py. It combines LoggingHooks (prints every callback) with an OTelTracer backed by ConsoleSpanExporter (prints every span). Topology is a fan-out → join graph that exercises the parallel-superstep path.

python examples/graphs/observability.py

For HITL suspend + resume coverage of the same observability surfaces, see examples/graphs/hitl.py — interrupt + resume cycles fire on_node_interrupt and stamp augments.graph.node.status="interrupted" on the per-node span at close.

Known Limitations

• Per-node attempts comes from the retry loop on the success path, or from the .attempts field of NodeRetriesExhaustedError / GraphNodeTimeoutError on failure. Any other path (nested-agent resume, GraphResumeError, other exceptions) records 1.