Observability¶

Sentinel AI is designed with observability in mind. It provides a consolidated event bus that broadcasts key lifecycle events of an agentic run, allowing you to monitor performance, debug tool calls, and track token usage in real-time.

Event Bus¶

Each Agent instance has an EventBus that broadcasts events. You can subscribe to this bus to log activities, push metrics to external systems (like Prometheus or Datadog), or trigger custom side-effects.

Subscribing to Events¶

You can access the event bus via the Agent or AgentSetup:

agent.getEventBus().connect(event -> {
    System.out.println("Received event: " + event.getType());

    // Use the AgentEventVisitor for type-safe handling
    event.accept(new AgentEventVisitor<Void>() {
        @Override
        public Void visit(InputReceivedAgentEvent event) {
            System.out.println("User input: " + event.getContent());
            return null;
        }

        @Override
        public Void visit(ToolCalledAgentEvent event) {
            System.out.println("Calling tool: " + event.getToolCallName());
            return null;
        }

        @Override
        public Void visit(OutputGeneratedAgentEvent event) {
            System.out.println("Model usage: " + event.getUsage());
            return null;
        }

        // ... handle other events
    });
});

Core Event Types¶

Sentinel AI broadcasts the following events during an agent's execution:

Event Type	Description
`INPUT_RECEIVED`	Broadcasted when a new request is received by the agent.
`MESSAGE_RECEIVED`	Broadcasted when a response is received from the model.
`TOOL_CALLED`	Broadcasted before a tool is executed. Contains tool name and arguments.
`TOOL_CALL_COMPLETED`	Broadcasted after a tool execution finishes. Contains the result or error.
`TOOL_CALL_APPROVAL_DENIED`	Broadcasted if a tool execution was rejected by a `ToolRunApprovalSeeker`.
`OUTPUT_GENERATED`	Broadcasted when the final response is generated. Contains token usage and the response object.
`OUTPUT_ERROR`	Broadcasted if an error occurs during processing.
`MESSAGE_SENT`	Broadcasted when a message is sent to the model.

Model Usage Stats¶

For every run, Sentinel AI tracks detailed token usage statistics in the ModelUsageStats object. This is available in the AgentOutput and also broadcasted in the OutputGeneratedAgentEvent.

Token Details¶

The usage stats include breakdowns for: * Request Tokens: Tokens used in the input (system prompt, tools, history, facts). * Response Tokens: Tokens generated by the model. * Cached Tokens: Tokens retrieved from the model's cache (if supported by the provider). * Reasoning Tokens: Tokens used by reasoning models (like OpenAI's o1 series) for internal thought.

Example of accessing usage stats from an event:

@Override
public Void visit(OutputGeneratedAgentEvent event) {
    ModelUsageStats stats = event.getUsage();
    log.info("Request Tokens: {}, Response Tokens: {}, Reasoning: {}", 
        stats.getRequestTokens(), 
        stats.getResponseTokens(),
        stats.getResponseTokenDetails().getReasoningTokens());
    return null;
}

Logging¶

By default, Sentinel AI uses SLF4J for logging. You can enable debug logging for com.phonepe.sentinelai to see detailed information about prompt generation, tool discovery, and internal processing.

# Example logback or log4j config
logging.level.com.phonepe.sentinelai=DEBUG