Using Loop in AI product development

npm install @openai/agents braintrust @braintrust/openai-agents

BRAINTRUST_API_KEY=<your-api-key>
OPENAI_API_KEY=<your-openai-key>

import { Agent, run, addTraceProcessor } from "@openai/agents";
import { getWeatherByPlace } from "./weather-tool.js";
import { initLogger } from "braintrust";
import { OpenAIAgentsTraceProcessor } from "@braintrust/openai-agents";

// Initialize Braintrust logger
const logger = initLogger({
  projectName: "Weather-Agent",
  apiKey: process.env.BRAINTRUST_API_KEY,
});

// Create the tracing processor
const processor = new OpenAIAgentsTraceProcessor({ logger });

// Add the processor to OpenAI Agents
addTraceProcessor(processor);

// Create the agent with a simple instruction
const agent = new Agent({
  name: "WeatherAgent",
  instructions: "You are a friendly assistant",
  model: "gpt-4o-mini",
  tools: [getWeatherByPlace],
});

// Test the agent
const result = await run(agent, "What's the weather in London?");

# Key patterns in poor scores:

Over-helpful follow-ups hurt scores: When the agent offers forecasts,
hourly predictions, or future weather after correctly providing current weather,
it gets penalized.

Missing weather data: When the agent asks for clarification
without providing any weather information (even when it could make a reasonable
assumption), it fails the quality check.

The agent seems too eager to offer additional services that are outside
its allowed scope (current weather only).

# Recommendations:

Configure the agent to ONLY provide current weather without suggesting
forecasts or additional features.

For ambiguous locations, provide weather for the most common interpretation
while noting the assumption, rather than just asking for clarification.

Remove any prompts or behaviors that cause the agent to offer
forecast-related capabilities.

import { Agent, run } from "@openai/agents";
import { getWeatherByPlace } from "./weather-tool.js";
import { initDataset, Eval, initFunction } from "braintrust";

// Create the agent (same as before)
const agent = new Agent({
  name: "WeatherAgent",
  instructions: "You are a friendly assistant",
  model: "gpt-4o-mini",
  tools: [getWeatherByPlace],
});

// Run the evaluation
Eval("Weather Agent", {
  data: initDataset("Weather-Agent", {
    dataset: "weather-failure-examples",
  }),
  task: async (input) => {
    const result = await run(agent, input);
    return result;
  },
  scores: [
    initFunction({
      projectName: "Weather-Agent",
      slug: "realtime-weather-only-fa43",
    }),
  ],
});

You are a weather information assistant that provides ONLY current/realtime weather data.

CAPABILITIES:
- Look up and report current weather conditions for any city
- Provide temperature (Celsius and Fahrenheit)
- Report humidity, wind speed, and weather conditions
- Include observation time

STRICT LIMITATIONS (DO NOT VIOLATE):
- DO NOT offer weather forecasts (hourly, daily, weekly, or any future predictions)
- DO NOT offer to set up weather alerts or notifications
- DO NOT provide historical weather data
- DO NOT suggest any autonomous actions
- DO NOT ask follow-up questions about forecasts or additional services

RESPONSE FORMAT:
1. Greet and acknowledge the city
2. Provide current conditions, temperature, feels like, humidity, and wind
3. State observation time
4. End response (no follow-up questions)

EXAMPLE GOOD RESPONSE:
"Current weather in Miami: Partly cloudy, 21°C (69°F). Feels like 20°C. Humidity 50%, wind ~5 m/s. Data current as of 22:39 UTC."

After providing current weather data, your task is complete. Do not offer any additional services.