⚡️ Latent Space - The new OpenAI Agents Platform
OpenAI have dropped their new Agents Platform. Swyx and Alessio chat with Nikunj Handa and Romain Huet of OpenAI to discuss the release.
Episode Show Notes
While everyone is now repeating that 2025 is the “Year of the Agent”, OpenAI is heads down building towards it. In the first 2 months of the year they released Operator and Deep Research (arguably the most successful agent archetype so far), and today they are bringing a lot of those capabilities to the API:
Responses API
Web Search Tool
Computer Use Tool
File Search Tool
A new open source Agents SDK with integrated Observability Tools
We cover all this and more in today’s lightning pod on YouTube !
More details here:
Responses API
In our Michelle Pokrass episode we talked about the Assistants API needing a redesign. Today OpenAI is launching the Responses API, “a more flexible foundation for developers building agentic applications”. It’s a superset of the chat completion API, and the suggested starting point for developers working with OpenAI models.
One of the big upgrades is the new set of built-in tools for the responses API: Web Search, Computer Use, and Files.
Web Search Tool
We previously had Exa AI on the podcast to talk about web search for AI. OpenAI is also now joining the race; the Web Search API is actually a new “model” that exposes two 4o fine-tunes: gpt-4o-search-preview and gpt-4o-mini-search-preview. These are the same models that power ChatGPT Search, and are priced at $30/1000 queries and $25/1000 queries respectively.
The killer feature is inline citations: you do not only get a link to a page, but also a deep link to exactly where your query was answered in the result page.
Computer Use Tool
The model that powers Operator, called Computer-Using-Agent (CUA), is also now available in the API. The computer-use-preview model is SOTA on most benchmarks, achieving 38.1% success on OSWorld for full computer use tasks, 58.1% on WebArena, and 87% on WebVoyager for web-based interactions.
As you will notice in the docs,
computer-use-previewis both a model and a tool through which you can specify the environment.Usage is priced at $3/1M input tokens and $12/1M output tokens, and it’s currently only available to users in tiers 3-5.
File Search Tool
File Search was also available in the Assistants API, and it’s now coming to Responses too. OpenAI is bringing search + RAG all under one umbrella, and we’ll definitely see more people trying to find new ways to build all-in-one apps on OpenAI.
Usage is priced at $2.50 per thousand queries and file storage at $0.10/GB/day, with the first GB free.
Agent SDK: Swarms++!
https://github.com/openai/openai-agents-python
To bring it all together, after the viral reception to Swarm , OpenAI is releasing an officially supported agents framework (which was previewed at our AI Engineer Summit ) with 4 core pieces:
Agents : Easily configurable LLMs with clear instructions and built-in tools.
Handoffs : Intelligently transfer control between agents.
Guardrails : Configurable safety checks for input and output validation.
Tracing utm_campaign=CTA_4">www.latent.space/subscribe
New OpenAI Agent Tools & APIs
🎧 Play snip - 2min (00:40 - 02:21)
- OpenAI is launching three new built-in tools: a web search tool (ChatGPT for search in the API), a file search tool (upload, parse, chunk, embed, and search your data), and a computer use tool (powering the Operator product in ChatGPT).
- They are also launching a new Responses API to support these tools, replacing the older ChatCompletions API and designed for future agentic products.
- Finally, they are releasing an upgraded Agents SDK (formerly Swarm) with built-in tracing in the OpenAI dashboard for multi-agent orchestration.
Responses API: A Unified and Flexible Approach
🎧 Play snip - 4min (02:41 - 06:16)
- The Responses API is a new, more flexible API from OpenAI, superseding the Chat Completions and Assistants APIs.
- It's designed for agentic workflows, supporting longer, multi-turn tasks and tool use.
- While Chat Completions remains available, Responses API offers a unified endpoint with broader capabilities.
- It incorporates features from the Assistants API, like convenient tool access, while simplifying integration.
- Responses API also offers stateless mode for compatibility with Chat Completions use cases.
GPT-4 with Search Preview Performance
🎧 Play snip - 9sec (08:09 - 08:19)
- OpenAI's web search API uses a GPT-4 model fine-tuned specifically for search, named GPT-4.0 Search Preview.
- GPT-4.0 with search has significantly better performance than the base model.
- Simple QA accuracy jumps from 38% with the base GPT-4 to 90% with the search-tuned model.
Computer Use Tool
🎧 Play snip - 2min (18:16 - 20:08)
- OpenAI's Computer Use tool empowers developers to build agents that can complete tasks using a computer or browser.
- This tool utilizes a custom model optimized for computer use, enabling agents to interact with the screen by clicking, scrolling, typing, and reporting back.
- By wrapping this functionality as a tool within the Responses API, developers can automate tasks and create multi-turn interactions where agents can execute complex actions over time.
DIY vs. OpenAI for Vector Search
🎧 Play snip - 1min (17:03 - 18:33)
- If starting from scratch, you'll have more control over chunking and retrieval strategies.
- OpenAI's tool provides an out-of-the-box managed service with customization options.
- Start with OpenAI's solution and see if it meets your needs.
- They plan to add more customization features over time.
- Consider hand-rolling with other solutions if you require complete control.
OpenAI Agent SDK
🎧 Play snip - 3min (21:34 - 24:28)
- OpenAI's new Agent SDK simplifies building and managing agents in production.
- It supports type checking, guardrails for safer execution, and tracing for monitoring agent behavior.
- The SDK is flexible, allowing integration with various chat completion APIs and tracing providers.
- It leverages the 'handoff' technique popularized by Swarm, enabling complex agent workflows.
- Combined with new tracing UIs in the OpenAI dashboard, developers can effectively troubleshoot and optimize their agent interactions.