Fix OpenAI cua agent

[tkattkat]

why

• OpenAI now requires a screenshot on initial message for computer use agents

what changed

• added screenshot to initial message
• imported types from openai sdk for stronger typing

test plan

• tested locally using operator-example script

---

Summary by cubic

Send an initial screenshot with the first message in OpenAICUAClient to meet OpenAI’s computer-use requirement and prevent startup errors. Also adopt openai SDK response types for stricter typing.

• Bug Fixes

  • Capture and attach a high-detail screenshot on the first request when available; gracefully skip on failure.
  • Build the first message using EasyInputMessage, ResponseInputText, and ResponseInputImage; include an optional system message from userProvidedInstructions; make createInitialInputItems async and update method signatures to accept a unified OpenAIRequestInputItem type.

• Dependencies

  • Add changeset for a patch release of @browserbasehq/stagehand.

^{Written for commit f01d8a8. Summary will update on new commits. Review in cubic}

[changeset-bot]

🦋 Changeset detected

Latest commit: f01d8a8

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages

Name	Type
@browserbasehq/stagehand	Patch
@browserbasehq/stagehand-evals	Patch
@browserbasehq/stagehand-server-v3	Patch
@browserbasehq/stagehand-server-v4	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[cubic-dev-ai]

No issues found across 1 file

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

Architecture diagram

sequenceDiagram
    participant App as Client Application
    participant CUA as OpenAICUAClient
    participant Driver as Browser / Page Driver
    participant OpenAI as OpenAI API

    Note over App, OpenAI: Initial Message Flow for Computer Use Agent

    App->>CUA: step(input)
    
    rect rgb(240, 240, 240)
    Note right of CUA: NEW: Requirement for initial screenshot
    CUA->>CUA: CHANGED: createInitialInputItems() (async)
    
    opt Screenshot capture
        CUA->>Driver: screenshot()
        alt Success
            Driver-->>CUA: base64 image data
            CUA->>CUA: NEW: Create ResponseInputImage object
        else Failure
            Driver-->>CUA: error
            Note right of CUA: Gracefully skip image attachment
        end
    end
    
    CUA->>CUA: NEW: Construct OpenAIRequestInputItem list
    Note right of CUA: Includes both User Text and Screenshot
    end

    CUA->>OpenAI: chat.completions.create(messages)
    Note over CUA, OpenAI: Using NEW: Stronger SDK Types (ResponseInputText, etc.)
    
    OpenAI-->>CUA: Assistant Response (Tool Calls)
    CUA-->>App: Agent Action / Observation

Loading