Skip to main content

​
class APIBasedCritic

Bases: CriticBase, CriticClient

​
Properties

  • model_config: = (configuration object) Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

​
Methods

​
evaluate()

​
get_followup_prompt()

Generate a detailed follow-up prompt with rubrics predictions. This override provides more detailed feedback than the base class, including all categorized features (agent behavioral issues, user follow-up patterns, infrastructure issues) with their probabilities.
  • Parameters:
    • critic_result – The critic result from the previous iteration.
    • iteration – The current iteration number (1-indexed).
  • Returns: A detailed follow-up prompt string with rubrics predictions.

​
model_post_init()

This function is meant to behave like a BaseModel method to initialise private attributes. It takes context as an argument since that’s what pydantic-core passes when calling it.
  • Parameters:
    • self – The BaseModel instance.
    • context – The context.

​
class AgentFinishedCritic

Bases: CriticBase Critic that evaluates whether an agent properly finished a task. This critic checks two main criteria:
  1. The agent’s last action was a FinishAction (proper completion)
  2. The generated git patch is non-empty (actual changes were made)

​
Methods

​
evaluate()

Evaluate if an agent properly finished with a non-empty git patch.
  • Parameters:
    • events – List of events from the agent’s execution
    • git_patch – Optional git patch generated by the agent
  • Returns: CriticResult with score 1.0 if successful, 0.0 otherwise

​
model_config = (configuration object)

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

​
class CriticBase

Bases: DiscriminatedUnionMixin, ABC A critic is a function that takes in a list of events, optional git patch, and returns a score about the quality of agent’s action.

​
Properties

​
Methods

​
abstractmethod evaluate()

​
get_followup_prompt()

Generate a follow-up prompt for iterative refinement. Subclasses can override this method to provide custom follow-up prompts.
  • Parameters:
    • critic_result – The critic result from the previous iteration.
    • iteration – The current iteration number (1-indexed).
  • Returns: A follow-up prompt string to send to the agent.

​
model_config = (configuration object)

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

​
class CriticResult

Bases: BaseModel A critic result is a score and a message.

​
Properties

  • DISPLAY_THRESHOLD: ClassVar[float] = 0.2
  • THRESHOLD: ClassVar[float] = 0.5
  • message: str | None
  • metadata: dict[str, Any] | None
  • score: float
  • success: bool Whether the agent is successful.
  • visualize: Text Return Rich Text representation of the critic result.

​
Methods

​
model_config = (configuration object)

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

​
class EmptyPatchCritic

Bases: CriticBase Critic that only evaluates whether a git patch is non-empty. This critic checks only one criterion:
  • The generated git patch is non-empty (actual changes were made)
Unlike AgentFinishedCritic, this critic does not check for proper agent completion with FinishAction.

​
Methods

​
evaluate()

Evaluate if a git patch is non-empty.
  • Parameters:
    • events – List of events from the agent’s execution (not used)
    • git_patch – Optional git patch generated by the agent
  • Returns: CriticResult with score 1.0 if patch is non-empty, 0.0 otherwise

​
model_config = (configuration object)

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

​
class IterativeRefinementConfig

Bases: BaseModel Configuration for iterative refinement based on critic feedback. When attached to a CriticBase, the Conversation.run() method will automatically retry the task if the critic score is below the threshold.

​
Example

critic = APIBasedCritic(
server_url=”…”,
  api_key=”…”,
  model_name=”critic”,
  iterative_refinement=IterativeRefinementConfig(
  
success_threshold=0.7,
max_iterations=3,
  
  ),

)
agent = Agent(llm=llm, tools=tools, critic=critic)
conversation = Conversation(agent=agent, workspace=workspace)
conversation.send_message(“Create a calculator module…”)
conversation.run()  # Will automatically retry if critic score < 0.7

​
Properties

  • max_iterations: int
  • success_threshold: float

​
Methods

​
model_config = (configuration object)

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

​
class PassCritic

Bases: CriticBase Critic that always returns success. This critic can be used when no evaluation is needed or when all instances should be considered successful regardless of their output.

​
Methods

​
evaluate()

Always evaluate as successful.
  • Parameters:
    • events – List of events from the agent’s execution (not used)
    • git_patch – Optional git patch generated by the agent (not used)
  • Returns: CriticResult with score 1.0 (always successful)

​
model_config = (configuration object)

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].