Skip to content

README.md

Latest commit

 

History

History
61 lines (47 loc) · 1.76 KB

README.md

File metadata and controls

61 lines (47 loc) · 1.76 KB

Judges

LLM-as-judge implementations. Each judge implements the Judge interface.

Interface

interface Judge {
    name: string
    initialize(config: JudgeConfig): Promise<void>
    evaluate(input: JudgeInput): Promise<JudgeResult>
    getPromptForQuestionType(questionType: string, providerPrompts?: ProviderPrompts): string
    getModel(): LanguageModel
}

Adding a Judge

  1. Create src/judges/myjudge.ts
  2. Implement Judge interface
  3. Register in src/judges/index.ts
  4. Add to JudgeName type in src/types/judge.ts
  5. Add default model in src/utils/models.ts (DEFAULT_JUDGE_MODELS)

Required returns:

  • initialize() - Set up client with API key and model
  • evaluate() - Return { score: 0|1, label: "correct"|"incorrect", explanation: string }
  • getPromptForQuestionType() - Return prompt string for question type
  • getModel() - Return the initialized LanguageModel

Use these helpers from ./base.ts:

  • buildJudgePrompt(input) - Builds full prompt from JudgeInput
  • parseJudgeResponse(text) - Extracts JudgeResult from LLM response

Models

Add model config in src/utils/models.ts:

interface ModelConfig {
    id: string
    provider: "openai" | "anthropic" | "google"
    displayName: string
    supportsTemperature: boolean
    defaultTemperature: number
    maxTokensParam: "maxTokens" | "max_completion_tokens"
    defaultMaxTokens: number
}

Provider-Specific Prompts

Providers can override judge prompts. See providers/README.md.

Existing Judges

Judge SDK Default Model
openai @ai-sdk/openai gpt-4o
anthropic @ai-sdk/anthropic sonnet-4
google @ai-sdk/google gemini-2.5-flash