Skip to content

ai-sdk.mdx

Latest commit

 

History

History
343 lines (263 loc) · 10.1 KB

ai-sdk.mdx

File metadata and controls

343 lines (263 loc) · 10.1 KB
title Vercel AI SDK
sidebarTitle Vercel AI SDK
description Use the @runpod/ai-sdk-provider package to integrate Public Endpoints with the Vercel AI SDK.
tag NEW

The @runpod/ai-sdk-provider package integrates Runpod Public Endpoints with the Vercel AI SDK. This gives you a streamlined, type-safe interface for text generation, streaming, and image generation in JavaScript and TypeScript projects.

The Vercel AI SDK is a popular open-source library for building AI applications. By using the Runpod provider, you can access Runpod's Public Endpoints using the same patterns and APIs you'd use with other AI providers like OpenAI or Anthropic.

Why use the Vercel AI SDK?

  • Unified interface: Use the same generateText, streamText, and generateImage functions regardless of which AI provider you're using.
  • Type safety: Full TypeScript support with typed responses and parameters.
  • Streaming built-in: First-class support for streaming text responses.
  • Framework integrations: Works seamlessly with Next.js, React, Svelte, and other frameworks.
  • Provider switching: Easily switch between Runpod and other providers without rewriting your code.

Installation

Install the Runpod provider alongside the Vercel AI SDK:

npm install @runpod/ai-sdk-provider ai

Configuration

Default configuration

The provider reads your API key from the RUNPOD_API_KEY environment variable by default. Import the runpod instance and start using it immediately:

import { runpod } from "@runpod/ai-sdk-provider";

Set the environment variable in your shell or .env file:

export RUNPOD_API_KEY="YOUR_API_KEY"

Custom configuration

For more control, use createRunpod to create a custom provider instance:

import { createRunpod } from "@runpod/ai-sdk-provider";

const runpod = createRunpod({
  apiKey: "YOUR_API_KEY",
  baseURL: "https://api.runpod.ai/v2",
  headers: {
    "X-Custom-Header": "value",
  },
});
Option Description Default
apiKey Your Runpod API key RUNPOD_API_KEY env var
baseURL Base URL for API requests https://api.runpod.ai/v2
headers Custom HTTP headers to include with requests {}

Text generation

Basic text generation

Use generateText to generate text from a prompt:

import { runpod } from "@runpod/ai-sdk-provider";
import { generateText } from "ai";

const { text, finishReason, usage } = await generateText({
  model: runpod("qwen3-32b-awq"),
  prompt: "Write a Python function that checks if a number is prime:",
});

console.log(text);
console.log(`Tokens used: ${usage.totalTokens}`);

The response includes:

  • text: The generated text
  • finishReason: Why generation stopped (stop, length, etc.)
  • usage: Token counts (promptTokens, completionTokens, totalTokens)

Chat conversations

For multi-turn conversations, pass a messages array instead of a prompt:

import { runpod } from "@runpod/ai-sdk-provider";
import { generateText } from "ai";

const { text } = await generateText({
  model: runpod("qwen3-32b-awq"),
  messages: [
    {
      role: "system",
      content: "You are a helpful coding assistant. Be concise.",
    },
    {
      role: "user",
      content: "How do I read a JSON file in Python?",
    },
  ],
});

console.log(text);

Generation parameters

Control the generation behavior with additional parameters:

const { text } = await generateText({
  model: runpod("qwen3-32b-awq"),
  prompt: "Write a creative story about a robot:",
  temperature: 0.8, // Higher = more creative (0-1)
  maxTokens: 500, // Maximum tokens to generate
  topP: 0.9, // Nucleus sampling threshold
});

Streaming

For real-time output (useful for chat interfaces), use streamText:

import { runpod } from "@runpod/ai-sdk-provider";
import { streamText } from "ai";

const { textStream } = await streamText({
  model: runpod("qwen3-32b-awq"),
  prompt: "Explain quantum computing in simple terms:",
  temperature: 0.7,
});

for await (const chunk of textStream) {
  process.stdout.write(chunk);
}

Streaming with callbacks

You can also use callbacks to handle streaming events:

import { runpod } from "@runpod/ai-sdk-provider";
import { streamText } from "ai";

const result = await streamText({
  model: runpod("qwen3-32b-awq"),
  prompt: "Write a poem about the ocean:",
  onChunk: ({ chunk }) => {
    if (chunk.type === "text-delta") {
      process.stdout.write(chunk.textDelta);
    }
  },
  onFinish: ({ text, usage }) => {
    console.log(`\n\nTotal tokens: ${usage.totalTokens}`);
  },
});

Image generation

Text-to-image

Generate images using models like Flux:

import { runpod } from "@runpod/ai-sdk-provider";
import { experimental_generateImage as generateImage } from "ai";
import { writeFileSync } from "fs";

const { image } = await generateImage({
  model: runpod.image("black-forest-labs-flux-1-dev"),
  prompt: "A serene mountain landscape at sunset, photorealistic",
  aspectRatio: "16:9",
});

// Save the image to a file
writeFileSync("output.png", image.uint8Array);

// Or access as base64
console.log(image.base64);

The response includes:

  • image.uint8Array: Binary image data
  • image.base64: Base64-encoded image
  • image.mimeType: Image MIME type (e.g., image/png)

Image editing

Edit existing images by providing reference images:

import { runpod } from "@runpod/ai-sdk-provider";
import { experimental_generateImage as generateImage } from "ai";

const { image } = await generateImage({
  model: runpod.image("google-nano-banana-edit"),
  prompt: {
    text: "Add modern Scandinavian furniture to this room",
    images: ["https://example.com/empty-room.png"],
  },
  aspectRatio: "16:9",
});

For models that support multiple reference images:

const { image } = await generateImage({
  model: runpod.image("google-nano-banana-edit"),
  prompt: {
    text: "Combine these into an epic band photo",
    images: [
      "https://example.com/drummer.png",
      "https://example.com/guitarist.png",
      "https://example.com/bassist.png",
      "https://example.com/singer.png",
    ],
  },
});

Provider options

Pass model-specific parameters using providerOptions:

const { image } = await generateImage({
  model: runpod.image("black-forest-labs-flux-1-dev"),
  prompt: "A sunset over the ocean",
  providerOptions: {
    runpod: {
      negative_prompt: "blurry, low quality, distorted",
      num_inference_steps: 30,
      guidance: 7.5,
      seed: 42,
      enable_safety_checker: true,
    },
  },
});
Option Description
negative_prompt Elements to exclude from the image
num_inference_steps Number of denoising steps (higher = more detail)
guidance How closely to follow the prompt (0-10)
seed Seed for reproducible results (-1 for random)
enable_safety_checker Enable content safety filtering
maxPollAttempts Max polling attempts for async generation
pollIntervalMillis Milliseconds between status polls

Supported models

Text models

Model ID Description
qwen3-32b-awq Qwen3 32B with AWQ quantization. Good for general text and code generation.
gpt-oss-120b GPT OSS 120B. Supports tool calling.

Image models

Model ID Description
black-forest-labs-flux-1-dev Flux Dev. High quality, detailed images.
black-forest-labs-flux-1-schnell Flux Schnell. Fast generation, good for prototyping.
google-nano-banana-edit Nano Banana Edit. Supports multiple reference images.
google/nano-banana-2-edit Nano Banana 2 Edit. Image editing with 14 aspect ratios and resolution options (1k/2k/4k).
bytedance-seedream-4-0-t2i Seedream 4.0. Text-to-image with good prompt adherence.
tongyi-mai/z-image-turbo Z-Image Turbo. Fast 6B parameter model with text-to-image support.

For a complete list of available models and their parameters, see the model reference.

Example: Chat application

Here's a complete example of a simple chat application using streaming:

import { runpod } from "@runpod/ai-sdk-provider";
import { streamText } from "ai";
import * as readline from "readline";

const rl = readline.createInterface({
  input: process.stdin,
  output: process.stdout,
});

const messages: { role: "user" | "assistant"; content: string }[] = [];

async function chat(userMessage: string) {
  messages.push({ role: "user", content: userMessage });

  const { textStream } = await streamText({
    model: runpod("qwen3-32b-awq"),
    system: "You are a helpful assistant.",
    messages,
  });

  let assistantMessage = "";
  process.stdout.write("\nAssistant: ");

  for await (const chunk of textStream) {
    process.stdout.write(chunk);
    assistantMessage += chunk;
  }

  messages.push({ role: "assistant", content: assistantMessage });
  console.log("\n");
}

function prompt() {
  rl.question("You: ", async (input) => {
    if (input.toLowerCase() === "exit") {
      rl.close();
      return;
    }
    await chat(input);
    prompt();
  });
}

console.log('Chat started. Type "exit" to quit.\n');
prompt();

Next steps