Yelp Adaptive

Accessible conversations for Deaf diners powered by Yelp AI search + Google Gemini Generative UI.

Inspiration

For millions of people with hearing impairments, aphasia, or speech disorders, walking into a new restaurant can be stressful. Traditional AAC apps are static—they rarely reflect the specific dishes, staff norms, or sensory cues of the room someone just entered. Yelp Adaptive grew from the realization that context is the missing accessibility layer. By letting Yelp’s data “read the room” and pairing it with an LLM that acts as a UI architect, we can craft a bespoke communication surface the moment Sarah (our Deaf persona) steps inside.

What it does

Deaf-friendly contextual search

Sarah can type natural descriptions like “A quiet place with good lighting and patient service.” Instead of simple keyword matching, searchYelpBusinesses() hits the Yelp AI Chat endpoint, which reasons over review sentiment, ambiance, and accessibility attributes so results prioritize comfort and safety.

Generative UI (core innovation)

Selecting a venue fetches its menus, photos, reviews, and attributes. Gemini 3 Flash then generates a JSON schema that SmartRenderer turns into UI sections (menu, service, logistics, issues, payment). A steakhouse yields doneness selectors, a boba shop spawns sugar/ice sliders, and a sushi bar emphasizes rolls and ingredient swaps—none of these layouts are hard-coded; they’re inferred live.

Visual-to-audio communication bridge

Each card combines Yelp imagery, iconography, and polite TTS phrases. When Sarah taps “Cheeseburger, medium-rare” and adds modifiers like “no onions,” the docked CommunicationStrip assembles a sentence (“I would like the cheeseburger, medium-rare, with no onions, please”) and streams Gemini 2.5 Flash TTS so the browser speaks for her.

How we built it

We designed the system as a Generative UI pipeline that separates intelligence from rendering.

Intelligence layer (backend concept)

• FastAPI service (not included in this repo) orchestrates Yelp Fusion AI API for conversational queries plus Yelp Fusion data for structured menus/photos.
• Prompts send all contextual data to Gemini 3 Flash, instructing it to obey a strict JSON schema so hallucinated widgets can’t break the UI.
• Imagen 3 (or curated stock photos) can backfill imagery when Yelp lacks photography.
• Gemini 2.5 Flash Preview TTS produces PCM audio buffers for high-quality speech output.

Rendering layer (this repo)

• React + Vite front end renders whatever schema arrives. SmartRenderer builds sections, ActionCard handles selection states, and CommunicationStrip manages audio playback.
• Modifier chips, helper CTAs, and “New need” modals let Sarah customize requests or ask Gemini for more cards on the fly.
• Tailwind via CDN provides styling inside index.html, while TypeScript types (types.ts) keep all schema data aligned.

Request flow

1. User describes a venue in the landing search (App.tsx, idle state).
2. searchYelpBusinesses() posts to Yelp AI Chat via YELP_PROXY_URL, using the API key in constants.ts. Failures fall back to MOCK_YELP_DATA.
3. Selecting a business triggers generateAdaptiveUI() which prompts Gemini 3 Flash for the UI schema, validated by responseSchema.
4. The schema feeds SmartRenderer, which renders sections and listens for selection/modifier changes.
5. Pressing “Speak” calls generateSpeech(), decodes inline base64 PCM via pcmToAudioBuffer(), and plays it with the Web Audio API.
6. “New need” buttons open a modal that calls generateExtraCards() for on-demand cards scoped to the current section.

What’s inside

• Smart trip planning – natural-language search surfaces Yelp AI recommendations with rich metadata.
• Adaptive communication kit – Gemini-generated schema drives the five-section UI without hard-coded layouts.
• Modifier-aware speech – selected cards stream to Gemini TTS, honoring any modifier chips.
• On-demand expansion – modal prompts Gemini for extra cards per section to keep the UI bespoke.
• Offline-friendly demo – mock data keeps the flow demoable when Yelp API access fails.

Architecture

Layer	Files	Notes
React UI	App.tsx, components/*	Tailwind via CDN powers styling. SmartRenderer orchestrates schema rendering, selection state, modifiers, modal, and the CommunicationStrip.
Data contracts	types.ts	Shared TypeScript types for Yelp payloads and generated UI schemas keep React + services in sync.
External services	services/yelpService.ts, services/geminiService.ts	Yelp proxy fetches recommendations; Gemini handles UI schemas, extra cards, and PCM speech. pcmToAudioBuffer decodes audio for playback.
Config	constants.ts, vite.config.ts, tsconfig.json, metadata.json	Contains proxy URL, default mock business, Vite env wiring (injects GEMINI_API_KEY), and AI Studio metadata.
Entry points	index.html, index.tsx	Boots the React app, injects Tailwind, fonts, and an import map (for AI Studio hosting), and mounts <App />.

Challenges we ran into

• Generative UI hallucinations – early prompts produced invalid JSON or unsupported widgets; strict schema validation plus clearer instructions stabilized outputs.
• Latency – real-time schema generation can lag. We rely on Gemini 2.0/3 Flash for faster responses and kick off UI generation while users review Yelp results.
• Data consistency – parsing unstructured menu text into modifiers (e.g., “Add avocado +$2”) required iterative prompt engineering to teach the model how to classify extras vs. core dishes.

Accomplishments we’re proud of

• True generative UI – sushi spots, pizzerias, and coffee shops all yield different interfaces without bespoke React branches.
• Contextual helper actions – helper chips like “Chopsticks” or “Restroom code” appear automatically when Yelp context hints they are needed.
• Accessibility-first use of Yelp AI – demonstrates Yelp AI as a tool for comfort and autonomy, not just food discovery.

What we learned

• Accessibility is personal – no single “accessibility mode” works for every disability; generative UI scales personalization.
• Data is a superpower – Yelp photos/reviews become structured, assistive interfaces once interpreted by AI.

What’s next

1. Multi-persona expansion – add blind (audio-first) or wheelchair (logistics-first) profiles using the same schema pipeline.
2. Wearable integration – sync cards to smartwatches or AR glasses for hands-free communication.
3. Offline mode – cache generated schemas for favorite locations so users can communicate without connectivity.

Prerequisites

• Node.js 20+ (tested with Vite 6 / React 19).
• A Yelp AI API key.
• A Google Gemini API key with access to Gemini 3 Flash Preview and Gemini 2.5 Flash Preview TTS.

Quick start

git clone <repo-url>
cd yelp-adaptive
npm install

# Create an .env.local file
echo "GEMINI_API_KEY=your-key" > .env.local

npm run dev

Visit http://localhost:3000.

Configuration

Variable / File	Purpose
.env.local → GEMINI_API_KEY	Injected into the Vite build so services/geminiService.ts can instantiate GoogleGenAI.
constants.ts → YELP_API_KEY	Demo key bundled for hackathon use. Replace with your own secrets manager in production.
constants.ts → YELP_PROXY_URL	Defaults to https://corsproxy.io/? so the browser can call Yelp. Swap for your own proxy/back end in production.
constants.ts → MOCK_YELP_DATA	Returned if the live Yelp request fails (useful for demos/offline screenshots).
metadata.json	Used by Google AI Studio Apps to request microphone permissions for TTS playback.

Key UI states (App.tsx)

• idle: Landing hero with natural-language search field and suggestions.
• searching_yelp: Animated spinner while Yelp results load.
• results_found: Grid of recommendation cards; clicking one runs the Gemini UI generation.
• generating_ui: Spinner while Gemini responds.
• ready: SmartRenderer renders the generated schema, modifiers, add-new modal, and the communication strip.
• error: Displays the stored error message with a reset CTA.

Development scripts

Command	Description
npm run dev	Starts Vite dev server on 0.0.0.0:3000.
npm run build	Emits a production bundle under dist/.
npm run preview	Serves the production build locally.

Project structure

.
├── App.tsx
├── components/
│   ├── ActionCard.tsx
│   ├── CommunicationStrip.tsx
│   └── SmartRenderer.tsx
├── services/
│   ├── geminiService.ts
│   └── yelpService.ts
├── types.ts
├── constants.ts
├── metadata.json
├── index.html
├── index.tsx
├── package.json
├── tsconfig.json
└── vite.config.ts

Extending the app

1. Swap search providers – implement a new function in services/yelpService.ts (or add parameters) and update handleSearch in App.tsx.
2. Add schema sections – update GenUISection['category'], extend getSectionIcon() in SmartRenderer, and adjust the Gemini prompt to include the new category.
3. Persist generated cards – sync schema state to a backend or local storage whenever generateExtraCards() resolves.
4. Improve security – move API calls server-side, store keys securely, and issue short-lived tokens to the client.

Limitations / notes

• API keys live on the client for hackathon speed—move them server-side before production.
• Tailwind currently loads via CDN inside index.html; install it locally if you prefer the standard PostCSS pipeline.
• Audio playback relies on window.atob and the Web Audio API, so it must run in the browser (not SSR).

License

MIT — feel free to adapt for your own accessibility hackathon projects.