YC Intent Radar

An automated scraper that pulls job listings and company data from YCombinator's Workatastartup platform. It bypasses login bottlenecks by utilizing authenticated sessions and ensures no duplicates are recorded by saving everything directly to a local SQLite database (jobs.db).

Features

• Deduplication: Utilizes better-sqlite3 to store state, ensuring you never scrape the same job twice.
• Robust Extraction: Identifies hidden JSON payloads on YC pages to grab accurate backend job_id values.
• Filtered Exports: Includes an export script (export_radar_candidates.js) that queries the SQLite database for intent-based hiring (e.g., GTM, DevRel, Growth, Content) and outputs it as a JSON payload for secondary research tools.

Setup

1. Clone the repository.

2. Navigate to the scripts/ directory:

   cd scripts
   npm install
   npx playwright install

3. Authenticate (First Time Only): Run the following script and log in to YC via the browser that opens. This creates a state.json file.

   node auth.js

4. Run the Scraper:

   node scraper.js

5. Export Targeted Jobs:

   node export_radar_candidates.js

   This will query the DB and produce radar_candidates.json containing the targeted companies and matching roles.

Note on Sensitive Files

The .gitignore strictly protects your state.json (authentication cookies) and jobs.db (local history). Do not commit these files to a public repository.