JobScraper AI is not your typical keyword-based job board. It utilizes a curated dataset of job offers collected by a high-performance Go (Golang) scraping engine and leverages Artificial Intelligence to understand the meaning of job titles.
Using a dedicated Python microservice, the system generates vector embeddings for your search queries in real-time. This allows users to search for "Software Engineer" and find results for "Backend Developer" or "Fullstack Programmer" thanks to semantic similarity search powered by sqlite-vec.
Don't worry about exact keywords. Type what you are looking for, and our Vector Search Engine will find the most contextually similar offers.
- Powered by
sentence-transformers&sqlite-vecfor high-performance local vector similarity.
We prioritize user experience with intuitive controls:
- Recency Sorting: Automatically displays fresh listings first, ensuring you never miss the newest opportunities.
- Navigation: Browse through hundreds of offers using navigation buttons.
- Custom Limits: You decide how many offers to see per page using the dropdown selector.
Each card provides essential insights at a glance, with a detailed view available on click (displaying all provided data):
- 🛠️ Tech Stack & Skills: Instantly see required technologies alongside nice-to-have skills.
- 📍 Location & Mode: Check if the offer is Remote, Hybrid, or On-site.
- 💰 Salary & Contract: View salary ranges (Net/Gross) and contract types (B2B/UoP).
- 📝 Full Description: Click to expand the card and read the complete requirements and responsibilities.
- 🔗 Direct Application: One-click redirection to the original offer page.
The system is designed to perform fast read-heavy operations on a pre-built dataset generated by an external high-concurrency pipeline.
-
Data Collection (External Pipeline):
- A specialized scraping engine written in Go (Golang) crawls multiple job boards to aggregate offers.
- The AI Service calculates vector embeddings for all collected titles.
- The combined data (metadata + vectors) is saved into a portable SQLite database.
-
Runtime Application (This Repo):
- The application mounts this pre-populated
.dbfile, ensuring zero-latency startup and no runtime scraping overhead.
- The application mounts this pre-populated
-
AI Service (Python):
- Acts as a runtime vectorizer using the
all-MiniLM-L6-v2model. - Converts the user's search query into a high-dimensional vector to match it against the database.
- Acts as a runtime vectorizer using the
-
Core Backend (NestJS + Prisma):
- Orchestrates the flow between the UI, the AI Service, and the Database.
- Serves the frontend using Handlebars (HBS) templates.
-
Database (SQLite + sqlite-vec):
- Performs the actual mathematical heavy lifting.
- Uses
sqlite-vecto execute search between the query vector (from Python) and the job vectors (stored in DB).
- Docker & Docker Compose
-
Clone the repository
git clone https://github.com/JobScrapper-io/JobScraper.git cd JobScraper -
Build and Run
docker-compose up --build
- The App will start at:
http://localhost:3000 - The AI Service will start at:
http://localhost:8000
- The App will start at:
- Open your browser at
http://localhost:3000. - Search: Enter a job title (e.g., "Python Developer"). The system sends your text to the AI Service to create a vector, then queries the database.
- Browse: Use the dropdown to change the number of results per page.
- Navigate: Use the page navigation buttons to scroll through results.
- Apply: Select a job card to view full details and navigate to the original source to apply.
- Backend: NestJS (TypeScript)
- ORM: Prisma (Schema management & Client)
- AI Model: all-MiniLM-L6-v2 (via SentenceTransformers)
- Database: SQLite with
sqlite-vecextension - Frontend: Server-Side Rendering with Handlebars (HBS), HTML5, CSS3
- Containerization: Docker & Docker Compose