Core Ingestion & Content Engine (AdonisJS / Elasticsearch)

Overview

The core backend that powers the platform, a long-lived (2020-2025) AdonisJS service responsible for ingesting media content, extracting and enriching article metadata, and serving it via search and feeds. It is the central engine the rest of the product cluster orbits.

The Challenge

The platform needs to continuously crawl, normalise, deduplicate and enrich a very large stream of articles and social posts, then make them instantly searchable and rankable. That requires a resilient, queue-driven backend that can scale ingestion independently from serving.

What We Built

A structured AdonisJS application with clear separation across Controllers, Services, Models, Jobs, Commands, ExternalData and Middleware. It integrates Elasticsearch for full-text search, PostgreSQL (via Lucid ORM) for relational data, Redis and RabbitMQ for queues/SSE, and a spacy-nlp module for entity and language processing (with franc-min for language detection). External enrichment includes Open Graph scraping, node-readability article extraction, RSS feed emitters, GetStream activity feeds, SharedCount social metrics, Recombee recommendations and Firebase. Custom SQL (indexes.sql, procMode.sql) tunes the data layer.

Technologies & Approach

AdonisJS gives a batteries-included MVC backend; Elasticsearch handles search scale; RabbitMQ/Redis decouple ingestion from serving. The Python spaCy sidecar handles NLP that JS does poorly. Containerised with Docker and deployed to Kubernetes.

Outcome / Impact

A production engine sustained and evolved over five years, the durable backbone for ingestion, enrichment and search across the whole product family.

Capabilities Demonstrated

• Designing resilient, queue-driven ingestion backends in Node.js/AdonisJS
• Operating Elasticsearch + PostgreSQL together for search and relational workloads
• Integrating NLP enrichment and third-party recommendation/social-metrics APIs
• Long-term maintenance and evolution of a core platform service