News/Web Data Ingestion Microservice
A media-monitoring / data-orchestration platform
Overview
A TypeScript microservice that ingests news and web content from the Webz.io data API and feeds it into a media-monitoring platform via a RabbitMQ message queue. It handles query parsing, API access, response mapping and queue publishing as a self-contained, containerised service.
The Challenge
A media-monitoring platform needs a steady, normalised feed of news/web articles from external data providers. That means a reliable integration that translates platform queries into provider calls, maps the responses into the platform’s own shape, and hands them off asynchronously without coupling the producer to the consumers.
What We Built
An Express + TypeScript service structured with clear separation: services/ (webz-api-service for the Webz.io API, webz-article-service for orchestration, rabbitmq-service for AMQP publishing), dto/ (request/response and article DTOs), util/ (query parsing, response mapping, env loading), config/ (Awilix dependency-injection container and logging), and decorator-style annotations/ for logging. It pulls credentials via Google Secret Manager (google-secrets-as-env), is built with tsc, tested with Jest (unit + integration), and ships with a Dockerfile and Kubernetes manifests for deployment.
Technologies & Approach
TypeScript/Express for the service; Awilix for dependency injection and testable wiring; amqplib/RabbitMQ for decoupled, queue-based delivery; axios for the Webz.io API; Google Secret Manager for secret handling; Docker + Kubernetes for deployment. The DI-and-DTO design keeps the provider integration cleanly isolated and easy to test.
Outcome / Impact
A production-shaped ingestion microservice that integrates an external news/web data source into a larger media-monitoring system over a message queue, with full unit/integration test coverage and a Kubernetes deployment path.
Capabilities Demonstrated
- Integrating third-party news/web data APIs into a platform
- Queue-driven (RabbitMQ/AMQP) ingestion architecture
- Clean DI-based TypeScript microservice design with DTO mapping
- Kubernetes + Docker deployment and secret management