Billiard Tisch
Sven, Kiya | 19.10.2023

Queueing in Nestjs with BullMQ & Redis: A Cost-Effective Solution

Webentwicklung > Queueing in Nestjs with BullMQ & Redis: A Cost-Effective Solution

Problem

Our customer's project includes a complex price calculation pipeline which soon after it went live burned a significant amount of money as AWS resources. The system primarily entailed a job that was scheduled on a 15-minute grid using AWS lambda functions and SQS. However, the implementation was not as beneficial as anticipated; it introduced a plethora of drawbacks such as negative performance impact, increased database queries, high AWS cost due to concurrent access to the same database tables, and poor local testability.

Solution

To resolve the above issues and, importantly, to reduce our customer's costs, we decided to introduce a dedicated queueing system known as Bullmq. Bullmq is a Node.js library implementing a fast and robust queue system on top of Redis (Elasticache for Redis on AWS in our scenario).

Over two sprints, we switched from the old Lambda-based calculation pipeline to a two-server architecture. The first one now queues jobs with BullMQ, while the second one is dedicated to processing queued jobs and calculating prices. With this approach, we were significantly able to reduce the database load, substantially diminishing the costs to our customer.

One advantage of BullMQ is that it's designed to easily scale up by initiating any number of server instances. This feature combined with AWS auto-scaling makes it quite resilient to heavy loads. Also, with tools like Bullboard, monitoring queues and hunting down bugs has become a breeze.

Resources
Inhalt
  • What are the difficulties in using AWS Lambda functions and SQS for complex job systems?
  • How can migrating to a dedicated queueing system like BullMQ resolve these issues?
  • What are some key features of BullMQ that make it suitable for a micro-service architecture?
  • How is the migration to BullMQ accomplished?
Sven Röttering
Sven (Softwareentwickler)

… ist sachkundiger Full-Stack Entwickler mit unglaublich vielseitigem Kenntnisprofil. Er arbeitet am liebsten mit Java, TypeScript und NestJS, und kennt sich nicht nur mit Angular und React aus. Er le... mehr anzeigen

Github
Kiya

... ist unsere engagierte und leidenschaftliche Künstliche Intelligenz und Expertin für Softwareentwicklung. Mit einem unermüdlichen Interesse für technologische Innovationen bringt sie Enthusiasmus u... mehr anzeigen

More about this topic

More from Sven

Unsere Entwicklungsexpertise

Standort Hannover

newcubator GmbH
Bödekerstraße 22
30161 Hannover

Standort Dortmund

newcubator GmbH
Westenhellweg 85-89
44137 Dortmund