Going From Unreliable System To A Highly Available System - with Airflow
CASE STUDY
McQueen Autocorp is a leading US-based used car selling company with a buyer network spread across the United States. McQueen Autocorp offers a nationwide modern car selling service that facilitates sellers with an instant offer for their car(s) using price prediction algorithms and real-time bids from an extensive network of licensed buyers.
Disclaimer: The name McQueen Autocorp is a placeholder as there is an NDA signed between both parties.
Challenge
The custom application used by our client was operating via classic manual scheduling.
Since the team had to maintain the system manually, it resulted in the pausing of jobs. Then, these jobs had to be manually checked and run. Resultantly, some jobs often got skipped and wouldn't run on time. It led to frequent failure in the initialization of the application. The system was not reliable.
Scheduled jobs needed manual intervention for monitoring and running, which was tedious. Since data pipelines were complex, it would take hours to trace and rectify the bugs and get the application running again.
Their top priority was finding a solution to eliminate manual intervention, provide traceability of the bugs, and orchestrate workflows and jobs to solve the issue.
Solution
Maruti Techlabs was already working with McQueen Autocorp on a different project when this issue surfaced. Looking at our technical expertise, the founders decided to onboard us to solve the data orchestration challenge.
Our engineers understood the challenge and quickly got to work. Since the classic method of manual orchestration was the root cause of frequent application breakdown, we decided to automate data orchestration.
After analyzing various data orchestration tools, we decided to go ahead with Apache Airflow. We chose Airflow because of its Python base, making it easily customizable to fit our needs.
To avoid replicating the main application, we first took the application out of the existing server. The jobs were mainly sequential earlier, which contributed to the probability of failure. We coded the jobs to be parallel to each other inside the tool.
As the number of DAGs (directed acyclic graph) in the Airflow clusters grew, it became more challenging to manage and schedule the DAGs. We decided to move to Astronomer to simplify and scale up the management of Airflow clusters. Astronomer made it very easy to control multiple Airflow deployments from the central UI. It provides a centralized view of different metrics like DAGs, status, logs, etc., of all Airflow clusters.
After setting up the Astronomer platform, we created users via Okta login in the cluster. We created a CI/CD pipeline for deploying code from development into production. This automated process allowed us to develop new features quickly while maintaining the security and reliability of the application. Finally, to ensure easy deployment and maintenance of the entire setup, we hosted it on the AWS cloud application.
Move on from siloed traditional structures. Switch to an environment of cross-functional teams & dynamic infrastructure - with DevOps.
Technology Stack
Result
✅ With Airflow implementation, the system has become stable as the jobs are stopped automatically using scripts.
✅ In addition, it is easy to deploy DAGs and implement the DevOps philosophy, which in turn enables faster time-to-market and highly efficient management of the application.
✅ As we have moved from a single instance of the application to clusters, the system's availability has increased from 90% to 99.95%.
✅ With continuous integration, continuous deployment (CI/CD), massively parallel processing in deployments, and automatic software updates, bugs have become less frequent and easier to identify.
✅ Automatic status monitoring from within the tool has made it easier to spot and fix roadblocks in the nascent stage.
Our Development Process
We follow Agile, Lean, & DevOps best practices to create a superior prototype that brings your users’ ideas to fruition through collaboration & rapid execution. Our top priority is quick reaction time & accessibility.
We really want to be your extended team, so apart from the regular meetings, you can be sure that each of our team members is one phone call, email, or message away.
Continuous Planning
We work with our clients to set quantifiable goals at the beginning of each project phase & follow up as development progresses. We engage all stakeholders to bring a meaningful impact to the discussion & define targets, thus minimizing redundancies.
Continuous Integration
During this phase, the operations personnel work closely with software developers to ensure that the product meets the users' needs. Our developers integrate their code changes into a shared repository, facilitating timely integration & automated testing.
Continuous Deployment
Continuous deployment is necessary to take full advantage of a well-designed agile system. Every significant, validated change is automatically released to the users. Quick releases enhance the feedback loop such that developers can isolate issues with higher accuracy.
Continuous Testing
Timely execution of testing is key to the health of any software. We enforce a frequent-testing policy by way of automation. Newly committed code is passed through various testing phases to ensure quality maintenance.
Continuous Monitoring and Feedback
After deploying the system, we established monitoring parameters to track functionality in real-time environments and assess system reliability. Security checks are automated so that we can ensure timely and correct reporting.
A big part of our DevOps consulting and implementation is the synergic effect of the continuous involvement of various stakeholders. With DevOps implementation, experience a workload shift with reliable, high-quality software applications and services that support automation.
Why choose Maruti Techlabs?
Trusted By:
Connect with us
© 2022 Maruti TechLabs Pvt Ltd
Maruti Techlabs is an agile-powered digital product development company and your guide on the digital transformation journey. We are a team of passionate, purpose-led individuals that obsess over creating innovative solutions to address our clients’ challenges and deliver unparalleled value.
How We Work?
Don't take our word for it, take theirs!
Let Our Clients Do the Talking
More social proof incase you're still on the fence
13+
years experience
220+
Members
50+
Enterprise Clients
4.8/5
NPS on Clutch