Services We Delivered
Devops
Industry
Automotive
The Client
McQueen Autocorp is a leading US-based used car selling company with a buyer network spread across the United States. McQueen Autocorp offers a nationwide modern car selling service that facilitates sellers with an instant offer for their car(s) using price prediction algorithms and real-time bids from an extensive network of licensed buyers.
Disclaimer: The name McQueen Autocorp is a placeholder as there is an NDA signed between both parties.
Challenges
Our client's application operated using classic manual scheduling. Here are the challenges observed following this practice.
- Manual maintenance of the system required manual checking and restarting, leading to inefficiencies.
- Some jobs were skipped or failed to run on time, affecting overall operations.
- Frequent initialization failures of the application resulted in system unreliability.
- Manual monitoring and running scheduled jobs was tedious and time-consuming.
- Complex data pipelines made bug tracing and resolution lengthy.
- The lack of automated workflows disrupted the efficient orchestration and management of jobs.
- The absence of robust traceability tools made identifying and fixing bugs challenging.
Solution
Our experts discovered that manual orchestration was the root cause of application breakdown and, therefore, decided to implement automatic data orchestration.
- We selected Apache Airflow as the data orchestration tool due to its flexibility and customizability.
- Extracted the main application from the existing server to avoid replication.
- We reduce the likelihood of failures by reconfiguring jobs to run in parallel instead of a sequence.
- Addressed the challenges of managing growing DAGs (directed acyclic graph) by transitioning to Astronomer, which simplified Airflow cluster management.
- Leveraging Astronomer's centralized UI to monitor, we controlled multiple Airflow deployments efficiently.
- Gained insights into critical metrics such as DAG statuses, logs, and cluster health through Astronomer’s interface.
- We integrated Okta for user authentication, enabling secure and streamlined cluster access.
- Implemented a CI/CD pipeline for automated code deployment from development to production, enhancing feature delivery speed and application reliability.
- Hosted the entire setup on AWS cloud, ensuring scalability, easy deployment, and robust maintenance.
Results
The system upgrades delivered transformative results:
- System availability surged from 90% to 99.95%.
- Time-to-market accelerated with efficient application management.
- CI/CD implementation reduced bug frequency and simplified identification.
- Airflow added stability and streamlined operations.
- Automatic status monitoring ensured early detection and resolution of roadblocks.