We are seeking a highly skilled Senior Data Engineer to join our clients dynamic team in Phoenix, Arizona. The ideal candidate will have extensive experience in big data processing and analysis, with a proven track record of working with Apache Spark. You will play a pivotal role in designing, implementing, and optimizing data pipelines and applications for large-scale data handling in both real-time and batch processing environments.

Key responsibilities:

• Design, develop, and maintain distributed, large-scale data pipelines using Apache Spark.
• Apply best practices to optimize Spark jobs for performance, scalability, and reliability.
• Work with diverse data sources, including HDFS, NoSQL databases, relational databases, and cloud storage.
• Collaborate with data scientists, analysts, and business stakeholders to understand requirements and deliver robust data solutions.
• Develop real-time data streaming applications using Spark Streaming or similar technologies.
• Write clean, maintainable, and reusable code, adhering to best practices in version control, testing, and documentation.
• Identify and resolve issues in Spark jobs and data pipelines to ensure smooth operations.

Required skills and qualifications:

• 8 to 12 years in software development, with a strong focus on big data solutions.
• Minimum 3+ years of hands-on experience with Apache Spark (batch and streaming).
• Proficiency in programming languages such as Scala, Python, or Java.
• Deep understanding of distributed computing principles and big data ecosystems (e.g., Hadoop, HDFS, Hive, Kafka).
• Strong SQL skills and experience with relational databases.
• Familiarity with cloud platforms like AWS, Azure, or GCP.
• Expertise in version control systems (Git, GitHub, Bitbucket).
• Experience with CI/CD pipelines and DevOps practices.
• Ability to work in Phoenix or New York or Fort Lauderdale in a Hybrid environment (Mandatory)

Preferred qualifications:

• Knowledge of machine learning frameworks and integration with Spark (e.g., MLlib).
• Experience with containerization (Docker) and orchestration tools (Kubernetes).
• Certifications in Big Data or Cloud technologies.
• Experience in developing solutions for payment networks or financial transaction processing.