We are looking for an experienced Senior Data Engineer to design, build, and optimize scalable data pipelines and architectures. The role involves working closely with data scientists and analysts to ensure data availability and integrity. Candidates should have strong expertise in SQL, Python, big data technologies, and cloud platforms.

Skills

• SSIS
• SQL
• Azure
• AWS
• Databricks
• Power BI
• Technical Skills
• Interpersonal Skills
• CICD
• Python
• ETL

Requirements

• Job Role Senior Data Engineer
• Job Type Full Time
• Workplace Type Onsite
• Industry Software Development

Secondary locations

About

Not provided

Responsibilities

• Senior Data Engineer will be responsible for designing, implementing, and maintaining data solutions on the Microsoft Azure Data platform and SQL Server (SSIS, SSAS, UC4 Atomic)
• Collaborate with various stakeholders, and ensuring the efficient processing, storage, and retrieval of large volumes of data
• Technical Expertise and Responsibilities
• Design, build, and maintain scalable and reliable data pipelines.
• Should be able to design and build solutions in Azure data factory and Databricks to extract, transform and load data into different source and target systems.
• Should be able to design and build solutions in SSIS
• Should be able to analyze and understand the existing data landscape and provide recommendations/innovative ideas for rearchitecting / optimizing / streamlining to bring efficiency and scalability.
• Must be able to collaborate and effectively communicate with onshore counterparts to address technical gaps, requirement challenges, and other complex scenarios.
• Monitor and troubleshoot data systems to ensure high performance and reliability.
• Should be highly analytical and detail-oriented with extensive familiarity with database management principles.
• Optimize data processes for speed and efficiency.
• Ensure the data architecture supports business requirements and data governance policies.
• Define and execute the data engineering strategy in alignment with the company’s goals.
• Integrate data from various sources, ensuring data quality and consistency.
• Stay updated with emerging technologies and industry trends.
• Understand the big picture business process utilizing deep knowledge in banking industry and translate them to data requirements.
• Enabling and running data migrations across different databases and different servers
• Perform thorough testing and validation to support the accuracy of data transformations and data verification used in machine learning models.
• Analyze data and different systems to define data requirements.
• Should be well versed with Data Structures & algorithms.
• Define data mapping working along with business and digital team and data team.
• Data pipeline maintenance/testing/performance validation
• Assemble large, complex data sets that meet functional / non-functional business requirements.
• Analyze and identify gaps on data needs and work with business and IT to bring in alignment on data needs.
• Troubleshoot and resolve technical issues as they arise.
• Optimize data flow and collection for cross-functional teams.
• Work closely with Data counterparts at onshore, product owners, and business stakeholders to understand data needs and strategies.
• Collaborate with IT and DevOps teams to ensure data infrastructure aligns with overall IT architecture.
• Implement best practices for data security and privacy.
• Drive continuous improvement initiatives within the data engineering function
• Optimize data flow and collection for cross-functional teams.
• Understand impact of data conversions as they pertain to servicing operations.
• Manage higher volume and more complex cases with accuracy and efficiency.

Other Requirements

• Design and develop warehouse solutions using Azure Synapse Analytics, ADLS, ADF, Databricks, Power BI, Azure Analysis Services
• Should be proficient in SSIS, SQL and Query optimization.
• Should have worked in onshore offshore model managing challenging scenarios.
• Expertise in working with large amounts of data (structured and unstructured), building data pipelines for ETL workloads and generate insights utilizing Data Science, Analytics.
• Expertise in Azure, AWS cloud services, and DevOps/CI/CD frameworks.
• Ability to work with ambiguity and vague requirements and transform them into deliverables.
• Good combination of technical and interpersonal skills with strong written and verbal communication; detail-oriented with the ability to work independently.
• Drive automation efforts across the data analytics team utilizing Infrastructure as Code (IaC) using Terraform, Configuration Management, and Continuous Integration (CI) / Continuous Delivery (CD) tools such as Jenkins.
• Help build define architecture frameworks, best practices & processes. Collaborate on Data warehouse architecture and technical design discussions.
• Expertise in Azure Data factory and should be familiar with building pipelines for ETL projects.
• Expertise in SQL knowledge and experience working with relational databases.
• Expertise in Python and ETL projects
• Experience in data bricks will be of added advantage.
• Should have expertise in data life cycle, data ingestion, transformation, data loading, validation, and performance tuning.

Skillsets Required

SQL, PL/SQL

SSIS

SSAS

TFS

Azure Data Factory

Azure Databricks

Azure Synapse

ADLS

Lakehouse Architecture

Python

SCD Concepts and Implementation

UC4

Power BI

DevOps CI/CD

Banking Domain

Good to have

Not provided

About the Company

Softnotions Technologies Pvt Ltd stands as a premier software & app development company in India, boasting a stellar 15-year track record of excellence. Specializing in cutting-edge solutions, we cater to clients across the Middle East, Europe, & key regions such as the UK, US, Spain, Germany, Australia, Canada, Singapore, & India. Our dedicated team commits to delivering top-tier services in software development, web & app development, data analytics, AI, digital process automation, agile transformation, experience & design, digital marketing, research, & innovation. Utilizing the latest technologies & industry best practices, Softnotions Technologies ensures unmatched client satisfaction & project excellence.