Senior Data Engineer with Databricks experience
Role: Sr. Data Engineer - with Databricks experience Location: Bangalore, India – Remote Key Responsibilities 1. Databricks Development Develop ETL/ELT pipelines using Spark (PySpark/Scala/SQL). Work with Delta Lake for ACID-compliant data processing. Build and optimize bronze–silver–gold architecture. Handle large-scale batch and streaming data. 2. Data Integration Ingest data from various sources (API, SQL Server, ADLS, Kafka, Event Hub, Service Bus). Manage Databricks Jobs, Workflows, and Clusters. Optimize pipelines for performance and cost. 3. Data Modeling & Storage Create and optimize data models for analytics. Work with Data Lake, Delta Lake, Hive Metastore. Implement schema evolution, time travel, merge, and CDC. 4. Reporting & BI Build dashboards in Power BI / Tableau / Looker. Write optimized SQL for reporting layers. Create semantic models, DAX measures, and calculated fields. Publish reports and handle incremental refresh. 5. Governance & Quality Implement CI/CD for Databricks using Git. Apply data validation, logging, monitoring, and alerting. Follow security best practices (RBAC, Key Vault, PAT tokens). Excellent analytical and problem-solving abilities Understanding of HR processes, compliance requirements, and industry best practices