[Remote] Senior Data Engineer- Databricks SME
Note: The job is a remote job and is open to candidates in USA. Cloud7Works, Inc. is a small business federal IT contractor specializing in cloud modernization, enterprise architecture, cybersecurity, and advanced analytics for federal civilian agencies. They are seeking two Senior Data Engineers with deep Databricks expertise to support a mission-critical, FedRAMP High-authorized data platform on AWS, focusing on workspace architecture, data governance, and secure AWS integration.
Responsibilities
- Administer and optimize Databricks workspaces, clusters, and SQL Warehouses via Terraform and GitHub Actions CI/CD pipelines
- Design and maintain medallion pipeline architecture using Delta Live Tables and Auto Loader for large-scale data filtering and transformation use cases
- Govern data access end-to-end through Unity Catalog — enforce RBAC/ABAC, PII tagging, audit logging, retention policies, and data lineage
- Enable self-service analytics through Databricks Genie, SQL semantic layers, and materialized views for business consumers
- Integrate Databricks with mainframe sources (JCL/COBOL/DB2), streaming platforms (Kafka/Kinesis), CDC tools (Debezium/DMS), and SaaS systems
- Drive performance optimization using Photon, Z-ordering, liquid clustering, and rightsizing with continuous cost monitoring
- Maintain platform observability — data quality expectations, metrics, alerting, and resilient error handling
- Support 24x7 platform operations including proactive monitoring, P1 incident response (initial response within 4 hours), and continuous health checks
- Collaborate with cross-functional technical teams and government technical leads during normal business hours
- Maintain requirements traceability, architecture review artifacts, and records-compliant documentation repositories
Skills
- 8+ years in cloud technology
- 3–5 years hands-on Databricks (non-negotiable)
- Unity Catalog — RBAC/ABAC, dynamic row-level security, column masking, tagging, lineage, Delta Sharing
- Delta Live Tables and Auto Loader — medallion architecture design and operations (bronze/silver/gold)
- Databricks SQL Warehouses and Genie — semantic layer and self-service analytics enablement
- Performance optimization — Photon engine, Z-ordering, liquid clustering
- Workspace, cluster, and SQL Warehouse provisioning and administration via Infrastructure as Code
- S3, IAM, VPC, PrivateLink, CloudWatch — secure connectivity and observability
- Streaming integrations — Kafka, Kinesis, EventBridge, SQS, SNS
- CDC and batch ingestion — Debezium, AWS DMS, AWS DataSync
- Amazon EKS — Kubernetes workload integration with Databricks
- Terraform and GitHub Actions — automated Databricks and AWS provisioning
- Databricks Asset Bundles — environment promotion (dev/test/prod)
- SAST/DAST security scanning, policy-as-code, approval gates
- Blue/green and canary deployment strategies
- Active Databricks certifications — Databricks Certified Data Engineer Professional
- Active AWS certifications — Solutions Architect or Data Analytics Specialty
- Experience in federal, regulated, or large-scale tax/financial data environments
- FedRAMP High or FISMA-governed environment experience
- Clean Rooms and cross-account Delta Sharing configuration
- Informatica Enterprise Data Catalog (EDC) / Axon metadata and lineage integration
- Power BI and Tableau dashboard development on top of Databricks SQL
- Bachelor's degree in Computer Science, Information Technology, or related field
Company Overview