Data Platform Manager

The Manager, Data Platform is responsible and accountable for the operational management, technical implementation, and continuous optimization of the Databricks- based trading data platform. This role manages the day-to-day operations of data ingestion from 30+ external market data sources and 12+ internal systems, ensuring high availability, performance, and reliability of the platform. The position requires strong hands-on technical expertise in Databricks, data engineering, and platform management, combined with leadership skills to guide and develop a team of data and platform engineers in delivering mission-critical data services to the trading ecosystem.

Duties and Responsibilities

• • Manage day-to-day operations of the Databricks platform ensuring 9%+ uptime and availability
  • Own platform architecture implementation including workspace configuration, cluster management, and job orchestration
  • Configure and optimize Databricks clusters for different workloads (streaming, batch, interactive analytics)
  • IBuild and maintain data ingestion pipelines for 30+ external market data sources (Bloomberg, Reuters, ICE, exchanges)
  • Develop and manage integration workflows for 12+ internal data feeds from trading, risk, and operations systems
  • Optimize Spark job performance through code review, partitioning strategies, and query tuning
  • Implement data quality frameworks with automated validation, reconciliation, and alerting
  • Build and maintain data quality dashboards and metrics
  • Manage cloud infrastructure supporting Databricks (AWS/Azure/GCP networking, storage, compute)
  • Implement infrastructure-as-code using Terraform, CloudFormation, or similar tools
  • Establish comprehensive monitoring and observability across all platform components
  • Configure alerting for platform health, pipeline failures, data quality issues, and performance degradation
  • Lead incident response for platform issues including root cause analysis and remediation
  • Write production-quality PySpark and SQL code for data transformations and pipelines

Educational Requirements

• Bachelor's degree in Computer Science, Engineering, Data Science, or related technical field.
• 10-13 years of experience in data engineering, platform engineering, or related technical roles