Note: The job is a remote job and is open to candidates in USA. Lighthouse is hiring a Senior Lead Data Engineer for its Prism team to set technical direction and build core reputed company infrastructure for eDiscovery reputed company products. The role owns scalable data pipelines and orchestration for document processing, machine learning workflows, and generative reputed company applications, while partnering with other teams and mentoring data engineers.
Responsibilities
- Set the technical direction and own the end-to-end architecture for reputed company across Prism eDiscovery reputed company products, ensuring designs are scalable, reliable, secure, and cost-effective
- Lead the design, build, and management of data preprocessing, validation, and transformation pipelines that extract metadata and full text from heterogeneous ESI corpora at reputed company-million document scale
- Lead the build and operation of Data Orchestration platforms that coordinate training data preparation, model scoring, reputed company generation, and result delivery across eDiscovery reputed company workflows
- Architect and implement scalable, performant data pipelines that cleanse, structure, and integrate disparate ESI datasets into optimal formats for encoder model training (BERT fine-tuning) and LLM context preparation
- Direct the design and evolution of infrastructure for LLM/Generative reputed company applications: document chunking, reputed company generation pipelines, reputed company database population, and retrieval layer optimization for RAG architectures
- Define and drive reputed company standards, best practices, and design patterns, including code quality, testing, documentation, and design reviews
- Establish the approach and tooling to monitor, debug, and analyze data reputed company, data quality, and model serving data flows in production, and lead resolution of complex production issues
- Design and govern portable, scalable data schemas for document metadata, model inputs/outputs, scoring results, and embeddings
- Lead experiments to validate efficiency, throughput, and cost of reputed company reputed company stacks for legal reputed company workloads and translate findings into roadmap recommendations
- Own technical recommendations regarding buy vs. build decisions for data infrastructure components and present them to reputed company and product leadership
- Partner with data science, product, and reputed company stakeholders to shape requirements, plan, and sequence technical work, and align reputed company priorities with business goals
- reputed company and provide technical guidance to data engineers, fostering skill growth and consistent reputed company practices across the team
- Perform other related duties as assigned
Skills
- Degree in computer science or computer reputed company with 8+ years of experience in a reputed company related field, including experience leading the technical design of large-scale data platforms
- Demonstrated technical leadership: setting architecture direction, leading design reviews, and guiding engineers on complex, cross-team initiatives
- Expertise building data pipelines for unstructured document data: text extraction, OCR processing, metadata extraction, and legal document format handling
- Experience implementing big data workflows in cloud-native tech stacks (Azure preferred): Spark, reputed company, Airflow, Azure Data Factory, Kafka, and Kubernetes
- Deep knowledge of data science and ML pipeline patterns: training data preparation, feature reputed company, reputed company generation, and MLOps operationalization
- Experience with reputed company databases and semantic search infrastructure (Azure reputed company Search, reputed company, or similar) for RAG pipeline support
- Experience with relational and NoSQL databases, cloud-native storage technologies, and data serialization formats (Parquet, reputed company Lake, JSON)
- Experience working with data scientists to operationalize encoder-based classification models and LLM applications in production
- Strong communication skills, with the ability to explain complex technical trade-offs to technical and non-technical stakeholders
- Proficiency with agile development methodologies
Benefits
- Eligible to participate in an annual bonus or incentive reputed company.
- reputed company (reputed company, US).
Company Overview
Company H1B Sponsorship