Lead Data Engineer

Lead Data Engineer     Job Description My client tracks threat and opportunity across fast moving online ecosystems, providing ethical, high-quality data products for a range of national security customers. They specialise in the development of custom data products for the government sector. Past and present clients include government and defence clients as well as web-based companies such as shopping and social media. They are a small team of subject matter experts, infrastructure and data engineers and following the development of their technical platform in 2020, they’re now seeking to rapidly scale their cutting-edge offer. The Role: The Lead Engineer will lead the technical team, directly managing the backend data engineers and having oversight of frontend developers. They will work closely with the infrastructure, research, and business leads to ensure that data ingest, storage, and delivery is both robust and secure, as well as meeting the needs of the end users. The aims of the technical team are: to maintain, develop, and scale up existing technical products; to scope, pilot, design, plan, and build new technical products from scratch; to actively investigate new and expanding technical capabilities and methods; to provide the business and research teams with a clear and accurate picture of current and possible future technical capabilities. The Lead Engineer will be comfortable running daily stand-up and presenting to/interacting with clients on technical matters. They will work closely with the Client Servicing team to ensure best-in-class products continue to evolve and maintain the precision and quality the government customers expect. This is an opportunity to be part of a small but ambitious team looking to scale rapidly, with excellent career progression for the right candidate. Successful Candidates Will Need: Have experience building technical data products for government or intelligence/OSINT customers. Have excellent knowledge of data pipelines and data management principles. Have an in-depth understanding of the open-source information environment in particular the opportunities and challenges presented by such an environment. Be excited, motivated and have knowledge of analytics and OSINT products in the national security space. Comfortable working for Defence and Government customers in areas such as Disinformation, Terrorism and ExtremismResponsibilities: Setup and maintain secure environments for data ingest, management, and access on AWS. Build, maintain, and perform basic administration on relational databases. Automate monitoring and evaluation of data quality. Work with the Client Servicing team and project manager to deliver data products to senior government clients. Work closely with senior stakeholders to outline project scope and requirements. Be able to rapidly and professionally advise on feasibility of business requirements in terms of data availability, appropriateness, and quality. Work on pre-sales to respond to immediate business opportunities and longer-term development projects. Overall responsibility for the setup of data pipelines for batch and near real-time data collection across platforms such as Telegram, Twitter, or Youtube via API, or web scrapers/third parties when necessary. Scale projects to ensure data is available as required. Scope out new data collection opportunities and manage access. Scope requirements and target architecture for project propositions. Manage and support the rest of the tech team in debugging, blockers etc. Distribute and prioritise tasks amongst the tech team, to reflect goals and deadlines set by the business team. Work in conjunction with the project manager to ensure good oversight on ongoing and upcoming work. Run daily standups. Architect data solutions for new projects and new technology. Liaise with CEO and Business development on pitching new work, including resource and timeframe estimates. Comfortable with being ‘hands-on’ as well as manage the more strategic / technical aspects of code. Have experience in working with data scientists to productionise data science and AI models. Professional attitude at all times and confident speaking directly to senior clients if required.Essential Skills: SQL Relational database design Python API consumption AWS pipelines – Lambda triggers, SQS, SMS, SES Serverless functions (AWS Lambda) Docker/AWS Fargate Virtual machines/EC2 Cloud storage/S3 buckets Master data management/single source of truthDesirable: Security Cleared to SC Terraform CI/CD pipeliningOverview Position – Lead Data Engineer – Permanent Location – Homeworking (Travel to London once or twice a month) Salary – up to £80k Keywords: Lead Data Engineer, building technical data products for government or intelligence/OSINT, Data Management, pre-sales, secure environments, AWS, project propositions, management experience, real time data, data quality. Only apply for this role if you currently hold the specific Government Security Clearance or are eligible for Government Security Clearance. Due to the nature of work undertaken at our client's site, incumbents of these positions are required to meet special nationality rules and therefore these vacancies are only open to sole British Citizens. Applicants who meet this criteria will also be required to undergo security clearance vetting, if not already security cleared to a minimum SC level. Electus Recruitment Solutions provides specialist engineering and technical recruitment solutions to a number of high technology industries. We thank you for your interest in this vacancy. If you don't hear from us within 7 working days please presume your application has been unsuccessful on this occasion. You are of course free to resubmit your CV/details in the future and we shall assess your suitability at that time. This is a Permanent Role