Senior Linux Systems Administrator

Location: Remote — United States Department: Information Technology — Infrastructure Engineering Reports to: Infrastructure Engineering Manager Employment Type: Full-time, individual contributor Experience: 8+ years of relevant hands-on experience Base Salary: $125,000–$165,000 annually, plus equity and benefits Eligibility: U.S. citizenship required Senior Linux Systems Administrator RapidFort, Infrastructure ======================================================================= ABOUT RAPIDFORT --------------- RapidFort helps enterprises and government organizations reduce software supply-chain risk through hardened container images, runtime-based attack surface reduction, and automated vulnerability remediation. We own and operate the infrastructure that supports our engineering teams and core systems. ABOUT THE ROLE -------------- RapidFort is looking for a senior Linux systems administrator to own and run its infrastructure: bare-metal Linux servers in data centers, the virtual machines and containers that run on them, the networks that connect them to each other and to our staff, self-hosted development services such as GitLab and CI, and our resources in AWS. You will keep these systems reliable, secure and documented, plan and carry out changes to them, and lead the response when something breaks. This is a hands-on senior individual-contributor role with direct responsibility for technical execution. We need someone whose knowledge runs from disks and kernels up to routing tables, firewalls and DNS. RESPONSIBILITIES ---------------- Linux systems administration - Build, configure, administer and harden Linux systems: installation, system services, packages, users, permissions, authentication and access control. - Operate a fleet of remotely managed physical Linux servers and the virtual machines and containers on them: storage, capacity planning and hardware replacement. - Own the operating-system lifecycle: patch testing, staged deployment, operating-system and kernel upgrades, tracking exceptions, and moving systems off releases that reach end of life. - Keep configurations consistent, and find and correct configuration drift. Networks, cloud and development services - Design, run and troubleshoot the network: routing, traffic shaping, firewalls, DHCP and DNS, and the overlay network that connects the data centers, the cloud and remote staff. - Manage our resources in AWS, mainly EC2 and S3. - Run the self-hosted development services, including GitLab, CI runners, build machines and package mirrors. Troubleshooting and incident ownership - Independently diagnose production problems involving CPU, memory, disk I/O, filesystems, systemd, networking and service dependencies, such as failed services, boot problems, resource exhaustion, package conflicts and intermittent performance issues. - Use logs, metrics and command-line tools to find root causes and to confirm that a fix worked. - Monitor the systems, and improve monitoring and alerting so that problems are caught early without needless noise. - Lead incident response, contribute to incident reviews, and complete the corrective actions that keep a problem from recurring. - Work with developers, IT, data-center staff and hardware vendors when a problem crosses system boundaries. - Escalate when needed, with clear findings, the supporting evidence, and a record of the troubleshooting already done. Automation and configuration management - Write, debug and maintain Bash scripts for provisioning, diagnostics, remediation, reporting and routine operations, and Python automation where it makes operational work more reliable or simpler. - Write and maintain Ansible roles and playbooks to the team's standards. - Replace repetitive manual tasks with reliable, maintainable automation. - Keep configuration and automation in git, test changes, and make changes through merge requests, review and automated deployment rather than by editing live systems. Security and operational reliability - Own access control and security: SSH keys and hardware security keys, network access policies, least privilege, security baselines, timely patching, vulnerability scanning, audits of exposed services, and software supply-chain risk. - Run the backups and prove that they restore. - Plan changes and maintenance with a written procedure that gives the steps, how to check the result, and a way back. - Write and maintain documentation, including runbooks, configuration records and troubleshooting notes, and review other engineers' changes. - Take part in a shared on-call rotation and in scheduled maintenance windows. REQUIRED EXPERIENCE AND SKILLS ------------------------------ Typically eight or more years administering Linux servers in production with root responsibility, including physical hardware and not only cloud instances, with recent hands-on responsibility for operations and troubleshooting. Solid working knowledge of most of the following. Linux systems - Linux processes, services, users and permissions, package management, filesystems and logging, in depth. - Strong experience with Debian and other Linux distributions. - Boot and kernel: UEFI, GRUB, initramfs, kernel configuration and building, kernel modules (including DKMS) and sysctl. - Resource control and diagnosis: cgroups v2, memory management and the out-of-memory killer, pressure-stall information (PSI), and tools such as strace, perf, iostat, ss and /proc. - Both systemd and sysvinit. Networking - TCP/IP fundamentals: IPv4 and IPv6, subnetting, bridges, VLANs, bonding and NAT, and troubleshooting connectivity. - Lin