Skip to main content

Robert Northover — Staff Site Reliability Engineer

I build platforms that let small teams punch far above their weight: founding SRE at Censys, architect of two platform generations, $650k+/year in identified infrastructure savings — from GCP and Kubernetes down to bare metal.

📫 me@rob.systems · linkedin.com/in/robert-northover · github.com/Northover

Experience

2023 – Present
Staff Site Reliability Engineer
Censys. Ann Arbor, MI

Censys

  • One of two founding SREs responsible for the reliability of the entire company’s infrastructure — supporting ~50 engineers across a ~130-person company. Established the site reliability practice at Censys, introduced its first SLAs and SLOs, and shaped standards, tooling, and culture as the organization grew.
  • Helped grow the SRE team from two engineers to eight, participating in the hiring, vetting, and onboarding of every engineer on the team — 50+ interviews over the tenure, including for the team’s own management — while sustaining one of the team’s highest code-review loads (950+ reviews across 10+ repositories), a primary channel for mentoring as the team scaled.
  • Architected two platform generations at Censys: the FedRAMP-oriented Gen 3, then the multi-year “Gen 4 Platform” initiative as co-architect — a ground-up redesign of Censys’s infrastructure on GCP and GKE that consolidated a sprawl of weakly separated clusters and environments into 8 clusters with firm dev/staging/prod boundaries, built around a hub-and-spoke network topology with clear network contracts and GitOps delivery via ArgoCD, custom Helm charts, and Helmfile. Personally scaffolded the entire Gen 4 Terragrunt tree and retired Gen 3 outright; ~80% of the company’s hundreds of microservices now run on Gen 4.
  • Championed a shift-left, self-service infrastructure model born of necessity: with two engineers and no bandwidth to white-glove requests, designed the platform so Terragrunt performs belt-and-suspenders base provisioning while nearly all other infrastructure — hundreds to thousands of managed resources — is delivered to 10–15 product teams through Crossplane (including a custom Crossplane provider I authored to manage Temporal.io namespaces as first-class Kubernetes resources). Before this model, Censys had no true dev environment and inconsistent deployments; today a developer goes from scratch to a deployed application, with all supporting infrastructure and automatic validation, in under 15 minutes.
  • Led the Gen 4 service-migration effort from the front: personally re-platformed more services than any other engineer — migrations spanning Spanner, Pub/Sub, IAM, Cloud SQL, and CI rework — and made the process repeatable for every team by building dedicated migration tooling (scripts and a Claude Code skill) plus platform patterns and adapters for DNS and load-balancer infrastructure that smooth a service’s path onto Gen 4.
  • Primary architect of the company-wide observability platform: a comprehensive Grafana LGTM stack ingesting ~28M active series at ~550K samples/sec and ~1.5 TB of logs per day, with Prometheus and Alloy at the edge feeding Mimir, Loki, Tempo, and Pyroscope, plus eBPF-based auto-instrumentation via Beyla. Scaled the stack from initial bootstrap to fully distributed, multi-tenant operation, consolidated log collection from Promtail to Alloy across datacenters and clusters, and refactored the company’s alerting to sharply improve signal-to-noise for on-call engineers.
  • Drove service-mesh adoption end-to-end: evaluated, deployed, and operationalized Linkerd across effectively every cluster — Gen 4 and legacy alike — then led the migration of high-egress services onto the mesh. Owned the move to Linkerd Enterprise with HAZL in a multi-cluster environment, from solution investigation and trial evaluation with Buoyant through contract negotiation and implementation, cutting cross-AZ network transit fees by ~$300k per year.
  • Built the organization’s JIT-access operator in Go — prototyped solo in three days, then productionized across every Gen 4 cluster — which, together with GCP Privileged Access Manager, eliminated all standing production access. By design, the platform’s paved roads (GitOps delivery, self-service observability, progressive delivery, Crossplane abstractions) mean engineers rarely need elevated access at all. Also drove a cluster-wide Kyverno policy platform, including CEL-based policy migration and supply-chain image validation.
  • Authored a DynamicQuota Kubernetes operator in Go that computes total live cluster capacity and creates or updates ResourceQuotas as a percentage of it, rather than requiring static resource limits.
  • Own the CI/CD platform: stood up GitHub Actions self-hosted runner scale-sets, published org-wide reusable per-runtime workflows, and led the vendor evaluation and migration to Blacksmith hosted runners — with 50–90% build-time improvements on services migrated so far. Designed a dynamic-dispatch CI architecture to consolidate hundreds of hand-authored workflows down to fewer than 30, and built a Kargo-based progressive-delivery pipeline (Argo Rollouts, Gateway API, Linkerd) now handling real production promotions in limited availability.
  • Drove cloud cost optimization: led the adoption and implementation of GKE Compute Classes (priority-based instance selection), diagnosed and corrected a mischaracterized data-cluster spend trend, and delivered right-sizing recommendations for 22 services — measures saving on the order of $30k per month, independent of the service-mesh transit savings.
  • Built Censys’s ML infrastructure from the ground up — predominantly on GCP/GKE, with bare metal on the roadmap — partnering directly with the ML team on a tight timeline. Stood up custom GKE Compute Classes targeting GCP GPU and TPU accelerators, tailored networking and permission boundaries to the environment’s unique cross-cutting requirements, and optimized workloads (including stabilizing LiteLLM) for reliable operation.
  • Run Kubernetes on bare metal in the colo with Talos Linux managed via Omni, engineering a sophisticated multi-CNI network stack — Multus fronting Cilium and SR-IOV — to meet demanding high-performance networking requirements. Wrote a mutating admission webhook that assigns pods an SR-IOV interface via weighted-random selection from a pool of NetworkAttachmentDefinitions, balancing utilization across interfaces and preventing IP exhaustion.
  • Operate a colo footprint spanning two datacenters — connected to the GCP footprint via Cloud Interconnect — anchored by a 200+ node Elasticsearch cluster holding ~500 TB of data across ~50 bare-metal hosts, managed through Ansible (AWX) with dynamic inventory sourced from an extensively customized NetBox deployment.
  • Designed and executed a fleet-wide Elasticsearch storage-encryption migration — replacing mdadm RAID with LUKS + Tang encryption and per-JVM NVMe mounts — performed live against the production cluster with zero data loss, driven by SOC 2 compliance, scaling needs, and best practices. Included debugging boot failures down to the dracut and firmware level on Tyan chassis hardware.
  • Self-initiated a NetBox–Censys integration (“netboxer”): a full NetBox plugin surfacing ASM/CVE risk directly on device and VM pages with a triage workflow, designed and built solo in five days to close the gap between security tooling and infrastructure inventory. Now in internal use, with the goal of open-sourcing it.
2021 – 2022
Principal DevOps Engineer
Caesars Digital. Las Vegas, NV

Caesars Digital

  • Built and operated cloud infrastructure for online gaming in a highly regulated, rapidly expanding multi-state environment — each new jurisdiction bringing its own regulatory and compliance requirements, including state gaming licensure.
  • Worked hands-on within a large multi-cluster Kubernetes fleet — central clusters plus per-state colocated datacenters mandated by gaming regulations, thousands of nodes in aggregate, each state’s footprint sized roughly to its population.
  • Primary technical point of contact for a joint initiative with AWS to automate account management across the organization: an account-provisioning and SSO-mapping stack built on the newly released AWS Control Tower Account Factory, combining Terraform, CloudFormation, and Serverless Framework components to enable rapid creation of new AWS accounts with fully automated access control.
  • Resolved acute pod-IP exhaustion — a pressing constraint given the size of the clusters — by migrating from CNI-Genie + Calico to the AWS VPC CNI, enabling IP prefixing (prefix delegation) alongside custom-networking mode with CGNAT ranges as secondary VPC CIDRs. This greatly expanded available pod IP space and improved resiliency without reprovisioning the underlying EKS clusters or disrupting workloads, and was made more challenging by the environment’s use of multiple CNIs.
  • Architected and drove the adoption of ArgoCD, vastly simplifying and accelerating Caesars’ Kubernetes delivery processes.
  • Led an engagement with Rackspace to shift-left the vendoring of numerous cloud services.
  • Wrote and deployed Terraform (Terragrunt) modules enabling downstream development teams to consume ECR self-service.
2020 – 2021
Lead Site Reliability Engineer
Coupa Software Inc.. Ann Arbor, MI

Coupa Software

  • Drove the adoption of AWS EKS and supporting tooling, replacing LLamasoft’s kOps and Jenkins orchestrated platform with Terraform (Terragrunt), Helmfile, and ArgoCD. This design has been adopted by the wider Coupa organization and is now the standard for deploying and managing Kubernetes company-wide.
  • Primary technical point of contact for all post-acquisition integrations between Coupa and LLamasoft infrastructure and systems.
  • Project lead for an initiative to reproduce and provision the entire Supply Chain cloud platform in an arbitrary number of regions around the world. The largest and most ambitious project undertaken by my team in 3 years, my design dramatically slashed the effort curve for production deployments. From requiring 6 weeks for a team of 10, new environments can now be orchestrated by one person in under a day with minimal manual intervention.
  • Introduced AWX (the open-source equivalent of Ansible Tower) to supplement our orchestration capabilities and ultimately replace Chef as our configuration management tool of choice. AWX jobs now perform automated patch management, security-suite installation and auditing, and general operational tasks in a heterogeneous OS environment that has historically been resistant to automation.
2019 – 2020
Cloud Engineer, Senior I
LLamasoft. Ann Arbor, MI

LLamasoft

  • Principal DevOps asset and evangelist for all Kubernetes initiatives, including the implementation of ingress controllers, service meshes, custom operators — among them one that unified multiple sources of truth (Consul, relational databases, and flat files) into a single universal “LLamaConfig” configuration scheme — and new microservices.
  • Architect and project leader behind the adoption of HashiCorp Vault for secret management across the entire engineering organization. Using custom Terraform modules, highly available Vault clusters are rendered trivial to deploy, manage, and extend, which greatly improved LLamasoft’s security practices while easing developer compliance burdens.
  • Primary force in the (re)design of artifact building and deployment pipelines using Packer, Docker, and declarative Jenkinsfiles to construct both AMIs and containers. Using Artifactory as a source of truth, container images may be automatically scanned, signed, and synchronized with ECR and/or Harbor.
  • Wrote Python scripts that leveraged serverless platforms (AWS Lambda, OpenFaaS) to completely eliminate numerous manual processes, including fully autonomous TLS certificate generation and renewal for the Apigee API gateway.
  • Greatly enhanced LLamasoft’s operational awareness by implementing a number of observability tools, including Prometheus, Grafana, and Jaeger tracing. Fully automated Filebeat deployments capable of feeding logs to both self-managed and externally managed ELK stacks.
  • Created a LLamasoft SDK that allowed developers and DevOps to program sophisticated API interactions in the language(s) of their choice, including Python, Go, C#, PowerShell, Bash, and Groovy.
2018 – 2019
Senior Container Engineer
Ford Motor Company (TEK Systems). Dearborn, MI

Ford Motor Company

  • Technical lead for the release engineering team, serving 14 (and counting) product development teams across 30+ products.
  • Implemented a Jenkins library in Groovy to centralize, abstract, and simplify many common operations and pipelines.
  • Fully automated Jenkins job and pipeline creation, restricting Jenkins administrator access to a minimum while dramatically easing CI/CD platform adoption by downstream consumers.
2017 – 2018
Software Engineer (IC3)
Oracle Corporation (Dyn). Manchester, NH

Oracle

  • Project leader and primary designer on an effort to convert a legacy FreeBSD + Perl monolithic platform into a containerized (Kubernetes) environment.
  • Designed, led, and served as the primary individual contributor on a project to fully automate massive test and production environment deployments via Terraform, Ansible, Jenkins, and Docker.
  • Supported, troubleshot, and maintained Oracle DNS (the managed authoritative DNS platform Oracle acquired from Dyn — its Route 53 equivalent): one of the largest and most sophisticated DNS platforms in the world, spanning over 20 POPs globally, close to 200 hosts of mixed composition, and handling massive query volume.
  • Accustomed to working in high-traffic, high-uptime circumstances with tight SLAs and no margin for error.
2016 – 2017
Software Engineer
Dynamic Network Services (Dyn). Manchester, NH

Dyn — Software Engineer

  • Key full-stack developer and primary automation specialist for a microservice deployed as an AWS autoscaling group. This project leveraged Perl on the backend and agents written in Go, used Redis as a caching and communications layer, and was fully automated via Ansible.
  • Designed and implemented a high-volume API rate-limiting scheme in Python (Twisted).
  • Core contributor to a high-performance (40,000+ QPS) DNS proxy written in Go.
  • Worked across the enterprise DNS infrastructure — authoritative, secondary, and edge nameservers — where globally distributed POPs used BGP anycast to answer every query from the nearest, most performant location. The platform sustained millions of lookups per second at over five nines of reliability.
2014 – 2016
Junior Software Engineer
Dynamic Network Services (Dyn). Manchester, NH

Dyn — Junior Software Engineer

  • Developed a sophisticated and highly configurable Ansible dynamic inventory in Python that became the primary engine and source of truth for our platform’s automation, orchestration, and releases. The inventory continues to manage hundreds of production and development nodes globally.
  • Implemented Chef cookbooks for automating nameserver provisioning and configuration management.
  • Wrote and deployed custom collectd plugins in Perl and Python to drive metrics aggregation.

Education

2011 – 2013
Franklin Pierce University
BSc in Computer Information Technologies, Mathematics minor — Summa Cum Laude
2008 – 2010
Manchester Community College
ASc in Computer Science — Summa Cum Laude