July 2024 - Present
Remote
Beast Industries (@MrBeast)
Software Engineer II
- Senior-scope engineer on a team of about fifteen, reporting to the Director of Engineering and the Director of Data, AI and ML. Most days start with other people's pull requests. New backend hires usually get me as the person who walks them through the codebase, and questions about Python, Java or the ML side tend to end up with me.
- The company's Python codebase started as an empty repository I was handed. I chose the architecture, how services are laid out, the CI and how they deploy. Today it holds twelve microservices that process thousands of videos, posts and accounts a day, the embedding jobs, and the collection services that feed them through rotating proxy pools. Databricks takes the heavier batch work.
- The embedding pipelines are mine. They used to call third-party model APIs one record at a time and topped out near 60,000 records a day. Moving inference onto our own GPUs in batches got that to about 2 million. Transcript analysis, topic classification, sponsor detection and brand-safety scoring all sit on top of those embeddings.
- Took an AI image and thumbnail generation service from an empty repository to production: model integration, the async job queue, storage and the public API. It handles hundreds of jobs a day and has been stress-tested at a thousand. I got per-job latency down from around 90 seconds to 30.
- Built the services that aggregate and estimate metrics over millions of videos and channels, and a retrieval-augmented generation service that lets more than 2,000 users ask questions of the analytics data in plain language.
- Wrote the design documents behind several of the team's infrastructure choices and saw them through to adoption: Qdrant for vector search, Manticore and Quickwit for full-text search, and the schemas and data contracts for ClickHouse, MongoDB and Cloud Spanner.
- Design event-driven systems on RabbitMQ, backed by PostgreSQL, Cloud Spanner or ClickHouse depending on whether the workload is transactional or analytical. Some of the consumers I reworked now handle up to 100x more messages per second than before, and a set of queries that used to run for over ten seconds come back in 300 ms or less.
- Maintain roughly 70 Helm-deployed services on GKE through Argo CD. About twenty of them run on a deployment pattern and architecture I designed. Moving service-to-service traffic onto private VPC networking cut the network bill sharply. The infrastructure-as-code, ingress, certificate and DNS automation and the OpenTelemetry tracing are mine as well.
- Built the observability around the payments integration: status-code tracking for the payment provider, exception capture, and alerts to email, Slack and Discord. I am on the on-call rotation and respond to production incidents.
- Write the Kotlin authentication services (token issue and refresh, session handling) that the web and mobile clients sign in through, and work across a large Java and Kotlin monorepo covering analytics, data collection, ranking, outlier detection, search, payments and alerting. Currently adding NestJS and TypeScript services alongside it.
- Work on the Next.js frontends and the React Native app when a feature needs someone to take it end to end.
