Staff Engineer, Observability

We are seeking a Staff engineer to design, build, and operate the internal and external Observability stack for the MongoDB platform. Tens of thousands of customers depend on our Observability stack to monitor their database clusters and to generate actionable alerts to safeguard critical workloads. This is an opportunity to join a team that is responsible for all Observability systems that support metrics, metric visualization, logs, traces, and alerts for MongoDB. We are looking for engineers with high standards, and experience in setting direction and technical leadership for large engineering teams in designing and operating complex distributed systems, with strict SLO on security, durability, availability and performance.

As MongoDB Atlas and its supporting infrastructure continue to experience rapid growth, the demand for high-cardinality observability data for internal and external use cases means we need to continually innovate and scale our systems to the next level. For example, MongoDB Observability systems need to handle 10’s of billions of metrics time series, all whilst processing petabytes of logs, traces, and events. Our stack includes VictoriaMetrics, Splunk, Flink, WarpStream/Kafka, Java, Golang Fluentbit. In addition to owning critical components of our observability infrastructure, as a Staff engineer on the team, you’ll also work closely with other SWE, Product and SRE teams to promote and implement best practices in instrumenting and monitoring their services. This is a highly collaborative role, and you will get to own some of the most relied upon internal infrastructure at Mongo.

Our team champions a strong culture of inclusivity, diversity, and collaboration. If you want to be a deeply technical leader on a collaborative team that applies low-level systems expertise to build the foundational infrastructure of a popular database, join us! Let’s build a faster, more reliable, and exceptionally observable database system together.

We are looking to speak to candidates who are based in Dublin for our hybrid working model.

Candidate Profile

Solving complex problems at scale, and to high standards, excites you

Minimum 10 years of experience in designing, programming, debugging, and tuning distributed and/or highly concurrent C/C++/Java/Rust mission critical software systems.

Experience running latency sensitive, high throughput systems

Strong systems fundamentals, including multi-threaded programming, performance profiling, and expert-level programming

Familiarity with database internals or building core components for data processing systems

(Nice to have) Indexing or database performance tuning experience

Familiarity with observability ecosystem and best practices

Excellent verbal and written technical communication skills and a strong desire to collaborate with colleagues and mentor engineers

Excellent time and project management skills including the ability to make realistic assessments of project cost and complexity

Has a good understanding of information security management

Responsibilities

Help build and architect systems for the mission-critical observability platform, leading the architecture and implementation of components that drive performance, scalability, cost-efficiency, and resiliency

Design and implement observability improvements that enable MongoDB engineers and customers to quickly and accurately diagnose the root cause of production issues.

Handle production customer escalations from Technical Support team and coach teammates to do the same

Write production-ready database code, improve the existing code, and mentor their team to write higher quality code

Own areas of code that the Observability Team maintains, ensuring it achieves a high standard for quality (including security, durability, availability, and performance) and maintainability

Diagnose test failures, identify bugs in existing code, fix them, and prevent bugs from being introduced in new code

Invest