On debugging

A mental model

1. There’s a knowledge graph of the software engineering, tooling, and the systems in your head. As you gain more knowledge, you expand this graph.
1. This includes a collective knowledge graph of the documentation, wikis, and messages of your organization. As the organization produces more docs, wikis, and chats, this graph expands.

2. There’s a problem graph of the stack, code, components, API calls, services, etc.

Traversing the graph

1. DFS on the problem graph: Put a debugger through whole stacktrace and trace through the code. Navigate up and down code trees.

2. BFS on the knowledge graph: Apply a bunch of different ideas and tools that you know (React Component Tools, Debugger, Datadog, etc), ask your neighbors for ideas.

3. BFS on the problem graph: Look at all the recent commit messages and edited files and see if anything stands out.

4. Binary search on the problem graph: Do a git bisect to narrow down the exact offending commit.

• There’s an incident! The SEV manager helps delegate. One person may try to git bisect, another might look at regression code profiles, another might look at user and employee reports. They all contribute to the collective knowledge graph by reporting their findings.

Improving the graph traversal

1. Getting better at understanding programming languages, frameworks, architecture.

2. Getting better at understanding tools and applying them.
1. Debugger, React Component Tools, Datadog, Sentry, Git, etc.

3. Getting better at searching internal tools / google / LLM / stackoverflow / Slack / etc for similar issues.

1. Creating better organizational processes and communication norms for managing problems and incidents.
1. Problems: Asking good questions. Adding engineering investigation notes to tasks. Communicating problems proactively (e.g. dev infra is broken).
2. Incidents: Solid incident management practices, blameless Postmortems.

2. Creating better product, technical, and tooling documentation.
1. General docs. FAQs. Collections of past incidents and solved problems. Debugging and tooling docs.

3. Organizing channels of collective knowledge.
1. Categorizing docs, auditing and improving wiki organization, archiving old docs, organizing Slack channels in a more efficient way.

1. Navigating up and down code stacks, files, and versions very rapidly.

2. Becoming more familiar with the codebase, services, APIs.

3. Narrowing down the problem very fast, e.g. ruling out the problem areas (frontend or backend? which component? which versions?)

Practical tips

1. Solve problems outside of your direct domain, because this builds significant expertise and adds breadth & depth to your knowledge graph. Often, many bugs are caused at a layer outside of your ownership (eg you work on product components but the problem is actually in core web components), and just triage ping ponging the task away or solving the bug at the wrong level costs significant time.

2. Pair often. Pay attention to how other people apply strategies and ask them questions to fill in your knowledge graph!

3. Document the systems (product and technical), the strategies (tooling), and past and present investigations! This expands the collective knowledge graph everybody has access to.

4. Strategize. When debugging, take a step back and consider the two graphs and your debugging strategy. How can you navigate knowledge, tooling, or the code faster to solve the problem as quickly as possible?
1. Maybe this is (1) realizing you need to ask for help sooner because you are not making forward progress or rubber duck debugging (2) timeboxing how long you try a particular approach, or (3) document your investigation, assumptions, and learnings more clearly.

5. Check your assumptions. Often, we get stuck because we’ve marked the problematic nodes in our problem graph as not problematic, but we made a logical error. Writing down all our assumptions and then checking them can help us identify whether this is actually true!
1. Perhaps we thought the offending code MUST be in a particular set of versions or MUST be in a particular part of the codebase, but this actually isn’t true!
Example assumptions:
• The code is in our frontend components, because the issue occurs on frontend but not when querying via the API call directly.
• ^ This assumption could be wrong if either the API call doesn’t match exactly the way the frontend’s calling it or if there’s some middleware or server-side rendering shenanigans.
• This issue began on July 23 2024, so only new code written after that is problematic.
• ^ This assumption could be wrong if we only started getting a few reports on July 23 but we did not do a proper bisect yet to verify.
• The problem is in product code, because it only affects my product.
• ^ This assumption could be wrong, because we may not realize that the issue affects other parts of the app, so the problematic area might actually be a core component and not a product component!
2. Sometimes it’s a random issue from an external vendor (…like CloudStrike).

6. Try other ideas before bisecting. I typically try to solve the problem directly (via inspecting the product, reading the code, traversing git blames, or version diffs) instead of bisecting. I think if you bisect, you rob yourself of debugging a problem directly and all the knowledge you get from it!