Tracing Python Exceptions with static analysis to find production bugs open source projects like httpbin, Redash, Airflow, and Label Studio

The problem with Python backend APIs

Backend APIs ideally should never raise unexpected, uncaught errors. The user ends up getting back a generic 500 error message that will make debugging slow and require someone to look at logs. In the worst case, uncaught exceptions can end up with your program saving bad data or breaking user flows. For example if an endpoint transfers money by calling an external API, and you don’t handle the possible failures from requests.get() the user request might get stuck in some sort of “processing” state that requires manual intervention. For this reason, amongst others, many API driven products end up writing their backends in languages like Go and Kotlin that force the developer to handle exceptions explicitly.

Example bug found in HTTPBin

Here is a simple bug found in httpbin, a popular repository owned by Postman that people use to test HTTP clients. Bubble found an uncaught ValueError when you send an invalid qop (quality of protection) parameter in the digest authorization header.

Try it yourself (this works on the live httpbin.org right now):

# Bug: Invalid qop parameter returns 500 instead of 401
curl -s -o /dev/null -w "%{http_code}\n" \
  -H 'Authorization: Digest username="u", realm="r", nonce="n", uri="/", qop=INVALID, nc=1, cnonce="c", response="r"' \
  'https://httpbin.org/digest-auth/auth/user/passwd'
# Returns: 500

# Expected: Valid qop parameter returns 401 (wrong password, but proper error)
curl -s -o /dev/null -w "%{http_code}\n" \
  -H 'Authorization: Digest username="u", realm="r", nonce="n", uri="/", qop=auth, nc=1, cnonce="c", response="r"' \
  'https://httpbin.org/digest-auth/auth/user/passwd'
# Returns: 401

Is this a real bug or a design choice? It’s a bug - RFC 7616 states:

If a parameter or its value is improper, or required parameters are missing, the proper response is a 4xx error code.

Issues found in Open Source Python Projects

The tool flags endpoints where exceptions escape to global handlers. Human review then confirms which are real bugs.

PROJECT          FRAMEWORK    ENDPOINTS   FLAGGED      BUGS
─────────────────────────────────────────────────────────────
httpbin          Flask              55     3  (5%)        1
Redash           Flask             135   119 (88%)        3
Airflow          FastAPI           122    31 (25%)        5
Label Studio     Django/DRF        124    85 (69%)        3
─────────────────────────────────────────────────────────────
                                   436   238            12

High “flagged” counts (like Redash’s 88%) indicate shared code paths - e.g., a get_setting() function called on every request that can raise KeyError. Static analysis flags all of them; human review determines which are actually reachable.

The pattern across all confirmed bugs: the correct exception handling already exists in the same codebase. These are inconsistencies, not design decisions. (Bug reports written by Claude Code, the rest written by me)

GET /digest-auth/auth/user/passwd
  → digest_auth()           core.py:1120
    → check_digest_auth()   helpers.py:353
      → response()          helpers.py:311
        → HA2()             helpers.py:296
          → raise ValueError

curl -s -o /dev/null -w "%{http_code}" \
  -H 'Authorization: Digest username="u", realm="r", nonce="n", uri="/", qop=INVALID, nc=1, cnonce="c", response="r"' \
  'https://httpbin.org/digest-auth/auth/user/passwd'
# Returns: 500 (should be 401)

GET /api/queries/123/dropdowns/456
  → QueryDropdownsResource.get()   query_results.py:204
    → dropdown_values()            query_results.py:178
      → raise QueryDetachedFromDataSourceError

# Line 194 - CORRECT
class QueryResultDropdownResource(BaseResource):
    def get(self, query_id):
        try:
            return dropdown_values(query_id, self.current_org)
        except QueryDetachedFromDataSourceError as e:
            abort(400, message=str(e))

# Line 204 - BUG
class QueryDropdownsResource(BaseResource):
    def get(self, query_id, dropdown_query_id):
        return dropdown_values(dropdown_query_id, self.current_org)  # No try/except!

GET /api/users?disabled=invalid
  → UserListResource.get()   users.py:115
    → parse_boolean()        helpers.py:22
      → raise ValueError

# BUG: No validation
disabled = parse_boolean(disabled)

# Same file, CORRECT pattern:
if page < 1:
    abort(400, message="Page must be positive integer.")

POST /api/settings/organization
  → OrganizationSettings.post()   settings.py:42
    → set_setting()               organizations.py:58
      → raise KeyError

POST /api/storages/s3/123/sync
  → ImportStorageSyncAPI.post()     api.py:129
    → storage.validate_connection() base_models.py:298
      → raise KeyError

# api.py:129 - BUG (sync endpoint)
def post(self, request, *args, **kwargs):
    storage.validate_connection()  # No try/except!
    storage.sync()

# api.py:86 - CORRECT (create endpoint)
def perform_create(self, serializer):
    try:
        instance.validate_connection()
    except Exception as exc:
        raise ValidationError(exc)  # Returns 400

# data_manager/functions.py:21 - BUG
class DataManagerException(Exception):  # Should extend APIException!
    pass

class LabelStudioAPIException(APIException):
    status_code = status.HTTP_500_INTERNAL_SERVER_ERROR

class AnnotationDuplicateError(LabelStudioAPIException):  # Correct!
    status_code = status.HTTP_409_CONFLICT

GET /auth/menus
  → get_auth_menus()      auth.py:32
    → get_auth_manager()  app.py:158
      → raise RuntimeError("Auth Manager has not been initialized")

# BUG - global state access
def get_auth_manager() -> BaseAuthManager:
    if _AuthManagerState.instance is None:
        raise RuntimeError("Auth Manager has not been initialized")
    return _AuthManagerState.instance

# CORRECT - dependency injection (exists in same codebase)
AuthManagerDep = Annotated[BaseAuthManager, Depends(auth_manager_from_app)]

GET /structure_data
  → get_upstream_assets()   structure.py:37
    → raise TypeError("Unsupported type: ...")

PATCH /{task_instance_id}/run
  → ti_run()              task_instances.py:101
    → raise ValueError("DagRun not found")

Bubble

Finding bugs across Flask, FastAPI, and Django required understanding how each framework handles exceptions differently. What Bubble does is take in entrypoints to programs and determine which errors will be caught and which end up propagating past the entrypoint to the high level exception handler. The reason I made it was partially because it seems to be a genuine gap in the Python tooling ecosystem and because I have been exploring “AI Agent first” development tools in the repos I work with. In many of the repos I work in I create fully custom tools to help Claude/ AI agents better understand and interact with the codebase - but this case is more general.

Doing this pattern - understanding entrypoints, how errors are caught in a framework or library, understanding the relationships between subtypes and whatnot is something that needs to be handled differently for each framework or library. The great and awful thing about Python is that it lets you do “code magic” with decorators, context managers, and lots of other tricks. This has been a serious blocker for great dev tooling in Python since it is hard to make a one size fits all tool for these and in the past it was unrealistic for a tool to be able to keep up with all this. Sentry, for example, started out as a django specific tool.

Claude Code as the System Integrator

Every framework has different exception handling practices. Before it would’ve been unrealistic to do lots of custom work to integrate a software quality tool - it needed to work out of the box. Now a new class of tool or system that needs custom integration work per repository or per use case is possible since AI agents like Claude Code can do large portions of the integration work. This pattern is particularly strong for developer tools or domains like data pipelines where there is strong machine readable feedback and the agent can run it multiple times, get feedback and sanity check it, and adjust accordingly. Not to say that Claude Code is perfect at this now but it is sufficient and only getting better.

In this program specifically the “custom integration” is implementing an interface to detect which functions are exposed API routes, what functions are error handlers, and what exceptions are “handled out of the box” by the framework like HttpException in Flask/ FastAPI. So what Claude Code did to integrate FastAPI for example is define custom AST vistors to detect routes and exception handlers.

I think we will see this pattern play out across the software economy not just for developer tools. There are entire industries around software with high integration costs and the promise of a big payoff - companies like Palantir, Salesforce, Oracle, and others have full ecosystems around their system integrators. In many cases they only go after 7+ figure contracts. This is an area I will be watching closely - products and strategies that weren’t even possible 12 months ago because the the unit economics of custom integration work made them unworkable. It might even make some of these kinds of services that were only possible in a B2B context available to end consumers.

The future of Python and backend development

It will be interesting to see if Python’s share of backend development will grow or shrink in the next era. My hope is that Python will get increasingly strong tools and we are just entering a golden age.

Try it yourself

pip install bubble-analysis

# Audit a Flask project
bubble flask audit -d /path/to/your/project

# Audit a FastAPI project
bubble fastapi audit -d /path/to/your/project

# Trace exceptions from a specific function
bubble escapes some_function -d /path/to/your/project

The tool supports Flask, FastAPI, and Django right now.