scraping_browser_* tools in Bright Data mcp hang for 5 minutes in Docker/Render but return instantly locally #11

Activity

1. meirk-brd commented on May 19, 2025

   

   meirk-brd

   on May 19, 2025

   Contributor

   More actions

   Hi,

   Thank you for reaching out!

   Ive looked into this issue and I have a direction on why this is happening on Docker and not on Mac, to clarify, the issue here is not related to our MCP, we have many different use cases on Docker, non of them faced this issue.

   After diving a bit into Pydantic AI library, there is what I observed:

   run_mcp_servers() leaves its async with, so AsyncExitStack.aclose() awaits each server’s aexit

   In Docker the child gets SIGTERM (graceful), stays alive while the WebSocket is open and After 300 s the watchdog fires → tab closes → Node exits → aclose() unblocks.

   The difference is that locally, the MCP child is force-killed on shutdown (macOS sends SIGKILL), so you never notice the timer.

   The fix: launch the MCP server once at app start-up and keep it alive across requests; close (or time-out) the browser only after your final multi-step tool call.

   Here is an example, using your code snippet and adding the relevant context manager to handle this issue:

   
   from fastapi import FastAPI
   from pydantic_ai.mcp import MCPServerStdio
   from pydantic_ai import Agent
   from dotenv import load_dotenv
   import os
   from pydantic_ai.settings import ModelSettings
   from contextlib import asynccontextmanager
   
   load_dotenv()
   
   app = FastAPI()
   model = "openai:gpt-4.1-mini"
   
   glassdoor_server = MCPServerStdio(
       command="npx",
       args=["@brightdata/mcp"],
       env={
           "API_TOKEN": os.getenv("BRIGHTDATA_API_TOKEN"),
           "WEB_UNLOCKER_ZONE": os.getenv("BRIGHTDATA_GLASSDOOR_UNLOCKER_ZONE"),
           "BROWSER_AUTH": os.getenv("BROWSER_AUTH_GLASSDOOR", ""),
       },
       log_level="debug"
   )
   
   system_prompt = (
       "You are a tool-using agent connected to Bright Data's MCP server."
   )
   
   glassdoor_agent = Agent(
       model,
       mcp_servers=[glassdoor_server],
       retries=1,
       system_prompt=system_prompt,
       model_settings=ModelSettings(
           request_timeout=120,
           max_tokens=5000
       )
   )
   
   @asynccontextmanager
   async def lifespan(app: FastAPI):
       servers_ctx = glassdoor_agent.run_mcp_servers()
       await servers_ctx.__aenter__()          # startup: launch MCP once
       try:
           yield                                # << app serves requests >>
       finally:
           await servers_ctx.__aexit__(None, None, None)   # shutdown
   
   app = FastAPI(lifespan=lifespan)             # FastAPI handles cleanup
   
   
   @app.get("/glassdoor")
   async def analyze_company(name: str):
       prompt = (
           "Using scraping_browser_navigate - navigate to this article "
           "https://dev.to/olgabraginskaya/when-small-parquet-files-become-a-big-problem-and-how-i-ended-up-writing-a-compactor-in-pyarrow-20b6 "
           "and return null immediately without timeout, don't hung out"
       )
   
       result = await glassdoor_agent.run(prompt)
       print(result)
       return {"result": str(result.output)}
   
   

   Let me know if that works for you, or if not - if you are seeing any issues with the MCP implementation that can be solved to make it work, feel free to contribute if needed!
2. olgazju commented on May 21, 2025

   

   olgazju

   on May 21, 2025

   Author

   More actions

   Thank you so much for the detailed explanation and the fix - I’m genuinely impressed by the level of involvement and how quickly you dove into the root cause.

   I’ve tested the solution in Docker and so far it seems to be working well! I really appreciate the support and if I run into anything else or find something useful to contribute back, I definitely will. Thanks again!

   ❤️React with ❤️2Reacted by Meir Kadosh and Ranjan Dailata
3. meirk-brd commented on May 22, 2025

   

   meirk-brd

   on May 22, 2025

   Contributor

   More actions

   Glad I could help-thanks for confirming!

   I’ll go ahead and close this issue now. If you need anything else, don’t hesitate to reach out.
4. meirk-brd

   closed this as completedon May 22, 2025
5. olgazju

   added a commit that references this issue on May 23, 2025

   fixed app for docker image according to brightdata/brightdata-mcp#11 …

   ...

   cd17b5b