Merge @jupyterlab/debugger into JupyterLab core

[afshin]

We propose merging @jupyterlab/debugger into JupyterLab core as a 3.0 feature that only appears for users who have a kernel that supports debugging. Because the end-user experience is transparent, this feature should not cause disruption.

As of July 2020, the only kernel that supports the Debug Adapter Protocol is the xeus-python kernel, but we are eager for other kernels to support debugging. The front-end debugger is kernel-agnostic and will only show up for users who have installed at least one kernel that supports DAP.

cf. VSCode support for debugging Jupyter notebooks:

• June 2020 (version 1.47)
• debugging with xeus-python example

[blink1073]

I support this proposal

[ellisonbg]

I support

…

On Wed, Jul 15, 2020 at 10:15 AM Steven Silvester ***@***.***> wrote: I support this proposal — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#75 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAGXUHCOBMHOUS5LL3V5PTR3XP35ANCNFSM4O2XB7HQ> .

-- Brian E. Granger Principal Technical Program Manager, AWS AI Platform (brgrange@amazon.com) On Leave - Professor of Physics and Data Science, Cal Poly @ellisonbg on GitHub

[saulshanabrook]

I can understand the pressure to include this in core, especially since VS Code does now. And I am very thankful to all the work put in from folks to push this feature along, it will obviously have huge benefits to users.

However, I wanted to articulate one potential objection.

If we merge this in and release a 3.0 blog post announcing this feature, then users of JupyterLab will be presented with a choice:

1. Run JupyterLab by default, and run notebooks, don't get access to new debugger feature.
2. Install xeus-kernel after installing jupyterlab, get access to debugger but not to magics or IPython runtime library (I believe)

This will lead, I believe, to some pressure from users to move the community to a place where they can have a default kernel that will both have backwards compatible support and the debugger support.

So that means at some point in the future, if we merge this PR, we will either want to:

1. Add more features to xeus-python and make it a default install with JupyterLab, switching the default kernel
2. Add debugging support for ipykernel.

When we (at Quansight, including @Carreau) explored that question for a client of ours, we tentatively came to the conclusion that we should support bring xeus-python up to ipykernel support, because no one really wanted to keep maintaining ipykernel and adding debugger support would require a large rework.

But it required more funding than we had to even explore that question in a comprehensive way. If anyone has already done this exploration, it would be really helpful to share any findings from that to understand how much money and time that would require.

---

So my preference at the moment would be to wait on including debugger support until our default kernel we include has support for it.

If we include it, since we are a high visibility project, we will effectively have to announce it. And it's unclear what win including it brings, if we aren't advertising it as a new feature. Of course, it's more troublesome to install a third party extension, but that is true for many other extensions as well which have higher usage than this one.

[Carreau]

It's still unclear how much work is needed in ipykernel to get it to the right point, there are also a couple of things in the current jupyter implementation protocol that are incorrect/impossible, for example the control channel message receiving is blocked by the GIL, so it would need to be moved to a thread which has its own implications – xeus does that. It for example affects killing and restarting the kernel and there is hack in the notebook server that kill the process instead of sending a graceful shutdown message.

Also AFAICT after talking with Sylvain is that Xeus will not (and cannot) be 100% compatible with IPykernel (WRT configuration & extensions), and is not yet packaged on all platforms because it has to be compiled; So we will likely always have a need for an ipykernel. I've worked with sylvain to make the IPython utils better delineated so that xeux en

If core of JupyterLab want to merge support that's your decision, but I do think this may fracture the ecosystem, and that kernels maintainer (me), are stating to get tired of being single maintainer of projects no one want to spend critical time on (IPython, IPykernel, Traitlets). I did work with Sylvain to make IPython more modular and importable from Xeus, but I'm at the limit to the number of support request I can handle.

The client @saulshanabrook talked about really want an official way of using a debugger, but they do have a number of extensions/magics, and having their user switch kernels depending on the features supported will not make them happy.

There have already been a big fracture on the Python ecosystem with the move to JupyterLab, as people are still using classic notebook, and there is little time spent on it. So I would be mindful.

[blink1073]

I am currently maintaining ipykernel, and will continue to do so. Members of the JupyterLab team are actively working to ease the transition between classic notebook and JupyterLab, and have always done our level best on that front.

Other kernel authors are working on adding debugger support. I don't think it is fair to say that we are causing a rift by offering a feature that is standard in IDEs.

[afshin]

The way I think about rolling out new features is based on these simple heuristics:

1. Does this benefit our end users?
   A feature that is totally invisible and inert unless it can benefit an end user's experience after they already have a debugger-friendly kernel seems only good for users. If we are concerned about advertising it in a confusing way, I'd propose: let's not advertise it in a confusing way.
2. Does this benefit our ecosystem partners / extension authors?
   The answer to this seems clear-cut too, insofar as it is a no-op.

If our own tools are shipping in other products, I submit they should ship with ours as well.

[SylvainCorlay]

Quick comments on xeus:

• xeus-python supports IPython magics as of 0.8.2. The matplotlib magic is not yet supported.
• it is packaged for Linux, OSX, and Windows on conda. I have not yet pushed PyPI wheels for windows over questions with respect to the static build of openssl. It is a matter of a couple of days to sort that question out.

[lresende]

* xeus-python supports IPython magics as of 0.8.2. The matplotlib magic is not yet supported.

I still see the magic issue open - jupyter-xeus/xeus-python#63

* it is packaged for Linux, OSX, and Windows on conda. I have not yet pushed PyPI wheels for windows over questions with respect to the static build of openssl. It is a matter of a couple of days to sort that question out.

So, this means it has a dependency on anaconda? Or is pip install also supported? When I looked at the last year Python Developer Survey conda was used by only about 23% of python developers.

[SylvainCorlay]

I still see the magic issue open - jupyter-xeus/xeus-python#63

Cool.

So, this means it has a dependency on anaconda?

No. Why would you think so?

Or is pip install also supported? When I looked at the last year Python Developer Survey conda was used by only about 23% of python developers.

Cool. Pip install is supported on osx and linux. We build windows wheels as well in our CI but as I said, I have not uploaded them yet.

[SylvainCorlay]

Does this benefit our end users?
A feature that is totally invisible and inert unless it can benefit an end user's experience after they already have a debugger-friendly kernel seems only good for users.

👍 I agree and I support the proposal.

[timkpaine]

Does xeus have this critical feature of ipykernel?

from ipykernel import IPKernelApp; IPKernelApp.launch_instance()

Without the ability to launch from an existing python process it doesn't have much utility for those of us with large, custom python deployments.

Also the idea of having certain python kernels support certain features and not others seems bad. If you try to install non "big 3" kernels the user experience degrades substantially and it becomes very confusing very quick.

[SylvainCorlay]

Does xeus have this critical feature of ipykernel?

from ipykernel import IPKernelApp; IPKernelApp.launch_instance()

Not quite. You are free to use ipykernel if xeus-python does not fit your need.

Note that this issue is about incorporating the debugger front-end in lab. The debugging protocol was devised in a language agnostic fashion and other kernel authors are already looking at supporting it in their implementation.

[timkpaine]

Would it be possible to leave it out of the assets so as to not bloat the build with something we cannot use? If it's not going to work for most cases it should probably stay separate.

[SylvainCorlay]

Would it be possible to leave it out of the assets so as to not bloat the build with something we cannot use? If it's not going to work for most cases it should probably stay separate.

You can certainly use the debugger if you install a kernel that support debugging.