Modernizing my Python development setup in Emacs

Updated Feb 2024: Starting properly on MacOS; Completion demo; "peek" find demo.

For some years, I've got by with a gradually evolving set of Emacs extensions and configuration to make me productive and happy while working with Python. I tend to be slow to adopt new tools once I've got something working. But my old setup was getting pretty crufty, and there's so much buzz around LSP that -- rather than yell at the VSCode kids to get off my lawn -- I wanted to see what I was missing.

As always, my complete emacs config lives on my github.

Thanks to the folks in the "tool talk" meeting at Recurse for inspiring me to finally do this.

I'm just starting to get comfortable with it, but here's where I'm at currently:

LSP

Emacs has good support for LSP and it was pretty easy to set up. I use straight to manage my Emacs packages, so all I had to do was this, adapted from the lsp-mode docs:

(straight-use-package 'lsp-mode)
(use-package lsp-ui
  :commands lsp-ui-mode
  :straight t)

(add-hook 'python-mode-hook 'lsp-deferred)

This didn't actually do anything until I had a language server installed. That's where things got thorny...

Per-buffer environment setup

Spoiler alert: we're going to use direnv! Skip to the next section if you're not curious about why.

The long-winded problem

To run an LSP server, you typically need to install the server in the environment (eg. for python, something like venv, virtualenv, or pyenv) where your "project" is being developed/run. This means running a command such as pip install python-lsp-server or pip install pyright.

But... what about when you frequently work on multiple projects, each with their own environment? You don't want them to clash with each other, or you might run into trouble such as your language server being unable to find completions for code you know is in your project - because it's looking at the wrong project.

In order to find the right server, emacs needs to activate the right environment when working on relevant files, so that the PATH environment variable gets updated and Emacs can find the right server(s) to run.

Over the years, I've tried various Emacs extensions that are supposed to help with working in virtual python environments: pyvenv, virtualenvwrapper, and the like. The problem is, they typically make changes to the global environment for the entire Emacs editing session. This means I couldn't really work on multiple projects at once. And, each one of those Emacs modes is tied to a specific solution such as virtualenv, which makes it harder to switch between those tools - and doesn't help with other languages.

Solution, part 1: Direnv

direnv is a really neat tool for the shell that supports automatically updating environment variables when a directory is entered.

For python, this means you can easily activate / deactivate a virtualenv just by cding into / out of a directory. That's pretty handy regardless of emacs! I've been overdue for a tool like this for a long time. It does require you to put .envrc files in directories of interest, but they are easy to write, and it seems worth the trouble.

Direnv has pre-built recipes for setting up a python 3 virtual env; I'll probably use that in the future, but I have often lazily done python3 -m venv in the root of a project I'm working on, and I found that it was easy to support my existing projects by dropping in the following .envrc file:

test -f bin/activate && source bin/activate

Even better, direnv is not at all Python-specific, so I will probably leverage it when working in other languages too.

Solution, part 2: Envrc mode

There is of course an Emacs mode to integrate with direnv: envrc. The best thing? It's buffer-local!

This means I can open files in different projects, each with their own virtual env, and emacs will see different PATH etc. for each file I have open.

This means I can run correctly isolated LSP sessions for each project!

Setting it up turned out to be easy as well:

(straight-use-package 'envrc)

... with one gotcha, you have to do this after all other config (for reasons explained in the docs):

(envrc-global-mode)

Installing the LSP server and related tools for each project

I haven't automated this for new projects, but a basic recipe to start a new Python project would look something like:

mkdir my-new-project
cd my-new-project
git init .

# Set it up for direnv
echo "layout python3" > .envrc
direnv allow
# Congrats, you now have a virtual python 3 environment in .direnv/python-3.11
# and it's active already!
# You probably want to git ignore those.

# Now we need our python packages:
pip install python-lsp-server pylsp-mypy flake8

It took a bit of trial and error, but now it Just Works.

Fixing a MacOS environment issue

I found that when launching emacs via Spotlight, which is my preferred way on the mac, it did not properly see my $PATH because Spotlight doesn't use the whole environment from your preferred shell.

This led to some weird behavior such as direnv loading the wrong version of Python. For example, if I have an .envrc file consisting of layout python3, direnv would apparently find the system Python 3.9 and look for (or bootstrap) a new virtual environment for that version.

But when direnv was run in a login shell, it would find the python 3 version I'd installed via homebrew, eg 3.12, and use that. So I'd have all my packages such as python-lsp-server installed for 3.12. Then when Emacs looked for the pylsp executable, it was wrongly looking in the bin directory for 3.9. So LSP would fail.

Solution: inspired by a stackoverflow post, I make an Automator script to launch Emacs.

  • Launch Automator
  • Select "Application" type
  • Choose "Run Shell Script"
  • Paste in this source:
EMACSDIR="$(dirname $(dirname $(readlink -f /opt/homebrew/bin/emacs)))"
bash --login -c "$EMACSDIR/bin/emacs"
  • Then save this as Emacs.app in /Applications

The first line attempts to get the path to wherever Homebrew put emacs, regardless of specific version number. The second line starts a subshell runs bash in login mode, thus loading my .bash_profile and setting $PATH.

Now when I run emacs via spotlight, any commands it launches should work exactly the same as in my shell.

What does all this buy you?

I'm still exploring the features of LSP, but I'm liking these basic IDE-like features. Some of them I had before via other means, but this is already nicer:

Completion

This didn't take any configuration, it Just Worked. (I'm unclear exactly how since TAB still is bound to py-indent-or-complete from python-mode.el, but it's great.)

Find references, find definition

Find references is bound by default to s-l g r. On my mac, I have Command as super.

Even nicer is the "peek" interface via s-l G r:

Find definition is likewise bound to s-l g g, or s-l G g for the "peek" version.

This replaces language-specific mode features such as jedi:goto-definition, or my previous favored solution of using Emacs' xref and tags functionality. Xref was useful because it's fast in some huge repos I had to work with. LSP seems better at finding the actual definition; xref often showed me a bunch of false positives. And, now I never have to generate a TAGS file again, hopefully.

We'll see how well LSP holds up next time I have to work on a giant monolith ...

One thing I really miss from xref though; it had a "back" button, AKA xref-pop-marker-stack, bound to M-, which would take me back where I was before following the definition. lsp-mode doesn't seem to have that :-(

Inline documentation on hover!

Nice.

Style errors / warnings

I had this without LSP via flycheck but it didn't always find the right executables; that's solved now thanks to direnv, yay, as long as flake8 or another compatible linter is installed in the environment.

The way errors are currently displayed inline is a little weird for me, I may look into how to tweak that, but it's usable already.

Type error hints via MyPy

These are handy! Just need to have pylsp-mypy installed in the environment.

Reformatting via plugins

This one tripped me up at first because it turns out they are off by default as per the docs. For pylsp, you have to explicitly enable one of the plugins. So for example, to use black for python, install the plugin in your shell:

pip install python-lsp-black

And then configure it in emacs:

(setq lsp-pylsp-plugins-black-enabled 't)

Class / method / function breadcrumbs at top of the window.

I had this without LSP via which function mode but I like the breadcrumb style better.

More things to try in future

References

Other noteworthy blog posts that inspired me:

Tags