Bug#1040297: gnome: Gnome fails to start on login and falls back to GDM3

[Simon McVittie]

Control: reassign -1 gnome-shell 43.6-1
Control: retitle -1 gnome-shell: fails to start on login: failed to allocate 51540049304 bytes
Control: tags -1 + trixie sid moreinfo

On Tue, 04 Jul 2023 at 08:55:38 +0200, Bastian Venthur wrote:
> Jul 03 16:48:14 fiona gnome-shell[2112]: ../../../glib/gmem.c:136: failed to allocate 51540049304 bytes
> Jul 03 16:48:14 fiona kernel: __vm_enough_memory: pid: 2112, comm: gnome-shell, not enough memory for the allocation

I think this is the key thing here: some upgraded package has caused
gnome-shell to try (and fail) to allocate 51G of RAM at once. The rest
of what you quoted is just fallout from this.

> Since there was no upgrade in Gnome recently, I suspect that one of the
> dependencies got an upgrade. I do vaguely remember evolution, systemd and libc6
> being the biggest updates today.

Please check /var/log/apt for recent upgrades. Mesa is another possibility,
and if you have any GNOME Shell extensions installed/enabled, so are those.

It would be useful to know whether a different user account on the
same system shows the same issue (create a new one with adduser from a
text-mode login if necessary) - that would narrow down whether this is
triggered by some user-specific setting or whether it's a system-wide
issue. gnome-shell is clearly not *completely* unusable on your system,
because you say gdm3 is working, and the gdm3 UI is just gnome-shell in
a special mode - but that special mode turns off a lot of code paths,
including the whole application chooser, which seems to be involved in
the crash.

If you can get a core dump/backtrace from the crash (the syslog entries
that you quoted include a Javascript-level backtrace, but not a C-level
one) then that would probably also be useful information. Installing
system-coredump and using `coredumpctl gdb` is usually the easiest way
to get that, especially for components larger than a single application.

    smcv