Alright I’m at the end of the rope with my Linux knowledge. So I need your help.
I’m on EndeavourOS and just recently upgraded to Linux 7.1.5, all upgrades thus far have been without issue. I am dual-booting but have Linux on its own SSD, and grub handles booting into Linux or Windows.
Now, after booting into Windows and back into Linux, Linux won’t start.
The error:
A start job is running for /dev/disks/by-uuid/<CORRECT UUID>

Let me tell you what I’ve already done:

  • Regenerate mkinitrd with dracut
  • Regenerate grub.cfg and verified via diff it is identical to the current one
  • On a live usb verified that the UUIDs in the grub and fstab files are correct using the KDE partition manager (they are)
  • Verify on a live usb the SSD still works (data and everything is still there)
  • Changed the grub boot option on boot to check that the UUID is not hardcoded somewhere and can be changed (this is true, inputting a garbage UUID changes the boot error to A start job is running for /dev/disks/by-uuid/garbage-uuid

One last idea I had was to drop myself into the dracut emergency shell and actually check what’s actually under /dev/disks/by-uuid and lo and behold: it’s actually missing the drive.

Here’s whats nuts tho: grub loads the drive by UUID as well, loads the initrd, and all that works. When I regenerated the initrd with dracut I could also tell that grub was now loading the newer initrd.

WHAT THE FUCK???

Help?

At this point I REALLY don’t know wtf to do. Why doesnt the initramfs detect the drive? The only useful info I can add here is that all drives listed under /dev/disks/by-uuid had that shorter UUID with no dashes like ABCD1234, whereas the UUID for the correct drive/partition is the longer type UUID like abc1234-1234-0000-defg

  • jwt@programming.dev
    5·
    2 hours ago

    Did the problem begin right after a kernel upgrade? Slim chance, but only thing I can think of is maybe some module your boot process depends on is no longer built-in the kernel and needs to be added to initrd manually. I always have an LTS kernel installed as backup to be able to exclude these kind of fucky issues.

    If you don’t have an LTS kernel installed, you could try to boot a live usb, chroot into your EndeavourOS installation (don’t forget to do a mount -a inside the chroot environment to mount boot and efi partitions) and install the LTS kernel. If you’re able to reboot with an LTS kernel that makes debugging a whole lot easier. Best of luck.

    (The ‘shorter uuids’ are probably the vfat boot and efi partitions btw. Maybe you have access to lsblk or blkid in rescue mode to give some hints? You could compare them to live usb outputs of those commands)

    • undrwater@lemmy.world
      2·
      2 hours ago

      If the previous kernel is still available, can you boot that? That should help determine if the kernel is at fault.

      • ThunderComplex@lemmy.todayOP
        1·
        2 hours ago

        Yeah I up upgraded the kernel, finished my song in windows, then Linux go boom.
        Tried downgrading to 7.1.4 the previous version no luck, will try lts kernel tomorrow

        • undrwater@lemmy.world
          1·
          2 hours ago

          The specific grub command wasn’t listed in the OP, I don’t think?

          grub-mkconfig -o /boot/grub/grub.cfg

          That’s Gentoo syntax anyway, and your grub config may live elsewhere, but you should get the idea. It’s supposed to find the new kernel.

          I’ve never used an initrd, so I can’t comment on that.

          • ThunderComplex@lemmy.todayOPEnglish
            2·
            1 hour ago

            I did do that with a new cfg file then diff’d the current and newly generated cfg files and they were identical

            • undrwater@lemmy.world
              1·
              1 hour ago

              Fascinating.

              I’d like to recommend rEFInd once you figure this out.

              Windows has never been able to squash it.

              • ThunderComplex@lemmy.todayOPEnglish
                1·
                1 hour ago

                thanks but since my boot manager already doesn’t live on the same drive as windows, it can’t squash it anyway hehe

  • corsicanguppy@lemmy.caEnglish
    5·
    4 hours ago

    Yeah so grub can see the disk but then the kernel can’t. Efi and other tricks aren’t involved.

    I think your route is to find out why the kernel hates your ass. Probably an upd pulled a driver … But who removes the SATA or nvme stuff?

    Either way, it’s in the kernel and not the boot or the core apps. I don’t use endeavour, where I could provide better help, but I think the kernel is your best suspect for this crime.

    • ThunderComplex@lemmy.todayOPEnglish
      1·
      4 hours ago

      well since endeavour is very close to arch I hope I’ll find some solutions in that realm. But yeah I’m thinking something must’ve gone awry with the kernel… hm I should downgrade that one too, only downgraded systemd but maybe 7.1.5 has some fuckery

  • BCsven@lemmy.ca
    10·
    6 hours ago

    Smarter people than me can answer, but I had two similar issues before. 1) I added nofail to fstab because in my situation it wasn’t finding a certain drive and getting stuck, and 2) windows fastboot or hybernate had somehow made drive locked or appear dirty to linux

    • ThunderComplex@lemmy.todayOPEnglish
      2·
      5 hours ago

      Its not the fstab, I mean what am I gonna do, not mount the root drive? idk if its windows fastboot but this setup has been working for a couple months and survived windows and linux updates so this seems unlikely

      • BananaTrifleViolin@piefed.worldEnglish
        2·
        3 hours ago

        Are your windows drives mounted in linux? If so comment them out in fstab or nofail them as a minimum, although it wouldn’t normally manifest as a root drive error (unless you’ve set up a strange mount point for the windows drives outside /mnt on root or even in home folder? or do you have them listed above your root drive in fstab so they are trying to mount first and that fails?).

        Also you said elsewhere your linux drives can be “seen” in windows. Just to be clear, you’ve not mounted the drives in Windows right using a tool like Ext2Fsd? Because if so, go into windows and unmount the drive.

        And have you disabled fastboot? Not just rebooted but disabled fastboot completely?

        • ThunderComplex@lemmy.todayOPEnglish
          1·
          2 hours ago

          Yes, I mount my NTFS drives in linux, and they’re all nofail already, it’s literally the root drive it can’t find which makes no sense.
          No, I don’t mount my linux drive on windows, I am not insane :D
          Just checked, fast startup is disabled

      • BCsven@lemmy.ca
        4·
        4 hours ago

        No I realize that, for me it was stuck on a windows partition , which was irrelevant to Linux running. It sounds like it found some corruption and wants to do a full disk check. Or maybe their is a hardware fault arising.

        Others may have better answers for you.

  • Fleppensteyn@sh.itjust.works
    3·
    4 hours ago

    Check again whether windows reenabled fast boot. Maybe disable secure boot in bios as well.

    In grub recovery mode, see which drives are there and try to cd into them.

  • ScoffingLizard@lemmy.dbzer0.com
    3·
    4 hours ago

    This sounds like could be the bit locker function that windows usea. I had this issue with it locking my extra ntfs drive when I installed Linux over my whole primary partition. However, your Windows Drive should not be able to read your Linux drive.

    If Windows bit locker function still caused the trouble, there is a way to fix it but it will continue to happen until you remove Windows.

    If that is the case. Maybe something like ntfsfix command.

  • SavvyBeardedFish@reddthat.com
    4·
    5 hours ago

    Did you also update systemd at the same time? It might be worth reverting that update to see whether it’s that causing the issues.

    To me, this sounds more like an fstab/systemd-automount issue rather than a bootloader/initrd issue

  • lime!@feddit.nu
    6·
    6 hours ago

    can windows see the ssd? did you shut windows down the normal way or the hard way? because there’s one way to shut down windows that is basically a hard hibernate, where it keeps control of the drives. i don’t remember exactly how it works but it can lock other oses out.

      • scutiger@lemmy.world
        4·
        4 hours ago

        The “normal” Windows shutdown is not a real shutdown. I don’t know if it will change anything for your situation, but for a full power off, you have to hold left shift while you click shutdown from the start menu, otherwise it is basically just hibernating.

        • ThunderComplex@lemmy.todayOPEnglish
          1·
          4 hours ago

          I’m constantly doing this. rebooting, shutting the whole PC down for a min then starting it again. def not windows’ fault

    • ThunderComplex@lemmy.todayOPEnglish
      1·
      5 hours ago

      Yeah it has to be, since the grub bootloader I use to boot either linux or windows is on the same disk as linux. If the UEFI firmware could not see the linux disk it couldn’t load grub

  • TruePe4rl@lemmy.mlEnglish
    2·
    6 hours ago

    Not sure if my suggestion will be any help, but in the past I found out that macOS installer changes drive flags and removed some flags from my partitions. Fixed with efibootmgr.