[syzbot] [block?] possible deadlock in blk_mq_freeze_queue_nomemsave

Hello,

syzbot found the following issue on:

HEAD commit: aa918db707fb Merge tag 'bpf_try_alloc_pages' of git://git...
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=1776ec3f980000
kernel config: https://syzkaller.appspot.com/x/.config?x=9c654e2bc7f36976
dashboard link: https://syzkaller.appspot.com/bug?extid=9dd7dbb1a4b915dee638
compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/81273eda411b/disk-aa918db7.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/a2d1a79f3011/vmlinux-aa918db7.xz
kernel image: https://storage.googleapis.com/syzbot-assets/df851fa81fd2/bzImage-aa918db7.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: [email protected]

loop5: detected capacity change from 2048 to 64
======================================================
WARNING: possible circular locking dependency detected
6.14.0-syzkaller-10764-gaa918db707fb #0 Not tainted
------------------------------------------------------
syz.5.738/7855 is trying to acquire lock:
ffffffff9068dfe8 (uevent_sock_mutex){+.+.}-{4:4}, at: uevent_net_broadcast_untagged lib/kobject_uevent.c:317 [inline]
ffffffff9068dfe8 (uevent_sock_mutex){+.+.}-{4:4}, at: kobject_uevent_net_broadcast lib/kobject_uevent.c:410 [inline]
ffffffff9068dfe8 (uevent_sock_mutex){+.+.}-{4:4}, at: kobject_uevent_env+0xb36/0x1870 lib/kobject_uevent.c:608

but task is already holding lock:
ffff888025874ea8 (&q->q_usage_counter(io)#22){++++}-{0:0}, at: blk_mq_freeze_queue_nomemsave+0x15/0x20 block/blk-mq.c:215

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (&q->q_usage_counter(io)#22){++++}-{0:0}:
blk_alloc_queue+0x619/0x760 block/blk-core.c:461
blk_mq_alloc_queue+0x179/0x290 block/blk-mq.c:4349
__blk_mq_alloc_disk+0x29/0x120 block/blk-mq.c:4396
loop_add+0x496/0xb70 drivers/block/loop.c:2067
loop_init+0x164/0x270 drivers/block/loop.c:2302
do_one_initcall+0x123/0x6e0 init/main.c:1257
do_initcall_level init/main.c:1319 [inline]
do_initcalls init/main.c:1335 [inline]
do_basic_setup init/main.c:1354 [inline]
kernel_init_freeable+0x5c2/0x900 init/main.c:1567
kernel_init+0x1c/0x2b0 init/main.c:1457
ret_from_fork+0x48/0x80 arch/x86/kernel/process.c:153
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245

-> #1 (fs_reclaim){+.+.}-{0:0}:
__fs_reclaim_acquire mm/page_alloc.c:3914 [inline]
fs_reclaim_acquire+0x102/0x150 mm/page_alloc.c:3928
might_alloc include/linux/sched/mm.h:318 [inline]
slab_pre_alloc_hook mm/slub.c:4089 [inline]
slab_alloc_node mm/slub.c:4167 [inline]
kmem_cache_alloc_node_noprof+0x57/0x3b0 mm/slub.c:4239
__alloc_skb+0x2b2/0x380 net/core/skbuff.c:658
alloc_skb include/linux/skbuff.h:1340 [inline]
alloc_uevent_skb+0x7d/0x210 lib/kobject_uevent.c:289
uevent_net_broadcast_untagged lib/kobject_uevent.c:326 [inline]
kobject_uevent_net_broadcast lib/kobject_uevent.c:410 [inline]
kobject_uevent_env+0xca4/0x1870 lib/kobject_uevent.c:608
kobject_synth_uevent+0x7d4/0x8a0 lib/kobject_uevent.c:207
bus_uevent_store+0x3d/0x90 drivers/base/bus.c:832
bus_attr_store+0x74/0xb0 drivers/base/bus.c:172
sysfs_kf_write+0x11a/0x170 fs/sysfs/file.c:139
kernfs_fop_write_iter+0x354/0x510 fs/kernfs/file.c:334
new_sync_write fs/read_write.c:591 [inline]
vfs_write+0x5bd/0x1180 fs/read_write.c:684
ksys_write+0x12a/0x240 fs/read_write.c:736
do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
do_syscall_64+0xcd/0x260 arch/x86/entry/syscall_64.c:94
entry_SYSCALL_64_after_hwframe+0x77/0x7f

-> #0 (uevent_sock_mutex){+.+.}-{4:4}:
check_prev_add kernel/locking/lockdep.c:3166 [inline]
check_prevs_add kernel/locking/lockdep.c:3285 [inline]
validate_chain kernel/locking/lockdep.c:3909 [inline]
__lock_acquire+0x1173/0x1ba0 kernel/locking/lockdep.c:5235
lock_acquire kernel/locking/lockdep.c:5866 [inline]
lock_acquire+0x179/0x350 kernel/locking/lockdep.c:5823
__mutex_lock_common kernel/locking/mutex.c:587 [inline]
__mutex_lock+0x199/0xb90 kernel/locking/mutex.c:732
uevent_net_broadcast_untagged lib/kobject_uevent.c:317 [inline]
kobject_uevent_net_broadcast lib/kobject_uevent.c:410 [inline]
kobject_uevent_env+0xb36/0x1870 lib/kobject_uevent.c:608
set_capacity_and_notify+0x1ca/0x240 block/genhd.c:102
loop_set_size drivers/block/loop.c:210 [inline]
loop_set_status+0x94a/0xb90 drivers/block/loop.c:1306
loop_set_status64 drivers/block/loop.c:1415 [inline]
lo_ioctl+0x729/0x26d0 drivers/block/loop.c:1582
blkdev_ioctl+0x277/0x6d0 block/ioctl.c:698
vfs_ioctl fs/ioctl.c:51 [inline]
__do_sys_ioctl fs/ioctl.c:906 [inline]
__se_sys_ioctl fs/ioctl.c:892 [inline]
__x64_sys_ioctl+0x193/0x200 fs/ioctl.c:892
do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
do_syscall_64+0xcd/0x260 arch/x86/entry/syscall_64.c:94
entry_SYSCALL_64_after_hwframe+0x77/0x7f

other info that might help us debug this:

Chain exists of:
uevent_sock_mutex --> fs_reclaim --> &q->q_usage_counter(io)#22

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(&q->q_usage_counter(io)#22);
lock(fs_reclaim);
lock(&q->q_usage_counter(io)#22);
lock(uevent_sock_mutex);

*** DEADLOCK ***

3 locks held by syz.5.738/7855:
#0: ffff8880259e4b68 (&lo->lo_mutex){+.+.}-{4:4}, at: loop_set_status+0x2a/0xb90 drivers/block/loop.c:1275
#1: ffff888025874ea8 (&q->q_usage_counter(io)#22){++++}-{0:0}, at: blk_mq_freeze_queue_nomemsave+0x15/0x20 block/blk-mq.c:215
#2: ffff888025874ee0 (&q->q_usage_counter(queue)#21){+.+.}-{0:0}, at: blk_mq_freeze_queue_nomemsave+0x15/0x20 block/blk-mq.c:215

stack backtrace:
CPU: 0 UID: 0 PID: 7855 Comm: syz.5.738 Not tainted 6.14.0-syzkaller-10764-gaa918db707fb #0 PREEMPT(full)
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/12/2025
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:94 [inline]
dump_stack_lvl+0x116/0x1f0 lib/dump_stack.c:120
print_circular_bug+0x275/0x350 kernel/locking/lockdep.c:2079
check_noncircular+0x14c/0x170 kernel/locking/lockdep.c:2211
check_prev_add kernel/locking/lockdep.c:3166 [inline]
check_prevs_add kernel/locking/lockdep.c:3285 [inline]
validate_chain kernel/locking/lockdep.c:3909 [inline]
__lock_acquire+0x1173/0x1ba0 kernel/locking/lockdep.c:5235
lock_acquire kernel/locking/lockdep.c:5866 [inline]
lock_acquire+0x179/0x350 kernel/locking/lockdep.c:5823
__mutex_lock_common kernel/locking/mutex.c:587 [inline]
__mutex_lock+0x199/0xb90 kernel/locking/mutex.c:732
uevent_net_broadcast_untagged lib/kobject_uevent.c:317 [inline]
kobject_uevent_net_broadcast lib/kobject_uevent.c:410 [inline]
kobject_uevent_env+0xb36/0x1870 lib/kobject_uevent.c:608
set_capacity_and_notify+0x1ca/0x240 block/genhd.c:102
loop_set_size drivers/block/loop.c:210 [inline]
loop_set_status+0x94a/0xb90 drivers/block/loop.c:1306
loop_set_status64 drivers/block/loop.c:1415 [inline]
lo_ioctl+0x729/0x26d0 drivers/block/loop.c:1582
blkdev_ioctl+0x277/0x6d0 block/ioctl.c:698
vfs_ioctl fs/ioctl.c:51 [inline]
__do_sys_ioctl fs/ioctl.c:906 [inline]
__se_sys_ioctl fs/ioctl.c:892 [inline]
__x64_sys_ioctl+0x193/0x200 fs/ioctl.c:892
do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
do_syscall_64+0xcd/0x260 arch/x86/entry/syscall_64.c:94
entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f74ac58d169
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f74ad427038 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007f74ac7a5fa0 RCX: 00007f74ac58d169
RDX: 00002000000007c0 RSI: 0000000000004c04 RDI: 0000000000000006
RBP: 00007f74ac60e2a0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 00007f74ac7a5fa0 R15: 00007ffc990fb088
</TASK>


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at [email protected].

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

syzbot has found a reproducer for the following issue on:

HEAD commit: a1b5bd45d4ee Merge tag 'usb-6.15-rc1' of git://git.kernel...
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=1711cfb0580000
kernel config: https://syzkaller.appspot.com/x/.config?x=adffebefc9feb9d6
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=12ee494c580000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=13df5998580000

Downloadable assets:
disk image (non-bootable): https://storage.googleapis.com/syzbot-assets/7feb34a89c2a/non_bootable_disk-a1b5bd45.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/588b4d489b63/vmlinux-a1b5bd45.xz
kernel image: https://storage.googleapis.com/syzbot-assets/51ead797f7ae/bzImage-a1b5bd45.xz
6.14.0-syzkaller-12886-ga1b5bd45d4ee #0 Not tainted
------------------------------------------------------
syz-executor170/5939 is trying to acquire lock:
ffffffff906953a8 (uevent_sock_mutex){+.+.}-{4:4}, at: uevent_net_broadcast_untagged lib/kobject_uevent.c:317 [inline]
ffffffff906953a8 (uevent_sock_mutex){+.+.}-{4:4}, at: kobject_uevent_net_broadcast lib/kobject_uevent.c:410 [inline]
ffffffff906953a8 (uevent_sock_mutex){+.+.}-{4:4}, at: kobject_uevent_env+0xb36/0x1870 lib/kobject_uevent.c:608
ffff888025aa6228 (&q->q_usage_counter(io)#24){++++}-{0:0}, at: blk_mq_freeze_queue_nomemsave+0x15/0x20 block/blk-mq.c:215
-> #2 (&q->q_usage_counter(io)#24){++++}-{0:0}:
do_one_initcall+0x120/0x6e0 init/main.c:1257
ret_from_fork+0x45/0x80 arch/x86/kernel/process.c:153
__fs_reclaim_acquire mm/page_alloc.c:4028 [inline]
fs_reclaim_acquire+0x102/0x150 mm/page_alloc.c:4042
might_alloc include/linux/sched/mm.h:318 [inline]
slab_pre_alloc_hook mm/slub.c:4102 [inline]
slab_alloc_node mm/slub.c:4180 [inline]
kmem_cache_alloc_node_noprof+0x57/0x3b0 mm/slub.c:4252
bus_attr_store+0x71/0xb0 drivers/base/bus.c:172
sysfs_kf_write+0xef/0x150 fs/sysfs/file.c:145
kernfs_fop_write_iter+0x351/0x510 fs/kernfs/file.c:334
new_sync_write fs/read_write.c:591 [inline]
vfs_write+0x5ba/0x1180 fs/read_write.c:684
__mutex_lock_common kernel/locking/mutex.c:601 [inline]
__mutex_lock+0x199/0xb90 kernel/locking/mutex.c:746
loop_set_size drivers/block/loop.c:211 [inline]
loop_set_size drivers/block/loop.c:208 [inline]
loop_set_status+0x9c7/0xb90 drivers/block/loop.c:1306
loop_set_status_old+0x162/0x1d0 drivers/block/loop.c:1405
lo_ioctl+0x695/0x26d0 drivers/block/loop.c:1575
blkdev_ioctl+0x274/0x6d0 block/ioctl.c:698
__x64_sys_ioctl+0x190/0x200 fs/ioctl.c:892
uevent_sock_mutex --> fs_reclaim --> &q->q_usage_counter(io)#24
lock(&q->q_usage_counter(io)#24);
lock(fs_reclaim);
lock(&q->q_usage_counter(io)#24);
lock(uevent_sock_mutex);

*** DEADLOCK ***

3 locks held by syz-executor170/5939:
#0: ffff888025c4fb68 (&lo->lo_mutex){+.+.}-{4:4}, at: loop_set_status+0x2a/0xb90 drivers/block/loop.c:1275
#1: ffff888025aa6228 (&q->q_usage_counter(io)#24){++++}-{0:0}, at: blk_mq_freeze_queue_nomemsave+0x15/0x20 block/blk-mq.c:215
#2: ffff888025aa6260 (&q->q_usage_counter(queue)#20){+.+.}-{0:0}, at: blk_mq_freeze_queue_nomemsave+0x15/0x20 block/blk-mq.c:215

stack backtrace:
CPU: 0 UID: 0 PID: 5939 Comm: syz-executor170 Not tainted 6.14.0-syzkaller-12886-ga1b5bd45d4ee #0 PREEMPT(full)
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014
__mutex_lock_common kernel/locking/mutex.c:601 [inline]
__mutex_lock+0x199/0xb90 kernel/locking/mutex.c:746
loop_set_size drivers/block/loop.c:211 [inline]
loop_set_size drivers/block/loop.c:208 [inline]
loop_set_status+0x9c7/0xb90 drivers/block/loop.c:1306
loop_set_status_old+0x162/0x1d0 drivers/block/loop.c:1405
lo_ioctl+0x695/0x26d0 drivers/block/loop.c:1575
blkdev_ioctl+0x274/0x6d0 block/ioctl.c:698
__x64_sys_ioctl+0x190/0x200 fs/ioctl.c:892
RIP: 0033:0x7f5b0284a229
Code: 48 83 c4 28 c3 e8 37 17 00 00 0f 1f 80 00 00 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffeda6302f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007ffeda6304c8 RCX: 00007f5b0284a229
RDX: 0000200000000080 RSI: 0000000000004c02 RDI: 0000000000000003
RBP: 00007f5b028bd610 R08: 00007ffeda6304c8 R09: 00007ffeda6304c8
R10: 00007ffeda6304c8 R11: 0000000000000246 R12: 0000000000000001
R13: 00007ffeda6304b8 R14: 0000000000000001 R15: 0000000000000001
</TASK>


---
If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

syzbot has bisected this issue to:

commit ffa1e7ada456087c2402b37cd6b2863ced29aff0
Author: Thomas Hellström <[email protected]>
Date: Tue Mar 18 09:55:48 2025 +0000

block: Make request_queue lockdep splats show up earlier

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=172e2fb0580000
start commit: 92b71befc349 Merge tag 'objtool-urgent-2025-04-01' of git:..
git tree: upstream
final oops: https://syzkaller.appspot.com/x/report.txt?x=14ae2fb0580000
console output: https://syzkaller.appspot.com/x/log.txt?x=10ae2fb0580000
kernel config: https://syzkaller.appspot.com/x/.config?x=47cc4b0c03278250
dashboard link: https://syzkaller.appspot.com/bug?extid=9dd7dbb1a4b915dee638
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=1680d998580000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=1180d998580000

Reported-by: [email protected]
Fixes: ffa1e7ada456 ("block: Make request_queue lockdep splats show up earlier")

For information about bisection process see: https://goo.gl/tpsmEJ#bisection

On Thu, 03 Apr 2025 01:34:24 -0700
#syz test

--- x/block/blk-core.c
+++ y/block/blk-core.c
@@ -456,12 +456,6 @@ struct request_queue *blk_alloc_queue(st
lockdep_init_map(&q->q_lockdep_map, "&q->q_usage_counter(queue)",
&q->q_lock_cls_key, 0);

- /* Teach lockdep about lock ordering (reclaim WRT queue freeze lock). */
- fs_reclaim_acquire(GFP_KERNEL);
- rwsem_acquire_read(&q->io_lockdep_map, 0, 0, _RET_IP_);
- rwsem_release(&q->io_lockdep_map, _RET_IP_);
- fs_reclaim_release(GFP_KERNEL);
-
q->nr_requests = BLKDEV_DEFAULT_RQ;

return q;
--

...
#syz test: https://github.com/ming1/linux.git syzbot-test


Thanks,
Ming

Hello,

syzbot has tested the proposed patch but the reproducer is still triggering an issue:
unregister_netdevice: waiting for DEV to become free

unregister_netdevice: waiting for batadv0 to become free. Usage count = 3


Tested on:

commit: 665c7e64 loop: replace freezing queue with quiesce & f..
git tree: https://github.com/ming1/linux.git syzbot-test
console output: https://syzkaller.appspot.com/x/log.txt?x=12321178580000
Note: no patches were applied.

commit: e48e99b6 Merge tag 'pull-fixes' of git://git.kernel.or..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=1564823f980000
kernel config: https://syzkaller.appspot.com/x/.config?x=c79406130aa88d22
patch: https://syzkaller.appspot.com/x/patch.diff?x=14783998580000