summaryrefslogtreecommitdiff
path: root/include
diff options
context:
space:
mode:
authorJan Kara <jack@suse.cz>2023-03-13 10:30:02 +0100
committerJens Axboe <axboe@kernel.dk>2023-03-14 09:25:50 -0600
commit34e0a279a993debaff03158fc2fbf6a00c093643 (patch)
tree93134b8c6bef8f8a2e33a71c24e16a006c4eb26b /include
parente2f2a39452c43b64ea3191642a2661cb8d03827a (diff)
block: do not reverse request order when flushing plug list
Commit 26fed4ac4eab ("block: flush plug based on hardware and software queue order") changed flushing of plug list to submit requests one device at a time. However while doing that it also started using list_add_tail() instead of list_add() used previously thus effectively submitting requests in reverse order. Also when forming a rq_list with remaining requests (in case two or more devices are used), we effectively reverse the ordering of the plug list for each device we process. Submitting requests in reverse order has negative impact on performance for rotational disks (when BFQ is not in use). We observe 10-25% regression in random 4k write throughput, as well as ~20% regression in MariaDB OLTP benchmark on rotational storage on btrfs filesystem. Fix the problem by preserving ordering of the plug list when inserting requests into the queuelist as well as by appending to requeue_list instead of prepending to it. Fixes: 26fed4ac4eab ("block: flush plug based on hardware and software queue order") Signed-off-by: Jan Kara <jack@suse.cz> Reviewed-by: Christoph Hellwig <hch@lst.de> Link: https://lore.kernel.org/r/20230313093002.11756-1-jack@suse.cz Signed-off-by: Jens Axboe <axboe@kernel.dk>
Diffstat (limited to 'include')
-rw-r--r--include/linux/blk-mq.h6
1 files changed, 6 insertions, 0 deletions
diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
index dd5ce1137f04..de0b0c3e7395 100644
--- a/include/linux/blk-mq.h
+++ b/include/linux/blk-mq.h
@@ -228,6 +228,12 @@ static inline unsigned short req_get_ioprio(struct request *req)
*(listptr) = rq; \
} while (0)
+#define rq_list_add_tail(lastpptr, rq) do { \
+ (rq)->rq_next = NULL; \
+ **(lastpptr) = rq; \
+ *(lastpptr) = &rq->rq_next; \
+} while (0)
+
#define rq_list_pop(listptr) \
({ \
struct request *__req = NULL; \