block: optimize bvec iteration in bvec_iter_advance

There is no need to only iterate in chunks of PAGE_SIZE or less in
bvec_iter_advance, given that the callers pass in the chunk length that
they are operating on - either that already is less than PAGE_SIZE
because they do classic page-based iteration, or it is larger because
the caller operates on multi-page bvecs.

This should help shaving off a few cycles of the I/O hot path.

Reviewed-by: Johannes Thumshirn <jthumshirn@suse.de>
Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>

diff --git a/include/linux/bvec.h b/include/linux/bvec.h
index 87e82e503a52fbd55ac184656950c0511237c94d..f6275c4da13a765fd60f3d34a9ca491d1d56db93 100644 (file)
--- a/include/linux/bvec.h
+++ b/include/linux/bvec.h
@@ -112,14 +112,15 @@ static inline bool bvec_iter_advance(const struct bio_vec *bv,
        }
 
        while (bytes) {
-               unsigned iter_len = bvec_iter_len(bv, *iter);
-               unsigned len = min(bytes, iter_len);
+               const struct bio_vec *cur = bv + iter->bi_idx;
+               unsigned len = min3(bytes, iter->bi_size,
+                                   cur->bv_len - iter->bi_bvec_done);
 
                bytes -= len;
                iter->bi_size -= len;
                iter->bi_bvec_done += len;
 
-               if (iter->bi_bvec_done == __bvec_iter_bvec(bv, *iter)->bv_len) {
+               if (iter->bi_bvec_done == cur->bv_len) {
                        iter->bi_bvec_done = 0;
                        iter->bi_idx++;
                }