IO-less Dirty Throttling.pdf

IO-less Dirty Throttling Fengguang Wu fengguang.wu@ LINUXCON JAPAN 2012 Write and writeback USER APPLICATION KERNEL FLUSHER THREAD ================== ===================== dirty pages and writeout pages after immediately return dirty expired (30s) avoid blocking apps aggregate write IO The flusher thread(s) Initiate writeback IO in background One flusher per storage device . . $ ps ax PID TTY STAT TIME COMMAND 2322 ? S 0:01 [flush-8:0] 12681 ? S 0:00 [flush-btrfs-1] Dirty throttling To limit dirty pages.. by slowing down heavy dirtiers. IO-full balance_dirty_pages() sys_write() balance_dirty_pages() i f (over_bground_thresh()) start_background_writeback(); i f (dirty exceeded) writeback_inodes(pages_dirtied * 3/2); Problem: disk seeks + lock contentions flusher task 1 task 2 task 3 *************** concurrently working on **************** ++ ++ ++ ++ | | | | | | | | | | | | | | | | | | | | | | | | | inode 1 | | inode 2 | | inode 3 | | inode 4 | | | | | | | | | | | | | | | | | || || || || ++ ++ ++ ++ disk seeks small IO lock contentions cache bouncing Problem: large latencies tasks in deep IO path are not killable hurts responsiveness hurts throughput (pipeline stalls) . . loop { READ from net WRITE to disk # long sleeps = idle network } rsync to a loaded server: pause time: 0-3s 0-300ms 60-70ms throughput: 1 +12.4% +63.5% 30s pause (NFS, 1-dd) NFS, 1 dd, 3G mem, 2.6.38-rc7 3s pause (xfs, 8-dd) xfs, 8 dd, 4G mem, 2.6.38-rc7 Let’s do controllable sleeps! I/O ? sleep i f (dirty exceeded) - writeback_inodes(pages_dirtied * 3/2); + sleep(pages_dirtied / task_ratelimit); and rethink dirty balancing …… Balancing dirty rate apps’ dirty progress disk write progress Balancing dirty pages global dirty limit global dirty pages global dirty setpoint per-disk dirty pages per-disk dirty setpoint IO-less dirty throttling/balancing Do proper sleep()s during sys_write(): .1 throttle each task’s dirty rate around write_bandwidth / N .2 balance the global dirty pages around global