Saturday 13th April 2013 12.20am

As part of my blue sky non work side project, I've had to write a large scale batch asynchronous file i/o engine, which I've based on Boost.ASIO. Some quick benchmarks:

For 1000 file open, write one byte, fsync + close, then delete:

Windows Overlapped IOCP backend on Windows 7 x64, 2.4Ghz Sandy Bridge on 256Gb Samsung 840 Pro SSD:

It took 0.010001 secs to dispatch all operations
It took 0.375038 secs to finish all operations

It took 0.154015 secs to do 6492.86 file opens per sec
   It took 0.144014 secs to synchronise file opens
It took 0.0130013 secs to do 76915.4 file writes per sec
It took 0.0860086 secs to do 11626.7 file closes per sec
It took 0.132013 secs to do 7575 file deletions per sec

POSIX compat backend on Windows 7 x64, 2.4Ghz Sandy Bridge on 256Gb Samsung 840 Pro SSD:

It took 0.010001 secs to dispatch all operations
It took 0.39804 secs to finish all operations

It took 0.135014 secs to do 7406.67 file opens per sec
   It took 0.125013 secs to synchronise file opens
It took 0.0490049 secs to do 20406.1 file writes per sec
It took 0.0920092 secs to do 10868.5 file closes per sec
It took 0.132013 secs to do 7575 file deletions per sec


I'm more than pleased with these results. Need to freshen up the Linux port as it's been a bit neglected for a while now, though given I haven't implemented a native linux aio backend I'd assume it'd be little faster than the POSIX compat backend on Windows. One huge problem with Linux which no other POSIX implementation has is the need to fsync the containing directory after file creation - and fsyncs are not quick.

Also, I really must get round to porting it to ARM NEON. It uses a lot of 128 bit and 256 bit vectors, currently only have SSE2 and generic implementations :)