[Win32utils-devel] Playing with ReadFileScatter()

Berger, Daniel Daniel.Berger at qwest.com
Tue Oct 9 12:09:26 EDT 2007


> -----Original Message-----
> From: win32utils-devel-bounces at rubyforge.org 
> [mailto:win32utils-devel-bounces at rubyforge.org] On Behalf Of 
> Park Heesob
> Sent: Tuesday, October 09, 2007 8:57 AM
> To: Development and ideas for win32utils projects
> Subject: Re: [Win32utils-devel] Playing with ReadFileScatter()

<snip>

> Here is a complete working source using ReadFileScatter.
> 
> Notice that VirtualAlloc shoud be redifined. Since 
> ReadFileScatter requires page 
> aligned memory buffer, I allocated buffer using VirtualAlloc. 

<snip>

Thanks! That works.

I was curious about performance, so I did some benchmarks. It seems to
get dramatically worse as the file size gets larger. Is this to be
expected?

Here's a benchmark program I created. The 5 files end up at the
following size (in bytes):

      550 file1.txt
    5,590 file2.txt
   56,890 file3.txt
  578,890 file4.txt
5,888,890 file5.txt

A little profiling indicates that the worst of it is the buffer.split &
map at the end. Here it is with split and map:

                                         user     system      total
real
IO.readlines file1                   0.000000   0.000000   0.000000 (
0.000000)
IO.readlines file2                   0.010000   0.000000   0.010000 (
0.010000)
IO.readlines file3                   0.030000   0.000000   0.030000 (
0.030000)
IO.readlines file4                   0.891000   0.020000   0.911000 (
1.111000)
IO.readlines file5                   0.922000   0.020000   0.942000 (
0.942000)
WinIO.readlines file1                0.000000   0.010000   0.010000 (
0.010000)
WinIO.readlines file2                0.000000   0.000000   0.000000 (
0.130000)
WinIO.readlines file3                0.120000   0.030000   0.150000 (
0.280000)
WinIO.readlines file4                1.522000   0.040000   1.562000 (
2.824000)
WinIO.readlines file5               12.448000   0.431000  12.879000 (
23.514000)

Here it is if we remove the split and map. Much better, but still slow.

                                         user     system      total
real
IO.readlines file1                   0.000000   0.000000   0.000000 (
0.000000)
IO.readlines file2                   0.000000   0.010000   0.010000 (
0.010000)
IO.readlines file3                   0.030000   0.010000   0.040000 (
0.050000)
IO.readlines file4                   0.811000   0.040000   0.851000 (
1.042000)
IO.readlines file5                   0.902000   0.020000   0.922000 (
1.031000)
WinIO.readlines file1                0.010000   0.000000   0.010000 (
0.010000)
WinIO.readlines file2                0.000000   0.010000   0.010000 (
0.110000)
WinIO.readlines file3                0.060000   0.010000   0.070000 (
0.200000)
WinIO.readlines file4                0.420000   0.030000   0.450000 (
1.743000)
WinIO.readlines file5                4.427000   0.140000   4.567000 (
13.980000)

# winio_bench.rb
$:.unshift Dir.pwd
require 'benchmark'
require 'winio'

fh1 = File.open("file1.txt", "w")
fh2 = File.open("file2.txt", "w")
fh3 = File.open("file3.txt", "w")
fh4 = File.open("file4.txt", "w")
fh5 = File.open("file5.txt", "w")

s = "The quick brown fox jumped over the lazy dog's back "

10.times{ |n| fh1.puts s + n.to_s }
fh1.close
puts "File 1 created"

100.times{ |n| fh2.puts s + n.to_s }
fh2.close
puts "File 2 created"

1000.times{ |n| fh3.puts s + n.to_s }
fh3.close
puts "File 3 created"

10000.times{ |n| fh4.puts s + n.to_s }
fh4.close
puts "File 4 created"

100000.times{ |n| fh5.puts s + n.to_s }
fh5.close
puts "File 4 created"

MAX = 10

Benchmark.bm(35) do |x|
   x.report("IO.readlines file1"){
      MAX.times{ IO.readlines('file1.txt') }
   }
   x.report("IO.readlines file2"){
      MAX.times{ IO.readlines('file2.txt') }
   }
   x.report("IO.readlines file3"){
      MAX.times{ IO.readlines('file3.txt') }
   }
   x.report("IO.readlines file4"){
      MAX.times{ IO.readlines('file4.txt') }
   }
   x.report("IO.readlines file5"){
      MAX.times{ IO.readlines('file4.txt') }
   }
   x.report("WinIO.readlines file1"){
      MAX.times{ WinIO.readlines('file1.txt') }
   }
   x.report("WinIO.readlines file2"){
      MAX.times{ WinIO.readlines('file2.txt') }
   }
   x.report("WinIO.readlines file3"){
      MAX.times{ WinIO.readlines('file3.txt') }
   }
   x.report("WinIO.readlines file4"){
      MAX.times{ WinIO.readlines('file4.txt') }
   }
   x.report("WinIO.readlines file5"){
      MAX.times{ WinIO.readlines('file5.txt') }
   }
end

Regards,

Dan


This communication is the property of Qwest and may contain confidential or
privileged information. Unauthorized use of this communication is strictly 
prohibited and may be unlawful.  If you have received this communication 
in error, please immediately notify the sender by reply e-mail and destroy 
all copies of the communication and any attachments.


More information about the win32utils-devel mailing list