[Win32utils-devel] Playing with ReadFileScatter()

[Park Heesob]

Hi,
----- Original Message ----- 
From: "Daniel Berger" <djberg96 at gmail.com>
To: "win32utils-devel" <win32utils-devel at rubyforge.org>
Sent: Tuesday, October 09, 2007 8:27 PM
Subject: [Win32utils-devel] Playing with ReadFileScatter()


> Hi all,
> 
> Looking at the IO.readlines source in io.c, it looks to me like they 
> grab 8k chunks, split on the input record separator, and buffer accordingly.
> 
> Since it looks like ReadFileScatter() does some of that work 
> automatically (in page file sized chunks), I thought I'd give it a try. 
> Here's what I've got, but it doesn't work. I have an incorrect parameter 
> in the call to ReadFileScatter(). So, I've either got the size wrong, 
> bad alignment or I need to pass in a packed data structure of some sort.
> 
> Any ideas? BTW, you'll want to grab the latest windows-pr from CVS in 
> order to run this code.
> 
> Thanks,
> 
> Dan
> 
<snip> 
Here is a complete working source using ReadFileScatter.

Notice that VirtualAlloc shoud be redifined. Since ReadFileScatter requires page 
aligned memory buffer, I allocated buffer using VirtualAlloc. 


Regards,

Park Heesob

# WinIO.readlines

require 'windows/handle'
require 'windows/error'
require 'windows/system_info'
require 'windows/nio'
require 'windows/file'
require 'windows/synchronize'
require 'windows/msvcrt/io'
require 'windows/msvcrt/buffer'
require 'windows/memory'
      
class WinIO
   extend Windows::Error
   extend Windows::Handle
   extend Windows::NIO
   extend Windows::File
   extend Windows::Synchronize
   extend Windows::MSVCRT::IO
   extend Windows::MSVCRT::Buffer
   extend Windows::SystemInfo
   extend Windows::Memory
   
   include Windows::File
   include Windows::Memory
   
   PAGE_READWRITE   =   4  
   ERROR_IO_PENDING =   997
   MEM_RELEASE      =   0x8000     

   def self.readlines(file, sep = $INPUT_RECORD_SEPARATOR)
      handle = CreateFile(
         file,
         GENERIC_READ,
         FILE_SHARE_READ,
         nil,
         OPEN_EXISTING,
         FILE_FLAG_OVERLAPPED | FILE_FLAG_NO_BUFFERING,
         nil
      )

      if handle == INVALID_HANDLE_VALUE
         raise SystemCallError.new(GetLastError())
      end

      sysbuf = 0.chr * 40
      GetSystemInfo(sysbuf)
      page_size  = sysbuf[4,4].unpack('L')[0] # dwPageSize shoud be sysbuf[4,4] not sysbuf[8,4] 
      file_size  = File.size(file)
      page_num = (file_size*1.0 / page_size).ceil

      API.new('VirtualAlloc', 'LLLL', 'L')  # redefined VirtualAlloc

      base_address  = VirtualAlloc(nil, page_size * page_num, MEM_COMMIT, PAGE_READWRITE)
      buf_list = []
      for i in 0...page_num
        buf_list.push(base_address+page_size*i)
      end
      
      seg_array = buf_list.pack('Q*') + 0.chr * 8
      olapped = 0.chr * 20
      
      unless ReadFileScatter(handle, seg_array, page_size * page_num, nil, olapped)      
         error = GetLastError()
         if error != ERROR_IO_PENDING
           raise SystemCallError.new(error)
         end
      end
     
      sleep 0.01 unless HasOverlappedIoCompleted(olapped)

      unless CloseHandle(handle)
         raise SystemCallError.new(GetLastError())
      end

      buffer = 0.chr * file_size
      memcpy(buffer, buf_list[0], file_size)
      VirtualFree(base_address,0,MEM_RELEASE)
      
      buffer.split(sep).map {|x| x+"\n"}
   end  
end

a = IO.readlines('c:/work/java.txt')
b = WinIO.readlines('c:/work/java.txt',"\r\n")
p a==b