multithreading - Processing multiple files concurrently in c++ -
i working on vc6 project run on multicore processor pc. require process large number of files. going use multiple threads process them. need experts' advice go ahead. knowledge case of data parallelism. plot this.
description of files :=
file structure same names arbitrary.
size of around 100 kb each.
file number few hundreds few thousands.
each file processed same way.
for each file,
i read hdd -> process -> write hdd
saving may in same file or may different folder same name ( not decided ignore )
i thinking use multiple threads process files. 1 file processed per core. (i know file processing,thread creation, getting number of cores)
doubt 1.
now given 1500 files , 2/4/8 cores how should divide files (appox.) equally among multiple threads, each file gets processed once.
doubt 2. have 1 hdd i/o how many threads create.
one thread doing both input , output or 2 threads, 1 reading , 1 writing.
thank in advance kanade
unless data processing slow, limiting factor here disk access, assuming files on same physical disk , you're not using ssds, wonder if multitasking @ all.
besides, if try process multiple files in parallel, generate lot of random disk accesses, slower accessing files sequentially.
trying read , write in parallel cause same problem on hard disk. hd's heads go , forth between read , written files crazy, dramatically reducing effective disk bandwidth.
my advice have 1 thread per physical disk, doing first read , writes. can still have worker threads process data in memory though.
Comments
Post a Comment