Results 1 to 5 of 5

Thread: Multi-Threaded file copying?

  1. #1
    .NET Framework
    .NET 4.5
    Join Date
    Dec 2018
    Posts
    3
    Reputation
    0

    Cool Multi-Threaded file copying?

    Hello,

    Im a noob but am trying to learn. Ive found a way to make VB useful for a task ive been given to copy thousands of files to a slow SAN server daily. Right now i copy one file at a time. it works. painfully slow but it works.

    What id like to do is come up with some kind of stack where i can add filenames to copy to the stack. Call some code to spawn X number of threads and one by one it chews away at the stack. popping each file off the stack, copying it and then
    taking another off the stack until it's done then ending gracefully.

    Umm yeahhh. my code isnt even close. I got the multi-threading to work. But i cant seem to get control of one thread taking a file from the top and the second thread takes the next one down etc. The way i have it all three threads seem to grab the same top item in the stack. its a mess. Well any ideas on how i should be doing this? I appreciate the help. Thank you!

    Imports System


    Module Module1
    Public FileStack As New Stack(Of String)


    Sub FCT1()
    Try
    Console.WriteLine("FCT1: " & FileStack.Peek)
    FileStack.Pop()
    Catch ex As Exception
    Console.WriteLine("Error: FCT1")
    End Try
    'Do actual file copy to SAN
    End Sub
    Sub FCT2()
    Try
    Console.WriteLine("FCT2: " & FileStack.Peek)
    FileStack.Pop()
    Catch ex As Exception
    Console.WriteLine("Error: FCT2")
    End Try
    'Do actual file copy to SAN
    End Sub
    Sub FCT3()
    Try
    Console.WriteLine("FCT3: " & FileStack.Peek)
    FileStack.Pop()
    Catch ex As Exception
    Console.WriteLine("Error: FCT3")
    End Try
    'Do actual file copy to SAN
    End Sub


    Sub Main()
    FileStack.Push("1.PDF")
    FileStack.Push("2.PDF")
    FileStack.Push("3.PDF")
    FileStack.Push("4.PDF")
    FileStack.Push("5.PDF")


    If FileStack.Count > 0 Then
    Console.WriteLine("Copying " & FileStack.Count.ToString & " files...")
    Else
    Console.WriteLine("No files to copy.")
    End If


    Do While FileStack.Count > 0
    Dim A As System.Threading.Thread = New Threading.Thread(AddressOf FCT1)
    Dim B As System.Threading.Thread = New Threading.Thread(AddressOf FCT2)
    Dim C As System.Threading.Thread = New Threading.Thread(AddressOf FCT3)
    A.Name = "File_Copier_Thread_1"
    B.Name = "File_Copier_Thread_2"
    C.Name = "File_Copier_Thread_3"
    A.Start()
    B.Start()
    C.Start()


    Loop


    Console.ReadLine()
    End Sub
    End Module

  2. #2
    .NET Framework
    .NET 4.0
    Join Date
    Aug 2004
    Location
    Sydney, Australia
    Posts
    14,010
    Reputation
    1731
    These days, you should use Tasks. They will be executed on ThreadPool threads and that will manage the number of concurrent threads. E.g.

    Private Sub CopyFiles(filePaths As IEnumerable(Of String))
    Dim tasks As New List(Of Task)

    For Each filePath In filePaths
    tasks.Add(Task.Factory.StartNew(Sub() File.Copy(CStr(filePath), "destination path here")))
    Next

    Task.WaitAll(tasks.ToArray())
    End Sub

  3. #3
    .NET Framework
    .NET 4.5
    Join Date
    Dec 2018
    Posts
    3
    Reputation
    0
    Hello jmcilhinney,

    Ok that works so much better than the way i tried. lol. A few questions if i may? I tried to read up on tasks first. I liked the way you used a Lambda expression when adding the task.

    1. It sounds like all of the tasks run in one thread pool. i.e. does that mean they all run on one cpu? Or would they be distributed against all logical cores?
    2. Is there an upper limit / practical limit on the numer of tasks one can add? i.e. say i have 100,000 files to copy.
    should I:
    A- Add all files in that iEnumerable list and pass it in to the subroutine?
    B- Only pass in say 100 at a time, wait for them to finish, then take another chunk of files?

    What are your thoughts on this?

    Thanks

  4. #4
    .NET Framework
    .NET 4.0
    Join Date
    Aug 2004
    Location
    Sydney, Australia
    Posts
    14,010
    Reputation
    1731
    Quote Originally Posted by MNVBGuy View Post
    1. It sounds like all of the tasks run in one thread pool. i.e. does that mean they all run on one cpu? Or would they be distributed against all logical cores?
    You should do some reading on the ThreadPool class for more specific information because I don't actually know the answer to that question, but I would assume that all cores would be utilised. I'm not sure that that would even be up to .NET but rather the domain of the OS, but I'm not really sure. The ThreadPool always has at least a certain number of threads available to do work and will create more if the number of work items queued grows too large. It will continue to create new threads as needed up to an upper limit. Once the work item queue drops to a certain level, the ThreadPool will start terminating unused threads after a while. I don't even know whether a single thread is executed on the same CPU core all the time. I'm guessing that it is most efficient to do so but, if another core becomes available, I would guess that threads waiting on a busy core would be executed on that available core. Again, I would guess that that is up to the OS to manage rather than .NET or the ThreadPool itself, but I could be wrong about that.
    Quote Originally Posted by MNVBGuy View Post
    2. Is there an upper limit / practical limit on the numer of tasks one can add? i.e. say i have 100,000 files to copy.
    should I:
    A- Add all files in that iEnumerable list and pass it in to the subroutine?
    B- Only pass in say 100 at a time, wait for them to finish, then take another chunk of files?
    A Task is an object so I would expect that any relevant limits would be the same that apply to any other type of object. If you use the ThreadPool directly you call QueueUserWorkItem and pass a delegate and I would imagine that that's what happens inside a Task too, amongst other things. Other than available memory and the like, I would think that the only limits on that would be that the queue cannot have more than Integer.MaxValue items, which is way bigger than 100,000.

  5. #5
    .NET Framework
    .NET 4.5
    Join Date
    Dec 2018
    Posts
    3
    Reputation
    0
    You rock. Thank you for pointing me in the right direction. I hope to be able to help others on this forum once i get better myself. I'll do more reading on that and some testing and see how it goes. Thank you!!

Similar Threads

  1. Debugging a multi-threaded vb2005 app
    By bamasmith in forum Testing, Quality Control, Debugging
    Replies: 2
    Last Post: 12-07-2009, 4:59 PM
  2. File IO in Multi-threaded Environment
    By KeysCoder in forum VB.NET General Discussion
    Replies: 5
    Last Post: 11-04-2008, 8:00 AM
  3. Multi-threaded TCP redirection (proxy)
    By Berry-tan in forum Net / Sockets
    Replies: 1
    Last Post: 04-23-2007, 3:32 PM
  4. copying a file
    By RatDan in forum VB.NET General Discussion
    Replies: 5
    Last Post: 10-19-2006, 7:45 PM
  5. Replies: 0
    Last Post: 04-12-2006, 5:15 PM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •