I wrote 300,000 single values to a binary file for testing. Reading the contents into a List(Of Single) took about 5 seconds on my system (low end Core 2 Duo).
Code:
Dim singleCollection As New List(Of Single)
Using reader As New BinaryReader(File.OpenRead("C:\Temp\BinaryFile.dat"))
Do Until reader.PeekChar() = -1
singleCollection.Add(reader.ReadSingle())
Loop
End Using