[DeStilleGast]

Hello HackForums people,
For a while I have seen many "Help me" topics and not much "challenges" so I got here a challenge for you guys and also maybe some usefull code

Here is a piece of code that gives a "OutOfMemory" error:

Button 1

Code

Dim openFile As New OpenFileDialog
        openFile.InitialDirectory = New DirectoryInfo(Application.StartupPath).FullName
        If openFile.ShowDialog = Windows.Forms.DialogResult.OK Then
            File.Create(openFile.FileName & ".gz").Dispose()
            File.WriteAllBytes(openFile.FileName & ".gz", Compress(File.ReadAllBytes(openFile.FileName).ToArray))

        End If

Compress function

Code

Public Function Compress(ByVal Raw() As Byte) As Byte()
    Using memory As MemoryStream = New MemoryStream()
        Using gzip As GZipStream = New GZipStream(memory, CompressionMode.Compress, True)
            gzip.Write(Raw, 0, Raw.Length)
        End Using
        Return memory.ToArray
    End Using
End Function

Trying to compress a big file (600MB) will result in this:


If done correctly it will result is this:

The ".pak" file is the original file.
The ".pak.gz" file is the compressed file.

THIS IS NOT A HELP REQUEST, I JUST WANT TO CHALLENGE SOME PEOPLE HERE
and also to give a small source for gzip (haven't seen it here yet)

If need I can provide a video with the error (with source) and I my solution (without showing the source).
Also I WILL release the source at some time.

[Multibyte]

The code you provided is working fine for me:

[Josh‎]

Compress function literally stolen from another site...

[DeStilleGast]

The code you provided is working fine for me:
<picture>

True, but it doesn't work work on BIG files

Compress function literally stolen from another site...

True, the code here is from an another site, I never said it was my code

[Dentmen]

i bet its memory problem try junk cleaner or sumthing
ram out of memory check your ram usage during this

[DeStilleGast]

i bet its memory problem try junk cleaner or sumthing
ram out of memory check your ram usage during this

Correct, but I got already the solution, its your task to fix it.
And if you can see, it is imposible to cleanup (with this sample) the memory.

I got 8GB ram on this computer. and there is minimal 2GB free while it is compessing 600MB. So its not the problem with my RAM.

[Shoсkwave]

Well, I don't really believe that this is a challenge. I believe you're just stuck and you're to proud to ask for help.

Getting help is fine and nothing to be embarrassed of.

Regardless here is my "solution" I split the file into 2mb chunks, compressed the chunk and appended it to the output file, very simple and only uses about 20k memory and ~12 CPU on my machine

Code

Imports System.IO
Imports System.IO.Compression

Module Module1

    Sub Main()
        CompressLargeFile("bigfile.dat", "bigfile.gz", 2048 * 1024)
    End Sub

    Private Sub CompressLargeFile(input As String, output As String, bufferSize As Integer)
        Dim FileSize As Long = New FileInfo(input).Length
        Dim ChunksRequired As Integer = FileSize / bufferSize
        Using FileStream As FileStream = New FileStream(input, FileMode.Open, FileAccess.Read)
            While FileStream.Position < FileStream.Length
                Dim Chunk As Byte() = New Byte(bufferSize) {}
                If FileStream.Length - FileStream.Position < Chunk.Length Then
                    Array.Resize(Chunk, FileStream.Length - FileStream.Position)
                End If
                FileStream.Read(Chunk, 0, Chunk.Length)
                If ChunksRequired < ChunksRequired Then
                    ChunksRequired += 1
                End If
                Chunk = Compress(Chunk)
                Dim AppendStream As New FileStream(output, FileMode.Append, FileAccess.Write)
                AppendStream.Write(Chunk, 0, Chunk.Length)
                AppendStream.Close()
            End While
        End Using
    End Sub

    Private Function Compress(Raw() As Byte) As Byte()
        Using memory As MemoryStream = New MemoryStream()
            Using gzip As GZipStream = New GZipStream(memory, CompressionMode.Compress, True)
                gzip.Write(Raw, 0, Raw.Length)
            End Using
            Return memory.ToArray
        End Using
    End Function

End Module

I think the file I compressed was already compressed as it's from GTA's game resources.



If you want I can comment the code, but it's pretty self explanitory.

[Odus]

Well, I don't really believe that this is a challenge. I believe you're just stuck and you're to proud to ask for help.

Getting help is fine and nothing to be embarrassed of.

I'm with this as well. Post your solution.

[DeStilleGast]

Well, I don't really believe that this is a challenge. I believe you're just stuck and you're to proud to ask for help.

Then you haven't read the whole post, I know a solution that works. You can blame me if it isn't, but I got already a working code:

Proof:

(Used another recorder since my WIP recorder didn't work correct)

And yes, I'm also a little bit curious how other people are using/fixing this.

---

I'm with this as well. Post your solution.

Aw but that ruins my challenge 😟

Code

Using toOpen As Stream = IO.File.OpenRead(Arg)
                            Using toWrite As Stream = IO.File.OpenWrite(Arg & ".gz")
                                Using gzip As GZipStream = New GZipStream(toWrite, CompressionMode.Compress, True)
                                    gzip.Write(GetStreamAsByteArray(toOpen), 0, toOpen.Length)
                                End Using
                            End Using
                        End Using
          GC.Collect()

Note: Arg is from the arguments (so I can proceed more the 1 files at once)

And to get the bytes from the Stream I got this function

Code

Private Function GetStreamAsByteArray(ByVal stream As System.IO.Stream) As Byte()
        Dim streamLength As Integer = Convert.ToInt32(stream.Length)
        Dim fileData As Byte() = New Byte(streamLength) {}

        stream.Read(fileData, 0, streamLength)

        Return fileData
    End Function

Maybe not the best code, but it works fine for me ATM

[Odus]

Aw but that ruins my challenge 😟

Most challenges actually come with the challenger's (right word?) solution. Those actually up for the challenge won't look and those who can't will try and improve using it.

I've seen people make the challenge without and the thread quickly fills up with posts saying "just make it a help thread". That's why I asked you to post your solution.

[DeStilleGast]

Most challenges actually come with the challenger's (right word?) solution. Those actually up for the challenge won't look and those who can't will try and improve using it.

I've seen people make the challenge without and the thread quickly fills up with posts saying "just make it a help thread". That's why I asked you to post your solution.

Kinda true what you said there.😵

[BahNahNah]

The error is self explanatory, you are trying to use too much memory.
You should be using a file stream, not a memory stream.