OutOfMemoryException. Don’t panic, your applications are ok, this is not a real exception. It will be more familiar to those people that have been working with I/O operations with heavy files and most of you would wonder “if I have 8GB RAM and I’m working with a 500MB file, how am I out of memory?”. Well, to know why we are getting this kind of exceptions, first we have to understand how the RAM memory works.
Let’s take a moment to think about an empty room of 25m2. We start filling it with furniture: a coach( 3m2) in front of the tv table (2m2) and next to a foot lamp (0.5m2), then a coffee table (2m2) and a confortable reclining chair at a side (1.5m2)… At this point, we have 16m2 of free space in our room, but what happens if we want to put a bed in it? Well, probably, besides having a lot of free space, the current placement of the rest of the furniture won’t allow us. This is a elementary example of how RAM memory works.
While our server is running, RAM slots are filled and emptied, and some holes will appear between the occupied memory slots. Our free memory is composed for all this holes and it’s in here while our application will try to allocate the 500MB file which we are working with. The problem is, most of the time we won’t have a continuous hole of 500MB, so fitting that file will be like fitting a bed in our living room. This is a concept known as fragmentation.
There is another problem while we are working with big files, so even when we can find a hole to fit our 500MB file, ¿what happens if our application has to work with 2 files at the same time? ¿and if it has to work with 100? We are not going to find 100 holes for 500MB files, that’s for sure. We will find this problem when we are creating a web application which is used by a lot of users at the same time, who need to upload or download big files.
The solution is simple: use a buffer. Instead of storing all the file in our RAM memory, we can try to temporarily store just a small part of it, a part that we can allocate easily in our free holes, working fail proof and taking less space of our RAM memory. If we create a 1MB buffer and we are working with 100 files at the same time, it will only take 100MB RAM and our application can easily find 100 holes of 1MB.
Enough chit chat, let’s get to work. This piece of code is written in C#, but can be transported to any other language, the important think here is the concept. You can download a complete example application, to debug or whatever, from my GitHub. In this example, we are reading a file and writting it to disk, but you can actually modify it to perform any other operation, simply changing the line 16.
public void ReadFile(HttpPostedFile inputFile)
//Buffer. PART_SIZE es una constante de, por ejemplo, 1MB (1048576 bytes)
byte buffer = new byte[PART_SIZE];
//Esta es la cantidad de bytes que vamos a leer, inicializada al tamaño del buffer
int readingSize = PART_SIZE;
//Creamos el fichero donde vamos a escribir
using (FileStream fileStream = File.Create(Config.outputFilePath))
//En cada iteración, lee 1MB del fichero y lo mete en el buffer
while (inputFile.InputStream.Read(buffer, 0, readingSize) > 0)
//En cada iteración, escribe el contenido del buffer en el fichero de salida
fileStream.Write(buffer, 0, readingSize);
//Si queda menos de 1MB por leer, ajustamos el "readingSize" para no pasarnos
if ((inputFile.InputStream.Length - inputFile.InputStream.Position) < PART_SIZE)
//Podemos hacer el cast sin miedo, porque sabemos que es menos de 1MB
readingSize = (int)(inputFile.InputStream.Length-inputFile.InputStream.Position);