The go standard library package net/http makes working with the HTTP clients and servers very easy. While most parts of the package are straightforward, some can cause nasty incidents in production if not used with care. In this article we explore common issues with the net/http package and how to avoid them.
Free resources after parsing multipart forms
Dealing with file uploads is really simple: Just call Request.FormFile("my_file") and close the file handler when done, right? Not so fast: Request.FormFile() calls Request.ParseMultipartForm(), which has a note in it's doc string that's really easy to miss:
[...]The whole request body is parsed and up to a total of maxMemory bytes of its file parts are stored in memory, with the remainder stored on disk in temporary files [...]
Temporary files don't magically disappear from disk when you close the file handle, which is why the mutlipart.Form type has a RemoveAll() function. When not called after working with multipart forms like file uploads, these temporary files will slowly fill up your disk until no more space is available, potentially bringing the whole production server down.
To clean up temporary files from your http handler, just defer a cleanup goroutine:
f, fh, err := req.FormFile("my_file")
defer f.Close()
defer func(){
if req.MultipartForm != nil{
req.MultipartForm.RemoveAll()
}
}()
...
Don't forget to add the nil check, as a parse error may leave the field with a nil value!
Beware of http.DefaultClient
When using functions like http.Get() or http.Do(), the net/http package uses the variable http.DefaultClient to make the request. When interacting with a single server, you will quickly exceed remote rate limits, because by default this client may create an unlimited amount of connections (and keep 90 of them around when idle). To prevent yourself from effectively sending a ddos attack to some remote host, you should create your own http.Client and set the fields MaxConnsPerHost and optionally MaxIdleConnsPerHost in it's http.Transport.
tr := http.DefaultTransport.(*http.Transport).Clone()
tr.MaxIdleConnsPerHost = 10
tr.MaxConnsPerHost = 10
client := http.Client{
Transport: tr,
};
We first clone the http.DefaultTransport to keep all the sane defaults of the net/http package, then we add our missing limits per host and create a custom http.Client using our adjusted transport. If we make more than 10 concurrent requests to the same host using this client, the first 10 will run immediately and all others will simply block until one of the first 10 completes, freeing a slot for a new request.
The myth of calling Request.Body.Close()
Many outdated tutorials will contain a warning to always close Request.Body after reading it from an http handler. This has been wrong for some time now, but the advice sticks like glue. To quote the documentation:
// The Server will close the request body. The ServeHTTP. Handler does not need to.
You do not need to add defer request.Body.Close() to your http handlers!