Jon’s recent Find the Time to First Byte Using Curl post reminded me about the additional timing details that cURL can provide.
cURL supports formatted output for the details of the request ( see the cURL manpage for details, under “-w, –write-out <format>” ). For our purposes we’ll focus just on the timing details that are provided.
Step one: create a new file, curl-format.txt, and paste in:
time_namelookup: %{time_namelookup}
time_connect: %{time_connect}
time_appconnect: %{time_appconnect}
time_pretransfer: %{time_pretransfer}
time_redirect: %{time_redirect}
time_starttransfer: %{time_starttransfer}
———
time_total: %{time_total}
Step two, make a request:
curl -w "@curl-format.txt" -o /dev/null -s http://wordpress.com/
What this does:
-
-w "@curl-format.txt"
tells cURL to use our format file -
-o /dev/null
redirects the output of the request to /dev/null -
-s
tells cURL not to show a progress meter -
http://wordpress.com/
is the URL we are requesting
And here is what you get back:
time_namelookup: 0.001
time_connect: 0.037
time_appconnect: 0.000
time_pretransfer: 0.037
time_redirect: 0.000
time_starttransfer: 0.092
———
time_total: 0.164
Jon was looking specifically at time to first byte, which is the time_starttransfer line. The other timing details include DNS lookup, TCP connect, pre-transfer negotiations, redirects (in this case there were none), and of course the total time.
The format file for this output provides a reasonable level of flexibility, for instance you could make it CSV formatted for easy parsing. You might want to do that if you were running this as a cron job to track timing details of a specific URL.
For details on the other information that cURL can provide using -w
check out the cURL manpage.
42 replies on “Timing Details With cURL”
[…] Timing Details With cURL | Joseph Scott […]
related and useful: http://stackoverflow.com/questions/10793332/php-curl-long-redirect-time-in-curl-getinfo
Thanks, very instructive, I wrapped this in a bash script:
https://github.com/mat/dotfiles/blob/master/bin/curlt
That is a nice way to to do it.
In Windows, curl-format.txt needs “n” instead of “n”.
[…] timing, like “time_namelookup”, “time_connect”, etc. Checking a post by Joseph, I remembered that curl supports formatted output. This way we can create a “template” […]
Superb. Thanks for writing this.
Re-posted at http://stackoverflow.com/a/22625150/1175496 with some advice for Windows
Nice. How do I also include the current date+time of when the curl initiated the HTTP request?
Adding a call to
date
to the shell script would give you that.@Joseph,
That’s what I did. But I was wondering if there was better way using just the curl command.
Thanks! 🙂
[…] está demás decir que durante la redacción de este post he visitado https://josephscott.org/archives/2011/10/timing-details-with-curl/ y he visto como añadir formateo a la información de cURL con más detalle y me he hecho uno […]
Can anybody tell me when ever I am sending a curl request (curl http://x.x.x.x/file_name) if I want to parse the size of the file then how to do?
I already trace the size from header.(i dump the header bu “-D” and redirect it).but it is taking so much time.
If the server includes the ‘Content-Length’ header in the response then that will give you the size of the file. If it is a large file and you don’t want to download the whole thing just to get the size, try making a HEAD request. Here is an example:
curl -v --compressed -I https://s1.wp.com/home.logged-out/css/homepage-one-field-long.css > /dev/null
The response includes “Content-Length: 35326”.
thank you sir,
but the response still contain 6-7 line of information including “200 OK”,”last modification”,”accept-range” ,”content-length” etc i just want 1 line information i.e “content-length : 23456 “.can it be possible?
regards
swa
cURL has several options, docs are at – http://curl.haxx.se/docs/
Here are all the pieces that just output the content length: curl –compressed -I -o /dev/null -s -D – https://abs.twimg.com/a/1424163092/css/t1/twitter_core.bundle.css|grep Content-Length
[…] were able run a controlled experiment in EC2 and gather response timing data via information returned by cURL . As we suspected, the response times from the load balancer versus its back-end instance were not […]
If in curl I want to extract an exact size data let curl http://x.x.x.x/ 1.doc 1234
where 1.doc is of size 30000 bytes and i want 1234 bytes from that file 1.doc. Whenever I will send the request the curl request will extract for 1234 byte data only. Is there any way to do this.
You could do that with a range request, but the server will need to support it. Check out –range in cURL.
Can we dump the header or extract the content value by using curl-loader.
I don’t know, I’ve never used curl-loader.
OK still thank you. Then is there any way to send multiple curl request in parallel . Multiple means not 10 or 100 or 1000 its like 100,000 request. I tried the method
curl http://x.x.x.x/{1.doc,2.doc,3.doc………} but it limits upto 10500.
You’d need to look at curl multi. Or a separate program that manages making all of the requests, via threads or forking. Or something else entirely, like Node.js / Go.
"n"
should be replaced by"n"
(tested in Linux).Thanks, not sure what ate the slashes.
What about setting timeout option?
From manpage:
–connect-timeout
-m, –max-time
In this case what I wanted was the time info about the request, I wasn’t worried about setting a timeout.
[…] previously talked about how to get timing details with cURL. On that post is a comment by Matthias Lüdtke linking to […]
Hello,
Need help!
Why is total_time not equal to (time_namelookup + time_connect + time_appconnect + time_pretransfer + time_redirect + time_starttransfer)?
Does total_time really count in milliseconds the request time from the beggining to the end {page loaded} ?
BR
Christopher
The cURL man page – http://curl.haxx.se/docs/manpage.html – describes total_time as:
While the time breakdown from cURL can be useful, it doesn’t breakdown all of the various times for a request. For instance WebPageTest will report times for:
– DNS lookup
– Initial connection
– TLS negotiation
– Time to first byte
– Content download
[…] 上面命令中的w参数表示指定输出格式,time_connect变量表示TCP握手的耗时,time_appconnect变量表示SSL握手的耗时(更多变量请查看文档和实例),s参数和o参数用来关闭标准输出。 […]
[…] Timing Analysis section. This uses curl’s –write-out option and was inspired by this post, this post, and my co-worker […]
Very Nice Post for Basic Understanding.
[…] 上面命令中的w参数表示指定输出格式,time_connect变量表示TCP握手的耗时,time_appconnect变量表示SSL握手的耗时(更多变量请查看文档和实例),s参数和o参数用来关闭标准输出。 […]
[…] /2011/10/14/timing-details-with-curl/ […]
Nice post.
Can you please help by specifying how to get these timing details while using libcurl in C/C++?
I don’t have any examples on that. I’d say curl_easy_getinfo() might be a good place to start though – https://curl.haxx.se/libcurl/c/curl_easy_getinfo.html
Thank you so much for this tip, this is awesome!
20 years using curl and never figured out about using a file to format the output.
Could someone explain the meaning of each parameter?
time_namelookup:
time_connect:
time_appconnect:
time_pretransfer:
time_redirect:
time_starttransfer:
time_total:
They are described in the cURL man page – https://curl.haxx.se/docs/manpage.html
How do we print out in newline in windows dos prompt ?
Almost – in later versions, you have to put “n” at the end of each line in the format file.