Jon’s recent Find the Time to First Byte Using Curl post reminded me about the additional timing details that cURL can provide.
cURL supports formatted output for the details of the request ( see the cURL manpage for details, under “-w, –write-out <format>” ). For our purposes we’ll focus just on the timing details that are provided.
Step one: create a new file, curl-format.txt, and paste in:
n
time_namelookup: %{time_namelookup}n
time_connect: %{time_connect}n
time_appconnect: %{time_appconnect}n
time_pretransfer: %{time_pretransfer}n
time_redirect: %{time_redirect}n
time_starttransfer: %{time_starttransfer}n
----------n
time_total: %{time_total}n
n
Step two, make a request:
curl -w "@curl-format.txt" -o /dev/null -s http://wordpress.com/
What this does:
-w "@curl-format.txt"tells cURL to use our format file-o /dev/nullredirects the output of the request to /dev/null-stells cURL not to show a progress meterhttp://wordpress.com/is the URL we are requesting
And here is what you get back:
time_namelookup: 0.001
time_connect: 0.037
time_appconnect: 0.000
time_pretransfer: 0.037
time_redirect: 0.000
time_starttransfer: 0.092
----------
time_total: 0.164
Jon was looking specifically at time to first byte, which is the time_starttransfer line. The other timing details include DNS lookup, TCP connect, pre-transfer negotiations, redirects (in this case there were none), and of course the total time.
The format file for this output provides a reasonable level of flexibility, for instance you could make it CSV formatted for easy parsing. You might want to do that if you were running this as a cron job to track timing details of a specific URL.
For details on the other information that cURL can provide using -w check out the cURL manpage.
Mon 4 Jun 2012 at 2:05 pm
related and useful: http://stackoverflow.com/questions/10793332/php-curl-long-redirect-time-in-curl-getinfo
Fri 29 Mar 2013 at 7:42 am
Thanks, very instructive, I wrapped this in a bash script:
https://github.com/mat/dotfiles/blob/master/bin/curlt
Fri 29 Mar 2013 at 10:25 am
That is a nice way to to do it.
Fri 24 Jan 2014 at 2:56 am
In Windows, curl-format.txt needs “n” instead of “n”.
Fri 27 Jun 2014 at 8:39 am
Superb. Thanks for writing this.
Tue 26 Aug 2014 at 5:46 pm
Re-posted at http://stackoverflow.com/a/22625150/1175496 with some advice for Windows
Sat 13 Dec 2014 at 4:27 pm
Nice. How do I also include the current date+time of when the curl initiated the HTTP request?
Mon 15 Dec 2014 at 4:26 pm
Adding a call to
dateto the shell script would give you that.Mon 15 Dec 2014 at 6:23 pm
@Joseph,
That’s what I did. But I was wondering if there was better way using just the curl command.
Thanks! 🙂
Mon 16 Feb 2015 at 5:21 am
Can anybody tell me when ever I am sending a curl request (curl http://x.x.x.x/file_name) if I want to parse the size of the file then how to do?
I already trace the size from header.(i dump the header bu “-D” and redirect it).but it is taking so much time.
Mon 16 Feb 2015 at 4:27 pm
If the server includes the ‘Content-Length’ header in the response then that will give you the size of the file. If it is a large file and you don’t want to download the whole thing just to get the size, try making a HEAD request. Here is an example:
curl -v --compressed -I https://s1.wp.com/home.logged-out/css/homepage-one-field-long.css > /dev/null
The response includes “Content-Length: 35326”.
Fri 20 Feb 2015 at 7:53 pm
thank you sir,
but the response still contain 6-7 line of information including “200 OK”,”last modification”,”accept-range” ,”content-length” etc i just want 1 line information i.e “content-length : 23456 “.can it be possible?
regards
swa
Fri 20 Feb 2015 at 8:09 pm
cURL has several options, docs are at – http://curl.haxx.se/docs/
Here are all the pieces that just output the content length: curl –compressed -I -o /dev/null -s -D – https://abs.twimg.com/a/1424163092/css/t1/twitter_core.bundle.css|grep Content-Length
Tue 24 Feb 2015 at 7:18 am
If in curl I want to extract an exact size data let curl http://x.x.x.x/ 1.doc 1234
where 1.doc is of size 30000 bytes and i want 1234 bytes from that file 1.doc. Whenever I will send the request the curl request will extract for 1234 byte data only. Is there any way to do this.
Tue 24 Feb 2015 at 3:28 pm
You could do that with a range request, but the server will need to support it. Check out –range in cURL.
Thu 26 Feb 2015 at 11:48 am
Can we dump the header or extract the content value by using curl-loader.
Thu 26 Feb 2015 at 4:32 pm
I don’t know, I’ve never used curl-loader.
Fri 27 Feb 2015 at 6:30 am
OK still thank you. Then is there any way to send multiple curl request in parallel . Multiple means not 10 or 100 or 1000 its like 100,000 request. I tried the method
curl http://x.x.x.x/{1.doc,2.doc,3.doc………} but it limits upto 10500.
Fri 27 Feb 2015 at 4:02 pm
You’d need to look at curl multi. Or a separate program that manages making all of the requests, via threads or forking. Or something else entirely, like Node.js / Go.
Tue 31 Mar 2015 at 11:21 pm
"n"should be replaced by"n"(tested in Linux).Tue 31 Mar 2015 at 11:24 pm
Thanks, not sure what ate the slashes.
Wed 1 Apr 2015 at 7:24 am
What about setting timeout option?
From manpage:
–connect-timeout
-m, –max-time
Wed 1 Apr 2015 at 3:30 pm
In this case what I wanted was the time info about the request, I wasn’t worried about setting a timeout.
Thu 3 Dec 2015 at 3:56 am
Hello,
Need help!
Why is total_time not equal to (time_namelookup + time_connect + time_appconnect + time_pretransfer + time_redirect + time_starttransfer)?
Does total_time really count in milliseconds the request time from the beggining to the end {page loaded} ?
BR
Christopher
Thu 3 Dec 2015 at 9:00 am
The cURL man page – http://curl.haxx.se/docs/manpage.html – describes total_time as:
While the time breakdown from cURL can be useful, it doesn’t breakdown all of the various times for a request. For instance WebPageTest will report times for:
– DNS lookup
– Initial connection
– TLS negotiation
– Time to first byte
– Content download
Tue 24 May 2016 at 12:47 am
Very Nice Post for Basic Understanding.
Tue 6 Dec 2016 at 4:06 am
Nice post.
Can you please help by specifying how to get these timing details while using libcurl in C/C++?
Tue 6 Dec 2016 at 9:03 am
I don’t have any examples on that. I’d say curl_easy_getinfo() might be a good place to start though – https://curl.haxx.se/libcurl/c/curl_easy_getinfo.html
Thu 9 Feb 2017 at 3:38 am
Thank you so much for this tip, this is awesome!
20 years using curl and never figured out about using a file to format the output.
Fri 24 Mar 2017 at 12:08 am
Could someone explain the meaning of each parameter?
time_namelookup:
time_connect:
time_appconnect:
time_pretransfer:
time_redirect:
time_starttransfer:
time_total:
Sun 26 Mar 2017 at 9:34 pm
They are described in the cURL man page – https://curl.haxx.se/docs/manpage.html
Sun 8 Apr 2018 at 10:40 pm
How do we print out in newline in windows dos prompt ?