Timing Details With cURL

Jon’s recent Find the Time to First Byte Using Curl post reminded me about the additional timing details that cURL can provide.

cURL supports formatted output for the details of the request ( see the cURL manpage for details, under “-w, –write-out <format>” ). For our purposes we’ll focus just on the timing details that are provided.

Step one: create a new file, curl-format.txt, and paste in:

n
            time_namelookup:  %{time_namelookup}n
               time_connect:  %{time_connect}n
            time_appconnect:  %{time_appconnect}n
           time_pretransfer:  %{time_pretransfer}n
              time_redirect:  %{time_redirect}n
         time_starttransfer:  %{time_starttransfer}n
                            ----------n
                 time_total:  %{time_total}n
n

Step two, make a request:

curl -w "@curl-format.txt" -o /dev/null -s http://wordpress.com/

What this does:

  • -w "@curl-format.txt" tells cURL to use our format file
  • -o /dev/null redirects the output of the request to /dev/null
  • -s tells cURL not to show a progress meter
  • http://wordpress.com/ is the URL we are requesting

And here is what you get back:

            time_namelookup:  0.001
               time_connect:  0.037
            time_appconnect:  0.000
           time_pretransfer:  0.037
              time_redirect:  0.000
         time_starttransfer:  0.092
                            ----------
                 time_total:  0.164

Jon was looking specifically at time to first byte, which is the time_starttransfer line. The other timing details include DNS lookup, TCP connect, pre-transfer negotiations, redirects (in this case there were none), and of course the total time.

The format file for this output provides a reasonable level of flexibility, for instance you could make it CSV formatted for easy parsing. You might want to do that if you were running this as a cron job to track timing details of a specific URL.

For details on the other information that cURL can provide using -w check out the cURL manpage.

32 Comments

  1. That is a nice way to to do it.

  2. In Windows, curl-format.txt needs “n” instead of “n”.

  3. Superb. Thanks for writing this.

  4. Nice. How do I also include the current date+time of when the curl initiated the HTTP request?

  5. Adding a call to date to the shell script would give you that.

  6. @Joseph,

    That’s what I did. But I was wondering if there was better way using just the curl command.

    Thanks! 🙂

  7. Can anybody tell me when ever I am sending a curl request (curl http://x.x.x.x/file_name) if I want to parse the size of the file then how to do?
    I already trace the size from header.(i dump the header bu “-D” and redirect it).but it is taking so much time.

  8. If the server includes the ‘Content-Length’ header in the response then that will give you the size of the file. If it is a large file and you don’t want to download the whole thing just to get the size, try making a HEAD request. Here is an example:


    curl -v --compressed -I https://s1.wp.com/home.logged-out/css/homepage-one-field-long.css > /dev/null

    The response includes “Content-Length: 35326”.

  9. thank you sir,
    but the response still contain 6-7 line of information including “200 OK”,”last modification”,”accept-range” ,”content-length” etc i just want 1 line information i.e “content-length : 23456 “.can it be possible?

    regards
    swa

  10. cURL has several options, docs are at – http://curl.haxx.se/docs/

    Here are all the pieces that just output the content length: curl –compressed -I -o /dev/null -s -D – https://abs.twimg.com/a/1424163092/css/t1/twitter_core.bundle.css|grep Content-Length

  11. If in curl I want to extract an exact size data let curl http://x.x.x.x/ 1.doc 1234
    where 1.doc is of size 30000 bytes and i want 1234 bytes from that file 1.doc. Whenever I will send the request the curl request will extract for 1234 byte data only. Is there any way to do this.

  12. You could do that with a range request, but the server will need to support it. Check out –range in cURL.

  13. Can we dump the header or extract the content value by using curl-loader.

  14. I don’t know, I’ve never used curl-loader.

  15. OK still thank you. Then is there any way to send multiple curl request in parallel . Multiple means not 10 or 100 or 1000 its like 100,000 request. I tried the method
    curl http://x.x.x.x/{1.doc,2.doc,3.doc………} but it limits upto 10500.

  16. You’d need to look at curl multi. Or a separate program that manages making all of the requests, via threads or forking. Or something else entirely, like Node.js / Go.

  17. "n" should be replaced by "n" (tested in Linux).

  18. Thanks, not sure what ate the slashes.

  19. What about setting timeout option?
    From manpage:
    –connect-timeout
    -m, –max-time

  20. In this case what I wanted was the time info about the request, I wasn’t worried about setting a timeout.

  21. Hello,

    Need help!

    Why is total_time not equal to (time_namelookup + time_connect + time_appconnect + time_pretransfer + time_redirect + time_starttransfer)?

    Does total_time really count in milliseconds the request time from the beggining to the end {page loaded} ?

    BR
    Christopher

  22. The cURL man page – http://curl.haxx.se/docs/manpage.html – describes total_time as:

    time_total The total time, in seconds, that the full operation lasted. The time will be displayed with millisecond resolution.

    While the time breakdown from cURL can be useful, it doesn’t breakdown all of the various times for a request. For instance WebPageTest will report times for:

    – DNS lookup
    – Initial connection
    – TLS negotiation
    – Time to first byte
    – Content download

  23. Mikita Prajapati

    Tue 24 May 2016 at 12:47 am

    Very Nice Post for Basic Understanding.

  24. Nice post.
    Can you please help by specifying how to get these timing details while using libcurl in C/C++?

  25. josephscott

    Tue 6 Dec 2016 at 9:03 am

    I don’t have any examples on that. I’d say curl_easy_getinfo() might be a good place to start though – https://curl.haxx.se/libcurl/c/curl_easy_getinfo.html

  26. Thank you so much for this tip, this is awesome!
    20 years using curl and never figured out about using a file to format the output.

  27. Could someone explain the meaning of each parameter?

    time_namelookup:
    time_connect:
    time_appconnect:
    time_pretransfer:
    time_redirect:
    time_starttransfer:
    time_total:

  28. josephscott

    Sun 26 Mar 2017 at 9:34 pm

    They are described in the cURL man page – https://curl.haxx.se/docs/manpage.html

  29. How do we print out in newline in windows dos prompt ?

Leave a Reply

Your email address will not be published. Required fields are marked *

© 2018 Joseph Scott

Theme by Anders NorénUp ↑