{"id":1696,"date":"2024-07-12T10:23:01","date_gmt":"2024-07-12T14:23:01","guid":{"rendered":"https:\/\/www.peteonsoftware.com\/?p=1696"},"modified":"2024-07-12T10:23:01","modified_gmt":"2024-07-12T14:23:01","slug":"core-tools-to-know-curl","status":"publish","type":"post","link":"https:\/\/www.peteonsoftware.com\/index.php\/2024\/07\/12\/core-tools-to-know-curl\/","title":{"rendered":"Core Tools to Know: curl"},"content":{"rendered":"

\"'curl'-yAs all bloggers eventually find out, two of the best reasons to write blog posts are either to document something for yourself for later or to force yourself to learn something well enough to explain it to others. That’s the impetus of this series that I plan on doing from time to time. I want to get more familiar with some of these core tools and also have a reference \/ resource available that is in “Pete Think” so I can quickly find what I need to use these tools if I forget. <\/p>\n

The first tool I want to tackle is curl<\/em>. In the world of command-line tools, curl<\/em> shines as a flexible utility for transferring data over various network protocols. Whether you’re working on the development side, the network admin side, or the security side, learning how to use curl<\/em> effectively can be incredibly beneficial. This blog post will guide you through the very basics of curl<\/em>, cover some common use cases, and explain when curl<\/em> might be a better choice than wget<\/em>.<\/p>\n

What is curl<\/em><\/h3>\n

curl<\/em> (Client for URL) is a command-line tool for transferring data with URLs. It supports a wide range of protocols including HTTP, HTTPS, FTP, SCP, TELNET, LDAP, IMAP, SMB, and many more. curl<\/em> is known for its flexibility and is widely used for interacting with APIs, downloading files, and testing network connections.<\/p>\n

Installing curl<\/em><\/h3>\n

Before diving into curl<\/em> commands, you need to ensure it is installed on your system. Lots of operating systems come with it. In fact, even Windows has shipped with curl<\/em> in Windows 10 and 11.<\/p>\n

For Linux:<\/strong><\/p>\n

\r\nsudo apt-get install curl  # Debian\/Ubuntu\r\nsudo yum install curl      # CentOS\/RHEL\r\n<\/pre>\n
For macOS:<\/strong><\/p>\n
\r\nbrew install curl\r\n<\/pre>\n
For Windows<\/strong>
\nYou can download the installer from the official curl website<\/a> if it isn’t already on your system.  To check, just type curl –help<\/em> at the command prompt and see if it understand the command.  If you get something back like this, you’re all set.<\/p>\n
\r\nC:\\Users\\peteonsoftware>curl --help\r\nUsage: curl [options...] <url>\r\n -d, --data <data>           HTTP POST data\r\n -f, --fail                  Fail fast with no output on HTTP errors\r\n -h, --help <category>       Get help for commands\r\n -i, --include               Include response headers in output\r\n -o, --output <file>         Write to file instead of stdout\r\n -O, --remote-name           Write output to file named as remote file\r\n -s, --silent                Silent mode\r\n -T, --upload-file <file>    Transfer local FILE to destination\r\n -u, --user <user:password>  Server user and password\r\n -A, --user-agent <name>     Send User-Agent <name> to server\r\n -v, --verbose               Make the operation more talkative\r\n -V, --version               Show version number and quit\r\n\r\nThis is not the full help, this menu is stripped into categories.\r\nUse \"--help category\" to get an overview of all categories.\r\nFor all options use the manual or \"--help all\".\r\n<\/pre>\n
The Most Simple Example<\/h3>\n
The simplest way to use curl<\/em> is to fetch the contents of a URL. Here is a basic example that will will print the HTML content of the specified URL to the terminal:<\/p>\n
\r\nc:\\\r\nλ curl https:\/\/hosthtml.live\r\n<!doctype html>\r\n<html data-adblockkey=\"MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBANDrp2lz7AOmADaN8tA50LsWcjLFyQFcb\/P2Txc58oYOeILb3vBw7J6f4pamkAQVSQuqYsKx3YzdUHCvbVZvFUsCAwEAAQ==_M6heeSY2n3p1IRsqfcIljkNrgqYXDBDFSWeybupIpyihjfHMZhFu8kniDL51hLxUnYHjgmcv2EYUtXfRDcRWZQ==\" lang=\"en\" style=\"background: #2B2B2B;\">\r\n<head>\r\n    <meta charset=\"utf-8\">\r\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\r\n    <link rel=\"icon\" href=\"data:image\/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAADElEQVQI12P4\/\/8\/AAX+Av7czFnnAAAAAElFTkSuQmCC\">\r\n    <link rel=\"preconnect\" href=\"https:\/\/www.google.com\" crossorigin>\r\n<\/head>\r\n<body>\r\n<div id=\"target\" style=\"opacity: 0\"><\/div>\r\n<script>window.park = \"eyJ1dWlkIjoiZDFhODUxY2ItOTUyZi00NGUyLTg4ZWMtMmU3ZGNhZmE1OTk0IiwicGFnZV90aW1lIjoxNzIwNzMyMzQxLCJwYWdlX3VybCI6Imh0dHBzOi8vaG9zdGh0bWwubGl2ZS8iLCJwYWdlX21ldGhvZCI6IkdFVCIsInBhZ2VfcmVxdWVzdCI6e30sInBhZ2VfaGVhZGVycyI6e30sImhvc3QiOiJob3N0aHRtbC5saXZlIiwiaXAiOiI3Mi4xMDQuMTY5LjE1NCJ9Cg==\";<\/script>\r\n<script src=\"\/bwjblpHBR.js\"><\/script>\r\n<\/body>\r\n<\/html>\r\n<\/pre>\n
Some Useful Examples to Actually Do Stuff<\/h3>\n
Downloading Files<\/strong><\/p>\n
\r\n# To download a file and save it with a specific name:\r\ncurl -o curlypigtail.jpg https:\/\/peteonsoftware.com\/images\/202407\/curlytail.jpg\r\n\r\n# If you want to save the file with the same name as in the URL:\r\ncurl -O https:\/\/peteonsoftware.com\/images\/202407\/curlytail.jpg\r\n<\/pre>\n
Sending HTTP Requests<\/strong><\/p>\n
\r\n# GET requests are used to retrieve data from a server. The basic example is already shown above. \r\n# To include headers in the output, use the -i option:\r\ncurl -i https:\/\/hosthtml.live\r\n\r\n# POST requests are used to send data to a server. \r\n# This is particularly useful when interacting with APIs. \r\ncurl -X POST -d \"param1=value1&param2=value2\" http:\/\/somereallygreat.net\/api\r\n\r\n# Many APIs now accept JSON.  This is how you'd send that\r\ncurl -X POST -H \"Content-Type: application\/json\" -d '{\"key1\":\"value1\", \"key2\":\"value2\"}' http:\/\/somereallygreat.net\/api\r\n\r\n# Without explaining it, we included a header above (-H).  That added a Content-Type header.  \r\n# To add an Auth header, you might do something like this\r\ncurl -H \"Authorization: Bearer token\" http:\/\/asitethatneedsbearertokens.com\r\n<\/pre>\n
Cookies<\/strong><\/p>\n
\r\n# To save cookies from a response\r\ncurl -c cookies.txt https:\/\/www.google.com\r\n\r\n# To send cookies with a request\r\ncurl -b cookies.txt https:\/\/www.google.com\r\n<\/pre>\n
When to Use curl<\/em> Over wget<\/em><\/h3>\n
While both curl<\/em> and wget<\/em> are used to transfer data over the internet, they have different strengths.  Daniel Stenberg is the creator of curl<\/em> (and also contributes to wget<\/em>) and he’s published a more lengthy comparison here<\/a>.  I defer to the expert, but here are some of my big takeaways.<\/p>\n
curl<\/em> Advantages<\/strong><\/p>\n