A comparison of two popular Python HTTP request libraries using the GitHub API
Image by author
Introduction
When requesting information from the web with an API call, one fundamental resource is Postman. There are, of course, other platforms on the market, as well as tools and programming languages that we can use to request information in an automated way. One of those ways is by using the requests library, which, along with Postman, are both totally free!
In this article, I will give you practical examples of using these tools, and then show you how to integrate them into your daily work as data analysts, data engineers, and general data enthusiasts.
But what are APIs at the end of the day, and what is Postman?
Regarding APIs and how they work, I wrote a little bit about them in this article:
[Data Extraction from APIs: From ETL to Visualization
heartbeat.comet.ml](https://heartbeat.comet.ml/data-extraction-from-apis-from-etl-to-visualization-9d774e40ce66 "heartbeat.comet.ml/data-extraction-from-api..")
So let’s talk about Postman!
What is Postman?
Postman is an API client that makes it easy for developers to create, share, test, and document APIs. This is done by allowing users to create and save simple and complex HTTP/s requests, as well as read, analyze and save their responses. These API responses are typically expressed in JSON format.
In one friendly platform, Postman gives you access to:
- An API repository, where you can easily store, catalog, and collaborate on all your API artifacts on one central platform;
- A comprehensive set of tools that help accelerate the API lifecycle — from design, testing, documentation, and mocking your APIs;
- Advanced intelligence and insights about all of your API operations by leveraging alerts and security warnings, search, reporting, and more;
- Workspaces, which help you organize your API work and collaborate across your organization or across the world;
- And last but not least, integrations, which allow you to integrate the most important tools in your software development pipeline to enable API-first practices.
Creating an account is very easy and straightforward, so you can start to make calls to the WorldWideWeb in no time!
Image from Postman Desktop App
Once you have completed this process and installed the Desktop App, you’ll get an environment like this one:
Image from Postman Desktop App
HTTP requests with Postman
In order to start making some simple calls to the web, we will use a free and well-known platform that has become many developers’ best friend; GitHub has an API that allows us to extract valuable information, so that we can integrate it into apps, do simple research, self-training, and much more. You can find the GitHub API documentation in this link.
Let’s make a request from one of its endpoints:
Image from Author
In Postman, we introduce the URL of the API endpoint in the GET field of the workspace, and click “Send,” producing the same response as in the browser in JSON format. Here we see the data corresponding to all GitHub users:
Image from Author
Once you’ve completed the previous API endpoint request, you might notice that the users listed correspond to the first GitHub users ever created, and are organized by their corresponding id. The first one is actually one of GitHub’s creators (Tom Preston- Werner, GitHub alias: mojombo), created back in 2008. If you want to read the full story check out his wikipage.
Image from GitHub
You can also save that response from the API for further analysis locally as a JSON-formatted file.
The previous call showed the body of the response, but the header of that response contains some other useful information about that request, such as the Server , the Date (when the call is performed), the Content-Type of that request, among other things:
Image from Author
There are 29 users in this API response — but, that doesn’t make much sense! Certainly there are a lot more GitHub users than that! To request more results, just add “query strings” (more abore APIs query strings and parameters in this link) at the end of the initial URL (e.g., ?since=109200990). Using this query string shows that there are a lot more users than what showed up in the initial GET request — over 100 million users more!
Image from Author
You could easily spend a lot of time querying all of GitHub user’s pages, but this won’t be very beneficial, because after a certain number of calls you will exceed the API rate limit:
https://docs.github.com/en/rest/search#rate-limit
Otherwise, you will receive a message like this one:
Image from Author
I can also access the information of my own GitHub repo using the API, by sending https://api.github.com/users/fvgm in the GET field.
[fvgm-spec — Overview
You can’t perform that action at this time. You signed in with another tab or window. You signed out in another tab or…github.com](https://github.com/fvgm-spec "github.com/fvgm-spec")
That request will generate a response corresponding to the information of my own GitHub repo. Note that my id field corresponds to the 60,470,633ᵗʰ user! Cool, isn’t it? You can also play around and get your own GitHub repo data:
Image from Author
Now let’s try to request some other information about a specific repository on GitHub:
Image from Author
This specific call makes a request for data from issues reported to the GitHub pandas repository.
Innovation and academia go hand-in-hand. Listen to our own CEO Gideon Mendels chat with the Stanford MLSys Seminar Series team about the future of MLOps and give the Comet platform a try for free!
Making HTTP Python Requests
In order to start performing the same calls with Requests, we will start importing the required packages in a Jupyter Notebook environment:
Image from Author
Note that you can find the full code in my GitHub repository.
Next, I instantiate some variables, an empty pandas DataFrame in which I’ll store the JSON, and a function to parse the DataFrame data:
image from Author
Next, I execute a for loop to iterate over the https://api.github.com/users endpoint (here, I did 10 iterations to make it simple and not exceed the API rate limit), then store the response in the empty pandas DataFrame:
Image from Author
When the data is stored in the DataFrame, we can now parse it using the function created above, and voila! — the JSON response will be formatted in tabular (flat) format.
Image from Author
From here, you can modify the loop going from the 0ᵗʰ page to 100,000ᵗʰ page, for example, while setting time.sleep() for 65 seconds after every 10 iterations in order to not exceed the API rate.
Image from Author
Inside the data extracted from every GitHub repo, you can find even more data to play around with and store in a DataFrame — and this may be useful for further analysis. Maybe you want to determine how many repos were created in a certain period of time, or even the different technologies and languages used across repos.
Conclusion
In this tutorial, you have learned how to use Postman to test API endpoints and quickly retrieve data from APIs. You also learned how this can be achieved similarly by using the Python requests library, but in a more automated way. Below, you can find some more free APIs to play with data, and maybe even build your own program to extract and analyze data!
[5 Awesome API for your Projects!
Here are some APIs that will take your projects to another level!medium.com](https://medium.com/codex/5-awesome-api-for-your-projects-cc61c795ac39 "medium.com/codex/5-awesome-api-for-your-pro..")
Hope this has been useful for you and happy coding!
Sources:
[How To Use the GitHub API - Build a Simple Web App - Code Snippet
GitHub is one of the best ways to collaborate on software and discover new projects using Git, the popular Version…codesnippet.io](https://codesnippet.io/github-api-tutorial/ "codesnippet.io/github-api-tutorial")
[Resources in the REST API - GitHub Docs
This describes the resources that make up the official GitHub REST API. If you have any problems or requests, please…docs.github.com](https://docs.github.com/en/rest/overview/resources-in-the-rest-api "docs.github.com/en/rest/overview/resources-..")
Editor’s Note: Heartbeat is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We’re committed to supporting and inspiring developers and engineers from all walks of life.
Editorially independent, Heartbeat is sponsored and published by Comet, an MLOps platform that enables data scientists & ML teams to track, compare, explain, & optimize their experiments. We pay our contributors, and we don’t sell ads.
If you’d like to contribute, head on over to our call for contributors. You can also sign up to receive our weekly newsletters (Deep Learning Weekly and the Comet Newsletter), join us on Slack, and follow Comet on Twitter and LinkedIn for resources, events, and much more that will help you build better ML models, faster.