Introduction

Hey there!!
Last time we created a REST API that scrapped pinned repos from a user's GitHub Profile. This time we will add 3 new features to our app.

1. Get Social Preview
2. Get pinned repo's Github API data
3. UI at root route

All of these are standalone features so if you wish to skip any or all of them then, feel free to do so. That being said, I encourage you all to try experimenting and adding your features to this app.

Utility function

Before we move forward, we would create a new utility function for later use.
Create a file stringToBoolean.js in utils folder
OR
run the following command:

echo "" > util/stringToBoolean.js # for windows terminals

touch util/stringToBoolean.js # for linux terminals

stringToBoolean()

Inside stringToBoolean.js create a function with the same name that takes one parameter str of type string. We will use a ternary operator to return true if the string matches "true" otherwise it returns false.

const stringToBoolean = (str) => {
    return str === "true" ? true : false;
};

module.exports = stringToBoolean;

Get Social Preview

So far we were only getting textual data from our REST API but, wouldn't it be cool if we could get an image of our app as well. We will scrape social preview from repo's <meta> and return it with other repo details.

Create a function getRepoImage() that takes one parameter repos of type array. Call this function inside try block of getPinnedRepos() just above the return statement and pass repos as argument. We will await for its returned values and update our repos variable with these new values.

Now, coming back inside our getRepoImage(), we will insert this snippet:

const requests = repos.map(async (repo) => {
  const { data } = await axios.get(repo.url);
  const $ = cheerio.load(data);
  const repoImage = $("meta[property='og:image']").attr("content");
  return {
    ...repo,
    image: repoImage,
  };
});

return Promise.all(requests);

and understand what is going on here.

1. We are looping through our repos array that already has pinned repo's name, url and description properties and making get calls to each url.
2. Just like last time, we use cheerio.load() to convert the DOM string (received from get call) to actual DOM.
3. Then, we get the url of the image by first, accessing a <meta> that has an attribute property with value 'og:image' and then, extracting the value of its content attribute using Cheerio's attr().
4. Finally, we return a new object that has all the previous keys and values along with this new image URL.

Now, you might have noticed that our getRepoImage() is not asynchronous but instead, the callback function inside repos.map() is asynchronous. We did this because we are making our get() call inside this callback function so, there is no need to make getRepoImage() asynchronous.

Moreover, whenever we are making asynchronous calls in a loop, it is a recommended practice to use Promise.all() because like in our case, it will wait for all the asynchronous calls to be resolved and then only it will return the values.

You can now run your app and open localhost:3000/[username] to check out the image URLs.

[Optional] You can also customize your repo's social preview (click here to know more).

Great!! Our 1st feature is complete 🥳. Let's move on to the next.

Get pinned repo's GitHub API data

GitHub itself provides us with a REST API that provides a lot of data of our repos. You can access yours at: github.com/[username]/github-pinned-repos.

This data can be very handy so, what if we could get all of that data but just for our pinned repos. That's what our 2nd feature is, we loop through the response of this official API and filter out data of only the pinned repos we have scraped so far.

Create an asynchronous function getGithubApiData() with 2 parameters username and projects of type string and array respectively.

We will call this function inside getPinnedRepos() just below getReposImages() and pass username and repos as arguments. Then, we will await for its resultant values as well and update our repos variable again.

Now, insert the following snippet in this function:

const githubApiUrl = `https://api.github.com/users/${username}/repos`;

try {
  const { data: ghApiRepos } = await axios.get(githubApiUrl);
  let finalRepos = [];

  ghApiRepos.forEach((ghApiRepo) => {
    repos.forEach((repo) => {
      const ghApiRepoUrl = ghApiRepo.html_url;
      const repoUrl = repo.url;

      if (ghApiRepoUrl.toLowerCase() === repoUrl.toLowerCase()) {
        finalRepos.push({
          ...repo,
          ghApiData: ghApiRepo,
        });
      }
    });
  });

  return finalRepos;
} catch (error) {
  console.log(error);
  return error;
}

1. First, we make a get() call to the GitHub API and then start looping through the returned data. We also initialised an empty array finalRepos.
2. Then, we will start another loop inside this loop and compare the URLs of our scraped data and the new data we got from GitHub API.
3. Whenever the condition is fulfilled we push the GitHub API data along with the scraped data to the finalRepos array.
4. Finally, we return this finalRepos array.

Make the 2 features OPTIONAL

Now, we have successfully added 2 new features to our app and you will notice that our app follows this flow:

It first scrapes pinned repos from the user's profile and then, scrapes each repo's social previews and finally, returns along with GitHub API data.

It's great but, what if a user only needs the scraped repos, or they don't need GitHub API data, or no social preview? We can handle these situations by taking URL query parameters and calling the functions accordingly.

Note: The amount of data transferred over the API affects the speed of an API so, it's better to give choice to users for what data they need.

Firstly, inside our /:username route we pass 2 new arguments: req.query.needrepoimage and req.query.needghapidata to getPinnedRepos(). We will also apply our stringToBoolean() utility function (import it like capitalize.js) to them both because req.query.[queryname] will provide us a value of type string and we need a Boolean value.

This is what your updated /:username route should look like:

app.get("/:username", async (req, res) => {
    const result = await getPinnedRepos(
        req.params.username,
        stringToBoolean(req.query.needRepoImage),
        stringToBoolean(req.query.needGhApiData)
    );

    if (result.status === 404) {
        res.status(404).send(result);
    } else {
        res.send(result);
    }
});

And inside the try block of getPinnedRepos(), we replace this:

repos = await getRepoImage(repos);
repos = await getGithubApiData(username, repos);

with this:

if (needRepoImage && needGhApiData) { // If user wants the social preview and GitHub API data
  repos = await getRepoImage(repos);
  repos = await getGithubApiData(username, repos);
  return repos;
} else if (needRepoImage) { // If user only wants the social preview
  repos = await getRepoImage(repos);
  return repos;
} else if (needGhApiData) { // If only wants GitHub API data
  repos = await getGithubApiData(username, repos);
  return repos;
}

We will also, add 2 new parameters: needRepoImage and needGhApiData to getPinnedRepos() of type Boolean with default values as false.

You can now open localhost:3000/[username]?needRepoImage=tru.. in your browser to test and play with your API. Try changing the queries to false one by one or even removing them.

UI at root route

Alright, so far we have added 2 new features to our REST API and now we will add a very basic User Interface to it. We will create an HTML form at our / (root) route that takes a username and hits our /:username route.

Create a new index.html file

touch index.html

and populate it with the following code:

<!DOCTYPE html>
<html lang="en">

<head>
  <meta charset="UTF-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Github Pinned Repos</title>
</head>

<body>
  <h1>Enter your Github Username</h1>
  <form onsubmit="return false">
    <input type="text" name="username" id="username" placeholder="Enter your Github Username...">
    <button type="submit" onclick="getData()">Submit</button>
  </form>
</body>

<script>
  function getData() {
    const username = document.querySelector("#username").value
    window.location.href = window.location.href + username
  }
</script>

</html>

1. In <body> we created a basic HTML form that takes a text input and executes a function when the button is clicked.
2. In <script> we create a function getData() that hits our /:username route with the input value as the username.

Now, coming back to our server.js, we will create a / route that will use res.sendFile() to send the index.html file as a response.

app.get("/", async (req, res) => {
    res.sendFile(__dirname + "/index.html");
});

We are using __dirname to get the absolute path of the directory containing index.html.

That's it, you can now access your HTML file at localhost:3000 and check out your newly build interface.

Concluding

Next time, we will deploy our app into production. Meanwhile, if you have any doubts or issues, feel free to use the comment section and I will try my best to help you out.

Thank you for reading!! 🙏