Posts by briansia (8)

Resource Monitoring for Developers

Motivation

For computationally intensive tasks, monitoring the workstation's resource usage is important. Said workloads can range from gaming, to high performance computing, or training deep learning models.  Knowing how much CPU/GPU/RAM/network bandwidth is used can tell developers if their machines can handle the workload. If necessary, the developers can modify their code to reduce resource usage, or add more hardware to accomodate for the heavier load.

Monitoring Tools

Depending on the platform, different built-in solutions are offered. Linux users have the GNOME System Monitor, Windows users have Task Manager, and Mac users have Activity Monitors. Sadly, these off-the-shelf solutions share a few problems. They are quite inaccurate, their graphical interfaces are unfriendly, and their logs cannot be exported.

These tools are useful for taking a quick glance at the resource usage. Serious developers, however, would require more than basic tools. They often have to export their findings and share it with their colleagues.

Third Party Alternatives

There are many third-party tools catering to various platforms and needs. For example, GPU-Z for monitoring NVIDIA chips or s-tui for graphical monitoring on Linux terminals. My personal favorite monitoring tool by far is netdata. 

Netdata is a powerful open-source tool designed to collect a plethora of real-time metrics, and then displaying them on aesthetically pleasing, easy-to-interpret graphs and charts. Unfortunately, Netdata is not supported natively on Windows. Netdata's engineers decided to focus their effort on Linux, since that is what the vast majority of the production systems use. Windows developers have to jump through a bunch of hoops to use netdata with Windows (Just hop onto the Linux bandwagon already. Why would anyone still run production systems on Windows?). In the future, a whole post may be dedicated to just exploring Netdata.

I hope this post introduced you to some monitoring tools. If there is any takeaway, it is for developers to move away from built-in monitoring tools and adopt better third-party solutions. Feel free to share other monitoring tools that you used for your job.

Happy monitoring! 

Learning Flask With a Personal Project

What is Flask?

Flask is a web framework written in Python. It is a microframework that does not require particular tools or libraries. As such, Flask does not come with database abstraction layer, form validation nor any other components, unlike Django. Flask and Django are the most popular Python webdev frameworks, so supports for both are extensive.

Flask Components

Flask operates with these components.

- Werkzeug, a toolkit for Web Server Gateway Interface (WSGI)

- Jinja, a template engine for displaying Python variables in markup

- MarkupSafe, a string handling library that escape characters so it is safe to use HTML and XML

- ItsDangerous, a data serialization library used to store Flask app sessions without allowing users to tamper with session contents

Toy Project – Pack Cracking Simulator

The toy project is a simple web application that simulates the opening of a card game booster pack, or colloquially called ‘pack cracking’. A Magic: The Gathering pack (only the best trading card game in the world) is cracked open, revealing 16 cards. A standard MTG pack (not counting special products) will contain 1 rare or mythic rare card, 3 uncommon cards, 11 common cards, and a token card. Promotional cards are not counted since they do not provide any gameplay value.

Setting Up the Flask Project

Setting up a Flask project can be a daunting task for new web developers who are not familiar with the Flask project structure. Instead, running a boilerplate generator is faster and much more convenient. The one used in this project is https://github.com/isakal/create-flask-app, a Flask version of the popular Create React App generator. Spending time for development is much more productive than spending time on project configuration.

Card Data Request

Card data is acquired using the Scryfall API. Scryfall is the most well known search engine for Magic: the Gathering cards. Requests for sets and individual cards can be performed by following the request format stated in the docs. The response will be returned as JSON blobs.

As per usage policy, each request comes with a 50 milliseconds delay so that the Scryfall server does not experience heavy load.

Since all request information can be contained inside the URL, all Scryfall requests are GET requests.

Request Caching

Sending a set request to the Scryfall API, getting the JSON blob and rendering the cards takes around 12 to 20 seconds, which is quite slow for a simple app. Caching the requests would bypass the need for the application to fetch data from Scryfall’s server. With the cache, the first request takes more time, but subsequent requests of the same set would take a mere fraction of a second.

Since this is a simple app, there is no need for long term storage nor a sophisticated caching policy. Setting request TTLs to one day will wipe out stale data.

Architecture

The application works as follows:

1.) The user selects the set with the interface.

2.) The application sets a get request to the Scryfall API.

3.) If the GET request is cached, get the JSON blob from the cache, send it to the web server,

and render the cards on the browser. Else, get the data from the server, cache the request, and send the data to the web server before rendering it on the browser.

Sample Output

Summary

The project shows how easy it is to set up and run a Flask project. Learning how the Scryfall API work took the most time. The request cache shows how a few lines between each request significantly speeds up the performance.

The project can be found at https://github.com/zerovirus123/pack_cracking_simulator.

Learning Web Development

Before Web development

Before learning, the applications that I wrote are usually standalone and runs only on individual machines. I had previous exposure to coding in my electrical engineering courses, however they are either microcontroller code or hardware description languages like Verilog.

What Is Web Development

Web development, however, offers so much more potential. Every major applications these days run on the web. These applications could be search engines, e-commerce sites, or in-house solutions. All these examples have one thing in common: decentralization. This could be the decentralization of computation or data storage. A one machine task can be done much more effectively if the task is performed across many machines. Standalone applications that do not communicate with other machines will soon be obsolete, if they haven't already been. With the exception of low-level hardware engineers, programmers are losing out big time if they do not learn web programming. Knowing the basics will be useful, even if one is just trying to code up a toy project.

Learning Web Development

Getting started with web development is a steep learning curve. For starters, there are numerous frameworks out in the wild. Learning a framework is one thing, but knowing the project architecture and requirements before choosing one is another. Serious web development does not just end with mastering a framework. Front-end design, setting up back-end infrastructure, security and traffic handling are just some of the challenges that one would face. Each field is so deep and complex that there are teams dedicated just for that one aspect. No wonder full-stack engineers only exist in startups.

Where to Go From Here

The path to become a web developer is tough, but the rewards of being a good web developer is too good to pass up. A good starting point is to learn a simple framework like Flask. I am quite proficient with Python and have working knowledge with HTML and JavaScript, so Flask seems like a good starting point. Compared to Django, Flask also has a lower learning curve. Flask is probably the most suitable framework for learning web development with personal projects.