Python

tl;dr When using Python to write web scraping / general automation scripts, run Python using the -i flag preserve and interact with the latest state (i.e python -i script.py). This let’s you avoid re-running the whole script every time a change is made. Problem When developing a bot or web scraper that operates behind a login screen, development, testing, and debugging can take longer than you might expect. The process of write code --> test --> debug --> repeat hits a bottleneck every time the browser starts, pages load, login details are entered, the welcome page loads, and so on.

tl;dr It’s hard to underestimate the importance of typing. Unless you are a journalist, legal secretary, or professional writer, you probably don’t get enough practice to ever become truly fast. I built Typr and used it to simultaneously read & type “Surely You’re Joking, Mr. Feynman!”. Two weeks and 100,195 words later, I was maintaining between 100-110 WPM after starting out at 50 WPM. If you like reading, kill two birds with one stone by using Typr to practice typing material you would otherwise be reading.

tldr: Auto-scrape let’s you focus on writing web scraping scripts, while it takes care of logging, data persistance, data presentation and data export, all through a modern browser-based UI. It can be run locally or deployed remotely. Here are some screencasts of the UI. Get it on Github. Why scrape the web? Building a Selenium web scraper is almost a rite of passage for programmers starting out. Watching a computer fill out forms, click links and collect data before your eyes is not only a highly satisfying and suitably non-abstract exercise for beginners to complete - browser automation forms a foundation for frontend testing, can be used for automated research, and of course can be used to replace those expensive and unreliable humans to accomplish a wide range of business-related tasks.

Expressing time intervals in a format that is readable by humans is an old problem that every frontend developer will eventually face. While there are plenty of solutions available (this one is my favourite), I required a function that could provide different levels of granularity in it’s “human-readable” description of time. Take, for example, a component that tells the user how long in aggregate they have been using a service. If they have been using it for a few hours, it would be appropriate to display ## hours, ## minutes, ignoring the seconds and milliseconds.

This function takes an SQLAlchemy query object as it’s input (SQLAlchemy is a popular Python SQL toolkit and Object Relational Mapper). It’s strength lies in not needing to hard-code column names, making it scaleable and suitable to “set-and-forget”. While developing enterprise software, every developer has surely have been faced with a client asking “how do I get the data out of the system?”, and after digging a bit deeper, it becomes clear they want the ability to save a snapshot of data as an Excel spreadsheet.