Spidey – the faster, less distracting way to do research online

general assembly singapore web development course in session
It’s the end of the week, a project week, a project week-eeeennnnd! (Wong Fu Productions reference.)

Today is Friday, 23rd September 2016, week 6 of the intense 12-week GA web development course. Each of us have just completed a full-stack web app—the first for many of us including me—and presented it to the class. It was great!

Before describing the project I worked on, I’m going to list the things I’ve learned from this week of intensive, all-in coding.

Lessons from a week of intensive coding

  • You won’t know until you try. Good news with web stuff is that trying is (mostly) free. (Unless you need to pay to use certain data sets from APIs…) So whenever I think something should work a certain way, I don’t ask someone. I try it out first. It is often faster and, even better, easier to internalise.
  • There are plugins for almost everything, so search before trying to build another wheel. The caveat here is if you are building a business and are worried about being at the mercy of external library owners (what if the creator removed it from GitHub?!), I think it’s ok to try to build your own. Otherwise, it’s best to build your stuff on what others have built.
  • It’s ok to leave CSS to the last, but nevertheless, do your CSS. A technically sophisticated product without good packaging is destined for failure. Internet users—everyone, basically—are human, and humans privilege vision.
  • When learning to code, most of us will quickly notice the chasm between what we are creating and the finished products available out there in the world wide web. It is there, and yes, it is huge. Like huge huge. We expect a button to just do that thing. Nobody even thinks about how the hell it does – they just expect it to be because there are great developers out there who built the web to be as intuitive as it is to use in 2016. But it’s useful to bear in mind that once you get to the peak of the first mountain (called HTML), it becomes that much easier to scale CSS. The skills are transferrable. Then take on JavaScript, then jQuery. Then Node.js and Express and Postgres. Eventually, the chasm will begin to close up as you will the other side towards you. We’ll get there.
  • To learn as much as possible, always factor some time for experimenting with things that you haven’t used or done before. Plugins, different approaches to writing code that do the same thing but potentially more effectively, APIs, etc. Walk through a door but occasionally walk back out and check out the rest. You won’t know what you’re missing out if you don’t.
  • Reading other people’s code is only interesting and useful at the right moment. Immediately after doing a project with the same specifications (but with wildly different implementations) is such a moment.

As you can see, I’ve learned a lot this week. I owe it to the good programming of the Web Development Immersive course and to my classmates who did their best and inspired me along the way.

beer and wine
A great way to celebrate after a week of hard work and growth!

Introducing Spidey 1.0

So, on to my project for the week. I made a dashboard for people who need to do research on the web, but need to do it in an efficient and non-distracting way. A live version of the project can be accessed here. Updates to the project will likely be pushed to that link.

The process of using it is something like this:
Enter search term —> Browse summarised results —> Click interesting search results to read full article on same page —> Digest info, synthesise into the project you’re working on —> Save article URL for later use/reference

Simple and straightforward enough, right? Here, if I wasn’t already 6 weeks into the web development course, I would’ve said, “Yeah, sounds simple and easy enough!” Alas, I am a month and a half into the course and I knew what I was getting myself into when I chose to try and create this project in one week. It wasn’t extremely difficult, but it was challenging.

The bits I found most difficult to grasp were the server-side stuff. Using Node.js, we can use JavaScript to configure a server. But Node.js is a tricky son of *. To get a server running you’ll need to create several folders and put in the relevant pieces of code in them. Authentication goes to the ‘auth’ folder, passport.js/bcrypt/express-session external libraries must be initialised in the root folder inside index.js, and oh, passport.js requires configuration, which should be done in a separate configuration file and in a folder called ‘config’. Full rant here.

Somehow, as a web developer, we’re supposed to know where every single thing goes and how they relate to one another. If you’ve ever looked left and right 20 times in quick succession, you’ll more or less know how it feels to create a server using Node.js.

Anyway, I feel good about how the project turned out, but if I were to be honest, I’m not completely satisfied with what it does yet. Spidey is still in its early development. It’s not good enough even for me to want to bring it into my writing workflow, even though it was designed to be. The good news is that unlike the previous project–a Spot the Difference game–I will continue to work on Spidey. I can already think of one or two features that will make it worthwhile to use. One of them that I’m most excited about is the ability to add customised markdown to your article and let Spidey find the best materials online that you should read and append as hyperlinks to your own. That should easily cut down the time needed to write a great piece by half!

I like where I am right now. I feel like some of the worst days are behind me, at least when it comes to creating a server for a web application. I know conceptually and practically how to set one up and how it works, even though it might take a while. It wasn’t easy getting here, but interest and grit got me through, and I look back smiling.

More adventures ahead!

PS. If you have seen my research dashboard project Spidey and have feedback, feel free to leave a comment.

The nightmare of coding a node.js server

lion lying restlessly on the ground

Setting up a node.js server is not the same as writing HTML, CSS or JavaScript. It’s much more complicated than that, and honestly, I think it shouldn’t be.

So, HTML handles what is displayed on the page, CSS alters the way these HTML elements look, and JavaScript manipulates these elements dynamically according to user action. While the process of writing JavaScript code, which is the only real programming language among the three, can be at times very challenging, it is almost always easy to identify the reasons you’re stuck. You might be stuck because you were experimenting anyway, or because you used a string method on an array or something like that. After coding (and staring at code I’ve written) for a while, I’d usually recognise a pattern and be able to fix the problem.

On the end of the spectrum as I’ve recently stumbled upon is node.js and its host of libraries. Ok, I didn’t really stumble upon them so much as I was told to use them by my instructors at General Assembly. Anyway, Node.js is an engine that ports the JavaScript language to be used in projects other than those that run on browsers. In my case, I’m learning to use it as a server side language to build a server for my web project, and it’s been a nightmare. Here’s why…

Node.js comes with a slew of external libraries that do things. In practice, these are more like essentials than addons. Express, body-parser, ejs view engine, passport.js, sequelize database handler, and much, much more are almost requirements for node.js to work as a server.

If it were merely about dependencies I would just live with it, especially at this learning stage, but that is definitely not the case with this house of cards. The biggest issue I’m facing with writing code for the back-end (ie. server) using node.js is the acrobatics it requires of me.

I’ll use the example of trying to set up user authentication to bring you through the source of my recent headaches.

The nightmare of setting up user authentication with passport.js

Ok, here goes. To set up a web app with user sign up and login (what we call authentication and, when it pertains to permission to view certain pages, authorisation), we need to use an external library called passport.

A little on what that means. Passport is a package for Node, and Node in turn is the platform on which our entire server is built. It’s called node.js because it ports the JavaScript language, which was designed and built around browsers, to the server side of things. It’s a big, great deal to have something like node.js exist. For one, it saves me from having to learn a server side language like PHP.

Before I can write a line of code that defines the logic for authentication and authorisation, I need to set up my server to “require” these and various other packages. In this case, I also need the express-session package because passport needs that to work.

After writing a bunch of “require” and “app.use” statements, I then have to create new folders to store a few other files that are absolutely necessary for it to function properly:
1. /config/ppConfig.js to set up the behaviour of passport
2. /controllers/auth.js to receive the signup, login and logout routes and control what happens when the user lands on the page

After this, let me see… Oh ya, after this, I’m supposed to create a “hook” to make sure that when a new user signs up, the info passed into the form and into server are sanitised and the password is hashed (ie. encrypted) before they get stored in the database. That requires using–and thus, learning–two more node packages (bcrypt for encryption and sequelize for communicating with a SQL database using JavaScript instead of Structured Query Language), so I think I’m going to take a break first.

Ok, I’m back, and quite frankly I don’t feel inclined to spell out the rest of the process. It’s just mind-numbing! (If you have questions though, I’ll gladly help as I can. Just post your question as a comment.)

What is the point?

Yes, I have a point I’m trying to make, and it’s this: it shouldn’t be this hard!

I’ve read in several places about a particular kind of professional whose main goal is to build systems so that life can be better, less repetitive and mundane, more colourful…

Programmers are not supposed to have to deal with unnecessary complexity, and this is clearly what it is. It should be easier. With a community as lively as it can possibly be, there should be a project to unite these common server tasks (how many websites don’t need authentication and authorisation nowadays?) into a much leaner machine.

Until that happens, I’m afraid the only way out is through.

(image: Joel Herzog)

A writer’s tool for project two

a young lady in a beautiful office

Week 6 of web development school is coming up! That means we’re going to build stuff – yay!

I’m particularly excited about this coming week’s project. We’ll have from tomorrow (Friday) to next Thursday to start and complete to an extent a full-stack web application. The point is to create a web-based application that demonstrates our understanding of front-end web technologies (HTML, CSS, JavaScript) and back-end technologies and frameworks (Node.js + Express, Postgres SQL), and how to hook them up nicely to do useful stuff for us/other people.

For someone who hasn’t built a full-stack application before, this is a tremendous opportunity for me to make mistakes, learn, and flex my brainpower. It’s going to be great.

And for the first time in a long time, I’ve actually decided beforehand on the project I’ll put myself to work on. It isn’t something I’ve always dreamt of making since I was kid, but it is something I care about creating for myself to use, and so to me, that counts as a good project to commit some time to working on. Definitely beats working on a project just as an exercise that’s for sure.

The problem to try to solve

For the second of four total projects, I’ll be creating a writer’s dashboard. Let me first explain the problem…

As a person who loves to write (I don’t really know why), I write almost every day and publish selected articles on my blog. Now, as a person who is neither a self-made billionaire or a celebrity, I have to earn my reader’s trust by showing that each article is fact-checked if they are meant to be. If I write about Facebook’s bullying antics toward the open source community for example, I’d have to include links to credible news sources or nobody would believe me. Also, for Google’s bots to find my blog and rank it reasonably high in search results, every article should contain a few links to relevant external resources and internal blog posts.

Here comes the problem: I love writing but I hate finding URLs to embed into my articles! The process of writing is fun and transcendental, but finding and appending URLs is banal and at times even soul-numbing. Sometimes I just don’t feel like doing it and I just hit publish. No matter how factual-sounding my article is, though, I know new readers will not trust me.

Another problem in the post-creative phase of writing I’ve found is that when I research online for facts, I tend to drift deep into the open ocean of the internet. I wouldn’t even call it a distraction – it’s a total attention blackhole. Google returns too many entertaining results…

What I’ll be working on for project week

So I’ve come up with a potential solution. I think it will work for me, and I’ll only be focusing on catering it to my needs for this now, to keep things manageable.

Here’s the idea. Create a dashboard that, based on a single search input, pulls articles from around the web and displays them in so-called cards. A little bit like what Oscar Otero did with his jQuery cheatsheet, but with a different presentation.

Each card will contain a title and a text snippet of the article, and clicking on it will launch an in-page window with the full, original article (eg. from Wikipedia) from which I can browse and extract information. These cards are laid out across a single browser page to provide a quick overview of the articles relevant to my blog post.

For added convenience, each card also comes with a short text input field that I can use to copy the original article’s URL and paste into my blog post. Hitting a ‘favourite’ button also saves the link to my account for later, periodic use.

There are also a number of other features that I’d love to have but won’t be able to create in just a week. One of the most useful would be an algorithm that automatically appends internal blog links to my newest, yet to be published blog post on my behalf. That’d be fantastic. Alas, I am only a novice programmer and I shall be realistic.

With that, on this humid Thursday evening, I’m laying my mind to rest about the project. Tomorrow is project brief day. For now, I have other things (like how to integrate user authentication to my server) to think about…

(image: Bench Accounting)

Continually undo knots

a whiteboard with a sketched diagram of the relationship between database, server, and client
Our instructor, Jeremiah, illustrating the relationship between the client, server, and database in a web project

Once you’re over a certain age, enough knowledge of how the world works will be stored in your memory. That knowledge can then serve as the basis from which to draw parallels to other things, and by doing so, our imaginations can relax because you’d already have an inkling of how it might work out.

The parallel between a box and a database is telling of this. Because I have lived for 26 years, I have a good idea what boxes are and what they are good for. People use boxes to store stuff or transport or contain or any combination of those things, right? So when I’m made to think about a database, I immediately relate it to a kind of box. “It stores stuff that can later be retrieved.”

Of course, sometimes we draw false parallels. A database isn’t exactly a box – it’s more like a whole collection of small pull-out drawers that contain items (or data in the virtual world). Even that may not be as accurate as saying that a database is a “glorified spreadsheet”, as our instructor said in web development class today; but that hardly matters because the value of drawing parallels is really in its ability to help us visualise the abstract. As long as it’s kind of the same, it will do the job.

Metaphors and parallels may be cool kids in the conceptual block, but in application town, they’re about as interesting as a piece of unbaked clay. Come building time, visualisation must give way to implementation thinking – and that’s the hard part.

To write code that sets up a database that can communicate with your application is a lot harder than visualising the database being somehow hooked up to your webpage.

It is difficult for everyone, not just me or you. Every aspiring web developer will need to put fingers to keys and create that bridge to learn how the whole thing works at least once.

My first try at creating a database-linked server application was met with countless problems, like knots in a long string. I know where I’m supposed to head, but the knots keep me from going through.

First it was the new framework called Sequelize, which apparently is used by many developers as a way to use the JavaScript language (practically every web developer’s bread) to communicate with the database, which uses something called SQL (Structured Query Langauge) – where “sequelize” got its name from. Then it was how to run Sequelize commands on the command line, as well as how to write code in the server side hosted on Node.js.

Learning to code is a continual process of de-knotting to get to the end of the line. Well, except there’s no end of the line, but really just the end of this or that line. When I finally untied the knots today that were blocking me from implementing an exercise project that linked to a database through Sequelize, it was magnificent. I think I might have even said “it’s so straightforward after you get it!”

While having a straight rope now feels great, I know this is not the last time I’ll be spending hours in cognitive high-gear trying to chew away a knot. I look forward to undoing many more knots to come, because that’s how I know I’m getting better.

Don’t code for code’s sake

person welding metal

Coding is just a means to an end. Unless you’re using programming languages as a medium of art or doing computer science research, there is no good reason to get romantically involved with your code. It’s best to not get overly attached to the (idea of) code but instead, focus on what the code is meant to achieve.

For the past month in class we’ve had an instructor who’s passionate about programming and great at it. He even used to work at Google as a developer for a number of years. His enthusiasm about programming is infectious, and I think I became more in love with the fact that I’m now a “coder” because of it. Little did I know that I’m starting to romanticise the idea of coding a little too much for my liking…

Our “new” instructor came in today and, I think unknowingly, threw me out of my daze. He declared to the class that he sees code merely as a means to an end, and although he keeps himself up to date with the latest and the coolest, he does it only to know about the most up-to-date ways to be lazy and write less code. I don’t know what to tell you, but it was really effective in bringing me back to reality.

What reality? Simple. Coding is just a means to an end. Our real calling as a developer is to write programmes that do useful stuff for people.

I knew that. That’s one of the biggest reasons I decided to dive into learning to code full time in the first place – so that I can make cool stuff that make people’s lives easier and more meaningful.

I’d temporarily lost focus, getting caught up with the code itself. If I were made to choose between being a great programmer who writes beautiful code and one that writes only decent code that works, I would choose the latter knowing myself. By spending less time worrying about the “most efficient way” to do things, I know I’d be able to spend more time on fixing annoying bugs, user testing, and anything else that makes the programme more delightful to use.

Why are you learning to code?

Your reason will decide how you should think about programming. As an aspiring entrepreneur, I’m learning to code so that I can implement projects on my own, and eventually manage a team of developers to build the rest of it. Coding is but a part of my toolkit – albeit an extremely important one that without which I’d cease to be an effective entrepreneur. Marketing, business development, user testing and product development, legal stuff, and much more are other things that an entrepreneur needs to be effective. Seeing that I’m not superhuman, it’s best to proportion time for getting decent at each of these rather than got amazing at one – even if it’s the most important.

In fact, spending time romanticising the idea of “conjuring things out of thin air” as I’ve been saying a lot lately isn’t necessarily going to make you a great programmer. Sure it may mean that you have a disposition that is primed for learning new things about your programming language and using them to create programmes with more robust and clean code; but on the flip side, it may be that you’re spending a disproportionate amount of time keeping close to the cutting edge in order to write the latest and coolest programmes.

It really depends on who you are. For me, I’d rather stand closer to the mainstream but make a lot more useful programmes that create value for people than at the edge making one or two cool ones that fellow developers would admire and respect me for.

I think that makes me a pragmatic programmer. How about you?

(image: Rob Lambert)

Two modes of programming

rack of old vinyl records

I had an interesting conversation with a programmer yesterday about programming. Halfway through our conversation about how much complexity we have to handle when writing programmes, she said something that captivated me: “my friend is a very experimental programmer”. How fascinating!

In my short programming career so far (some might say it’s not even a career yet), I’ve come to know of two modes of programming. First and the more straightforward and common is the problem-solving mode. This is the state of mind that we get into when a specific problem has been handed to us or that we have identified for ourselves to solve. We then proceed to hold the problem in our mind’s hands, turn it around to examine it, get a feel for it, and devise a way or multiple ways to crack it. Most programmers are in this mode most of the time, because to write good, useful programmes, we need to literally consider every potential use case under which it will work, and if it breaks, fix it.

While it’s certainly an engaging state to be in, the problem solving mode is inherently restrictive. Programming is the act of solving one big problem by breaking it down into its tiniest constituents, but when we choose to enter a room, the others are forfeited. Once we set our mind to solving a specific big problem, it can be extremely difficult if not impossible to zoom out and be experimental.

The experimental programmer

What exactly does experimental mean in the context of programming? Broadly, it means being in a state of mind of imagining things, of making things up, and then trying them out to see if they yield any interesting outcomes.

The lady I met yesterday gave me an example of this in action. She described her friend, who wrote a Tetris programme, as someone who is very experimental. When I asked what she meant by that, she replied comfortably as if it was a natural thing, that her friend was someone who would write a whole bunch of different code just to see what happens each time, and later, combine them to see what kind of effects can be achieved.

I told her that that’s what most programmers do, but then I realised a crucial difference between her friend and most other programmers bent on problem solving on my own, before she has to explain. Her friend, the experimental programmer, was only letting the big problem he was trying to solve (how to create a two-player Tetris) sort of guide him in a very loose way. Being experimental, he mostly tried things for the value of discovery, and trusted that some of these will lead to a potentially better game design. There’s an important difference between the higher plane of experimentation and the lower, on-the-ground plane of thinking – dwelling on the higher plane can potentially prevent a programmer from coming up with a brilliant solution to the wrong problem. It is also a lot more fun.

Pacing around higher, experimental plane is the key to finding what Tim Ferriss likes to call “the lead domino” that will easily topple all others along the way. Better to find the right domino to kick than to go straight to kicking dominoes.

(image: Mr Cup)

Why programmers make visually ugly projects

people laughing and talking with one another general assembly wdi singapore

As a user, it used to be hard for a website to impress me. I see certain features here and others there, and after a while, I get a good idea of what is possible and is usually practiced, and that guides my behaviour on the web. I’d expect that hitting ‘enter’ will submit a login form.

Now, I’m much more easily impressed. Why? Because I now know how much thinking and coding goes on behind the scenes to make simple things such as updating a user profile work. Here’s an illustration with editing a profile:

  1. user clicks ‘edit profile’ button
  2. client-side sends a GET request to server-side (to fetch the edit profile page with the form to edit)
  3. server-side finds the exact match of GET request, gets a lot of different info from database, sends data that makes up the edit page to client-side
  4. client-side continues to run…
  5. client-side receives data, manipulates data, and refreshes the webpage with form (pre-populating the form with existing info)
  6. user makes edits, hits ‘save’
  7. client-side makes POST request to server-side with user’s data input
  8. server-side receives data
  9. server-side manipulates data, updates database
  10. server-side redirects within itself to the GET request (a la step 3)
  11. server-side sends response to client-side via GET
  12. user’s browser refreshes to profile page

What used to be intuitive is not so intuitive. For the first iPhone to feel intuitive, a lot of work was put in by Steve Jobs and the Apple team behind the scenes. As internet users, things feel intuitive when they behave in ways that are consistent with how things normally behave in reality.

But the internet is completely virtual—you can never hold it in your hands and get a feel for its texture—and that makes it a big challenge for programmers and engineers and designers to to make things feel right. So far though, I think they have done a great job (hat-tip to the people working on the first iPhone, which changed everything with a large, multi-touch touchscreen with gesture controls and more).

I found myself wow-ing a lot in class today alongside my classmates. That said, I bet you would never guess what we were looking at. Steve, our instructor, showed us this:

screenshot of simple website mockup of Best Buy

Yep, an extremely simple rendition of Best Buy’s website with admin controls to create, edit, and delete product pages. Having only just learned how to set up a (locally-hosted) server using Node.js, this was a big leap forward in technicality for us, and I stopped to wonder just how complex some programmes out there really are.

Here’s one last observation from today: programmers are so busy making sure that the code works (ie. doesn’t break) that styling a webpage is one of the last priorities for them. It’s true – I almost always leave CSS for last now. I think of it as a veneer that is important, yes, but my job is to ensure that the product functions as it is intended – and that is a full-time job in itself.