Illuminated Computing

What to Learn

2017-08-02T00:00:00Z

I once heard a tech speaker say that in programming her job description was “learn new things,” and I’m happy to steal that way of putting it. In seventeen years of professional work I’ve never done a project that didn’t require me to learn something new on-the-job. It’s what I love about programming. But how do you decide what to learn?

Some things you learn because your project demands it, and those lessons are small and focused (hopefully most of the time): a new library here, some protocol detail there, today something about Linux, tomorrow something about Docker. This is stuff you do on the job, practically every day.

That’s not what I’m talking about here, but I’ll offer some advice in passing: when you learn a new tidbit, try to write it down. You don’t have to spend time polishing it, and if it helps then write it somewhere private. But write it down. I use personal man pages for this—and I’m not very good at it myself. If you’re bad at it too, then at the very least spend an extra 20 minutes making sure you actually understand, and come up with a few experiments to test that your take is correct. Try to put your understanding into words, at least in your own head.

But instead of that on-the-spot learning, I want to talk about things we learn that take more time and have a more long-term payoff. We typically do this off the job, without pay. I think most programmers love learning (or they would soon find a different job), so we can’t help ourselves. But also it pays to keep your skills current and sharp. Every professional has to do this. My favorite book about professional services work (which despite the title is about way more than managing) talks about developing your “asset”—you—by continuously learning. Lawyers, architects, accountants, doctors—all have to keep learning. Car mechanics too. With programmers the pace is different, but I expect the world doesn’t exactly stand still for anyone else either.

There is so much to learn! And the hype is everywhere, stealing your attention and diffusing your time. Prototype, jQuery, Backbone, Knockout, Angular, Ember, React, Vue, … Less, Sass, Uglify, Asset Pipeline, Npm, Bower, Babel, Gulp, Grunt, Ember-cli, Webpack, … Oracle, MySQL, Postgres, Memcached, Cassandra, Mongo, CouchDB, Redis, Riak, DynamoDB, … Aaah! You can’t learn it all, so you have to be deliberate.

I often hear advice to learn one new language a year, two new languages a year, whatever. The best versions of this advice say to learn a new “kind” of language, like Lisp or Haskell or Prolog. For a new programmer, that’s pretty good advice, and I still follow it myself. (For me the last few years it’s been Haskell, Rust, and Elixir.) But for several years I’ve tried to adopt a more strategic approach. One of the problems with learning another language is that either it’s something you won’t actually use, or you mostly leave behind the old one, so it’s like starting over from zero. (Not really, but a little bit.) After a couple dozen you start to wonder how to make the investment more worthwhile. Is there a way to make our learning build on itself, so we aren’t throwing away so much time? Here is my own approach to having some “continuity” in what I learn:

First, realize that there are so many more categories besides language! There are operating systems, cloud environments, back-end frameworks, front-end frameworks, databases, build tools, deployment tools, networking protocols, specialties like GIS or machine learning, industries like e-commerce or finance or health care, “soft” skills like writing, requirements gathering, design, management, financial planning, sales. Don’t get stuck in a rut of thinking in only one dimension.

Second, don’t be too focused. Go ahead and mix in some “useless” learning. I’ve had fun lately reading about the history of transistors, the integrated circuit, and the Internet. Or to take it to an extreme, you could learn some Greek or Latin or Chinese. :-) Whatever you like. One of those books (The Chip) actually talks about how Jack Kilby, the co-inventor of the IC, would read several newpapers and a bunch of magazines every day, plus every new patent granted by the government. Maybe that’s an extreme, but it’s good to have some breadth because you never know what will come in handy or inspire you. But more than that, recreation is important. Read some trashy science fiction or something.

But when you are being deliberate, I think there are three good alternatives to “learn another language”. The first is to learn something that complements your current skills. Suppose you are (or want to be) a “full-stack web developer.” Okay, learn some Rails, pick one Javascript framework and learn it, but then also learn some advanced Postgres, learn some details about HTTP or SSL or CORS, learn Wireshark and IP, learn HTTP Canvas, learn a configuration management tool like Chef or Ansible. I think Chef is a great complement to Rails (or Ansible to Django). My own current “expansion of territory” is down the stack, learning some Rust and reading The Linux Programming Interface. Learn the things that border on what you do, so you are gradually expanding. This is “breadth”, but in a calculated, not desultory way. You’ll probably put those skills to use right away, so they’ll sink in and make you better at your job.

Second is to dive deep somewhere. I’ve really enjoyed getting to know Postgres. I’ve been hired to scale it, to replicate it, to write C extensions for it. Maybe for you it is React or Datomic or AWS or reverse engineering or Ruby performance tuning. But get on the mailing list, follow what problems the community is trying to solve right now, write some blog posts, get to know someone in the community. Whatever it is, use it enough to find some friction points and maybe even fix one or two. You don’t have to make it your whole identity (though you could), but instead of learning a new thing, go in the opposite direction: go deep.

So far this is a lot like the classic “T-shaped person” advice, but I’m saying that for the breadth, pick things connected to your specialty, and for your specialty, have a “specialty within the specialty”. Keep trying to push a little further out, a little further down.

Third and finally is to learn something truly new, at the cutting edge of research. For the last couple years I’ve been reading about temporal databases, which have 20-30 years of academic study but few well-developed practical tools (especially open source ones). This isn’t something I’ve been able to use on a real project (yet), but it’s been great fun, and it feels like a way to find opportunities to build something before anyone else does. How you find your topic is by listening to your pain and seeing if there are other people trying to solve the same problems. Some other things I wish I could become an expert in: Bayesian statistics, operational transforms, HTTP/2, column-store databases, RDF, type theory, vector CPU instructions. There is so much happening! Pick something that people are writing papers about and learn a little.

So that’s what I’ve learned the last few years about learning. Instead of “learn another language”, try to be strategic. Try to build on what you have. Careers are long, so try to find some long-term problems you can grapple with. Maybe like Kilby you will even solve one!

Doing Many Things

2017-05-05T00:00:00Z

I often say that making code do one thing is easy, but making it do lots of things at once is hard. You especially see this in UI code, although it happens on the back end too. The worst of all is front-end web development, where so many things are outside your control.

For example, a customer asks to add a real-time “suggestions” feature to a search box, so that as users type they see possible things to search for. That’s pretty easy, until you think about all the details you have to support:

It works in IE/Firefox/Chrome/Safari, across lots of versions.
It works on Windows/OS X/iOS/Linux.
It works on desktops/tablets/phones.
It works when the user resizes the browser window.
It works when a user has configured text to appear larger than normal.
It has a hover effect that changes the background color and the cursor.
It has a hover effect that bolds the suggestion text, but doesn’t resize the row’s height/width.
It works when there are two search boxes on the page.
It works in the header/body text/left sidebar.
It works inside a dropdown menu.
It works inside a div that is floated.
It works inside a div with absolute positioning.
It works inside a div with relative positioning.
It works inside flowed text.
It works in an iframe.
It works without resizing the containing element.
It works when the containing element has overflow: hidden.
It works on lots of different pages across the website.
It works on all our different websites.
It works when the doctype is HTML5, HTML 4 Loose, HTML 4 Strict, XHTML, and missing.
It works in Bootstrap/Foundation/our custom CSS. (Hopefully you can just pick one here!)
It works with vanilla Javascript/jQuery/Angular/React/Ember. (Ditto.)
It works with or without Turbolinks.
It works with Ajax.
It works when the Javascript/CSS is minified and concatenated.
It works when the HTML and other static assets are served from a CDN on a different domain name from the back end.
The suggestions cover whatever is below the search box.
… but new modal dialogs cover the suggestions.
It works inside a modal dialog.
The suggestions don’t extend outside the modal dialog, but you can scroll to see more of them.
It works when a modal dialog opens another modal dialog.
Clicking a suggestion takes you to the search results.
Clicking elsewhere hides the suggestions.
Clicking a link elsewhere on the page still works when suggestions are visible.
When in a modal dialog and showing suggestions, clicking in the dialog hides just the suggestions.
When in a modal dialog and showing suggestions, clicking outside the dialog hides the suggestions and the dialog too.
When the search box is close to the bottom of the page, suggestions appear above it instead of below.
Clicking a suggestion doesn’t show that funny dashed line.
Clicking a suggestion row in the margin around the text still chooses the suggestion.
It works when suggestions include quotes and angle brackets (", ', <, >).
It works in English/Spanish/French/Mandarin Chinese/etc.
It works with right-to-left text (e.g. Arabic).
When a suggestion is too long to fit, it appears with an ellipsis, but hovering shows a tooltip with the whole suggestion.
The ellipsis works correctly with Arabic and Chinese text.
The ellipsis doesn’t change when the text becomes bold from a hover.
The back button works as expected.
It works when there are no suggestions.
It works when there are 1,000 sugestions, but it doesn’t show them all.
You can use a scroll bar to see more suggestions.
Scrolling down far enough loads more suggestions from the back end.
A suggestion is guaranteed to have at least some search results.
Suggestions show how many search results they have before you click.
It is fast.
It has tests.

I’m not saying it’s reasonable to do all these things every time. You may have customers for whom supporting Arabic and Chinese is a waste of money. I didn’t even list the things users will ask for that are just crazy! I did throw in “search box inside a dropdown menu” to give you a mild taste, and I may have included others that are … questionable. But you have to at least think about them.

In any case, there is a lot to get right, and these details are the difference between polished and unpolished. I learned early in my career that if I deliver something where a few details are still not quite right, I will surely get caught and have to fix it, and the customer will be a lot happier if they never see the broken version in the first place. Polish takes time, but it is still required.

The problem is that completing the list gets harder and harder as you approach the end. No item by itself is hard, but adding the next item without breaking the others is hard. In relativity, accerlating at high velocity takes more energy than accelerating the same amount at low velocity, and the energy to reach c is infinite. Web development has a similar law: requirement n + 1 costs more than requirement n. Adding an ellpisis isn’t hard. Adding it in right-to-left text with an Arabic font on IE 6 in quirks mode when the user has large text enabled is hard.

And now, just to draw one more connection: one thing you learn from experience is ruthless simplification, and skepticism toward adding “concepts” to your app. If you have n features, and they all interact, the complexity grows not by O(n) but O(n^2). I’ve talked mostly about front-end work, but I think it is the same everywhere, and we just see it more on the front end because there it is so unavoidable. Some of it is multiple browsers, some of it is features users expect from any UI, but there our n is already high before we even do anything. In our own feature design we can at least be ruthless. On the front end, we just have to keep polishing. It makes me pity people who sell Wordpress plugins for a living.

As much as you can, it pays off tremendously to manage the combinatorial explosion. You can limit scope, you can stop supporting old browsers. You can design components to be isolated, since if they don’t interact they don’t raise each other’s n. Somehow you have to keep that number under control. The good news is that the marginal benefit of cutting n is large.

You can also write tests, which let you tolerate a higher n by making it less expensive to test in an n^2 world. Sadly browser tests are the most expensive to write, the most brittle (i.e. the shortest-lived), and the longest to run, but still I am always glad to have them.

I guess I have written myself into two morals here. One is: don’t permit any more complexity than you have to. The second is: sometimes you just have to keep polishing, also known as “That’s why they call it work!” Painfully, these really tug you in different directions. Choosing when to apply them is not easy.

Good luck!

When the Inner JSON Effect Works

2016-07-29T00:00:00Z

Hacker News has a fun discussion today about an article on The Daily WTF called The Inner JSON Effect. It’s about how a lead-programmer mad genius built an entire web application on top of svn, not by using svn to track history, but by using a separate revision number for each individual function, and having each class contain nothing but a list of the revision numbers of each function it wants as a method. It is a hilarious example of what not to do. Obviously it’s fun to laugh at something as crazy as JDSL, but it’s even more interesting to try linking it to things we might find more normal. So a few provocative thoughts:

jsonapi? I work on a lot of Ember projects, where jsonapi is the blessed format for frontend/backend communication. It seems to me awfully verbose without adding a lot of new information. Just as the WTF story adds extra indirection where you could just define functions, so jsonapi makes you list out the attributes and references—even though in regular JSON they are right there. It feels to me like the worst of the old XML excesses. I admit this is a bit trollish, but I’ve yet to understand what I get from jsonapi. Maybe someone can explain it to me.
Ansible? I’m a Chef guy and haven’t tried ansible, and I know a lot of developers-with-sysadmin-chops love it for its simplicity. Personally I’m fearful of using something so restrictive. I have similar doubts about Puppet, although at least there I know there are good “escape hatches”. I want to be able to throw in an if statement when I need to. Here I feel my sentiment is wrong though, since these are such successful projects. So I’m curious if anyone can articulate principles that make Ansible’s YAML “work” but not TDWTF webapp’s JSON.

And going the other direction, I think sometimes—not very often—you can get huge wins out of creatively building on top of an existing tool. But it is always a risk and requires a lot of good taste. A few examples:

I built a product with user-customizable questionnaires, and we knew up front the logic for branching/skipping/computations was going to be super complicated. I could tell it was heading toward what I call a “Turing-Complete Feature”. (Enterprise Workflow Engine anyone?) Instead of trying to model and store all the rules in a database, or writing a new language (you can see the looming Inner Platform Effect), we basically made each questionnaire a server-side underscore.js template, with a simple Javascript API to access information about the current context. The feature was done quickly and we had great flexibility building the actual surveys. Our surveys were built by an internal team of not-quite-programmers, so the developers could work on other things. We even had the potential of letting customers write their own surveys, since Javascript is the lingua franca of modern programming, with good security isolation. (After all your browser uses it to run untrusted code!) Many customers told us our advanced survey functionality is what set us apart from the competition.
Another project also has user-customizable forms, with something close to an EAV pattern in the database. We wanted to offer customers reporting capabilities, so we started talking about building an API. It would be for the existing “power users”, who were doing reports and analysis using R, SAS, Tableau, and similar tools. But the API was going to be high effort, slow, and hard to join data. So we decided our API would be . . . SQL! Of course you can’t share your real tables, for security but also because if those become a public API they can never change. Plus the EAV format made them too abstract to use easily. So instead we gave each customer a private database with one table for each of their forms. This lets them ask questions in a very direct way, using tools they already know, with great performance and flexibility. (Okay I am using the past tense but this one is not actually turned on for external customers yet. Still, using a derived, flattened database for SQL reporting is not so strange, even if young CRUD developers haven’t seen it before.)
Even the idea of building on top of a version control system is appealing. (Maybe not svn though! :-) I would love a self-hosted password manager with a Firefox plugin, where my passwords are stored (encrypted) in a git repository I keep on my own server. Not only would using git save the author time, but it would be like using a “standard protocol” rather than something proprietary. Same thing for a self-hosted bookmarking service.

I think all three of these “good” examples are a little audacious, but in my opinion they all work well. So are there any general guidelines that can help us make better decisions about when to make an unexpected swerve? So much of programming comes down to taste. That leads us into pointless vi-vs-emacs wars, but it also sets good programming apart from bad. How can we make that taste teachable? Or should it be entrusted only to the old and wise? Are there guidelines we can identify, so that we can discuss them, evaluate them, and make decisions about them? (Apparently one rule is don’t write your own TCP/IP stack. :-)

One principle of my own is a “risk budget”: you have to take risks—they are a fertile source of advantage—but you should have a list of them and not take too many. Risks can include: tight deadlines, unreasonable clients/management, new clients/management, unclear requirements, new tech, new-to-me tech, high scalability requirements, etc. One risk is “cutting out a layer” like in the Javascript survey API and the SQL reporting interface. I’m not saying it’s always a good idea, just that it can be. Building your own filesystem on top of NFS is another example of cutting out a layer. I can’t speak to how well that one worked out, but it’s an interesting idea.

What are some other guidelines people use? I’m especially interested in guidelines about “when to leave CRUD behind”, or when to make a choice that seems strange at first glance but pays off. If programming were chess, when do you sac a piece?

Hello world!

2010-01-18T00:00:00Z

I’ve been tackling a lot of new languages lately–ruby, Objective C, C#, a tiny taste of lisp–as well as learning the Windows API (C++). My experience is really in the open source Unix world: Java, Perl, Python. There has been so much to figure out, I thought I’d start a notebook to keep track of it all. And why not put it online so other people can benefit? Lately my projects have been in C#/C++, so that will probably dominate the posts for a while.