The Future of Human Computer Interfaces and How We Work

idea blog

I'm going to start out with the pile of question's I asked myself as I wrote this.

What do you see when you sit down to work? My guess is a desk pressed against a wall, maybe a few shelves, one, maybe two monitors in front of you, and a keyboard and mouse dominating the work area of your desk. How do you access information? Do you simply Google something and if it's not on the first page give up? How do you get ahold of experts when you have a question? How do you interpret results that you find? How do you store information you've collected? How do you filter information to get exactly what you're looking for?

If your answers where typical at all, I don't think this is a good way to work. I think we, as individuals and as business need to invest more in our work spaces, probably well beyond what most people would even consider. I'm not talking about adding a third monitor or giving everyone an artistic environment. I'm talking about setting up a work environment that's conducive to productivity as instead of merely being the medium on which it takes place, the work environment should actively contribute to finding, accessing, retrieving/storing, consuming, and creating. - where data can be anything from art to documentation.

Finding Information

We've grow accustomed to the all knowing Google answering any request, and really, I don't see anything wrong with that. I even think anyone at Google that may read this will probably share some of this vision. I would very much like to see a world where data searching is context aware. For example, say I'm searching for a data sheet for an old vacuum tube and I have a schematic of an old amplifier open in another tab, I would love if the engine for finding this data saw the context, saw I was searching for that, and changed the label in the schematic into a link to this datasheet. Furthermore, it would be great if it crawled the web and finished finding datasheets and hot linking them, possibly well before I even even got to that page in the first place. Another point is the summery of information and omission of the irrelevant info. Say I were to look up bits in a byte, I don't necessarily need the historical context as to why bits are named bits and bytes, bytes, though even though I didn't search it, presenting that a nyble is 4 bits, and how to tell endinan-ness is more relevent, unless I had recently searched historical information or stated I wanted that explicitly. In my opinion we should be less concerned with finding relevant results and more concerned with discarding the irrelevant, though making the verbose available.

I also think that brining people together, potentially anonymously, based on search and knowledge could be extraordinary valuable. If as I'm looking up 'How to do X' and somebody else is looking up 'How to Do Y, an advanced topic from X' it would be fantastic if we could talk, albeit unobtrusively to the person being requested. It seems to me that searching is desperately lacking a social element and ranking system. How great would a simple reddit like upvote system on any given search be? What if browsers added a comment system that was hosted though some sort of distributed network and tied to each page so people could leave comments on any site regardless of functionality?

I also think the boundaries of physical and digital should be more blurred. I'd love if I could set a book on my desk and search though it for an idea or concept by mere image recognition of the cover, or if it's an unknown book at least being able to digest any pages shown to it explicitly. Say a section was highlighted? It would be great if that were automatically added to a personal journal file of sorts for future reference, especially if related data were automatically associated from online sources, or links made to people who are interested in similar subjects. The digital world doesn't have to be lonely pages indexed like a book, why are we treating it as such? Today each page can point to any other page in a beautiful web of recursively indexed information, where each topic has lines of association spanning such that no two pages are unconnected. Wikipedia sort of has the simplest form of this, but what if we had systems so capable of automatic understanding - not just tagging - of information that any new info could propagate though that web naturally. Social linkage to people in the same graph, even if anonymous, could help connect people that together, due to their very specific knowledge, drive man kind further. I should clarify to, I literally mean a web/graph, possibly in 3D, relating and indexing information, possibly like this 3D representation of wikipedia. (or this one)


Accessing Data

So, even if all of this data is able to be interconnected, distributed, and hyper-hyperlinked, what good is it if to access it is anything but simple and intuitive, like reading a book. Obviously this overly linked system is something that people would need to get used to. Until the advent of the WWW we've read information linearly, page by page. The web has allowed this tree traversal style of navigation so that any missed topic can be reviewed, but generally this is a system where the tree only build down, to simpler information, from the current node. It seems weird to think about an algebra book where it suddenly references multidimensional calculus, but this is exactly what I'm implying. In my education there was uncountable times I had to learn something because 'it will be used later' with no explanation as to how or why. Linking back up the tree allows for information traversal in both directions, eliminating this problem.

But that's still a bit of the 'finding' side of things, how should the information actually be accessed? I'm not advocating for the destruction of books by any means, but I do think the way we actually view and access this data is due for an upgrade. Sure our screens are progressively getting to be higher resolution and have better color and readability qualities- as OLED is breaking the no white text on black backgrounds rule- but I don't think this is enough. I think we need to actually fundamentally make a new type of viewing device that is more adaptable to different content styles. the 16x9 monitor needs to die, my suggestion, 24x18 (4x3), now hear me out, I'm envisioning a screen that is 2.5 of a normal monitor long, and twice as 'tall', though it's this height were the magic happens, I'm proposing the top half of the monitor be flat while the bottom half has a curve to it approaching about 45 degrees by the bottom lip, while the overall horizontal is still curved. This has the effect of basically the inside of a sphere for the bottom half and a simple curved wall on the top. This, purely anecdotally, seems to make sense to me as it would provide a good encompassing but non distorted view for video or images, while at the bottom proving a more natural angle for reading- when you read from your phone do you want it held in font of your face or lower and tilted? Exactly. Obviously I view this as being a touch screen to allow the navigation of the web of pages as well.

Beyond this I think it's time we rethink our main input devices as well. Today we primarily use three input devices: Mouse, Keyboard, and a touchscreen. I recognize a few use a pen and tablet, but I don't see that catching on anytime soon, though as with any prediction, I may be wrong. I think we should retain a physical keyboard at frequent use terminals/computers as the tactile feedback of switches (not rubber dome) is something that can't be beaten, though I think we do need to rethink the shape, layout, and even way we use a keyboard. Plenty of research has shown we have a better way, and I think it's time we start abandoning what we know to be inferior. I don't want to hammer this too much though. Furthermore, I think we need to standardize a mouse that actually has useful inputs. The Roccat Tyon, with it's shark fin in the middle, and the Logitech Mx Master with the horizontal scroll have both done very innovative things to make something that is truly much better than a three button mouse, but I think combining these ideas into one great, standard mouse could be a game changer. I also think we need to use more of our body- our feet sit there doing nothing, but imagine all the things we could control with two analog pedals.

Another thing is natural interaction without a physical connection: think Leap Motion or Microsoft's Kinect. These systems I think have limited practical use, but given the necessity for depth sensing camera's in some of what I described above it's kind of a 'why not'. Furthermore, obvious motions like twisting a knob in the air to control volume are simply convenient. Also, complex 'digitally analog' controls such as color balance of a picture could be controlled in a way that doesn't distract the user with the actual values or knob positions, instead focusing on the raw creation.

Physical controllers are also something I've debated the practicality of repeatedly, as usually they're single purpose, like a MIDI keyboard. I think these controllers, for those that use them frequently enough to justify it, should not be standardized and the form factor they use should be something where as many choices as possible are available; however, I think a much better universal protocol than MIDI, HID, OSC, etc. needs to be made. I think exposing a raw form of data in the OS that allows the protocol to be redesigned on the fly would allow for more general purpose uses of hardware, like how some people are using midi controllers in Photoshop or WiiMotes for projection mapping. Nothing but good could come from making off label uses more available, particularly for the disabled.

Another point is the idea of biohacking and body augmentation. Examples include implanted RFID tags (which I actually have) and magnets for sensing electromagnetic fields, though I think a lot of biohackers are missing the point. This is the future I see in store.

Another point is the presentation of the data. Ironically, I think text on a screen is kind of a shit method for this. More so, I think a lot of graphs and charts leave a lot to be desired. I don't want to imply AI and machine learning even more, but I think this is authentically a great application of the tech. If graphs could be analyzed and the data aggregated to produce more fitting visualizations on the fly it would be incredible. Having something that could, for example, take two 2D graphs with a common axis and turn it into one 3d graph would be incredible. In computer science it's well understood fact that any data can be represented graphically in some way, I think finding better ways to dynamically link and graph data would be a huge step in the right direction, paralytically if that data could come from multiple sources. It's amazing how many different ways there are of representing the same data too, and this can help expose otherwise non obvious trends.

Retrieving and storing data

The theme of this section is going to be decentralization. Because I think this is a subject that has been covered to death, I'm simply going to link to content that I think gets to the point I'm trying to convey

One point of contention I have with almost all existing systems is the basis on crypto currency for all storage, I think that public data should be free to store on the network, with encrypted/private data being the only thing that costs money. This has the side effect of promoting using public data, which via hashing can prevent unnecessary storage of duplicated data and the private data can be used to fund the public data overhead costs.

I think lbry is currently the closest to an implementation that I'm describing, but to my knowledge lbry doesn't host private data. I think builiding upon the ideas of lbry, having data stored as base://user/../folder/../contentname would be best, that way duplicate user names are possible, and then users can chose to either bid for that content name or use base://uuid/samecontentname if they don't want to pay, and then any private content can simply be relegated to the uuid system as well. Sharing data could rather easily be based on unix like file permissions. This idea could also relatively easily be integrated with aforementioned anonomized user profiles/chatting when finding data. Having this user account data be stored on the same decentralized network could also allow for many interesting possibilities.

I'd also like to mention the idea of distributed computation here as well, as I think it's relevant both for the sake of compression and encryption of the data. I think that having a system like 'this' - I'm referring to all the ideas up until now- in place should ask for contribution from users in turn for it's use, the obvious ask is to get it to be self sufficient. So if the distributed computational power of all these systems were used for everything I've described until now that should be more than plenty to allow it to function. This does bring up the idea of balanced usage to contribution, I think the easiest solution is to simply use a system of computational debt tied to each user account. If the user is creating more computational debt than the average debt the system can sustain then that user should be handicapped in bandwidth accordingly. This does sort of bring us full circle in 'can I just trade debt with someone, or sell them my computational time' though I don't like this ieda for two reasons: 1, this system needs mealtime computation, like electricity peak hours are worth more and 2, this incentives simply paying for compute time instead of actually contributing computational power to the network like it actually needs. Fortunately, as time goes on the amount of computational power should actually get closer and closer to scaling linearly with the amount of content produced as old content is indexed and linked into the network.


I think it's been pretty well discussed throughout this, but a few extra points about consumption of information include the idea of making strictly ethical design decisions. For example we've all seen biased user interfaces where there's something like this:

where the design is actively pursing an agenda. Instead the affirmative action should be stated on the button that triggers it and both actions given equal weight to the user:

With destructive or irreversible actions, such as deletion (not recycling), given a confirmation dialogue.

Beyond that, keeping the design minimal but powerful. I think markdown is a great example of this. User's aren't as dumb as people seem to think, we can, and do, learn the ways to make interaction with the things we use daily faster, so make the 'speed limit' as fast as it can be. Putting a frequently used option into a menu that needs to be clicked at all is much slower than assigning it a keyboard shortcut.

Furthermore, the design should promote health. An example of this is Netflix's 'Are you still watching?' while this was implemented on their end to prevent unnecessary usage of data, it has the side effect of letting a user know they've been on the couch longer than should probably be advised. I'm not advocating for interruptions at every corner, just affirmative action by the user before bombardment with data. I do think as much data as possible should be linked to or aggregated, but don't show me more than what I request plus some surface level information. For something like YouTube this might mean playing a playlist is fine, but don't start playing another 'related' video when that list is over. For something like an interactive data sheet this would mean showing the most relevant info on the first page, then linking into in one way or another (table of contents, drop down selection menu, etc) into the more detailed information, with the option to remember preferences on displayed info in the future. This actually leads well into my next point:

Information overload is increasingly becoming a problem globally. As more and more information is accessible at our fingertips and more advertisements have the opportunity to be beamed via any one of a number of surrounding screens directly into our retinas we need a way to filter it down to levels the human brain can cope with and digest.

{{< figure src="/images/times-square.jpg" caption="Bobby Mikul, Times Square :CC0(" >}}

Worse than the effect of information overload on the brain though is how the way we use technology has trained us to think in the same was as the tools we use. Thinking like I am as I write this, associating various subjects and grabbing bits and pieces form my life of experiences is looking to me to be a skill that people don't use as much when using technology. Don't get me wrong, I think as time goes on people are not only getting smarter, but also getting better at this abstract thinking but I think online, when using technology, these skills are just applied less and we resort back to the linear associations between things. I'm saying this completely anecdotally, partially due to a lack of research. (or at least any that I could find. Then again I'm an engineering major, not a psych major, I may just be missing the right words to ask the question) I think we're just now approaching the stage where instead of merely extending our linear thinking capacity- our memory and computation skills- that computers, as tools extending the mind are starting to be able to extend our abstract and relational thinking as well, so that we can do all the amazing things that come from that type of thought. I fully expect this to be a controversial idea, but I'd love to hear why you think otherwise if you do.


Creation vs consumption is an ever fought battle. How should you spend your time? should it be balanced? should you create more than you consume? Obviously we're hitting hard core life choices and philosophy here. Truth is, I don't care as long as you're (I'm) happy. I love going into YouTube induced comas on a semi-regular basis, but if I didn't also have a creative outlet I think I might explode. Thankfully to me, that creative outlet doesn't need to be something that's classically artistic, but rather anything from programming to making these blog posts works for me. What I will say is for those that chose to create, having the best environment possible, with the best information, tools, and space around them is a huge boon to the output and quality of work. I'm not going to bother finding the evidence because I don't think I need it to support the claim that when we're happier we work better. So, that begs the question, what makes us happy? Look, I'm not about to claim to know the meaning of life here, but I think I can at least point out some relatively obvious things and state what I want in a work environment:

First of all is a low noise floor. This is sort of an odd one, as different noise matters in different ways dependent on its 'musicality' that is it a repeated pattern, frequency (low is less annoying than moderate, but high is much more annoying than low), etc... For example, right now I'm in a room that has a rather loud computer on part of its fans, giving this room a noise floor of about 50db, making it about the same voulme as a large office but because that sound is roughly consistently at 120hz, it's an at lest not unplesent background hum. Sadly, on the rare occasion I have that computer off (Let's not talk about how much time I spend in this space) I can actually almost physically feel the change in atmosphere, and is undoubtedly relaxing.

Next is adequate space for interruptions. While I'm a strong proponent of not eating where you work or consume media as I think it should be an either social or self reflective time, I understand that sometimes it's necessary, and there's nothing worse than not having a flat surface to put your bowl of soup on. More practically though, as mentioned before, it's ideal if the mouse and keyboard aren't in the way of desk space that would otherwise be used for physical craft, note taking, art, etc. So I think three spaces total are ideal: one for primary input devices- today that's a mouse and keyboard -; a second for papers, a main project, etc; and a third that is easily accessible added for the interruptions and side projects in life.

Next is a visually appealing space. Wires dangling over things, pealing paint, unorganized shelves, etc. are obviously off-putting, but I'd go the step further to say they actively interrupt productivity as they stick out and beg to be fixed. It's the standard scenario of not wanting to do homework until the room is clean. But idealy, I'd go a step further. I think a nice minimal design that accentuates useful things is a good start. Adding a bit of tactile flair can go a long way too. I personally don't want art or static words in my work space as again, they just distract. A bit of sound dampening foam on the wall your facing can go a long ways both in the visual and sound department, and it's pretty cheap too. Until my dreams of a monitor utopia come, a good start is just getting rid of the base and using a VESA mount to the wall or back of desk. the flexability in position and extra available desk space goes a long ways, and it's much, much more visually appealing. Rather paradoxically, I do see value in motion in the workspace too. For example, MIT's reactive table or those fancy marble in sand tables can add much needed visual motion to prevent a space from becoming stale. Hell, even a simple fish tank or plant that adds a bit of change with time make's a huge difference.

To round off the environmental side of creation, and arguably most important is lighting. In recent years redlight shifting/ blue light filtering of screens has come into fassion and for good reason, our eyes and brain are tied together into a biological clock, and it turns out, we can manipulate it pretty well based on how we set our lights. But at the same time the light can be a huge problem too. Without going to in depth white≠white≠white≠white... There's cool and warm and natural white sure, but the 'quality' of this light varies a lot too, there's a lot to be said for getting high quality lights that put off a natural spread of sunlight emulating frequencies instead of just peaks that make a white, this actually makes photography way better too.

Next I'd like to talk about the tools used for creation. This is where I really have a hard time deciding about how money should work because on one end I could just say "Everything you use should be free and open source (FOSS)!" like a hippy, but honestly I don't believe that. What I support instead is free to use personally and open source software that takes a cut of any profit made in return for usage at any sufficiently large corporate scale. This would mean your average joe is free to get their hands dirty with professional software and that in turn means they have experience with and would prefer to continue using the same software for anything commercial. It's a win-win for everyone.

As far as how all of these tools should work and interact, well, I think they should all use standardized file formats, even if they have to abuse them a little to do so and that they should all have a common Application Programming Interface (API) for interaction. This would hopefully mean that any extension written for one program would work for another, and any program could talk to another. Currently the world of music software has a little bit of this but it still leaves a lot to be desired. I'd actually like to take it a step further though and ask that all data of any kind use a common enough format that it can be processed with any extension/program written with this API in mind. Imagine if you could use a synthesizer as a static generator for image manipulation, or color management controls as an EQ. Both would and should behave in strange way, and it's this very lack of defined behavior that could lead to interesting art forms. I'd love to see a 'Master' Api that works across all formats and ideas with a common data type that allows for program⟺program, program⟺extension, program⟺hardware, etc. communication even in long, complicated chains, in any nodal connection system: (


Potentially this could also plug into the entire OS as well, making it so an image manipulation program's extension could for example modify anything output to the screen in real time, or an audio program could effect the output of anything. For developer's this may even offer more power, making possible things such as interprocess communication (think pipes, like $ls -la | grep *.png) a matter of connecting two nodes, or reading disk information such as activity, space, or even writeback and inode information, this would literally allow any one piece of information to be accessible to any other. This does have obvious permission issues, but unix permissions should already have this under control. If something like this could also be tied into the originally mentioned web searching and socialization web without massive security concerns the potential use cases are as simple as getting color information from an image hosted online to as complicated as remote access or distributed computing.


or even supporting a full programming environment sort of like Luna:


Wrapping up

In all honestly I'm not exactly sure what everything I just wrote is about. Mostly it's just a brain dump, but hopefully it's an interesting one. To round things off with a bit of a closing note though, I don't actually foresee many of these things being possible, if not simply because they're require so many people to agree on standards, but there is one glimmer of hope, and it's one of proof of uniformity. The terminal. Yes. This terminal:


The terminal emulator above is still compatible with the VT220 from 1983 (as are most terminal emulators) yet from it I can do everything I can really think of: browse the web, chat with friends, listen to music, basically anything. I'm not saying we should all stop using chrome (You totally should though) but I think part of the reason so many neck beards and sys admins still use the terminal is you can do so much with it, and because everything uses it as a common interface and it has programming capabilities (or at least bash/zsh/whatever does) you can automate or string together just about anything, exactly as I described above. In fact, I think it would be amazing if all the graphical 3D node based exploration and data flow editing I described above had underlying syntax that could be written directly for the nitty gritty when desired, sort of like aforementioned Luna.

Finally, I'd like to say I understand we don't all get the choice, be it by monetary, physical, or other restrictions to have a 'perfect' work environment. If you live in the city, there will be noise, I get that. Obviously I don't expect everyone to go out and make their own versions of some of the high tech borderline art installations I linked either. I also don't think everyone's down to get an RFID tag in their hand. I just wanted to present what I see as 'the future' weather it comes in 2018 or 20018. I do, however, hope this has inspired you to look at the way you work, the environment you work in, and how you can improve it. Weather it be by switching software, tidying up those cables, or making a bad ass desk, I hope something comes of this post.

Other interesting and sort of related things that didn't seem to fit anywhere nicely

ReacTable, Netsukuku (2),

Previous Post Next Post