We are visual animals. But before we can see the world in its true splendor, our brains, just like our computers, have to sort and organize raw data, and then transform that data to produce new images of the world. Beginning Python Visualization: Crafting Visual Transformation Scripts talks about turning many types of small data sources into useful visual data.

The about page adds some bullets, in particular this one: “Write ten lines of code and present visual information instead of data soup.”

Yes! We’ll never get rid of lists and columns of numbers but a simple visualization is often all you need. Sparklines? Little additions. Make Tufte proud.

I think this data soup problem is responsible for me wanting to spend more time with Mathematica. Mathematica makes it relatively easy to do visual data analysis. Check out the Flag Analysis with Mathematica post from the Wolfram blog for a good example.

In a nutshell….

3rd party bucket apps make it easier to actually grab stuff while native tools are generally multi-step processes. Most of the bucket tools emphasis one-click storing or provide easy access to system-wide scripts.

This might sound crazy, but the filesystem really isn’t a very good general storage system. It requires up front thinking and decision making. Features such as links and aliases only give you so much flexibility in changing your system later. Alex’s article sorta waffles on this point.

The Finder is only the bare minimum for an everything bucket client. 3rd party tools, like Pathfinder, get closer to meeting the needs.

Currently an unadorned Spotlight barely meets the minimum needs for search. With some 3rd party tools and adding some constraints it can be better. But out of the box, no.

Here’s my story….

One of the first things I installed after getting my Mac was Devonthink. After the indignity of Windows and OneNote, I felt I finally found my home with Devonthink. I always just assumed a dedicated catch-all was the way to go. Devonthink is my everything bucket.

But Devonthink is a true island and totally opaque to OS X. It needs to be running to get things into it. Searching happened inside too. Granted it was a really fast search, much better than Spotlight in that regard. But again, it had to be running to be effective. Eventually, I found myself not using it as rigorously as I once did. I went in spurts. Days with it running, days without.

Over a holiday break last year I thought about how I was using Devonthink. As it turned out my use had degenerated to nothing more than collecting & filing webarchives, PDFs, images, text snippets. Nothing more. No fancy AI. No outlines. Nothing really Devonthink specific. If that’s the case, I reasoned, there’s another path available to me.

From Devonthink I exported my entire database to a folder on my hard drive. Everything. And then I began piecing together a system that mimicked what Devonthink was doing.

What I learned….

First, get your story straight. If this was a research project, you’d have a very clear statement describing what it is you’re after. Same here. If you google around for Devonthink stories you’ll find success comes from a pretty focused use of it. Steven Johnson, as an example, uses Devonthink to save text clippings and notes and then uses the fancy clustering/relating tools to find interesting connections. His use isn’t something I’d call an everything bucket.

So my goal was to manage the rich media and chunks of text I was keeping in Devonthink with the file system and native & 3rd party apps. Eventually, I settled on these as the most important pieces:

Leap
Pathfinder & contextual right-click extras
Quick Look extras
Applescript

Leap is just a fantastic tool for digging through your hard drive. Leap makes Spotlight something special. Since I restricted it to just my Devonthink archive it was always fast and immediate. Leap can do tagging but I never really used it all that much. The standard Spotlight metadata is actually incredibly rich and provided enough flexibility for me to find and group what I needed.

With Pathfinder as a Finder replacement, I relied on the tabs and the ability to create new files in a given folder. Simple tasks lacking any native OS X analogs oddly. But Pathfinder sucks up RAM so I’d have a back-up, NuFile, when I felt like living an austere Finder-only life. NuFile adds customizable right-click context menu file creation. So if I was in my RFID folder and wanted to add a new note I could just right-click, create a new text file and edit in TextMate.

Devonthink made it easy to see things since it had native viewers for every sort of media you could import. My Mac, on the other hand, has a smaller set of native viewers. Luckily with Leopard you get Quick Look and a large library of extra 3rd party Quick Look plugins for viewing non-native things. I’ve installed plugins for webarchives, Excel, iWork, zip files, etc. etc. Tapping the space bar generally brings up a preview of whatever it is I happened to have selected.

I had high hopes going into this project that I’d be able to quickly glue a bunch of things together with Applescript. The best I was able to do was create some Safari scripts for saving off webarchives to a common Inbox folder. The biggest hurdle wasn’t the Applescript language so much as the inconsistent treatment the various apps gave Applescript. Safari, for instance, is pretty bad while iTunes is really good. Devonthink ships with a great collection of very useful scripts. Furthermore the Devonthink support forums are full of very helpful script writers willing to provide code; really top-notch.

So my cobbled together system was working for me. I had various ways of looking through my archive, both native and non-native. I had various ways of finding items in my archive. And I had many ways of creating new items in my archive. Things were good. I was living with this low-tech system for many months.

And then the Devonthink people had to go and create v2. The latest goes native! Stores items as-is in your filesystem, uses Quick Look for viewing, uses Spotlight (but doesn’t rely on it) and provides a universal inbox that doesn’t need the app running to work! I’ve been using the betas and liking it–my biggest gripes have been seemingly answered. Once again, time for some hard questions.

Ultimately, I agree with Alex with one caveat. One app to store everything doesn’t work for me. However one file system doesn’t work for me either. Your particular situation will drive that decision. What’s important to you? Want a quick one-click way of grabbing stuff? Need a lot of flexibility in structuring and organizing stuff? Have to handle fancy exotic media? Need to search through gigs and gigs of data? Hey maybe Excel is all you’ll ever need.

Studios are a bit like VCs. They even share similar language, “green lighting” works for both movies and start-ups. They both invest in something risky with the hopes of a big pay day sometime soon. They both construct hand-picked teams to finish or ship. They both exert subtle and not-so-subtle control on the creative direction.

The various tech conferences/showcases like Demo or Techcrunch act like film festivals. There are winners and losers, panel discussions, buzz generation, etc.

Both industries are equally clueless when it comes to predicting success.

Granted, I’ve done nothing but list a few superficial similarities but this idea seems pretty obvious, others surely have written more. This is a tough thing to Google tho—no matter what terms I use I keep getting Marc Andreesen’s Rebuilding Hollywood in Silicon Valley’s image. Not quite the theme.

There is, however, a book called Hollywood Economics: How extreme uncertainty shapes the film industry by Arthur De Vany.

Just how risky is the Movie industry? Is screenwriter William Goldman’s claim that “nobody knows anything” really true? Can a star and a big opening change a movie’s risks and return? Do studio executives really earn their huge paychecks? These and many other questions are answered in Hollywood Economics. The book uses powerful analytical models to uncover the wild uncertainty that shapes the industry. The centerpiece of the analysis is the unpredictable and often chaotic dynamic behavior of motion picture audiences.

It sounds like a fun project to mis-use De Vany’s math by shoehorning start-up industry numbers into his models.

Just to show you can’t extend this analogy too far, here are some places where it breaks:

Scale: everyone is equipped to see a movie. Most start-ups never get bigger than your college graduating class.

Impact: a great movie makes an emotional connection to the viewer. Short of the Pets.com sock puppet I haven’t made an emotional connection with a start-up since.

Success: a movie’s success is measured by the amount of money that trickles in over its lifetime, a start-up’s success is that it goes public, a big chunk all at once.

Amble Time / “A shortcoming of standard maps is their inability to convey a sense of temporal scale. Can I stroll to the park for lunch, or would it take me all day? Amble Time adds an element of time to a PDA-based tourist map. By using a GPS system and your average walking speed, it creates a bubble that indicates everywhere you could walk in an hour.” This could be a lot of fun.

Don’t bother clicking, the url is kaput. One quick Google finds Amble Time alive and well:

A steadily shrinking area of a city map shows where you can walk as time ticks by. The bubble shows everywhere you could go within timing constraints that you provide. Researchers used this “travel-sensitive alarm clock” to explore ways that location-based information and ad-hoc networking could support participation in interactive stories.

There’s even a nice PDF available, Time, Voice, and Joyce, that discusses the project Amble Time was built to support:

We present a design for recapitulating walks through Dublin’s City Centre by characters in James Joyce’s Ulysses. Our computationally supported walkers will avail themselves of a “map with a sense of time” and a system that translates their hand lettering gestures as attributes of colourful typographic forms.

Now I definitely have to see it working again. Amble Time has iPhone written all over it.

Testing out some new Tumblr-like code.

This is a time-lapse of an Adobe Illustrator master doing his thing. One of my favorite new Portishead songs provides the soundtrack.

Biomimicry is a survey of solutions the natural world has come up with to solve problems. Everything from spider webs to photosynthesis to the way chimps medicate themselves in the wild is covered. The end chapter takes a look at what we could learn from nature and apply to various industrial processes. After all nature works at room temperature, gets power from the sun and generates no harmful by-products.

Gabriel’s essay takes a look what it would take to build Ultra Large Scale Systems, which he defines as something impossible to build because of today’s software engineering technology. For impossible, substitute “trillions of lines of code, millions of computers, real time requirements with life critical applications.” His essay spins off on a number of fascinating tangents and brushes upon the natural world, both from a civic planning and a biology perspective.

So from Biomimicry, we see an end result (e.g. oyster nacre, incredibly strong, stronger than man-made composites, safe) without really knowing how to recreate and apply it. And in Gabriel’s essay, he makes a good case for extending the Biomimicry lessons and applying them to these ultra large scale systems as well. What are the technical analogs to “room temperature, sunlight, water and no harmful by-products” for ultra scale system design?

From A New Kind of Building?, Maurice Martel “was interested in generating architectural structures subject to spatial constraints (such as a given area in which they need to fit).” So he “settled on a project in which he would run 2D cellular automata on irregular grids determined by arbitrary polygons.” His ultimate goal was to design an actual 3D structure.

Using Mathematica he was able to go from this:

To this:

I think what’s interesting is just how benign the result of the Mathematica-assisted design process. Give certain architects free reign, no restrictions and no context and you could very well end up with more folly than form (or function.)

Here’s a short description I found (dunno if it’s an official description but I like it), “Twitter is for staying in touch and keeping up with friends no matter where you are or what you are doing.” (Incidentally, found that while looking for a Mac app for dealing with Twitter.)

I don’t have a problem staying in touch and keeping up. I do plenty of IMing & IRCing & emailing & SMSing. Sometimes I even use the phone. For those more distant friends (for whatever definition of “distant” you prefer) there’re weblogs to read and Flickr photo streams to follow.

I do have problems with the second part of the quote: “no matter where you are or what you are doing.” Last time I heard something like that was when I was 10 and my Mom was reminding me to call home if I was going to be late. Since when does being online & available require real-time Intensive Care Unit-like availability? If you have to let me know you’re going to the bathroom then we’re getting too close. Let’s put some healthy distance back into the relationship.

It frustrates me that I can’t figure it out; many folks who I respect are off and running with it, seemingly have a grand time. What am I missing?

I think, for lack of a better term, this “context smearing” is something I’ll have to get used to. Perhaps one of the reasons Twitter seems so in your face is because I lack the means to actually make it useful. Every single operation involved in a “twittering” involves me actively detaching from whatever context I was in, starting up another tool (or picking from a number of different windows already open), attaching to another context, detaching again and then flipping back to the original context so I can finish whatever it was I was doing. This isn’t multi-tasking, it’s something more akin to meta-multi-tasking. And it is exhausting.

Twitter makes it easy to broadcast my presence across IM, email, the web and my phone. Regrettably it feels more like Twitter is broadcasting my already feeble attention span everywhere as well.

In a nutshell, this little script grabs up to 15 items recently tagged “readinglist” from my del.icio.us account, applies some simple formatting and then builds a 3×5 PDF suitable for your Hipster PDA. It’s stream of consciousness coding with lots of hardcoded stuff, but it works just fine for me. Bugs exist, e.g. it’ll get confused if you use certain characters in your item titles. I had a few extra minutes so now you can also build arbitrary lists, like this or this. The hardcoded formatting can either work for or against you.

Of course, there’s no support for this. If you can find something useful out of it, great. Otherwise, sorry.

Interactive City Summit on the 7th & 8th.

ISEA2006 from the 7th until the 13th.

Survival Research Labs on the evening of the 11th.

WWDC06 from the 7th until the 11th.

Going to go? Let me know.