Steve Jobs “just says no” to Flash on iPhone. Well, on first glance, that’s just what he says now, and we all know, like a good episode of Lost, there’s always more to unpack and nothing is what it seems. Considering Adobe’s product line, the so-called “missing middle Flash product” suitable for the iPhone doesn’t yet exist. The middle product refers to something between Flash Player for the desktop and Flash Lite for mobile devices. But, considering the pipeline, it’s only a matter of time before Adobe AIR Mobile hits iPhone and just about every other mobile device, smack dab in the middle of the entire mainstream interactive media ecosystem.¹

Despite all the nay-saying, Adobe seems determined to get Flash on OS X Touch.

The brilliance of both Apple and Adobe waiting for Adobe AIR Mobile to launch is that it addresses all issues and pleases both parties politically:

• AIR Mobile is likely to be built upon AJAX, WebKit and Flash Player with the ActionScript 3.0 VM, as it is on the desktop, so it will be robust, efficient, modern and support both Apple’s and Adobe’s standards. It will please developers, designers, open standards proponents, lovers of proprietary goodness, and every other regular user who just wants everything to just work.
• AIR Mobile will solve the installation and distribution problem inherent in Flash Lite. Flash Player for the desktop has never been about standalone application installation. AIR on the desktop bridges this gap. It’s only a matter of time before Adobe sends AIR into the mobile device space and allows creators to put mobile app icons on standby screens with a few clicks. Just think Apple AppStore, or jailbreak AppTapp for Adobe Mobile AIR.
• AIR Mobile will bridge the fading distinction between web – desktop – mobile by allowing creators to write software in one environment (eg. Flex Builder) and distribute codebase to all three of these crucial platforms in a truly hybrid sense.
• It’s really no coincidence that Apple and Adobe are both investing heavily in products named “Air” — the notion of the data-cloud, cloud-sourcing, everyware, and webware is nascent. Narrow-minded jargon-lovers will call it Web 3.0, but intelligent folks will hopefully leave this lame version number moniker behind and use the appropriately visionary language espoused here.

Multi-touch API is the key

The central unresolved issue that remains is a multi-touch API. Any version of Flash for iPhone will need to have its intrinsic APIs updated for multi-touch and that will need to translate to a higher-level ActionScript object so that designers and developers can trap events related to multi-touch gestures. Without gestures like pinch, flick, zoom and others, it’s really pointless to put Flash on iPhone.

At the end of the day, talk about politics, performance and battery life are probably just the red-herrings that both Apple and Adobe need to work out the vexing issue of multi-touch APIs. In fact, it will probably take a few years before all platforms (Windows Mobile, Symbian, Android and iPhone) all reckon with multi-touch on all levels of hardware and software.

The alleged YouTube Mobile Safari plugin in OS X Touch 2.0 beta is probably all the beehive needs to chill out and give Apple and Adobe the breathing room they need to get multi-touch worked out, and deploy Mobile AIR on a dizzying and divergent array of devices, platforms and crotchety carriers.

Previously:

• YouTube plugin for Mobile Safari Suggests No Flash in iPhone 2.0←
• Will iPhone Ever Run Flash?←

Footnotes

1. This is based only pure postulation and not informed by any confidential Adobe insight.

Specula-palooza Rocks On

The latest in Flash and iPhone rumor-mongering suggests a YouTube plug-in for Mobile Safari will accompany this summer’s Touch OS 2.0 update. An uncorroborated claim indicates that this plug-in is contained within the recently seeded developer iPhone SDK 2.0 beta firmware.

This would theoretically enable embedded YouTube movies to work on the billions of pages that currently flaunt the dreaded blue question mark icon. Presumably, a page with an embedded YouTube SWF player might show a thumbnail with a play icon that when tapped would load the clip in the native YouTube player or some embedded player within Mobile Safari. This would be similar to what happens when you load an embedded MP4 video file via the native QuickTime player within the iPod function.

In technical terms, Mobile Safari may parse the OBJECT and EMBED tags that point to the YouTube SWF player and redirect the path to the video into its own native player.

It’s important to note that not all YouTube videos would work. In fact, numerous clips not encoded to MP4 H.264 remain only available in the Flash codecs Sorenson Spark and On2 and therefore would not play in the Mobile Safari YouTube plugin. Although, Apple and Google seem to be colluding to convert the vast user-generated video library over into the H.263 format. It’s truly doubtful that OS X Touch 2.0 contains additional video codecs beyond H.263.

However, rather than subscribing to comment stream appearing below rumor sites, this YouTube Mobile Safari plugin is very likely not a clue towards a Flash player for iPhone. In fact, this could be the kiss of death for the chances of seeing it soon.

This kind of Mobile Safari plug-in access is precisely what Apple is shielding from Adobe (and other third-party developers, thanks to the sandbox intrinsic to the SDK). In many ways, a YouTube plugin acts as a trojan horse to usurp dominance from Flash in favor of WebKit and open standards. For most people who crave a Flash Player for iPhone, it’s the frequent dead-end to embedded YouTube clips that has them most irked. Once that kink is worked out, will users really miss Flash?

The other crucial importance of a Mobile Safari YouTube plugin is that it bypasses the main sticking point: how to implement multi-touch via an ActionScript API in the Flash Player!

We’re not suggesting Flash Player won’t ever appear on OS X Touch, but if a native YouTube plugin appears for Mobile Safari this summer, it has big implications for Adobe’s mobile strategy, and concerns the long-term viability of Flash as a de-facto standard if the mobile medium cannot be captured.

We’d love to see Flash on OS X Touch. There is a vast designer and developer community out there fluent in ActionScript that would thrive in a Touch world. It’s all about a multi-touch API from here on out. We’ve tinkered with both Mobile Safari web application design with WebKit and AJAX and also attempted to pick apart Cocoa Touch. There is no middle ground yet, and the Touch application market is nascent and fractured as a result. You’ve either got really crappy web applications or sketchy jailbreak apps. Yes, this summer’s launch of the AppStore will change the game forever. But, until the vast Adobe-enabled developer community is employed to create, the market will be constricted by limitations and learning curves.

Previously: Will iPhone Ever Run Flash? ←

It has been awhile since posting to mercurious. However, we’ve been hard at work updating our academic site. This site now documents recent student work:

• Freshmen Processing work
• Muzzi, the digital pet —music player hybrid
• Info Wraps, a mobile informatics research project

It also now features a commercial work samples page that highlights an eight-year career in the interactive media industry. Perhaps the most interesting additions include ancient works from the vault, some almost a decade old.

In fact, getting this site together in this manner has been long overdue.

» New and improved professorial site on parsons.edu

On Teaching Mobile Application Design at Parsons

Dr. Dobb’s Journal, legendary software design publication, published our phone interview on its web portal yesterday on the subject of teaching mobile media design at Parsons Communication Design & Technology. John Dorsey, the editor of Dr. Dobb’s, offered probing questions that got at the nature of the program, the challenges of teaching Flash for application design, and how various tools and platforms can fit together in a technology curriculum at the service of the arts. This is all very interesting when you consider Dr. Dobb’s target audience, clearly hard-core coders and application developers. It’s a good sign when engineer-types are starting to take what’s going on in art and design schools more seriously. These distinctions are very much starting to blur.

» Interview

Web Ink for Mobile Media Curriculum

Adobe published a case study on my work with students at Parsons Communication Design & Technology in the area of mobile media design.

Specula-palooza

Risking iPhone coverage overexposure, today we ponder one of the most interesting questions about the future of Flash, iPhone and web standards. Despite assurances by Uncle Walt that Apple and Adobe are hard at work on a Flash Player for iPhone, plenty of naysayers, skeptics, and player-haters have voiced strong speculations that Flash will never appear on the iPhone for strategic, practical and technical reasons. A quick scan of comments on various iPhone related entries across the web reveals an almost universal plea amongst everyday users indicating that a dearly missed feature from Mobile Safari is the presence of a mainstream multimedia plugin. In fact, the world’s most popular piece of software in history, is well known to be absent from iPhone.¹

Mobilizing the Means of Production

Those who have written about why iPhone should not have a Flash Player don’t mask their agendas. These voices are usually programmers and developers who have always been hostile to Flash, mostly because it threatens their grip on the means of production, by bringing software and interface design to the masses. Indeed, we’ve seen the worst and the best of the web, as a result. Furthermore, because Flash has always been a mainstream, populist, albeit proprietary media format, it has been deployed for reach and ease of creation, rather than robust performance. When imagining a Flash Player for iPhone, its high-octane thirst for processor cycles does not bode well for battery life.

Monopolizing Web Standards

The second significant argument against iPhone Flash is Apple’s strategy to deploy its WebKit “web standards” platform for all third-party application development. Indeed, the recently redesigned Apple.com site has reduced its use of Flash significantly, in favor of JavaScript and browser based features, so-called AJAX. The argument goes that Apple doesn’t want to forsake its influence on the consumer-level interface design market by inserting Adobe’s trojan horse into the battlefield.² This perspective does make a lot of sense, but leaves out a tremendous amount of nuance, that we’ll investigate here.

A Muted Voice

The assumption that web standards based technology, such as JavaScript, can wholly replace Flash functionality is only somewhat true, especially on iPhone.³ Certainly, the support of JavaScript and its embrace by the web developer community over the past three to five years has changed the face of the web, earning the popular title of version 2.0. However, Flash offers some essential multimedia capabilities that JavaScript alone cannot yet offer. This includes audio support. It’s no coincidence that Bejeweled for iPhone is mute. Especially considering that many iPhone users may have stereo headsets plugged in during use, there are unimaginable uses for audio based applications, especially when combined with locative media technologies. Imagine a sort of audiopedia. The differences don’t start and end with audio support, however. Even Apple’s newly touted Web Gallery feature, part of a .Mac subscription with the iLife personal media suite needs to use Flash for its carousel photo browsing interface.⁴ Indeed, only the Flash Player offers the multimedia engine to manipulate the images as a responsive interface with the reach required on this consumer grade product.

I Want My MTV

Certainly, the most common deployment of Flash Player on the web recently is for web video. It was almost shocking to watch the FLV video format surpass RealPlayer, Windows Media and QuickTime as the most important, influential web video format, in what seemed like a matter of months, with much thanks to YouTube and others. A recent Adobe Flash Player 9 public beta featured H.264 video support, which seems part of its strategy to preserve the dominance of Flash video, especially as Apple and Google migrate towards this non-Flash based video standard. However, until the myriad of embedded SWF FLV players on perhaps billions of web pages get updated to auto-detect the client and deliver the appropriate video by codec, the web will still appear to be littered with missing plugins on Mobile Safari.

Assumptions and Speculations

Let’s proceed with the assumption that Walt Mossberg was correct, and indeed, Apple and Adobe have reached an agreement to release Flash Player iPhone in some manifestation, at some time. Of course, he could be blindly speculating like the rest of us, just running on the fact that it feels crazy for Flash not to be there. But, let’s hope he’s as well-connected and respected as they say he is.⁵ There are several scenarios for the future of Flash iPhone, which should only contribute to the over-saturated discourse by further complicating the biased opinions with an understanding of Adobe’s perspective, previously, and conveniently left out of the discussion.

A Straight Port of Flash Player 9

In this scenario, Adobe compiles the current version of the desktop Flash Player 9 for the ARM processor of iPhone. This scenario would allow all of the existing web-based Flash content to function within Mobile Safari. Authors who create content in ActionScript 3 would enjoy a noticeable improvement of performance and energy efficiency on the iPhone, since this type of content would play in the more recent Flash Virtual Machine, a marked improvement over previous runtime environments, across Windows, Mac OS X and Linux. However, the vast majority of Flash content on the web right now was created in ActionScript 1.0 and 2.0, and so does not take advantage of the improvements to the runtime. Indeed, the skeptics are correct when they assume that running web pages with Flash content, even animated banners, would put an unfortunate drain on the battery.However, what’s crucially missing from a simple straight-on port of Flash Player 9 for iPhone is a substantial confrontation with the multi-touch interface. This is likely the deal-breaker for this scenario. Although, ActionScript and Flash button symbols might offer some means of developing and designing for the iPhone’s multi-touch interface, it’s more than likely that despite the best efforts of Flash coders, the existing means to respond to mouse-based interactions will fall short of the requirements needed to respond to multi-touch. Gestures like pinch, tap-zoom, and flick are difficult to imagine as ActionScript events. Indeed, Adobe probably needs to reckon with the reality that ActionScript needs a true multi-touch API. This could be one of the many reasons why we can only assume that iPhone Flash is in development now, and may be for quite some time, especially based on Apple’s delay in releasing an official iPhone SDK.

Probability: Very Low

Flash Player 10

Looking into 2008, perhaps third or fourth quarter, we have to assume that Adobe will continue to release improved versions of the Flash Player. Version 10 is likely to provide Adobe with the required release cycle needed to fully contend with Apple’s native iPhone application API and SDK release schedule. It’s much more likely that ActionScript libraries will be written that allow true response to the multi-touch gestures, such as pinching, flicking and zoom-tapping. It’s really difficult to imagine a Flash Player on iPhone without this crucial ActionScript API.Furthermore, this allows Adobe to potentially release new versions of the Flash and Flex authoring tools that will compile in Flash Player 10, and subsequently, support a runtime environment that is tuned to the needs of the ARM architecture and precious battery life. It’s been frustrating to read the Flash iPhone haters and their blatant neglect for Adobe’s expertise in the area of building a Flash Player for mobile devices. There has simply been no mention of the possibility of Flash Lite for iPhone.

Probability: Medium

Flash Lite 2.1

Flash Lite is a very special version of the Flash Player for mobile devices, such as Symbian Series 60, Windows Mobile and others. It is not the Flash Player that typically sits in an embedded browser, like the vast majority of Flash content out there. Instead, Flash Lite content exists as standalone, full-screen, mobile applications, or more appropriate mobile media implementations such as a standby screens, wallpapers, screen-savers, and even the device’s native UI.Not only is Flash Lite compiled to the particular device, and so is limited by its processing and memory capabilities, but it also has proven to be very energy efficient, accordingly. Indeed, authors must specifically design and code for Flash Lite. It is in no way, a conversion from desktop Flash content. In this manner, designers and developers alike, are rightly forced to contend with the requirements of mobile media, in terms of interface, content and use-case considerations.It’s possible that Adobe could port Flash Lite to iPhone instead of the expected desktop Flash Player. In this regard, Flash Player would exists as a widget on the SpringBoard home-screen of iPhone, and not as a Mobile Safari plugin. Although it would not fix countless broken plugins in pages that use Flash Player, it would offer mobile media design opportunities for Flash on iPhone. Specifically, Flash Lite already offers APIs to interact with mobile phone specific features, such as triggering the vibrate, detecting battery life, and cell network signal strength. These capabilities are not offered on the desktop version of Flash Player, and may or may not be available in Apple’s official native iPhone application API or SDK. Only time will tell.In this scenario, Flash Lite becomes an avenue for designers and developers familiar with Adobe’s toolsets to create applications that exist as standalone native experiences, rather than embedded modules of Mobile Safari. In many ways, this supports the special needs of mobile media more appropriately than simply making the familiar Flash just work. Imagining a port of Flash Lite for iPhone could mean the opening of a vast market for native iPhone widget applications, designed by designers, and not restricted to hard-core Objective C programmers. Indeed, this is a threatening prospect for those that are eager to carve out a niche in iPhone application development. However, Flash Lite 2.1 probably does not offer the means to react to multi-touch gestures like pinch, flick and tap-zoom, and so it’s probably not the version we will eventually see.

Probability: Low

Flash Lite 3

Just as the desktop Flash Player is constantly updated, the mobile Flash Lite player is also expected to be evolving. It is said, that a forthcoming version of the Flash Lite player, perhaps 3, will bridge the gap between the embedded browser Flash content and the standalone mobile specific Flash Lite content.⁶ In other words, Flash Lite 3 could play not just in the mobile device’s web browser, but could also run within its native operating system environment. This jives with Adobe’s efforts to seed the use of Flash outside of the browser and distribute a desktop based native runtime, called AIR, the Adobe Integrated Runtime. In addition, a Flash Lite 3 would address the concerns of a multi-touch API, and the required energy efficiencies for battery life, memory usage and processor cycling, as well as provide ActionScript to trigger mobile device specific features intrinsic to Flash Lite.In this case, Flash would exist in two manifestations on iPhone: as standalone native applications on the SpringBoard home screen, and as a Mobile Safari plugin, playing the usual desktop based content. This gives developers, designers and end-users the best of all worlds. Flash content can be created with mobile use in mind, considering the unique user interface and energy efficiency required. Everyday users will enjoy not only the full multimedia web they’ve grown accustomed to on the desktop, but also will enjoy a market of native mobile applications that arguably, the Objective C programmers of the world, simply cannot singlehandedly service.

Probability: Medium-High

Conclusions

Flash will appear on iPhone eventually. There is no doubt that Adobe will roadmap this device into its strategy for Flash Player, Flash Lite, or both.Yes, there are significant performance, interface and user considerations that must be addressed as Flash appears on iPhone. Adobe has already demonstrated an accomplished ability to service the mobile media market. It’s only a matter of when, and what form iPhone Flash takes as it appears beneath our beloved glass multi-touch screens.One can only imagine the pressure Adobe’s product planners are putting on its Flash Player team to fit iPhone into the needs of the short-term and long-term strategy. In the short term, Adobe needs to get Flash on iPhone within the next three to six months to capture the required developer and designer audience, and compete with the ascendancy of Mobile Safari and native iPhone applications. In the long term, Adobe needs to get it right, and release a iPhone Flash Player that addresses the specific needs of both mobile media and the vast legacy of desktop Flash content out there. A premature release could spell long term disaster for Flash, as it needs to compete with the rapidly expanding Open Source and Web Standards movement. We haven’t mentioned Microsoft’s entry into the web media space with its recently launched Silverlight platform, but until we stumble upon a site that actually uses it, it’s irrelevant.Although there are many who would like Flash to just go away, because it’s not open source, not free, and tends to be used to bombard us with annoying banner ads and horrible interface design models, Flash is not going away anytime soon. However, how Apple and Adobe navigate the uncharted territory of merging mobile and desktop user experiences along with multi-touch interfaces, will certainly determine the relevance of Flash in the years to come.

Footnotes

1. See Adobe statistics on Flash Player downloads.
2. See Roughly Drafted’s analysis.
3. See A List Apart’s analysis.
4. See Cabel Sasser’s Twitter which claims dibs on this observation.
5. See the profile of Walt Mossberg in the New Yorker.
6. Speculative, but based on reliable, but undisclosed interactions with Adobe. Also, see Bill Perry’s entry on the subject.

Read Packing Lite: A mobile media interface design primer on Adobe’s Developer Center to get my perspective how to adapt to directional-pad based devices in Flash Lite. I detail the process of transferring SWF files to Nokia Series 60 devices and analyze a 4-way menu example from Adobe’s Content Developer Kit (CDK).

» Packing Lite on Adobe.com

Article to be published on Adobe Developer Center

Packing Lite: Getting Started Designing Interfaces for Mobile Media

I wrote an article/tutorial for Adobe’s Developer Center to be published at the end of August covering how to get started designing interfaces in Flash Lite on Nokia Series 60 devices. It details how to get equipped for mobile media design using Flash and the Flash Lite platform, install Flash Lite content on Nokia S60 phones with Bluetooth or Nokia’s PC Suite, discusses the unique interface design challenges, and looks at an example, highlighting the issues.

Stay tuned, as I’ll be sure to post the link to the article when it goes live. Article pushed live at 5 PM PST, August 17, 2007←

A collaboration between the Academy of Arts and Design, Tsinghua University, Beijing and Communication Design and Technology, Parsons The New School for Design

Objective

Develop a design prototype to present to the Art and Design Academy and BOCOG proposing an interactive Mobile Visitors Guide for the 2008 Beijing Olympiad. Comprising of a desktop web product and a downloadable mobile media component, the Mobile Visitors Guide will offer travelers to Beijing helpful information about Olympic events, venues, the visual identity system, transportation, language assistance, and the unique and vibrant culture of China. In particular, the mobile component offers visitors a useful electronic assistant to navigate the city and manage culture shock and language barriers.

Goals

The workshop students (RedStudio) will present prototypes and design documents suggesting and demonstrating how this product might be further defined and developed. The short timeframe (three weeks) will not allow the creation of a fully functional product, but we hope that it inspires further study and development.

Solution

Adobe Flash technology offers the best platform to rapidly design and prototype the Beijing Olympics Mobile Visitors Guide. Using Flash Player to deploy the desktop web feature prototype and Flash Lite to deploy the mobile media prototype on Nokia Series 60 handsets, the students will easily integrate design and content across delivery mechanisms for a unified and integrated user experience.

Mobile Media Enabling World Class Hospitality

Everyone knows that the Chinese are famous for great hospitality, and preparing for the 2008 Summer Olympics is no exception. However, the rapid pace of development and the intricacy of the Chinese language and culture creates
barriers for visitors to enjoy their experience in Beijing. Now that most people carry a mobile phone or laptop when they travel, the Mobile Visitors Guide solves common problems for foreigners to help them communicate across languages, discover the delights of the city, manage its challenges, and more easily find their way around.

An Essential Companion

In the New China, driven by high technology and media innovations, visitors to the Olympics Beijing 2008 should enjoy digital services to help them make the most of their time here, whether on their laptop or mobile phone. Consider these modules:

• Find your way. Visually link event schedules to venue locations and find your way by subway, taxi, bicycle or walking.
• Keep score. Follow sports events by medal counts, in real-time, as winners are announced.
• Communicate. Blaze through the language barrier with common translations for useful phrases and situations on your mobile. Get pinyin transliterations, hanzi characters and audio pronunciations. Point your way out of a tricky situation using Chinese characters on your mobile’s screen. Hand over your mobile and let the Beijinger listen to your needs.
• Participate in China’s Cultural Olympics at the National Art Museum of China and use a mobile phone in an experimental large-scale group interaction within the gallery spaces.
• Explore and discover the Old Beijing and take a guided tour through the famous hutongs, an intricate maze of traditional culture and cuisine.
• Meet the Five Friendlies. The Fuwa have a great life story and getting acquainted with them will help you better enjoy their intricate symbolism which elegantly evokes the many layers of Chinese language and culture, not to mention the universal spirit of the Olympics.

Conclusion

This project prototype was presented to the Academy of Arts and Design, Tsinghua University on June 20, 2007. Relevant industry critics from Nokia and Microsoft R&D laboratories in Beijing attended the presentation.

Presentation Photos

See also

Red Studio: Teaching Design at Tsinghua↑