While Nokia is leading the way with its Nokia Refocus now available for Nokia Lumia [Nokia Conversations, Nov 13, 2013]* being “a great example of computational photography” now Qualcomm is beginning to deliver such, albeit not so advanced yet, capabilities in that realm for its latest Snapdragon 805 SoCs.
* For more information see the Leading edge Nokia phablets for both entertainment and productivity: Lumia 1320 targeting the masses at $339, and Lumia 1520 the imaging conscious business users and individuals at $749 [‘Experiencing the Cloud’, Oct 26, 2013] post of mine.
** INSERTED LATER For the S805 SoC itself watch this CES 2014 recorded video:
Qualcomm Snapdragon 805, Adreno 420, HEVC 4K decode, 802.11ac, 4G LTE and more [Charbax YouTube channel, Jan 12, 2014]
END OF INSERT ***
Back in March 2013 Qualcom touted the following capabilities for its (then) top of the line Snapdragon 800 processors: High Performance Camera—Voice Activation—High Performance HD Video—3D Gaming—CPU Performance—Embedded Applications. With short videos included in a whole Youtube playlist you could easily understand each of them.
Then this was evolved into Professional photography, now in your pocket [Oct 21, 2013] in which the Snapdragon related message was formulated into:
Snapdragon 800 series mobile processors are designed to give your mobile devices the features and functionality of standard video cameras available today, including image stabilization, picture-in-picture and touch-to-track selections. With Snapdragon processors, you also receive 4K “UltraHD” resolution, with dual ISPs for higher performance and thermal efficiency. This means faster, higher-quality video, increased functionality and sleek portability for performance on the go.
Now that was further evolved into the starting message of CES 2014 [Jan 3, 2014]:
Qualcomm mobile technology is going everywhere. Your home. Your car. Even your body. Qualcomm 4G LTE Advanced connects you to a whole new world of possibilities. And that’s just the beginning. We’re inventing exciting breakthroughs here, so you can have incredible experiences—everywhere.
So computational photography arrived first time to Qualcomm’s high-end SoCs. This is even just the beginning of an immense set of new capabilities upto computer vision and augmented reality all enabled by Qualcomm moving to Applications DSP (ADSP) [‘Experiencing the Cloud’, Feb 9, 2013].
It is time therefore to examine Qualcomm’s recent Snapdragon 805-based offerings with some details about their research roots which will also make possible to take a glimpse into the future as well.
First let’s watch the Qualcomm Snapdragon 805 Processor Action Shot Demo [Qualcomm YouTube channel, Jan 3, 2014]:
This actually demonstrates the touch-to-track (T2T) capability the essential enabler for action shots. It was already mentioned in the messages about professional photography (see in the begining). In fact on the Qualcomm’s Snapdragon 800 Performance Review [July 17, 2013] webinar for industry analysts T2T was already recognized as a “cool concept”.
Smartphone cameras are typically not equipped with optical zoom due to size and cost considerations. As a result, the image captured by zooming in with a smartphone is done through cropping and interpolation. This results in an image that is both blurry and highly pixelated. Opti-Zoom technology significantly improves the clarity of images captured using zoom, through the use of sophisticated image processing technology which enhances the true resolution of the image.
There are certain challenges associated with images taken in low lighting conditions. For example, using a flash can cause the image to become over-exposed, altering the color of the image making it look unrealistic. On the other hand, not using a flash can cause the image to appear very dark. Chroma-Flash technology mitigates these challenges algorithmically and produces an image that preserves the colors, texture and brightness of the scenery.
So the above research was productized in Qualcomm Snapdragon 805 Processor Chroma Flash Demo [Qualcomm YouTube channel, Jan 3, 2014]
For this kind of research Qualcomm Research Austria is one of the company’s major bases (along with Qualcomm Research San Diego and Qualcomm Research Korea) which is running a “Qualcomm Augmented Reality Lecture Series” to provide high level talks given by highly recognized speakers from Academia in the field of computer vision and augmented reality. The RGB++: How “Side Information” Improves Computational Photography and Computer Vision [Apr 25, 2013] was one of that: (The chosen slide shown below is also referring to the original academic roots of Chroma Flash.)
Information theory and signal processing have classically used the notion of “side information” to formally describe and analyze situations where providing more information to either the encoding or the decoding process improves system performance. We consider and extend this viewpoint to modern day imaging systems, where in addition to images representing visual information, devices also capture a variety of side information. In the ubiquitous smartphone, for example, multiple sensors (microphone, GPS, accelerometer, compass, etc.) augment the two cameras that have become the norm. Additionally, these devices are usually connected to a large network of digital data. This rich “side information” can improve the performance of imaging applications and enable completely new functionality. Using research examples from our group, ranging from near-infrared to semantics, we present applications of such “side information” enabled functionality and improvements for computational photography and computer vision.. Prof. Sabine Süsstrunk, EPFLausanne
EPFL (Ecole Polytechnique Fédéral de Lausanne, i.e. Swiss Federal Institute of Technology, Lausanne) is indeed one of leading academic research places in Europe. The EPFLNews – Dynamic video-tracking for sports without physical tags [epflnews YouTube channel, Nov 2, 2011] video is the best illustration not only for that research but also of the immense possibilities for computer vision in the future:
Finally let’s examine what Qualcomm Technologies Announces Next Generation Qualcomm Snapdragon 805 “Ultra HD” Processor [press release, Nov 20, 2013] was offering in terms of new capabilities:
Mobile Technology Leader Announces its Highest Performance Processor Designed to Deliver the Highest Quality Mobile Video, Camera and Graphics to Qualcomm Snapdragon 800 Tier
NEW YORK – November 20, 2013 – Qualcomm Incorporated (NASDAQ: QCOM) today announced that its subsidiary, Qualcomm Technologies, Inc., introduced the next generation mobile processor of the Qualcomm® Snapdragon™ 800 tier, the Qualcomm Snapdragon 805 processor, which is designed to deliver the highest-quality mobile video, imaging and graphics experiences at Ultra HD (4K) resolution, both on device and via Ultra HD TVs. Featuring the new Adreno 420 GPU, with up to 40 percent more graphics processing power than its predecessor, the Snapdragon 805 processor is the first mobile processor to offer system-level Ultra HD support, 4K video capture and playback and enhanced dual camera Image Signal Processors (ISPs), for superior performance, multitasking, power efficiency and mobile user experiences.
The Snapdragon 805 processor is Qualcomm Technologies’ newest and highest performing Snapdragon processor to date, featuring:
- Blazing fast apps and web browsing and outstanding performance: Krait 450 quad-core CPU, the first mobile CPU to run at speeds of up to 2.5 GHz per core, plus superior memory bandwidth support of up to 25.6 GB/second that is designed to provide unprecedented multimedia and web browsing performance.
- Smooth, sharp user interface and games support Ultra HD resolution: The mobile industry’s first end-to-end Ultra HD solution with on-device display concurrent with output to HDTV; features Qualcomm Technologies’ new Adreno 420 GPU, which introduces support for hardware tessellation and geometry shaders, for advanced 4K rendering, with even more realistic scenes and objects, visually stunning user interface, graphics and mobile gaming experiences at lower power.
- Fast, seamless connected mobile experiences: Custom, efficient integration with either the Qualcomm® Gobi™ MDM9x25 or the Gobi MDM9x35 modem, powering superior seamless connected mobile experiences. The Gobi MDM9x25 chipset announced in February 2013 has seen significant adoption as the first embedded, mobile computing solution to support LTE carrier aggregation and LTE Category 4 with superior peak data rates of up to 150Mbps. Additionally, Qualcomm’s most advanced Wi-Fi for mobile, 2-stream dual-band Qualcomm® VIVE™ 802.11ac, enables wireless 4K video streaming and other media-intensive applications. With a low-power PCIe interface to the QCA6174, tablets and high-end smartphones can take advantage of faster mobile Wi-Fi performance (over 600 Mbps), extended operating range and concurrent Bluetooth connections, with minimal impact on battery life.
- Ability to stream more video content at higher quality using less power: Support for Hollywood Quality Video (HQV) for video post processing, first to introduce hardware 4K HEVC (H.265) decode for mobile for extremely low-power HD video playback.
- Sharper, higher resolution photos in low light and advanced post-processing features: First Gpixel/s throughput camera support in a mobile processor designed for a significant increase in camera speed and imaging quality. Sensor processing with gyro integration enables image stabilization for sharper, crisper photos. Qualcomm Technologies is the first to announce a mobile processor with advanced, low-power, integrated sensor processing, enabled by its custom DSP, designed to deliver a wide range of sensor-enabled mobile experiences.
“Using a smartphone or tablet powered by Snapdragon 805 processor is like having an UltraHD home theater in your pocket, with 4K video, imaging and graphics, all built for mobile,” said Murthy Renduchintala, executive vice president, Qualcomm Technologies, Inc., and co-president, QCT. “We’re delivering the mobile industry’s first truly end-to-end Ultra HD solution, and coupled with our industry leading Gobi LTE modems and RF transceivers, streaming and watching content at 4K resolution will finally be possible.”
The Snapdragon 805 processor is sampling now and expected to be available in commercial devices by the first half of 2014.
The Meet the Snapdragon 805 “Ultra HD” Processor [Nov 20, 2013] blog post gave some more detail about the dual camera ISPs:
- Snapdragon 805 processors also enable users to take, edit and share higher quality photos in low light conditions. The world’s first commercial mobile 1GPixel/s (Giga-pixel per second) ISP (image signal processor) packs a large increase in ISP and CPP (camera postprocessor) speed and throughput, empowering users to take sharper, higher resolution photos with advanced post-processing features for low light conditions.