The end result of a benchmark run in SPMarkJava06 is an SPMark score than can be used to compare performance across handsets. The benchmark program (itself a MIDP application) is run on the handset and cycles through a series of tests including a 2D Games represenstation, a 3D Game / rendering representations, a video test, various graphics and images tests and a series of JVM tests (to test general Java performance). Further information on the tests is available on the Futuremark website.
Through the benchmark application users can also access detailed information about the Java implementation present on the test device including supported JSRs.
New to Futuremark's mobile benchmarking lineup is the inclusion of the Online Results Browser (ORB) on the phone itself (using the built in browsers) allowing users to compare results between different phones via the online database. As with other Futuremark products tests can be run individually (via user slection) or as part of a series to generate the overall benchmarking score.
Aside from the obvious uses in comparison test for end users and in reviews in the media the benchmark is also aimed at manufcaturers, operators, and developers who can now used a standardised performance testing solution to help with development and related business decisions. The source code is made avilable through Futuremark's BDP program to allow for the development of custom benchmarks by manufacturers and other interested companies.
There are three versions available, two for the conusmer (a free personal edition, and a $3 advanced edition with full testing features) and one for professional use ($20,000) with optional source code licensing.
The application is easy to use and provides a comprehensive set of tests and subsequent results and facillitates realistic Java performance comparisons across handsets (both Symbian and non Symbian handsets), something which has only be avilable in a limited form until now.
Results:
Results for the Nokia 6680 from SPMarkJava06 Benchmark:
SPMarkJava06 Score: 962
2D Game: 0.8 FPS
3D Game: 11.2 FPS
Fillrate: 2712 kTexels/s
Polygon Throughput: 37.6 kVertices/s
PNG Score: 21
JVM Score: 1720
In order to run the full set of benchmark test (and thus gain a final overall score) devices must support all the necessary Java standards (JSR 184, JSR 135, using the MIDP 2.0 Profile on the and CLDC 1.1 which is required to run tests that require floating point calculations, and H.263 for video).
If they do not only a limited subset of the tests will be run. Both the Nokia 9500 (Mobile 3D [M3G] not supported, and 3D tests are therefore disabled) and Sony Ericsson P910i (CLDC 1.1 and Mobile 3D not supported) fall into this category. While this may mean it is difficult to make comparisons between current devices (there are few that meet all requirements to run the benchmarks) this is largely because the benchmark is aimed at being useful for comparing future handsets. Many of the handsets released in 2006 will meet all the requirements required to run the benchmarks.
The Nokia 9500 Results:
2D Game: 0.5 FPS
PNG Score: 18
JVM Score 896
The Sony Ericsson P910i Results:
2D Game Score: 0.7 FPS
PNG Score: 33
The Nokia 670 Results:
2D Game: 0.7 FPS
PNG Score: 29
From the Futuremark website:
Key Features
- MIDP 2.0 / JSR 184 Java benchmark application for handheld devices
- Extensively researched workloads evaluate performance across a broad range of Java APIs and profiles
- Source code licensing available under separate license
- Version available for business users with full functionality and results publishing rights
- Free consumer version available for personal testing
- Small fee consumer version available with playable game and extensive features
- Includes high and low detail 3D game tests, 2D game test, video playback, image processing, Java Virtual Machine (JVM) testing and battery life testing tool
- Option to upload and compare results in Futuremark Online ResultBrowser (ORB).
- SPMarkJava06 is the most comprehensive benchmark for measuring Java performance on mobile devices
- Developed in close cooperation with industry leading companies, to ensure well designed, valid workloads with unbiased, reliable results comparison across different architectures and environments
- Consists of popular end user application workloads including 2D and 3D gaming, video playback and image processing, in addition to measuring JVM operations and battery life
- Uses common Java APIs and will run on a wide variety of devices which have MIDP 2.0 and JSR 184
- Source code licensing option allows customizing and compiling benchmarks to suit semiconductor manufacturers, hardware and handset manufacturers, application developers and operator requirements
Press Release:
Futuremark Announces SPMark(TM)Java06 Benchmark
A Reliable Performance Measurement and Online Comparison Tool for Mobile Devices
Saratoga, California USA - November 16, 2005 - Futuremark Corporation today announced the SPMarkJava06 benchmark, a richly featured, handheld device benchmark for measuring and evaluating a system's performance in running Java applications. Developed in close cooperation with such industry leading companies as ARM, ATI, Bitboys, DMP, Falanx, Imagination Technologies, Intel, Khronos Group, NVIDIA and Symbian, SPMarkJava06 benchmark provides reliable, unbiased results across different architectures and environments.
"SPMarkJava06 was developed to offer a broad group of users a rich variety of features," said Tero Sarkkinen, Executive Vice President of Sales and Marketing for Futuremark. "Futuremark offers versions for business users as well as consumers, and the benchmark includes comprehensive, extensively researched workloads to accurately assess performance across a broad range of Java Application Program Interfaces (APIs) and profiles."
Saratoga, California USA – November 16, 2005 – Futuremark® Corporation today announced the SPMarkJava06 benchmark, a richly featured, handheld device benchmark for measuring and evaluating a system's performance in running Java applications. Developed in close cooperation with such industry leading companies as ARM, ATI, Bitboys, DMP, Falanx, Imagination Technologies, Intel, Khronos Group, NVIDIA and Symbian, SPMarkJava06 benchmark provides reliable, unbiased results across different architectures and environments.
"SPMarkJava06 was developed to offer a broad group of users a rich variety of features," said Tero Sarkkinen, Executive Vice President of Sales and Marketing for Futuremark. "Futuremark offers versions for business users as well as consumers, and the benchmark includes comprehensive, extensively researched workloads to accurately assess performance across a broad range of Java Application Program Interfaces (APIs) and profiles."
Test Profile and Technology
A valid benchmark should reflect the types of applications likely to be running on the systems for which it is designed. With this approach, users get reliable measurements and evaluations rather than theoretical information. As such, Futuremark's SPMarkJava06 benchmark includes performance tests for high- and low-detail 3D games, 2D games, image processing and Java Virtual Machine (JVM). In addition, SPMarkJava06 provides tools for measuring the battery-life and checking the video (H.263) playback functionality of the device. SPMarkJava06 also shows extensive system information about the device's Java implementation.
For greatest applicability, the benchmark uses common Java APIs and runs on a wide variety of devices compatible with MIDP 2.0 and JSR 184 (M3G). In addition, it works with handsets and adapts to their user-interface codes and Java implementations. The SPMarkJava06 benchmark utilizes JSR 135, and the MIDP 2.0 Profile on the CLDC 1.0 & 1.1 configuration.
Result Comparison: Online ResultBrowser
SPMarkJava06 is the first mobile phone benchmark from Futuremark with online services enabled. The Online ResultBrowser (ORB), which has so far only been accessible via Futuremark's PC performance benchmarks, is the ultimate tool for any user to search and compare benchmark results and system information online. Futuremark first implemented the ORB concept in 1997, to allow their users to upload their 3DMark and later PCMark benchmark results. By expanding the service to include SPMarkJava06 results, mobile device benchmarkers can join the nearly three million current ORB users and harness the power of a global database of over 13 million real-life benchmark results.
Performance results and system information can be uploaded directly from SPMarkJava06, which uses the device's default web browser. New users can create a free ORB account, and existing ORB users can log onto their existing account. Once the data is loaded into the ORB, the results can be browsed using their internet connected PC or the device's browser. This will permit the user to compare virtually any phone against each other and see which J2ME features are supported by various phones.
"Enabling users to submit their benchmark results and system info from SPMarkJava06 to our online database is a tremendous step forward in mobile phone benchmarking. To be able to compare various mobile phones' performance differences from anywhere in the world, proves Futuremark's leadership in internet-enabled benchmarks," said Sami Niemela, Manager Mobile Products for Futuremark. "The Online ResultBrowser is the worlds most used and known online performance comparison tool based on a database consisting of 13.5 million benchmark results and almost 3 million registered users."
Pricing
Futuremark offers different versions of SPMarkJava06 to meet the varying needs of a broad user base.
The Professional Edition is the most comprehensive and provides industry companies with rights to publish results. Binary versions of the Professional Edition are priced at USD 20,000 for the first license, USD 5,000 for each additional license. Operators will be provided a 12 month free trial Professional Edition license. Optional source-code licensing for members of Futuremark's Benchmark Developer Program (BDP) supports technology developers and device manufacturers. For example, semiconductor manufacturers, hardware and handset makers, application developers and operators can create benchmarks very much tailored to their specific requirements and analyze performance in platforms under development.
Consumers have two versions available to them. The Advanced Edition, available for USD 3 per license, offers full features and a playable game. The Basic Edition is available at no cost to allow testing of the user's personal devices.
Availability
The SPMarkJava06 benchmark is available immediately via source-code licensing to Futuremark's BDP members and interested third parties. Compiled versions are also available immediately to professional users and consumers as downloads from the Futuremark Web site – http://www.futuremark.com/products/spmarkjava06.
About Futuremark® Corporation
Futuremark ® Corporation is the leading provider of performance analysis software and services for PCs and smartphones. Futuremark® is known around the world for its benchmark products, including the 3DMark® and PCMark® Series and SPMark™ (with more than 30 million copies distributed worldwide) and value-added services powered by a database of over 13.5 million real life benchmarking results. Futuremark® maintains offices in Saratoga, California and Helsinki, Finland. For more information, please visit http://www.futuremark.com.
© 2005 Futuremark® Corporation. 3DMark®, PCMark® and SPMark™ trademarks and logos, Futuremark® character names and distinctive likenesses, are the exclusive property of Futuremark Corporation. OpenGL® and the oval logo are trademarks or registered trademarks of Silicon Graphics, Inc. in the Untied States and/or other countries worldwide. All other trademarks are property of their respective companies.