Fork me on GitHub

ImageDecoderFramework Examples

PAGE IS STILL IN CONSTRUCTION

ImageDecoderFramework.js is a Javascript library for heavy decoding of images which are decoded by custom logic. It was designed especially to show the images in a web viewer like Cesium or Leaflet, besides an API that supports image decoding for wide applicative needs.

The main advantages of using the library over implementing your image directly on top of one of the popular viewers API (e.g. Cesium's ImageryProvider or Leaflet's layers) are:

  • Heavy use of Web Workers, which allow better interactivity.
  • Support progressive fetch of gradually improved quality.
  • An intermediate level between image implementation and viewer: Once an image is implemented, it may be used easily in both Leaflet and Cesium. Moreover, once implemented a plugin for your favorite viewer you may use it for any image.

Once an image is implemented, the API of using it (for example to show part of it in HTML page or put it on a viewer) is relatively simple. However implementing the underlying image requires training on the API. In addition, the spatial calculations of the exact data to be fetched and pixels to be drawn on decode may be unintuitive without some experience with viewer infrastructure implementations.

Using the library requires the following script imports: 

   <script src="http://cdn.leafletjs.com/leaflet/v0.7.7/leaflet.js"></script>
   <script src="Cesium-1.20/Cesium.js"></script><!-- downloaded locally -->
   <script src="http://MaMazav.github.io/cdn/image-decoder-framework.dev.js"></script>

Image Implementation Overview

An image requires implementing a top-level Image class for all internal implementation. The image should hold three other classes that together implement the image functionality:

  1. Fetcher - fetches the image parameters and the image data from server.
  2. InputRetriever implementation for Dependency Workers - manages the Web Workers that perform the actual decoding. It should be implemented by the depdendency-workers.js library API. If using gridimagebase some of the dependency-workers.js requirements are already fulfilled.
  3. LevelCalculator - provides conversion between screen resolution and level index, for multiple resolution images.

Whereas manually implementing all the functionality of those classes gives a lot of power, it may be too complicated for simple images. The GridImageBase and FetcherBase classes that are explained below are recommended for simple image implementation.

Notice that each object of these classes may live in another Web Worker. Thus all method arguments, method results and even exceptions should be serializable. You don't need to treat the serialization and manage the Web Workers, just make sure that all data supports structured clones.

Let's look on a simple example. This example shows an image that contains some resolution levels. Each resolution level is splitted into rectangular tiles (as in a standard "pyramid" - a common technique in the area of graphic viewers). The image is blue except of the tile borders which are cyan.

GridFetcherBase

First we need to implement the Fetcher. Inheriting the GridFetcherBase is an easy way to create a fetcher that fits our purpose. It requires implementing of three methods: open, fetchTile and getImageParams. The open method is called once and should return a promise that successfully terminates when the image is ready. The fetchTile performs the actual fetch of a single tile within a specific resolution level. The getImageParams should return an object contains the parameters of the image. The image parameters object should be returned also in the open method Promise result.

The image parameters should contain at least the following fields:

  • imageLevel - a reference resolution level of the image. The reference level is used only for consistency of calculations (sizes returned by various methods of the image should be in terms of the reference level) and doesn't have any applicative meaning for viewers. To simplify calculation it's usually better to choose the best resolution level as a reference level.
  • imageWidth, imageHeight - the size of the image, in terms of the imageLevel.
  • numResolutionLevelsForLimittedViewer - count of resolution levels. Even if the image is dynamic and may adapt to any resolution, some viewers may not have the ability for that and you'll have to choose some limitation for the resolution level count.
  • lowestQuality, highestQuality - An advanced feature that may be used to define progressive quality within a given resolution level (e.g. as in Jpeg2000's quality layers). A lot of image implementations doesn't need it, and may put here an arbitrary value.
  • tileWidth, tileHeight - Single tile size, Required only for images that inherit GridFetcher and GridImage.

Besides those arguments, additional arguments may be added to the image parameters object. The image parameters object is later passed to the image by the opened method (see below). Besides being a consistent way to pass this parameters, it also provides a way to pass this argument across Web Workers if the Fetcher and Image are not on same Web Worker.

GridImageBase

Now that we have a Fetcher implemented, we need the Image and DependencyWorkers classes. Fortunately the GridImageBase implements most of their functionality given that our image is a conventional grid-structured. We only need to implement two methods: The opened method which is called after the Fetcher.opened terminated successfully (don't forget to call the base's opened method!), and the getDecodeWorkerTypeOptions which returns the data required to initialize a Web Worker that decode a single tile (same Web Worker data required for the Web Workers in depdendency-workers.js library).

Notice that we also overrode the getLevelCalcualtor. This method can optionally be overriden to change the default LevelCalculator behavior provided by the GridImage. The default implementation represents an image with infinite resolution levels, where each level is four times bigger than the previous level (two times for each axis). In that example we override the default behavior of the GridImage to show how we create a finite-resolution image. You also may choose to implement an absolutely different LevelsCalculator (see the default GridLevelCalculator implementation as an example).

Decoder Worker: Web Worker for actual decode

Finally we need the Web Worker implementation which actually decode a single tile:

That's all! We have the implemented image. Now let's see how can you easily convert this code into a graphic view using Leaflet.

Showing image in Leaflet

Using the following several lines, your image will be shown in leaflet:

Notice that you the workersLimit argument is optional and has a default value of 5. This argument limits the count of Web Workers used for decoding (excluding the fetch Web Worker). We limitted it in this demo because this page contains four images, and too many Web Workers causes instability issues in mobile device browsers.

Now that we've seen the whole picture of the simplest use of the image-decoder-framework.js library, we would like to extend our knowledge about additional features that the library allows.

Another image: Sierpinski carpet

The following example is more complex example of an image which represents Sierpinski carpet image. It uses a graphics library which luckily supports by-region partial calculation of the Sierpinski squares coordinates (well, actually this "library" was written especially for this demo page...).

The Sierpinski carpet image does not demonstrate a new feature of the framework. It is just another more complex reference example of the framework. In addition we will improve this example to demonstrate progressiveness later (please ignore the getLowestQuality and getHighestQuality methods now - as mentioned above, they are not effective without progressiveness support).

Leaflet Sierpinski ImageDecoder Demo

Top level image implementation of Sierpinski carpet:

Sierpinski carpet image fetcher implementation:

Sierpinski decoder worker implementation:

Progressive fetch

Assume you have a heavy image, which is hard to be loaded or decoded. However this image may be splitted into lower qualities, which allows showing a quick lower quality image until the better image will be shown. In this case you may find progressiveness a very useful feature. Unfortunately not all viewers support it easily.

Imagine that calculating Sierpinski square positions is a very heavy task. (later 'll demonstrate it by adding 500ms delay). As we show squares of smaller size, the count of the square increases and the quality improves. Luckily again, the graphics library enable bounding the square size to lower value to improve performance over quality. Let's reimplement our fetcher to support progressive calculation of Sierpinski squares (the progressive version of Sierpinski carpet image inherits the simple one and overrides only the required changes for progressiveness):

Leaflet Progressive ImageDecoder Demo

The top level image implementation of Sierpinski carpet is slightly changed:

The fetcher implementation, however, is more sensitive to progressiveness:

The quality does not affect the decoder worker which only takes a list of squares as an input (although there may be cases in which the worker implementation should be changed to support progressiveness).

Showing image in Cesium

The natural way to implement an image in Cesium is to implement an ImageryProvider. This API has a lot of advantages like natural integration into Cesium, tile caching, etc. However this API does not support progressiveness. We provide the two alternatives: CesiumImageDecoderLayerManager is an implementation which support progressiveness, and ImageDecoderImageryProvider which is more stable and enjoys other Cesium's ImageryProvider advantages if you don't care about progressiveness:

CesiumImageDecoderLayerManager

CesiumImageDecoderLayerManager Demo

(In new Cesium versions we have a serious blinking issue, hope to solve it soon).

The code behind showing an image in Cesium (given that the image is already implemented as above):

ImageDecoderImageryProvider

The API of ImageDecoderImageryProvide is more similar to the standard Cesium API than the CesiumImageDecoderLayerManager demonstrated above. However, one point which is counter-intuitive is the fact that ImageDecoderImageryProvider requires its parent viewer or widget as the constructor parameter. The reason is performance: Cesium tends to request a lot of tiles, especially when the user moves quickly but also for prefetch reasons. It may significantly degrade performance in the context of heavy decoding images. Thus the framework uses scheduling mechanism to prioritize and even abort some requests. The framework uses the information about the current view area for this prioritization, thus it needs the viewer.

ImageDecoderImageryProvider Demo

Render image to canvas

To render an image to canvas, ImageDecoder.renderToCanvas may be used. The region from the image and region in the canvas can be set by the passed arguments. Once started, the renderToCanvas function takes control over the canvas and render the pixels to it. Click on the button below the code to activate the demo:



Programmatically get pixels of region in image

Until now we saw only high level interfaces for using the image with Leaflet and Cesium viewers. One can also use the ImageDecoder class to get the pixels programmatically and do it whatever he would like. The following example demonstrate how to decode a region of interest from the image and use it within a simple canvas - i.e. achieve almost same behavior as in previous demo, but manually (click on the button below the code to activate the demo):



Showing image in arbitrary viewer

Besides the ImageDecoder demonstrated above, there is also API which is more optimized for viewers. The class ViewerImageDecoder will manage a canvas for you, including caching already-rendered regions, canvas resizing, and always rendering low-resolution image in the back to avoid empty areas blinkin. You only need to let the ViewerImageDecoder know when and where the viewer moves, and update the canvas position according to the ViewerImageDecoder callbacks.

The example below shows how to use the image with a super simple viewer implemented in the graphics library (same graphics library mentioned above which was written for this demo).