ArcCatalog no longer seeing Raster Data?

I am providing this information for the benefit of ERDAS IMAGINE customers who are using ArcGIS as well.

The problem: ArcGIS can no longer see raster data in ArcCatalog using ESRI’s File Chooser. The only way the customer can add raster data to ArcMap seems to be a drag and drop from Windows Explorer.

A friend told me said the problem started appearing with ArcGIS 9.2. I have experienced it in 9.2 as well. This is the result of a setting in ArcCatalog being changed somehow. The friend said the problem occurred on systems with both ESRI and ERDAS products loaded as well as with ESRI products alone. To complicate things, it doesn’t happen on every machine (a phantom problem).

Notwithstanding the illusive nature of the problem, it is easy for the customer to correct on their computer.

To resolve the issue, go to ArcCatalog – Tools – Options. From the General Tab under the "Which types of Data do you want the Catalog to show?" section, check the checkbox next to Raster Files and apply the change. This will restore ArcMap and ArcCatalog’s ability to display rasters in the file chooser dialog and catalog listing.

Create an IMAGINE Image File List (.fls) for use in MosaicTool and MosaicPro.

I originally created this procedure while at the Georgia Institute of Technology (Georgia Tech) Center for Geographic Information Systems.

The structure of .fls format is documented in ERDAS IMAGINE as follows:
n number-of-images
i reference-image
0 imagename_1
1 imagename_2
.
.
.
n-1 imagename_n

An example is as follows:
3 number-of-images
1 reference-image
0 R:\Data\wasia1_mss.img
1 R:\Data\wasia2_mss.img
2 R:\Data\wasia3_tm.img

How do I create a .fls for all TIFF images in a directory and in all sub-directories?

First, use a DOS Command Window
1. Open a DOS Command Window
2. Change to the appropriate directory
3. Use "dir" command as follows: dir *.tif /b /s > mosaic-list.txt
(The /s parameter looks in sub-directories; the /b makes the list 'bare')

Next, use MS Excel to organize the list
4. Open "mosaic-list.txt" in Excel
5. Insert a column to the left of the list
6. Insert 2 rows at the top of the list
7. Add 0 in the first column of the third row
8. Add =A3+1 to the first column of the forth row
9. Copy formula from #8 above to all rows with images
10. Copy the number for the last image to the first column of the first row
11. Select an image number for the reference image (can change in Mosaic later) and put in the first column of the second row.
12. Find / Replace all \" with "/", in the image column.
13. Save as the file to a text file to your desired txt filename.

Next, use MS Word to remove tabs introduced by Excel:
14. Replace tab "^t" with a space " " (do not enter quote marks)
15. Save as text with the extension of .fls

Finally, open in Mosaic Tool or Mosaic.

ERDAS IMAGINE / LPS 9.3 Shipping complete - SP 9.3.1 available

The final few media packs containing the ERDAS 2009 DVDs (which include ERDAS IMAGINE and LPS 9.3) are being shipped from ERDAS, Inc. we I write.

Look on the ERDAS, Inc. Support Web site to the latest Service Pack for ERDAS IMAGINE and LPS 9.3 as well. The service pack upgrades ERDAS IMAGINE and LPS 9.3 to 9.3.1.

There is a Service Pack for ERDAS ER Mapper 7.2 as well.

Have y'all started using ER Mapper 7.2 yet? It is included on the ERDAS 2009 DVD and in the IMAGINE Professional license files. Give it a try.

ERDAS IMAGINE / LPS 9.3.1 available

ERDAS IMAGINE / LPS 9.3.1 and ERDAS ER Mapper 7.2 SP1 are now available for download from the erdas.com support website. There are customer reported issues and some enhancements available in these service packs.

ERDAS IMAGINE / LPS 9.3.1 requires a pre-installed version of ERDAS IMAGINE / LPS 9.3. ERDAS ER Mapper 7.2 SP1 requires installed version of ERDAS ER Mapper 7.2.

ERDAS IMAGINE 9.3 Shipping

ERDAS took receipt of the ERDAS 2009 DVD Media Kits on Oct. 31, 2008 and began shipping ERDAS IMAGINE 9.3 on the following Monday November 3, 2008. I have spoken with the ERDAS shipping department recently and they expect to have all DVD Media packs shipped out to ERDAS customers and ERDAS Distributors by early December.

The ERDAS 2009 DVD Media Kits consist of two dual-layer DVDs containing all ERDAS software and data examples for all ERDAS products. Be sure to load ERDAS ER Mapper 7.2 and give it a try, especially the mosaic tools.

Also, if you have not tried ERDAS TITAN, you might want to give it a look. I believe Universities will especially benefit from ERDAS TITAN’s sharing ability. I know when I was at the Georgia Tech Center for GIS I always had staff and students in the School of Architecture want to know if I had data in this area or that area. If ERDAS TITAN was around then, it sure would have saved me time and maybe kept me from being grumpy when the tenth student emailed about this data or that data for the project they have due in one week. Students…. Gotta love ‘em.

Back to ERDAS IMAGINE 9.3, you could download the install executable from the ERDAS Website under the Products > ERDAS IMAGINE > Download page. Here's the link.

http://www.erdas.com/Products/ERDASProductInformation/tabid/84/currentid/1050/default.aspx

Strange Rumor Concerning Backwards Compatibility in 9.3

I have recently heard a strange rumor concerning backwards compatibility in ERDAS IMAGINE 9.3. In case you hear the rumor, just ignore it. ERDAS IMAGINE 9.3 has image and vector backwards file compatibility to pervious verisons of .img, .lan, .gis, .tif and many more raster data, as well vector data created within the product line since the introduction of commercial ERDAS software in 1982.

I had another question on "backwards compatibility," dealing with license files rather than data files. Here is a comment on that:

The entire licensing technology was changed for ERDAS IMAGINE 9.3. The FlexNet license manager is now used and the system IDs are thus required to be different. Nevertheless, customer can have ERDAS IMAGINE 9.2 and earlier versions of license managers on the same system as ERDAS IMAGINE 9.3. As with all versions, only one version of IMAGINE can run at a time.

When ERDAS switched to the new licensing system, ERDAS streamlined the process and addressed many of the places where errors occurred for customers installing license files. So, ERDAS hopes the pain of obtaining new license files has been minimized.

Finally, note that there is no price difference between node-locked and floating licenses for ERDAS IMAGINE. If a customer receives node locked license and rather wanted floating licenses (and are not changing the license manager machine), they need only ask for floating licenses. This has been ERDAS policy since ERDAS IMAGINE supported floating license files in about 1992.

ERDAS IMAGINE 9.3 and ER Mapper 7.2 now share licensing codes

On May 21, 2007 ERDAS announced the purchase of Earth Resource Mapping Ltd (ER Mapper). Among the products ERDAS acquired in that purchase was ER Mapper Professional. For almost two decades, ER Mapper Professional and ERDAS IMAGINE competed for market share in the remote sensing segment of the mapping community. Until ER Mapper moved its focus to enterprise development, ER Mapper Professional owned the second largest market share, second only to ERDAS IMAGINE.

ERDAS announced on October 6, 2008, ERDAS IMAGINE 9.3 and ER Mapper 7.2 now share licensing codes at the IMAGINE Professional level. This means customers purchasing new licenses or receiving licenses through software maintenance (SWM) for IMAGINE Professional 9.3 and ERDAS ER Mapper 7.2 will be able to use both products at the same time on the same desktop, with one license of either of these ERDAS products.

At first glance it may appear customers will now receive two licenses for every one license. This is not correct. The customer will continue to have one license.

To help explain, allow me to introduce the idea of an “ERDAS Session.” An ERDAS Session begins when the customer launches an ERDAS product on a computer. Each session consumes a single license of an ERDAS product. ERDAS allows only one session per computer.

When a customer launches ERDAS IMAGINE 9.3 or ERDAS ER Mapper 7.2, they launch an ERDAS Session. Because ERDAS ER Mapper 7.2 and ERDAS IMAGINE 9.3 use the same license codes, they also use the same session. Thus, the customer can launch both products on the same computer at the same time.

To help make this clearer, let's look at a few scenarios. Assume we have one floating license of ERDAS Professional 9.3 and one floating license for ERDAS ER Mapper 7.2 on a network to be used among three people.

Scenario 1:

1. Shirley begins an ERDAS Session by launching ERDAS ER Mapper 7.2 on her computer. She then needs to use a classification tool found in IMAGINE Professional 9.3, and launches it as well (not closing her ER mapper tools). No problem, she has occupied 1 license within her ERDAS Session.
2. Soon thereafter, Stephanie begins an ERDAS Session by launching ERDAS IMAGINE 9.3 on her computer. No problem, she has occupied the second (and final) license within her ERDAS Session.
3. A few moments later, Jessica tries to launch ERDAS ER Mapper 7.2 and is denied a license by the license broker. Jessica’s problem is both licenses have been consumed by Shirley and Stephanie’s two ERDAS Sessions. Her solution is to ask Shirley or Stephanie to close their sessions, or purchanse another license.
   

Scenario 2:

1. Matt begins an ERDAS Session by launching ERDAS IMAGINE 9.3 on his computer.
2. Soon thereafter, Chenda begins an ERDAS Session by launching ERDAS IMAGINE 9.3 on her computer. No problem, she has occupied the second (and final) license.
3. A few moments later, Trey tries to launch ERDAS ER Mapper 7.2 and is denied a license by the license broker. Trey’s problem is both licenses have been consumed by Matt and Chenda two ERDAS Sessions. His solution is to ask Matt or Chenda to close their sessions, or purchase another license.
   

Scenario 3 (added 2 days after original post by popular request). Assume we have one floating license of ERDAS Professional 9.3 on a network:

1. Troy begins an ERDAS Session by launching ERDAS IMAGINE 9.3 on his computer.
2. Soon thereafter, Michael tries to open an ERDAS ER Mapper 7.2 session on his computer and is denied by the license broker. Michael's problem is the one license is occupied by Troy's session. His solution is to ask Troy to close his session or purchase another license.

So, why the change?

ERDAS is creating the next generation of desktop geospatial authoring tools. These new tools are being developed from customer’s demand of a fusion of the best of ERDAS IMAGINE, ERDAS ER Mapper, other existing ERDAS technology (desktop and enterprise), as well ERDAS' desire to introduce new technologies under development at ERDAS (not yet released).

Because the fused product will have a 100% capability match at the ERDAS ER Mapper 7.2 and IMAGINE Professional 9.3 level, ERDAS decided to provide both tools to customers, today.

It is my opinion, ERDAS is providing a future vision while delivering some of the capability in that vision, today. The combined licensing of ERDAS IMAGINE 9.3 and ERDAS ER Mapper 7.2 delivers the most powerful geospatial imagining offering available today. Making the change today provides ERDAS customers an opportunity to experience the power of both products, and knowing their SWM has value today and in the future.

A few benefits to existing ERDAS ER Mapper 7.2 Customers are:

• Expand ortho-rectification capability with many different sensor models supported in IMAGINE Professional
• Expand the raster data format support to >150 formats
• Provide robust native vector editing/attributing capability found in ERDAS IMAGINE (arc coverage & shapefile)
• Provide pathway to all of the IMAGINE Add-on Modules (VirtualGIS, Vector, Radar Mapping Suite, Objective, and so forth)

A few benefits to existing ERDAS IMAGINE 9.3 Customers

• Rapidly mosaic, color-balance and compress massive volumes of image data. Using the 32-bit compressor on a 32-bit OS, mosaic terabytes of imagery (limit between 1 and 5TB) Using new-64-bit compressor on a 64-bit OS, mosaic many terabytes of imagery (limit estimated above 72TB)
• All image processing is dynamic / on-the-fly; process to screen rather than process to disk
• Domain specific workflows and wizards suitable for a range of industries and specialized applications (e.g. Oil and Gas)

A few benefits to new ERDAS Customers

• ERDAS customers will have both ERDAS IMAGINE and ER Mapper on the same desktop
• ERDAS customers do not have to pick between the products, making their decision easier and quicker
• No need to study both products to determine which meets their need
  

Native and fast lossless compression within ERDAS IMAGINE

While ERDAS IMAGINE supports lossless compression through several different means (MrSID, JPEG2000, Packbits, etc.), this discussion focuses on IMAGINE's native run length encoding (RLE) compression. Typical RLE algorithms compress adjacent duplicate pixel values and are a common lossless compression. The implementation of RLE in ERDAS IMAGINE uses a dynamic range run length compression (DR-RLE) rather than the typical RLE. DR-RLE not only compresses adjacent duplicate pixel values but also compresses unused pixels values within the data’s defined data range. This has been the case since the earliest days of .img for both athematic (continuous) and thematic data.

What do we mean when we say it compresses the dynamic range of the data? An example provides a good description. Many modern data types are collected as 11 or 12-bit data, yet we must store it as 16-bit data. If we have an 11-bit dataset, all possible pixel values between value 2048 (unsigned 11-bit +1) and value 65,535 (unsigned 16-bit max) are not used by the dataset, but must take up disk space. When we store these 11-bit data in the .img data format using a DR-RLE compression, all unused pixel values are compressed to a small fraction of its initial 16-bit size.

Elevation data can also benefit from the .img lossless compress because few DEMs need values anywhere near the 16-bit maximum value of 65,535. Thematic data also can benefit. Often have 100 or less categories (classes) and thus must be stored 8-bit. When encoded using DR-RLE, unused values are compressed.

One gottcha discovered by Donn Rodekohr at Auburn Univ., floating-point data stored in the .img DR-RLE should not be processed with intensive math functions in the modeler. Using these floating-point data in complicated models are shown to have some degradation in processing speed. We will address this issue in future versions of IMAGINE. So, keep your DR-RLE data storage limited to integer data for the time-being.

Below are a few size comparisons. From my previous GeoTIFF post (http://field-guide.blogspot.com/2008/05/what-is-wrong-with-my-geotiff.html) you will recall that Packbits is TIFF’s RLE compression. Packbits compresssion is not a DR-RLE compression.

DR-RLE comparison using a 512 x 512 single band .img file:
271KB 8-bit uncompressed (with pyramid layers)
271KB 8-bit RLE compressed (with pyramid layers)
527KB rescaled to 16-bit uncompressed (with pyramid layers)
272KB rescaled to 16-bit RLE compressed (with pyramid layers)

Packbits comparison using a 512 x 512 single band .tif file:
258KB 8-bit uncompressed (without pyramid layers)
258KB 8-bit packbits compressed (without pyramid layers)
514KB rescaled to 16-bit uncompressed (without pyramid layers)
514KB rescaled to 16-bit packbits compressed (without pyramid layers)

But what about the speed? When ERDAS implemented DR-RLE back in 1993, we focused on it being efficient. This effort, and because DR-RLE is so very simple to compress and decompress we see some speed gains from DR-RLE compressed img data over uncompressed .img data. This is the case for today’s processors as well. Having an efficient access mechanism, small data footprint stored in an ultra-simple compression & de-compression format helps processing speeds to disk or to screen.

When an .img file is >2.1GB, IMAGINE rolls the data over in an .ige file and the .img file stores metadata, spatial indices and so forth. The DR-RLE compression is not supported within .ige files. The original design of the .ige is for the simple and fast access of very large uncompressed data files. The .ige design does not allow for any file compression whatsoever. (See: http://field-guide.blogspot.com/2010/04/when-does-img-image-roll-over-to-ige.html)

For files >2.1GB which can stand a little value loss, we suggest our own ECW format or LizardTech’s MrSID format. We have added a ECW read capability to IMAGINE Essentials 9.2 and an encoding capability with IMAGINE Professional 9.2. While IMAGINE supports MrSID Generation 2 and Generation 3 (MG2 & MG3 respectively), we must use MG3 for MrSID files >2.1GB and lossless compression. ECW does not support lossless compression.

Are there any side effects (good or bad) when pulling these .img DR-RLE data into ArcGIS?” Not that I have seen in 15 years. In-fact, ESRI and ERDAS both use the same DR-RLE algorithm. ESRI uses it in GRID, ERDAS in .img. As well, ERDAS provides (writes) many of the raster data objects (RDO) in ESRI products, and .img support is one of them (RLE being part of .img support).

Any positive side effects in ArcGIS? Where ESRI uses the ERDAS / ESRI RDO within their product lines, the user will see the same performance improvements in ArcGIS. In other words, other than the issue with floating-point data, rock-n-roll in ArcGIS.

How do you take advantage of the .img dynamic range run length encoding in ERDAS IMAGINE?
Set “RLE” as your “Default Compression” in the "IMAGINE Image Files (Native)" preference category.
Set “Default” as your “Data Compression” in the "Spatial Modeler” preference category. This will cover Import, Save As, Subset (which is currently a modeler function), Mosaic Tool, MosaicPro, Spatial Modeler and all IMAGINE for files less than 2.1GB. We are planning to make RLE 'on' in the preferences in 9.3, if I hear a demand growing from the user community. Comments?

IMAGINE now supports encoding and decoding DR-RLE, JPEG, LZW, TIFF Packbits, MrSID, JPEG2000, ECW, et al. What other compressions do you folks wish to see within the ERDAS IMAGINE suite?

What is wrong with my GeoTIFF?

Maybe, it is not really a GeoTIFF?

Many people have tried over the last decade to standardize on GeoTIFF as a neutral data delivery format. This was the intent and design of the format. So, why are many still having problems with GeoTIFF?

GeoTIFF is built upon Adobe’s TIFF Revision 6 Specifications (see: http://partners.adobe.com/public/developer/en/tiff/TIFF6.pdf). While this data format is well documented, it is not followed by some software when creating and modifying data. This creates a challenge because some very prominent software companies are among the offenders. We are not immune to this non-standard software tweaking in the remote sensing, photogrammetry and GIS (geospatial) communities. Typically, today’s geospatial data providers have painstakingly located software that conforms to the GeoTIFF specifications (see: http://www.remotesensing.org/geotiff/geotiff.html) in a very strict manner. So, when we take receipt of data from today’s data vendors, we can be confident we have a proper GeoTIFF.

Again, some very prominent software does not conform to the TIFF and GeoTIFF specifications. When GeoTIFF data are modified subsequent to delivery to GIS, remote sensing and photogrammetry practitioners; often the practitioner will use software that does not conform to TIFF/GeoTIFF specifications to modify the data. This is where problems occur. When software creates a TIFF with non-standard GeoTIFF tags, it ceases to be a GeoTIFF, and is simply a TIFF with geographic information associated. These associated files are seldom readable by packages outside of the creating package, thus decreasing the portability of the image data. When you must modify GeoTIFF data, be sure you can create a true GeoTIFF.

While ERDAS IMAGINE is flexible and forgiving when reading many non-standard TIFF data with geographic information associated, ERDAS IMAGINE follows GeoTIFF specifications very carefully when creating GeoTIFFs. Notwithstanding ERDAS IMAGINE’s careful implementation of TIFF and GeoTIFF support, there are several things to be aware when creating the most portable GeoTIFF data possible. If we follow a few simple rules, we can provide data with a high level of portability, even to those beyond the geospatial community.

First, determine what level is the lowest level of support needed. I typically assume non-geospatial people will try to use the data in Microsoft Windows Picture and Fax Viewer or will insert the image into Microsoft Word. Hey, I know that is a bad idea, but a whole lot of people do it by accident, or because that is the best they have. Now, let’s not call them idiots because my mom has done this by accidently double-clicking on the TIFF. (Sorry mom.)

So, what TIFF parameters will create an image which will be supported the ‘lowest common software?’

1. GeoTIFF tags do not decrease portability. Software not using these GeoTIFF will ignore the tags. Because GeoTIFF followed TIFF6 specifications, true GeoTIFF’s will not cause a problem.

2. Unless you intend to open a lot of TIFF images at the same time, do not use tiling. If you need to open a lot of images at the same time, use tiling. While tiling is a standard TIFF option, and improves display speed in many software (especially for large images and lots of images), it decreases the portability of data. This is not a problem within ERDAS IMAGINE. It will handle a few dozen either striped or tiled images rapidly. But again, many lower-end software packages cannot handle TIFF data using tiling option, so use stripped TIFF (BIL) when in doubt.

3. Use the Red, Green, Blue (RGB, denoting 3 colors in the TIFF) option rather than Multispectral option. Multispectral is a standard TIFF option allowing for placement of more than three bands into a single TIFF. Many software products cannot support more than 3 colors per TIFF, and thus will not allow the display of the data.

In the modern geospatial industry, 4+ band data delivery is becoming more prevalent. Many providers are required to deliver data as a red, green, blue image and a NIR, red, green image for each frame. This means the deliverable has duplicated 2 bands, green and red. Sadly, until the key software vendors decide to support Multispectral TIFF, this will continue to be a problem for TIFF and thus GeoTIFF.

4. You typically want to keep compression to None or Packbits. Packbits is the TIFF implementation of run-length compression (RLE). I have never run access a product which does not support Packbits. I have in the past run access software which did not support JPEG and LZW inside a TIFF wrapper, but these are becoming less and less frequent.

My lowest common test software for TIFF data is Microsoft Windows Picture and Fax Viewer. It supports: Striping, RGB, No Compression; or Striping, RGB, Packbits Compression; or Striping, RGB, LZW Compression. If these tools can read TIFF, everything can (Microsoft is kind of cheap.)

5. While BigTIFF (aka TIFF-64) is the standard for TIFF data >4GB in file size, it is not widely adopted yet. Consider delivering a compressed image until it is more widely adopted. ECW and MrSID are excellent solutions within the geospatial community. In the future JPEG 2000 may become a good format for portability in and out of the geospatial community.

So, how do I know I have a true GeoTIFF? A true GeoTIFF will have all horizontal map coordinate, projection information contained within the TIFF tags. It even supports vertical projection information. For a test, move any ancillary files (non-TIFF files) to another folder and display the TIFF in ERDAS IMAGINE 9.1 SP4 or higher version (9.2 SP1 is the current version). Use the TIFF and GeoTIFF tag viewer within the Image Info capability to examine the projection information. If you see your projection defined correctly, then the GeoTIFF is a true GeoTIFF.

Hello, Geospatial Community.

After years of helping students, professionals, and colleagues with various remote sensing, GIS, and photogrammetry challenges... I have now found a forum where I can express my opinions in the wider community. I hope this blog will be a tool where I can facilitate communication on remote sensing and related geospatial technologies.

I have done my time in private sector data development, academia and private sector software development. Some of my strongest interests are in change detection and vegetation analysis.