Spatial Times's Blog

Velocity Grid Transformations in ArcGIS

Wed, 08 Apr 2026 22:00:00 GMT

Coordinate reference systems and datum transformations are one of those GIS fundamentals that usually ‘just work’, right up until they don’t. Well, we all assumed it just worked, but not always in the way we thought. “Just use the first transformation on the list” isn’t a scientific approach, but thankfully, the software did attempt to guide us. However, the software could only do so much, especially when the best transformations weren’t even an option.

While the implications are broad, the real impact is felt most strongly in Canada, where modern datum management relies heavily on velocity-based transformations. We will focus on NAD83(CSRS) NTv2 velocity grid transformations since these are the first ones to be implemented in ArcGIS Pro 3.6, but more are coming in future releases.

A Quick Recap: Why CSRS Transformations Were a Problem

For anyone working with Canadian data, transformations between NAD83 CSRS realizations have always been a bit uncomfortable in ArcGIS (or any desktop GIS). For years, we’ve known the issue: NRCan moved to NTv2 velocity grid transformations for later CSRS re-adjustments, but ArcGIS didn’t support velocity grids. That left GIS users in an awkward position. We either accept partial transformation support or resort to external tools like NRCan’s TRX when accuracy of transformations really mattered.

That finally changed.

With ArcGIS Pro 3.6, Esri has added support for velocity grid transformations, aligning ArcGIS Pro with modern geodetic practice. In this initial release, Esri leveraged NRCan’s publication of velocity grids to make them the first ones to use this new approach in GIS software.

This table lists all the transformations based on the new NTv2 velocity grid approach in Pro 3.6

Blame Canada?! Nah, we are just fashionably late early:

NRCan defines transformations between NAD83(CSRS) realizations using NTv2 velocity grids, not traditional static NTv2 grids. These velocity grids account for crustal motion over time and are essential for more accurate transformations between CSRS versions.

Until recently, desktop GIS software had some related limitations:

No support for published NTv2 velocity grids;
Many CSRS realization transformations were not available;
Results between GIS transformations and NRCan’s TRX tool didn’t align for NAD83(CSRS) realizations;
EPSG didn’t even register these transformations until March 2024;

In short, the math existed, but ArcGIS Pro couldn’t leverage the file format. Side note as of writing: this still isn’t available in QGIS yet.

NAD83v80VG East Uncertainty Map ~NRCan

What Changed in ArcGIS Pro 3.6

As of ArcGIS Pro 3.6, Esri has implemented support for velocity grid transformations using the published NRCan NTv2 velocity grids. Specifically:

ArcGIS Pro now supports transformations between NAD83(CSRS) v2 through v8;
These transformations use the latest velocity grid, NAD83v80VG.gsb;
The required grid files are included via the ArcGIS Coordinate System Data installer;
Transformations are available directly through standard ArcGIS Pro projection workflows;

This is a big deal. It means ArcGIS can now perform the same class of transformations NRCan has been promoting for years.

ArcGIS Pro using a 2-step transformation including a NTv2 Velocity Grid

Why This Matters

This update closes a long-standing gap for datum transformations in the software:

ArcGIS Pro can now perform modern, authoritative velocity grid transformations;
Aligns with EPSG registered definitions;
Is the first phase to supporting NATRF2022 migrations in North America;
Differences between ArcGIS and NRCan transformations are effectively eliminated (at the anchor epoch);
No more awkward disclaimers in project documentation (because we were all doing that… RIGHT?!);

This is also a first, and critical step for ArcGIS Pro as we get closer to the release of the North American Terrestrial Reference Frame of 2022 (NATRF2022). Moving from older NAD83 projections into more modern systems of NAD83(CSRS) or NATRF2022 will be possible with this enhancement and expansion over time.

Transformations that could be leveraged from NAD83 > NAD83(CSRS)v3 > NAD83(CSRS) v6/7/8. Leveraging NTv2 and a second NTv2 Velocity Grid.

Epochs: One Important Limitation in ArcGIS

There is one important caveat to the ArcGIS Pro implementation of the velocity grid transformations. Although NTv2 velocity grids support transformations at different epochs, ArcGIS Pro currently supports transformations only at the anchor epochs of the CSRS realization. In practice, this is fine for the majority of GIS workflows. If you’re doing time-aware tectonic modelling, you’re probably not using GIS software anyway. Still, it’s something to be aware of if you are comparing results against the full desktop NRCan TRX tool.

Why Everything Uses the v80 Velocity Grid

If you’ve poked around the Esri Coordinate System Data installation folder, you might have noticed something interesting: ArcGIS installs three velocity grids:

NAD83v6VG.gsb
NAD83v70VG.gsb
NAD83v80VG.gsb

Although all 3 are installed, only v80 is actually used. Why? This comes straight from the EPSG Geodetic Registry (and confirmation from Esri’s Bojan Savric): Transformations for CSRS v6 and v7 that used their matching velocity grids were superseded by newer transformation variants that always reference the latest velocity grid (v80). According to EPSG and NRCan: Users should use the most recent velocity grid, with older velocity grids retained only for backward compatibility. To note: CSRS v6, v7, and v8 all share the same anchor epoch (2010.0).

The result? Transformations from CSRS v3 to v6, v7, or v8 produce identical coordinates in ArcGIS Pro, and that is the expected behavior. So why include the older versions?

“users who wish to work with superseded transformations can create their own custom geographic transformations and ensure possible backward compatibility with older product releases, should we later add support for those superseded transformations”. ~ Esri’s Bojan Savric on Esri Forums

Final Thoughts

This is one of those changes that looks small on a release but is going to be critical as organizations update their datums over time. Even if you don’t plan to move from NAD83 (EPSG: 4269) to the NAD83(CSRS)v8 (EPSG: 10414) datum just yet, NATRF2022 is coming, and it is likely we are all headed in this direction.

FYI: GNSS correction services in Canada are already using epoch 2010 in most cases, so if you want to align other/older data with it, say NAD83, these transformations produce better (and different) results than the options in ArcGIS Pro 3.5 and earlier.

Final, final thought: No matter how you transform between datums, please don’t forget to document the transformation used. The software default transformation changes over different releases (first in the list), and no, we will not remember it in our GIS brains.

Updated final, final thought: This enhancement is also available in ArcGIS Enterprise v12.0! But no, you can’t use the new Coordinate System installer with older software versions as they can’t handle the velocity grids.

Helpful Links

_{Cover image: Photo by Simon Lee on Unsplash. April 08, 2026.}

Say Goodbye to Concurrent Use Licenses

Wed, 13 Aug 2025 22:00:00 GMT

The GIS landscape is changing once again, and this time it’s the familiar Concurrent Use (CU) license model that’s being phased out. Esri has announced the deprecation of both Concurrent Use licenses and the ArcGIS License Manager. This will directly affect users of ArcGIS Pro and ArcGIS CityEngine currently using CU. While this shift aligns with broader trends in software licensing, it does mean that organizations relying on these legacy models will need to start planning for a transition.

Why the Change?

Esri is transitioning to a licensing framework that emphasizes secure, permission-based access and centralized management of GIS content. The recommended path forward is through ArcGIS user types, which offer “a more modern and scalable approach to managing access across teams and platforms” (Esri). Unlike Concurrent Use, which leverages a shared pool of licenses, named users are assigned to an individual person.

Don’t Want to Change?

Sadly, you will need to decide if you want software upgrades or Concurrent Use. ArcGIS Pro can still use CU licenses with the existing software (3.5.x and possibly 3.6.x) if you have perpetual CU licenses. However, CU licenses are tied to a maximum version, and moving forward, there will be no way to request newer licenses as you upgrade ArcGIS Pro. There will definitely be an impact on companies that use CU for large numbers of casual users to manage costs, global CU pools, and some emergency management mobilization scenarios.

Key Dates to Keep in Mind

Here’s a breakdown of the timeline and what it means for your organization:

December 1, 2025: Maintenance renewals will transition to equivalent ArcGIS user types. MyEsri will also remove the ability to convert ArcGIS Pro add-ons to Single Use or Concurrent Use licenses—or revert them back.
March 1, 2026: Last date for software releases that support Concurrent Use licenses.
Q2 2026: ArcGIS Pro and CityEngine versions released after this point will no longer support Concurrent Use.
Through 2028: ArcGIS License Manager will continue to support older versions of ArcGIS Desktop, Pro, and CityEngine until those products reach the Mature phase of their lifecycle.

Time to Evaluate Options

If your organization is still using Concurrent Use licenses, now is the time to evaluate your options and discuss them with your Esri reps. Before updating to newer versions of ArcGIS Pro or CityEngine in 2026 and beyond, you’ll need to transition to a supported license type. This means using ArcGIS user types or Single Use licenses (for ArcGIS Pro only) and/or evaluating future GIS software needs. Failing to do so will result in updated software that simply won’t be setup to work properly. There are options beyond ArcGIS Pro, but for many GIS departments, the integrations run too deep.

While this change may require some adjustment, it’s part of a broader effort to modernize how GIS software is accessed and managed. For many organizations, the move to user types will offer greater flexibility and control. Regardless of why it’s happening, it’s important to plan ahead to avoid disruption.

Helpful Links

_{Cover image: Photo Generated with support from Copilot. August 2025.}

ArcGIS Enterprise 11.3 Released

Sun, 26 May 2024 22:00:00 GMT

ArcGIS Enterprise 11.3 is here! There are many significant updates and new capabilities in ArcGIS Enterprise 11.3, emphasizing its continuous evolution to meet technology and industry needs. This also adds many of the new ArcGIS Online features to in-house Enterprise deployments with new Apps and features.

What’s New

ArcGIS Enterprise 11.3 introduces enhancements for data management, administration, and applications, with a focus on new features for numeric and temporal data types, and updates to ArcGIS Living Atlas content.

11.3 is the new Long-Term Release (LTS) version of Enterprise: Yes, this is the one to install/upgrade to (or wait for the first patch).
Installation tweaks: The Portal for ArcGIS and ArcGIS Server packages now come in two install files (.exe and .exe.001), and there’s a new .NET 8-based ArcGIS Web Adaptor (IIS) requirement.
Data Types: Four new field data types for numeric and temporal data are available based on the same field types released in ArcGIS Pro 3.2:
- Big Integer: Numeric 64-bit field type storing values up to 53-bit safe integers and has a storage size of 8 bytes. FYI: Some client applications may not fully support this field type.
- Date only: Store date values of day, month, and year. No time values are stored for date-only fields.
- Time only: Store time values in local time with millisecond precision. No date values are stored.
- Timestamp offset: Store date and time values. The time values include information about the time offset from coordinated universal time (UTC).
Retirement Notices: ArcGIS GeoPlanner is retired in version 11.3, with additional details and transitioning recommendations available in the deprecation announcement.
ArcGIS Urban: New App to design, manage, and measure urban developments in your city and collaborate with stakeholders. ArcGIS Rural isn’t a thing, but Esri does have an Ideas Page if you feel left out in the field.

New in ArcGIS Portal

Lots of additions to Portal in this release to hopefully catch up to ArcGIS Online:

Organization Administration: Many new administration options. Thankfully you can bulk delete members and transfer content now. Living Atlas also gets some love with the ability to update Atlas items when new content is available.
Data Management:
- You can schedule the cache to be rebuilt for hosted vector tile layers that have an associated hosted feature layer or map image layer.
- A new editor for metadata (and the old editor renamed to… you guessed it “classic”).
- Additional authentication methods (Entra ID and SAS token).
- Can now set a schedule to synchronize bulk published layers with their source datasets.
Mapping and visualization: Many new visualizations and colour ramps, chart type (Box plot); a Box plot chart; and Many enhancements to sketch layers.
A very cool addition for custom tools within the MapViewer: Custom web tools are geoprocessing workflows, such as models, scripts, or notebooks, that are published and hosted in ArcGIS Enterprise as Tool or Geoprocessing Service items and are accessed from the Map Viewer Analysis pane.

New in ArcGIS Server

This is more of a hardening release for LTS, but it does provide some new features:

Administer custom data providers in Server Manager.
New cloud data store authentication methods added (Microsoft Entra ID and SAS).
ArcGIS Enterprise 11.3 is the last release to support Microsoft SQL Server 2017 (no more SQL Server 2016 or PostgreSQL 12x).

Installation Requirement Details

There are some requirement changes before you do a fresh install or migrate an existing Enterprise deployment. ArcGIS is only supported on 64-bit CPUs with x86-64 architecture at this point, so don’t run out and buy a fancy new Microsoft Surface laptop with a Snapdragon X Elite (Arm 64 architecture) just yet. Some more important changes:

ArcGIS Enterprise components require the Microsoft Visual C++ 2015–2022 Redistributable (x86 and x64) minimum version 11.3.50790. If installing using the setup.exe files, this will be automatically installed if missing. However, if using the setup.msi the file will not install automatically. If you are savvy enough to use the MSI, you are savvy enough to install this yourself.
ArcGIS Web Adaptor: New to this release, the Web Adaptor will require the ASP.NET Core Runtime - Windows Hosting Bundle 8.x. The Web Adaptor also requires Microsoft Web Deploy 3.6, although this is the same requirement as 11.2.

Helpful Links

_{Cover image: Photo by KickABrick on FLICKR. Modified May 26, 2024.}

ArcGIS Pro 3.3 Released

Thu, 09 May 2024 22:00:00 GMT

ArcGIS Pro 3.3 was just released by Esri. Although this is a minor release, it has some bigger implications and features. Overall, this is related to one of the new Long-Term Support (LTS) releases for ArcGIS Enterprise (11.3) that will be coming in the next few weeks. This release also changes a few of the pre-install/upgrade requirements so you might not be able to just click the upgrade button. This is also a designated “Network Management Release”, important for those in the Utilities and Telecom sector.

New ArcGIS Pro 3.3 Requirement: Microsoft .NET Desktop Runtime 8x

Before installing ArcGIS Pro 3.3, you will need to install Microsoft .NET Desktop Runtime 8.0.x (x64). Don’t worry, it won’t install/upgrade otherwise. The version linked in MyEsri is preset to download 8.0.0 as of writing, but Microsoft updates this very frequently so you can already grab 8.0.4 from Microsoft if you want the latest and greatest.

New ArcGIS Pro 3.3 Requirement: WebView2 Runtime 117+

You will also need the Microsoft WebView2 runtime version 117 or later. This is likely already installed on your computer unless you did some fancy Windows SlipStream install. This runtime is used to display HTML and JavaScript content in Pro (popups, etc).

What’s New in ArcGIS Pro 3.3

Too much to cover here, but some of the highlights are:

Enhanced Water Flow modelling with Flood Simulation: ArcGIS Pro 3.3 introduces improved capabilities for modeling water flow. Whether you’re analyzing drainage patterns or managing water resources, these enhancements allow you to create new mapping products.
Storytelling: The new Presentations feature enables you to create compelling narratives directly within ArcGIS Pro. Combine maps, charts, images, and text to communicate your spatial analysis results effectively. Don’t want to use PowerPoint? Create a “ProPoint” presentation (patent pending). It’s like giving your maps a bedtime story—complete with dragons, unicorns, and maybe a sprinkle of GIS magic.
Simplified PDF Integration: In previous versions, adding PDF files to your project required converting them to other formats, such as TIFF. ArcGIS Pro 3.3 eliminates this step, allowing you to directly import PDFs into your map layouts. This enhancement simplifies document integration and preserves the original quality of your PDF files. Way cooler AND less pixelated.
Hyperlinks in Text Elements: Now you can add hyperlinks to text elements in your layouts. Link to external websites, documents, or other resources directly from your map annotations. This feature enhances interactivity and provides additional context for your audience. I did some preliminary tests on this and the results are great within the Layout itself, but don’t always work when exporting to PDF as tested below:
- Create a textbox with LINK tag only: Exports text only, no hyperlink created. <LINK url="https://www.spatialtimes.com">SpatialTimes</LINK>
- Create a textbox with LINK AND some text preceeding the link: Exports a hyperlink. Hi <LINK url="https://www.spatialtimes.com">SpatialTimes</LINK>
- Create a textbox with LINK where the link text is the URL, PDF reads as a hyperlink. <LINK url="https://www.spatialtimes.com">https://www.spatialtimes.com</LINK>
- Sadly, none of the exports to PDF meet WCAG 2.0 AA for hyperlinks. Further, no PDF Layout exports from ArcGIS Pro meet S.508/AODA as tag order and proper tag types are still lacking since being introduced.
Export Attachments: When exporting data from feature layers, ArcGIS Pro 3.3 allows you to include attachments associated with features. This is especially valuable for field data collection workflows where photos, documents, or other files are linked to specific features.
Geodatabase Schema Replication: The Schema Report tool generates a detailed report of your geodatabase schema, including feature classes, domains, relationships, and more. In ArcGIS Pro 3.3, you can now use this report to create a new geodatabase with the same schema. This feature simplifies database replication and ensures consistency across projects.
Combined Visualization Techniques: ArcGIS Pro 3.3 enhances your cartographic options by allowing you to combine multiple visualization techniques within a single layer. For example, you can display point features as both graduated symbols and heat maps. Let’s all start making those Graduated Heat Symbols.

For more details, you can refer to the official ArcGIS Pro 3.3 release notes and explore the full list of enhancements and bug fixes. Also wanted to mention that existing 3.x custom SDK modules/extensions should continue to work with ArcGIS Pro 3.3. Recompiling isn’t required unless you need to crack the code open again.

Helpful Links

_{Cover image: Photo by Mockaroon on Unsplash. Modified May 9, 2024.}

ArcGIS Pro Layouts and Accessibility

Mon, 20 Feb 2023 22:00:00 GMT

Creating beautiful cartographic map layouts can be a difficult journey. Titles, fonts, and picking a north arrow that matches your design (much like your socks matching your belt). Yeah, those difficult decisions. Let’s not forget the accessibility of our design as well. Forgetting accessibility during the design process is like forgetting the map title - it shouldn’t happen. Accessibility should be part of the design process, not an afterthought. This will drastically reduce the time needed to remediate at the end and allows more people to enjoy your map. And yes, it’s also a legal requirement in many jurisdictions - this is where it gets interesting.

This started as a simple article on making a great, and accessible PDF layout with ArcGIS Pro. However, I went too far down the rabbit hole, and we end up with this article. In short: ArcGIS Pro (at version 3.0.4) doesn’t generate accessible layouts that meet Section 508 or AODA requirements. Note: An explaination to this statement is in the ArcGIS Pro section below. That said, ArcGIS Pro does have some accessible features and exports some PDF elements - which is a great start.

It helps when we have tools that support building and testing accessible layouts. As mentioned, ArcGIS Pro does have some layout “accessibility” tools, and they should expand over time. Adobe Pro has an “accessibility” checker to report on some accessibility issues. The problem is: “Accessibility” is a broad term. Neither ArcGIS Pro nor Adobe Pro is creating/checking WCAG 2.0 AA accessibility. What?!

Let’s step back for a second.

Accessibility and Section 508, AODA, and WCAG

Accessibility is something we strive for and there are many different levels of accessibility. There are also differences between Section 508 and Web Content Accessibility Guidelines (WCAG) although all 3 terms are used interchangeably by some. Section 508 and WCAG are, in fact, quite different. Since I live in Ontario Canada, I’m adding AODA in there too - it will make sense later.

Section 508

Section 508 is a federal law in the United States. This law requires federal agencies and those receiving federal funds to acquire, develop, use, and maintain information and communications so people with disabilities can access it. In 2018, Section 508 was enhanced with updated requirements for information and communication technology (ICT). This update adopted WCAG 2.0 AA as the criteria for a website, electronic document, and software accessibility.

AODA

Accessibility for Ontarians with Disabilities Act (AODA). This is Ontario provincial law that sets out a process for developing and enforcing accessibility standards much like Section 508. Public content must meet the WCAG 2.0 AA success criteria. There are 2 exceptions, but they don’t apply to our PDF map layouts.

WCAG

Web Content Accessibility Guidelines (WCAG) were “developed to provide a single shared standard for web content accessibility that meets the needs of individuals, organizations, and governments internationally” W3.org. These guidelines have been adopted for digital documents (Word, PDF, etc) over the years as well. There are many versions of WCAG, with version 2.0 categorized into three levels of conformance: A (minimum), AA (mid-range), and AAA (highest). Both Section 508 and AODA must meet WCAG 2.0 AA as their compliance requirement.

Accessibility in Adobe Pro

Adobe accessibility checker isn’t specifically checking for WCAG 2.0 AA compliance. It does validate many WCAG criteria, but passing the Adobe checker doesn’t mean the document passes WCAG 2.0 AA. Some items can’t be checked, and other items can be complex and might not apply to all accessibility standards. So if you don’t pass the Adobe Pro accessibility checker: You have a lot of remediation to do. If you do pass the Adobe Pro accessibility checker: You probably still have some remediation to do.

Don’t be mad at Adobe/ArcGIS Pro (that Pro-ism didn’t escape me), there are more than 2 accessibility laws in this world and many accessibility guidelines. Providing tools to help us along our accessible journey is what we are looking for.

ArcGIS Pro and PDF Layout Accessibility

Oh yeah, the original article title. Based on the information above, what I’m really talking about is using ArcGIS Pro to create WCAG 2.0 AA PDF layouts. And now we know, ArcGIS Pro and Adobe Pro can help get us started, but can’t get us there like an Uber. Although great tools such as the color vision deficiency simulator do highlight the efforts Esri is making in this area.

Esri does have a few good blog articles, help documents, and demonstrations on using ArcGIS Pro to generate accessible PDFs. I’m honestly grateful they are bringing attention to the subject and adding related functionality. They also do a good job of NOT saying “WCAG 2.0 AA accessibility”. It may look synonymous, but there is purposeful separation.

Is WCAG not possible in ArcGIS Pro?

*It is possible, but not by using regular ArcGIS Pro layout elements (scale bar, north arrow, legend). And other elements that do have tag structure aren’t complete (tables, headings). Tag types, tag order, and changing the order isn’t available at this time either. So it might be possible - but keep a copy of Adobe Pro for the time being.

Now What

If Section 508 or AODA apply to you - let’s celebrate Esri and Adobe for their accessibility efforts so far. They are making improvements and this does make our job easier when remediating PDFs. Once we export our maps to PDF, we run the Adobe Accessibility checker to fix obvious issues. Then we dive deeper into the remediation and continue on our journey.

I’m hoping to post a second article on this topic where I get into the map layout remediation process in the near future.

Helpful Links

_{Cover image: Photo by Fauxels on Pexels. Modified Feb. 20, 2023.}

ArcGIS Pro 3.0 Released

Mon, 27 Jun 2022 22:00:00 GMT

Initially scheduled for release in June, Esri was able to release ArcGIS Pro 3.0 with a few days to spare. With the Esri User Conference just around the corner, I’m sure it will get lots of attention on the UC’s big stage. There isn’t an official announcement of the release yet, but with the ArcGIS Pro update service going offline for a few hours, it seemed like something big was happening - now we know.

Ready to Upgrade to ArcGIS Pro 3.0

To upgrade to ArcGIS Pro 3.0, just open the currently installed version and you will be prompted that an upgrade is available. It doesn’t specify the significance of the update - but don’t be fooled, this is significant.

In short, the biggest things to know when upgrading to ArcGIS Pro 3.0 are:

You will require .NET 6 Desktop Runtime x64: This needs to be installed before ArcGIS Pro 3.0. If you just run the upgrade from pro 2.x it will fail to install. However, install the runtime, and head back to the upgrade and the install should be successful (without another download needed).

If using concurrent licenses, update to ArcGIS License Manager 2022.0 first.
ArcGIS Pro 3.0 is a breaking change release: Esri uses the semantic versioning specification - which means when the first number changes, it’s a major release. Major releases tend to break some areas of backward compatibility. There are some breaking changes with this release (including the update to .NET 6).

Breaking Changes

Projects created or saved using ArcGIS Pro 3.0 can’t be opened directly in ArcGIS Pro 2.x. After updating to ArcGIS Pro 3.0, when you first save a project created in an earlier version, you will receive a notification of this breaking change. You can still create a project package if you need to open it in a previous version as that functionality is still a workaround. Pro 1.x and 2.x projects can be opened in 3.0, but if saved they will become 3.0 projects.

Some geodatabase functionality is also not backwards compatible. Attribute Rules, Annotation, Dimensions, and Utility networks created in 3.0 will not work with 2.x releases. Other limitations exist and it’s recommended to read the migration page before upgrading geodatabase functionality.

What’s New

Beyond the new splash screen, Pro has many new features available - so I’m not going to attempt to list them all here. Here are some notable items:

New Package Manager: Allowing us to manage multiple Conda environments for use in ArcGIS Pro. Yes, Conda was always there but there is now an interface to manage versions, repair environments, and rename environments.
Add maps to reports: Sure you can add reports to maps, but what about maps to reports? This is now available for those that want to add some spatial goodness.
Export Map Presets: These presets allow us to save all the settings for a particular export type and shortcut to these settings in the future. When you export a map or layout, you can select a default preset or a custom preset created. This isn’t handled the same as other user favourites, but hopefully, this gets streamlined over time.
New MGWR Spatial Statistics: Multi-scaled Geographically Weighted Regression (MGWR) tools are here. Huh?! That’s right, some pretty advanced predictive and explanatory spatial analysis tools are now available - it is the Science of Where after all.
New Hydrology Tools: Hydro conditioning is not required with these new tools which aim to make it easier to create hydro surfaces with less effort.
Credit Consumption Estimation for Batch Geocoding: Worried you will use too many credits when Geocoding? A simple estimation tool is now available so you no longer need to keep an Excel Spreadsheet calculator off to the side.
And lots more: Improved 3D, more CAD/BIM support, Synthetic Aperture Radar Toolset, Mobile database publishing.

Helpful Links

_{Cover image: Photo by Daniel K Cheung on Unsplash. Modified June 27, 2022.}

Using JavaScript's Geolocation API

Sun, 23 Jan 2022 22:00:00 GMT

The location of people and devices is handy if you want to display local weather, show a map, get directions, or even tweak advertisements by region. These examples of GeoLocation are just the tip of the iceberg, but you get the idea - location, location, location. Web browsers and javascript started supporting geolocation back in 2009 (Firefox v3.5). This happened with very little fanfare since very few people owned smartphones, and tracking user location wasn’t big business (yet).

Today, we assume our phone knows where we are, where we are going, where the device is, and if we are near them. Sounds complicated, but we are just tapping into an API that can ask the device for location details. With GNSS chips (AKA “GPS”) in all our phones, how easy is it to tap into this location? With JavaScript’s GeoLocation API, it’s easier than you might think.

JavaScript GeoLocation API

The JavaScript Geolocation API ’provides access to geographical location information (lat/long) associated with the hosting device’ W3C. This enables web apps to request location data - whether it be your phone, laptop, or another smart device. Even without a GNSS/GPS chip, device location is still possible via cellular triangulation, network IP, etc. The accuracy will vary and is impacted by this and a few other factors. Thankfully, we don’t need to worry about those details, we just ask the GeoLocation API for the location and the device will take care of the rest - if permission is granted.

The GeoLocation API provides a few different ways to access this information:

getCurrentPosition: Get the device’s current location
watchPosition: A handler function that is called automatically when the location of the device changes

There are three arguments with both approaches:

A success callback (required)
An error callback (optional)
An Options object (optional)

A Simple Request for Location

Call the getCurrentPosition once, we can use: navigator.geolocation.getCurrentPosition(successCallback, errorCallback);. If the browser supports ES6, you can always use the arrow approach too:

navigator.geolocation.getCurrentPosition((pos) => {
  myFunction(pos.coords.latitude, pos.coords.longitude);
});

By default, the API answers as quickly as possible with a low accuracy result. Great if you just want to know the general area or city. If you want a more accurate result you also need to supply the Options object and enable high accuracy.

Extra options object

The options object is optional but has some important settings worth reviewing. From MDN:

maximumAge: A positive long value indicating the maximum age in milliseconds of a possible cached position that is acceptable to return. If set to 0, it means that the device cannot use a cached position and must attempt to retrieve the real current position. If set to Infinity the device must return a cached position regardless of its age. Default: 0.
timeout: A positive long value representing the maximum length of time (in milliseconds) the device is allowed to take to return a position. The default value is Infinity, meaning that getCurrentPosition() won’t return until the position is available.
enableHighAccuracy: A boolean value that indicates the application would like to receive the best possible results. If true and if the device can provide a more accurate position, it will do so. Note that this can result in slower response times or increased power consumption (with a GPS chip on a mobile device for example). On the other hand, if false, the device can take the liberty to save resources by responding more quickly and/or using less power. Default: false.

We can add the options by sending them with our request:

var geoOps = {
	enableHighAccuracy: true,
	timeout: 10000 //10 seconds
}
navigator.geolocation.getCurrentPosition(successCallback, errorCallback, geoOps);

Geolocation callback

With our call complete, we now wait for our results in the success callback where we can use the location to enhance the user experience.

function successCallback(pos) {
    var lat = pos.coords.latitude;
    var lng = pos.coords.longitude;
    alert("Latitude : " + lat + " Longitude: " + lng);
}

That is pretty much it. This extremely useful API can be used with your app, such as: showing location on a map, tracking, directions, or grabbing some weather data.

GeoLocation Limitations

Geolocation functionality is available in all major browsers (desktop and mobile), including IE - yeah, crazy I know. But just because it’s supported, doesn’t mean our code will always bring back a valid location. Here are a few main reasons why we might not get what we are looking for:

The Geolocation API will only work on secure contexts such as HTTPS. If your site is hosted on a non-secure origin (such as HTTP) the requests to get the users’ location will no longer function in most modern browser versions.
- An exception to the HTTPS rule is local file:// testing which still works.
The Geolocation API might be blocked by the user/settings. In most desktop/mobile operating systems users can block location services from being exposed to the browser. In addition, the user will be prompted to provide permission for an individual website as per W3C specifications.
A request timeout can occur if the data takes too long to return. An optional PositionOptions object has a timeout setting representing the maximum length of time (in milliseconds) the device is allowed to wait for a return a position. The default timeout value is Infinity, which is a long time to wait. It’s a good idea to set a reasonable timeout. Even slower response times should be expected if using this in combination with enableHighAccuracy set to true.

Mapping the Location

Popular mapping APIs have their own techniques to get the user location. In most cases, this is just a wrapper around the Geolocation API with some additional properties that focus on use ease of use within the map environment. For example, Leaflet’s location function uses Geolocation and also provides some shortcuts to working with their map object, providing a copy of the lat/long pre-formatted to the leaflet standard. Overall this means we can use the GeoLocation API directly, or using Leaflet’s map.locate wrapper if we want to set additional options such as the map’s setView and maxZoom. The returned object just has a few minor differences.

Leaflet with vanilla Geolocation API

A brief example of getting the user location with the Geolocation API and adding to a map.

And below is the javascript code using the Geolocation API. With the results, it grabs the date, latitude, longitude, and accuracy from the return object and uses them with the map, popup, and prints some details to the page.

// Setup Geolocation API options
const gpsOptions = {enableHighAccuracy: true, timeout: 6000, maximumAge: 0};
const gnssDiv = document.getElementById("gnssData");
// Geolocation: Success
function gpsSuccess(pos) {
    // Get the date from Geolocation return (pos)
    const dateObject = new Date(pos.timestamp);
    // Get the lat, long, accuracy from Geolocation return (pos.coords)
    const {latitude, longitude, accuracy} = pos.coords;
    // Add details to page
    gnssDiv.innerHTML = `Date: ${dateObject} 
        <br>Lat/Long: ${latitude.toFixed(5)}, ${longitude.toFixed(5)} 
        <br>Accuracy: ${accuracy} (m)`;
    const radius = accuracy / 2;
    layerGpsGroup.clearLayers();
    // Zoom to the location
    map.setView([latitude,longitude], 12);
    //Add a marker and radius based on accuracy to map 
    L.marker([latitude,longitude]).addTo(layerGpsGroup)
        .bindPopup(`Lat/Long : ${latitude.toFixed(5)}, ${longitude.toFixed(5)}`)
        .openPopup();
    L.circle([latitude,longitude], radius).addTo(layerGpsGroup);
}
// Geolocation: Error
function gpsError(err) {
    console.warn(`Error: ${err.code}, ${err.message}`);
}
// Button onClick, get the the location
function getLocation() {
    navigator.geolocation.getCurrentPosition(gpsSuccess, gpsError, gpsOptions);
}
// Setup the leaflet map
const map = L.map('map').setView([44, -79], 9); // Toronto area set as default
const osmTileLayer = L.tileLayer("https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png", {
    maxZoom: 19,
    attribution: '&copy; <a href="https://www.openstreetmap.org/copyright">OpenStreetMap</a> contributors'
}).addTo(map);
const layerGpsGroup = L.layerGroup().addTo(map);

ArcGIS and Log4j Vulnerabilities

Wed, 15 Dec 2021 22:00:00 GMT

The investigation continues into the Log4j vulnerability (AKA Log4Shell) as CVE-2021-44228 was disclosed on December 9, 2021, and related CVE’s (2021-45046, 2021-4104) were disclosed on December 14, 2021.

Article updated January 23, 2022: Added links to more software instructions (Workflow and GeoEnrichment Servers) and a Related Technologies section.

Is ArcGIS Software at Risk?

Like many enterprise software packages, ArcGIS server software uses the OpenSource Log4j module - so regardless if exploitable, take all mitigation measures available.

As of writing, no exploits have been identified with ArcGIS products. However, closing the potential vulnerability is still best practice and highly recommended. Errr, this is more than a recommendation - mitigate by patching regardless. Even if you are using the Windows version of ArcGIS Server/Enterprise/Portal, the ArcGIS software uses Java behind the scenes - the Web Adaptor is then used to proxy traffic from Windows IIS to the real GIS server itself. So Linux or Windows, whatever version, fix it.

Props to Esri as they’ve been researching and posting on the situation since the start on their blog. Keep checking their pages as they update daily. At this point, Esri has already posted scripts to help mitigate the threat across many of their server products. This ST article was delayed until now for this reason.

Which Esri Software is Vulnerable?

There are only potential vulnerabilities at this point. Esri currently stated there are no known exploits with ArcGIS products. However, be diligent and take the recommended mitigation measures now.

“No known exploit available for any version of a base ArcGIS Enterprise deployment (including the ArcGIS Server, Portal for ArcGIS, and ArcGIS Data Store components) or stand-alone ArcGIS Server at this time” ~esri blog.

So if you are using ArcGIS server technologies (all 10.x releases), use the links below to start patching.

Software to patch with links to the script downloads and instructions:

Script Success

After going through the workflow, it is pretty straightforward. I successfully ran the --list command with no issues. The only hiccup I had was running the actual --delete command. I was logged in as an Administrator, but I still had to open the Command Prompt with “Run as administrator” for the second execution to work properly. Otherwise, everything went smoothly. Note: It automatically makes backups of the files for you too which is a nice touch. Update: instruction pages do explain the admin access in more detail now.

Now, stop reading, start patching.

Related Technologies

Related technology/software notes (added Jan 06, 2022 from Esri blog updates):

ArcGIS Notebook Server: The underlying framework does not contain or for 10.7x does NOT include the vulnerable JMSAppender class. The Docker container image contains Log4j, however for a person to be able to execute the component they would need to be granted permissions to the notebook container, so Log4j does not present additional RCE risk in this configuration. A patch will be made available in the future regardless.
ArcGIS Online: Esri continues to perform patching of ArcGIS Online systems including some updates with the latest log4j version of 2.17 and is continuing to evaluate the CVE as well as all relevant third-party fixes as they become available.
Esri Managed Cloud Services: EMCS Advanced and Advanced+ have implemented web filter mitigations for Log4j vulnerabilities and have applied the scripts that remove the JNDILookup class to all affected systems.
ArcGIS Monitor: Does not contain Log4j
ArcGIS Pro: All ArcGIS Pro versions under General Availability support contain Log4j, but are not known to be exploitable as the software does not listen for remote traffic.
Esri Geoportal Server: This open source product was updated to version 2.65 on Dec 17th to resolve Log4j issues, please upgrade to this latest release.

Helpful Links

Esri Blog post on Log4j, updated frequently
Esri Trust Center
CISA Log4j Vulnerability Guidance

ArcGIS Pro Favorites and Folders

Tue, 21 Sep 2021 22:00:00 GMT

ArcGIS Pro has been able to store favorites since the 2.x releases. How this is managed in Pro has changed slightly over the years. Well, I guess it started with no favorites, but we don’t like to talk about Pro 1.x so let’s just pretend it never happened. If you have corporate database connections or projection files, setting this up for users makes day-to-day GIS’ing a little easier.

A Short History of Favorites

Setting favorites in ArcGIS Desktop (ArcMap/Catalog) was a mixed bag of files and locations: database connections could be copied into your roaming profile folder; Custom/favorite projections files stored in a different folder; Folders shortcuts stored in the ArcCatalog.gx binary file. From that point forward they would be available in every MXD or ArcCatalog instance. Overall, this worked great with the files stored locally (per user), not great if you wanted shared favorites throughout the organization.

Some of this ArcGIS Desktop ’favorite location chaos’ has been fixed in ArcGIS Pro. The binary GX folder favorites have been replaced with a JSON file and extracted from the binary. As well, you can store some favorites on a shared network which makes corporate updates easier to deploy. However, there are still a few locations where favorites might exist and a new JSON schema that is worth reviewing.

Favorites Pro Every Occasion

There are many types of ArcGIS Pro favorites you might want to set up for yourself or your organization, this includes:

Database Connections
Server Connections (ags, wms, wcs, etc)
Folder Connections (shortcuts)
Projections (prj files)
Custom Transformations
Custom Style File

The abnormally wide table below compares the new ArcGIS Pro favorites locations with ArcGIS Desktop.

Favorite	Type	ArcGIS Pro	ArcGIS Desktop
Database Connection	.sde, .odc	%AppData%\Roaming\Esri\ArcGISPro\Favorites\Favorites.json	%AppData%\Roaming\Esri\Desktop10.x\ArcCatalog\
Server Connection	.ags, .wms, .wcs, .wfs, .wmts	%AppData%\Roaming\Esri\ArcGISPro\Favorites\Favorites.json	%AppData%\Roaming\Esri\Desktop10.x\ArcCatalog\
Folder Connection	UNC path	%AppData%\Roaming\Esri\ArcGISPro\Favorites\Favorites.json	%AppData%\Roaming\Esri\Desktop10.x\ArcCatalog\ArcCatalog.gx
Projection	.prj	%AppData%\Local\Esri\ArcGISPro\Favorites\	%AppData%\Roaming\Esri\Desktop10.x\ArcMap\Coordinate Systems\
Custom Transformation	.gtf	%AppData%\Roaming\Esri\ArcGISPro\ArcToolbox\CustomTransformations\	%AppData%\Roaming\Esri\Desktop10.x\ArcToolbox\CustomTransformations\
Toolbox	.tbx	%AppData%\Roaming\Esri\ArcGISPro\Favorites\Favorites.json	%AppData%\Roaming\Esri\Desktop10.x\ArcToolbox\My Toolboxes\
Style	.style, .stylx	%AppData%\Roaming\Esri\ArcGISPro\Favorites\Favorites.json	%AppData%\Roaming\Esri\Desktop10.x\ArcMap\

In the table, %AppData% refers to the local user’s AppData directory (C:\Users\<user_name>\AppData). This folder is hidden in Windows by default. Enable show hidden items to browse if needed. Beyond that, almost everything in ArcGIS Pro is in the roaming section except for Projections which uses the local subdirectory.

Most favorites are now managed by the Favorites.json file. This file is located in %AppData%\Roaming\Esri\ArcGISPro\Favorites and can contain more than just the Favorites.json file. By default, this is where new favorite files will be created and managed by the favorites pane in ArcGIS Pro. But remember, it’s the JSON file that is the real favorites list:

If you create a favorite by marking an existing item (sde/server connection file), it will copy the file into the favorites folder AND will add the entry to the favorites.json file.
If you create a new favorite from the favorites pane, it will create the new file in the favorites folder AND will add the entry to the favorites.json.
If you manually copy files into the favorites folder it will not show in your favorites list unless you update the favorites.json file manually.
Removing an item from your favorites list in ArcGIS Pro will update the Favorites.json but will not delete the file itself.
Creating a folder favorite doesn’t create a shortcut file, it just updates the JSON file with a new virtual entry.

Coordinate System Projection file favorites

The list of coordinate systems to show in your favorites list is an odd duck and managed separately:

All PRJ files in the user-based favorite directory are considered favorites automatically, with no related JSON configuration.
Marking an out-of-the-box projection as a favorite will create a PRJ file and place it into the user-based directory.
Removing a projection from your favorites will delete the local PRJ in the directory

Custom Transformations

Similar to projections, these have their own location and complexities. First off, these aren’t really favorites, more of a local user configuration. Still worth mentioning as it leverages the user profile and there is no interface to manage these inside ArcGIS Pro. Just place the GTF file into the folder and Pro will pick up the settings:

A custom GTF is stored in its user-based favorite directory.
A custom GTF may require a custom binary grid shift file (.gsb) in the Projection Engine resource directory (resources\pedata directory).
Make sure the transformation settings match the names of the projection files (if they are custom too) for the transformation to show as an option.

Note: Normally, the transformation will not show in any tools as being an available option if the complementing .gsb file is missing (i.e. Canadian NTv2 files). However, some tools will mistakingly still list them - such as the transformation properties of mosaic datasets. For the mosaic situation, this can lead to the issue where the transformation will not take place even if specified. Still, on mosaic properties, it appears there might be a bug where the transformation is still ignored even if both files exist (further investigation is still required as to the cause and might be isolated).

Pro Favorites, Folders, and JSON

As you can see from the favorites table earlier, ArcGIS Pro (currently 2.8.3) still has multiple favorite folder locations; and the roaming favorites folder doesn’t manage favorites at all beyond it being where the Favorites.json file resides. Most favorites can be managed directly in the JSON file or use ArcGIS Pro’s interface to keep the JSON file up-to-date. For deploying favorites to many users, just make a single JSON file and push to all user roaming directories and keep the related files in a single shared drive for easier updates.

Again, the JSON file doesn’t include coordinate systems or custom transformations at this time - both of these are managed in their own user-based favorites folder with no JSON file or shared network options. Maybe we can make an Ideas Page?!.

The User Config File

Although not a managed favorite, an honourable mention to ArcGIS Pro configuration files. These have a few settings that can be managed by users or administrators via XML files. Yes, because using JSON here wouldn’t make sense I guess?! In ArcGIS Pro, the user.config file stores personalized preferences such as geoprocessing, search, and mapping properties. The file also stores the location of recent projects accessed from the start page in ArcGIS Pro.

User user.config configuration file is an XML file located in the user profile:

In Pro 2.5 and below: C:\Users\<user_name>\AppData\Local\ESRI\ArcGISPro.exe_StrongName_\
In Pro 2.6 and above: C:\Users\<user_name>\AppData\Roaming\ESRI\ArcGISPro.exe_StrongName_\

Admin configuration can be created to control a subset of settings from a shared location. This can either be set during an ArcGIS Pro silent install or by editing the Windows Registry. It only controls 4 settings currently: UseLocalHelp, EnableIndexing, IndexNetworkConnections, and SkipServerConnections. This Pro.settingsConfig XML file is further described in the Online Help Application Settings.

Helpful Links

_{Cover image: Microsoft Stock Photo (modified) Sept 21, 2021.}

2021 AODA WCAG Content Compliance in Ontario

Sun, 08 Aug 2021 22:00:00 GMT

Public web content in Ontario must meet Web Content Accessibility Guidelines (WCAG) 2.0 AA as of January 1, 2021. This shouldn’t be a shock as the Accessibility for Ontarians with Disabilities Act (AODA) has been around since 2005 and is intended to “reduce and remove barriers for people with disabilities so that Ontario can become more accessible and inclusive for everyone”. Sounds great, no arguments here.

Although how are organizations in Ontario doing 16 years later with new compliance rules in effect? And what does AODA/WCAG really mean when it comes to accessibility? Let’s take a quick look into the world of AODA and content accessibility.

What is AODA and public content compliance

With the January 1, 2021 AODA compliance date now behind us, businesses and non-profits in Ontario with 50 or more employees had make their public web content accessible based on Web Content Accessibility Guidelines (WCAG) 2.0 AA (with a few exceptions). On top of that, companies are required to submit a compliance report to the Government on June 30, 2021 (extended deadline). Designated public sector organizations, including municipalities and other identified organizations must file by December 31, 2021 (and every two years).

Current AODA Web content exceptions:

WCAG Success criteria 1.2.4 (live captions)
WCAG Success criteria 1.2.5 (audio descriptions)
Web content published before Jan 1, 2012

The approach for accessible web content seems simple enough: verify your public web content (web pages and related public documents) meet WCAG 2.0 AA and file a report every few years to let the government know you are compliant. Easy peasy. There is also some incentive, oops, I mean possible penalties in the form of fines. Up to $50,000 for an individual, or $100,000 for a Corporation… per day! Oh, and potential fines up to $50,000 per day to directors or officers of a corporation. Incentive. However, the government isn’t driven by potential revenue (they can just raise taxes) so expect much smaller fines or even just some warnings and proof of a strategy to fix issues over the next few years. This makes sense as the intent is long term accessibility, not fines.

The Status of AODA web content in 2021

So how are we doing? Are most websites and public documents WCAG 2.0 AA compliant? In short, it doesn’t look very good. Almost all sites I visit have an abundance of errors - although some are more obscure than others. However, many organizations are most likely submitting their compliance reports stating their content is sufficient. But this isn’t as nefarious as it seems. Companies are likely attempting to be truthful, but they just aren’t aware of the non-compliance. And at what point do you say a website isn’t compliant? If 2 web pages aren’t compliant out of 500? 20 pages? What if the website is compliant but not the related documents? Is attempting compliance the same as being compliant? (answers: Unknow, maybe, maybe, not compliant, no/maybe)

2020 Accessibility Compliance Report Question #9: Other than the requirements cited in the above questions, is your organization complying with all other applicable requirements in effect under the Information and Communications Standards?

Issues exist everywhere. Case and point: The Ontario Government web page where you download the 2020 Compliance Report doesn’t pass an accessibility checker. That’s right, the “steps to open and complete form” list on the page, isn’t a proper list and doesn’t have proper list items. Technically, not passing WCAG 2.0 AA. As we will discuss below, this isn’t necessarily a complete fail or reflection of the bigger site. To be honest, the Ontario.ca site is one of the better sites for meeting WCAG 2.0 AA (especially considering the size and volume of tables/charts). Let’s say this specific example is more of a typo, and likely be be corrected when the next form is posted.

Wondering how other companies could think non-compliant websites are compliant? Well, a big part of this is based on there not being a fast, easy, automated way to check. But wait, what about all those “accessibility checkers”? Those are a great start but they can’t check everything. An automated check can tell you if the page is non-compliant, but it can’t tell you if a page is compliant. How can an automated check determine if a line of text is supposed to be a paragraph, a heading, or in a list? Is that cell in a table supposed to be a header or data cell? Sure, maybe one day the checkers will use AI to fill this gap, but we aren’t there yet. For now, final review of content accessibility continues to be manual process. This is true for both websites and documents (docx, pdf, etc). Side note: Manual review isn’t complicated, nor does it take long - but it’s still manual and you need to know what to look for.

Accessibility Checkers

There are lots of free and extremely useful accessibility checkers out there. As well, Microsoft Word continues to improve its built in Accessibility tools, along with Adobe Pro for PDFs. Please use these tools as they are a great starting point.

Great/free accessibility checkers:

There are also some amazing subscription/cost based checkers that have additional features, automated scheduling, and can crawl the entire site. But regardless of the checker used, we can’t stop there. A manual review is still needed. Remember, the automated checker can’t tell us if a page or document is fully WCAG 2.0 AA compliant. The Adobe Pro Accessibility checker can’t tell us if our PDF is fully accessibile either (it actually tells you a manual check is required for specific items like Reading Order).

Accessibility typos

If you are striving for accessibility and staff have the proper training to catch and fix issues, you are well on your way to success. We can’t expect accessibility perfection (sadly). There will always be spelling error (typos) that sneak by, and there will most likely be accessibility typos too. Since part of WCAG requires a manual check, the chance for human error/omission remains. Personally, I don’t think finding a single small accessibility typo means a website fails (Ontario.ca). Although if an issue is built into your website framework, it could mean that every single page technically isn’t compliant.

The goal of a good accessible website is to start with a good and accessible framework/template. This framework will set default fonts, sizes, proper heading styles, colours, and other features that give the opportunity to be WCAG 2.0 AA compliant. Next we need good content that uses the the appropriate styles and features to meet WCAG 2.0 AA. This really means that creating accessible content is everyones responsibility. If we are involved in the technology or the content (website, document), we should keep accessibility in mind.

Side-bar: The Co-operators Group

A shout-out to the Co-operators group for being open about their content accessibility struggles. In January 2021, the Co-operators group published a web page on PDF Requirements & Guides that acknowledged they were non-compliant with their public PDFs by the Jan 2021 deadline. They go on to explain their goal to fix this issue by setting project with target date (March 31, 2021) along with training and remediation plans. They would also send bi-weekly email updates on the status to various teams on the project.

“We are working towards a target date of completion of March 31, 2021. A few key things about this – if we are audited by the Ministry, we want to have a date to provide them as part of our overall action plan.” ~ Source: Co-operators (website quote)

As this Co-operators web page is focused on PDF Accessibility Requirements & Guides, there is also a wealth of information on making WCAG 2.0 AA compliant PDF. This is an excellent resource overall. Please note there are some minor guidance issues (section on PDF reading order). That said, this is still a great resources to add to your collection.

Accessibility can be sexy and good for business

Thankfully we don’t need to choose between visually appealing or accessible content. We can, and should, strive for both with public content (web, documents). Matter of fact, why limit this to public content?! If we update our approach to all organizational content, even better. This would add to awareness, everyone would have more experience making accessible content, and makes our organizations more inclusive to draw the best talent.

Why wouldn’t you want to promote your organization to the 61 million adults in the US that live with a disability? With 26% of adults in the US having some type of disability (CDC), an accessible website can lead to more business opportunities.

There is no sacrifice here. If we think of our visual and accessibity outcomes while we develop a document or website, both can be acheived with little extra effort. If we ignore accessibility until the end, the remediation effort can be significant. There are some really great examples of visually appealing websites and documents that are also WCAG 2.0 AA compliant (and Section 508 for that matter).

Lets keep moving forward with accessibility

I recognize that not every web page or app can be fully accessible. Having alternate formats or willingness to help the public are part of an accessibility plan too. Since this site discusses maps and apps in many cases, I realize this is one of the bigger gap areas for accessibility. But even if our maps can’t be fully accessible, keep moving forward and add as many accessible features as you can. When creating a map, use colours and shapes to your advantage. If you are working on developing a new app, add accessibility requirements to your success criteria (remember it’s easier to include accessibility upfront). If leveraging an Esri Story Map, use one with improved accessibility features already setup for you.

And have fun with it. Use your creativity to find new and innovative ways to include accessibility features in your public content. Contribute code to public repositories that have accessibility baked in. Share your findings and WCAG 2.0 tips with others. Hmmm, that gives me some ideas for future articles too.

Helpful Links

_{Cover image: Daniel Ali on Unsplash June 07, 2021.}

ArcMap to Retire

Fri, 09 Jul 2021 22:00:00 GMT

After 2 decades, ArcMap is set to retire. Sure there are still six more years of support, but ArcGIS 10.8.1 is the final release for good old ArcMap (ArcGIS Desktop). Esri is saying it for realzies this time. Sure, it’s been said for years: I remember when 10.3 was going the last release until it wasn’t. But the recent announcement on ArcGIS Enterprise removing ArcMap publishing support is the nail in the coffin. This is it, and this time they mean it.

Why Retire ArcMap Right Now

Well, you need to wait until a larger percentage of people have moved to ArcGIS Pro before you flip the switch. ArcGIS Pro was officially launched in 2015 by uptake was very slow. If you move too to retire ArcMap, other options (competitors) might bubble to the surface. Add a dash of integration with ArcGIS Online, and the ability to save database connections as favourites in Pro (finally), and you have reached your destination. We all knew it was happening, we just didn’t have dates until this past year.

Don’t Leave Me ArcMap

In most cases, ArcMap just worked. After a long run, thousands of issues have been worked out in the software. But the 32-bit ArcMap with Python 2x was showing its age. We live in a 64-bit world with Python 3x now. Our datasets are larger, integration with the web becoming more dominant, and we’ve run out of new names for ArcGIS Desktop. Still, can’t we go a little longer?

An ArcGIS Desktop by any other name would smell as sweet:

ArcGIS
ArcGIS Desktop
ArcGIS for Desktop
Desktop for ArcGIS
ArcMap
ArcDeskMap-View-Not-Pro

There could be some hiccups if you are migrating from ArcMap to ArcGIS Pro but there is lots of documentation to help get you through it and explain that you need to make some changes along the way. Concurrent licenses can work in Pro, but Esri would prefer the user subscription approach which is pretty handy if you use AGOL too; If you use terminal servers to access ArcMap, be prepared to crush your servers with Pro unless you start adding some serious GPU power; Let’s not forget to re-training users too, this will be a mixed bag of emotions unless you hire fresh graduates who never used ArcMap (or ArcInfo Workstation, or Idrisi32, or ArcView 3). I’m officially old.

Oh wait, they can’t get rid of ArcMap since I need to publish directly to ArcGIS Server. Well, that feature is finally included in ArcGIS Pro so this is no longer an admin/publishing issue thankfully. It was possible starting at Pro 2.3 (with ArcPy/tools), and now with GUI integration introduced in Pro 2.4. Now that it’s finally in the GUI, Esri will retire the ability to publish from ArcMap to Server in future ArcGIS Server releases. Seems a little harsh but your Admins wouldn’t migrate otherwise (and you know it).

Many of the other limitations to ArcGIS Pro have been resolved through software updates over the years. Remember when ArcGIS Pro couldn’t store Database connections outside a Project? Thankfully those days are over. And Pro has some real benefits since 64-bit - it can use a crap ton of RAM and native 64-bit processing tools with Python 3x support (yes, like all other GIS software). ArcMap couldn’t leverage extra RAM even if installed after long justifications with the IT department to have a powerful workstation. Let’s not forget one of Pro’s best features: you can auto-update the software (not to be underestimated and simplifies administration).

Pro Du Jour

Regardless of your thoughts on ArcGIS Pro, it is powerful and is the current Arc Du jour. Moving from ArcMap has a learning curve, but it’s surprising how quickly you get used to the Pro approach. “apPROach”?! Sorry. I still use both but I gravitate to ArcGIS Pro for my mapping these days. What will I miss the most? ArcCatalog. I just liked having that administrative tool that was separate from the mapping/analysis tools. Akin to Microsoft’s SSMS, or TOAD for MS/Oracle DB management. The dockable window just isn’t the same IMHO. I’ll get over it one day. Maybe a TOAD extension?

With today’s computers all being 64-bit, we were leaving processing power untapped with ArcMap (well, there are a bunch of 64-bit background GP tools but you know what I mean). Python 2x was also sunset in January 2020 and is no longer support or receiving bug fixes from the Python Software Foundation. Yeah, this is legacy software now. But still usable. I’m not crying ArcCatalog, you are.

In short, you have lots of time to use ArcMap still. However, if you keep your ArcGIS Server up-to-date, you are going to need to hold back at some point or you will no longer be able to publish from ArcMap/Catalog. This also gives you some time to figure out what you are going to do with your ArcMap Python Add-Ins, since… yup, those don’t exist in Pro. To the SDK we go!

Well, time to go manually patch ArcMap now as I’m not ready to move on just yet.

Helpful Links

_{Cover image: Austin Chan on Unsplash June 07, 2021.}

JavaScript map libraries with WMS Radar data

Sat, 08 May 2021 22:00:00 GMT

In a recent post, we discussed the Canadian Government’s open platform to share Meteorological data, MSC GeoMet. MSC GeoMet provides public access to the Meteorological Service of Canada (MSC) and Environment and Climate Change Canada (ECCC) data via interoperable web services and application programming interfaces (API).

Web Map Services

GeoMet provides direct access to meteorological data via Web Map Services. If you aren’t familiar with Web Map Services (WMS), it is a standard protocol for serving georeferenced maps over the Internet. These services are easily consumed in many web map libraries to build your mapping websites and applications. In our examples, we will use the GeoMet service and the main map request type, GetMap, to talk to the WMS server. Some of the JavaScript libraries also leverage GetCapabilities requests, but since we already know what layers we want to use, we don’t need to specify both requests. Both of these Get requests are required by WMS servers:

GetCapabilities – returns parameters about the WMS (such as map image format and WMS version compatibility) and the available layers (map bounding box, coordinate reference systems, URI of the data and whether the layer is mostly opaque or not)
GetMap – returns a map image. Parameters include: width and height of the map, coordinate reference system, rendering style, image format

~Wikipedia.org

When a map is published as a WMS, GetCapabilities is typically used to discover what is available in the service. The layers, symbology, time intervals, and metadata can be searched. Once you know what you want, you can request a map image using the GetMap request with some parameters to describe what you are interested in. Most of the GetMap parameters are handled by the JavaScript Library and are automatically configured. When panning and zooming on the map, the library will update the coordinate parameters and send a new request to update the map.

Building a Web Map

The goal is simple - Add the WMS data to many different web mapping API/frameworks. I hope these examples are easy to understand, as the steps taken will work with most WMS endpoints. We could get more advanced and query the time slices available in the radar datasets, but let’s start with a basic Hello Map for some of the more popular javascript mapping libraries: Leaflet, ArcGIS JavaScript API, and OpenLayers.

Keeping the examples simple and with similar requirements:

Map: Basic full-screen map
- Set zoom and map center
Map Options: Map library defaults
Basemap: Open Street Map
Layers: WMS service from GeoMet with 1 Radar layer
- Layer Name: RADAR_1KM_RRAI
- Add attribution
- Specify WMS version 1.3.0

Although we are using many JavaScirpt libraries, this isn’t meant to be a comparison/competition between them. Way too much to unpack there, and not the purpose of this article. In some ways, this article shows how it doesn’t matter which library you use - WMS data can be leveraged by all of these libraries. WMS is handled a bit differently, but these differences are minor - many bigger application factors will determine the library that works for you.

Leaflet

Leaflet is a very popular open-source JavaScript library for building interactive maps. It has a very simple and effective WMS implementation. It doesn’t natively support the GetCapabilities request - but that isn’t required for our examples. If you know what you want (map layers from the service), just set it up, and Leaflet will take care of the rest.

// Add a wms service and specify the layer(s) of interest
var wmsLayer = L.tileLayer.wms('https://geo.weather.gc.ca/geomet?', {
    layers: 'RADAR_1KM_RRAI'
}).addTo(map);

Beyond the basic layers parameter, we can specify a few additional parameters to handle transparency, preferred image format, and attribution. If you investigate the GetCapabilities request manually, you can find the parameters and values that are available for this service.

// Add the GeoMet WMS (Radar layer) with some additional parameters
var wmsLayer = L.tileLayer.wms('https://geo.weather.gc.ca/geomet?', {
    layers: 'RADAR_1KM_RRAI', //missing comma
    version: '1.3.0',
    transparent: true,
    format: 'image/png',
    attribution: '<a href="https://eccc-msc.github.io/open-data/licence/readme_en/">ECCC</a>'
}).addTo(map);

Now, all we need to do is create the basic HTML page, the Leaflet JavaScript library source code and our WMS code. Based on our general requirements, we will add a basemap.

ArcGIS JavaScript API (Multiple versions)

Esri’s ArcGIS API for JavaScript has been around for a long time. Version 3x is now considered Esri’s ”legacy” JavaScript API with no new feature development underway. The good news, it is a very mature library with some features still not available in the 4x replacement. ArcGIS JavaScript API 4x is at version 4.19 at the time of writing and has almost reached parallel functionality - and moved beyond 3x in many areas (2D/3D views). Both versions use an Asynchronous Module Definition (AMD) format - this was to ”make code easier to author and debug” (DojoToolkit) but it does come with its learning curve. This adds some additional lines of code for simple apps - with the benefit of a large functionality matrix and feature-rich modules for bigger applications that go beyond mapping and deeper into the world of GIS.

ArcGIS JavaScript API 3x

Unlike Leafet, the ArcGIS JavaScript API 3x default behaviour is to execute a GetCapabilities request when a WMS Layer is added to the map. Unfortunately, this might lead to some CORS issues - although you can use a simple proxy if you get stuck. To bypass the automatic GetCapabilities request (and possible CORS issue), we can create a resourceInfo object and include it with the WMSLayer. Much like Leaflet, by including the resourceInfo data, a getMap request is made directly.

// resourceInfo object to describe the WMS layers (and bypass the GetCapabilities request)
var resourceInfo = {
    extent: new Extent(-126.40869140625, 31.025390625, -109.66552734375, 41.5283203125, {
        wkid: 3857
    }),
    description: "Radar",
    layerInfos: [layer1],
    layers: "RADAR\_1KM\_RRAI",
    version: "1.3.0"
};
// Now we can add the WMS layer details with our resourceInfo object
//   in the wmsLayer constructor
var wmsLayer = new WMSLayer('https://geo.weather.gc.ca/geomet/?lang=en&service=WMS', {
    resourceInfo: resourceInfo,
    copyright: 'ECCC',
    visibleLayers: ['RADAR_1KM_RRAI']
});
map.addLayers([wmsLayer]);

Same with the last example, we build the remaining HTML page and add our basemap.

ArcGIS JavaScript API 4x

Similar to 3x, the ArcGIS JavaScript API 4x default behaviour is to execute a GetCapabilities request when a WMS Layer is added to the map. Missing in the new version is a way to work around this. So currently there is no resourceInfo object and we can’t bypass the getCapabilities request.

“The WMSLayer is used to create layers based on OGC Web Map Services (WMS). The WMSLayer initially executes a WMS GetCapabilities request, which might require CORS or a proxy page.” ~ Esri WMSLayer reference

// Create the WMS layer and specify sublayers
var wmsLayer = new WMSLayer('https://geo.weather.gc.ca/geomet/?lang=en&service=WMS', {
    sublayers: [
        {
            name: "RADAR_1KM_RRAI",
            title: "RRAI Radar",
            description: "Radar"
        }
    ],
    copyright: '<a href="https://eccc-msc.github.io/open-data/licence/readme_en/">ECCC</a>',
    description: "radarWMS",
    title: "Radar PN",
    version: "1.3.0"
});

OpenLayers

The OpenLayers map library continues to be a free, Open Source JavaScript option that doesn’t get much press but continues to have a strong user base. For some reason, it gets compared to Leaflet and mislabelled as “more complicated”. True in some ways, but it really will depend on what you are trying to accomplish. OpenLayers has more in common with the ArcGIS APIs: It crosses over the web-map to web-GIS threshold in similar ways: Feature-rich and able to do more complicated tasks with ease.

Back to OpenLayers… to add WMS layers we use the ImageWMS image source to connect to the GeoMet service. In the config, we add a params object with the list of layers (this is mandatory). The version defaults to 1.3, but added her to posterity, along with our attribution.

// Add the Image layer with WMS source
new ol.layer.Image({
    source: new ol.source.ImageWMS({
        url: 'https://geo.weather.gc.ca/geomet/',
        params: {
            LAYERS: 'RADAR_1KM_RRAI',
            version: '1.3.0'
        },
        attributions: [
            'Radar <a href="https://eccc-msc.github.io/open-data/licence/readme_en/">ECCC</a>', ol.source.OSM.ATTRIBUTION
        ]
    })
});

Map Libraries and WMS

Regardless of the mapping library, the GeoMet service has a vast amount of data available to enrich your app. Don’t go by the amount of code, as this can be misleading. Easy things might take more lines of code, but complex items might take less, or not available in all libraries. Also not saying my approach is crafted to perfection.

One thing to note, since the GeoMet WMS has many layers, the service is not small. The impact of running a GetCapabilities request is noticeable and may impact map load performance. If you can avoid this extra request, do so. Other WMS servers may also have CORS issues with a proxy required for GetCapabilities. Two good reasons if you can avoid it and you know the service isn’t going to change.

The GeoMet WMS works great in all libraries, which is a real benefit of WMS and its open/standard protocol. WMS also has time-aware layer information if you want to take your radar application to the next level - play the last 3 hours of radar data in a simulation, or specify from one of a few symbology settings, etc.

Helpful Links

_{Cover image: CC-BY Dirk Breuel (Flickr) May 07, 2021. Modified from original.}

ArcGIS JavaScript API Key Please

Sun, 25 Apr 2021 22:00:00 GMT

It’s not just technology and underwear that change frequently, software requirements and Terms of Service (TOS) can change quickly too. Even if you don’t do anything, external factors can turn your app into 404 without warning. This can happen for many reasons, including when changes come to the APIs we use. Don’t worry, you are just an API Key away from success in some cases. What API Key? Do I need one? Let’s dive in.

What API Key?

The ArcGIS JavaScript API has been around for a long time. Since the beginning, there has always been free (not Open) access to the JavaScript API, and additional developer subscription levels for the hard-core. With Esri, API Keys were focused on the subscriptions, while smaller and non-revenue generating sites weren’t impacted. This is still partially true, but things are continuing to evolve.

I was building an ArcGIS JavaScript API 4x sample to demonstrate Canadian radar data with Esri. Suddenly, the demo stopped working. I opened the chrome dev console to look for clues. The code syntax looked fine (no missing semicolons, etc). Everything worked 30 seconds ago and I only changed the basemap. It wasn’t until I jumped into the Chrome developer tools’ network window that I noticed something was blocking the Esri JavaScript API. Oh crap, a ”No API Key” error. Wait, what?! Was I no longer able to build my app because I changed the basemap? I tried removing the basemap, but it was too late, the cache had me. I could clear my cache or go and get a developer API Key, but I wanted to know why. Some searching led me to an old/familiar location - the “Getting Started” section of the ArcGIS JavaScript documentation (haven’t been to that page in a while). Looks like API Keys are becoming the norm now with some new requirements for some services (and an updated terms of service).

A Switcheroo?

I’ll admit, I don’t re-read Getting Started pages or TOS documents often. But I was pretty sure just a few years ago an API Key wasn’t a requirement for ArcGIS JavaScript 4x maps (unless you were a developer generating revenue).

As recently as 2018, Esri was promoting that Google developers migrate to the ArcGIS JavaScript API (4x) - One of the selling features mentioned: “You won’t need an API key; you can simply reference the API”. Yeah, that part. That’s only 14 months ago if we don’t include COVID months (since COVID months should never count). Fine, 2.5 years for the purists. To be clear - you don’t need a key to access the JavaScript API, just the services.

Source: Esri Blog Announcement Aug 2018 Migrating from Google Maps JavaScript API to ArcGIS API for JavaScript

Thankfully, the online documentation is now littered with reminders for API Keys if you are looking. The Map class basemap section has a list titled “Basemaps for use with API keys”. That explains the issue from earlier. Another example, the Getting Started section plays it more direct - it mentions these 3 things to get you started:

Setup the API
Get an API key
Start the tutorial

Not a sneaky switcheroo, another change to technology over time. Although, if you migrated from Google based on the 2018 marketing, I could understand if you are a little pissed. One silver lining is that the 2018 article mentioned 1 million free map transactions per month with a developer subscription. Today, that number has increased to 2 million. You can still be pissed, but take the 2 million map requests.

Things Change

This shouldn’t be a surprise as companies review their approaches, TOS, and bank accounts all the time. Providing API Keys for access to enhanced services is the norm, with free services still available in most cases. The ArcGIS JavaScript API integration with Portal Maps, specialized basemaps, enhanced search services (routing and demographics) were all signs it was moving more this way. Yup, Esri recently announced the ArcGIS Platform (PaaS) for developers. Use one of many APIs to access Esri enriched services to build your apps - which is where the API keys come in.

You can work directly with our location services using open-source mapping libraries like Leaflet, GL JS, OpenLayers, and ArcGIS REST JS. _{~Esri ArcGIS Platform (PaaS) blog}

This doesn’t mean it will take your beer money - there is still a free tier with a developer API key and… oh wait, it’s the same as Goog… Yes, a free tier for most services and some premium services are also available. And you don’t need an API key for everything (yet). My demo is back and running without a key and it gets the job done - it’s just a demo and will cap out to less than 2 million monthly requests even if I needed a key. Not that I make any beer money from it anyway.

So, you need an API Key for Google, Esri, and many other services. Not a big deal. If you want to avoid API Keys, you do have other options - but remember, API Keys aren’t evil. OpenStreetMap and the popular Leaflet API both have lots of options. And yes, Esri has an extension to Leaflet to integrate with Esri. However, if you want to use premium Esri services (basemaps, routing, etc), you will still require an Esri API Key (I looked it up, it’s in the docs). On the flip side, you can use whatever API you like and still access premium services with the key. Glass half full.

Esri Developer Terms of Service

Since I was digging into things, I also reviewed the current Terms of Service. There are a few items related to developers creating value-added applications (revenue-generating) that might be of interest to a select few.

Value-added applications built with:

The current generation client APIs (i.e. JavaScript 4.x and Runtime SDKs 100.x) is required to use API Keys in their applications by Aug 1, 2021.

The older generation client APIs (i.e. JavaScript 3.x and Runtime SDKs 10.x) is required to use API Keys in their applications by January 1, 2023.

Source: Developer Products footnote #91 of the Esri Terms of Service e300. The TOS has more details if interested. Also check out footnotes #16 and #64 for JavaScript Web API specific items.

What’s the Point?

The point is, web development is always changing, and so are the requirements to access services. Keep an eye on the APIs and services you use and the changes that are happening. Old applications can be impacted without warning, so don’t forget to check the documentation and requirements before you start pulling your application apart. API Keys are just one of many types of changes. Breaking changes can also be introduced with new API versions (ArcGIS 3x to 4x for example). Heck, my Wyze camera stopped its in-camera person detection one day - Why? The company that made that piece of software (Xnor.ai) was acquired by Apple, so Wyze had to change its terms and no more onboard AI software.

In short, RTFM once a year for all the APIs and web services you use just to make sure everything will keep ticking. As well, a big tech company might buy something and screw your recent home camera purchase.

Helpful Links

_{Cover image: Silas Köhler (Unsplash) April 17, 2021.}

Canadian Radar Online Access

Thu, 15 Apr 2021 22:00:00 GMT

When browsing the Canadian weather website the other day I was surprised to read the bold notice “New Weather Radar Map”. I then scrolled down to find a simple, clean, easy-to-use Weather Radar map. Thank you ECCC/MSC (AKA Canadian NOAA)!

New Weather Radar Map

This map is a huge improvement over the old GIFs. Now we have a single mosaic image, GIS data in a WMS API, an actual ‘map’, and accessible navigation too. The framework appears to use Vue with the OpenLayers API in a module called VueLayers. VueLayers is best described by their tagline “Web map Vue components with the power of OpenLayers”. Free to use by the way.

The map has a few options as well: rain/snow modes depending on temperature (rain and snow reflect differently), two animation timeframes, and two symbology options. From GIFs to web maps - that’s like french fries to poutine!

source: weather.gc.ca

Now that we have a real map, could that mean… a real API to weather data that is Open and available to developers? Sure does. Welcome to MSC GeoMet.

MSC GeoMet Online Access

MSC GeoMet has been around for a while now but has stayed under the radar (sorry). So what is MSC GeoMet? A platform that provides public access to the Meteorological Service of Canada (MSC) and Environment and Climate Change Canada (ECCC) data via interoperable web services and application programming interfaces (API). In short, open weather/climate APIs. Precipitation is just one of the hundred or so layers available at this point. Some even mosaic NOAA data for a seamless dataset. Weather needs no passport.

So now that the government radar site is using live Web Mapping Services (WMS), shouldn’t we all start mashing it into our apps?! Maybe we can all stop screen scrapping the GIFs?! Although, when it comes to consistency, those GIF scripts have worked without issue for a decade. Respect.

What’s Next

Hoping the next step for MSC GeoMet is to find a better way to search available data. Leveraging the OpenGov site and combing through the layers in the services is a good start, but hoping for a possible discovery/search tool. This would help more beginners navigate and find some interesting data and lower the bar to adoption. But right now I can’t complain - well done ECCC/MSC, and keep going. There are even a few examples of how to build a simple web map with their data. A few additional examples would be helpful as well, possibly for the layers that are harder to understand - such as Forecasting and modelling data.

I guess the best way to help would be to create some examples as well. I’ll start putting together a follow-up article where we can start leveraging these services with a bunch of web mapping APIs: Leaflet, ArcGIS JavaScript API v3x and v4x, and OpenLayers for good measure.

Helpful Links

Append using ArcPy cursors

Sun, 03 Jan 2021 22:00:00 GMT

If we are using Python with ArcGIS, there are existing tools to append data. However, calling a Geoprocessing tool from Python doesn’t feel very Pythonic. It also doesn’t give us access to the data before loading. Instead of calling the Append Tool, we can use cursors to append data and open the door to other data manipulation options. More specifically, we will use a SearchCursor on source input data, and an InsertCursor on a Target schema to show how we can replicate or enhance our Python script. Both the Append Tool and the Cursor approach have their pros and cons - just nice to have some options.

A Cursory Overview

Cursors are an extremely fast way to read/write features while giving you lots of control over the features, attributes, and filters. Cursors can include a SQL clause or deconstruct features into their individual points or vertices.

A cursor is a data access object that can be used either to iterate through the set of rows in a table or to insert new rows into a table. In our case, we are going to read and write features. If using ArcMap, there are two different classes of cursors - old and new. No need to discuss the old ones, just disregard them and just use the new arcpy.da.cursor classes.

SearchCursor

First, we need to read the input features from a Feature Class that we want to append somewhere. Since we just need to read these features we can use a SearchCursor. This SearchCursor requires a minimum of two parameters, the input features (Feature Class, Layer, Table, Table View), and a list of field names. Use an asterisk (*) instead of a list of fields if you want to access all fields from the input table. There are some limitations with the asterisk which we will get into later.

# Setup a search cursor on our new data and iterate the rows
with arcpy.da.SearchCursor("C:/input.gdb/newPoints","*") as cursor:
    for row in cursor:
        #Do stuff with each row

Once we have access to every row of input data, we can insert the records into another layer, or even manipulate the attributes before loading. The issue with the wildcard attribute approach (asterisk) is knowing how to access the correct attribute of interest. If the attributes are specified, the SearchCursor will maintain the attribute order as listed so we know how to access them.

Append data with the InsertCursor

The great thing about the Append Tool is the ability to auto-match fields with the schema_type parameter. You can either specify the matching fields or if the field names/types are the same, allow the tool to take care of it for you. When working with SearchCursors and InsertCursors, we don’t have the same luxury. If you know your input and target layers have the same schema and field order you can use the wildcard approach. If not, you will need to define the fields.

Let’s assume our schemas are identical for now. How would we load into our target layer? Let’s add the InsertCursor to our working sample. We will set up our new InsertCursor at the beginning, and with one more line of code inside the SearchCursor loop, we can add the data to our target dataset.

# Create an InsertCursor on our Target dataset
targetCursor = arcpy.da.InsertCursor("C:/out.gdb/Points","*")
# Setup a search cursor on our new data and interate the rows
with arcpy.da.SearchCursor("C:/input.gdb/newPoints","*") as cursor:
    for row in cursor:
        #Insert into the target dataset (only Points or tabular data if using an asterisk)
        targetCursor.insertRow(row)

If we are loading point data or table data, this will work without issue if the fields list and field order match. However, there is a small issue with the default SHAPE field when updating multi-point, line, or polygon data.

A SHAPE issue?

This is technically documented, but it can be confusing why our previous sample would work on tables and point layers, then suddenly fail (silently) on a line or polygon layer. It will finish running with other attributes populated, but previewing the map will not return any spatial data.

Why does this happen? Performance defaults. Accessing geometry can be an expensive operation, so the default Geometry token is set to SHAPE (SHAPE@XY) to save precious time. This means if we want to load point data, the performance will be much better than for other geometry types. Yes, even faster than the Append Tool itself.

When using * (field list), geometry values will be returned in a tuple of the x,y-coordinates (equivalent to the SHAPE@XY

token) ~Esri

Again, this default is fine if inserting points with an InsertCursors (and non-existent for tables). But polygon, polyline, or multipoint features can only be created using the SHAPE@ token which will cost us some performance and extra lines of code.

SHAPE it up

We can deal with this SHAPE issue in many ways. The overall goal is to replace the SHAPE (SHAPE@XY) token reference with the full geometry SHAPE@ token. We can grab the attributes with arcpy.Describe or the arcpy.ListFields function. ListFields is faster, while Describe provides multiple properties, such as data type, fields, indexes, and many others. Once we have the fields, we pass into the cursors. Again, still assuming the fields are the same. If there is a change the order is different, just sort the list.

inFC = "C:/input.gdb/newRoads"
#Collect all fields except the Geometry field
lstFields = [field.name for field in arcpy.ListFields(inFC) if field.type not in ['Geometry']]
lstFields.append("SHAPE@") # add the full Geometry object
# Similar to previous with fields specified
targetCursor = arcpy.da.InsertCursor("C:/out.gdb/Roads",lstFields)
with arcpy.da.SearchCursor(inFC,lstFields) as cursor:
    for row in cursor:
        targetCursor.insertRow(row)

Next Steps

We now have some raw code to append data using ArcPy cursors. From here we could start refining the code:

Check the datatype in advance to see if we need to change the SHAPE geometry
Verify the fields match b/w the inputs and target
Append multiple datasets
Manipulate attributes before loading
Optionally remove other read-only fields if you aren’t using them (ObjectID, shape.length, shape.area)
- These are ignored if you try to update anyway

Example: Append multiple datasets

Here is one additional example where we add multiple datasets to the append list. This example still assumes all the layers have an identical schema. Since we are loading lines, we will update the Geometry object.

import arcpy
lstFCs = ["C:/input.gdb/Road_North", "C:/input.gdb/Road_South",
     "C:/input.gdb/Road_East", "C:/input.gdb/Road_West"]
#List all the fields and add the full Geometry object
lstFields = [field.name for field in arcpy.ListFields(inFC) if field.type not in ['Geometry']]
lstFields.append("SHAPE@")
# Set the target and Loop all input layers to insert
targetCursor = arcpy.da.InsertCursor("C:/out.gdb/Roads",lstFields)
for inFC in lstFCs
    with arcpy.da.SearchCursor(inFC,lstFields) as cursor:
        for row in cursor:
            targetCursor.insertRow(row)
del targetCursor

Helpful Links

_{Cover image: CC-BY Yohanes Sanjaya (Flickr) Jan 03, 2021. Modified from original.}

Using multiple versions of Python with ArcGIS

Wed, 14 Oct 2020 22:00:00 GMT

As Python continues its dominance as the defacto scripting language for GIS software, we hit a point when one Python environment isn’t enough. Even if we try to keep a clean system, ArcGIS applications need to install their version(s) of Python. ArcGIS Desktop has two flavours alone, and a third if we include ArcGIS Pro (which can have many itself). How do Desktop users open a .py file in the desired version of Python? Can we make this a little easier? If you don’t mind updating the Windows Registry, some little tweaks can make a big difference and save you some hassle.

To preface, this article is for “GIS Desktop power users” with multiple ArcGIS Python versions that want to create/run scripts on their local machine. If installing Python independently, this article isn’t for you. Great package managers like Conda (Anaconda/Miniconda) exist for these purposes. Conda makes managing multiple environments a breeze - and cross-platform too. If you are a more dedicated ArcGIS developer and using the ArcGIS API for Python, you might still want to consider Conda. If you just need ArcGIS Desktop and/or ArcGIS Pro and want to open/edit a specific version of Python to run some code, keep reading.

ArcGIS and Python

ArcGIS for Desktop software (ArcMap and ArcCatalog) are tied to Python v2.x. The initial install of Desktop will install the 32-bit version since ArcGIS Desktop is… 32-bit. Desktop Background Geoprocessing installs the 64-bit version of Python. Both 32/64-bit will run 99% of v2.x scripts without issue. There are a few Esri tools that fail if run in the wrong version which is why I started down this path in the first place.

ArcGIS Pro on the other hand is based on Python 3.x 64-bit. If only using Pro, you shouldn’t have any issues. In this case, you can even manage additional Python environments directly in ArcGIS Pro! Yup, ArcGIS Pro has a Python Package Manager built-in (err, you guess it - Conda is integrated into Pro). Side note: GDAL is also installed with Pro - a topic for another time.

Selecting the right IDLE/IDE

At this point, we have three Python environments installed. Setting a default can be done as an environment variable but switching between them isn’t as simple. If wanting to select a specific environment we can take advantage of a small registry update. For this example, we will use the installed IDLE application but the same approach can be used for the editor/IDE of your choice.

Registry Disclaimer

Don’t screw up the registry. You’ve been warned. If you don’t understand what you are doing, this isn’t the sandbox to learn in. Try it out on a virtual machine, or ex-friends’ computer first.

Registry CLASSES

To add options to the right-click menu, we just need to update the CLASSES registry and file name extension information. This info is stored in three places:

HKEY_LOCAL_MACHINE (HKLM)
HKEY_CURRENT_USER (HKCU)
HKEY_CLASSES_ROOT

We should never touch HKEY_CLASSES_ROOT directly as it provides a view of the registry that merges the information from these other two sources. HKEY_CLASSES_ROOT also provides this merged view for applications designed for historic versions of Windows.

Where we update depends on who we want this to work for. HKLM if we want these changes to show for all users - or HKCU for the current user only. If you accidentally write values to a key under HKEY_CLASSES_ROOT and the key already exists under HKCU\Software\Classes, the system will store the information there instead of under HKLM\Software\Classes.

Registry changes

First, we should confirm the HKLM\SOFTWARE\Classes\Python.File\shell section exists. It is created when you install Python or ArcGIS for the first time but worth a quick check. From there we will add all options that don’t already exist and/or can tweak the naming of existing. The name of the keys inside will be the name shown in the context menu (r-click). To each key, we add a command sub-key to provide the IDLE environment reference.

The final results will look similar to this:

Here are all three ArcGIS IDLE environment settings for the Registry assuming the default install directories were used:

Python v3 64-bit installed with ArcGIS Pro

[HKLM\SOFTWARE\Classes\Python.File\shell\Edit with IDLE (ArcGIS Pro)]
[HKLM\SOFTWARE\Classes\Python.File\shell\Edit with IDLE (ArcGIS Pro)\command] : @="\"C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\pythonw.exe\" \"C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\Lib\\idlelib\\idle.pyw\" -e \"%1\""

Python v2 32-bit installed with ArcGIS Desktop

[HKLM\SOFTWARE\Classes\Python.File\shell\Edit with IDLE (v2 x32)]
[HKLM\SOFTWARE\Classes\Python.File\shell\Edit with IDLE (v2 x32)\command] : @="\"C:\\Python27\\ArcGIS10.7\\pythonw.exe\" \"C:\\Python27\\ArcGIS10.7\\Lib\\idlelib\\idle.pyw\" -e \"%1\""

Python v2 64-bit installed with ArcGIS Desktop Background GP

[HKLM\SOFTWARE\Classes\Python.File\shell\Edit with IDLE (v2 x64)]
[HKLM\SOFTWARE\Classes\Python.File\shell\Edit with IDLE (v2 x64)\command] : @="\"C:\\Python27\\ArcGISx6410.7\\pythonw.exe\" \"C:\\Python27\\ArcGISx6410.7\\Lib\\idlelib\\idle.pyw\" -e \"%1\""

At this point, we could add more options for additional IDE’s/editors (VS Code, PyCharm, Atom, NPP), or start scripting. Don’t forget to backup your registry before you start just in case.

Helpful Links

_{Cover image: CC-BY-SA Opensource.com (https://opensource.com/article/19/5/python-3-default-mac) Oct 14, 2020.}

ArcGIS Server Print Service Produces a Pixelated Legend

Tue, 18 Feb 2020 20:00:00 GMT

Custom print services allow us to create unique layouts with custom elements such as dynamic legends, scales and automated text labels. Dynamic legends can be a great feature to automatically toggle layers in the output legend area - but sometimes printed legend patches may appear garbled/pixelated/grainy. This is a known issue that may occur but thankfully there are some fixes available. The issue may appear after a server or software upgrade and will happen consistently when requesting the same print size and DPI. There are many possible causes including the map document (MXD) layout, ArcGIS Server configuration, or even much deeper into the server settings.

Troubleshooting

Before jumping into the server settings, make sure the legend issue doesn’t occur in ArcGIS Desktop when printing from the same template. Second, do a quick check of the print service DPI settings in your print service and your application. Low DPI requests can generalize the output graphics, especially if using custom fills or transparencies. Try using a value of at least 96 DPI, up to 600 DPI to see the impacts this setting has on your outputs. Beware, setting DPI too high results in each print request taking more time, so don’t go overboard unless your users are prepared to wait. If DPI isn’t specified, 96 is the default for the ExportWebMap specification. Print services are Synchronous by default but can be set up as Asynchronous if requests are expected to take a while do to size or high DPI. Side note: print services used in the ArcGIS Enterprise portal must be Synchronous.

Using Windows Server and still not working?

On Windows, ArcGIS Server printing services rely on the underlying Windows Graphics Device Interface (GDI) Print API. The GDI Print API provides applications with a device-independent printing interface. ArcGIS Server uses this GDI to render text and graphics returning the results to ArcGIS Server which passes the results back to the user.

When leveraging the Windows OS environment for AGS, Esri notes that “If a legend patch with a picture fill or marker symbol type falls outside the default page size of the Windows default printer, Windows GDI drawing calls used by Print Service may not properly draw the object or image on output.” Cool. In short, the GDI configuration is based on your default printer setup. The most likely culprit is requesting a map output larger than the default printer’s default paper size. Since the print GDI rendering starts in the top-left, legend issues become more common the further they are to the right and lower corner where they can extend beyond the default page.

Possible Windows GDI fixes

Since ArcGIS Server leverages the default printer settings for the GDI, we can update the default settings to fix it. To update, we need to log into the server under the ArcGIS Server Account.

Note: This is the account that was chosen during the ArcGIS Server setup. Windows Task Manager can also be used on the server machine to find out which user is being used to run the ArcSOC.exe process.

Once logged in, review and update the default page size and orientation printing settings. When complete, you will need to restart the ArcGIS Server service as an admin, or reboot. It’s recommended to set the page size to the same or larger than the biggest map layout needed.

If a pixelated legend persists, change the default printer completely. This isn’t a proposed solution in the bug report but it has been the one solution that has worked in the past for me: Changed the default printer and still used the same page settings. To change to a new printer you may need to log in as a local administrator to add the printer to the server first, then it will be available to the ArcGIS Server account (the ArcGIS Server account is a local user account with very limited permissions). Again, you will need to restart the ArcGIS service as an administrator or reboot for changes to take place. Side note: Esri recommends stopping the service first (admin account), change the print defaults (arcgis account), start the service again (admin account).

After the problem is resolved it’s recommended to binge the movie Office Space so you can watch someone take a baseball bat to a printer for a few minutes - very therapeutic.

Touch GIS Provides Free Access with GeoInt

Fri, 31 Jan 2020 16:00:00 GMT

Touch GIS is one of the latest mobile data collection apps to be launched for the field workforce. It has many of the professional field collection and visualization capabilities needed for successful fieldwork including offline workflows, and support for many file-based spatial formats. SpatialTimes hasn’t done a full review of the app yet, but it’s worth mentioning that Touch GIS just announced it is now available at no cost to credentialed U.S. government employees, first responders, and the Department of Defense.

“We’re very excited to offer Touch GIS on the GEOINT App Store” says Joe Wilson, Head of Product. “Touch GIS is useful for a wide variety of field uses, and offers best-in-class data collection, mapping, and visualization tools. The app is especially helpful when working offline. The best part? It runs on the device that’s already in your pocket!”

GEOINT App Store Access

Touch GIS is now available at no cost from the National Geospatial-Intelligence Agency (NGA) for approved users via the GEOINT App Store. The GEOINT App Store provides online on-demand access to GEOINT content and services through mobile apps, web apps, and web services. GEOINT is available to all Department of Defence (all apps), Federal Government and Mission Partners (limited apps).

Public Access, NPO, and Education

If you don’t qualify for the no cost GEOINT version, Touch GIS is still available to the general public from Apple’s App Store. Pricing is reasonable and Touch GIS also offers a free trial and special rates for non-profit groups and educational institutions.

Helpful Links

Python Script to Overwrite Existing Service in ArcGIS Server

Thu, 12 Sep 2019 22:00:00 GMT

ArcGIS Desktop’s ArcPy package gives us the ability to publish a map service from a map document (MXD) inside Python. This can be very helpful for migrations, backups, and maintenance activities such as changing layers/symbology and overwriting the existing service. Wait a sec, there isn’t an option to overwrite when using the GP Tool in Python… or is there.

Publish a map service with Python

Before we overwrite a map service, let’s start with the basics of publishing an MXD to a new service with Python. The code here is based on Esri’s online help sample. This will require an existing ArcGIS Server connection file (.ags) created in ArcCatalog.

import arcpy, os
cwd = os.getcwd()
# Setup main variables
agsCon = cwd + os.sep + "AGS_Publish_Connection.ags"
agsDirectoryName = "ServiceDirectory"
agsServiceName = "ServiceName"
mapDoc = arcpy.mapping.MapDocument(cwd + os.sep + "helloWorld.mxd")

# Setup Service Details
sddraft = cwd + os.sep + agsServiceName + '.sddraft'
sd = cwd + os.sep + agsServiceName + '.sd'
# Create service definition draft
arcpy.mapping.CreateMapSDDraft(mapDoc, sddraft, agsServiceName, 'ARCGIS_SERVER', None, True, agsDirectoryName, None, None)
# Analyze the service definition draft
analysis = arcpy.mapping.AnalyzeForSD(sddraft)

# Stage and upload the service if the sddraft analysis did not contain errors
if sdAnalyze['errors'] == {}:
    # Execute StageService to create the service definition
    arcpy.StageService_server(sddraft, sd)
    # Execute UploadServiceDefinition to upload the service definition and publishes the service
    arcpy.UploadServiceDefinition_server(sd, agsCon)
    print "Service Published"
else: 
    # if the sddraft analysis contained errors, display them
    print analysis['errors']

Overwrite an existing service with Python

Once we have the map service published, a few tweaks can add the ability to overwrite. The trick here is to open the .sddraft file before we stage and upload the service. It’s inside this file where a specific element can be modified to allow for overwriting a service. Thankfully the sddraft file is really just an XML file rebranded with a different extension. A few online community members provide solutions by going the extra mile and use Python XML modules (minidom, ElementTree) to modify the XML data. This is a much more robust approach, especially if you want to modify other values of the XML/sddraft file. You can see an example of the XML module approach in the Helpful Links section at the end if interested.

In our example, we just want to overwrite an existing service so an XML module won’t be needed. We just need to replace the service type Esri constant esriServiceDefinitionType_New value inside the XML with the overwrite constant esriServiceDefinitionType_Replacement. We can leverage Python’s String replace() method as this constant is a unique value.

## Change the SDDraft to overwrite existing service
# Read the file
with open(sddraft, 'r') as file:
    filedata = file.read()
# Replace the target string
filedata = filedata.replace('esriServiceDefinitionType_New', 'esriServiceDefinitionType_Replacement')
# Write the file out again
with open(sddraft, 'w') as file:
    file.write(filedata)

The code above should be inserted into the main script after creating the sddraft file. You could also place it directly before creating the staging service (arcpy.StageService_server) as it is only needed if there are no errors during the analysis review. I should also mention that this isn’t offically supported - so remember to test, test, test.

Tips and Tricks

I was receiving some Python errors when running this script with Python 64-bit. The StageService_server (10.7) not liking the 64-bit version. Both 32-bit and 64-bit could be installed by ArcGIS (32-bit installed with ArcGIS, 64-bit installed with Background Geoprocessing). Everything did work fine when making sure to use Python 32-bit.
As previously mentioned, the sddraft file is really just an XML file. If you rename to XML, you can open in Excel/NPP to checkout some of the other interesting elements. If wanting to modify more values it’s recommended to use an XML module for greater support and functionality.
What about the SD file, is it an XML file too? Nope, the the .sd file is actually just a renamed .7z extension. If you have a copy of 7-zip (Open Source), you can rename to .7z and extract the contents of the SD to see what’s inside this file as well. This compressed file can also include data if it needs to pass it to the server which is why it can get pretty big.

Helpful Links

Parallel Processing in ArcGIS

Sun, 05 May 2019 22:00:00 GMT

We’ve all been there - Staring at the progress bar, wondering why task manager continues to say our 32-cores are sitting nearly idle while we run a geoprocessing tool on a huge dataset. OK, that huge dataset isn’t so large anymore - it’s the new normal. But why is ArcMap/Pro toying with us? We did the usual tweaks: data is local, fast drives, 64-bit BGP (for ArcMap), lots of free RAM. Still, the abnormally powerful “GIS workstation” computer (the one we had to beg for and write a justification document to get) is sitting there barely rotating the computer fans? Could be we aren’t taking advantage of the Parallel Processing Factor in ArcGIS.

Parallel Processing?

Parallel processing is a technique which splits up a task into many smaller chunks. The chunks are then assigned to multiple CPUs, cores, or processes to work on parts of the job at the same time. This technique can often result in faster processing times for larger datasets. For the past few releases, Esri has been increasing the number of tools that can take advantage of parallel processing. This allows some tools to distribute their work across multiple cores. Very handy if you have a good computer and want to take advantage of the extra power.

There are some limitations of course. First, not all tools currently take advantage of this. Over time the number of tools has been expanding, so worth reviewing your favourite tools after an upgrade. Second, you might not get the performance boost you’d expect due to the data or analysis you are trying to perform. If using ArcMap, make sure the tool is running in 64-bit mode via Background Geo-Processing as well (BGP).

64-bit and multithreading

Parallel processing is related to 64-bit and multithreading, but there are differences. For most geoprocessing tools, performance is roughly the same between ArcGIS Pro and ArcMap (if using BGP), while some tools in Pro are being updated to be faster/improved over time. 64-bit geoprocessing does not automatically make tools faster - but it does allow for more memory allocation during processing. 32-bit is limited to 2GB memory, while 64-bit doesn’t have this limitation.

64-bit geoprocessing is more robust, results will be more accurate, and processes that previously hung, crashed, or ran out of memory may be able to complete successfully ~ Esri

Likewise, multithreading doesn’t increase the performance of geoprocessing directly. However, with a decent computer, multithreading allows geoprocessing to take place in its own thread. Running independently, this allows other functions to continue on the main thread such as pan/zoom on a map, work with symbology, etc. So multithreading is still very useful if you want to work on other items in your map while the analysis works away on its own thread. Multithreading is used in both Pro and ArcMap (if using background processing in ArcMap).

ArcGIS Parallel processing

So how can we speed up the process? One possible way is to take advantage of the Parallel Processing Factor in ArcGIS. Again, not all tools are set up to take advantage of Parallel processing but Esri has mentioned that more tools will have this feature at every release. ArcGIS Pro 2.1 has 70 geoprocessing tools and ArcMap 10.6 has approximately 30 tools.

ArcGIS Parallel processing is managed by the Parallel Processing Factor environment variable. Some tools now default to parallel processing, while most tools still require you to set the parallel processing factor. The documentation does a good job at letting you know which ones, however you can always set the parameters just in case - or to override the default. In ArcGIS, the environment section will list this variable if the option is available. You can set this variable using the following options:

The Parallel Processing Factor is also available when using the ArcPy module with Python. This can be set the same as other variables at the top of your script if you want to leverage for all tools that accept it: arcpy.env.parallelProcessingFactor = "80%".

Final tips

Following the usual best practices, along with leveraging Parallel Processing should reduce the amount of time you spend watching the progress bar. However, don’t go overboard. If you specify more processes than you have available, you could negatively impact performance. One exception to that rule is when the analysis is I/O heavy or processing directly to an Enterprise database connection:

The Add Rasters to Mosaic Dataset tool is I/O bound when the mosaic dataset is stored in an enterprise database. Also, the Build Overviews tool is primarily I/O bound to the disk. You can use more processes than your machine has cores by either specifying a percent value greater than 100% or a number of processes greater than the number of cores on your machine. ~Esri

Helpful Links

Create a JavaScript Weather App with Location Data Part 2

Thu, 21 Mar 2019 08:00:00 GMT

In part 1 we went over the main components to create a JavaScript weather app with location data. From finding the device location, getting weather data from the internet machine (DarkSky API), and building a basic HTML interface. However, one limitation mentioned was how to keep the DarkSky “secret key” a secret.

In part 2 we will build a simple reverse proxy to hide our key on a server. A reverse proxy is when we redirect resources on behalf of the client from one or more servers. These resources are then returned to the client, appearing as if they originated from the proxy server itself. To create the proxy, we will use Node.js (server-side JavaScript). There are many other approaches, but Node.js allows us to use JavaScript and is one of the most powerful runtimes in use today. And who doesn’t love a little JavaScript!

The team at DarkSky also highly recommends you to protect your key. They feel strongly enough that they disabled cross-origin resource sharing (CORS) as a security precaution.

Your API call includes your secret API key as part of the request. If you were to make API calls from client-facing code, anyone could extract and use your API key, which would result in a bill that you’d have to pay. We disable CORS to help keep your API secret key a secret. To prevent API key abuse, you should set up a proxy server to make calls to our API behind the scenes. Then you can provide forecasts to your clients without exposing your API key. ~ DarkSky FAQ

Yeah, let’s do that… with JavaScript (Node.js)!

Node.js in a nutshell

Node is simply a runtime (environment) that allows us to run JavaScript outside a browser. With Node, we can build web servers, server APIs, web apps, desktop apps, and much more. When running JavaScript server side, you also don’t need to worry about Web Browser compatibility since the JavaScript engine is always the same - and built on Chrome’s JavaScript engine.

Node.js® is a JavaScript runtime built on Chrome’s V8 JavaScript engine. Node.js uses an event-driven, non-blocking I/O model that makes it lightweight and efficient. ~ Node.js

Node.js also uses packages to provide additional functionality, much like modules in Python. Many are included by default, but there are thousands of other great external packages that can help when needed.

For more details on Node.js, there are lots of amazing resources out there. Check out the Node.js website, along with a large library of YouTube videos, and Medium articles should give you an in-depth look into this very popular runtime.

The Proxy

For the proxy, we are going to use 3 modules. We will add a few more optional ones later for some additional security and validation. Dot-Env is also listed here, but could also be considered optional. Dot-Env will keep the key outside of the main code (if you want to post your code to gitHub, but not your key), there are many other ways we could perform that step as well. Here are the main packages:

Express: Web application framework that provides a robust set of features for web and mobile applications.
Request: Request is designed to make HTTP calls. It supports HTTPS and follows redirects by default.
Dot-Env: A zero-dependency module that loads environment variables from a file. This stores the configuration in the environment separate from the code.

The Project Setup

We will use a very simple project structure for this work. A server-side JavaScript file (server.js), and a Dot-Env file (.env) to store our secret key.

The .env file will contain a single variable with the key. The variable can be named whatever you would like - however, no spaces, and all CAPS is recommended. Let’s set this up as “DARKSKY_SECRET”. When we are ready to access this variable inside the server application, we just prefix the variable name “process.env.[variable]“. Based on our variable name above: process.env.DARKSKY_SECRET.

//.env file
DARKSKY_SECRET="abc123"

The Code

To build the reverse proxy server we start by creating the server.js file and add some of the modules we will be using.

//server.js - load the modules
const express = require('express');  
const request = require('request');
require('dotenv').config();
const app = express();

With the modules loaded it’s time to define a route (API endpoint). This endpoint will take the request from the client web application along with URL parameters. Once we have the request on the server, we will add our key and send our own request to the DarkSky API from the server. The response from DarkSky will be handed back to the client.

In the client app, we will change our original code from Part 1 that talks to DarkSky directly - subbing in our server route /darksky/:lat,:long. Both the “:lat” and “:long” keywords are placeholders on the server that will be populated when the client makes a request. A sample request from the client would look like https://myDomain.com/darkSky/-80.123,45.123. The server will interrogate the request and if it matches route name, the request will be forwarded. To access our lat/long variables in the request, we prefix with req.params..

//server.js - setup the route
app.get('/darkSky/:lat,:long', (req,res) => {
    //DarkSky URL we will send the request to
    var dsURL = 'https://api.darksky.net/forecast/';
    //Define a variable pointing to our secret key
    var dsSecret = process.env.DARKSKY_SECRET;
    //Additional DarkSky  URL parameters (optional)
    var dsSettings = '?exclude=minutely,hourly,daily,alerts,flags&units=auto';
    //Build the full DarkSky URL
    var url = dsURL + dsSecret + '/' + req.params.lat + ',' + req.params.long + dsSettings;
    //Send the request and direct the results back to the user
    req.pipe(request(url)).pipe(res);
});
//app starts a server and listens on port 3000 for connections
app.listen(3000, () => console.log('server ready'));

If we were to stop here, this entire exercise wouldn’t be worth the effort. We’ve created a reverse proxy that hides the key, but it doesn’t stop anyone else from using our proxy/routes with external apps. So anyone could leverage the proxy - while DarkSky sends us the bill. We aren’t going to add a full authentication tier, but it’s something to consider for production sites.

Basic protection

With the key hidden, let’s focus on adding some server protection. More could/should be done, and we should consider the below example a bare minimum. To keep this painless, let’s add a few additional modules to do the heavy lifting for us.

Helmet: Helps secure your Express apps by setting/blocking various HTTP headers
host-Validation: Extends Express (middleware) that protects Node.js servers from DNS Rebinding attacks by validating Host and Referer headers from incoming requests

//server.js - add protection
const helmet = require('helmet');
const hostValidation = require('host-validation');
//configure hostValidation to only accept requests from a specific host () and referer
//  if the following hosts/referers don't match the client, the request will be rejected
app.use(hostValidation({ hosts: ['myDomain.com'], referers: ['https://myDomain.com/weather.html'] }));
//Use helmet's default header rules for added security
app.use(helmet());

Rate Limiting

Rate limiting is used to control the rate of traffic received by the server and is used to prevent DoS attacks. We aren’t going to throttle (slow) each request to our server, just make sure each client doesn’t abuse the server and make thousands of requests per minute (remember, we want to keep this API from spending our beer money). We can set both a timeframe and the maximum number of requests to allow within that timeframe - for each IP requesting a weather update.

//server.js - add rate limiting
const rateLimit = require("express-rate-limit");
//app.enable("trust proxy"); //if already behind a reverse proxy
const limiter = rateLimit({
  windowMs: 5 * 60 * 1000, // 5 minutes
  max: 20 // limit each IP to 20 requests per windowMs
});
app.use(limiter); //apply to all requests

Putting it all together

Here is all the code put together. In total, we will need server.js and the .env file in the root of the project along with all of the modules in the default node_modules folder.

//server.js
const express = require('express');  
const request = require('request');
require('dotenv').config();
const helmet = require('helmet');
const hostValidation = require('host-validation');
const rateLimit = require("express-rate-limit");
const app = express();
app.use(hostValidation({ hosts: ['spatialtimes.com'], referers: ['https://www.spatialtimes.com/weather.html'] }));
app.use(helmet());
const limiter = rateLimit({
  windowMs: 5 * 60 * 1000,
  max: 20
});
app.use(limiter);

//Setup route for DarkSky
app.get('/darkSky/:lat,:long', (req,res) => {
  var dsURL = 'https://api.darksky.net/forecast/';
  var dsSecret = process.env.DARKSKY_SECRET;
  var dsSettings = '?exclude=minutely,hourly,daily,alerts,flags&units=auto';
  var url = dsURL + dsSecret + '/' + req.params.lat + ',' + req.params.long + dsSettings;
  req.pipe(request(url)).pipe(res);
});

const port = 8001;
app.listen(port, () => console.log('server ready'));

Now that our server is ready with the reverse proxy we would just need to change the URL request in the HTML file to call our new endpoint instead of going to the DarkSky API directly.

Create a JavaScript Weather App with Location Data Part 1

Tue, 15 Jan 2019 20:00:00 GMT

In the previous post, we discussed the possibility of writing our own location-based weather application. So let’s try it! In part 1 we will build a simple app that gets the users’ location and fetches weather data from an API using HTML and JavaScript. This approach is similar to using various mapping APIs to find the user location and will take us a bit behind the scenes on how some of the pre-built location map widgets really work.

Here are the steps we will work through to build the app:

Get user coordinates using the HTML5 GeoLocation API
Request weather data for the coordinates using an online weather API
(optional) Reverse Geocode the coordinates to get additional location details

* Not required for most weather APIs, but worth touching on

Before we begin: The GeoLocation API

To request a user location (lat/long coordinates) within the web browser we will leverage the HTML5 GeoLocation API. This functionality is available in all major browsers (desktop and mobile) and includes IE - yeah, crazy I know. But just because it’s supported, doesn’t mean our code will always bring back a valid location. There are a few reasons why we might not get what we are looking for:

When using the API in most browsers the GeoLocation results are only available over secure connections (HTTPS). So the final product should be hosted with HTTPS (we can enable this for free these days, so this shouldn’t be a show-stopper).
- An exception to the HTTPS rule is local file:// testing which still works.
The Geolocation API might be blocked by the user/settings. In most desktop/mobile operating systems users have the ability to block location services from being exposed to the browser. In addition, the user will be prompted to provide permission for an individual website as per W3C specifications.
A request timeout can occur if the data takes too long to return. An optional PositionOptions object has a timeout setting representing the maximum length of time (in milliseconds) the device is allowed to wait for a return a position.
- The default timeout value is Infinity, which is a long time to wait. It’s a good idea to set a reasonable timeout. Even slower response times should be expected if using this in combination with PositionOptions.enableHighAccuracy.

Note: As of Chrome 50, the Geolocation API will only work on secure contexts such as HTTPS. If your site is hosted on a non-secure origin (such as HTTP) the requests to get the users location will no longer function ~ w3schools.com

Thankfully the Geolocation API has some error checking available to provide some details when troubles occur, including timeouts. It’s then possible to add alternate location approaches such as IP Address lookup, prompt the user for a city, or to change their browser settings.

1. Get user coordinates using the HTML5 GeoLocation API

To start, we should determine if the browser has basic support for the Geolocation API. Again, even IE9+ supports this but still a good idea to test and very simple to implement. If for some reason there is no support, we can send the user a message letting them know their browser sucks.

//Check if the geolocation API exists
if (navigator.geolocation) {
  //true
  alert('Lets get the location (placeholder)');
} else {
  //false
  alert('geolocation not available?! What browser is this?');
  // prompt for city?
}

Assuming everything goes well we can swap our placeholder with some code to request the actual location coordinates. The GeoLocation API has three main methods:

getCurrentPosition() - used to return the user’s position. A single request per call. This is the one we will use for the example
watchPosition() - Returns the current position of the user and continues to return updated position as the user moves
clearWatch() - Stops the watchPosition() method

The getCurrentPosition and watchPosition methods are asynchronous, which means we create a callback to deal with the location data whenever it’s returned (or when an error is returned). A successful return can have up to 8 properties, however, only 3 of these are guaranteed: latitude, longitude, accuracy.

Here is a short sample with no error checks with an inline success callback so we can easily identify the basics:

//Short sample version with inline success callback
if (navigator.geolocation) {
    //Initial a request for the location
    navigator.geolocation.getCurrentPosition(function(pos){
      //'pos' return object has many properties we can grab
      var geoLat = pos.coords.latitude.toFixed(5);
      var geoLng = pos.coords.longitude.toFixed(5);
      var geoAcc = pos.coords.accuracy.toFixed(1);
    });
}

Here is a more complete sample with both the success and error callbacks in place. Error messages are broken out just in case we want to add more options in the future.

//More complete version
if (navigator.geolocation) {
    // Request the current position
    // If successful, call getPosSuccess; On error, call getPosErr
    navigator.geolocation.getCurrentPosition(getPosSuccess, getPosErr);
} else {
    alert('geolocation not available?! What year is this?');
    // IP address or prompt for city?
}

// getCurrentPosition: Successful return
function getPosSuccess(pos) {
  // Get the coordinates and accuracy properties from the returned object
  var geoLat = pos.coords.latitude.toFixed(5);
  var geoLng = pos.coords.longitude.toFixed(5);
  var geoAcc = pos.coords.accuracy.toFixed(1);
}

// getCurrentPosition: Error returned
function getPosErr(err) {
  switch (err.code) {
    case err.PERMISSION_DENIED:
      alert("User denied the request for Geolocation.");
      break;
    case err.POSITION_UNAVAILABLE:
      alert("Location information is unavailable.");
      break;
    case err.TIMEOUT:
      alert("The request to get user location timed out.");
      break;
    default:
      alert("An unknown error occurred.");
  }
}

That’s all it takes. If all goes well we now have the latitude and longitude coordinates. With this information available, we can to pass it to a weather API and/or convert the coordinates into a known location (reverse geocode).

2. Request weather data for the coordinates using an online weather API

There are many weather APIs available, and most of them have a free tier (for low usage and/or testing). Each will have some pros and cons, including various advanced features (15-day forecast, multiple forecast models, etc). One major change in this area is the recent shutdown of the very popular Yahoo! Weather API, probably one of the most used weather APIs over the past decade. As of January 3rd, 2019 the service is offline with a replacement service just starting its “by-request onboarding” phase. Since that’s currently not available, we are going to use DarkSky. This API has an easy to use free tier and accepts Lat/Long inputs. If using for production purposes it’s advised to review all of the APIs to find one that best fits your needs along with appropriate Terms of Use.

Side note on Weather APIs: Looks like this world is collapsing into more of an oligopoly. Weather Underground was purchased by The Weather Channel (2012), which was then acquired by IBM (finalized in 2017). Weather Underground was set to retire their API Dec 31, 2018, but have extended this to February 15, 2019, to allow more transition time.

Side note on Dark Sky: This is not a sponsored post and I have no affiliation/relationship with DarkSky. The use of their API is based on it being easy to use, good documentation, and a free usage tier.

Using DarkSky’s Weather API

The DarkSky developer API will require you to register an account to get access to a key. Once complete, a request is fairly simple:

<!-- API Params -->
https://api.darksky.net/forecast/[key]/[latitude],[longitude]
<!-- Sample Request -->
https://api.darksky.net/forecast/myFakeKey123abc/43.642567,-79.387054

The resulting JSON data returned from a request like this can be pretty substantial. We can reduce the response using query parameters (filters) to focus on what we want, and also set the measurement units (si, ca, us, uk2, auto). Now we can make a more streamlined request:

<!-- API Params for filters and units (auto) -->
https://api.darksky.net/forecast/[key]/[latitude],[longitude]?exclude=minutely,hourly,daily,alerts,flags&units=auto
<!-- Sample Request with filters and units -->
https://api.darksky.net/forecast/myFakeKey123abc/43.642567,-79.387054?exclude=minutely,hourly,daily,alerts,flags&units=auto

Here is the returned JSON object at the CN Tower in Toronto, Ontario, Canada (43.642567, -79.387054) with filters and units set:

{
   "latitude":43.642567,
   "longitude":-79.387054,
   "timezone":"America/Toronto",
   "currently":{
      "time":1546832805,
      "summary":"Partly Cloudy",
      "icon":"partly-cloudy-night",
      "nearestStormDistance":219,
      "nearestStormBearing":75,
      "precipIntensity":0,
      "precipProbability":0,
      "temperature":-5.11,
      "apparentTemperature":-10.56,
      "dewPoint":-13.61,
      "humidity":0.51,
      "pressure":1032.58,
      "windSpeed":14.42,
      "windGust":26.78,
      "windBearing":38,
      "cloudCover":0.59,
      "uvIndex":0,
      "visibility":16.09,
      "ozone":247.23
   },
   "offset":-5
}

Much easier to work with this smaller dataset, but it does remove some advanced attributes we might want to tap into later. For now, we sent a request based on user location and received the weather data. Time to build a basic GUI and grab a beer (who cares if it’s -5 celsius outside). While we let our beer warm up and breathe a little, we will wrap this request into JavaScript.

Building the request in JavaScript

With the DarkSky URL syntax ready to go, we just need to make the call in our code. Well, almost ready. There is one last hurdle: CORS. DarkSky has disabled Cross-Origin Resource Sharing (CORS) on their servers, and for good reason. As recommend by Dark Sky, you should leverage a secure proxy where you can store your Secret API Key (don’t put this in client-side JavaScript or bad people will do bad things with it).

Your API call includes your secret API key as part of the request. If you were to make API calls from client-facing code, anyone could extract and use your API key, which would result in a bill that you’d have to pay. We disable CORS to help keep your API secret key a secret ~ Dark Sky FAQ

Argghhh, the CORS limitation doesn’t help when building a JavaScript example. As a quick workaround, I’m going to leverage the “Heroku method” which will use the Heroku public CORS proxy and send the secret key directly from the client. THIS IS FOR TESTING ONLY so please don’t use this approach in your published code. Only a private proxy or create your own with your secret behind the scenes. It’s called a “secret” for a reason.

CORS limitations behind us, we just need to make our JavaScript request and use the results for our interface. For this, we are using jQuery’s getJSON() method. jQuery isn’t required here, it’s just a shorthand call to the XMLHttpRequest() constructor which you could use instead. I’ll place this in a separate function to keep things easy, and because you could put this entire function in your proxy if desired (just send geoLat and geoLong to the proxy to keep your secret key hidden).

jQuery’s getJSON() will request JSON-encoded data from the server using a GET HTTP request. In general, it takes a URL (like the weather ones we crafted earlier), and returns a Deferred object. In this case, it implements the Promise interface so we can provide multiple callbacks. A successful callback will route to .done(), while an error response will be directed to .fail().

_dsSecret = "yourSecret"; //Again, for testing only, should be hidden in proxy

function fn_getWeatherByLL(geoLat,geoLng){
  //API Variables
  var proxy = 'https://cors-anywhere.herokuapp.com/';
  var dsAPI = "https://api.darksky.net/forecast/";
  var dsKey = _dsSecret + "/";
  var dsParams = "?exclude=minutely,hourly,daily,alerts,flags&units=auto";
  //Concatenate API Variables into a URLRequest
  var URLRequest = proxy + dsAPI + dsKey + String(geoLat) + "," + String(geoLng) + dsParams
  //Make the jQuery.getJSON request
  $.getJSON( URLRequest )
    //Success promise
    .done(function( data ) {
      var wSummary = data.currently.summary;
      var wTemperature = data.currently.temperature;
      // lots of results available on the data object
      // use the results to populate the GUI here
    })
    //Error promise
    .fail(function() {
      alert('Sorry, something bad happened when retrieving the weather');
    }
  );
}

Have Weather > populate GUI

Once we have weather data for the location, we can populate the user interface. In the code above, the data object contains all the information needed to build your weather dashboard. From current temperature, daily max/mins, and short-term forecast - how you present this to the user is up to you.

The image above is based on an example JSFiddle adding results that I’ve already requested in advance using the methods above. The only thing needed is to update the HTML text of the IDs when the data object is returned in the promise.

Notice we don’t have a city/place name on the page, just latitude and longitude - Turns out this information isn’t available in the return object. If you want to find the place name of the current location, see the Reverse Geocoding section below.

Dark Sky Forecast Embeds

At this point, I should probably confess that Dark Sky also has a number of embeddable weather widgets. Don’t be mad, if I started with this information, we wouldn’t know what it takes to make it all happen - and possibly miss out on some JavaScript goodness. You’re welcome. Dark Sky has many ways to include their widgets if you can agree to their widget terms of service. These can be accessed directly from Dark Sky as scripts, an iframe (via Forecast.io), and from Weatherwidget.io.

If you just want a widget on your page, good news, no secret keys! Just get the location from the Geolocation API and create an iframe on the page dynamically.

function getPosSuccess(pos) {
  // Get the coordinates of the current possition.
  var geoLat = String(pos.coords.latitude.toFixed(5));
  var geoLng = String(pos.coords.longitude.toFixed(5));

  //Create an iframe and use the current location data
  var iSource = "https://forecast.io/embed/#lat=" + geoLat + "&lon=" + geoLng + "&name=Woot&color=#00aaff";
  $('<iframe>') // Creates the element
    .attr('src', iSource) // Sets the attribute spry:region="myDs"
    .attr('height', 245) // Set the height
    .attr('width', "100%") // Set the width
    .appendTo('#id-weather'); // Append to an existing element ID
}

Code Complete

That covers the basics of finding the device location, getting weather data from the internet machine (DarkSky API), and building a basic HTML interface. Now you can play with the weather data and determine how to enhance the interface.

This is really it for part 1. In part 2 we will build a proxy using Node.js (server-side JavaScript) to show one approach to dealing with secret keys.

Oh, I did mention we could reverse geocode the coordinates as well…

3. Reverse Geocode the coordinates to get additional location details (optional)

For weather data, we don’t need amazing accuracy because weather data doesn’t change at the street level. So even if the accuracy sucks, it should have little consequence. But if you want to find and show the current city, address, or include a pin on a map, reverse geocoding to the rescue. Using a Reverse Geocoding API is pretty much identical to using a weather API - Provide coordinates in a URL and wait for results.

Again, we have lots of API sources to reverse geocode a lat/long to an address: Esri search API, Google Maps/places API, Bing Maps API, and OpenStreetMap API to name just a few. A future article will explore these options in more detail. For now, let’s take a quick look at OSM Nominatim.

OSM Nominatim

OpenStreetMap “Nominatim” has a REST endpoint to reverse geocode a pair of coordinates. The wiki help does an excellent job getting into all the options here but it is actually handled the same way as DarkSky. Nominatim requires three parameters: format, latitude, and longitude. Using the format option in the example below, a JSON object is returned with lots of information.

<!-- Sample request -->
https://nominatim.openstreetmap.org/reverse?format=jsonv2&lat=43.642567&lon=-79.387054

// return object
{  
   "place_id":"84050944",
   "licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright",
   "osm_type":"way",
   "osm_id":"32742038",
   "lat":"43.6425637",
   "lon":"-79.3870871832047",
   "place_rank":"30",
   "category":"tourism",
   "type":"attraction",
   "importance":"0.466188092284943",
   "addresstype":"tourism",
   "name":"CN Tower",
   "display_name":"CN Tower, 301, Front Street West, Entertainment District, Old Toronto, Toronto, Ontario, M5V 2X3, Canada",
   "address":{  
      "attraction":"CN Tower",
      "house_number":"301",
      "road":"Front Street West",
      "neighbourhood":"Entertainment District",
      "city_district":"Old Toronto",
      "city":"Toronto",
      "state":"Ontario",
      "postcode":"M5V 2X3",
      "country":"Canada",
      "country_code":"ca"
   },
   "boundingbox":[  
      "43.6423338",
      "43.6427965",
      "-79.3874416",
      "-79.3867985"
   ]
}

This provides lots of location information. Worth checking in various rural/urban areas to see how the results differ, but overall this gives a good idea of what to expect.

Weather apps and location data in the news

Tue, 08 Jan 2019 20:00:00 GMT

For some reason, my news feeds have recently been filled with articles about Weather Apps and how many of them are selling our location data to the highest bidder. After reading a few, I started to look at my weather apps (yes, I have a few) in more detail.

Location data in the news

Location-based Weather apps are everywhere but lately, they’ve been in the news for the wrong reasons. Turns out, many App Developers are taking our location data, even if we say not to, and selling it to third parties. If we grab a “free” app, we should assume that most of the time the App Developer still wants to make money. How do they make bank? By selling the data they collect. Location data among it. So why do weather apps get stuck holding the headlines as they surely aren’t the only ones doing this (Facebook)? Well, a few reasons off the top of my head why weather apps are at the forefront:

There are so many of them:
- Website 42Matters stats reveal iOS having 4,951 free weather apps listed for iOS (on 2019-01-07)
Weather apps require the user to share location for them to work fully (might still work otherwise, but we tend to share our location for these apps)
They are used daily, so lots of ongoing data gathering
Weather apps are pretty easy to create
Some smart people looked into this and yikes, it scary:
- New York Times: Weather Bug and Weather channel sending data directly to third parties
- The Wall Street Journal popular-weather-app-collects-too-much-user-data-security-experts-say
- Motherboard: Weather apps including The Weather Channel, Accuweather, and WeatherBug—have shown that they share your location data

Location, Location, Location

Location data - there used to be only a small niche working with this data - the GIS Geek. We would go to conferences and repeat one of our favourite truisms: “80% of all data is spatial”. Sure, we have a hard time proving where that stat came from, but if anything, in today’s world we are probably underestimating.

These days, no matter what you are doing with technology, there is a chance someone is getting the tracking data of where you are while doing it. Why not?! Location analytics has also come a long way with big data, AI, and demographic profiling tools at our fingertips. As someone who’s worked in GIS for many years, I’m not shocked this is happening, more shocked people now know more about my chosen profession.

Technology is Location Aware

With phones and watches having embedded GNSS chips, and in an always-connected world where we can even trace an IP address to a city block - location data is proliferating. Every site we sign-up with can get our IP address, and probably ask us for our home address or zip/postal. Companies see the value in this data and are finding more ways to collect and leverage it. This is no different than the old Nielsen ratings that started back in 1947. Companies have always wanted to know how they were doing and to know more about who their audience is since this can help sell ads and better understand the customer. The data itself is also worth a pretty penny. Combining location and usage data with census information and profiles - can give a very detailed picture.

How To Protect Yourself

To be honest, most people are probably somewhat aware this is happening but don’t care. Privacy seems like a thing of the past in the digital world. We may even trust some companies with our data. We also assume that the data is secure, and if shared/sold, we pretend they are secure too. Really?! In Canada, there is a federal privacy law (PIPEDA) setting rules for acquiring “meaningful consent” if a company wants to share your personal information. After reading a few Terms of Use, pretty sure some are even in violation of federal laws in their sharing of data with third parties. Especially if they track your location without permission (you can still block this using your os privacy settings).

Time and time again we hear about another database being hacked. That data we were comfortable sharing initially is now in the hands of the unknown. Still, some might not care (if it doesn’t include their favourite/only password). From a GIS perspective, just think about what you could do with this location data. Hopefully, criminals don’t have a GIS background, or they could do some pretty crazy stuff - beyond understanding their “potential customers” better. Perhaps even a choropleth map.

So think about the vast mountain of data you are sharing with apps and in what combination. Then put on your hacker/GIS hat and think about what you could do with a dataset of 500 million records (Marriott).

How To Protect Others

Education and implementation. We should remind ourselves and others that our data is ours, and has value - so worth guarding. Use a password manager or at least make sure to use unique passwords that aren’t easy to guess. Don’t use the same password for your email account (Gmail, etc) on any other website where you use the same email to register. Only share location data if you think it is worth it and adds value to your use of the app.

Most importantly, there are lots of great articles out there on how to secure your digital self. A good time to explore a few. Then delete some of the 4,951 weather apps from your devices.

Where do we go from here?

Maybe it’s time to try writing our own location-based weather app?! Hmmm, stay tuned…

Upgrading Spatial Times

Wed, 02 Jan 2019 16:00:00 GMT

Spatial Times is getting an upgrade. OK, more of a rebuild from scratch. After 5 years, it was time to update the interface and clean-up some server code. There will be some growing pains over the coming weeks, but it will be worth it (he said with trepidation). The biggest changes are behind the scenes, as the site is moving from a sluggish Wordpress site to a newer Static Site Generated (SSG) approach leveraging React and GraphQL. Selfishly, this also reduces my hosting cost which isn’t a bad thing either.

Static Site Generated with Gatsby

After lots of trial and error, I decided to go with Gatsby to build the new version of the site. This gave me the opportunity to continue my dive into modern development approaches with React.js, Webpack, GraphQL, and TypeScript (honourable mention to NPM, git, Node.js and VS Code of course). Over time I will add more Progressive Web App (PWA) functionality for better offline experience - but one step at a time.

All of the posts have been converted to markdown files as I embrace the JAMstack way of life. With all these changes, some things are bound to break. But if I miss something today, I will hopefully fix it tomorrow. Although if you see something wonky that persists for a while, let me know.

Why Upgrade Spatial Times?

Beyond what’s already been said, the upgrade lends itself to a learning opportunity. As most of the posts are GIS related, this learning isn’t limited to Spatial Times. GIS and the web are constantly evolving. Companies like Google, Esri, and Microsoft are all moving in this direction: Esri recently added the JavaScript API to NPM for creating custom builds; Google is focused on making PWA’s the future of web/app development; Microsoft bought GitHub (and big contributor), makes VS Code freely available (best code editor IMHO and built on Node/Electron), and Edge is moving to Chromium engine.

And there’s performance. Below is an audit report from Lighthouse to the original Spatial Times site using Wordpress. You can see the site was pretty good in the SEO and Best Practices categories, has decent accessibility, and dismal performance. To be fair, I didn’t spend too much time trying to squeeze performance out of the site - but shouldn’t it be fast by default?!

lighthouse audit of the old site

Without any tweaking of the starting template, the new site is getting a more favourable audit. The new site has a drastic improvement in many areas with Performance and Accessibility being my primary focus. I will spend some time making updates for PWA and SEO, but not going to lose any sleep with these new results.

lighthouse audit of the new site

A Few Changes

I’ve been working to keep the links to articles the same as to not impact any existing bookmarks or hyperlinks to the site. There are some changes as well:

The site is moving to HTTPS. Seems simple enough, but this means the entire site URL is changing. Even with post slugs remaining the same, the root will be different. All inter-site links are updated, but external sites may have an issue. I’m working on a redirect now but might take some time.
The Search widget has been removed. Let’s be honest: almost nobody used the search feature anyway. Search engines (Google, Bing, Duck-Duck-Go) are how we all search these days. They even have features to search within a specific site. Not worth adding to the competition at this point.
Post content has streamlined styling. Markdown files keep it simple when writing a new post. Custom CSS is possible, but not needed. This is a good thing as sometimes too much time is spent finding a custom style to highlight a point. Way too much time.

What’s Next

Hopefully everything goes as planned, but I’ll let the dust settle before making more changes. Maybe even find some time to publish some posts that have been sitting in the attic.

ArcGIS Online User Levels are Changing

Fri, 16 Nov 2018 16:50:25 GMT

A major change to ArcGIS Online User Levels are coming December 4th, 2018. This is a welcome change to the current “Level 1” and “Level 2” setup, but it will come with some growing pains. These User Levels will be converted to the newer ‘User Types’, and each User Type will have a mix of capabilities and Apps available.

What’s changing?

With the December release of ArcGIS Online, all existing Level 1 and Level 2 users will be changed to new User Types, along with 3 additional User Types being added. The new names are: ArcView and ArcIn… just kidding! For existing accounts, this is more of a re-branding since nobody wanted to be Level 3 with a sibling that was Level 4 (that is dating a Level 1 that used to be a level 5). Drama.

The bigger change here is the introduction of a new “GIS Professional” type. It’s this new type that has access to ArcGIS Pro, and isn’t directly related to the current Level 2 account. Maybe a chart and picture will help:

Compare old and new User Types

New User Types (via Esri)

Comparing User Types

User Types will be the new way to setup users in your organization based on the Capabilities and Apps required to do their work. Some of these new User Types will also have additional add-ons that can be configured to enhance the standard set of apps available.

The primary capabilities for each User Type is an expansion on the Level 1 and Level 2 users of today. I’ve taken the list from Esri and removed the App bundles to focus on the primary capabilities, but I strongly encourage reviewing their documentation for full details.

AGOL Capabilities

The Apps provided for each level are grouped together in named bundles. This is because we didn’t have enough new terminology at this point (and much easier to refer to a long list of apps if Esri bundle’s them). Either way, here is the breakdown with a selection of pastel colours I was able to find in Excel:

Apps by bundle and user type

Wondering where some of the niche Apps went? Not to fear - Navigator for ArcGIS, ArcGIS Business Analyst, Insights, Drone2Map, and Maps for Adobe Creative are all still available to add to your subscription.

Mark your calendars!

More information

Use ArcPy to Connect to SDE and Versions

Mon, 01 Oct 2018 15:52:15 GMT

This post will discuss how to use ArcPy to Connect to SDE, create a new version, and switch to that version all inside a Python script. At first glance this should be pretty straight forward - provide some connection details and a version name. Well, ArcPy isn’t really designed for that approach. There currently isn’t an ArcPy function to connect to SDE within code, or to deal with new/changing versions. The current documented approach is to use Toolbox tools or create the version in advance, create a connection file pointing to a version, and use this as a reference in a script.

There are many reasons why the documented approach might not meet our needs: We want everything inside one script without any related files required; We don’t know the new version name in advance; We want to run on Linux (ArcGIS server); Or we simply just want the option to use a single script for our scripty stuff.

Creating a SDE connection file in ArcPy

At this point, it might be easier to open ArcCatalog, create the SDE connection file, and point your script to it. But if you want the “everything we need is in this script” approach, how far can we go? Well, not too far unless we are willing to use some Toolbox tools along the way.

Creating the SDE connection file isn’t a problem since there is a toolbox tool to create one. Inside our Python script we can call the CreateDatabaseConnection_management tool directly and save the connection file for later use.

CreateDatabaseConnection_management(out_folder_path, out_name, 
    database_platform, instance, {account_authentication},
    {username}, {password}, {save_user_pass}, {database},
    {schema}, {version_type}, {version}, {date})

We are going to use Windows Authentication so we don’t need to expose any user credentials for now. When creating the connection, we need to supply a few parameters that impact our script:

‘version’: The name we specify here must already exist. In this example, we will use the Default version.
An output folder path (directory) for the file that is created. If on a server, there is a chance we aren’t sure where to save, or the script directory is read-only, and saving to c:\temp makes some admins a little nervous. Hmmmm, where to put the file?

What if we used the ArcPy “inMemory” space for the output? This would meet our needs, but inMemory doesn’t currently support this technique. Thankfully there is a great module called “tempfile” that comes included with core Python to handle this much the same as inMemory. The tempfile module generates temporary files and directories. It also works on all Python supported platforms.

tempfile.mkdtemp([suffix=''[, prefix='tmp'[, dir=None]]])

Within the tempfile module, there are commands for both directories and files that we can use much like the inMemory feature. The directory will be secure and accessible by the account running the script, just make sure to cleanup the temp directory when you are done.

sdeTempPath = tempfile.mkdtemp()
arcpy.CreateDatabaseConnection_management(sdeTempPath,
  'ConnName.sde','SQL_SERVER','db\\instance',
  'OPERATING_SYSTEM_AUTH','#', '#', '#','Database')

Creates a temporary directory in the most secure manner possible. There are no race conditions in the directory’s creation. The directory is readable, writable, and searchable only by the creating user ID. The user of mkdtemp() is responsible for deleting the temporary directory and its contents when done with it. ~Python Software Foundation, Overview of Tempfile

Create a new version in ArcPy

Toolbox is again required to create a version, using the CreateVersion_management tool.

CreateVersion_management (in_workspace, parent_version,
  version_name, {access_permission})

When creating a version in a DB like SQL Server, it will prefix the version name with the name of the user - even though we didn’t supply ours directly, it was determined when we created the connection file with reference to the current Windows Account. So before we can use the version, we will need to find it. Side note: We are assuming the version doesn’t already exist - you might want to check before creating.

arcpy.CreateVersion_management(sdeTempPath + os.sep +
  'ConnName.sde', 'sde.DEFAULT', 
  sdeVersionName, 'PUBLIC')

Using the new version

To continue complicating things, we can’t just switch our newly created version in the connection we already created. The only way to reference a version in ArcPy is to use a connection file that already points to that version. A little redundant, but good news is that we just learned how to create a SDE connection file, so we just need to find the full version name (user.version) and create a connection one more time with the new version name included.

The Code

Here is a script that will use ArcPy to Connect to SDE, create a new version, and switch to that version all inside a Python (ArcPy) script. This is just a sample and it is recommended to add more error handling, trap for correct licenses, check if the version exists, etc.

References

Collector for ArcGIS Updated v18.0.1

Sun, 22 Apr 2018 13:59:57 GMT

Collector for ArcGIS v18.0.1 is now released for all supported platforms. For iOS users, this includes the use of Leica’s Zeno GG04 plus. All platforms get an enhancement for offline tile usage security, and many bug fixes.

The bug fixes for iOS include many of the minor, yet annoying, issues that create some confusion with field staff:

BUG-000091922: After searching for a feature in Collector for ArcGIS (iOS) on an iPad Air, there is no option to Copy, Edit, Delete, Zoom to, or Get Directions to the feature
BUG-000092762: Orphaned replicas are seen under the REST endpoint of a hosted feature service after removing a map from a mobile device via the Collector for ArcGIS (iOS 8.x) application without synchronizing the edits
BUG-000093776: Collector for ArcGIS (iOS) shows duplicate results if multiple searches and selections are executed for the same feature using the Find Locations By Layer application setting
BUG-000095584: Unable to add attributes to a related table after searching the feature with a hosted feature layer in Collector for ArcGIS (iOS)
BUG-000109015: After searching for a feature in Collector for ArcGIS (iOS), updating a field that controls a feature’s symbology from these search results causes the symbology of the feature not to update, even though the field is updated.
BUG-000109894: Collector for ArcGIS (iOS) location tracking stops when the basemap is changed.

Make sure to update all your devices, and have fun in the field!

SXBlue Platinum, In the Field

Wed, 07 Feb 2018 22:45:48 GMT

It was finally time to get bundled up and test the SXBlue Platinum in nature. Not only to test the functionality, but with the recent -20c outside, to test the device in a traditional Canadian winter. Before heading out, it was time to setup the device with ArcGIS Collector (v17.0.4). Just in case, the device comes with a great quick start guide to explain the status lights, and how to pair with iOS and Android. It goes even further and helps you setup the SXBlue Platinum with ArcGIS Collector which was a great reminder on the “Provider” adjustments that may be needed.

SXBlue Platinum and ArcGIS Collector Setup

There are many ways to use the SXBlue Platinum with ArcGIS Collector. The basic setup is to configure the Provider to SXBlue and leave the Location Profile defaults (AKA, do nothing). This is the one to use when the device is reading SV’s from GPS/GLONASS, etc. without any additional augmentation/corrections. Here in Ontario, there is a very likely chance the device will pickup SBAS (WAAS). With this bonus augmentation, there is a setting change to the Location Profile to get the optimal config in Collector (you need to tell Collector about the signal). Don’t worry, the Quick Start guide covers that too!

By checking the SXBlue status indicators or the status screen in Collector, you can determine if you are currently connected to SBAS.

SXBlue Platinum - SBAS for all!

Update Note Feb 08, 2018: The items in the image to help verify SBAS are the “Fix Type”, “Station ID”, and “Correction Age”. The Location Profile name “SBAS” is my own naming of a profile and doesn’t reflect the current WAAS status.

If you are using additional correction services for increased accuracy such as Atlas, CAN-Net, Top-Net, etc - you will also need to change the Location Profile to match the signal from the correction service.

Vertical Elevation

ArcGIS Collector doesn’t currently support adding a GEOID for Orthometric height. The SXBlue Platinum, like every GNSS device, will output Ellipsoidal elevations directly to the application it’s paired with. The device doesn’t store GEOIDs inside (not unlimited space), but does work with Apps/Software that do support GEOID files. So no matter what you are collecting in ArcGIS Collector, all vertical data is being stored in Ellipsoidal height. If you really want to store Orthometric elevation values, the SXBlue Platinum does work with other iOS/Android apps that have this ability. In short, this is a limitation of ArcGIS Collector, not the SXBlue Platinum. Maybe a future release from Esri will incorporate this feature - although you can always post process.

In the Field

My preference it to mount the antenna to a pole or a small extender on a backpack depending on accuracy needs. These approaches give the antenna 360 degrees of SV potential. In collector you can set a mounting height in the Provider setup screen. With the DGPS light glowing and Collector Status showing that I have WAAS, I can see that the Blue circle on the map is nice and tiny - a good sign of high accuracy beyond the status window. At this point, the accuracy was… 49cm (1.6 ft). Reviewing with some controls, accuracy was definitely up there. So overall, the SXBlue Platinum is sending data at the level of accuracy we would hope for.

As mentioned in the SXBlue Platinum Introduction, the RTK package comes with 1Hz by default. This is great for any foot powered mission. And if you want to upgrade to 10Hz or 20Hz and mount to a race car, those options are still unlock-able at any time. As I walked through the trails and on the sidewalk I would keep my eye on the status window - nothing much to report as I had strong accuracy readings all afternoon. Every once in a while I would switch to the SXBlue app for a more detailed look at what the device was doing.

iSXBlue RTN App for iOS

Lots of SVs, strong signals: so boring - and boring is great. I tried picking up my pace and running (relatively speaking) through a few sections and the SXBlue was doing a great job updating the paired Collector app.

Overall, the SXBlue Platinum did a great job - and this is without some of the more advanced features like an additional correction services. In reflection, it was just easy, worked as expected, and exceeded my accuracy expectations (full report on that another time). All good stuff when getting out into the elements!

More Info

SXBlue Platinum, an Introduction

Sun, 14 Jan 2018 22:00:05 GMT

The SXBlue Platinum is one of the newer members to Geneq’s SXBlue family. As discussed below, there is really more than one Platinum - with 3 packages and some additional options available. This first post on the SXBlue Platinum is an introduction to the device, not an in-depth review. The goal is to help explain the features and capabilities of this little blue box. Once the -30 degrees C (-22 F) cold snap ends I’ll get my butt outside to do some field testing with ArcGIS Collector and related apps.

What is the SXBlue Platinum?

What is the SXBlue Platinum? Well, like many GNSS devices these days, it depends on the features you have enabled. What’s physically included in the little blue box? Everything. However, it just might not all be enabled by default. Depending on your needs, this can be a good thing, or a little confusing. Want to start out with an update rate of 1 Hz, and possibly unlock 10 Hz or 20 Hz at a later date? No problem. Or maybe use DGPS using the Atlas L-band correction service? Sure thing. Or possibly start leveraging RTK? You can do that too. But if you are only interested in basic functionality, there is an aptly named “basic” package that neuters the features (completely reversible procedure of course).

Specs and Accuracy

Ah yes, what most of us are interested in: Specs and accuracy. This varies by package/options, but the stated horizontal accuracy ranges from 8 mm + 1 ppm (RTK) to 1.2m (autonomous). I should also mention the accuracy measurement used here is Horizontal Root Mean Square (HRMS). The range is based on which components are enabled and available. The usual location considerations also have an impact (multi-path, signal strength, SVs in view, PDOPy stuff).

The SBAS/WAAS accuracy is up to 30cm, which is great considering no RTK or rovers are needed for this. Vertical accuracy isn’t listed on their website (yet) but Geneq confirmed it is 60cm VRMS for SBAS which is pretty typical considering the horizontal (V is usually 2 to 3 times H). It should be noted that the older SXBlue II spec sheets used the 2dRMS SBAS accuracy stats - so this isn’t a change from the older model specs, just a different representation. The 30cm HRMS of today is the same as 60cm 2dRMS of yesterday .

This switch to stating accuracy using RMS shouldn’t be a surprise to anyone as it aligns with many other vendors (and looks better in brochures). I do like that they include this information in their media as some vendors still mislead by omitting the statistics being communicated (looking at you Garmin).

But accuracy isn’ the only spec worth checking out with the SXBlue Platinum. The signals are definitely worth mentioning here as well. As stated earlier, everything is included inside this device. Access to 5 systems (4 global and 1 regional), as well as 3 GPS bands/signals, with 394 channels, and a replaceable/rechargeable Lithium-Ion battery (~10 hours on a full charge).

SXBlue Systems & Signals

Multiple bands, SBAS, and Sat. Systems will definitely help over the long run. The addition of planned GPS L5 band is beneficial as the rollout of this newest modernization continues through 2024. This band is more powerful allowing to track weaker signals, and better “ionospheric effect attenuation” when using in combination with another signal (in laymen terms: 2 signals are better than one). Ditto with 3 signals:

Through a technique called trilaning, the use of three GPS frequencies (L1,L2,L5) may enable sub-meter accuracy without augmentations, and very long range operations with augmentations. ~ GPS.gov

For a full list of specifications, check out their product page and brochure for a slew of geeky stuff. But one last feature worth mentioning is the inclusion of a battery meter on the device. I know it’s a small thing but the ability to see how much battery you have remaining in the field will help you plan ahead - and stay more productive.

SXBlue Platinum Packages

As there are many different options and combination of features, the SXBlue Platinum currently comes in 3 customizable packages: Basic, RTK, and Survey. “Basic” is pretty much what you would expect, it’s a bare-bones setup with very few features enabled (GPS L1). You still have SBAS, and you’ll definitely want to use it - although this package seems to be lacking many desired features.

The RTK and Survey packages are where the hardware is really leveraged. With both packages all Sat. systems are unlocked (GPS, GLONASS, BeiDou, Galileo, QZSS), multi-frequencies are unlocked as well (L1,L2,L5; G1,G2; B1,B2; E1BC,E5A,E5B; QL1,QL2), and RTK available. It even comes with a rugged carrying case to protect your fancy new gear. I do wish these higher end packages also came with 10Hz by default (like the Arrow Gold and Garmin Glo) - but I guess it saves a few bucks if you don’t need it.

SXBlue Packages

So what differentiates the RTK and Survey package? The difference is in the antenna. The Survey package comes with a “survey antenna” which is bigger, more sensitive, and has a Gain of 39dB. The regular antenna is a bit smaller with a Gain of 28dB. In short, it’s a better/upgraded antenna that allows it to track weaker signals - potentially giving it more satellites in view for a longer period of time.

Weather Watching

ArcGIS Collector and a few other apps are on the agenda for the SXBlue Platinum. Once the snow and cold temps have passed I will get outside to do some field testing. Truth be told I don’t do field work during NFL playoff games either.

Multiple Field Key to Single Field Key for Relates using ArcPy

Thu, 04 Jan 2018 20:52:33 GMT

An issue with multi-field unique keys (composite key) is that we can’t use them to setup a join or relate between objects (Feature Classes or Tables) in ArcMap. In this article we will use Python and the ArcPy module to update a unique Multiple Field Key to Single Field Key for Relates using ArcPy.

Multiple Field Key to Single Field Key for Relates using ArcPy?

Python scripting isn’t the only way to do this as there are some OOTB tool combinations that can give us the same result. As well, there are some ArcMap/Pro tricks and database objects that can all be potential solutions. But Python is fun, fast, and can make quick work regardless of the data format used.

The Scenario

For this scenario lets assume we have a Feature Class (FC) and a Table with a 1:M relationship. The 4 fields that make each record unique in the parent FC are: F1, F2, F3, F4. These four fields also exist in the related table, although there can be zero to many related records here. Please note, in this scenario it is assumed the 4 field combination in the parent FC is unique with only once instance of the combination. If there is a chance your data has some duplicates, you can use the Summary Analysis tool with the case fields option to do some QC before you begin. This solution works on most editable formats including File/SDE Geodatabases and Shapefiles.

Multi-field relationship diagram

There are two ways we can create the unique IDs for the objects:

Use the ObjectID field as the unique key on the parent FC, and add a new field to related table and populate with the same IDs.
Add a new field to both tables, allowing the field names to be identical. Then use a number sequence or the ObjectID to populate the new fields. We will use this approach as it gives more flexibility if needed.

And let’s try to do it in 30 lines of code or less.

The Approach

Since we are updating 2 objects at the same time we will need to use the ArcPy Editor class. If you’ve done simple ArcPy insert/update/delete statements before you may not have used this class - but it should be considered even if not required as it allows use of edit sessions and operations to manage transactions.

The Editor class can be used to start and stop edit sessions and edit operations for file, personal, and enterprise geodatabases and shapefiles. The Editor class can be used to start an edit session with versioned or nonversioned datasets. ~ Esri Online Help

Using the Editor class is pretty straight forward - pass the edit workspace, and start editing:

edit = arcpy.da.Editor(edit_ws)
edit.startEditing(False, False)

Once we are editing, we will use a cursor to update each of the parent FC’s linkID based on the ObjectID. Again, we could just use the ObjectID directly in this case, but the example also shows how to update 2 objects at the same time. As we pass over each parent record we will jump to the related table to find all matching items and update with the same linkID. The field LinkID is assumed to already exist with a Long Integer field type.

To loop and update the parent FC, we start with an UpdateCursor:

with arcpy.da.UpdateCursor(xs_fc, xs_att) as cursor:
  for row in cursor:
    #FYI: row[0] is LinkID field, row[5] is the ObjectID
    row[0] = row[5]
    cursor.updateRow(row)

Now that the LinkID field has been updated with the current ObjectID in the parent, we need to find all of the features with the same multi-field combination in the related table to update with the same LinkID. For this, we create a cursor on the related table passing in an optional query parameter so we only loop the records matching the parent attributes (F1,F2,F3,F4). The code has this section separated into a function for readability purposes since it looks very similar to the parent FC cursor snippet. The query parameter is the part of the code making sure we have the multi-field relationship matching between the two objects.

xs_tbl_qry = xs_att[1] + " = '{0}' AND ".format(row[1]) 
  + xs_att[2] + " = '{0}' AND ".format(row[2]) 
  + xs_att[3] + " = '{0}' AND ".format(row[3]) 
  + xs_att[4] + " = {0}".format(row[4])

Pseudo query from above: F1 = val AND F2 = val AND F3 = val AND F4 = val.

Once we’ve looped through all parent/child relationships, we just need to save and stop our edit session to commit the changes.

The Code

If we remove all the commenting lines, this only takes 19 lines of code, so we did stay in our 30 line max. The workspace is set to a File Geodatabase, however this also works on SDE Geodatabases and Shapefile/DBF workspaces.

Other Options You Say?

Not interested in using Python? Below are some additional approaches that can also work:

Create a new field and concatenate the multi-fields into a big string: This solution works most of the time, but joining based on a long text based field isn’t desired nor efficient. If using a Shapefile/DBF with lots of rows, this can also bloat the size of the objects, possibly even hitting the 2GB limit. In any case, when updates are made to either table in the relationship, the fields will need to be recalculated (could automate very easily).
Use database views or Query tables to create a virtual field on both tables. This is very similar to option 1 but is dynamically updated. Database views are server side while query tables are client side and not available to all users. Although query tables also work on non RDBMS formats.
Use ModelBuilder. There are numerous ways this can be accomplished, for example: using option #1 above as the temporary join, then swapping out with a unique number ID. The unique number can be generated using the Frequency Tool or Summary Statistics Tool. Then create the temp concatenate field on the 1:M objects, and the output from the stats tool. Use the concate field to temporarily join, then update the final relate ID field using the OID from Stats. Delete the temp concatenate field when done. Lots of issues with this approach since you will need a schema lock on the objects, as well as permissions to add/delete fields (fine for local files, not so much for database users).

ArcMap Layers to LYR Files with Python

Mon, 30 Oct 2017 22:03:10 GMT

A tiny Python script to show how to export ArcMap Layers to LYR Files with Python. Although it’s a short script, it shows how easy it is to use Python with ArcPy to jump into ArcMap layer properties. To add some variation, we will only export layers that sit at the root level of the map’s table of contents. The child layers are still exported inside their parent LYR file.

Requirements

Pretty basic requirements for this task and based on using almost any version of ArcGIS 10.x (tested on 10.3.1 and 10.5). The goal is to export LYR files by looping through the table of contents. We need the Python OS module for easy access to some path options - this isn’t required, but makes life easy. Second, the ArcPy module is required to read the Map document (MXD) contents.

As mentioned earlier, we could export every layer, including all child layers. Most of the time it seems like people are looking to accomplish one of the following:

Export just the Layers that have data, and none of the group layers. This means we would export single layers, or the deepest children of group layers (could have a group of groups with layers, etc). Small changes to the code below would be required for this approach.
Export just the layers at the root level of the table of contents. This could be a single layer of data, or a group layer. Saved group layers do include the details about the children layers. This is the requirement and approach that will be used in this code example.

How can we determine if we are at a root level? Line 17 in the code below checks if the layer.name value is equal to the layer.longName value. If they are equal, it’s a root level layer (group or stand-alone).

Long Name Property: This property is valuable when trying to determine whether a layer belongs to a group layer. If a layer does not belong to a group layer, the long name will equal the layer name. If a layer does belong to a group layer, the group layer structure will be included in the long name. For example, the name of a layer nested inside a group layer within another group layer may look something like Group1\Group2\LayerName. All layer types support this property. ~ Esri help on the longName property.

The Code: ArcMap Layers to LYR Files with Python

Commenting inside the code sample will help explain most of of the details. One of the optional pieces included is to find and replace some text characters. In the Windows OS, there are a list of illegal characters that can’t be used in file or folder names. Since the LYR file name is set to be the same as the Layer Name in the Map, best to trap some of these in advance (see line 18 and 19 for some basic find/replace examples).

Happy coding.

Secure Map Service to Shapefile with Python Part 3

Wed, 23 Aug 2017 16:06:44 GMT

This post is inspired by the many emails I’ve received on the first two posts in this series. This article will address the most most popular question so far: How to work with a secure map service to Shapefile. Short answer, tokens!

Before We Begin

We aren’t covering Part 1 or Part 2 again, but we will continue to build on those concepts. Secure services use the same approach as before with one additional parameter that passes a security token with each request. So once we have the token, the original posts can be used for the rest of the plumbing.

What Are You Token About?

When ArcGIS web services are secured using ArcGIS token-based authentication, every request must be accompanied by a valid token. A token can be acquired in one of two ways:

Using the ArcGIS Server token request page: This method is very straight forward and requires very little change to your Python script. Just visit the token request page (http://gisserver.domain.com:6080/arcgis/tokens), add your details, and generate a token. I know, not very Pythonic. Even if you decide to use the programmatic approach, this page is very useful for testing and verifying the username password is valid.

Programmatically: Send an HTTP(s) request to the token endpoint, and receive the token back inside your code/script (Python in our case). But there are some limitations to this method - the username and password can’t be passed as query parameters directly in the URL as we do in every example in Part 1 and Part 2 with GET requests. Instead, parameters are passed in the body of a POST request. It is also recommended to use HTTPS when sending this information whenever possible.

Once the token is received using an above method, we can modify the parameters to include the token in each request.

Programmatically Request a Token From ArcGIS Server

To authenticate with ArcGIS Server we need to submit a name and password that has permissions to the service/folder. The server will return a token which must be include with all subsequent service requests we make to the server.

The token does not last forever; it is designed to time out so it cannot be stolen and used indefinitely by a malicious user. You have to request a new token each time you run your script (but not each time you make a request). ~ Esri Server Help

This relatively simple task does unfortunately require more lines of code (and modules) than one would expect when using Python 2x. There are lots of existing resources out there if you want to use urllib with either urllib2 or httplib modules to perform this task. However for this example we are going to use the requests module (manual pip install is required). Why use this external module? It only needs a few lines of code, and with a tag line of “Python HTTP for Humans”, how could you not want to use it?! To install on Windows 10, just open PowerShell and enter: c:\Python27\python.exe -m pip install requests

Once installed, requesting the token is pretty straight forward…

With this token now saved as the variable myToken, we can make service requests until the token expires. Since we didn’t specify the time in our request, it will be whatever default is set by ArcGIS Server (typical is 60 minutes). Now whenever making a queries to map services, just add an extra parameter to the service requests used in the previous articles. The order of the parameters is up to you.

Original request without token: https://domain/arcgis/rest/services/Public/Directory/MapServer/0/query?where=...
New request with token added: https://domain/arcgis/rest/services/Public/Directory/MapServer/0/query?token=myToken&where=...

The new requests will be validated by ArcGIS Server and we can continue to download the information as we did with open services. As a side note, this example of generating a token uses the ‘Requesting IP’. This also means the generated token can’t be shared on multiple computers. As well, if there is a proxy server between the client and ArcGIS Server, tokens generated using the IP Address and IP address of this request’s origin options must be bound to the IP address of the proxy server. If there are multiple load balancing proxy servers between the client and ArcGIS Server, the HTTP Referer option should be used [ArcGIS Server Help].

Secure Map Service to Shapefile with Python

That’s really all we need to access a secure map service. A valid token was the only missing piece of the puzzle. Although if trying to access AGOL, it’s easier to sign in, and download via the GUI.

Not all ArcGIS Server instances are configured the same unfortunately. There are some ArcGIS Server installs where the Generate Token request is unavailable. This is due to security concerns (HTTP vs HTTPS), or the lack of requirement to make it available. Some may also point out that it is possible to request a token via HTTP using GET. This is technically possible, although thankfully not enabled by default - since using a GET request may seem easier but credentials are provided as part of the URL and may be stored in browser history or in network components. Best to use POST method regardless.

About This Series

This is the third post in a series of how to extract data from an ArcGIS Server map services to a local shapefile FC or Geodatabase FC. The first post “Extract a Map Service Layer to Shapefile using Python” focused on the basics: query, save as JSON, convert to Shapefile. The second post, ”Map Service to Shapefile with Python Part 2 – Iteration” dealt with larger datasets, when the results were bigger than a single query would allow.

Additional Resources

Python.org - Requests module for Python 2.7
ArcGIS Server Help - Scripting with ArcGIS REST API (admin tokens)
ArcGIS Server Help - Acquiring Tokens (using the GUI)

Collector for ArcGIS update

Sun, 11 Jun 2017 14:21:07 GMT

Collector for ArcGIS has its first feature update in 2017. The list of new features includes a few community request items we should all be excited for.

What’s New in Collector for ArcGIS

This release is known as 17.0.1 (all platforms) and 17.0.2 (Android only added bug fixes), includes:

Rename Attachments: A very popular request and a great addition to Collector for ArcGIS. You can now add the attachment (photo, video, etc.) and then change the generic attachment name to your liking.
GPS Averaging: Having a small issue with the name of this new feature (GNSS v GPS), but that aside, another great new feature. When you want to average the point location over time from multiple readings. This could be for high-accuracy updates, or even to help when using a mid-range GNSS. You can turn on this feature along with setting the number of points required to average. The location isn’t stored until it averages based on that many points. Works for points and vertices of lines/polygons. Note: You can’t use streaming collection and averaging at the same time (mutually exclusive).
- Use the setting menu to enable Averaging. Then when you start a collection, you’ll see the header change to show “averaging x of y” and the current averaged accuracy. The dot on the map will turn red during the averaging process as another form of communication.
Use Basemaps from an SD card (Android only): No space on your tablet for a large basemap? For Android users, this update will drastically expand the space available for large basemaps. There is lots of potential here as you could have a small collection of various basemaps users could swap between, even when in offline mode. Since no Apple product has SD card slots these days (or any ports really), you are slot-out-of-luck.
Continuous Collect Improvements: Copying an existing feature or continuous collection didn’t work in all scenarios. The most noticeable was when the layer participated in a relationship class. These restrictions are now removed as this is now supported on all layers without relationships and on the parent layer when relationships are present. Side Note: Not sure if this is a feature or a bug fix - either way, I’ll take it!
Layer Refresh Support: Another new feature/bug fix is with auto layer refresh (online only). If the feature layer has a refresh interval set, Collector will now…. You guessed it - refresh!

Bug Fixes

A few bug fixes also made it into this release. There is also a small 17.0.2 release for Android that fixes a few additional items that got through the cracks during release. Esri lists a few of the bigger ones which including some limitations with PostgreSQL databases.

Android (17.0.1): BUG-000093003, BUG-000103220, BUG-000101786
Android (17.0.2): BUG-000105721, BUG-000105749
iOS: BUG-000101784, BUG-000104278
Windows: BUG-000101636

So what are you waiting for, go download the latest 37MB (iOS) release today and get back out in the field!

Web GIS metrics and patterns with Google Analytics

Thu, 27 Apr 2017 23:02:56 GMT

Scouring through some blogs the other day I came across an article from Esri regarding the “Most frequently used Web AppBuilder for ArcGIS widgets in publicly available apps”. Web GIS metrics aren’t discussed very often, so definitely worth a read. It’s actually a review of 10,000 public apps and most popular widgets in use. I would typically read the article, make a mental note or two, and move on – but this article led me to ask myself a few questions: How do the tools we add to the apps relate to web GIS metrics/analytics? What tools are used by end-users the most?

The Esri article breaks down the widgets by type (off-panel, in-panel) and also mentioning themes and default widgets that have an impact on the statistics. Beyond the defaults already built into a theme, widgets are added by AGOL administrators (or via JavaScript development) for a variety of reasons.

Now that we have functionality added, how can we track usage and patterns? In this case, Google Analytics to the rescue! After a quick disclaimer of course: This is based on one web application over a sample time period that tracks some, but not all functionality. So take the rest of this article and metrics with a grain of salt.

Web GIS Metrics – The application du jour

To review metrics, we needed access to an application that collects… metrics. Thankfully, the organization I work for has developed a custom (2 guys with Notepad++) web GIS application with lots of tracking. This application was built after reviewing many existing functions provided by widgets, our existing (older) web-GIS applications, many municipal web GIS applications, and discussed functionality/usability with staff (end users). The application is also available to the public but some functionality is removed to keep it a little more streamlined.

GRCA’s Web-GIS Interface

This is a “web-GIS”, not a “focused web map”. We have both types of applications at our organization, but the main web-GIS is easier to analyze a wider range of functionality. I should also point out that since staff use the application for specific purposes (public as well), metrics will be skewed to this use. Lastly, application design can impact user interactions as the layout will help (or hinder) the ability of the end-user to navigate functionality.

Web GIS Metrics – How we track stuff

We leverage Google Analytics (GA) to collect metrics. This also means we have 3 levels of event tracking available. For example, whenever an end-user interacts with the application, we store 3 pieces of data: draw a polygon while using the Planning theme. The “Planning Theme”, “Draw Tools” category, and “Draw Polygon” tool info is stored. This allows us to review not only the functionality used, but the type of use, and context (theme).

This isn’t to say we track every user interaction. Clicking on the map will automatically identify operational layers as set in the theme configuration. The general identify tracking is disabled since it happens so frequently the extra metrics (REST calls) annoyed some firewall appliance settings (for now). However the user can override the pre-set identify settings to focus on any one layer – this specific info is tracked in GA. Table of Contents (TOC) tracking is also limited to specialized events (Changing themes, viewing metadata, etc.) and doesn’t track the toggle of layers on/off.

Web GIS Metrics – Functionality

Statistics are sneaky. And like maps, they can be misused and/or misinterpreted, and not saying I’m going to get it all right here either. GA Events are updated each time users interact with related functionality. So if a user draws 3 lines, and only prints the map once – Draw technically has more usage, even if more 2 users print. With GA, we can look at events in many ways: Total events for all sessions, session where the event occurred at least once, events by category and type, etc. Let’s take a look at event totals and event occurrence within session for a small sample time period with 13,168 user sessions and 46,396 user events.

Google Analytics of Event types by Count and by Percent of Sessions for a small sample time period

Within this sample, Draw tools (draw lines, polys, callouts, text, points) have the highest number of total events (14,536) [left chart]. The Search tool (big search box) has the second highest number of events; however this metric alone can be misleading. 90% of all user sessions search at least once, while only half of user sessions (46%) actually draw something [right chart]. The users that do Draw, add lots of graphics/text in one session. The print tool has a similar pattern: it has half the number of total events compared to draw, but is used in more total sessions.

This makes sense: We search and find area of interest (hopefully on the first try), possibly sketch graphics on the map, print the map once and close. And if you did sketch something you took your time and added lots of details (graphics). Side note: time on the app is also tracked, >12 minutes on average if drawing.

The assumed workflow above is confirmed with Google Analytics Event Flow report. This allows us to review the most popular first event by session. For both public and staff, the most popular first event is Search (note: pan/zoom not tracked). The second event for Public users is typically to view the legend, while Internal Staff second event will vary more widely (since they see these layers every day and don’t require the legend as often).

Web GIS Metrics – A few extra details

This isn’t to say low metrics of a tool is bad. The “About” events provide information to those that need it (corporate info, training, disclaimers). Hopefully this info isn’t needed frequently, but it should still be available. So there might be some functionality you include without high usage intentions.

I should mention that the Esri article statistics and the ones described above can’t really be compared. The Esri article reviewed the number of apps with functionality added, not the usage within the app. And our web-GIS is just one implementation based on certain use cases. What I can say: Search and Print are the two of the most important tools in our web-GIS relatively speaking – not by total clicks, but used by the most people.

Even though the original questions have been answered, this is by no means close to describing what is possible with Google Analytics – more the tip of the iceberg.

ArcGIS Javascript API 3.20

Mon, 20 Mar 2017 16:42:20 GMT

Phew. The long hiatus of posting updates is over. As expected, the world continued to move on without me. This includes an update to Esri’s ArcGIS Javascript API 3.20.

ArcGIS JavaScript API 3.20 is Here

Not too much in this release, although I always appreciate bug fixes! But let’s start with the unique alien updates.

Starting at ArcGIS JavaScript API 3.20 there are Planetary measurements and scale bars. You read correctly. I can’t find a reference on the ideas page to see how many up-votes this had, but I guess there must be some demand. That’s right, you can now use many “non-Earth” projections including: Moon, Mars, Venus.

On to more earthly updates - the directions widget now supports barriers, the Hindi locale is now supported, along with a few API updates/enhancements. Of note, the Dojo v1.12.1 package is now included which has some nice browser support updates as well.

That’s It?

That’s all folks. To be honest, I’m a fan of the faster release schedules so I don’t take this release as a bad thing. Any time we can get a few minor updates and bug fixes, count me in! So, here’s hoping Esri keeps up with the fast release schedules and get ready for 3.21 in the coming months.

ArcGIS JavaScript API 3.19

Thu, 22 Dec 2016 13:40:31 GMT

As we are all waiting in line at the mall to buy some final holiday gifts, ArcGIS JavaScript API 3.19 and 4.2 were released so we could start updating our code while sipping back some rum and eggnog (eggnog is optional).

What’s New in ArcGIS JavaScript API 3.19

A few of the bigger changes include more support for vector tiles and the additional of Arcade expression language.

Vector Tiles are now supported in all ArcGIS-supported spatial references. This addition will help more of us get starting using the vector tile approach as it seems to be the future of basemaps. One downside - you still can’t print the vector tiles basemaps.

Arcade is Esri’s new, lightweight, and secure expression language. Arcade isn’t just limited to JSAPI, and will also work with ArcGIS Pro, ArcGIS Online, and the ArcGIS Runtime APIs. For now, the focus is on expressions for visualisation and labelling but it also allows for other expressions and geometry functions. The initial release only includes basic geometry functions (create and reference). In future releases, other geometry functions should be added to help calculate areas and lengths, and perform simple overlay operations within an expression - think Arcade as a light weight ArcPy Geometry object.

Changes to ArcGIS JavaScript API 3.19

Geo-location has been enhanced so you don’t get caught in the browser wars with HTTP. To helps developers still using HTTP, the JSAPI no longer displays the Locate Button for non-secure web apps in any web browser. Most of the recent browser updates now block the Geo-location API for HTTP, so this update will help keep us all away from making an app that isn’t all-browser friendly.

While on the browser topic, the list of supported browsers was also changed to state “IE9+” instead of “IE7”. If you still need IE7 this is the least of your worries.

Both JavaScript API 3.19 and 4.2 are now available from Esri’s CDN and on GitHub.

iOS ArcGIS Collector 10.4 with the Bad Elf GNSS Surveyor

Sat, 05 Nov 2016 13:22:15 GMT

I’ve wanted to try ArcGIS Collector with the Bad Elf series of devices to see how it stacks up against some other external GPS/GNSS. I’ve read some great reviews, but needed to try it with Collector to see for myself. Well, send Larry from Bad Elf to the rescue. His confidence in the Bad Elf devices is very high, so much so, he sent me a Bad Elf GNSS Surveyor to test out. And now - I understand where his confidence comes from.

Update 2016-11-08: GNSS/GPS hardware pricing corrections.

Setting up the Bad Elf GNSS Surveyor

The setup was straightforward on the iOS devices I tested with: iPad, iPad w/ 3G, iPhone 6. You can pretty much hook up the Bad Elf in the same way you would connect most other external GNSS with Collector:

Setup the bluetooth pairing (iOS settings > Bluetooth)
Open the ArcGIS Collector 10.4 App
Go to the settings screen and select Providers (Action button > Settings > Providers)
Add/Select the Bad Elf provider
Collect

ArcGIS Collector Provider Settings

Collector Providers aren’t all the same

As you are probably aware, you also have the option to use the ‘Integrated Receiver’ provider in Collector on iOS devices. You can actually use this option with the Bad Elf as well but it isn’t recommended (not recommended for any external GPS/GNSS for that matter).

Collector Bad Elf GNSS vs. Integrated Provider on the trail (medium canopy)

Why not you ask? The Integrated Receiver is really just letting iOS Location Services manage the location profile. iOS Location Services can communicate with WIFI, Cellular, internal GPS, and external GPS devices in order to determine the best location signal to use. Sounds great - but there are two main issues when using this approach with ArcGIS Collector:

The details of Collector’s Location Accuracy window will currently max out at 5m for iOS Location Services. The actual accuracy might be better, but Collector will still show 5m in the Location Accuracy Window. Although if using a Bad Elf GNSS device you can still check the LED screen to verify the accuracy.
You don’t actually know which input signal is being used (GPS? Cellular? Other?): If you purchase an external GNSS, you probably want to know if it’s being used. I had a Glo battery drain and the only way I noticed was the Collector Location Accuracy window was hovering around 8.5m to 10m under canopy.

Collector Bad Elf GNSS on the trail (dense canopy)

Bad Elf, please stand up

The Bad Elf GNSS Surveyor really does stand out from the crowd. The LED screen provides a huge amount of information on accuracy, number of satellites, Bluetooth status, and even allows for data logging on the device. I didn’t think I would use the LED screen very much since I use Collector, but it was great for quickly verifying battery life, and accuracy (and didn’t have one on my last GNSS device). That said, I will be hard pressed to go screenless again - as any questionable readings I see in Collector can easily be verified by the device itself.

A subtle yet refreshing addition is the Bad Elf App for iOS (available for free in the App Store). It has a clean design with the same information you can see on the LED screen, and has the ability to easily change device settings and perform firmware updates - checks when they are available and deploys the updates with the click of a button. This means no desktop PC is needed for changing settings, or providing upgrades.

Accuracy and Price

When it comes to using ArcGIS Collector and high(er)-accuracy GNSS, there are way too many factors to get into for one article. The 1cm to 50cm devices are for very specialized use cases and definitely not covered here. Overall the level of accuracy required will vary by required use. Price is also a factor for many, and there is definitely an Accuracy-to-Cost ratio that needs to be taken into consideration (along with RTX services and fees).

Bad Elf GNSS Suveyor

The Bad Elf GNSS Surveyor is in a very unique position when looking at the Accuracy-to-price ratio. A stated “1.0m Stationary Accuracy” (real-time) will cost $599.99 USD, it’s hard to compare to other devices as there aren’t many ~1m devices on the market. None of the other big players come close to this price in the meter/submeter zone (50cm - 1.5m): $1895.00 USD (iSXBlue II+ GPS), 1995.99 USD (Arrow Lite), $2495.00 USD (Trimble R1), $2895.00 (iSXBlue II+ GPS). This all said, each device has its niche and specific features that can definitely justify different price points (external mount antenna, replaceable batteries, accessories, RTX add-ons, etc). Some of the more expensive devices can also be upgraded to 10cm, or even 1cm, so please take these comparisons with a grain of salt.

I realize the cost is more significant when compared to the likes of Garmin Glo’s and DUAL’s, but as mentioned earlier, these devices aren’t really in the same competitive space - there are actually a few additional Bad Elf products (GPS Pro, GPS Pro+) that are much closer in accuracy/price if doing comparisons in the 2.5m-4.0m range. The good news is you still get the iOS app, LED screen, rugged case, and water-resistant features of the higher end Surveyor on the lower end devices too.

Summary

The Bad Elf GNSS Surveyor with ArcGIS Collector make a great team, and considering the level of accuracy and features: a good price. Collector and Bad Elf have a great thing going here and I look forward to seeing what the future may hold for this dynamic duo.

Bad Elf GNSS Suveyor - GNSS Specs:

~1 meter stationary accuracy with SBAS+PPP
2.5 meter accuracy while in motion
Up to 10Hz position update rate
Fast GPS lock times (45 seconds or less)
56-channel GPS/GLONASS/QZSS receiver
RTCM support
SBAS/WAAS/EGNOS/MSAS
Barometer for barometric altitude
Bright yellow enclosure to find easily outdoors

Bad Elf GNSS Suveyor - Device Specs:

IPX4 splash resistance
Barometric Altimeter
Access data logs directly from PC or Mac (USB)
Post-processing support via RINEX conversion available in the Bad Elf app
No internet connection or monthly subscription required
Bluetooth 2.1 technology + EDR
- Range of 33ft (10m)
- Support for 5 Simultaneous Connections
Battery
- Up to 28 hours bluetooth connected, streaming GPS data, while data logging
- Up to 32 hours while data logging

Additional Links:

ArcGIS Server 10.5 is now ArcGIS Enterprise 10.5

Wed, 26 Oct 2016 15:37:03 GMT

If you haven’t heard, Esri found out we finally understand the current naming conventions and licencing models for their server products. Naturally, this means it’s time for a name and licence change. Don’t be too hard on them, many didn’t like the existing names either: is it ‘ArcGIS for Server’ or ‘ArcGIS Server’? And sooo many different server products, all dependent on each other yet separate products at the same time. Arrgghhh. This is what Esri is hoping to fix with the new branding (assumption).

So let’s not dwell on what could have been and the latest ArcMarketing terms du jour. Instead let’s focus on what this change means to GIS administrators.

What is ArcGIS Enterprise?

The Enterprise is an interstellar spacecraft powered by warp-drive engines that… oh crap, sorry about that - ArcGIS Enterprise is the entire server stack of Esri software ‘components’ for your organizations’ server environment. This includes:

ArcGIS Server
Portal for ArcGIS
ArcGIS Data Store
ArcGIS Web Adaptor

There are lots of enhancements to many of the software components, such as Portal-to-Portal sharing, and new HTTPS defaults. The official list will be released soon but you can checkout the draft Esri 10.5 help pages to get an idea of some of the bigger changes.

So overall, the change doesn’t sound too crazy so far, since ArcGIS [for] Server really came with this product list before. However ArcGIS Enterprise will require new 10.5 authorization numbers as previous authorizations (10.1-10.4x) will not migrate like they did in the past. And as for licencing, this is where it gets fun!

Licencing Roles and Extensions

System capacity and capabilities are licenced ”by CPU cores according to one or more server roles” ~Esri docs. In layman’s terms, each ArcGIS Server Role has one or more licence options available. So you install the ArcGIS Server component and licence it as one of the following:

ArcGIS GIS Server: Still has editions - Basic/Standard/Advanced
ArcGIS Image Server
ArcGIS GeoAnalytics Server
ArcGIS GeoEvent Server
ArcGIS Business Analyst Server

From this list, it’s pretty clear licencing is still based on functionality/use, and we get a few new things to laugh at: “ArcGIS GIS Server” is redundantly redundant. And the words ‘ArcGIS’ and ‘Server’ are used to describe every licence which contradicts the overall name change. Sorry, I couldn’t help myself. To be honest, it doesn’t matter what the name is, the focus is the functionality/purpose ;). Shakespeare said it best [What’s in a name].

So, the software is upgraded, and the software components you are familiar with from 10.1-10.4 are still there (bigger change is licence lingo), and we licence by functionality. Oh, and yes there are still extensions (Schematics, Data Reviewer, INSPIRE, Workflow, Defense, etc). Some extensions are included, some can be purchased - depending on the Standard/Advanced Edition (extensions not available for Basic).

Should I upgrade?

So nothing has changed, but everything has changed - this doesn’t mean you should be worried about upgrading as you will be given the appropriate licences based on your current maintenance/setup. I usually recommend waiting for the first service pack before moving a production environment - but we do need a few early adopters to find those bugs for the rest of us!

For smaller shops with one licence of AGS Standard (and maybe Portal), it really is just a new authorization with little changing. Larger implementations will see focus on how the various ‘roles’ should be implemented in determining licence needs if you want to change where components are currently installed as part of the upgrade - Multi-server setups by Role are recommended. Either way, we need to update our server diagrams and PowerPoint slides.

Release Date

There is no public/official release date as of yet, but the pre-release is currently available from myEsri. If you are eagerly waiting the final cut, it is anticipated to be released before the end of the calendar year.

Other Names on the Shortlist

Here are a list of names on my 15-second brainstormed shortlist Esri can use for the next name change:

“ArcIMS Pro”
“ArcGIS 4k Neo”
“ArcInternet”
“ArcGIS Online Locally”

Supporting links

Map Service to Shapefile with Python Part 2 - Iteration

Thu, 29 Sep 2016 20:49:15 GMT

This post builds on the previous Extract a Map Service Layer to Shapefile using Python post. It seems as though many are interested in getting past the pesky limits ArcGIS Server has on querying/extracting data to a shapefile. The initial post does mention this limitation, which is typically 1000 or 5000 features by default depending on the version (and configurable by admins). Part 2 will focus on getting past that limitation to all the juicy goodness… errr, all the data.

Note 2016-10-31: Sample code updated to sort objectIDs before creating groups.

Before We Begin

It might be a good idea to refresh on the details of part 1 as we aren’t covering the same pieces again. And before we go too far, the same disclaimers apply: Make sure you have permission/rights; use this code for good; Don’t bombard servers with your requests; Donate to SpatialTimes (worth a try).

Map Service to Shapefile with Python, No Limits

The first thing to check is the maximum query results allowed by ArcGIS Server. Since this can be configured by the Admin, it might be worth performing a manual query to see what the limits are. For this example, I’m assuming the typical Max Results = 1000 (most Admins don’t bother changing the default).

Get All ObjectIDs, Yes All of Them

Once the maximum is set, we can extract all of the ObjectIDs from the layer and begin to divide this into chunks of 1000 (or other value). Although the maximum results are set, the &returnIdsOnly=true parameter doesn’t have this same restriction. You can return all IDs regardless of how many there are. This result sets up the rest of our queries since you can’t assume ObjectIDs are sequential with no gaps. Update 2016-10-31: The IDsJSON variable originally assumed the IDs were in order which isn’t always the case (added the sorted function).

IDsRequest = serviceURL + serviceMap + "/" + str(serviceLayerID) 
  + "/query?where=1%3D1&returnIdsOnly=true&f=pjson"
IDsResponse = urllib2.urlopen(IDsRequest)
IDsJSON = json.loads(IDsResponse.read())
IDsSorted = sorted(IDsJSON['objectIds'])

Divide and Conquer

Now that we have all the IDs, we need to turn these results into usable chunks, then setup our extraction queries - each chunk will run another query. A list will be created for the first 1000, then the next 1000, and the next. The last list will be any remainder (probably <1000).

args = [iter(AllOIDs)] * maxRequests
ListOfLists = ([e for e in t if e != None] for t in itertools.izip_longest(*args))

Putting it all Together

We are now ready to take the Map Service to Shapefile with Python. We have the Map Service, all IDs, and divided those IDs into n lists. To query each list, we just use a python loop and make the same request as in the original post.

for SingleList in GroupOfLists:
    Query the service with ObjectID query
    Make a shapefile

This will create a unique shapefile for each chunk/iteration. If you want to merge/append into a single output, you can add a few lines of code to the end, or just use the Merge Tool or Append tool.

Below is the full code with the only changes required to the URL, Service, and Layer ID variables.

ArcGIS JavaScript API 3.18

Mon, 26 Sep 2016 10:16:56 GMT

ArcGIS JavaScript API 3.18 was released by Esri last week. Great news for those still using the 3 series - and FYI, ArcGIS JavaScript API 4.1 was also release last week.

What’s New in ArcGIS JavaScript API 3.18

The biggest update in ArcGIS JavaScript API 3.18 looks to be an overhaul of the FeatureTable Widget. There are lots of new and improved features, including the long anticipated support for related records. Selection behaviour in the table has been improved, but if you have been leveraging this in the past you might want to test first, as there are some major changes here.

Another big update is for VectorTileLayer support. The new implementation is more baked-in to the API instead of seeming so isolated in the past. And if you need more browser compatibility, this update now also supports IE11 and Edge (a showstopper missing feature for many in the past).

With all these changes in place for VectorTileLayers, there is a small warning from Esri:

The internal implementation of vector tile layers in 3.x has changed. If you were using any of the undocumented APIs that were part of the previous implementation they will not work with the new version.

Minor changes

The usual small bug fixes, and enhancements to the Elevation Profile Widget, new Calcite theme (dark) are also included in this release. Not surprising, there are also bug fixes for the Measure Widget, which is still one of those tools that is surprising difficult to fully implement without hitting an issue. On the bright side, they keeping squishing bugs as we report them.

Overall, this release is a must for those who plan on using VectorTiles moving forward. That said, this is also setting up for VectorTiles to be supported in “any ArcGIS supported spatial reference”, which is a huge step forward to this being the new publishing standard (for Esri map services).

Collector for ArcGIS Windows Edition 10.4 AND it works

Wed, 07 Sep 2016 13:18:13 GMT

Phew! Collector for ArcGIS Windows Edition 10.4 is here… and it works!! That positive energy put forward in the related Collector 10.3 post seems to have paid off. The 10.4 version is now available for Windows 10 on the Windows Store. After updating the software, I was successful pairing Windows 10 with external receivers (without the Location API).

Collector for ArcGIS Windows Edition 10.4

This August, Esri released the Windows 10 and Android (new beta) versions of the popular Collector for ArcGIS application. Available from their related App stores, this update has been long awaited for allowing high accuracy GNSS devices to be connected. The real benefit of this update is allowing external receivers to use different connection types instead of relying on the default Location API (still available if you have a built-in GPS). This also removes the need for 3rd party hardware emulators when using the Windows version. The final version for Android is expected mid-September.

Collector Windows Settings

Compatible Receivers

To use a GNSS receiver with Collector, it must support NMEA 0183 sentance outputs. NMEA is the data specification Collector uses to communicate with the receivers. Most mid/high accuracy receivers already support the NMEA sentences that Collector uses, including most of the mid range receivers: SXBlue (setup NMEA sentance streams in config), Arrow, Garmin Glo, Dual XGPS, most Bad Elf devices.The NMEA sentances required/supported with Collector for ArcGIS can be found in the help guides (iOS, Windows).

Collector for ArcGIS (Windows) Receiver and Profile settings

Collector for ArcGIS 10.4.0 iOS

Thu, 04 Aug 2016 06:00:43 GMT

Collector for ArcGIS 10.4.0 iOS is now available in Apple’s App Store. This is a major update for those hoping to use high-accuracy GNSS with their iOS device in the field. If this isn’t enough of a reason to upgrade, or you are happy with your current Elf/Glo/Arrow/other, there are a few other enhancements which make this update a must.

One of the other additions in this release is the enhancements to attachments:

Preserve EXIF data on attachments (time, date, gps/compass info, etc)
Support for attaching additional file types (such as PDFs)

As for bugs, a few were fixed in this release for more niche users, along with more languages now supported in the application. Some of the bugs actually refer to the Windows version of Collector, which isn’t actually release yet but shows what can be expected in the Windows release.

Setting up Collector for ArcGIS 10.4.0 iOS with your high-accuracy GNSS

Collector for ArcGIS 10.4 settings page

You may notice the settings page for Collector for ArcGIS 10.4.0 iOS has also had some tweaks in order to accommodate more options. This is where you can configure your high-accuracy GNSS device.

Thankfully, this is a pretty painless process as long as you are aware of the details of your GNSS device and your Collector Map. Just add a new “Provider” (read/AKA: GNSS Bluetooth device), and a “Location Profile” (read/AKA: configuration/transformation from device-to-map setup). The Provider setup even allows for pole mount heights to be included directly without requiring any post processing.

The Location Profile’s list of Geographic and Projected coordinate systems is very extensive. Thankfully there is a search/filter function, and my favourite feature: EPSG numbers (thank you whomever put that one in)! Not exactly sure how Collector can handle so many potential transformations but looks very similar to a PROJ.4 or Proj4JS type of setup - and we know Esri has recommended this approach for JS developers in the past.

Collector for ArcGIS 10.4 Profile setup

Depending on where you are working the one issue that you might come across is that some grid-based transformations are not currently supported. If you are in Canada, the options are very limited for the NAD83 CSRS to other map projection. Some of the omissions aren’t because Esri doesn’t want them there - but based on what IP owners will allow to be included by commercial vendors - so let’s all work on making grid shift files more available while we are at it!

I hope in the future, Esri will list the supported/planned transformations as this could limit the use and/or value of having the high-accurate GNSS connected in the first place. That said, this is a great first step.

Another addition to hope for would be the listing in the online help of the grid-shift files that are used since this might help you decide which input<-> output projection to use based on what’s available. As having a survey-grade GNSS in the field, you want to keep as much detail as possible, and using the wrong transformation might set false expectations with staff.

As for working in the USA, this looks to be a great starting point for using with your high-end GNSS. The popular Geographic and Projected coordinate systems are all listed, much like working in ArcGIS desktop. You shouldn’t have a problem finding one that matches your GNSS or your map. Regardless of where you live, the best approach would be to have your map in the same coordinate system as your GNSS device so you don’t even need to worry about grid shift files and transformations.

Should I Update to Collector for ArcGIS 10.4.0 iOS?

Yes! If you are using a high-accuracy GNSS device, you have been waiting for this update. Even if you aren’t using a high-accuracy GNSS, it is more than worth the upgrade to keep EXIF information on your photos. The (listed) bug fixes are few, but there weren’t many big bugs to fix for the iOS version. Overall this is a great release, and the biggest drawback is that it’s currently only available for the iOS version. An update for both Android and Windows is expected to be released later in August.

Collector for ArcGIS Windows Edition 10.3

Mon, 01 Aug 2016 15:52:05 GMT

It sounds like a no-brainer: Use your Microsoft 10 deviced docked at the office, then disconnect, and your field ready with Collector for ArcGIS Windows Edition 10.3. For staff that frequent the field, yet have some office time here and there, a single device sounds like a dream come true. But sometimes dreams can lead to nightmares - which can be the case for some Windows + Collector users.

Collector for ArcGIS Windows Edition 10.3 Not Ready

At the time of writing, Collector for ArcGIS Windows Edition is at release 10.3 - and frankly, this isn’t ready for production use (IMHO). I’m remaining positive that the 10.4.0 version of Collector will fix these issues [what can I say, I’m a positive person].

The Nitty-Gritty

If you want to connect a Bluetooth GPS/GNSS to your Windows 10 device for use with Collector: It doesn’t work out-of-the-box. WTF. You can connect your device to Windows, issue is that Collector can’t read the signal from COM ports as it currently uses Location Services only.

The only scenarios where this actually works is if you have a built-in GPS/GNSS, or cellular based Surface tablet - forget about a vanilla Windows 10 device. HOWEVER, Esri does list a workaround that could work in some circumstances. The workaround does mean purchasing a 3rd party piece of software (free trial available) to get the job done as it can create a virtual connection from a COM port to the Sensor API (which can be leveraged by Location Services). You shouldn’t need to know this to use Collector, nor should I.

During testing, I wasn’t particularly happy with this additional step, but it boils down to a current issue with Windows and the Location API. Windows 10 doesn’t have the Location Service part of the software working as well as Android and iOS. The 3rd party software tries to bridge that gap, and there are cases online where people have been successful. Not so much in my case, but this might come down to how the W10 device has been setup and configured by IT as well - or the specific GPS/GNSS device. Read: It’s more complicated than it should be.

Chicken and the Egg

I’m sure Esri looks at this situation (which they are aware of) and cringes. Not many people use Windows devices with Collector so maybe it’s a lower priority?! - but then again, we can’t use it because it doesn’t work properly, keeping the numbers now. Regardless of where the issue truly lies: Microsoft bug; or Esri releasing even though you can’t use with Bluetooth GPS/GNSS devices - it will hopefully be addressed soon. Maybe this is why Collector 10.4 hasn’t been released on Windows yet? One can only hope 10.4 bypasses this issue by talking directly to Bluetooth devices (by creating their own workaround that is hidden/transparent to the user). If the 3rd party software currently recommended was written by a single individual using the Windows SDK, I’m hoping Esri’s team of Windows developers are allowed to use their skills to do the same and incorporate directly into the product.

Where do we go from here

For those still hoping for the single windows device, only time will tell. If 10.4 can alleviate the issues I think the Windows Edition of Collector has lots of potential moving forward and a huge user base that could leverage it - especially for large Microsoft based organizations (Government, NPOs, etc).

Use Bootstrap to Customize Esri's JavaScript InfoWindow Icons

Sat, 02 Jul 2016 22:08:52 GMT

I was recently working a small web mapping application and wanted to use Bootstrap to customize Esri’s JavaScript InfoWindow icons to match the style of the application. If you were unaware, this usually involves finding the sprite (image) file and making modifications or creating a new sprite. In this case, I was already using Bootstrap, so I wanted to use Glyphicons instead.

If you are interested in using the sprite approach, or just want to change the popup look/feel, there is a great blog article for your reference on Making Your Pop-up Pop.

Use Bootstrap to Customize Esri’s JavaScript InfoWindow Icons

This example assumes you are already using bootstrap in your application. If not, you can checkout the Bootstrap website for more information on how to get started. So now that we are ready to go, all we really need to do is turn off the sprite reference (image), and replace with our Glyphicon (HTML span). Of course you don’t need to use a Glyphicon for this work, the HTML <span> element can also refer to a font character, Font Awesome icon, or similar.

JQuery/JavaScript code inside the MapLoad{} section

This one line of code inside your MapLoad{} section uses the JQuery append technique to inject a span element to the Esri Popup’s close button. Create an ID for the span element so you can setup your CSS reference. Since this example uses a glyphicon, we also set the glyphicon of choice in the class attribute:

$(".esriPopupWrapper .titleButton.close").append( "<span id='SpatialTimes-info-close' class='glyphicon glyphicon-remove' aria-hidden='true'></span>");

CSS Updates

The CSS updates will hide the reference to the sprite and set the related Glyhicon styles. In this example, the Glyphicon is white, turning red on hover. The top:-3px setting may need to be tweaked based on the size of the Glyphicon or font being used.

.esriPopupWrapper .content .contentPane{
    background-color: #FFF; padding: 6px;
}
.esriPopupWrapper .titleButton.close{
    color: #FFF; opacity: 1; background: none; text-shadow: none;
}
.esriPopupWrapper .titleButton.close:hover {
    color: #ff0000; opacity: 1; background: none; text-shadow: none;
}
.esriPopupWrapper .titleButton.close .glyphicon{
    top: -3px;
}
#SpatialTimes-info-close {
    font-size: 14px; width: 100%; height: 100%;
}

Summary

This technique can be used to replace all sprites in the popup heading area if desired. Some of the CSS referenced can be changed to meet your style, font colour, and mouse hover needs.

Projecting between NAD83 and NAD83 CSRS in ArcGIS

Wed, 04 May 2016 23:31:39 GMT

Update 2020-11-04: Fixed a small, yet important, typo in the ‘Create transformation’ steps . Special thanks to Darren M. for catching this.

Update 2018-09-19: Please note that different Geographic systems need specific transformations. As of 2018, CSRS v6 and v7 are becoming very popular - however there is no specific transformations available at this time. If using ArcGIS 10.6x, the newer v6 and v7 Geographic systems are available - in older versions you need to create it. The Guru, Melita Kennedy’s quote: “NAD83 CSRS is a [ongoing] re-adjustment of the original NAD83 geographic coordinate system (datum). Generally, the differences between original NAD83 and the CSRS adjustment are less than a meter, so depending on your data, you may be able to forgo a conversion.” Wise words. So it’s important to think about your requirements, and to document in your metadata how you changed the data. If you are using high quality LiDAR or survey data you need to be careful about setting up the correct transformation (and still update the metadata on how you transformed!).

If you work with data in Canada, be careful when projecting data from one GCS to another (NAD83 and NAD83 CSRS). Why you ask? Even though many Canadian geographic transformations are supported in ArcGIS, the required files aren’t installed by default. Now, before we show up at Jack’s office in Redlands with pitchforks and torches, you should know one thing - it’s not their fault. Matter of fact: Esri, PCI, GeoMedia, QGIS, and other GIS software all have this exact same issue - the grid files are proprietary so the software companies traditionally haven’t been allowed to include them. Blame the pre-Open government era if we must.

I’ve subtly mentioned this before when discussing steps to upgrade ArcGIS. A frequent question I get from that post is “how do I set this up?”, usually followed by “why didn’t anyone tell us?“. To answer the first question, keep reading. As for the latter, hopefully you already knew - and if not, consider this post a rude awaking that your previous NAD83 to NAD83 CSRS transformations didn’t do what you thought (as the tools still “work” without any error messages or warnings). This is the case for many scenarios: Project tool (and raster project tool), ArcMap “on-the-fly” transformations via Data frame properties, and Mosaic dataset geographic transformation options to name a few of the big ones.

Enough background, how do we transform between NAD83 and NAD83 CSRS

Fine, not interested in the background. Let’s get to it then - although I’m still going to start generic. There are two things you should consider when using geographic transformations:

Is the grid installed: Again, if you are in Canada, the answer is most likely no. Educated guess since the grid folder for Canada doesn’t exist on a fresh install. If someone else installed the software, it’s a good idea to check anyway. Browse to \pedata\ntv2\canada. If the folder (all small case) doesn’t exist, create it (you will need it eventually). If it already exists, examine the list of NTv2 grid files (*.gsb) inside. If the one you need is there, move to step 2. If it’s not there you can search Google or just contact your Esri distributor as they know where to find the common ones.
Is the NTv2 Geographic transformation already setup/supported. Again, even if supported, if the GSB isn’t located in the directory in step 1 - you need this first. You can see the supported list for more details. If you have the grid file and your transformation and file are listed - congratulations, you are ready to go. However, for those unsupported (Ontario CSRS for example), you will need to create a custom transformation.

Argh, NAD83 and NAD83 CSRS custom transformation already

The supported list of NTv2 Geographic transformations does include a Canada wide transformation (NAD_1983_To_NAD_1983_CSRS_1). This is based on using the NAD83-98.gsb grid file. If this meets your needs, you are good to go - place the file into the “canada” directory, then select the transformation when prompted in the Project tool, ArcMap dataframe settings, etc. If you want to use the Ontario grid file (ON83CSv1.gsb) or Toronto grid file, we are almost there. Verify the grid is in the directory, and we are ready to create our custom transformation.

Create a custom transformation:

Open ArcToolbox > Data Management Tools > Projections and Transformations > Create Custom Geographic Transformation
Setup the input and output coordinate systems (this is a bi-directional process so order doesn’t matter)
Provide a name. No spaces, and you should follow the recommended syntax: Input_GCS_To_Output_GCS
For Method, select NTv2
In the parameters section, change the value to match the folder name followed by a forward slash (/) and the GSB file name. The example below uses the Ontario grid file: canada/ON83CSv1.gsb
Run the tool.

Custom Transformation Tool

Do I need to keep reading?

Nope, that’s it! With the geographic transformation and grid file in place, you can now select it from the transformation from the list. This can be used when projecting data, and even when ArcMap transforms on the fly! You will need to specifically tell ArcMap via the dataframe properties however (yup, on-the-fly projecting in ArcMap wasn’t doing it either). You will also notice that the Projection Tool and ArcMap data refreshing takes longer (since it’s actually doing something now).

Additional Links:

Any answer on support/geonet by Melita Kennedy. If anything in this post doesn’t align with something Melita said - Melita is correct.
http://support.esri.com/en/knowledgebase/techarticles/detail/21327
http://support.esri.com/en/knowledgebase/techarticles/detail/18317
http://support.esri.com/en/knowledgebase/techarticles/detail/35152
https://geonet.esri.com/thread/57508
http://gis.stackexchange.com/questions/127220/proper-transformation-file-to-go-from-nad83-csrs-utm-17-to-nad83-utm-17
https://www1.toronto.ca/City%20Of%20Toronto/Engineering%20and%20Construction%20Services/Standards%20and%20Specifications/Files/pdf/NAD%2083/nad_83_csrs.pdf

Extract a Map Service Layer to Shapefile using Python

Mon, 28 Mar 2016 10:22:53 GMT

How easy/hard is it to extract a Map Service layer to Shapefile using Python? Not as hard as you would think. Is it legal? Not as easy as you would think! And before I get into this any further, there are also ways of doing this directly in ArcMap - but that wouldn’t be very Pythonic now would it.

UPDATE: Part 2 is now available - Map Service to Shapefile with Python Part 2 – Iteration

Legal Schmegal (and a brief disclaimer)

I’m not a lawyer - but I have watched many TV shows where someone else played one. Oh, and I’ve actually read the Canadian Copyright Act (some of it sunk in), done a few presentations on the topic, and deal with the subject at work on a daily basis. Why do I bring this up? Just because we can do something, doesn’t mean we should. Please make sure you have permission from the owner of the Data/Map Service, or have permission via a License (“Licence” in Canada) before running around downloading shapefiles from the internet.

Oh, and if you are worried about someone downloading from your service: Best practice would be to secure the service, or at least limit the service functionality. Possibly adding a note into the service description on intended use could also help. But if you really want to keep your data secure, please refer to the helpful venn diagram below.

Venn Diagram by Rob Jewitt

Map Service to Shapefile using Python

Alright, now that the legal stuff is out of the way - let’s get down to some coding! We are essentially just going to wrap 3 small steps into a python file:

Query the web service layer for data
Save the results locally
Convert results to a shapefile (or other feature class type)

1 Query the web service layer

For this example we will download US States from Esri’s ArcGIS Server sample server. Once we have the service and layer, we can investigate the attributes and query options using the REST interface or just jump into Python. Either way, we are just going to call the same REST query inside Python using the urllib2 library. The trick here is to make sure the result format type is set to JSON.

myRequest = <URL>
response = urllib2.urlopen(myRequest)
myJSON = response.read()

2 Save the Results

Now that we have the data, we can save it locally as a text file with the JSON extension. Nothing fancy here just create a file object and save the results.

foo = open("jsonOutput.json", "wb")
foo.write(myJSON);
foo.close()

3 Convert the result to a shapefile

With the data successfully downloaded in a JSON file, all we need to do now is convert it to a shapefile. This task is made pretty simple seeing as there is a tool for that, JSON to features.

arcpy.JSONToFeatures_conversion("jsonOutput.json", "finalShapfile.shp")

What could possibly go wrong?!

This approach assumes the service you are connecting to allows query operations, otherwise you will get an error message immediately. Try the query operation directly from your browser first, then copy the parameters/syntax into Python for a quick sanity check.

Another thing with queries is that results are limited. There is a setting in ArcGIS Server to limit the number of features returned from a query operation (1000 by default). You may have noticed in the sample code, the _where _clause was set to where=1%3D1 (the %3D is web speak for equals). This is a simple true query that will return all features to the max results limit.

If more features exist or if you’re not sure how many features there are in the first place, just query it. Make a query and set the Result Option returnCountOnly=true. This returns the count so you can determine if one query will be enough. For the example with US States, there are only 51 features, so the 1=1 clause isn’t a problem. If the number was above the return limit, you would need to use an iterative loop until all features are returned. Don’t worry, guessing ObjectIDs starting at 1 isn’t required since you can send a returnIdsOnly=true query to get the actual list of ObjectIDs - and this query doesn’t have the max return limit applied. Just use the IDs to divide into chunks less than the limit, and loop away.

A final issue that can occur is that the output shapefile isn’t a shapefile - just a DBF. This occurs when a layer in a map service is published but the SHAPE field is hidden from the service. In this case, you still get all the attributes but no spatial component as the SHAPE field must be visible in the service for the geometry to be returned.

Map Service layer to Shapefile - Working Code

Here is the full python script. The URL is split into 3 parts only to help show the various components - a single URL parameter could be used instead. Please note: ArcPy is required for the JSON to Feature tool.

ArcGIS Collector 10.3.7

Wed, 09 Mar 2016 08:30:51 GMT

The new releases keep pouring out for the Esri Dev Summit. This time we get a very focused update for ArcGIS Collector 10.3.7. If you were hoping for GNSS receiver support, sorry to report it isn’t quite ready yet. But don’t fret, this update focuses on another major area: Offline editing.

ArcGIS Collector 10.3.7 - what’s new

“Advanced offline options” are the focus of this release. These new offline options are setup in ArcGIS Online, and improve download and sync operations in the client app. You can determine what information is retrieved from the web during editing - for both editable and read-only layers.

Collector Advanced Options

This update might not seem like a big deal, but it can have significant benefits when field staff want to sync days worth of photos and vector edits. Not only can syncing use large volumes of data (images), but watching the sync icon spin for minutes on end is time wasted in the field.

…your organization may want to limit the size and amount of data field workers retrieve from the server during initial download as well as subsequent sync operations. This can make it faster for field workers to share changes, and minimize data transfer costs. This is especially true when your field workers are in areas with low bandwidth. You can limit the types of data field workers retrieve from the server by adjusting the Advanced Offline Options in ArcGIS Online. These settings do not affect edits made by the field worker; those are always sent to the server. ~ Esri blog (accessed 03/08/2016)

I still think we should thank Starbucks and McDonalds for their free Wi-Fi since this is still a great way to take a quick coffee break and do large syncs - but maybe with these updates you can be back in the field before finishing your Grande Chai Tea Latte, 3 Pump, Skim Milk, Lite Water, No Foam, Extra Hot drink of happiness.

ArcGIS JavaScript API 3.16

Tue, 08 Mar 2016 21:09:11 GMT

With Palm Springs hosting the Esri Developer Summit this week - a few updates are coming out of Redlands just in time for the plenary. The ArcGIS JavaScript API 3.16 was released this past Friday and it has many new features to showcase on the big screen.

ArcGIS JavaScript API 3.16 what’s new

Just a few items highlighted below, or you can checkout the full list here - but after searching, the easy button is still missing from the features list (Spoiler):

A new theme: Sorry Claro, there is a new theme in town. “Calcite” is currently in beta and has some advantages - it will not look identical to all other API maps out there using Claro.
The feature table widget has been overhauled to allow for more visual customization, and more events available. Not only that, it now supports editing!
Directions widget: Improved visualization and route editing is now available.
Measurement widget: Not only a much needed update for accuracy, but also on-the-fly dynamic distance/areas as you move the mouse. Sure we all wrote our own 3 years ago to do this - but nice to see the core function making it to the prime time widget world.
Vector tile basemaps available as named basemaps. The tile basemaps are still beta, but access to them is ready to go.

More tweaks, improvements, and bug fixes are included as well. The vector tile basemaps are a very exciting feature, and we should expect to see much more related to vector tiles as we move forward (read: this is the future of base maps).

ArcGIS 10.4

Sun, 21 Feb 2016 16:38:58 GMT

ArcGIS 10.4 has finally arrived (most of it). Esri announced last week that their flagship product suite, ArcGIS 10.4 is here. This includes an updated release of Desktop, Server, and Portal to name a few. And also marks the [any day now] release of Pro 1.2.

There are lots of new features with ArcGIS 10.4, along with a slew of patches/fixes. The usual reference to software has also changed (“ArcMap” seems to be replacing “ArcGIS for Desktop”), and 1.2 Pro is part of ArcGIS 10.4 release, and… who cares, if you can’t figure this out ask someone to draw you a map 😉

ArcGIS 10.4: ArcGIS for Desktop (AKA ArcMap?! errr ArcMap and ArcCatalog?! Arrgh, Arc32-Bit):

Some fixes, tweaks, and new features to keep up with the times. Nothing groundbreaking, but then again, we are nearing the end of the product life-cycle here. Don’t get me wrong, many of the new features are great, and will help you delay moving to Pro a tad longer if you so desire - but it’s time to make sure Pro is in your long term implementation plan. Some updates to Arc32-bit are:

Lots of new and updated LAS tools
Tools to create SQLite databases
Python: Upgraded to Python 2.7.10. New libraries included: SciPy, pandas, Sympy, and nose. Many existing libraries also updated.
Lots of raster improvements, and new supported types (SPOT-7, UAS, etc).

ArcGIS 10.4: ArcGIS for Server:

Lots of updates and new features here. Since ArcGIS for Server is used in many different ways (AGOL, EC2, Azure, etc) this release not only has tonnes of fixes, but includes lots of new admin features to boot! For a full list, jump to Server’s What’s New page - although here is a quick summary:

ArcGIS Server read-only mode
Update passwords for registered and managed databases
Changes to geoprocessing service publication and service extension deployment: Only Admins can publish these by default at ArcGIS 10.4.
Default single cluster mode: Reducing load balance chatter by default.
Default install setup for HTTP and HTTPS protocols
Security fixes and enhancements
Scan ArcGIS Server for security best practices: Comes with a Python script that scans for some common security issues.
Microsoft Windows 10 support for testing and development (not recommended for production).
Microsoft .NET Framework 4.5 support
Chain together multiple server object interceptors (SOIs) for a service
Service enhancements: Including new extract capability and GeoJSON format to name a few.
Changes for ArcGIS Server on Amazon Web Services
Patch notification for extensions
Geodatabases and databases: Support for Damang databases, and new requirements for SQL Server

ArcGIS 10.4: ArcGIS Pro 1.2

The release date and official list of new/updated features will be available from Esri in the next few weeks. One very important feature of Pro 1.2 will be the ability to use Concurrent Licensing a feature that will help many organizations work with Pro in a familiar setting. Lots of enhanced 3D tools and new Vector Tile tools are anticipated as well. More on 1.2 after it’s officially released.

ArcGIS Collector Updates Galore

Tue, 22 Dec 2015 09:05:38 GMT

Some of the most anticipated updates for ArcGIS Collector have arrived this December! Just in time for… winter?! (Canada Joke). That said, I’ll take these updates any day - seeing as it will be 15 degrees Celsius (59 Fahrenheit) on Christmas day here in Ontario.

So what’s all this Collector fuss about you ask?

ArcGIS Collector Attachment Sizing (for iOS)

That’s right - you can now optionally set the attachment size for images within the ArcGIS Collector application (iOS users). A huge issue for many users is the significant storage size of all the images being synced at full resolution. Not only are the images huge, but the photo dimensions are likely larger than anyone will ever need. The work around until now has been to take photos with an external app, then add them to ArcGIS Collector afterwards - not ideal if you’re the one in the field. If you still want large photos you have that option too - but for those of us who feel syncing shouldn’t lead to as many Illegal Start of Token Errors , this will definitely help. There is no getting back the original photo however, so use these settings wisely.

ArcGIS Collector Attachment Size Options

ArcGIS Collector Sync Settings

It’s also now possible to collect new features in the field and push them up to the “Cloud” (feature service) without pulling down gobs of updates from other users at the same time. The new synchronization setting is pretty straight forward: Push Only - send your updates but don’t receive others’. A great enhancement to increase productivity when out in the field - and taking a break at Starbucks to sync your changes via WiFi.

ArcGIS Collector Sync Settings

ArcGIS Collector for Windows 10

Maybe not huge demand at this point, but if you’ve been watching what Microsoft has been up to lately, ArcGIS Collector for Windows 10 might be a contender when it comes to field data collection. Sure Windows Mobile/CE was a time we would like to forget, but Windows 10 is quickly gaining ground. Not only that, Windows is the primary OS at many Government Organizations - so there is some long term potential when it comes to standards. Of course, this is all based on quality hardware being made available - and Microsoft looks like they might have some of this covered with the new Surface Pro 4 and the Surface Book. Why do field staff need a laptop, an iPad, and a smart phone - soon, windows 10 will have that all covered in one device (AKA “Continuum”). Maybe a little hard to imagine at this point, but the laptop/desktop/tablet in one, we are already there. Collector for ArcGIS is now available in the Windows App store and runs in both desktop and tablet mode depending on your needs.

ArcGIS Collector for Windows 10

ArcGIS JavaScript API 3.15

Mon, 21 Dec 2015 15:58:13 GMT

In time to upgrade over the holidays, ArcGIS JavaScript API 3.15 is here with updates that enable us to use many of the new server features that have been announced over the past few months.

ArcGIS JavaScript API 3.15 - no more beta!

The biggest improvement to this release is the move from beta to production for both the GeometryEngine and the FeatureTable modules. Using before this release there was fear that something could change or disappear - thankfully now that they are in production, using them will not be as much of a gamble. A few new methods are also included with the GeometryEngine for Geodesic Densification (used to draw a line of the geometry along great circles).

ArcGIS JavaScript API 3.15 - New stuff

Lots of new features added to this release, mostly related to recent server side enhancements:

New layer type for Vector Tile Layers: One of the biggest updates on the server side is the addition of Vector Tiling which can now be accessed by the API. This includes client side rendering of individual layers on-the-fly.
New rendering (Smart Mapping) method called “PredominanceRenderer”. Which ”allow[s] you to generate a UniqueValueRenderer showing the predominant value in each feature among a set of two or more competing fields” (Esri).
A new Sample of samples: A new sample has been introduced to help with symbol styles, the ArcGIS Symbol Playground. Make a unique symbol with this sample, and copy the rendering code to add directly to your application. That’s right, spend less time in your browser hitting F5 to refresh!
Over 35 new events/methods and enhancements added

ArcGIS JavaScript API 3.15 - Squished Bugs

A total of 15 bugs were fixed - although expect many fixes in the next release as vector tiles are just getting warmed up. Many of the fixes are related to Geocoding, symbols and the legend. Oh, and the annoying Measure Widget always opened by default even if you didn’t want it to issue is now fixed - thank you!!

Side note - you can now create a custom build of the API using Bower. This is a much better alternative to the Web Optimizer which can be a little complicated to get working properly. Want to give it a try? Esri has a great tutorial on this very subject!

ArcGIS Collector Illegal Start of Token

Thu, 12 Nov 2015 20:32:04 GMT

After minutes, hours, nay - week in the field, you decide it’s time to sync your offline ArcGIS Collector edits so others in your organization can enjoy the vast quantities of field data you recently collected. Clicking the sync button, a familiar progress bar appears and we start thinking about our next Collector excursion and how easy that was. Alas, we jinxed it. Our positive, happy thoughts somehow caused a rift in the wifi continuum. It then appears: ArcGIS Collector Illegal Start of Token [<].

Sync Error: ArcGIS Collector Illegal Start of Token

You try syncing many times thinking the gremlins probably went away, but the consistency of the message makes you second guess you and your finger mashing of the sync button. You think possibly checking for an update in the App store could help - but hopefully you finish reading this before doing so, or risk losing all that glorious data.

ArcGIS Collector Illegal Start of Token [>]:

The illegal start of token message is very generic, and there are many different reasons you might get this message. The one described below is based on the following configuration:

Apple iPad Air 2, iOS 9.02
ArcGIS Collector v10.3.4 or v10.3.5
Service locally hosted on ArcGIS Server 10.3.1 (windows/IIS),
- Secured service
- Service has 1 line FC, 1 point FC, 1 photo attachment to point FC
Service is registered with ArcGIS Online (AGOL) as Feature Service
The Feature Service SAVES service credentials
Create AGOL map referencing the feature service
Easy peasy

The real culprit in our situation is the credentials being stored with the feature service when added to ArcGIS Online (AGOL). For some reason, the sync will start but there is a failure after a few minutes with the token. Well, it’s actually the combination of the stored credentials, and having a large sync process - not based on feature count, but total size (photos included in that size consideration). Photos being the main focus of contention here as they drastically increase the total size.

Solutions - Part 1: Worth Trying First

Since the error is generic, it’s a good idea to start with the simple solutions that appear to have worked for a few users. Although these didn’t work in our situation, GeoNet has many possible solutions that should be tested first:

Increasing the IIS uploadReadAheadSize on our ArcGIS Server web server
Using a fast and strong wifi connection
Wait until ~~10.3.5~~ errr, 10.3.6 when the app will allow photo resize to deal with total sync size and hopefully fix the sync bug regardless
Whatever you do - don’t upgrade the app from 10.3.4 to 10.3.5 thinking it will fix the issue. This introduces another bug: Upgrading from 10.3.4 to 10.3.5 will erase all offline edits (found out the hard way).

Solutions - Part 2: Deep Dive

If part 1 above doesn’t solve the sync issue, there are a few additional methods that may fix or at least save the synced edits as we wait for 10.3.6.

1: Recreate the Feature Service: This time don’t save the credentials with the user/pw. Then open the existing AGOL map, remove the old Feature Service, and add the new one. This will NOT fix the sync of existing offline edits - take care of those first or extract manually at least and try the second step below. The sync of existing offline edits will fail since the Feature Service has a new reference ID in AGOL - In a future release of AGOL, hopefully we can modify the existing service to prompt no longer storing credentials - this might allow existing edits to be saved with less effort.

With the new approach users will be prompted for the user/pw when first accessing the Collector Map with the Feature Service inside. This isn’t ideal, but it does save this information in Collector, so as long as the Collector account stays logged in, the service seems to keep the feature service user/pw stored. You can also store the user/pw in the Notes app just in case (we won’t tell).

This approach has allowed us to sync larger batches of data, however, we sync every night as we still don’t know if there is a cap on the effectiveness. Very large sets of attachments should still consider a different approach (online edits, or store Feature Service on AGOL) as there might still be a limit.

2: Manually sync edits: Yuck! If nothing else works buckle down and get ready for traditional GIS joins and calculates. Since we used an iPad, the first thing to do is get the data off the device. Yes, we need iTunes (sorry). With iTunes connected to the iPad, you can copy the sync edits by browsing Apps Data section. Just copy the user folder with edits to the desktop. The files inside have names like “3123456789.geodatabase” but don’t worry, they are legitimate GDBs.

iTunes Collector Documents: Get your data back!

Since this is a Geodatabase in disguise, we first need to get the data into a usable format. Thankfully, there are now tools to get the data into an XML workspace. To do this, just open the Toolbox or use python to extract the data: arcpy.ExportXMLWorkspaceDocument_management(source, XML). Once complete, you can import the XML into a new/empty File GDB using another Toolbox/python tool arcpy.ImportXMLWorkspaceDocument_management(XML, GDB). Congratulations, you now have a File Geodatabase with Collector edits.

This approach will keep the relationship and GUIDs intact. If there are no relationships or photo attachments, and only new features created - just use the Object Loader (ArcMap) or Load Data (ArcCatalog) tools to append the new data into your Enterprise Geodatabase.

If there are attachments/photo and possible modifications to existing features, you still have your work cut-out for you. The best approach I found so far is to add an additional “Original_GUID” field to all Feature Classes and Tables (in source/enterprise GDB, and in newly created file GDB). Calculate the File GDB “Original_GUID” field equal to the real GUID field in Feature Classes. In related tables, create a “Original_RelGUID” and calculate based on “Related_GUID”. Once complete on all tables and feature classes, you can start the old-school grind of loading, joining, and calculating fields.

For attachment tables, the “Related_GUID” field will not change when appended to a parent layer (so why did we create a copy?! You’ll see…). In the Feature Class, the “GUID” field will change when loaded, but the “Original_GUID” we populated will remain. You can now join the “Original_GUID” in FC to the “Original_RelGUID” in the related table. Next, calculate the Related_GUID in the table equal to the FC’s GUID field. Since we joined using the “Original_RelGUID”, the table will not go crazy since we aren’t changing the field the ArcMap Join is based on - otherwise, the table will lose track of the join and edits might not look like they have been made correctly. Also good for double checking your work (which is why we created it).

Notable Related Links for iPad/Android and ArcGIS Collector Sync Issues

ArcGIS Javascript API 3.14 and 4.0

Thu, 23 Jul 2015 11:56:03 GMT

Just in time for the Esri International UC in San Diego, two big updates have been announced for the ArcGIS Javascript API - the release of 3.14 and 4.0 (beta 1).

ArcGIS Javascript API 3.14

ArcGIS Javascript API 3.14 is officially released and all things considered is a pretty big enhancement to 3.13. One that users/developers have been asking for is finally here, the LayerList dijit class widget. The is debate if such functionality should even be used in web applications still remains, but the popularity of the request over the years has finally pushed Esri to release an officially supported widget (officially a beta widget still). The widget provides users with the ability to toggle layer visibility, however it doesn’t include any legend components at this time. Other new classes include:

ElevationProfile (beta): Allows a user to create an elevation profile based on a line
ImageServiceMeasure: Allows a user to perform measurements (length, area, volume, height, absolute location) on an image service with mensuration support
ObliqueViewer: To display images in their native coordinate system.
WFSLayer: Not a widget, but worth mentioning this new class since it supports OGC Web Feature Services. You guessed it, beta only.

There are also 13 new samples and 28 major bugs that have been fixed - including many that were causing some issues in IE10 or IE11 (FYI: Many government agencies will be on IE for a while still, so this is good news). Other notables are the significant enhancements that enable developers to integrate with the Operations Dashboard and DataReviewer. For more information on what’s new and fixed at version 3.14, check out the What’s New page.

ArcGIS Javascript API 4.0 (beta 1)

As the name suggests, this is just the first of many beta releases (beta site). Esri is planning a few betas before the final release which is expected next year (UC anyone?). The biggest update to 4.0 from the user perspective is the new 3D “Scenes” view with support for 3D symbology. On the back-end, almost everything is changing - but in a way to simplify the developer experience. Migration is also very straight forward: Don’t (just rewrite).

Consider rewriting applications instead of simply trying to update them. ~ Esri help on migration

If you’ve been holding out on switching to AMD, time to get in your Delorean and join us in 2015 (80’s movie reference just in case). I understand some people will be reluctant as we all have legacy apps built using non-AMD styles, but since 4.0 apps pretty much require a re-write anyway, it seems like a logical step for Esri to drop it at this point. Other notable changes from Esri include (listed in their beta guide):

Handling properties: Getting and setting properties is much more elegant and aligns with current dev approaches. Listening for property changes is also much easier by introducing a .watch(property,callback) method.
Map and layer separation: Views have been introduced to separate some of the map and view/drawing logic. The “Map” will contain some base details (core properties, basemap reference), while “Views” will handle the 2D/3D and drawing logic. That’s right, ArcView 3.1 has finally been integrated into ArcGIS Pro and the web APIs!
Module and package updates: Changes to the location of [some] classes and packages have been implemented, but it appears most of them are sticking around in some form. This renaming should be welcomed as the new structure uses cleaner names with consistent CaSe conventions.
Deprecation: The Geocoder widget has been dropped at 4.0 as the Search Widget will take the reins. Will help alleviate the confusion as well.

Remember, this is just the first beta - but I’d still recommend checking out the samples and the Discover 4.x documentation to start getting your ducks in a row as this is a significant overhaul for the ArcGIS Javascript API.

Print Text Elements with ArcGIS for Server

Sun, 28 Jun 2015 13:48:54 GMT

Creating your own map template, or using the generic ones, you have the ability to print text elements with ArcGIS for Server layouts. But how much text can you store and print properly without it being truncated or running off the printed page?

What’s in a name?

The generic print templates have custom “Title” and “Author” text boxes and these properties are exposed in the layoutOption object so you can change the text in your application. But just because the property says “Title” and “Author” doesn’t mean you can only use them for that specific purpose. You can store any text, as long as it fits in the limited text box. If creating your own MXD template, you have much more control over this. You can make the Title/Author text boxes much larger (fixed width, multi-line, etc), or you can even add more custom text boxes to the layout as you aren’t limited to just Title/Author when building your own.

How long is a piece of string?

When we print text elements with ArcGIS for Server, you should probably determine how much text you can truly include - this will help the end users, the application, and the printouts! But how long is a piece of string (sic)?! If you assume all characters of text are the same length, you will quickly find out the text boxes don’t agree with you. For example, a “W” character uses much more space than the “i” character. But how much more? Naturally, it depends.

To determine length, a much better approach than counting letters or words, is to count pixels. Assuming you are trying to avoid a custom python print task and using straight forward MXDs, the pixel width of the text box is constant since all the font properties have already been set in your Map Document. Just type some text into your text box until it’s full, then calculate the pixel length from that text and you will have your maximum size. Next, add some code to your application to determine the user input length and make sure it doesn’t exceed the maximum. Easier said than done? Keep reading.

Pixel Magic

How do you calculate pixel length, is it some crazy equation showcased in the movie ‘Goodwill Hunting’, or from your favourite String Theory textbook? Not even close, and not even complicated! Here is an example when working with the ‘Title’ text, although will work with any custom text too:

Open your custom MXD template (or Esri generic) and type random text into the title text box until full of text. Copy that text to your web app and remove from text box (doesn’t need to be saved in the MXD). This text is your maximum length placeholder.
In your web app, create 2 text boxes: One for the user to enter text dynamically (id=‘txtTitle’), a second one with your dummy text from step #1 included as it’s text (id=‘txtTitleLength’).
Both text boxes: Set identical font, font-size, font-type (important to compare pixel length)
txtTitleMaxLength: remove all width styling and make ‘inline-block’. Note: you can hide this text box later so users don’t see it. For now, just keep it while testing.
Calculate the length of the txtTitleLength text box using the following code (assuming jquery in this example): $(“txtTitleLength”).width();
- Swap the user text from txtTitle into txtTitleLength, and get the width again.

Phew, we have the pixel length of our max dummy text, and the user text. The outputs from step #5 are the number of pixels we have to work with. I recommend keeping the txtTitleLength textbox and dummy text as part of a JavaScript calculation so you can always determine the max length available based on user browser settings. Just hide txtTitleLength from the interface.

Help users help themselves

We now know the maximum number of pixels, and the user text pixels. As long as the width the user text less than the dummy text, you are good! Now when printing Custom Text Elements with ArcGIS for Server you can feel more confident that the text provided by the user will fit the layout. This also works with large text boxes with fixed size and multi-line text.

ArcGIS 10.3.1 Released

Sun, 17 May 2015 21:03:11 GMT

ArcGIS 10.3.1 has been officially released by Esri. Like most build number increments, there are both fixes (service packs, quick fixes), and enhancements (new features and functions). If you are already using 10.1 or later, you can directly upgrade your existing installation - just remember to upgrade Desktop License Manager first, and that your ArcGIS Server services will be offline during the upgrade process. For new installations, you can directly install 10.3.1 without any previous install required.

ArcGIS 10.3.1 What’s New

The Desktop 3D Analyst extension seems to be the real winner here as LiDAR/3D data continues to expand its use: Classify LAS By Height, LAS Point Statistics By Area, and Locate LAS Points By Proximity are all new tools included at 10.3.1. There are also updates for GeoData, Spatial Analyst, and a few other minor tweaks to export CAD r2013 to round up the main Desktop features.

Server 10.3.1 has many new features. 3D Scene services are one of the biggest new items, allowing you to publish 3D content from ArcGIS Pro directly to server as a 3D service. For developers, a new Server Object Interceptor (SOI) will allow you to execute custom/changed logic before returning to the client - moving us away from the Server Object Extension (SOE) approach (still exists, but SOIs seem to be the flavour of choice). Other enhancements have been made for OGC compliance, IIS tweaks, and a new extension for “Maritime”.

In short, Portal 10.3.1 has been updated to the same/latest version of ArcGIS Online. It also supports the features listed for Desktop/Server above with published 3D content and a specialized imagery services tweaked for elevation data. The “Smart Mapping” feature has also been included to help with symbology/style options as updated Online a few months ago.

Here are links to the full documentation on what’s new in 10.3.1 for each product:

ArcGIS 10.3.1 Bug Fixes

Hundreds of bugs have been fixed at this release and also includes a roll-up of all previously release 10.3 patches - so you don’t need to install them first. If you have a specific bug you want to verify is squashed, you can check out the full list at at Esri’s support site (pdf).

ArcGIS JavaScript API Printing Group Layers

Tue, 28 Apr 2015 23:38:13 GMT

You’re not hallucinating - sometimes the map you see in your ArcGIS JavaScript application doesn’t look like the one you printed via the Print Service. There are many reasons why this might happen, and one is related to Group Layers.

App vs Printing Group Layers?

You may notice your group layers are working fine in your web application - automatically turning on/off the sub-layers as configured in the map service (for max/min scale dependant display). Without specifying any additional settings in the web app this will display as it would in ArcGIS Pro/Desktop. However, the Print Service will not use the same approach for printing sub-layers. In short: It’s not always WYSIWYG.

Dynamic Map Service Layers Example

For web applications, turning on a group layer to make sub-layers visible is straight forward (if the group isn’t already visible by default). The code below would turn on the ”Points of Interest” and ”Roads” group layers based on the sample image:

DynMapService.setVisibleLayers([0,3]);

Since sub-layer min/max scales are set in this example [as noted in square brackets], as you zoom in/out, different sub-layers will toggle automatically based on their default settings. There is no need to determine which sub-layers are meant to be on/off in your application code.

However… the Print Service doesn’t exactly work the same way.

The Print Service will query the map service’s visible layers (DynMapService.visibleLayers) - in our case, layer 0 and 3 have been coded as the only visible layers. The result of this query will update the JSON web map parameter in the Export Web Map task. Since the sub-layers aren’t included in the JSON string, they will not be available in the export.

Not a Bug

To be clear, the ArcGIS JavaScript API Printing Group Layers issue is technically not a bug. More of a lesson learned and heads-up that you should keep this in mind when writing your application with potential printing requirements. You can completely work around this issue by specifying the sub-layers in your code as well:

DynMapService.setVisibleLayers([0,1,2,3,4,5,6]);

Turning on the sub-layers by code will not override the max/min settings. So specifying as visible should still give the desired results in both the application and printouts. Again, just like turning on/off a layer that is out of scale range in ArcMap will not visually show the layer if outside the min/max range.

Specifying the sub-layers is great if you are controlling the visibility in code, but what about users? If you have a more complicated scenario where you’ve created a Table of Contents only showing the group layers (for users to check on/off). If this is the case, you will need to programmatically find the sub-layers so the Print Service knows to request them. Here are a few ways to accomplish this:

Method 1: All layers and sub-layers in one request

Get the structure of all layers in the map service, including group layers and sub-layers (if exist). A benefit of this approach is that you can send one request to the service as the application loads and store the settings for future use. To get the details, use the following syntax:

http://domain/MapService/MapServer?f=json

The result is a JSON object of the entire service including the layers, and sub-layers for a parent (aka group). So we could find the layer ID for Roads (layer id 3), it will have the following sub-layer property:

subLayerIds:[4,5,6]

Method 2: Unique Layer Request for Sub-Layers

If you just want to get the details for a specific layer you can just send the request specifying the layer. This is great if you want to see what type of layer is it (group, feature, etc) and reduce the size of the response (if you have a large/complex map service). Just specify the layer ID including the JSON parameter:

http://domain/MapService/MapServer/3?f=pjson

The returned JSON data will look like the following (more info is available but not shown):

{
  "currentVersion": 10.3,
  "id": 3, "name": "Roads", "type": "Group Layer",
  "description": "", "geometryType": null,
  "copyrightText": "", "parentLayer": null,
  "subLayers":[
    {"id": 4,"name": "Roads - local"},
    {"id": 5,"name": "Roads - mid"},
    {"id": 6,"name": "Roads - overview"}
  ]
...}

Well, that’s about it. If you still think you are hallucinating, maybe ask your roommate what’s in those brownies sitting on the table!?

Printing with ArcGIS for Server 10.3

Sun, 19 Apr 2015 21:17:08 GMT

My relationship with the ArcGIS Server Print Service and the REST API was a little more turbulent than expected. Here’s how it all started…

Like many relationships these days, we met online. If you are like me, the Esri JS site is a great place to browse, and hopefully find what you are looking for. And I will admit, this wasn’t my first time. Maybe we moved too fast in the beginning - dinner and some sample code, brunch and a PDF export. It was so easy and comfortable. And that’s when it all started to go sideways.

After a short while, I wanted our relationship to be more secure. We could both print with other people, but I wanted it to be somewhat safe, and dare I say, protected. I looked into some self help books on how to address this and thought I had found a way to take us to the next level. It was a token at this point, short or long, only time and more experience could tell.

This isn’t a new story, we’ve all been down this path before, my Ex, which we can refer to as “ADF”, didn’t print with me for years, but we eventually made a secure connection - but it was already too late as I had stopped using SOAP. But now, things were looking up. A mature [software] package and many RESTful nights had me thinking this was it! Communication was strong, and flexible - but as in all relationships, you really need to work at it. Here are some lessons learned, and where I drop the corny relationship analogies…

Environment Configuration

The workarounds and issues described below are all based on the following:

ArcGIS for Server 10.3 - all patches as of April 10, 2015
ArcGIS JS API 13.3 (latest release as of tests)
Esri DotNet Proxy v1.1 beta
Custom PrintingTools service (to support different layout)

Method 1: Secure Map and Print Service with Long Tokens

If you specify a long token for your app, remember that this will not work with the OTB Print Service directly. The documentation is pretty clear on this, and does explain how to setup a custom print service with credentials.

If you follow the documentation setup for a custom Print service with credentials, you should be up and printing in no time. However, it will only work for 60 minutes on a default AGS installation. More specifically, every thread request will fail after 60 minutes from first request or starting the service. The issue is that the custom service will not request a new token with the stored credentials unless the service is restarted or recycled. After working with support, it is an official bug (BUG-000086183) that will be fixed in 10.3.1. [someone may have already found the bug, but first I heard of it]

If you can’t wait for the 10.3.1 bug fix (which will be release in May), here are two workarounds:

Change the Print Service recycle time to 1 hour to match the default short token time (short tokens are 60 minutes by default). If your recycle time is different, you will need match it to your current setting. If recycle time is <= 60 minutes already, it should already be working without issue.
[or] Modify the maximum length of short-lived tokens in AGS Manager (Security > Token Settings) to match your recycle time. The issue with this approach is that it impacts all short tokens in ArcGIS server, not just this service.

Method 2: Secure Map and Print Service with Short Tokens and Proxy

As I worked with Esri support on the issue above, I decided to play with the short token approach. The documentation for printing with short term tokens appeared to be much easier. No need to embed credentials in a customized print service. However, it didn’t work either?!

If you secure your services using short-term tokens (the default), the user will be challenged to provide the name and password upon opening the web app. The token is then passed to the PrintingTools service when the user requests a map to be printed. There is no need to create a custom print service (as described below) if you are using short-term tokens. ~ Esri help

Confused, since the help says “there is no need to create a custom print service…”, I thought I found a second bug in as many days. But reading it again, the kicker is that a “user will be challenged to provide the name and password”. If a user provides the credentials, yes, it will work. My setup using the proxy with short tokens was the culprit. I don’t want users to provide passwords - Now what?

The current version of the proxy (1.1b) will request a token to access the secure map and print services which works great for the JavaScript application. However, since the proxy sends referer information, the token is tied to that referer (even if your proxy and AGS are on the same server). Although not an issue for application, it is a big issue for the Print Service since it will send its own request for a map without a referer - so the request is rejected by the map service. [If my spelling of “referer” appears wrong, here is a short history on the typo.]

So how do I use short tokens in the application yet still use the ArcGIS Server Print Service without users typing in passwords - here are a few workarounds:

Use the custom print service with credentials stored for printing from Method 1, and a short token for the application via the proxy. It’s a hybrid, but it works (must also implement one of the workarounds previously mentioned).
[or] Modify the proxy. Some C# Kung Fu and you can remove the referer as short tokens don’t require one be provided. Although the downside is you will be out of the loop for any fixes/updates to the Esri version.
[or] Surrender the approach and have users authenticate in the app. The downside is that your users are required to have a user/pw. Pro is that you can control access and use the short token approach.

Just wait until users have 3D printers!

ArcGIS Collector 10.3 for iOS

Thu, 12 Mar 2015 21:49:40 GMT

It’s finally here. With little fanfare (relative to Apple Watch and the all-new MacBook earlier this week), the iOS release of ArcGIS Collector 10.3 has finally arrived. A little surprised there was no mention on Esri’s blog - although to be fair, there might be some Dev-Summit hangovers. Regardless, the update finally appeared in the App Store (in Canada).

ArcGIS Collector 10.3 for iOS

As previously mentioned, ArcGIS Collector 10.3 includes the much requested ability to view and edit related records, as well as the ability to edit version data, and some enhanced settings for “Add-only [new]” or “update only [existing]” editing options.

iOS 7 or later is required for this latest version, and some of the new features will require the data be published using ArcGIS for Server 10.3 or ArcGIS Online. Even if you don’t use these new features, there is a long list of bug fixes included and well worth the upgrade (< 50 MB).

ArcSDE 10.3 Administration

Mon, 23 Feb 2015 23:22:00 GMT

ArcSDE 10.3 administration has significantly changed. More specifically, ArcSDE no longer exists. Starting at 10.3, there is no more ArcSDE installer, SDK, or command line tools made available. Fear not, we still get to administer our Enterprise Geodatabase(s), just a new set of tools/scripts to do so. The new admin tools are located in toolbox and/or python commands. But it isn’t all unicorns and rainbows…

No support for 9.3 ArcSDE Geodatabases with 10.3 clients (ArcGIS for Desktop, ArcGIS for Server, etc).

If your organization is fairly nimble at upgrading ArcGIS Clients (Desktop/Server), but has been keeping the RDBMS/ArcSDE running as a 9.x Enterprise Geodatabase, this environment is not supported with 10.3 clients. There has always been backwards compatibility with the Enterprise Geodatabase, but it doesn’t last forever. 10.2.x was the last version to support ArcSDE 9.x. A note on Esri’s compatibility page states:

You can connect from an ArcGIS 10.3 client to a 9.3.1 geodatabase to upgrade it; however, the geodatabase must be in a supported database management system release. ~ Esri.

That all said, if your 9.3 enterprise Geodatabase is still using a supported RDBMS version, it still works! Just because it isn’t officially supported, doesn’t mean it stops working. Although don’t expect any bug fixes, or 10.x new features to work (edit tracking, ArcGIS collector editing, photo attachments…). I’d keep a 9.3 Desktop hanging around just in case - actually, just upgrade your Geodatabase to 10x already.

ArcSDE 10.3 Spatial Views with SDEBINARY

Sorry, keep forgetting, no ArcSDE 10.3. But on the topic of spatial views, they are still around. In most cases you can create Spatial Views in 10.3 using the new Create Database View tool. This tool is available with ArcGIS for Desktop Standard and Advanced (ArcEditor and ArcInfo for us dinosaurs). But be forewarned, this will not work with older binary storage types (SDEBINARY).

You can either blame Esri for this limitation, or blame your Geodatabase for it still allowing you to store SDEBINARY (after all those years of being the default spatial type). Either way, it is what it is. Our office was using SDEBINARY but switched to SQL Spatial Types when we upgraded - for many reasons including this one. Just think how happy your DBA will be when there are no more F-tables cluttering the RDBMS Enterprise admin console! Probably not as funny as when you tell them they will need ArcGIS Desktop if they want to fully administer the Geodatabase (or give you more permissions). [If you work in a small office, it’s all you regardless.]

ArcSDE 10.3 Service, almost extinct

I thought it was finally behind us, the 3-tier Enterprise Geodatabase connection. You can’t create them as easily as a 2-tier connections (aka “direct connect”), but there is still the Create ArcSDE Connection File tool that will allow you to connect using a 3-tier connection. Even if you can’t upgrade your Geodatabase, it’s a good time to switch to the 2-tier method - you have little choice if you upgrade to 10.3.

Other ArcSDE Admin Commands

Here is a list of some additional ArcSDE administration commands and their replacements: Migrate from Admin Commands. While you read, grab a coffee - and feel free to use your old ArcSDE install CD as a coaster to get a bit more use out of it.

SpatialTimes 2014 Year in Review

Mon, 09 Feb 2015 13:50:02 GMT

The numbers are finally in, and 2014 was a great year for SpatialTimes.com. Sure it wasn’t a full year, as the first post wasn’t even posted until April, but the year still ended right on time. There were many draft posts at the beginning but it was hard to hit the publish button as a rookie blogger. Eventually I tossed many of those draft articles as they became stale before they were finished, or I just wasn’t happy with the final product. Overall, the posts continue to be related to what I’m working on or discussing with other Geo-Geeks at the time. So let’s take a look at the SpatialTimes year-ish in review…

SpatialTimes: The numbers

These numbers are summarized from Google Analytics. They don’t necessarily align with the JetPack stats, but there is already lots of articles on that topic. That said, my learning curve with WordPress has also changed over the year, so some results can be a little misleading at times (which will hopefully be explained).

Who are We/You/Us

Being the first year, it’s no surprise the user base was mostly New Users (78.5%) compared to returning (21.5%). Total sessions/pageviews has been trending positive every week/month so far - even the Christmas season didn’t stop the trend! As a new blogger, this is very exciting and I hope you keep coming back for more. That said, this blog doesn’t generate revenue, just nice to know we aren’t alone when it comes to GIS.

2014 Sessions and Views - Yippee, the dots are headed in the right direction!

The top 10 Countries (by total sessions) held a little more interesting. The US and Canada accounted for over 50% of all visitors, with Brazil coming in 3rd with 12.01% (the only other Country with double digits).

2014 Top 10 Countries

These numbers, along with a few others, have changed dramatically since I started blocking more spam site referals over the winter holidays, but they are the only numbers I have at this point. Spam sites also inflated the bounce rate [def: Percent of single-page visits] for the first year, up as high as 53%. The bounce rate is now 17% since updating the htaccess file - thank you Internet machine and Google Search! Below is a chart showing the 2014 to 2015 change in bounce rate after the WordPress updates were complete. The Bounce Rate bounces lower while Sessions continue to flourish. Phew! If you notice sites like “semalt-semalt” clogging your referrals, I’d recommend reading a few articles and forums on the WordPress site on the many ways you can start blocking them from your site and stats. Not only will it help you better understand your visitors and what is really happening on your site, but also helps you dive a deeper into the world of WordPress and the never-ending battle with spam (if so inclined).

2014 Bounce Rate Fixed in December (Bounce rate bounces lower while Sessions continue to flourish)

The Browser Wars

An overwhelming 60% of visitors are using Google’s Chrome browser to visit SpatialTimes.com. I use all 4 of the top browsers, and my usage breakdown is very similar. It’s troubling, but if one was to breakdown the IE usage, you don’t want to know how many people are still using IE 6-8.

2014 Browser Usage

This is one reason why we are still developing websites that require so much backwards compatibility I guess. The total browser numbers do combined both mobile and desktop usage- however at this point mobile only accounts for 9.23% of total traffic. Take these results, along with time of day numbers, and it seems like most are reviewing on their work desktop. Makes sense, as I’d imaging there are more GIS industry people than weekend tinkerers. [Full Disclosure: I’m both]

Now What?

Back to the grind all 72.14% of you reading this at work. Or maybe add this site to your Feedly list, or your bookmarks if you haven’t already - or read a few more posts while you are here. A special thanks to all the SpatialTimes readers/followers for a great 2014, and looking forward to more in 2015!

Unexpected Google Announcements

Tue, 03 Feb 2015 08:00:46 GMT

A few big announcements from Google over the past few weeks. First, Google announced that support for the Google Maps Engine product will end on January 29th, 2016. And this past Friday, Google also announced an update to the Google Earth Pro software… it’s now free. That’s right, free as in beer! In a somewhat unexpected move Stafford Marquardt, Product Manager, Google Earth Pro announced the change on [Google’s Lat Long blog](Stafford%20Marquardt, Product Manager, Google Earth Pro). Over the weekend, Esri sent their users an email to let them know Google & Esri have a plan to deal with these recent changes:

In coordination with Google, Esri has prepared a special offer for Google Earth Enterprise and Google Maps Engine customers and partners looking to transition to Esri software. ~ Esri

The “coordination” is based on a no-cost replacement for Google Earth Enterprise and Google Maps Engine with Esri products and/or services for one year. You can find more details on the software/fees on Esri’s FAQ page. To start using Google Earth Pro for free, head over to Google’s Key Request page to sign up.

The end of Google Map Engine is a sad one. I’ve only used it on a few projects, but it has always been a very powerful way to make custom web applications (along with Google basemap styling options too). I only hope the lack of a major competitor doesn’t slow innovation in this vertical - so keeping pushing forward Esri, CartoDB, Mapbox, and iSpatial (to name a few)!

Wait, while we’re at it, will someone replace Google Reader too?!

ArcGIS Collector 10.3 Released

Fri, 30 Jan 2015 16:15:38 GMT

ArcGIS Collector 10.3 has just been released for Android OS, and will soon be available for iOS users as well. ArcGIS Collector 10.3 is more than just bug fixes - it has many of the features we’ve been waiting for. Probably one of the biggest enhancements is the ability to work with related records. Don’t jump in too quick however, there is a known bug involving archive enabled related records that Esri is aware of, and has stated that a patch is on its way shortly. ArcGIS Collector fans will also be able to edit versioned data starting at this release. Side note: Using versioned data is technically an enhancement, but it was more of a bug that it wasn’t incorporated until now - my humble opinion of course. Water under the bridge, as it’s finally here and can now be part of the same enterprise workflows (AKA ArcSDE) many have been using over the past 15 years.

If you are using ArcGIS Collector on Android: Time to update the app! For Apple users, myself included, be patient as Apple traditionally gets apps first, so it’s our turn to wait a few days. Patiently.

The list of what’s new from the Collector for ArcGIS site:

Ability to view and edit related records
- Finally! Let’s pause here for a second… Doesn’t that feel better?!
Support for taking additional data offline, including:
- Hosted tile layers (Tiled web layers hosted on AGOL account)
- Editable versioned data (from ArcGIS for Server 10.3 or later)
- Layers with editor tracking
- Layers with limited feature service capabilities, including add-only and update-only editing
Various bug fixes and improvements, including NIM101011, NIM101989, NIM102096, NIM103498, NIM103525, NIM103549, NIM103836, NIM104130, and BUG-000083110

Migrating to ArcGIS for Desktop 10.3

Wed, 14 Jan 2015 16:52:02 GMT

If you are migrating to ArcGIS for Desktop 10.3, there are a few additional steps not covered in the install guide. That said, if you are looking for those step by step instructions on installing or migrating to ArcGIS for Desktop 10.3, you can check them out here - they are straight forward and will help you get up and running.

Err… now what? Are we done already? That part is up to you…

Here are 6 additional steps to consider before calling it a day. Disclaimer: Some of these are done automatically if “upgrading” an existing 10.1 or 10.2 installation, some aren’t. If you use Citrix or other terminal server approach, the regular upgrade only does the user account that performed the upgrade and not the other users’ individual settings. Either way, worth checking out the following:

Install additional grid shift files (AKA: transformation files) for your country. If you work in Canada and never done this before, I hope you haven’t reprojected from NAD27 to NAD83 (it still runs, but it probably didn’t reproject they way you hoped). Here are instructions for Canadian users (check with your local distributor if your country is also affected):
- What you need: NTv2 grid files (*.gsb), you can download directly from Esri Canada.
- Where to put the files: Place the files in <ArcGIS Install>\pedata\ntv2\canada. You will need to create the “canada” folder. Make sure you use all small case letters. Copy the *.gsb files directly into this folder and the new transformation(s) will be available next time you need them if the transformation is already setup in the software.
Custom map templates: Wish your company map templates with logo, disclaimer, and north arrow were available to all users when ArcMap launched? Me either - but corporately it’s still a good idea.
- What you need: Create a template MXD with your logo and other flair including auto-update text. Possibly create 3: landscape, portrait, and large plotter templates.
- Where to put the files: Place the files in <ArcGIS Install>\MapTemplates. The templates will now be available when loading ArcMap on the start-up screen and also available on the layout toolbar.
Custom transformation files? If you want to transform to/from NAD83 original and CSRS (as one example), you can create a custom transformation since this one isn’t covered OOTB.
- What you need: To make a custom transformation file you can use a tool in the ArcToolbox: Create Custom Geographic Transformation (Data Management).
- Where to put the files: Place the final file (.gtf) inside <WindowsUser>\AppData[Roaming|Local]\ESRI\Desktop10.3\ArcToolbox\CustomTransformations. If the ArcToolbox folder doesn’t exist, you can open Toolbox inside ArcCatalog or ArcMap once and the structure will be created.
ArcCatalog mapped folders and database connections: Your database connection files can be copied from your old installation to the new one. Only downside here is that these need to be done on a user by user basis (every user on terminal/Citrix servers). Good news - you can do both database connections and mapped folders for ever user.
- What you need: Find your old connection files from your Windows user directory. The ArcCatalog.gx file stores all mapped folders; *.sde, *.ims, *.ags, *.odc files store database and web connections (one file for each connection):
  - 9.3: \AppData[Roaming|Local]\ArcCatalog
  - 10.1-10.2 (10.x): \AppData[Roaming|Local]\ESRI\Desktop10.x\ArcCatalog
- Where to put the files: Place the files into the newly created 10.3 user directory. Note: Open ArcCatalog once in advance to create the directory structure for you. Paste the files to: <WindowsUser>\AppData[Roaming|Local]\ESRI\Desktop10.3\ArcCatalog
- Note: If using SDE Application connections (3 tier SDE Service connections) it’s recommended to switch to using the Direct Connection method. In this case, don’t copy the *.sde files, just recreate them using the Direct Connect method from ArcCatalog.
Home Directory: Setup/verify your “home” location for the ArcMap Catalog window, then reinstall add-ins, custom toolboxes/tools, python scripts to name a few.
Check for patches and service packs: Last but not least, check to see if there are any patches or services packs available. Yes, there are already two available at time of writing (Jan 14, 2015):
- Geodesic Buffer Method Patch
- Spatial Statistics and Space Time Pattern Mining Patch

Happy upgrading!

More information:

ArcGIS Javascript API 3.12 Released

Mon, 29 Dec 2014 13:52:13 GMT

It’s only been a few months, but Esri has already released ArcGIS Javascript API 3.12. Phew! Normally, this isn’t a big deal - but if you were trying to use the Measure Tools in 3.11, this change couldn’t come sooner (it didn’t work in 3.11). There are also many exciting and notable updates in this release, including:

Two new basemaps options: Terrain (updated) and Dark Gray (new)
The HeatmapRenderer is now released (no longer beta)
10 new analysis widgets
Improvements to labelling for non-web mercator (and GCS) spatial references
New DataAdapterFeatureLayer and Location Provider (beta): A great addition that enables visually displaying a tabular dataset that doesn’t contain spatial information.
FeatureTable widget (beta): A great new beta feature that we’ve all been waiting for - a new widget that displays your features in a table format, and can be used both with and without a map!
Bug fixes: If you were exited for the 3.11 updates/enhancements to the measurement widget mentioned in my ArcGIS Javascript API 3.11 release post (among other fixes), but were saddened to find out the measurement widget actually broke at 3.11, these fixes are for you!

There are many more new and enhanced features in the ArcGIS Javascript API 3.12 release. For the full list of updates, jump to Esri’s What’s new in Version 3.12 page (links to most recent what’s new page).

ArcGIS 10.3 and ArcGIS Pro are released

Fri, 12 Dec 2014 10:01:22 GMT

That’s right, ArcGIS 10.3 (ArcGIS for Desktop, ArcGIS for Server, etc), and ArcGIS Pro are now officially released. In a post from Esri, they announced it is (finally) here. Although this is primarily a large service pack for Desktop 10x users, there is also many new features. On the Server side, there have been some significant improvements as well, including ArcGIS Portal now being included with ArcGIS for Server! That’s right, you can make your own AGOL! And I almost forgot, Pro comes in this release 😉

Here are some of the new enhancements to look forward to as you watch your download progress bar:

A new Space time pattern mining toolbox
Buffer and Spatial join tools have planar and geodesic distance options
New Geodatabase Administration tools added (9 tools)
Many tools have new parameters and enhancements
ArcGIS for Server (Std/Adv) comes with Portal for ArcGIS, including Web AppBuilder
Many GeoEvent Extension updates
10.3 includes Python 2.7.8 (Not ArcGIS Pro, it uses v3.x)
Support added for 1.0 OGC GeoPackage format
Many new raster functions
LAS version 1.4 files now supported

Some changes for Enterprise Geodatabases:

ArcSDE administration command line utilities are no longer available with ArcGIS for Server
Connections to 10.3 enterprise databases are Direct Connect only as the ArcSDE application server will not be available for 10.3 GDBs
Default connection limitations (64 connections) have been removed - now has no limits set

Happy downloading!

ArcGIS Collector External GPS Options

Mon, 24 Nov 2014 08:51:53 GMT

I’ve seen enough demos and presentations of ArcGIS Collector over the past year that I think it’s time to take a more in-depth look into what it has to offer. This may have something to do with our old GeoExplorer finally retiring after 8 years of great service. Do I replace the Geo with a new one running ArcPad or TerraSync? Or do I switch to an ArcGIS Collector based option? Either way, it was time to review some of the more recent GPS options available in the market.

For us, it’s a pretty easy decision based on the user requirements: ArcGIS Collector. However, this isn’t the solution for everyone, and our requirements aren’t exactly typical. We have the infrastructure, Esri licenses, basic field data collection needs, with accuracy requirements ranging from 30cm to 5m. This accuracy range will be discussed in more detail - since it’s the one variable that has lots of options and configurations. So let’s get started with some of our ArcGIS Collector External GPS Options…

The Requirements

When reviewing our ArcGIS Collector External GPS Options, I quickly realized there is an abundance of… options! Too many really. So first things first - back to the requirements and possible use cases. Since we still have other hardware and software that is field worthy (ArcPad, ArcMap, Laptops, Toughbooks), selecting an external GPS that can be leveraged by multiple devices also seems like a good idea. In summary, here are the GPS requirements we ended up with:

Must have Bluetooth connectivity
- Note: Some GPS device compatibility differs by connection method, the compatibility requirements below are based on Bluetooth only. USB connectivity is a bonus but not required.
Mobile OS connectivity: iOS and Android
Desktop OS connectivity: Windows 7/8 for laptop/toughbook use
Compatible Software: ArcGIS Collector, ArcPad, ArcMap
Satellite Systems: GPS and GLONASS. We require both systems due to better response under canopy cover (better chance of seeing more satellites).
Location Protocols: NMEA and Apple protocols (to support software requirements).
Accuracy (CEP): 30cm to 5m - varies depending on activity.
- Must be attainable in the field without using post-processing afterwards.
- No subscription service. It would definitely help (on-the-fly correction), but due to the vary in seasonal use, and budget constraints, we had to forgo at this point. Not many options either - although I hear Trimble will be releasing something in the near future.

The Hardware

The Apple/Android tablet debate isn’t something I want to get into. Talk to your resident [OS] fanboy for more information. If you have a mobile OS preference - go for it. If you have a company standard - stick with it. Since Apple controls its hardware and a similar Android tablet was more expensive, we ended up selecting Apple’s iPad Air (Cellular + Wi-Fi model). There are many similar Android tablets, but if you want to use the internal GPS, different Android devices have different specs so select with caution. With Apple, you get the GPS chip they provide.

iPad Air Location Services Comparison. Source: Apple (Location Services section). 5m accuracy assumed to be CEP based.

Even the iPad Air has some variation: The cellular version includes a GPS chip, while the Wi-Fi only doesn’t. The cellular version doesn’t require a cell plan for the GPS to work - although the benefit of a cellular plan is that Collector can perform real-time edit updates to your database! The Wi-Fi only model still has “location services”, but this is Wi-Fi/compass based only, and really isn’t a great option for field collection unless you always have an external GPS connected or plan to manually select locations - so it’s not bad; just not as good. With an embedded GPS, the cellular iPad does give us some flexibility to collect data without any additional/external GPS device when location and accuracy permit.

You may have noticed I listed other hardware devices in the requirements section, mainly Windows 7/8. Since we have some field laptops and toughbooks, we require the same external GPS be compatible with these units and software (ArcPad/ArcMap). If someone requires to use their toughbook in the field, we want to be able to provide the same external GPS device (via Bluetooth) - providing consistent accuracy, training, and GPS support regardless of hardware.

ArcGIS Collector External GPS Options

Finally, the part you probably skipped down to read - the ArcGIS Collector External GPS Options. Below are the devices I could get my hands on to review which related to the requirements. No sponsorship, freebies, or web ads determined the device list. Just a helping hand from friends who have the GPS, or if I already had one at work/home. That said, the device list contains most of the popular options available at this point (most are also discussed in GPS/esri forums as well).

Listing of some popular ArcGIS Collector External GPS Options. Source: Product websites.

*Notes on Accuracy: Most of the GPS devices didn’t specify the measurement accuracy they’re reporting even though they supply an accuracy value. This should be one of the most important statements, but it’s omitted way too often by manufactures and resellers (IMHO).

“Since accuracy is a statistical measure of performance, a statement of navigation system accuracy is meaningless unless it includes a statement of the uncertainty in position that applies.” ~ Navipedia.net

So where the accuracy measurement isn’t stated with the value, I think it’s best to assume Circular Error Probability (CEP) is being reported (50% of the locations have an error lower or equal to the accuracy value). Garmin has a history of reporting with CEP, even though it isn’t mentioned on their website anymore. Similar with Bad Elf products - no mention can be found, so assuming CEP. The DUAL XGPSS160 does communicate their device accuracy as 2.5m CEP. On the high end, Geneq states the accuracy in many variations, including RMS, R95, 2DRMS, and CEP. Thank you DUAL and Geneq for helping and understanding not all accuracy statements are the same!

** Although all 3 Bad Elf products appear nearly identical (except $$), there are some major differences and justifications for the variation in price that just aren’t as apparent from the chart. The BE-GPS-3300 does have the ability to get 10-50cm post processed (“Coming soon via SDK and external apps” ~ Bad Elf website), something that many other devices don’t offer. The accuracy listed in this table is based on how we would use the device based on requirements (real-time with Collector, etc). The BE-GPS3300 1.0m to 2.5m range is based on the website description: ”< 1 meter stationary accuracy with SBAS+PPP” and “2.5 meter accuracy while in motion”. I’m not sure if Collector can take advantage of the 1m listing or not, but 2.5m should be a breeze.

The BE-GPS-2200 (GPS only) modal didn’t meet our GPS+GLONASS requirements but was included in the table since it may help others. My previous tests of GPS vs GPS+GLONASS make me feel confident that the addition of GLONASS is worth the cost.

Although the Garmin Glo has the lowest accuracy of the bunch at 3m, it’s meets all other requirements. We’ve paired this unit with a laptop and toughbook but running ArcPad, and with Collector on an Apple device. Works with all units without issue. There seems to be two different bundles available for the Glo: Includes car charger; or includes belt/hat clip. The clip is also very light and clips to a baseball cap or shoulder strap for maximum visibility.

The Geneq iSXBlue II is the stand-out winner for both price ($3300) and accuracy. It is highly configurable and includes the ability to select which NMEA strings (GPGGA, GPGLL, GPGSA, etc) to enable/disable and at what update rate (1-10Hz). It comes with an external antenna that can clip to an extension rod, baseball hat, etc. The main receiver can be placed in a backpack, or strapped to your belt. The best setup I’ve seen is to have a small backpack with the receiver inside, with a telescopic mounting pole attached with the antenna. Also allows the iPad to be stored in the backpack when not needed. The price is relatively expensive to the other devices listed, but overall this is still a much more cost effective solution compared to the old GeoExplorer at the same specs. However, today’s Geo’s can get down to 1-10cm with real-time correction services. If you need this level of accuracy in your handheld device, check out the latest Trimble (AKA: Cansel in Canada) devices.

Final Thoughts

This article touches on External GPS options based on a specific set of requirements. If 5m accuracy (CEP) is all you require, you might not need to purchase an external device. If you want 2.5 - 3.0m accuracy (CEP) the Garmin Glo, Bad Elf GPS Pro+, and DUAL XGPS160 are all great choices. For 30cm, the best device will set you back a little more - but it can still be leveraged by all your Bluetooth devices and software. If you need better than 30cm, you shouldn’t be clipping an external GPS to a baseball cap or backpack anyway. Many high-end devices do have Bluetooth capabilities, but not all use the Apple protocol (yet). Overall, there is no perfect device for all situations.

The Bad Elf series has a LCD screen, Datalogger with Barometric Sensor, storage and config options, while the Garmin and DUAL devices are both super light and can clip to a baseball hat with ease (no config also means, it just works).

When working with other laptops/toughbooks, the Bad Elf still requires an iOS device for configuration. If you don’t have access to an iOS device… seriously, who doesn’t know at least one person with an iOS device?! As for Bad Elf and the NMEA protocol: The Bad Elf 2300 does have NMEA GPS Output, but it does state ”USB connectivity to PC or Mac provides streaming NMEA GPS data”. Again, not a show stopper, but I’m assuming this means NMEA isn’t available via Bluetooth to PC/Mac directly. Also means it wouldn’t pass our specific requirements. If you don’t need Bluetooth connectivity to a toughbook/laptop with NMEA, this shouldn’t be a concern.

Higher accuracy will cost you - but a real benefit is that external GPS devices can be paired with many different OS/software depending on the need. If 30-100cm is your goal, the Geneq iSXBlue is a great all-around option. But remember, the Bad Elf might also meet your needs at the 1m level (again, unconfirmed if Collector can leverage this spec).

On the lower end, I still recommend an external GPS to pair with your iPad. It seems like you are only getting a little additional accuracy, but remember the location of the GPS. If you are holding the iPad close to your body, you are blocking more satellites. Having a tiny GPS on a hat or backpack increases your chances to see more satellites - and if canopy cover is an issue for you as it is me, we need all the help we can get.

More Information on Devices:

W3C HTML5 Recommendation Complete!

Wed, 29 Oct 2014 09:11:31 GMT

It’s time to rejoice - the W3C HTML5 guessing game is over! After 17 years of HTML4 we have reached a new milestone as The World Wide Web Consortium (W3C) published their official Recommendation of HTML5. For those who have been developing on HTML5 for years already, we finally have some official standards to base our development on. Better yet - the web browser industry can finish implementing the standard.

HTML5 has brought us new elements such as <video> and <canvas> which are changing the way users interact with the web. User Experience (UX) is so important to web development these days - this announcement couldn’t come sooner.

We expect to be able to share photos, shop, read the news, and look up information anywhere, on any device. Though they remain invisible to most users, HTML5 and the Open Web Platform are driving these growing user expectations. ~Tim Berners-Lee, W3C Director (excerpt from W3C news release).

Here is a link to a video from W3C on Vimeo to celebrate this special occasion.

Now everyone go visit their parents and upgrade their Windows XP computer. Better yet, just recycle it and get them a tablet with a decent browser App (Chrome/FireFox).

More information:

W3C: HTML5 Recommendation
TechCrunch: W3C Declares HTML5 Standard Complete
Microsoft Opentech: W3C Charts a Course for the Future of the Open Web by Finalizing HTML5 Standard

ArcGIS for Server Long Tokens

Mon, 27 Oct 2014 09:30:36 GMT

I recently wanted to use a secure ArcGIS for Server map service with a new proof-of-concept (POC) web application. The service was secure, but I want to allow public access to the application without any credentials required by the user. I could take care of this with a username/password in the Esri Resource Proxy; or have the application request a token with a user/pw saved in JavaScript (bad idea), but I was hoping to keep it simple with a “HTTP Referer” token. However, when visiting the token request page (http://domain/arcgis/tokens/) the longest request I could make was for 1 year. Not that I needed a longer token for this POC, but now I was intrigued. How do I create an ArcGIS for Server long token?

I’m sure we all hope our sites will survive longer than one Super Bowl - but can we create a token for more than a year? Since the ArcGIS for Server Long Token request GUI really just helps the user populate and submit a REST request, why not just submit our own in its place with a longer Expiration setting?!

Yes, we need to be careful of HTTP Referer Spoofing and other questionable external attempts to our services, so putting the resulting token directly in a JavaScript file isn’t the most secure method. For more rigorous security you can put this long token in the Esri Resource Proxy configuration file for a great approach to both security and longevity. The proxy can even be configured with it’s own HTTP referer settings, rate limits, and additional authentication options if you so choose.

Check your configuration:

The ArcGIS for Server long token request is straight forward, but there is one catch (there always is): ArcGIS for Server has a maximum lifespan of long-lived tokens (this is a good thing). Even better - this is configurable! To set the maximum length for all long tokens, just log into ArcGIS for Server: Manager > Security > Settings, and you can set the length of time both the short and long tokens.

The long token lifespan is set in number of days, so for this example, we will set the number to 3654 to cover 10 years (to account for leap years and some rounding). Once the setting is updated, we can start making our longer token requests.

Generate the Token:

With the username, password, and HTTP referer site information ready to go, you can now make the token request by opening your web browser and typing a URL:

http://domain/arcgis/tokens/request=gettoken&username=USER&password=PW&clientid=ref.https://www.spatialtimes.com&expiration=5261750

You might notice the expiration date doesn’t look the same as the config setting. While the config setting is in days, the token expiration parameter is in minutes. Once complete, a successful request will return a new token in your browser window. Copy this token to your Proxy configuration or JS file and you are off to the races. Might be a good idea to put a reminder in your 2024 calendar to update the token just in case, hopefully you aren’t already busy that day.

Additional Information on ArcGIS for Server Long Tokens:

ArcGIS Server Help: About ArcGIS Tokens
ArcGIS Server Help: Acquiring ArcGIS Tokens
Esri JavaScript API: Working with Secure Resources
Github: Esri’s Resource-Proxy page

ArcGIS Javascript API 3.11 Released

Mon, 06 Oct 2014 17:00:05 GMT

Today Esri announced their latest release of ArcGIS Javascript API 3.11. Notable changes/updates include:

New Heatmap Renderer for feature layers (as a beta item).
Measurement widget enhancements including updated events and methods. This sounds like a minor thing, but has lots of potential.
Lots of performance enhancements to projecting features, InfoTemplates, Web Optimizer, API enhancements, and bug fixes.
And most notably, the “what’s new” screen shots are all taken from a Mac. And they thought we wouldn’t notice 😉

There are many more new and enhanced features in the ArcGIS Javascript API 3.11 release. For the full list of updates, jump to the What’s new in Version 3.11 page. FYI - some of the new samples aren’t listed in the Table of Contents yet, but you can still find them from the API Reference Section, use the ”Search for samples that use this class” button.

ArcGIS for Server, Security, and Bash Vulnerability

Wed, 01 Oct 2014 12:36:42 GMT

In light of the recent announcements on the bash (AKA: Shellshock) vulnerability, a quick update on how your ArcGIS for Server might be affected.

Bash/Shellshock

First, some good news if you if you are running your Esri installation on Windows: you are not affected. This vulnerability only affects the Linux/Unix operating systems, and at this point it means relatively few installations will be impacted. However, don’t stop reading yet, more security vulnerabilities for ArcGIS Server installations are discussed below.

Now if you are one of the few (but growing group) using Linux/Unix, you are probably impacted in some way by this vulnerability. If your distro is hosted on Amazon’s EC2 environment, rest easy - you have a team of EC2 employees who have been working on, and patching the environment. In fact, it’s already patched for any new instances you create. If you customized an AMI a while ago, or you aren’t sure if the patch is applied, check with Amazon here to see how to fix it.

If you have a locally hosted Linux/Unix install, I’m going to assume you already know how to patch it since, well, you already installed the OS and configured ArcGIS for Server in the first place (you already know what you are doing)! OK, just in case: update Bash and you are good.

Lastly, if you are using ArcGIS Online, hosted services, or Portal for ArcGIS - No worries, Esri has already released a communication mentioning how these services are not affected.

FYI, other recent vulnerabilities

Just in case you took the summer off, a few additional security issues have been identified in the past few months - not getting the same fan-fair as Bash (since limited to Esri, not Operating Systems):

1. A vulnerability that could allow unauthorized access to secured resources, as well as some cross-site scripting (XSS) vulnerabilities were identified in ArcGIS for Server (and patched) by Esri this August. This impacts all OS versions and releases since 10.1 (including the current 10.2.2 release at the time of writing). More info and download from Esri. Review: Easy and fast to install, no reboot required (<1 minute outage).

2. Another issue, this time limited to Windows and Web Adaptor for IIS (10.1 to 10.2.2) has related patches specifically for the Web Adaptor/IIS config to fix some additional authorization bypass and XSS vulnerabilities. More info and download from Esri. Review: Easy and fast to install, no reboot required although IIS and the Web Adaptor are restarted causing a short outage (<1 minute).

Happy Patching!

SQL Server spatial query, part 2

Sat, 13 Sep 2014 12:56:03 GMT

This post is an extension of ”SQL Server spatial query, part 1”. The first post primarily focused on how the basic query and geometries are put together and the types of results you should expect to see. In part 2, the goal is to wrap some of these techniques into a SQL procedure to evaluate if an input feature intersects a reference layer.

Scenario

Suppose we wanted downloaded some weather data from NOAA and store in our SQL Server database. In many cases this data is available in a vector format (point or polygon), or it’s available in grid/raster format. An efficient way to store this weather information is to store vector geometry only once (with a unique ID), and every time we download the data we store the weather value and the related geometry ID instead of storing the geometry every time we download. This way we can also store all historical data without the bloat of storing the redundant geometry and still be able to perform quick statistical analysis at any location(s) directly in query builder or with a SQL View. Since the ID of each feature from NOAA is unknown (in most cases), or can change, we can use a spatial overlay to determine an ID from our own lookup table (spatial reference layer).

Side note: This also works great with raster/grid data even thought it isn’t directly supported in SQL Server 2008 - I just convert the raster cells to points or polys before running the analysis using a Python script (something for another post).

For this scenario, let’s assume we downloaded a set of polygons in a gridded polygon format and we want to determine if each polygon intersects our reference point layer. If it does - we will store the new value and related reference point ID.

The SQL Server spatial query

Skipping over the many ways you can download data from NOAA and processing into many different formats (Shapefile, WKT, etc), let’s just jump forward to the part where we have thousands of WKT polygons with attributes: a date attribute and current weather parameters. If you need some help getting to this stage, you might want to checkout NOAA’s ”Weather and Climate Toolkit” and use a little Python magic.

Similar to part one we want to compare an input geometry, a WKT polygon, with our reference point layer. As we loop through each polygon record we will send the data to the following SQL procedure:

The procedure includes a “@wType” parameter just in case we want to download multiple types of NOAA data and still be able to use the same proc. Since I pre-process the NOAA download using python, I ca send data directly to the SQL procedure in Python (w/ PYODBC):

MyStatement = "exec prc_NoaaDataLoad '" + str(dt) + "', " + str(val) + ", '" + "Temperature" + "', '" + wktGeom + "'"
cursor.execute(MyStatement)

This code is executed for each row in the NOAA layer, and stores the data for each record (in tbl_NoaaData) when the polygon from NOAA intersects the reference point table. The WHERE clause determines if there is an intersect (=1) versus no intersect (=0). So if the result of the overlap = 0, the row isn’t loaded and we move on to the next. Seems so… simple - and it is.

Even though I’m using Python for this process, I should mention that I’m not using any ArcPy or other GIS module. Just straight python calling a sql server spatial query. This also has the added benefit of running extremely fast (< 6 seconds), and not requiring an ArcPy/ArcGIS license.

More Information

ArcGIS Pro licensing Update

Wed, 27 Aug 2014 16:25:17 GMT

Thought you could delay and/or ignore signing up for an ArcGIS Online (AGOL) Organizational account? Well, it’s going to get a little more difficult to stay away based on a recent ”ArcGIS Pro licensing Update” blog post from Esri. Starting with ArcGIS Pro beta 5 and continuing through the final release, licensing of Pro will be managed through AGOL Organization admin accounts.

ArcGIS Pro follows a named user licensing model, where each user account is assigned permissions to access the software. Starting with beta 5, licenses for ArcGIS Pro are administered through your ArcGIS Online organizational account. This will be the software authorization method used for the remainder of the beta program, as well as in the final release.

We aren’t sure how traditional 9x/10x desktop Concurrent Use licenses will be counted/ported to the new Organizational account licenses, but since it is based on named users (not concurrent connections), there will be some changes.

This will not impact your current 9x/10x ArcGIS Desktop licensing or license manager - just ArcGIS Pro moving forward. Might be a good time re-evaluate signing up for that account now.

More Information:

SQL Server spatial query, part 1

Thu, 14 Aug 2014 17:37:21 GMT

A traditional GIS analysis question - Does the polygon contain the point, is the point inside the polygon? There are lots of ways to use a GIS application to find the answer to this question, but let’s take a look at a few ways SQL Server Spatial can help us with this traditional exercise using a SQL Server spatial query.

The Setup

The first step is to have some polygons/points to test with. In the query example, we will create a polygon (@thePolygon), a point located inside the poly (@pointIN), and a point located outside the poly (@pointOUT). Although this article doesn’t get into detail on the GEOGRAPHY/GEOMETRY types or spatial references - it should be noted that in the first example I changed the standard WGS84 Geographic spatial reference (4326), to use NAD83 Geographic (4269) instead. No real reason, just wanted to mention in case you compare with other samples.

DECLARE @thePolygon GEOGRAPHY, @pointIN GEOGRAPHY, @pointOUT GEOGRAPHY;
SET @thePolygon = GEOGRAPHY::STGeomFromText('POLYGON((-78.50932668617881 45.024933647425115, -78.53403351361905 44.9898648154388, -78.48446979547693 44.97239241709962, -78.45973073293072 45.007441690111115, -78.50932668617881 45.024933647425115))', 4269);
SET @pointIN = GEOGRAPHY::STGeomFromText('POINT(-78.51 45.00)', 4269);
SET @pointOUT = GEOGRAPHY::STGeomFromText('POINT(-65.00 35.00)', 4269);

The SQL Server Spatial Query

Now that the features are created, we just need to use SQL Server’s intersect function (STIntersection) to perform the analysis.

Example 1 - where the point falls inside the polygon:
Query: SELECT @pointIN.STIntersection(@thePolygon).ToString();
Returns: POINT (-78.51 45)

Example 2 - where the point falls outside the polygon:
Query: SELECT @pointOUT.STIntersection(@thePolygon).ToString();
Returns: GEOMETRYCOLLECTION EMPTY

Example 3 - Reversing the order of the Base and parameter in the analysis:
Query: SELECT @thePolygon.STIntersection(@pointIN).ToString();
Returns: POINT (-78.51 45)

When the point is inside the polygon, it is returned by the query, otherwise, an empty geometry is returned. Notice how the result in example 3 returns the same result as Example 1, regardless of which feature calls STIntersection. This is because the STIntersection function only returns the points that are common between the two shapes. In this case, only the one point is common. If we intersected two partially overlapping polygons, the output would return a new polygon of the overlap area.

Aside: Some SQL Server spatial functions can even return a collection of geometries. For example, a union of a line feature and a partially overlapping polygon feature will return a collection containing both lines and polygons.

Here is a full SQL Server spatial script for Enterprise Manager:

Coming soon: Part 2 - Create a SQL procedure to evaluate if an input feature intersects a reference layer. Gives me time to think of a shorter title. Update 2014-09-13: SQL Server spatial query, part 2.

More Information

Python Relative Directory

Wed, 06 Aug 2014 10:00:59 GMT

When using sub-directories to store temp files, analysis outputs, or logs - a nice way to deal with the parent directory when moving the code and subfolders is to leverage the OS module’s current working directory function.

I tend to start developing scripts on my local machine, then move them to their testing/production resting place later in the process. An issue with this is the last minute code modifications to python relative directory variables. Here is a way to setup the parent directory so the relative sub-directories can find their way without any changes if you move the code.

OS.getcwd() returns the full path of the current working directory, which you can then append other directories to.

os.getcwd()
#returns C:\\Program Files\\Python'

For example, you have a script that you want to store all logs in a “log” sub-directory, you could setup the following:

import os
DirMain = os.getcwd() + os.sep #sets the absolute parent directory
DirLogs = DirMain + "log\\"

You can move the script and sub-directories without the need to further modify the code.

The OS module also has another function for a python relative directory, os.curdir(). This returns a simple OS-specific string used to identify the current directory.

os.curdir()
#returns '.'

This is also handy but doesn’t set an absolute parent directory, so you can’t chdir() to it later and expect to land at the original directory. However there are lots of samples online that also use this alternative approach and might work depending on your situation.

Binary flags with Python

Thu, 31 Jul 2014 16:27:14 GMT

Sometimes you just want to run a few specific sections of your python code for testing. Then a few more. Next, all sections. Now run just one. Or possibly need to run different parts of your code based on the time of day or week.

There are many ways you can accomplish this directly in your python script(s) or by using the python window. But what about passing a runtime argument with a little binary?! This technique is a great way to run different/multiple sections (functions) of your code without the need to modify the python script - binary flags with python. I even use this technique to keep some test functions to call them when needed - and also to run minor variations based on time of day.

The method behind this approach is to use binary (base 2 numbers), but not using a long string of 0’s and 1’s. Rather, converting regular long integers (base 10) into binary in order to determine what sections of code to run.

Setup

You have 7 sections of code broken down into functions. Each section is assigned a value - these values are set to the power of 2 (assign the number on the far right side), examples:

Section 0 = 2^0 = 1
Section 1 = 2^1 = 2
Section 2 = 2^2 = 4
Section 3 = 2^3 = 8
Section 4 = 2^4 = 18
Section 5 = 2^5 = 32
Section 6 = 2^6 = 64
and so on…

Usage

Whenever you want to call multiple sections, just add up the values of each section and pass as a runtime argument.

Run section 3 only: MyScript.py 8
Run section 4 only: MyScript.py 18
Run section 1, 3, and 5 (2+8+32): MyScript.py 42

Since you are passing the integer (base 10) number as the argument, we need add a small function to the python script to determine the sections to run. The functions/sections will always be run from lowest to highest (highest to lowest is possible with a small modification to the script).

The code

Using this small function, we can determine which section(s) to run.

def fn_bits(n):
    while n:
        b = n & (~n+1)
        yield b
        n ^= b

Putting it all together

Now that we can pass the commands we want to run as a single argument, we need to put it all together. Here is a working sample of binary flags with python.

Based on the full code sample above, here is how we would call different code combinations:

Section 0 only: BitFlagFunctions.py 1
Section 2 and 3 (4+8): BitFlagFunctions.py 12
Section 0, 2, and 3 (1+4+8): BitFlagFunctions.py 13
Section 1 and 3 (2+8): BitFlagFunctions.py 10

Summary

This is just one of many ways you can setup binary flags with python. I’ve seen other examples where people use “0100101” to run various sections which also works well. The method described in the above gist is just a way that works well for me, giving me the ability to number each section and run them in ascending order as needed.

Mosaic Large Images to a Single Raster in ArcGIS

Tue, 22 Jul 2014 23:07:00 GMT

I’ve had some question regarding large sets of ortho imagery tiles and how you can turn this into one seamless raster (and maintain performance). While the new Mosaic Datasets are great, what if you only have 9.3 - here is one way to merge them together using the Raster Mosaic tools available in both 9x and 10x. “Mosaic Large Images to a Single Raster in ArcGIS” is a case study based on ArcGIS Desktop 9.3 with decent, yet older, hardware - then comparing some different compression options and visual results. Please note, if you have 10x, many newer options exist and I would recommend taking a good look before jumping into this approach (although still a viable option).

The Scenario

Using 7,600 uncompressed 4-band TIFF orthos, each 1km in size and 120MB (approx. 1TB total). You can store the output to a file Geodatabase (can handle huge rasters, lots of viewers) or SDE (adds enterprise access and security). Before you start it’s recommended to try your approach with a subset of data to see how long it will take and if the quality of the results meet your needs.

The Steps [processing time]

Create a file Geodatabase [< 1 min.]
- Even if you are planning to save your raster in SDE, you might want to consider processing the mosaic into a file Geodatabase first. This way you can limit the processing required and run locally for faster results. Copying afterwards does add to the total processing time - but also makes sure there aren’t any issues with an input raster and you don’t need to worry about indexes or logs getting bloated if you make any mistakes.
Create a Raster Catalog: Make it “unmanaged” to refer to images without loading/copying [< 1 min.]
Run ‘workspace to raster catalog’ tool (all TIFFs in one directory) [10 min for all 7k files]
Set the spatial reference properties of the catalog to same as data source [< 1 min.]
Run ‘Raster Catalog to Raster Dataset’ tool: This varies depending on the approach. Two different approaches we tested. Note: Create pyramids afterwards, not as part of the process. [3 days (old hardware)]
- Option 1 - LZ77 compression: Before running the tool, open the environment settings and select not to create pyramids, also set compression option to LZ77. We also unchecked “raster statistics” since we didn’t want the default colour stretching. Raster stats are on by default but leads to the usual question “why does the colour look different now?” after processing imagery (although yes, you can force this off in ArcMap in the layer properties but it resets each time).
- Option 2 - JPEG compression. We decided on 80% compression, no specific reason why but the results were almost as good as LZ77. The quality and pixel size will impact the visual quality - again, best to play with a small sample area to get a better idea of what works for you.
If your final destination is SDE, copy your raster at this point using best practices for loading large data to your enterprise database. When loading to SDE set the environment setting to create pyramids. If keeping in the file Geodatabase, you will need to create the pyramids still - We used the default options and number of levels. You can also try setting Bilinear vs Nearest neighbour to see which you like better.

The Results

Using the method above, both LZ77 and JPG compression options generated outstanding results. The LZ77 (lossless) compression has a slightly better visual output at the cost of extra hard drive space. If your tests show little visual difference, I would recommend JPG compression to save space and network load - however, the clarity of LZ77 is hard to beat, especially if you have a decent network.

File Size: When using LZ77, we found the output size was the same as the total input size (1TB). The compression helped, but adding pyramids added back the savings. JPG with 80% compression was 350GB (including pyramids). Advantage JPEG.
Load times: Load times were comparable for LZ77 and JPG, but if your network is slow or constantly under heavy load it might make a difference since LZ77 using more bandwidth based on file size. Small advantage JPG.
Visual output: Most people don’t have an issue with JPG quality - just don’t show them LZ77 and they won’t know what they are missing. Depending on the landscape, there can be some differences that are hard to overlook. Overall, there is a difference when using lossless compression - your intended use of the data will help you determine which is more important. Advantage LZ77.

Overall, a small advantage to JPG compression - but depending on the raster, LZ77 definitely has its place.

A few lessons learned

Test, test test. Pick a small representative area and run many different tests until you find the right settings that give you results that meet your specific needs.
Overall, a good rule of thumb is that pyramids will add 33% more storage space.
Ensure that you don’t build pyramids until the very end of processing (i.e. when uploading the last tile or in a new process after the entire load process is complete). By default ArcGIS will rebuild pyramids for the entire raster dataset with each new tile. This adds an excessive amount of processing time.
Use JPG (75-80) compression unless you determine a requirement for LZ77. To me this is the best tradeoff for image quality vs file size/performance. SIDs and JPEG2000 need to be decompressed on load which adds to draw time. LZ77 is perfectly good but it might be a bit slower than JPG because of the larger network load with most users not seeing any difference.

Final Thoughts

The goal of this case study was to try various methods of creating raster mosaics as well as reviewing compression and visual outputs on a large scale. Both of the compression types listed worked great. Other approaches and settings were also tested but they didn’t meet our final requirements or the compression options forced applications to decompress which can create some additional client load.

There are many other approaches including Mosaic datasets or using ArcGIS server and creating a tiled service. This is one approach that works very well if you are using 9.x and can be leveraged by both server and desktop applications - and yes, even 10x. This approach also gives you the option to store the data inside an Enterprise or file Geodatabase.

Esri User Conference Q&A

Mon, 14 Jul 2014 20:16:40 GMT

It’s that time of year again… when 12,000 people (approximately) head to San Diego to refill their Esri Kool-Aid and see what’s coming down the pipe from the Redlands, California GIS software giant. It also means it’s time for the the annual Esri User Conference Q&A that goes over some tidbits and direction that Esri is moving towards over the coming year. Basically, it highlights some of the products, and new features we should be ready for.

Esri has written questions and answers to keep our customers informed on our efforts in software development, products, education, and support; future plans in these areas; and our thoughts on GIS and the industry as a whole. Our purpose for sharing this information is to help our users be successful in their use of GIS. Please feel free to share with your colleagues.

Some interesting things to note:

Yes, there is a 10.3 and it will focus on “improving the science of ArcGIS”. This includes some space-time tools and ability to work with new datasets (netCDF4, HDF, GRIB)
Q4 2014 is when ArcGIS 10.3 and ArcGIS Pro be released (together).
A big change in ArcGIS Pro will be the upgrade from Python 2 (2.7) to Python 3 (3.4.0).
There is an attempt to describe the new licensing model for Pro, but hard to interpret from the limited Q&A page to know what this really means (how concurrent licenses are handled, etc).
Lots of Geodatabase updates. From support for SQL Server 2014 (at 10.3) to the future of SDE command line access.
And many, many more - worth taking a look, then returning to the Esri site to watch some of the videos posted from the conference.

Check out the full Esri User Conference Q&A for more details: http://www.esri.com/events/user-conference/q-and-a.

Useful GIS Developer Tools

Sun, 29 Jun 2014 19:47:42 GMT

Looking for additional tools for your coding toolbox? Here are 4 useful GIS developer tools you may or may not know about. Oh, they are also FREE!

Notepad++

My go-to for Javascript/HTML and Python coding, and completely replaces any need for Windows Notepad (and can handle much larger files). Notepad++ isn’t a full IDE by any means, but it does the job in most cases. It has great/customizable syntax highlighting, some auto-complete abilities, app/browser launching features, and lots of plugins for various other tasks. A default install is enough for some, but by taking the time to review additional settings (default encoding, auto-backup, auto-complete, interface style customization - to name a few) you will see why this is a popular choice for many.

Some of the little things Notepad++ has will make it hard to use other notepad programs: highlight the corresponding open/close tag when you click on either tag, and also allows the collapsing of sections based on tags or indentation (python). Line commenting (CTRL+K) and uncommenting (CTRL+SHIFT+K) lines of code is also very handy and it understands the current language commenting syntax. On the topic of languages, there are tonnes supported by default, but you can also add custom styles, and add additional languages if yours is missing - such as Github markdown.

A few great plugins include:

Compare: Side-by-side file comparison with every line highlighting for new/deleted/changed content.
DSpellCheck: Simple spell checking options.
Tidy2: A plugin to add easy HTML tidying to Notepad++ with various config and pre-set options.
JSLint: A plugin that allows users to run JSLint (The JavaScript Code Quality Tool) against their open JavaScript files.
More great plugins are listed at Backslash.gr and Lifehacker.com.

7-Zip

If you’ve never heard of this application, be warned: You might be upset you haven’t heard about it sooner. 7-zip works with many formats you use every day: ZIP, DMG, ISO - and it also works with 32 other formats you may deal with from time to time: RAR, TAR, GZIP to name a few. 7-Zip also includes an open architecture, a “7z” format with LZMA/2 compression, AES-256 encryption, and Petabyte size support.

So why mention 7-zip when discussing developer tools? Well, beyond being a great compression/extraction tool, it also includes full access via command line! With two different ways to access: A limited functionality EXE; or full featured EXE+DLL, you have all the same features as the desktop GUI. These command line options are well document and work great with Python and .NET projects. You can create a copy of the EXE+DLL into any python directory to have a portable copy of python code with no installation steps required of the 7-zip tools. PS. It’s only a 1 mb download.

Python

Already mentioned a few times, Python is a great development environment for any size project. It’s included with many popular GIS software applications and can be extended way more than the custom scripting languages from the days of old (looking at you AML and Avenue). Python is available in both 32/64-bit, and runs on over 21 different operating systems and environments.

Python’s strengths start with an amazing support community and extend to the number of supported environments, easy to read/write/understand the language and syntax, and the module design. For the Esri community, Python has become an integral part of the ArcGIS architecture (both desktop and server thanks to a growing number of ArcPy functions).

As for notable modules, two always jump to the front of the for me:

PYODBC: Allows you to use ODBC to connect to almost any database from Windows, Linux, OS/X. It also simplifies database connections while built on the Python Database API specs. Define a connection string or ODBC connection to your favorite database and you are off to the races.
rPython: Adds the powerful R statistical computing and graphics programming objects into the Python environment. There are some great demos using Esri’s Spatial Analyst extension, and passing the output to R for additional analysis.

github and github for Windows

Are you currently using a revision control / source code management system? If not, Git is a great option. No longer will you need to make copies of your code with cryptic names and track each one by yourself.

Git is a very powerful revision and source code management system (command line focused), with GitHub and GitHub for Windows making it super easy to get started with a great GUI and little or no command line required. You can work on local repositories with GitHub for Windows as well as complete synchronization with your online GitHub account. GitHub also puts you into a community where other developers can add/enhance/fork your code to help move ideas and code forward. Many companies are also posting code snippets and applications online with open licensing to build their communities and development environments.

Shapefile vs Feature Class

Tue, 10 Jun 2014 08:44:26 GMT

Walking down the hallways of a GIS conference you hear lots of great debate and geek-talk on a variety of topics. A Shapefile vs Feature Class discussion caught my attention the other day at such an event. This can be a simple or difficult question to answer depending on the way you interpret it. This isn’t really a comparison on Shapefile vs Feature Class because as we will discuss, this isn’t an apples to oranges comparison.

Let’s start with the Shapefile: A Shapefile is a Feature Class. Although the intended question probably wasn’t answered here. But let’s go a little deeper into this before moving on, so what is a Feature Class then (from: http://wiki.gis.com/wiki/index.php/Feature_class?

Feature classes are homogeneous collections of common features, each having the same spatial representation, such as points, lines, or polygons, and a common set of attribute columns, for example, a line feature class for representing road centerlines.

By this definition, a Shapefile is a Feature Class. Back in the 1990’s, the Shapefile was referred to by it’s full name - the Shapefile Feature Class. Think of “Shapefile” as it’s Given Name, and Feature Class as the Surname. If we always referred to it’s full name, Shapefile Feature Class, the comparison (or lack of) would not need to be discussed. This also helps explain a Geodatabase Feature Class. The Given name for both, really describes the Workspace type. Hopefully a Workspace in ArcGIS python or ModelBuilder makes more sense now too.

This brings up another discussion on the Shapefile vs Feature Class topic. What is there to compare with a Shapefile Feature Class and a Geodatabase Feature Class? There are many similarities, since they are both collections of common features having the same spatial representation (Point, Line, Polygon, etc). The differences are mostly in the additional types and behaviours available to a Geodatabase FC. Not going to list them all, but domains, topologic relationships (geometric networks, terrains), and the ability to store high precision geometry are a few differences. And there are many Geodatabase storage options: Access (AKA Personal), File, and Enterprise (AKA RDBMS) which can have subtle differences as well. Whereas a Shapefile is always stored in a file folder and is more of a universal format support by many [more] GIS and CAD systems.

Both shapefile FC and GDB FC have their place. Your requirements and constraints will most likely have you working with both at some point. Here are some additional links on the topic:

ArcGIS JavaScript API Web Optimizer beta released

Fri, 23 May 2014 11:31:27 GMT

The ArcGIS JavaScript API Web Optimizer is a web application that will generate custom or lightweight builds of the ArcGIS JavaScript API. You can pick and choose the component you want, then generate your custom/slim version of the API. You may have heard about this at the Esri Dev Summit this past year, but things went silent for a while after that. However, a recent blog post by Esri has surfaced with a link to the open beta site.

Once a custom build is created, you can download a copy or have it hosted on js.arcgis.com (with an Organization or Developer account). The Esri CDN is still available for both the regular and compact builds of the API and this new offering expands the options available to support growing variability of Dev requirements (based on developer requests I’d imagine).

Why might you want to use this approach? Esri’s documentation explains some benefits:

The Web Optimizer offers an alternative to the “one-size fits all” builds that Esri offers via the CDN (js.arcgis.com/3.9/ and js.arcgis.com/3.9compact/) or as a download. This works well for development of smaller web applications. But as applications grow in size and functionality, this approach breaks down. When an application uses a large number of modules not included in the available builds, applications generate more http requests, download more JavaScript and take longer to load. A custom build combines the required modules which reduces the number of http requests and application load time.

I wish this was around in the ArcGIS JavaScript API version 2 days! Back when smart phones and tablets had limited memory and bandwidth. Moving forward I would think new devices are able to handle the standard/compact CDNs while leveraging the benefits of pre-caching, distributed servers, and bypassing old browser single-domain download concurrency limits (default is 5).

On the other hand, long wait times (and progress bars) for [down]loading applications is a UX no-no. The Adobe Flash loading bar, and Silverlight’s Blue Circle-of-Wait rank up there with the Windows XP 99% complete progress bar - why does the last 1% take forever?! I can’t get a coffee every time I see these - I drink enough java as it is! The ArcGIS JavaScript API Web Optimizer is one way to combat such issues. Wouldn’t you choose the shorter line at the grocery store?! Log in and try the beta for yourself.

ArcGIS License Manager Settings

Thu, 15 May 2014 11:45:01 GMT

This post will cover 3 aspects of ArcGIS License Manager:

Using ArcGIS Desktop 9.3x and 10.x with License Manager 10.x
ArcGIS License Manager common firewall issues
ArcGIS License Manager options for reserving licenses/extensions for specific users/groups

1. Using ArcGIS Desktop 9.3 with License Manager 10.x

If you use the default setup of ArcGIS License Manager 10.x you might notice that 9.x ArcGIS Desktop clients can’t connect by default. This is due to a minor change in how ArcGIS Desktop uses ports when connecting. At 9.x, Desktop will try to contact LM using port 27004, however, LM 10.x is setup to use a port range of 27000-27009 instead.

To fix this issue, open the service.txt file located in the LM install directory (C:\Program Files (x86)\ArcGIS\License10.2\bin). I recommend making a backup first. The first line will typically read SERVER this\_host ANY. Change this line to read SERVER this\_host ANY 27004. This will force LM to use port 27004. Both 9.x and 10.x will be able to connect. Don’t forget to restart LM for the changes to take effect. Test.

2. ArcGIS License Manager common firewall issues

Firewalls are typically your friend. However, they also can’t read minds to understand what ports to keep open or closed. Once your License Manager is setup, the firewall configuration is fairly straight forward if you are having issues connecting from a different machine. ESRI provides these steps to work through the firewall. Below is a quick summary:

Before configuring the firewall, you will want to update the same service.txt file mentioned in part 1. This time, update the line VENDOR ARCGIS to use a specific port, such as VENDOR ARCGIS PORT=1235. This specifies that the vendor daemon (ARCGIS) is to use the port 1235. Don’t use any of the ports in the 27000-27009 range since this is the default range for FlexNet lmgrd. Again, don’t forget to restart LM - or stop LM before updating and start afterwards.

To set the firewall options in Windows Server 2008, open the Windows Firewall with Advanced Security application. On the left panel/tree, click the Inbound Rules. In the Actions area click the New Rule… button. On the General Tab, provide a rule name such as ArcGIS10LMPorts. On the Protocols and Ports Tab set the following:

Protocol type: TCP
Local Port: Specific Ports; 1235, 27000-27009
Remote port: All Ports

The local port settings in this example are set to the full range of lmgrd. If you are also implementing Step 1 (using with Desktop 9.x) you can just set the ports to include 27004. The image here uses a third example, setting the range to 27000-27004. After your ports have been set, save the firewall rule and make sure it is enabled. Test.

3. ArcGIS License Manager options for reserving licenses

There are a few cases where you might want to reserve an ArcInfo (AKA Advanced) license for admin purposes, or possibly an extension license for a small group of users. To accomplish this, another edit is required to the service.txt file to point to an Options file with additional configurations set. In the options file we will create a User Group to contain a list of staff, and specify what license can be reserved by the group. Once complete, this license will not be available to users outside the group. If one user in the group is already using the reserved license, other group users can still grab a general license from the pool. The example steps through reserving 1 ArcInfo license for an admin group):

Create a new file called ARCGIS.opt (or whatever name you wish to use) in the same directory as the LM service.txt file and open in your favorite text editor.
In the ARCGIS.opt file, create a group called AdminGroup and specify users:
- Create the group: GROUP AdminGroup
- Add users by OS login (no commas, just spaces to list users): BryanMac AlGore BarnyRubble
- The full line will read: GROUP AdminGroup BryanMac AlGore BarnyRubble
On a new line, specify the license to reserve one (1) ArcInfo for the AdminGroup
- RESERVE 1 ARC/INFO GROUP AdminGroup
Save and close the option file. Note: you can also add comment lines to the file using the hash/pound (#) symbol
Setup LM to use the newly created option file:
- Open/edit the service.txt file
- Update the line reading VENDOR ARCGIS PORT=1235 to point to a options file, in this case called ARCGIS.opt: **VENDOR ARCGIS PORT=1235 options=“C:\Program Files (x86)\ArcGIS\License10.2\bin\ARCGIS.opt” **(don’t forget the quotes)
Save the service.txt file and restart License manager
Test

The final results

Here is an example service.txt file if using all of the steps above:

SERVER this_host ANY 27004
VENDOR ARCGIS PORT=1235 options="C:\Program Files (x86)\ArcGIS\License10.2\bin\ARCGIS.opt"
INCREMENT ACT ARCGIS 1 permanent 1 qweryqweryqweryqwery\
vendor_info=qweryqweryqweryqwery

And the final ARCGIS.opt file:

\# Reserve one Info/Advanced license for administration (also added timeout)
\# To return any hung license after 1200 seconds
TIMEOUTALL 1200
\# Setup the User Group
GROUP AdminGroup BryanMac AlGore BarnyRubble
\# Reserve an ArcInfo license for the group
RESERVE 1 ARC/INFO GROUP AdminGroup

Links for additional information

Map of the Day - The New York Times

Wed, 07 May 2014 09:22:58 GMT

A great web mapping example. The New York Times Crime map has been around for a while now (June 18, 2009), but it’s still worth discussing. I sometimes dive into my browser bookmarks just to find it when I need some inspiration. Why? It gets so many things right! The topic is a little sensitive (homicides in New York) but the way the information is communicated on a map is very well done.

With 10 ways of categorizing the data and the ability to filter the results by year, a visitor to this page can drill down into the details and interact with the map without requiring a GIS diploma. As well, the map fits into the existing page design, becoming an interactive newspaper article, not a GIS map. From a GIS point of view, the 10 categories are just layers that could be in a table of contents (TOC) - but the map isn’t made for GIS people. The TOC has been transformed into an interactive gallery that communicates information to the user: Number of homicides by year; Stats on previous year variation (using a variety of fonts/sizes to guide the user); Change the type of information being visualized. Even the Legend control doubles as a chart, showing statistical breakdowns and colour coding.

This map could be used as an image in a traditional newspaper that landed on your driveway. It would still communicate a subset of the information to the reader (albeit no additional interaction). A subset of the news article and the header/footer of The New York Times is still present, and a link to the full article is also included.

GIS on the Web

Some in the industry, including myself, are trying to change how we use GIS on the web: Create for the end user, and design to incorporate into your site based on the subject matter. Why are we still making “map portals”? People don’t make “Chart portals” to show off the entire organizations’ list of bar-charts and spline graphs. These are usually embedded into the subject matter pages directly (ROI graph on an investment page) – providing a way to communicate data/information that might add to the subject. Maps (apps) should be viewed in a similar way when there is a focus/purpose to map. The world of GIS, might not be completely ready for the transformation away from map portals, but something we should keep in the back of our minds when we have our next request for map website.

The Details

Rant complete, back to the Crime Map that has many subtle features as well: The grey-scale basemap as to not grab your attention, and the additional sentence to let you know how much data isn’t shown (example: “Map does not show 1 location that the police have not released”). The address and ZIP search box is still there for those want to find out if they are living in a hotspot , but it’s not the focus of the application or the news article. The time slider is also very useful and changes the views and statistics as you interact - long before this functionality was easy to implement with traditional tools.

The New York Times has produced many great maps like this one. They find ways to communicate the information based on users, the subject, and data. Well done NYT.

Update (May 07, 2014 @5:10pm est): For those interested in a New York Times map that doesn’t use flash, checkout the JavaScript Health Department Restaurant Ratings Map.

ArcGIS 10.2.2 is here

Wed, 30 Apr 2014 23:09:44 GMT

With very little fanfare, ArcGIS 10.2.2 was released last week. Don’t be fooled by the lack of attention, this is an important update to your ArcGIS 10.2x install (or 10.1 if you were able to deal with some hiccups until now). This release is more of a Service Pack with lots of bug fixes and can be installed on top of an existing 10.2 install and is highly recommended. There are a few new features, including support for the OGC GeoPackage which will be further developed through 10.3. That’s right, there will be a 10.3 release.

This is an “easy” update, but always remember to read the upgrade/migration documentation for your current configuration. Things get more complicated when the entire Esri stack of software is being upgraded on a single machine, or when moving from older versions (9x or 10.0).

So will 10.3 (or 10.3.x) be the final nail in the ArcGIS Desktop 32-bit coffin? It would make sense that it would be the end for this cycle since ArcGIS Professional will be hitting the scene later this year in overlap. Esri probably doesn’t want to maintain two desktop streams for very long (makes sense). Anyone else having a flashback to the ArcView 3.3 /ArcGIS 8.0 overlap days?

Feature Class export into native SQL Server table

Fri, 25 Apr 2014 17:23:54 GMT

Not too long ago, the only way to store spatial data inside a database was to have a custom/binary format managed in code or by software (such as ArcSDE). There have been huge strides forward in the database world - with many platforms now incorporating native spatial types. Oracle Spatial, SQL Server’s Geometry/Geography, to name just a few. These spatial types might still have some limitations when using in combination with GIS software, but it is now much easier to ETL this data into native formats for other use.

You can have the best of both worlds as well: Using native spatial types and still leveraging Geodatabase objects - however this post deals with extracting data and loading into a native table/spatial type only. You can use ArcCatalog for the Geodatabase stuff.

Scenario

Have a [File|ArcSDE|Access] Geodatabase Feature Class and want to load into a new SQL Server table with the Geography spatial type using Python. In this scenario it will be a polygon layer, but the example can be expanded to allow other feature types as well.

Steps

Create the SQL Server table. In this example I used SQL’s Enterprise Manager to create a table named SpatialTable with 2 fields, one for a unique ID and one to store the Geography:

CREATE TABLE SpatialTable ( id int IDENTITY (1,1), GeogCol1 geography );  
GO

Read the features and load into SQL Server using python. A python script leveraging ArcPy’s data access module is used to cursor through the features while exposing the shape/geometry in the well-known text (WKT) representation for OGC geometry (usable by SQL Server). This is one line of code using ArcPy! Next, to load into SQL Server, a Python module called pyodbc is used to connect to the corresponding database.

Note: SHAPE@WKT tokens were made available at ArcGIS 10.1 Service Pack 1

You now have your data stored in a native table using the Geography format.

More information

Open Data Institute Coming to Waterloo

Tue, 15 Apr 2014 22:28:06 GMT

The Federal Government of Canada’s Economic Action Plan 2014 recently announced it will be providing $3 million to Canadian Digital Media Network (CDMN) for the creation of an Open Data Institute in Waterloo.

To help ensure that Canada captures the commercial opportunities presented by Open Data, Economic Action Plan 2014 proposes to provide $3 million over three years, from the existing resources of the Federal Economic Development Agency for Southern Ontario, for the creation of the Open Data Institute, to be based in Waterloo, Ontario.

Many organizations have also stepped up to contribute to the institute - OpenText, University of Waterloo, Communitech, and Desire2Learn, already coming forward to provide support.

This announcement comes just days after the Fed’s completed the largest and most successful Canadian open data hackathon, the Canadian Open Data Experience (CODE). It was a great way to help showcase a variety of ideas and business opportunities with open data (new datasets were also released for the event). It’s a great start for the Government, and hoping it leads to more data, formats, and web services in the future.

Side note: Someone should inform Canada Post what the rest of the Federal Government is up to - as they are still pursuing legal action against Geocoder.ca for creating their own PC file. Maybe we just found the next candidate for an open dataset?!

ArcGIS Javascript API 3.9 Released

Thu, 10 Apr 2014 12:26:10 GMT

Esri just announced the latest release of ArcGIS Javascript API 3.9. Notable updates in this release are the new CSV Layer and a new symbol style for continues color (colour) ramps.

A full list of new features for the ArcGIS Javascript API 3.9 release can be found in the on-line documentation.

It hasn’t been that long since 3.8 hit the scene back in January - I think this shows how important the JavaScript is to Esri these days.

Another GIS Blog?

Wed, 02 Apr 2014 22:30:09 GMT

Why create another GIS related blog? We sometimes ask ourselves the same question. This blog is just as much for us as it is our [potential] readers. The goal is to share interesting news, ideas, and solutions related to the world of GIS. We get a place to store our information and share it with you at the same time.

As primary author, Bryan gets together with a small team of professionals to discuss various spatial topics over a pint or coffee. Some of these discussions end up making it here to this site. The team consists of a few GIS consultants/DBAs/Devs/Architects who like to share what they’ve learned over the years – and so they don’t forget what they’ve learned either. Sure there are 5 ways to accomplish any GIS task – we just provide some detail into which method(s) worked for us. That said, our methods and approaches will change as technology does. The right answer today might not be the best tomorrow or perfect for every situation – I guess that means there is always something to share.

There are lots of great resources out there – and when we discover or create a great workflow, development technique, or cool map, this will be our place to share and discover with you.