Metadata on GitHub

As anyone who has ever worked with public data can attest, metadata is a critical component of any data set. Without metadata, the person attempting to apply the data to an analysis is left fumbling to decode archaic values and codes;  guess on data accuracy and infer any possible limitations of the data. I could go on about the importance of metadata but that is not the focus of this post.

Quite simply the purpose of metadata is to provide a detailed understanding of a data set. Coverage includes details on origin, accuracy and the definition of columns and the values used. There are different metadata standards (FGDC, ISO) and formats (XML, HTML) that are in use. All of which serve the public well; however there are certain limitations that we looked to address.

First, the layout of metadata with its system of tagging is more geared toward machine reading and not human consumption. In line with this machine reading focus, metadata lacks visual examples and a more descriptive treatment geared towards a human.

A second shortcoming is the use of file-based metadata that can get out of synchronization with the data. This can happen due to mistakes in the metadata that require correction or updates to the data.

Although not an issue, it is also beneficial to have all of the metadata in a single location that also provides access to the data and service endpoints.

To address these shortcomings and to provide additional usability benefits, we are providing supplemental metadata on GitHub. We first released supplemental metadata on GitHub last year for our planimetric data. Due to the positive reception, we are now extending this for all our geospatial data. To view, click the link below.

https://github.com/CityOfNewYork/nyc-geo-metadata

 

Make your own map of NYC

Updated links and added new data set.

NYCityMap & Beyond

So you want to make a map of NYC. You are familiar with the NYC Open Data Portal but have had some difficulty in finding all the data you need. You have the requisite software to construct the map (e.g., QGIS, ArcGIS, MapBox Studio, etc.) and the basic skills to do so. Perfect, read on.

This post is intended to provide a single source for the map ingredients. It is not intended to be the map cookbook. It’s up to the reader (i.e., mapper) to decide on data to include, colors, symbols, labels, fonts, etc.

The table below lists NYC DoITT geospatial data on the Open Data Portal and provides a very basic description of each data set. For more detailed information, see the included metadata. The table is grouped into core basemap features and other DoITT data.

The first grouping of data comprises the data used to construct…

View original post 438 more words

NYC Projected

Much has been written about map projections and coordinate systems. I will not attempt to add to what has already been well covered by those with more technical expertise and writing skills than myself. If you are interested though the USGS is an excellent resource. You can start here.

I recently had a thoughtful email discussion with a colleague who questioned our (NYC DoITT) use of the New York State Plane coordinate system in this era of web mapping. This got my thinking. Did I really know why the choice was made? Do we just continue to use this coordinate system because we always have?

Unfortunately, none of the individuals that were involved in the first digital mapping effort to fly the City and capture planimetric data (more here) are around anymore. And there is no documentation on the reasoning that went into the decision. However it is fairly straightforward to work backwards to understand within reason why the coordinate system was chosen. The New York State Plane Long Island Zone (EPSG 2263) is a local coordinate system that provides a high degree of accuracy and balances size and shape well. I know with certainty that the flyover and basemap were developed in support of the NYC Department of Environmental Protection (DEP) sewer and water mapping initiatives. Taking a leap of faith one can presume that DEP required a degree of accuracy that was best supported by the linear accuracy of the New York State Plane coordinate system.

To maintain consistency, all subsequent flyovers and planimetric captures have used this same coordinate system. But the story does not end there. Updates have been made to the coordinate system over the intervening years. The High Accuracy Reference Network (HARN) introduced an upgrade to North American Datum of 1988 which was subsequently upgraded by Continuously Operating Reference Station CORS in 2011. These very slight adjustments lead us to a current EPSG 6539. We have yet to adopt this new coordinate reference system (CRS) due to the disruptive nature of the change for minimal difference, but may move to it in the future.

State Plane is popular within state and local governments across the US but is that what the geo and open data communities are using? Most likely not. Google developed and popularized the ubiquitous web mapping coordinate system Web Mercator for its online maps. GPS coordinates and OpenStreetMap use WGS 84.

Thankfully transforming between coordinate systems is not as cumbersome and as problematic as it was in the past. Many systems can reproject  data on the fly. But is there a real difference between these popular coordinate systems used for NYC?

The three maps illustrate how the NYC landmass appears in the three coordinate systems discussed. State Plane and Web Mercator maintain the landmass shape well but the Web Mercator size is slightly distorted. This is most evident in Staten Island. The most extreme difference can be seen in WGS 84. The entire landmass seems stretched and flattened.

To answer my previous questions, I think the right choice was initially made and staying with the choice was an equally good decision. That said, we can support more than one coordinate system.

In the coming months, we plan on publishing our aerial imagery (digital orthophotography) as map tiles (WMTS) in both 2263 and 4326. So please check back. And feel free to leave us a comment with your thoughts on the topic.

What is the City Map (of New York City)?

“The City Map is the official adopted map of the city. It shows the location, dimension and grades of streets, parks, public places and certain public easements.”

This definition was taken from the Applicant Portal of the Department of City Planning website under Actions Requiring Uniform Land Use Review Procedure (ULURP), Changes to the City Map, and while it may not be the “official” definition, it does get to the point of what the City Map is and what is shown on it. The next questions that most likely come to mind are likely “Where can I see it?”, “Who has it?”, “Can I get a copy of it?”, or maybe even “Why should I care about the City Map when Google has a perfectly good map of the city?”

Let me begin by first explaining what the City Map is in the physical sense. It is not one comprehensive map showing the entire city that you could roll out on a large table (or basketball court), but instead it is a collection of thousands of maps of small areas of the city, each of which shows “the official adopted map of the city” for a that limited area. These maps are called alteration maps. Here is an example of an alteration map.

070109mmq

An alteration map becomes part of the official City Map whenever “the location, dimension and grades of streets, parks, public places and certain public easements” are changed anywhere in the city and that map change becomes effective. The “becomes effective” bit is important because unlike some other ULURP Actions, City Map changes are NOT effective upon adoption by the City Council, but must wait until the day after they are “certified and filed” with the Department of City Planning. The certification and filing step that makes the map part of the official City Map may not take place for several years after adoption by City Council. For example, the adoption date for the map change shown above was 04/28/2010, as stated in the paragraph above the “certified and filed” date. However, the alteration map was “certified and filed” on 02/27/2015 and so it finally “became effective” on 02/28/2015, almost five years after it was adopted.

The enlargement shown below, taken from this particular alteration map, reveals another interesting point- note there is no mention of the City Council anywhere in the paragraph. This is because the ULURP process allows the City Council a limited amount of time to take action on a City Map change following adoption by the City Planning Commission. If the City Council takes no action before that time expires, then the City Planning Commission’s adoption is considered final and the map then becomes effective the day after it is “certified and filed.” Had the City Council elected to take action on this map, there would be additional text showing the City Council’s adoption date and calendar number. If either the City Council or the City Planning Commission had rejected the map change, there would be no paragraph at all, and no map either, for that matter. City Map changes that are rejected are discarded.

This is a good point at which to answer the question of “Why should I care about the City Map when Google has a perfectly good map of the city?” The official adopted map of the city is not the same as Google’s map (or Bing or OpenStreetMap or even NYCityMap). While these other maps are good for showing what exists in the physical world , the City Map also shows streets that are official but have never been actually constructed. Also, the City Map will not show any streets that are not actually official city streets. Well, that last statement is not entirely true. The City Map does show, usually with dashed lines, what are known as “record streets.” These are streets that were present when the alteration map was created, but are not actually part of the official City Map. Record streets may or may not still exist in the real world, but for official purposes they are known as “streets of record” and they may be used for legal descriptions.

The City Map can affect someone who is looking to build (or rebuild) in an area where the built streets do not conform to the official City Map. This condition applied to parts of the Breezy Point neighborhood in Queens, an area damaged extensively during Hurricane Sandy. Many structures that were destroyed by the hurricane were built on top of officially mapped streets, which resulted in delays in obtaining permits to rebuild. Therefore, if you are thinking of building a(nother) new pie store on a vacant lot you saw in Google Street View (or while out walking), it is helpful to consult the City Map first. There could be a mapped street on the lot.

If it seems incredible to you that the entire extent of New York City could be completely covered by these limited-area alteration maps, then you are justified in that belief. There are many areas of the city that are not covered by any alteration map at all. So then what makes up the official City Map in these areas where there is no alteration map coverage, for example, all of the pink areas in the image below?

The Borough Presidents’ Final Section Maps are the answer.

Within the Borough President’s Office of each of the city’s five boroughs can be found a Topographic Bureau and a borough engineer. These Topographic Bureaus are the custodians of that borough’s Borough Map (also known as Final Section Map) along with all of the alteration maps filed within that borough. For any area within a borough that is not covered by an alteration map, the Borough Map is the official City Map. Every borough has its own style of Final Section/Borough maps. Personally I find the Queens maps have the best cartography. Below is one of them.

Now we have established enough of the background understanding necessary to answer the question of what is the official City Map with a little more precision. The official City Map for any area within the city is the most recently filed alteration map that covers that particular area. If there is no alteration map coverage, then the official City Map is the Borough Map. This seems pretty simple in concept, but often the determination of the official City Map for a particular area requires consultation of multiple, overlapping alteration maps plus the Borough Map, making it difficult to get a complete picture of what is official for the exact area that you care about. Wouldn’t it be great if someone synthesized all those alteration maps and Borough Maps into one single map or set of maps so your City Map research could proceed at more of a crime-scene-investigation-TV-show pace? Well, for all you CSIs of urban planning out there, the DCP Street Sectional Maps and Zoning Maps are your answer, mostly.

The DCP Street Sectional Map is essentially a schematic representation of the official city streets. I say schematic because, while the maps drawn to scale, sometimes the density of line work requires a bit of out-of-scale manipulation to make the street lines clear. The point is to provide a quick reference to the official city streets, not to make accurate measurements on the map. Street Sectional maps only show official (and record) street lines, official (and record) street names, honorary street names, parks, and public places. There are no dimensions or grades shown on these maps. Record streets are shown as dashed lines… but so are hidden lines, like underpasses, for example. You have to go by context to judge which dashed lines are record streets and which are underpasses.

The Sectional map is available for free at some public libraries and for purchase at the DCP Bookstore. There are thirty-five, 24” x 36” map sheets covering the whole city. Unfortunately, the Sectional Maps are only current to around 1997- the exact “current to” date is printed at the bottom of each individual map. Here is Street Sectional Map 8 which covers a popular section of Manhattan.

Fortunately, the Street Sectional Map is also the base map for the Zoning Map. Each of the 126 Zoning Maps covers a quarter portion of a sectional map. For example, the above section map 8 is covered by four zoning maps: 8a, 8b, 8c, and 8d. Zoning changes and City Map changes are often related to each other, and, probably because zoning changes more frequently than the City Map, the streets as shown on the Zoning Map are kept current to the present date. So, if you need to make a quick check on what the City Map is for any area, skip the Street Sectional Map and go directly to the Zoning Map.

In summary, for researching the Official City Map of New York City:

A) If you just need a general idea of what the official streets are in a particular area- just look at the zoning map. This will show you what the official street layout is.

B) If you are involved in serious land use business and you really need to know exactly what the official streets are, then you should go to that Borough President’s Topographic Office and research the filed alteration maps and Borough maps for your area of interest.

Resources for City Map research:

Zoning Maps

Brooklyn Borough President’s Topographic Bureau

Bronx Borough President’s Topographic Office (see Title Maps, Grade Maps, Monument Maps)

Manhattan Borough President’s Topographic Services Office

Contact for Queens Borough President’s Topographic Unit

Contact for Staten Island Borough President’s Office

Zoning data

Zoning application (ZoLa)

 

NYC Planimetric Update 2014/16

Background

For those wondering exactly what planimetrics are and why they should care, read on. Otherwise, you can skip to the last paragraph and download the data here.

Planimetric mapping is the capture of geographic features from aerial survey (i.e., capture of aerial photography) that are traditionally mapped in two dimensions and are therefore exclusive of elevation.  Quite simply these are the visible features that can be digitized from aerial photography. Often referred to as planimetric features or simply planimetics, these geographic features in their sum total essentially represent the base map data (i.e., layers) for a specific area.

NYC DoITT first developed a planimetric database in 2000. The data was captured from the first ‘modern’ aerial survey of the New York City that took place in 1996. Referred to as the NYC Landbase, components of this effort were the establishment of:

• a ‘database design’ (the delivery was ArcInfo coverages);
• coverage parameters (e.g., scale, projection, precision, fuzzy tolerance and dangle length);
• the specific features to be captured;
• and a classification scheme (i.e., feature codes).

The delivery of the data was by 2,500′ x 2,500′ tiles, which directly corresponded to the orthophotography tiles.

Updates

A subsequent ad-hoc update to the planimetric database was done in 2004. This update was based on aerial surveys from 2001 (Manhattan and Staten Island) and 2002 (Bronx, Brooklyn and Queens). This update conformed to the previous database design. In 2006, DoITT formalized the update frequency of  the planimetric database and aerial surveys. The aerial photography would be captured on a two-year cycle and the planimetrics a four-year cycle. With the first regularly-scheduled planimetric update to be based on the 2006 aerial photography.

With each subsequent update, refinements have been made. New features and domains have been added,  obsolete features have been removed, features are captured in three dimensions, a seamless database is produced and the time between aerial capture and delivery of planimetrics shortened.

Year of Update

A lot of work goes in to producing the planimetrics. The orthophotography takes from nine to twelve months to deliver. A spring capture in 2014 is therefore delivered in 2015. The planimetric features themselves also take from nine to twleve months. All of this is a long way of explaining why data from 2014 is published in 2016. It takes time.

2014 Update

For the 2014 update, additional refinements were made. Skybridges were captured as separate features (sub type within building footprints). Below is an example of a skybridge connecting 3 and 4 MetroTech Center, Brooklyn.

Previous (2010) and current (2014) representation of skybridges.

Previous

Cooling towers are a new feature capture – see example below. This data will be published in the next couple weeks.

Cooling towers – black rectangles

 

 

Curblines are a new separate feature. Previously curblines were a subtype in Pavement Edge. Pavement Edge features were segmented at the apex of each edge and a unique ID was assigned. These IDs were then transferred to the Citywide Centerline.

Pavement edge with Blockface ID

As with the previous update, all of the individual data sets are on the open data portal. New with this update is a comprehensive database that contains all of the data sets. Additionally, the data were tagged with ‘planimetrics’ and ‘doitt gis’ to simplify search and discovery. Lastly, previous blog posts will be updated with any new or updated data urls. Happy mapping!

Geoclient Released Under Apache 2.0

On the continuum of greater collaboration, there is open data, open API’s (application programming interface) and open source. Today’s release of Geoclient under an Apache 2.0 license (press release) on GitHub (here) further strengthens NYC’s overall commitment to openness and collaboration.

NYC government provided data for free download prior to the landmark NYC open data legislation in 2012. One of the earliest examples, if not the first, was the Department of City Planning’s (DCP) release of the LION street centerline file in November 2001. However, the open data legislation greatly opened the data spigot with many more data sets being released and included a plan to release all applicable data sets by 2018.

NYC’s open data portal was followed by a developer portal that was populated with data feeds and API’s. Geoclient was first released to the public on the developer portal as a freely available API. All that was required for access was the provision of an API key, which has since been automated.

We are excited about the release of Geoclient and hope it provides additional benefits to its end users. Although not the first release of source code on GitHub – there are already numerous NYC projects including my team’s Pre-K finder – Geoclient represents arguably the most significant when measured by volume of transactions and number of client application integrations. Geoclient is used by startups, universities, research institutes, other governments and the local developer community. Geoclient has seen almost 300% growth over the last year alone.

We look forward to greater collaboration and hearing from the community of users. However, like any open source projects we cannot guarantee a response to every issue and question but we will do our best. Hopefully the user community gets involved to answer some of the questions. Please bear in mind that Geoclient is a moving target under active development because it is used internally by City systems and staff whose needs are constantly evolving. Yes, we do eat our own dogfood.

Finally, none of this would have been possible without the dedicated staff at DCP who have maintained and extended Geosupport for over 30 years. And of course I’d be remiss without recognizing the hard work of Matt Lipper at DoITT. Kudos to all!

Source code – https://github.com/CityOfNewYork/geoclient

Developer Portal – https://developer.cityofnewyork.us/api/geoclient-api

 

Step Streets: An Unusual Means of Network Connectivity

For accurate routing, network connectivity is essential. This is the case regardless of one’s mode of travel. A person may start a journey on foot, move to a bicycle, then to subway and finish off her trip again on foot. Nevertheless, all legs (e.g., street, bike lane, subway line) of the journey, regardless of the mode of travel, need to be connected.

In the course of a trip, we may on occasion encounter unusual means of ‘network’ connectivity. For example, we may need to carry our bicycle up a set of stairs to enter onto a bridge. In this post, I will cover what I find to be an unusual means of urban connectivity: step streets – more on the name later. Quite simply, a step street can be considered an outdoor stairway or a series of steps that connect two different elevations.

When I think of ‘step streets’ as access ways in urban environments, older European cities come to mind first. Cities with narrow streets and sometimes steep grades that were laid out well before the advent of the automobile. One of the most famous examples, albeit not of a narrow passageway, is the Spanish Steps in Rome that connects Piazza di Spagna and Piazza Trinità dei Monti.

Image via Wikipedia

A more classic example, of a steep narrow passageway comprised of steps, can be found in Lago D’Iseo, Italy. You could probably schlep a bicycle up these steps but not much more.

Narrow Step Street

Image via Martha’s Marvelous Munchies

In older city settings, stairs or paths were probably the only means for getting pedestrians from point A to point B. I do not profess to know the history of steps in urban settings, but I imagine they were built out of necessity. In most cases, one could imagine, due to limited remaining space and a large difference in elevation. And in these parts of a city, cars are often not permitted for obvious reasons.

Stateside, San Francisco comes to mind first due to its hilly terrain and a street layout that works with, not against, the terrain. Although an amazing site with beautiful landscaping and a notable tourist destination, Lombard Street demonstrates the impractical nature, and some might argue, misuse of space, in trying to move automobiles through extremely steep spaces.

Lombard St, San Francisco

Image via Wikipedia

In contrast there are a series of pedestrian-only ‘stairways’ that connect Jones and Taylor Streets in the Russian Hill section of San Francisco. In OpenStreetMap, these ways are tagged highway = steps. For more in San Francisco stariways see here.

San Francisco Stairways

Back here in New York City, where the terrain is considerably flatter and the rents slightly lower, there are a series of narrow passageways comprised entirely of steps that are limited to pedestrians and are referred to in the local vernacular as ‘step streets.’

Many step streets are official streets and therefore can be found on the official City Map. For background context, the official City Map is a collection of Alteration maps. Alteration maps record changes to the City Map including public streets, parks and public places – see here for more information. Alteration maps of a corresponding area supersede all previous Alteration Maps from previous dates).

Below is a portion of an Alteration map from 1955 recording the elimination (de-mapping) of West 230th Street in the Bronx. This map shows both West 230th and West 231st Street labeled ‘STEPS.’ This label is most likely how the term step street started: as one might say as a cartographer’s annotation.

Alteration Map: Elimination of W 230 St

Alteration Map courtesy of the Department of City Planning. To see the complete map click on the image.

Step streets can also be found in NYC Open Data where the Roadway Type (RW_TYPE) is 7.
LION
CSCL

One of the hilliest areas in NYC is northern Manhattan and the Bronx. And as one might expect that is where the greatest number of step streets are found. This is where you will find the densest clustering of step streets. And these are the ‘classic’ step streets (i.e., connect two streets). The example below connects two sections of Pinehurst Street in Manhattan north of W 181st Street.

Pinehurst Ave & W 181 St, Manhattan

Based on the data, the longest step street is West 230th Street between Netherland and Johnson Avenue in the Bronx. A length of approximately 295′ with an elevation difference of 38′ for a grade of 12.9 percent. A relatively modest slope compared to the 26.8 percent grade of W 187 Street between Overlook Terrace and Fort Washington Ave also in the Bronx.

The definition of a step street seems to have been expanded to include steps that connect streets to the boardwalks in the Rockaways. In any case, go ahead and download the data to explore these and other interesting anomalies or check out the map below.

Additional reading:
Forgotten New York
Sister Betty
Boredpanda

NYC Streets on Paper

As is usually the case with development projects, pen must be put to paper first followed by a series of reviews and sign-offs before a shovel is put to the ground. That is also the case with street construction. What is unique, however, is that a street must be added to a map before it is constructed.

In New York City, a newly proposed street must be added to the official ‘City Map’ (not to be confused with NYCityMap) through the Uniformed Land Use Review Procedure (ULURP) before it can be constructed. Thus a street will exist on paper before it becomes a reality. These streets are what have become to be known as paper streets. Paper streets are not unique to NYC but ULURP is.

Paper streets may exist on paper only for many years before they are ever constructed. The street’s configuration or name may change before construction takes place. There are even situations whereby a street could halted (de-mapped) – see definition below – before it ever becomes a reality.

The dashed lines on the map below represent paper streets in the Midland Beach section of Staten Island. It is clear from the area that these infill streets are intended to complete the planned street grid when fully built out. However there could be circumstances (e.g., being in a flood zone) that prevent the streets from being constructed.

Paper Streets: Midland Beach, Staten Island

Although originally on paper only, paper streets can be found in NYC digital data. The NYC Street Centerline (CSCL) data set on the NYC Open Data Portal and City Planning’s LION data set include paper streets. For those wondering, LION is an extract of CSCL that includes both single-line (generic) and dual-line (roadbed) representation of the street network plus additional geographies. Additionally, LION has more fine-grained segmentation (breaks occur whenever geographies cross or there are unique address range breaks). Whereas, CSCL is focused specifically on the actual street (roadbed) representation with segmentation by block. More on these data sets in a later dedicated post.

Paper streets can be found as follows:
LION – featuretyp values of 5 and 9;
CSCL – STATUS values 3 and 9.

The inverse of a paper street is a de-mapped street. As the name would apply, this is a case where a street was officially removed from the City Map. And as with paper streets, the street will appear on paper (City Map) as being de-mapped before they are actually removed.

De-mapped Street: Melrose Crescent, Bronx

De-mapped streets can be found in LION where status equals 5.

State of the Map US 2015 and the role of Governments in OpenStreetMap

A little over two weeks had passed since the closing of the State of the Map US (SOTMUS) conference in NYC. For those not familiar, SOTMUS is the yearly conference for the US chapter of OpenStreetMap (OSM). This period offered some much needed time to reflect on the conference as a whole: setting, presentations and sessions, exhibitors, organization and execution. On all points, I felt SOTMUS hit the mark and was a resounding success.

Yes, I’m sure there were some minor shortcomings as evident by some of the tweets I saw (#SOTMUS). Nonetheless, for a conference organized and executed by volunteers coupled with the comments I heard, it was clearly a success. I have a new found appreciation for the hard work that goes into organizing such a large event after having assisted the organizers in securing the Surrogate Court space for the opening night (NYC DoITT sponsored IT). Kudos to the organizers! But alas I digress.

This post is not intended to be a review of the event. Many others I’m sure have already covered that and a better positioned to do so. My objective was to dive further into the role local government could play in OpenStreetMap. This post can be seen as an extension of the panel I was on at SOTMUS, which as a demonstration of interest in the topic, was the second of two panels on OSM and government. I’m sure that many would even question whether government has a role at all. To that I would say, duplicating or recreating what has already been mapped and increasingly is available on open data sites, is time consuming and wasteful. On the international landscape that is often not the case but here in the US it is.

Consider the NYC building footprint (with height) and address import. To manually digitize approximately one million buildings would have been a labor intensive and lengthy process. On-screen digitizing over aerial photography of a lower resolution then NYC possess would have also resulted in lower quality and less consistent data. Contrast that with a careful import utilizing high quality preexisting *authoritative* data that resulted in nearly complete and consistent coverage of NYC is in my opinion hard to argue against. A bulk import then frees up the community to focus on keeping OSM current and filling in the gaps where needed. Certainly a less daunting task then starting – with respect to buildings – from a nearly blank canvas.

The NYC buildings and address import was largely undertaken by Mapbox. NYC DoITT assisted with planning and answering questions (NYC addressing is a challenge) throughout the effort and of course providing the data. Part of the effort included a change notification email that gets sent out each night. The email shows the changesets from the previous day. Since a changeset can be comprised of multiple edits, wading through numerous unrelated edits (primary focus is on buildings and addresses) can be time consuming; however the change notification has proven useful and has resulted in hundreds of edits to NYC data.

Each changeset comes with a map (see example below) to guide the reviewer to the specific location of the edit. NYC DoITT staff review the changeset and apply any valid changes to the internal repository. Due to schema differences and ODbL license restrictions, the OSM data is not imported into the internal repository. The changesets are used as a guide.

Tools such as MapRoulette can also be used to bring in changes made to *authoritative* data sets. This is the method being used by the local NYC OSM community to incorporate missing bike lane data into OSM (see Eric Brelsford’s lightning talk here).

I think it is undeniable that *authoritative* government data can further enrich OSM to the benefit of many. You may then be asking yourself, what is the benefit to a local government? To me there are both direct and indirect benefits.

From a strategic perspective, it is important to have options when making decisions. In the case of data, not all local governments can afford or have the technical capabilities to manage their own geospatial data. And even when they do, there are cases where governments use external data sources for routing and logistics. To have only a couple proprietary commercial options limits choice and drives up cost. Having a robust and complete open data set provides governments alternatives. And the benefit is twofold: direct cost savings and indirect alternatives.

OSM can also benefit a much wider audience. Open data is great. And the movement towards more open data is fantastic. What is often not discussed is the barrier to enter the open data space. Not only specific to geospatial data, a person needs a variety of skills and software (there are open source options in geo such as QGIS) to work with and analyze the data. This is not an intended barrier but a result of the complexity within the current geospatial technology space. This greatly reduces the number of people downloading and working with open data. Conversely with OSM, there is a platform and an ecosystem of tools already in place. There are tools for viewing, editing, analyzing, rendering and even downloading OSM data. This allows people to focus on what they want to do with the data (e.g., make or view a map) and less about the intricacies of setting up the data to work with it. And there are an amazing set of tools from independent open source developers to commercial entities. From the elegant and simple ID editor to the Tangram map renderer. OSM can open up a wealth of possibilities and can be a viable alternative.

Geoclient: An Open NYC Geocoding API

Overview
Geoclient is a geocoding API that recognizes and geocodes addresses, intersections and blockfaces (on street and two cross streets) located in New York City. The Geoclient service provides a RESTful web service interface to the Department of City Planning’s (DCP) Geosupport system. Geoclient provides “pass-through” style access to native Geosupport functions. It does not change or modify Geosupport functionality in any way. However, Geoclient does add some very useful features (some of which are considered standard by today’s developers). While Geoclient is intended for programmatic use, DCP hosts GOAT, a website that provides direct access to Geosupport native functions. Geoclient developers often use GOAT to compare results or test data interactively.

Access
There are two primary Geoclient installations. One is for use internally by official City agencies and the second intended for the public developer community. However, the data and logic behind all instances of Geoclient are exactly the same. By sharing the same code base and deployment scripts, it is easier to maintain and support. For the public, access to Geoclient is through the NYC Developer Portal. For City agencies, contact us directly.

Functionality
Geoclient provides two notable enhancements which can be used to simplify and optimize access to Geosupport. The first is single-field search functionality which parses a single input string into the discrete location elements that are required by the different Geosupport functions. For example, an input of “59 Maiden Lane, Manhattan” is parsed into its house number (59), street name (Maiden Lane) and Borough (Manhattan) for submission to Geosupport. Note that parsing is not case-sensitive, punctuation is ignored and most standard street pre- and post-modifiers, types and directionals are recognized.

A fully qualified (i.e., expanded) address is often referred to as a ‘normalized’ address. An example is ‘314 w 100 st mn’ which is the equivalent of the normalized ‘314 West 100 Street, Manhattan’. Normalization is native to Geosupport. If your data is already parsed into discrete address elements, using the /geoclient/v1/address endpoint directly is less ambiguous and slightly more efficient.

Building on it’s ability to parse single-field search locations, Geoclient can also be used to “guess” the intended target of an incomplete or ambiguous input. One example is the submission of an address without a borough. As long as Geoclient can recognize the search text as an address (house number and street), the /geoclient/v1/search endpoint will try that address in all five boroughs. By default, if the address exists in one or more boroughs, all locations will be fully geocoded and returned as possible matches. This behavior can be customized, for example, by calling the service with the optional ‘exactMatchForSingleSuccess’ parameter set to true. In that case, if the address exists in only one borough it will be geocoded and returned as an exact match.

Using the previous example, entering “59 maiden ln” without the borough will result in the same response as entering “59 Maiden Lane, Manhattan”. The figure below shows the response for “59 Maiden Ln” using the Pre-K Finder application.

In cases where the same address exists in multiple boroughs, Geoclient will return a candidate list of possible addresses. In each case, the candidate address is validated to ensure a successful response. See example below.

 

Developers can customize this and several other search features as documented by the Geoclient API.

Hopefully this will clear up some of the misconceptions associated with Geoclient and contribute to the knowledge base. And please check back in the coming weeks for a more detailed Geoclient post. In addition, expect some enhancements over the coming months. Nice work Matt!