The grant will allow the project team to develop infrastructure, content, and community for Version 3 of the WHG. The index will more than double in size; the suite of tools will evolve to better support teachers, contributors, and end users; and the team will expand opportunities to involve diverse and global communities of board members, scholars, learners and developers.
The NEH’s Digital Humanities Advancement Grants program (DHAG) supports innovative, experimental, and/or computationally challenging digital projects, leading to work that can scale to enhance scholarly research, teaching, and public programming in the humanities. In support of its efforts to advance national information infrastructures in libraries and archives, the Institute of Museum and Library Services (IMLS) provides funding through this program.
ABOUT THE NATIONAL ENDOWMENT FOR THE HUMANITIES
Created in 1965 as an independent federal agency, the National Endowment for the Humanities supports research and learning in history, literature, philosophy, and other areas of the humanities by funding selected, peer-reviewed proposals from around the nation. Additional information about the National Endowment for the Humanities and its grant programs is available at: www.neh.gov.
In recent months we asked several contributors to pause their data preparation in the WHG system while these improvements were made. We can finally “re-open the doors” so to speak, so we invite those efforts to resume, and again encourage new contributions and collaborations. We will respond quickly to any bug reports or general inquiries about using the platform.
Over the next several months we will be adding quite a bit more data that is already in the queue. Although much user interaction with the WHG platform is self-guided and semi-automated, we have found that contributions move most smoothly with staff support. WHG staff stand ready to help with data conversion strategies and with the planning of contributions generally. Please do get in touch with us (whg at pitt dot edu) or with any individual WHG project team members individually.
The Site Guide and several tutorials on the WHG site describe its features and their use in some detail. The following briefly summarizes what is new since Version 1.
Registered users can now create “collections,” linking sets of existing public datasets within the system for purposes of presentation and combined search. This new feature aims at supporting the development of “focus regions” within WHG by collaborative groups with overlapping region/period interests.
Previously, search capability was limited to records fully accessioned into the WHG “union index,” and returned sets of one or more “closely matched” attestations of a place. This kept from view public datasets that had not yet been indexed. An option to search all public data within the WHG database—indexed or not—has been added to give a more complete view of the data we hold.
We have adopted the term “linking” to refer to all tasks of reconciliation and alignment—to external the external sources Wikidata, Getty TGN held in our sytem and to our own WHG union index. All of these require a “Review” step, where the prospective matches discovered in the task are presented for closeMatch/no match/defer decisions. The progress of this process, which can sometimes extend over time and involve multiple people, is now tracked in the Dataset Browse screen available to the dataset contibutor (“owner”) and designated collaborators. The choice to “defer” is also new since v1.2; it permits maintaining a separate queue of records, allowing users to move quickly through the easier decisions and set aside those requiring more attention, or review by others.
Views and downloads of public data
We now provide summary descriptions and mapped browsing for all datasets, collections, and individual place records that have been flagged as public. Public datasets can now be downloaded, according to CC-BY-4.0 license terms.
We have implemented the MapLibreGL technology for our Dataset and Collection maps, dramatically enhancing the speed of rendering large numbers of features.
Local Wikidata index
Since Version 1.2 in May, we have maintained a local index of about 3.6 million Wikidata place records, making reconciliation tasks for that resource 3x faster than the earlier SPARQL queries over the web–processing about 150-180 records per minute.
More reliable upload validation
Accounting for every possible anomaly or error in upload files is tricky. We have significantly improved the validation algorithm, trapping more errors with more user-friendly responses.
Site documentation has been edited and extended, and a number of display problems were fixed. SSL protocol (https) has been implemented for secure transfer.
After three years of development, we are pleased to announce the launch of Version 1 of the World Historical Gazetteer (WHG), at whgazetteer.org. Version 1 follows six beta releases over the past year or so. The WHG presently indexes 1.8 million modern place references and approximately 60,000 temporally scoped records.
In addition to filtered search and API access to data, we have developed a suite of tools that allow you to upload place datasets into a private workspace, augment them by reconciling them against the Getty Thesaurus of Geographical Names and Wikidata, publish them as Linked Open Data, and contribute them for accessioning to the WHG index.
We have a long list of planned improvements and a queue of in-progress data contributions. More data is very welcome of course, and your feedback is essential! You can use the site contact form, create an issue on GitHub, or simply write to us at email@example.com.
We have completed a Site Guide that describes the purposes, functionality, and data of the present system. The Tutorials and About sections of the site provide additional information. We will continue keeping our nearly 500 Twitter followers current with news of new data, new features, and bug fixes. We also plan to keep our blog updated with relevant announcements and discussion of the project’s future course..
We are pleased to announce this major step in our project and we look forward to your
The World Historical Gazetteer Team
Ruth Mostern, Karl Grossner, and Susan Grunewald
This post aims to clarify the relationships between a few of the models now in development for various uses by members of the historical linked data community, particularly with regard to geography (place)—namely Linked Places, Linked Traces, and Linked Art. Figure 1 provides a conceptual overview (click to magnify).
In several key respects the World Historical Gazetteer project (WHG; now in beta release 0.3) builds upon software and data development work produced by the Pelagios project—particularly the historical gazetteer infrastructure underlying its Peripleo and Recogito software applications.
Peripleo is a pilot application (no longer in active development) built to demonstrate a few key linked-data-for-history functions: a) search of a central index aggregating historical gazetteer records published as Linked Data, b) the annnotation of web-published records about historical objects with identifiers for relevant places (mostly coins and inscriptions in this case), and c) the display of search results for both in a map interface. WHG performs those functions also, along with some others.
Recogito is an annotation platform that among other things makes use of that historical gazetteer index by facilitating association of place references tagged in textual sources with the identifiers, coordinates, and name variants found in the indexed gazetteer records.
I have collaborated with Pelagios developer Rainer Simon and a few other interested folks to develop a Linked Places model and format particularly for contributions to the Pelagios and WHG platforms. The Pelagios and WHG indexes will have considerable overlap in coverage, but we anticipate that of WHG will over time become considerably broader in space and time—due primarily to its built-in semi-automated data development and contribution pipeline and stated goal of global breadth.
Because both projects have interest in annotations, we have also begun jointly developing a Linked Traces format—more precisely a set of implementation patterns using the W3C Web Annotation format standard for digital history and GLAM applications.
With that introduction, what follows are some details about Linked Places and Linked Traces, and thoughts about their immediate and potential uses. Also, given the concurrent development of the Linked Art model and ontology, some thoughts about how all of these might in time relate to each other in practice. Figure 1 above should provide useful reference.
Linked Places: model and data format
The Linked Places model and interconnection format (LPF) were developed to meet the particular requirements of the WHG and Pelagios platforms: a common data structure that both could ingest routinely without the need to accommodate on a case-by-case basis the enormous variety of data models in use by digital historical projects, large and small. LPF is a set of extensions to GeoJSON-LD, itself a Linked Data enabling extension to the most widely implmented test-based format for representing geographic features, GeoJSON (an IETF standard).
LPF also adds a standard means for adding time to GeoJSON features, introducing “when” objects to permit temporally scoping of a) an entire Feature, and/or b) its individual names, place types, geometry, and relations to other places, in any combination.
Uploads to WHG (and accessioning to both Pelagios and WHG indexes) require creating a serialization (i.e. transforming export) of place data from whatever form it is maintained in to LPF. We have also developed an abbreviated delimited text file format (LP-TSV) to meet the needs of contributors with relatively simple records.
Figure 1 summarizes the LPF conceptual model.
Linked Art: model and format
A global consortium of organizations involved in the domains of art, cultural heritage and archaeology—principally large museums and universities—are jointly developing Linked Art, a “shared model based on Linked Open Data to describe Art,” along with software implementations of it. The conceptual model is being formalized in an ontology with a subset of CIDOC-CRM entities and relations, and expressed as a data format using JSON-LD, a syntax of RDF.
From the perspective of WHG, Linked Art is a format many prospective users of our platform may adopt to describe objects in their collections. Both WHG and Pelagios are agnostic as to what formats our users and data partners use, and as mentioned above, users will have to perform a serialization to Linked Places format to interact with our platforms.
Figure 1 shows how Place appears in the Linked Art model. The points of contact with Linked Places are identifiers. One kind of identifier in Linked Art is a URI to a linked data gazetteer resource. A serialization of Places from a Linked Art dataset to LPF should include as many such identifiers as can be managed. WHG can aid discovery of those URIs via reconciliation services to Getty TGN and Wikidata.
All that said, place data from Linked Art collections are unlikely to be good candidates for contributions to WHG; the great majority of places will already be indexed. Rather, it is Linked Traces data that will be more relevant.
Linked Traces: model and format
WHG is following on from the Peripleo pilot in experimentally indexing not only place data, but what we are calling trace data: “annotations of web-published records about historical objects with identifiers for relevant places.” We say “experimentally” because it seems likely that the most useful web interfaces to trace data will be distinct from those for place data. Certainly there will be significant scaling issues.
In order to continue exploring the linking of places and associated traces, we (Rainer Simon and I) have also initiated development of a Linked Traces format, as a potential standard for use by the WHG and Recogito platforms. Linked Traces is turning out to be a set of implementation patterns for the W3C Web Annotation format (WA).
Annotating records of “anything” with URIs for web-published place records is but one use case for WA. For example, in Recogito, users annotate texts with references to not only places, but also people, events, and relations between all three.
Figure 1 indicates the way that a set of one or more place records can form the body of an annotation. The JSON form of the body in that example corresponds to an early draft of a “Linked Traces place pattern” in development. The working group’s activities are paused at the moment, but WHG is developing some exemplar data according to that draft, to be explored in our Version 1 release, slated for late spring 2020.
At long last we are ready to offer a v0.2 beta release of the World Historical Gazetteer (WHG) at http://dev.whgazetteer.org. We hope that spatial historians and spatio-temporal infrastructure developers will be interested in taking a look at what we are building, experimenting with their data or provided samples. It is a “sandbox,” so nothing will be saved for the time being (that will change soon). There are 5-6 months remaining in the term of our initial NEH grant, time enough to complete most of what we planned for this phase, and to incorporate more suggestions from users and potential contributors as we move toward future planning and development.
The site includes a brief guide titled “WHG Beta Release: A Tour,” which outlines what is there, what you can do and how, remaining challenges, and what is in the works. What follows is a higher level introduction.
Places and Traces
The World Historical Gazetteer is a Linked Open Data platform for publishing, linking, discovering, and visualizing contributed records of attested historical places and traces. Our initial focus has been on places, but we are working experimentally to demonstrate their integration within the platform with what we now call traces–defined as web resources about historical entities for which location in time and space is of scholarly and general interest. We are considering three classes of traces for the time being: agents (people and groups), works (e.g. artifacts, texts, datasets), and events (e.g. journeys, conflict). Our objective has been to create the first large-scale spatial infrastructure for world history: oriented toward documenting the human past at the global scale, and particularly the geography of global and transregional connections.
Our accessioning process is intended to eventually be largely self-directed; getting it to that stage means working directly and hands-on with our early contributors.
Registered users of WHG can publish their place records as Linked Open Data simply by uploading them in Linked Places format (or the LP-TSV version intended for relatively simpler records). We see LOD publication as a key feature for researchers who are not in a position to stand up their own web interfaces with per-place pages. Once uploaded, each record will have a permanent URI and be accessible in our graphical interface and API; on their way to being LOD in good standing. The dataset can be browsed immediately by its owner in a searchable table and map, but turning the uploaded dataset into a contribution for accessioning requires some further steps. The data needs to have as many asserted links to name authorities as possible, and augmentations of geometry where that is missing and findable. We provide reconciliation services for that purpose.
Simply put, reconciliation is the process of identifying matches between records of named entities. In this case the records are for places, and the matches are between a researcher’s records and those in existing place name authorities. So far, we provide reconciliation services for the Getty Thesaurus of Geographic Names (TGN) and Wikidata; DBpedia and GeoNames are planned.
The reconciliation process has two steps: 1) sending records to the authority, and 2) reviewing the prospective matches returned and accepting or declining them as appropriate. The results of this somewhat laborious process are 1) links, and 2) more geometry. Once augmented in this way, a dataset is ready for accessioning.
The last step is another reconciliation effort — this time to the WHG index. Each record is compared to the growing WHG index to determine if we have a contributed attestation for the place yet or not. If we do, the incoming record becomes a “child” or “leaf” in the set of attestations for the place. If the place is not yet accounted for, the new record becomes a “parent” — the seed for a new set of attestations. At this stage, an automatic linking can be made if two records share an authority match, but the rest will have to be reviewed as described above.
The opening screen of WHG offers users search of places and traces. We try to offer enough context on the opening screen to identify the likeliest match. Once you identify a place of interest, clicking its name take you to a “place portal” screen–where everything we have about the place, or linked to it in some way, will appear: attestations from contributors, associated traces, nearby places, physical geographic context (rivers, watersheds, ecoregions). The place portal is very much a work-in-progress at this stage. Several other features are also on our near-term to do list, including advanced search; more and better maps; user data collections; project team ‘workspace’; batching of reconciliation tasks; and more.
A Word About Architecture
There are two data stores within the WHG platform: a relational database (PostgreSQL) and a high-speed index (Elasticsearch). All uploaded data gets imported to a set of relational tables whose names correspond to the elements of Linked Places format: places, place_name, place_type, place_geom, place_link, place_when, place_related, place_description, and place_depiction. Contributed data is most readily managed in that form. Upon accessioning, records are added to the index in the manner described under Reconciliation above.
This part of the WHG platform is one of the most important, and the least developed right now. Stay tuned for further developments. Our intention is to provide access to both contributors’ individual records and datasets from the database (when designated by their owner as public), and to the aggregating index records; both with numerous and useful filtering capabilities.
Our index has been instantiated with records from modern gazetteer resources: 1) about 1,000 of the world’s most populous cites from GeoNames, 2) ~1.8 million place records from Getty TGN, 3) about 1,500 societies from the D-Place anthropological repository; and 4) major rivers, lakes, and mountain ranges from Natural Earth and Wildlife Research Institute.
To this modern “core” we have begun adding historical data: 1) 10,600 entities harvested from the index of the Atlas of World History (Dorling Kindersley, 1995), offering broad but shallow global coverage; and 2) our first specialist gazetteer, HGIS de las Indias, which consists of approximately 15,000 settlements and territories in colonial Latin America. There are several additional large datasets in the queue, which we will be adding in partnership with contributors. Some are previewed as heat maps on our Maps page.
Our Pelagios Connections
The WHG platform borrows extensively from the Peripleo application developed by Rainer Simon of the Pelagios project, extending it significantly in a few ways. Our backend architecture closely mimics that underlying both Peripleo and the Recogito annotation tool, and we are actively collaborating with Rainer and the entire Pelagios Network team on several aspects of this work. In particular, we are co-developing the data format standards for contributions to both systems: Linked Places format, and a nascent Linked Traces annotation format.
We welcome suggestions, critiques, even praise :^) – and there is an email form on the site which makes it easy to offer it. Please bear with us in this active development stage and check back as we realize the system’s potential more fully over the next several months. Look for further blog posts and follow us on Twitter; we tweet progress and related information as @WHGazetteer and @kgeographer.
The World-Historical Gazetteer project (WHG) is now just past the halfway mark in our three-year NEH grant period. We are on track to produce what was promised in the proposal, notwithstanding some expansion of scope and unforeseen developments and opportunities. We expect future project milestones roughly as follows:
March 2019: Alpha release of functional contributions system and basic API
July 2019: Limited public beta for comment: contributions, API, place pages
July 2019: Second Advisory Committee meeting
September – December 2019: Rolling beta releases and data additions
1 April 2020: Launch
April – May 2020: Refinements
There are currently seven (!) activity tracks: 1) data development; 2) outreach to potential data partners; 3) format standards; 4) backend design and development (data stores and contribution pipeline); 5) frontend design and development; 6) API; and 7) ecosystem.
The primary focus right now is the contribution pipeline of #4 and some pilot screens below show progress. We have a considerable volume of data queued up for conversion to Linked Places format then accessioning, and much more is promised.
The functionality we are providing for contributing data has become more ambitious than originally envisioned. We need to accommodate contributions ranging in size from a handful to many thousands, with a minimum of hand-holding by WHG support staff.
Registered users will have a dashboard from which they can manage datasets through the entire workflow: (i) uploading files, (ii) ingesting to our database, (iii) initiating reconciliation against name authorities, (iv) reviewing and validating “hits”, and (v) submitting the resulting enhanced dataset for indexing. Users will also be able to create, download, and share “MyPlaces” collections of records tagged while using the site. That’s a lot!
I will also join presenters of the LatAm Gazetteer project funded by a Pelagios Commons Resource Development grant. The LatAm project has provided us two excellent datasets, which together will form within WHGazetteer a “seed” of sorts for Colonial Latin America. Progress working with that data is indicated on the following functional screens in our pilot (pre-alpha) site. Note that little to no attention has been paid to styling at this point.
The World-Historical Gazetteer project (WHG) will soon begin aggregating and indexing historical gazetteer datasets, and exposing them as Linked Open Data via graphical and programmatic web interfaces — just as Pelagios Commons’ Peripleo project has done for a few years. And like Peripleo, WHG will also index contributions of annotation records that associate historical “items” with place identifiers. Typical items for Peripleo have included coins, coin hoards, and inscriptions of the Classical Mediterranean. Items records WHG will focus on include journey events, regions, and datasets. In fact, annotated items could be anything for which location is relevant, e.g. people and various types of events.
We are almost ready to begin accepting contributions; this post previews the pipeline and formats involved.
Contributions to WHG can include, in some combination: 1) gazetteer data, i.e. place records drawn from historical sources; 2) annotation records that associate a published record about an item with a place identifier; 3) collections of item metadata records referenced in annotations; and 4) a file describing the contributed dataset(s) in Vocabulary of Interlinked Datasets” format (VoID).
Over the past several weeks we have collaboratively developed a new Linked Places format (LPF here for short) with Rainer Simon of Pelagios, to be used for contributions of historical place data to both WHG and Pelagios’ Peripleo. The Linked Places format is designed around the JSON-LD syntax of RDF (it is also valid GeoJSON, with temporal extensions, as explained in the GitHub README). The new format makes use of several existing vocabularies and also introduces some terms specific to our shared purposes.
Several expert colleagues contributed valuable input, including Graham Klyne, Richard Light, Lex Berman, Arno Bosse, and Rob Sanderson . We are in the process of updating the template Peripleo has used for annotation contributions (formerly Open Annotation in RDF Turtle, now its next-generation W3C Web Annotation in JSON-LD). Both are discussed in a little more detail below.
Contributing historical place records
There will be two separate workflows for contributions: from larger projects and from smaller ones. The distinction is whether a project has the capability and resources to meet two criteria which are accepted norms for publishing Linked Data: 1) publishing data in some syntax of RDF (in our case the new LPF); and 2) providing a unique URI and associated “landing page” for each resource described.
Case 1: Larger projects
If your project has (or will have) a web presence that provides public pages describing your individual places and/or “items”, (routes, regions, etc.), then we ask that you perform a transformation and export of your data in the standard formats mentioned above – Linked Places, a future annotation format (see Contributing Annotations below), and VoIF. Upon validation, we will ingest those records, link them with those already in the system, and expose them in a nice GUI and API. Details of WHG interfaces are forthcoming soon.
Case 2: Smaller projects
If your project does not entail creating a web site providing per-record landing pages, then we can accept your data contribution as CSV, mint unique URIs, and provide very basic landing pages for places and other items. The records will also be made available as JSON-LD (bonafide RDF) via our API. We will provide a Python program for converting CSV to LPF, but note that the CSV will have to conform to a template that aligns with LPF (available soon). Conversion from your native format to our CSV template will probably be more manageable than to LPF. In other words, upon submitting CSV data we can parse, a semi-automated conversion and ingest procedure will result their publication as Linked Open Data.
WHG will index metadata describing historical “items” annotated with gazetteer record identifiers. These annotation records assert, in effect: “this item is/was associated with this place, in this way;” and optionally, “at this time.”
The result of such annotations can be seen in the current Peripleo interface, where upon navigating to a given place, you can view metadata (including images) for coins and inscriptions associated with it in e.g. a foundAt or hasLocation relation. Annotations exposed in the WHG web interface will include historical journeys for which the given place was a waypoint, and regions, works, and datasets including or referring to the place.
Annotation contributions will comprise two sets of data: 1) collections of brief Item metadata records; and 2) collections of annotation records in W3C Web Annotation format. The contribution template in use by Pelagios’ Peripleo now is currently being updated to better account for typing of items and relations. Details of that new Linked Places annotation format (LPAF?) will be published soon. Collaborators in that modeling effort are most welcome!
 Twitter handles, in order: @gklyne, @RichardOfSussex, @mlex, @kintopp, and @azaroth42
The World-Historical Gazetteer project (WHG) has been under way for six months, and we’d like to let people know what progress we have made and our immediate next steps.
We digitized the index of “Atlas of World History,” a 1999 Dorling-Kinderley volume edited by historian Jeremy Black. This has given us approximately 10,000 places to seed our “spine” gazetteer, including cultural places like settlements, states, regions, peoples, and archaeological sites, and natural features like rivers and mountain ranges. Each entry is associated with one or more of the atlas’s 450 maps, each of which has a temporal coverage.
We have aligned approximately 70% of places in the atlas index with records in GeoNames and/or DBpedia so far, giving us geometry, additional name variants, and Wikipedia abstract text. In the near future we will also align records with the Getty Thesaurus of Geographic Names. A significant proportion of unmatched entries simply aren’t in existing gazetteers — but will be in ours!
We have gathered several data sets to augment the spine with additional cultural and natural features that will allow us to contextualize place records in our interfaces in some novel ways:
Societies (peoples) and related language regions (D-Place)
Rivers and lakes (Natural Earth)
Watersheds (World Resources Institute)
Mountain ranges (Natural Earth)
Terrestrial ecoregions (biomes; World Wildlife Federation)
Major ocean currents (NOAA)
In September, we held a kickoff meeting in Pittsburgh. Participants included members of our four data partner teams, several experts from the historical Linked Open Data “ecosystem,” and members of the Pitt and Carnegie Mellon digital humanities and library communities. We received a lot of valuable input, much of which we have compiled into a set of 70 “user stories.” These will inform design of our data models, graphical interfaces and API.
Our initial mapping of cultural features indicated that, as expected, some regions are somewhat under-represented. We are identifying a few significant historical print gazetteers and maps, digitization of which can help rectify the problem, and facilitating the work to extract and publish that data. The first such project will help fill a gap for 17-18c Latin America – the 1786 “Diccionario geográfico-histórico de las Indias Occidentales ó América,” and its 1812 English translation.
We also have worked to position our project in the the ecosystem referred to earlier. Over the past year, Technical Director Karl Grossner served as co-coordinator of the Linked Pasts Working Group of Pelagios Commons, which recently published a white paper – now open for comment as a shared document – “From Linking Places to a Linked Pasts Network.” We view WHG as joining Pelagios’ Peripleo system as a place-centered Linked Pasts Network Hub; more about that framing effort in due course.
We have begun experimental integration of the Getty Thesaurus of Geographic Names (TGN) into the WHG system. This extraordinary resource (~2.5 million geo-referenced places; ~4 million place names) has, like most existing gazetteers, limited temporal information. We’d like to facilitate temporal annotations to all indexed place records, as we move towards a truly historical gazetteer system.
Following a mid-January 2018 system design charrette, database and software development begins in earnest. Early design plans will be published for comment.
Data development will continue throughout the duration of the project, and discussions with several prospective data partners are ongoing. Our goal is supporting communities of researchers specializing in particular regions and periods, and guiding them to publishing place data we can incorporate in our “union index.” The initial focus is on Colonial Era Latin America and West Africa, Maritime Southeast Asia, and Early Modern Europe, but we welcome inquiries concerning any region/period combination.