| WENET Working Group, MIS Committee, Whatcom County Law and Justice Council |
R. Josh Nylander | |
| Memo: 24 | ||
| Status: FINAL | ||
| Obsoletes: Memo 18, WENET Protocol v1.2.2 | June 22nd, 2009 |
|
WENET
Protocol v1.2.3 |
||
Status of this Memo
This document specifies part of the WENET project standard. Distribution of this document is unlimited. This memo obsoletes previous versions of the WENET Protocol. This memo and the protocol it describes has been approved by the Management Information Systems Committee of the Whatcom County Law and Justice Council.
Copyright Notice
This document is released as a government work into the public domain.
Abstract
This document specifies the basic framework needed for participating in the WENET network. The purpose of the WENET project is to share law and justice data from local, state, tribal and federal law enforcement and justice agencies using a common language.
Table of Contents
This document defines a standard protocol enabling inquiry of disparate law and justice data systems. These data systems, called record management systems (RMS), are built on a variety of platforms with an assortment of technologies. On the surface, these RMS systems perform different tasks, but all store common, yet discrete pieces of data that are used throughout the law and justice process. This protocol and its companion schema document, define both a common data format for exchange and a standard process to authenticate, query, and retrieve commonly formatted data. This document builds on the concepts in the November 2002 report by the MIS committee of the Whatcom County Law and Justice Council entitled "Data Integration & The Criminal Justice System in Whatcom County ". This document also builds on the presentation given July 10th of 2003 in Bellingham to project members and vendors (Power Point portion of presentation is available). However in some instances it has been necessary to deviate from both the report and the presentation.
Keywords "MUST", "MUST NOT", "REQUIRED", "SHOULD", "SHOULD NOT" and "MAY" that appear in this document are to be interpreted as described in [RFC 2119].
For the purpose of this protocol the term "client" refers to an application, typically used by a person ("user") that queries a "service". A "service" (also sometimes referred to as a "service point") is considered an implementation of this protocol. A client typically makes requests and services typically provide responses. A single agency may provide multiple services. It is expected that a client will query against multiple services in order to compile the complete data set. Both services and clients may exist on the same or different hosts but must be able to communicate via HTTP over TCP/IP to the requested services.
This protocol defines a collection of methods available from a service. Depending on the purpose of the service, different methods will be required. Each method is defined throughout the rest of this protocol within the proper context. Finally, a table of the methods and a definition of the different service types along with required methods is provided.
This protocol uses Uniform Resource Identifiers as specified by the IETF [RFC 2396] for both URLs and for URNs [RFC 2141]. URLs are used by this protocol to specify the technology (often called protocol), the host, and the name of methods and data sources exposed by services implementing this protocol. An example of a URL used to identify the discovery method would be "http://wenet.co.whatcom.wa.us/as400/discovery.jsp". URNs are used to uniquely identify specific data sets such as a service description, an agency's/RMS's information about a person or an agency's/RMS's information about an event. An example of a URN identifying a person in an RMS would be "wenet.co.whatcom.wa.us/as400/person/johndoe". Each agency MUST be identified with a unique URN. Each service MUST be identified with a unique URN. Each unique data set, such as Person, Warrant, and Arrest data sets from an RMS, MUST be identified with unique URNs.
Clients and services MUST communicate via HTTP [RFC 2616] over TLS [RFC 2818] or SSL. Services must authenticate for all requests. Except for retrieval of service descriptor, which only requires service authentication, both services and clients MUST authenticate each other with valid and authorized digital certificates. Services MUST present a digital certificate that matches the certificate listed in the service document. Clients SHOULD verify that the certificate received from the server during SSL authentication matches the certificate listed in the service document of that respective server. Services authenticate one another by ensuring the other's service certificate is derived from one of the common “Service Signing Certificates”.
Most of the methods exposed by service points are accessed via the HTTP GET method. Each service provides (in their service descriptor) a list of methods, which includes their parameters and their corresponding URLs. In most cases, additional parameters are required, for example, the name of a person to search for. In this case this protocol specifies a specific text string, which MUST exist in the URL that is to be replaced with the appropriate parameter. In this way clients can, via simple string replacement and HTTP GET requests; retrieve results and documents from a service. All REQUIRED methods are implemented via HTTP GET. All HTTP GET URLs MUST be properly escaped.
Services may implement methods not documented in this protocol via POST. For each method requiring POST, the service document should specify the stream content and format expected by the service method. If not specifically identified the default is to accept parameters as application/x-www-form-urlencoded.
Whenever possible the service SHOULD respond with the most appropriate HTTP status code and SHOULD strive to extensively use status codes for error, missing or not found information. Specifically all clients SHOULD understand the following status codes: 200, 301, 305, 307, 400, 401, 404, 408, 410, 414, 500, 501, and 503.
A companion document [WENET Schema] to this one defines the schema for top level XML types used by this protocol. For each request there is a corresponding type specified in the schema that is returned by the service as a complete and properly formatted XML document with HTTP content type text/xml. This document will identify XML data types in italics with these custom types having the namespace prefix of wenet. For example, the WENET service document is of type wenet:service.
Discovery allows for clients to discover and connect to all available services and for services to connect to each other.
Clients and services may belong to more then one cluster. While outside of the scope of this document, it is anticipated that all agencies within a single cluster will come to common agreement on security and GJXDML rules.
Client software may combine results from multiple clusters.Each service will include in its service descriptor the list of clusters it belongs to. An example of a possible cluster configuration is with agencies A, B, C, D. Agencies A, B, C could be grouped in a cluster, while agencies B, C, D could be in another cluster. Users in agencies B and C can access both agencies A and D's data while users in agency A cannot access data from agency D and vice versus.
Within clusters, services MAY be organized into a hierarchy. This hierarchy allows for indexing services, proxies and other services in a priority fashion. Each service defines which, if any, service is its parent within a given cluster. Clients SHOULD first attempt queries against the services without parents and then move down the hierarchy until their request is met. A service that does not define a parent within a cluster is at the top of the hierarchy. Multiple services may exist at the same level in the hierarchy.
Each service MUST provide the discover method using HTTP GET for the retrieval of a XML document of type wenet:service. The service descriptor (a.k.a. service document) contains information about the service such as contact information, disclaimers, method access URLs, and digital certificate. The service descriptor also contains a list of the clusters it belongs to and the services in those clusters. In this way the client software SHOULD only need the URL of one service descriptor in order to discover other services participating in the data-sharing cluster. The service descriptor MUST be available without the client authenticating and MUST not contain privileged information.
Most methods, the URL and any parameters needed to execute them are defined in the service document. A few methods are intrinsic in the protocol and thus not listed in the wenet:service document, such as the discovery method itself or the result method which is listed in response to a query. The wenet:service document lists the methods supported by the service, the HTTP request type, a brief description, the URL to execute the method and the supported parameters. Only parameters which are optionally supported must be specified in the service document. Any required parameters MAY be specified but MUST be assumed if not present. In the case of HTTP GET methods, the parameters are applied to the URL with simple find/replace of the parameter name surrounded by exclamation points (!) found in the URL with the value. For HTTP POST parameters, they are applied as properly escaped form parameters.
Included in the service descriptor is a list of the xpath elements available to query against at that service. The client SHOULD verify that the query elements are supported by the service. Elements returned in the wenet:summary section of the wenet:results documents will be marked with a showInSummary attribute of type boolean set to true. A friendly name and description may also be provided as attributes.
Security in WENET is based on trust relationships between separate agencies. Agencies wishing to share data may enter into written contracts specifying the security relationship. This includes defining the types of users participating in each particular role. Such contract specifics are outside the scope of this technical document.
Services establish a trust relationship amongst each other so that they may directly query each other. As noted in section 6.1, it is anticipated that agencies in trust relationships will be grouped together within a cluster.
Authorization lists are specified by service point at the user level and define the roles a user belongs to. Since each agency is in the best position to know who of its users should have access to what type of data, it is expected but not required that authorization lists be provided by and maintained by each agency. The authorization list contains all of the names, other identifying information and what roles the agency says the users should be able to use.
Services MAY provide for method authlist via HTTP GET for the retrieval of a XML document of type wenet:authlist which contains the list of authorizing certificates and a list of users that the service represents. The document contains a list of users and the roles the users are a member of. Each user is identified by an e-mail style username. This username SHOULD be a valid e-mail address but at the least MUST be globally unique. The username must match the username identified by the user’s digital certificate(s). In addition to a username, each user has a URL that MAY be used to retrieve information about the user. This MAY be a XML document containing contact information. Each service MUST from time to time contact the other services in the clusters it belongs to in order to retrieve the latest copies of authorization lists. The service MUST only allow other services to retrieve the list. Services MUST identify themselves to each other by authenticating their end of the connection with a the wenet:serviceCertificate and this certificate MUST be signed (or have a parent signed) by a certificate trusted by the other service. It is anticipated that each service will have a list of one or more certificates that it will use to establish trust of service certificates.
Each service implementing the authlist method MUST provide in their wenet:authlist a list of digital certificates which are authorized by the service to sign user certificates for the users it represents. In this way the client authenticates, on behalf of the user, with a user digital certificate signed by an authorized certificate. The user's digital certificate MUST contain the X.509 v3 extended attribute "Subject Alternative Name" which MUST contain the value "RFC822 Name=username" where username is an authorized username (see section 7.1). The service authenticated against uses the username to determine if the user is authorized to make the request and to log the request.
Each service that provides query or instance methods MUST record the username, date, time, request URL, and POST content, if any, of each authenticated request made. This record MUST be made available as method auditlog via HTTP GET as a XML document of type wenet:auditlog. The auditlog method takes two parameters, "!xpath!" and "!namespace!". The client is to replace these strings with the XPath query statement and namespace respectively. The XPath query MUST be a valid XPath rooted in the wenet:auditlog element. This method MUST only be available to other services in the same way as the authlist method. Services MAY log unsuccessfully authenticated requests. If not logged here, services SHOULD log all attempts at web server contact in a log available for review by the local security auditors.
8. Query
The primary function of a service is to provide for querying of an agency’s RMS system and retrieval of RMS data. Services make RMS data available in the form of XML documents of type wenet:instance.
Services may provide a method to query via HTTP GET for the retrieval of a XML document of type wenet:results. This document contains a list of instances which match the XPath query passed via the GET URL. XPath v1.0 syntax SHALL be used to query the data sets made available by the service. The XPath statement is made against XML documents of type wenet:instance rooted in a root element supported by the service (see section 6.3). The service descriptor specifies a URL for this method. The URL is a standard URL except where the XPath statement should go the URL contains an unescaped string, "!xpath!". The client is to replace this string with the XPath query statement.
In addition to the "!xpath!" parameter, the URL also contains the "!namespaces!" parameter. In this parameter MUST go the list of namespaces and namespace prefixes used in the XPath. The format for this is "xmlns:[prefix]='[namespace]'" (without the double quotes). For example, to map prefix j to namespace http://www.it.ojp.gov/jxdm/3.0.2 would look like (without double quotes) "xmlns:j='http://www.it.ojp.gov/jxdm/3.0.2'". Multiple namespaces are to be seperated by a space character (properly escaped).
In this way the client may create an XPath statement, escape it if necessary, substitute it into the URL, authenticate to the service using a valid digital certificate, make an HTTP GET request with the URL and retrieve a collection of results in the form of a wenet:results document and then resolve that list to retrieve the requested data.
The wenet:results document contains a collection of result elements. Each result element contains the instance URI, the agency URI, the service URI, the date/time last updated, a subset of the data, and most importantly the URL that may be used to retrieve the instance document. If there were no matching results then the collection will be empty. In addition to the result elements, the wenet:results document contains information identifying what service it comes from, the url used to query it, a check back url and any errors.
In some instances the service point may not be able to deliver results within a reasonable HTTP time window. In those cases the service point MAY queue the request, return an empty result set and provide for retrieval of the result set later. In this case, the check back url element contains a URL that the client can use to check back with the service to see if the processing is finished and if so retrieve the results. This feature is OPTIONAL. The wenet:checkBackURL element should only exist with an empty elements collection and when the check back later feature is implemented by the service. The client MAY check back at the URL at any time. If the results are not ready then the service will return an empty result set with a wenet:checkBackURL element for the client to try again later. If the service implements the check back option, the service MUST keep check backs open until successfully retrieved or 24 hours have expired from the time the results become available. The client MAY give up at any time.
If there were errors in processing the query and an HTTP error other then 500 is not appropriate to describe the error, the wenet:results MAY contain one or more wenet:error elements with the description of the error and an attribute, wenet:errorNumber, which is from the list below. HTTP error 500 should be sent when returning one of these errors.
Error Number Description 1 invalid XPath 2 insufficient rights
3 unsupported node(s)
Each service must provide via HTTP GET for the retrieval of a XML document of type wenet:instance. The document identifies the unique URI for this data, when the data was last updated in the RMS, the unique URI for the agency owning the the data, the unique URI of the service providing the data and a URL for retrieval. The wenet:instanceElement of the document contains a top level data element from the list of supported root elements (see Section 6.3).
In support of data warehouse “clients”, two optional methods are defined to allow the updating of a data warehouse based on what instances have changed and which ones have been deleted. Both methods should return a wenet:results document but without summary elements. Services implementing these one or both of these methods may need adjustments to the underlying RMS to track record updates to the date/time and to track which records have been deleted.
The getchanged method MAY be implemented by a service. It takes three parameters, datesince, root and namespaces. The root and namespaces parameters define which type of instances to return for. The datesince parameter should specify a date and optionally time in the xsd:dateTime format to return the results list for. Any records changed since the datesince parameter and matching the root/namespaces parameters MUST be returned in a wenet:results document, minus the wenet:summary element.
The getdeleted method MAY be implemented by a service. It takes three parameters, datesince, root and namespaces. The root and namespaces parameters define which type of instances to return for. The datesince parameter should specify a date and optionally time in the xsd:dateTime to return the results list for. Any records deleted since the datesince parameter and matching the root/namespaces parameters MUST be returned in a wenet:results document, minus the wenet:summary element.
Method
In Methods List
HTTP
Parameters
False
GET
True
GET
True
GET
xpath
namespaces
True
GET
xpath
namespaces
False
GET
True
GET
datesince
root
namespaces
True
GET
datesince
root
namespaces
By mixing and matching methods different types of services can be defined. This protocol defines six named types of services. All services MUST implement the discover method. It is expected that most services will be Sharing with Users and Changes Services.
A Sharing Service implements the discover, auditlog, query and result methods.
11.2. Sharing with Users Service
A Sharing with Users Service implements the discover, auditlog, authlist, query and result methods.
11.3. Sharing with Changes Service
A Sharing Service implements the discover, auditlog, query, result, getchanged, and getdeleted methods.
11.4. Sharing with Users and Changes Service
A Sharing with Users and Changes Service implements the discover, auditlog, authlist, query, result, getchanged, and getdeleted methods.
A Users Only Service implements the discover and authlist methods.
A proxy is a special type of service that doesn’t have to retrieve data from a local RMS. Rather it can choose where and how to get the data to return. A Proxy Service implements the discover and auditlog methods. It may optionally implement any of the other methods.
One option would be to store a subset of data from other service points in a local data warehouse to speed common searches. For example, to speed name searches it could cache locally all the names available on the service points it proxies for. If the XPath matches fields it caches, it can present a wenet:results document using local data but with instanceURLs pointing back to original services.
Another option would be for it to add value to a result list or instance document. An example might be a Geographic Mapping proxy service that intercepts data returned by other service points and builds a map based on all geographic information contained in the results, and then includes the map data with the returned results. This could involve GeoCoding the address and amending the latitude and longitude to the XML of the result, instance or both prior to returning the XML to the client.
Changes made since initially published on October 30th of 2003.
- 20090615 - final draft of v1.2.3 published
- Included spelling and grammar fixes
- Updated table of contents
- Restructured document content for better flow and reading.
- Fixed a number of grammatical errors.
- Improved and renamed the two new methods, getchanged and getdeleted.
- Added concept that all methods except for discover are optional and that different combinations define different types of services.
- Defined 6 types of services including moving the proxy concept to these new definitions.
- Modified the methods section to support the concept of a users only service point. Removed reference to a xpaths list method that does not exist.
- Added authlist to list of methods in table in section 4.5.1.
- Added two new optional methods in table in section 4.5.1, getallkeys and getdeletedkeys.
- 20051201 - v1.2.2 initial draft
- Modified sections 4.1 and 6.2 to reflect that retrieval of the discovery document shall be over HTTPS with server authentication only.
- 20050714 - v1.2.1 final draft
- Removed namespace parameter from simple query
- Added requirement that XPath use the namespace prefixes noted in the service document
- 20050601 - Added note about existance of the wenet:summary data and replaced instanceRoots with supportedXPaths.
- 20050418 - v1.2.1 released as initial draft, major changes include:
- Removed support for complex xpath
- fixed spelling, typos, and poor grammer
- clarified that each service has ONE certificate
- clarified clusters
- removed further dependence on the GJXDM and transfered dependence to the Technical Policy
- clarified user information retrieval
- clarified audit logging to require only authenticated requests and what SHOULD be done with others
- Added error number 3, for unsupported node(s) in a query
- 20041123 - removed from DRAFT
- Marked document as FINAL
- Updated date
- Changed order of "Change Log" to latest first
- 20041104 - major changes
- increased version number to 1.2
- spelling and grammar fixes
- defined methods, their names and parameters
- added XPath roots method
- fixed URI examples
- enhanced services trust model
- cleaned up cluster and hierarchy model
- users belong to roles not groups
- moved authorizing digital certificates to authlist
- refined both instance and results definitions
- improved results error handling and moved list of error messages to protocol
- added checkBackURL feature
- defined XQuery and changed how they are sent
- Made auditlog method only available to other services
- Removed requirement that instances be based on the GJXDM
- Enhanced auditlog method with an xpath query parameter
- 20040514 - major changes
- Increased version number to 1.1
- spelling and grammar fixes
- fixed name of Justice XML dictionary to GJXDM
- added 404 and 414 HTTP error codes
- service parent/child relationship within a cluster
- removed inter-local agreement reference in cluster and replaced with just agreements
- created a section on wenet:results
- added error messages to the wenet:results description to detail error messages
- added support for namespaces
- added better support for complex queries, including XQuery
- changed security model from method/document type to groups
- 20040316 - Added XPath examples to 8.2.