Technical Specifications for MassLNC August 2011 RFP
Created by:  Equinox Software, Inc.
Date:  11/29/11

ESI Service 01:  Infrastructure supporting multiple requests

ESI Service 02:  Data flattening service

ESI WIDGET-01:  Dojo-based interface component to present read-oriented interface for viewing tables of data

SC-A1:  Per-Screen Column Layout (high priority)

SC-B5:  Options for double clicking (high priority)

SC-B8:  Improved navigation of ACQ providers and funds (high priority)

SC-E3:  Individual screens use its own settings for receipt templates (high priority)

SC-A5:  Line numbers for display screen (medium priority)

SC-B1:  Customizable toolbar (medium priority)

SC-E1:  Automatic reload (medium priority)

SC-A4 (Circ06 is dependent on):  Multiple-level sorting (low priority)

SC-B4:  Easy return to search results from MARC record view (low priority)

SC-C1:  New tab button (low priority)

SC-D3:  Removal of unused fields from copy editor (low priority)

Acq-02:  Record matching and overlay in Acquisitions (no assigned priority)

Pac-04:  Flexible scopes (no assigned priority)

Pac-05:  Greater flexibility in library selector display (no assigned priority)

Sys-07:  Last date of authentication (no assigned priority)

User Activity Additions:  Last Date of Patron Authentication Part II*

(*This development is being donated by Equinox Software, Inc. to the Evergreen OS ILS Community.)

Circ-06 (depends on ESI Widget-01, SC-A1 & SC-A4):  Configurable holds pull list (no assigned priority)

ESI Service 01:  Infrastructure supporting multiple requests

This document explores the changes that we will make to the open-ils.pcrud service to meet the requirements of ESI-Service-01 as described in our response to the  MassLNC Aug 2011 RFP.

The open-ils.pcrud service today exists as a public-oriented variant of the open-ils.cstore service with comparatively limited functionality.  Cstore acts a general purpose data retrieval and modification service, with advanced features supporting the retrieval of multiple layers of related objects per request and no access control per se, as it is intended for use by other trusted Evergreen services.  Pcrud, by contrast, does not support the retrieval of multiple layers of related objects, and in fact only allows retrieval of one class of object at a time, in order that an authorization scheme may be easily enforced.

Many interfaces in the Evergreen staff client rely on pcrud today to deliver the data used to populate grid-based interfaces.  Because of pcrud’s current limitations, any information that must be presented involving more than a single class of object (for example, a hold plus a targeted title, volume, and copy, and the requesting user involves five different classes) necessitates multiple requests to pcrud or other backend services to retrieve related data, and any efforts to retrieve and manipulate those objects in tandem require code to specifically written for the occasion.  The production of such code becomes highly repetitive over the course of implementing several similar but distinct grid-based interfaces, but if pcrud is made to support multiple layers of related objects per retrieval request (fleshing) as cstore does, great amounts of time to produce similar grid-based interfaces can be saved.

Enabling fleshing for pcrud is therefore the first change we will make. This is simple by itself. But the reason why pcrud doesn’t already support fleshing is that it has not been designed to check for user authorization to access multiple classes of objects that may be related at different layers of a retrieval request.  This brings us to the second requirement of this service.

Extending authorization testing for pcrud to include fleshed objects is also necessary.  The pcrud service, let us remember, is publicly exposed to the Internet (because it is used by the staff client as well as by the OPAC, among other things), and so it enforces access control. If it is to be improved to provide access to multiple layers of objects at a time, it must be also improved to verify authenticated users’ permission to access all of the different classes and instances of objects that might be yielded in response to a given request.

Finally, since ESI-Service-01 is also intended to enable the open-ils.pcrud service to serve as the basis for grid-based interfaces that exchange large amounts of data (e.g. holds pull lists and holds shelf browsing) and not just interfaces that let users configure small tables (as it currently tends to be used), it must also be made to respond to requests faster.  Therefore retrieved objects should be returned as soon as they are ready instead of accumulated as a batch and returned all at once.  The underlying OpenSRF layer supports this kind of streaming operation, and pcrud simply needs adjustments to take advantage of it.

ESI Service 02:  Data flattening service

Evergreen supplies a broad API for accessing, processing and interacting with library data.  Because of the relatively complex nature of the underlying database schema, and the flexibility required by clients when, in the simplest case, performing CRUD operations, focus has been given to providing a nearly direct view of various data source.  When, however, the verbosity or complexity of full object access gets in the way of performant or straight-forward UIs, there has been a tendency to create one-off data simplification calls targetting specific use cases.

A generalized API which accepts a simplified query structure and field mapping, calculates the required backend query, and flattens the hierarchical data returned for each top level row into a would facilitate the simplification of existing UIs and provide for new UIs based on simple, reusable components.

Overview

The existing, public open-ils.fielder server will be extended with two new OpenSRF methods, contained in a separate package so that they will be reusable in a private service which does not require authentication.

These methods will be supported by code which takes simplifed cstore/pcrud search and order-by hashes and computes the full data structure required for the query.  The simplification will leverage the IDL to discover links between classes.

Underlying the simplified search grammar will be a path-field mapping structure.  This will describe the layout of fields, how they are to collapse from fleshed objects, and how the shape of both the query and result data structures should be interpreted by and presented to the caller.

Mapping Structure
Implemented as a JSON object, each property name will represent a data element that can be displayed, filtered or sorted, and each property value will represent either a simple path (in which case it is usable for display, filtering or sorting), or an object providing the path and available uses.

Example Map
Assuming a core class of circ:

{ "patron_barcode": "usr.card.barcode",
  "circ_lib_name":  "circ_lib.name",
  "circ_lib":       "circ_lib.shortname",
  "id":             "id",
  "xact_start":     { "path": "xact_start",  "sort": true, "display": true },
  "checkin_time":   { "path": "checkin_time",  "filter": true, "sort": true }
}

Based on this mapping structure simplified queries can be constructed by the caller, which will then be expanded in the flattening service to produce join and where clauses that can be used by open-ils.pcrud.

Example Query
Assuming the above example map:

{ “xact_start”: { “>”: “today” }, "circ_lib”: “BR1” }

This example would expand to a PCrud query based on the map provided above, containing not only the complex where clause, but a join tree and the necessary fleshing structure.

Expanded PCrud Query

{ “xact_start”: {“>”: “today”},
  “+circ_lib_aou”: {“name”: “BR1”}
},  {
  “join”:    {
    “aou”:     {
      “alias”: “circ_lib_aou”,
      “fkey”: “circ_lib”
    },
    “au”: {
      “alias”: “usr_au”,
      “fkey”: ”usr”,
      “join”: {
        “ac”: {
          “alias”: “usr_card_ac”,
          “fkey”: “card”
        }
      }
    }
  },
  “flesh_fields”: {
    “circ” : [“circ_lib”,”usr”],
    “aou” : [“card”]
  },
  “flesh”: 2
}

API

OpenSRF Method name: open-ils.fielder.pcrud.search
Parameters:

• Authentication token
• IDL class

“circ”

• Path map hash

{“patron_barcode”:“usr.card.barcode”,“circ_lib_name”:”circ_lib.name”,

“circ_lib”:”circ_lib.shortname”,”xact_start”:{ “path”: “xact_start”, “sort”: true, “display”: true},“id”:”id”,“checkin_time”:    { “path”: “checkin_time”, “filter”: true, “sort”: true }}

• Simplified query hash

{“xact_start”: { “>”: “today” }, “circ_lib”: “BR1”}

• Simplified sort/limit/offset hash
  • { “sort”:[{“checkin_time”:”desc”,”circ_lib”:”asc”}],”limit”:10}
• Returns: stream (or array, for .atomic) of hashes having the shape described in the path map

{“patron_barcode”:“3123456789012”,“circ_lib_name”:”Branch 1”,

“circ_lib”:”BR1”,”xact_start”:”2011-10-17T12:00:00-05:00,“id”:”123”}

ESI WIDGET-01:  Dojo-based interface component to present read-oriented interface for viewing tables of data

To take advantage of ESI Services 01 and 02 (pcrud fleshing and data
flatenning), we need UI components.

This describes how we’ll get from the output of the data flattening service
to interfaces we can use.

The data flattening service proposed API suggests a method that takes the path
map hash, the simplified query hash, and the simplified sort/limit/offset hash
as arguments, returning a stream or a list of result rows.

0) The data flattening service should get a secondary API method that takes only
the path map hash as an argument, and returns a cache key.  The cache key should
be usable in place of the path map hash argument to the first API method described.  This saves repetition of a potentially large object in the scenario
described next.

1) A mod_perl service should be constructed to offer the following RESTfully:
given a key to a cached path map hash, a query object, and and a
sort/limit/offest object (all in url parameterized form), offer search results
in the form of a JSON object oriented for consumption by Dojo’s
ItemFileReadStore.  Such a service would make read-only grids almost trivial
to implement using Dojo’s DataGrid.  Thise service may also offer other output formats.  Other formats may include JSONP, XML, CSV, Display HTML (full page, including query description and embedded data), Print HTML (self-printing on page load, with print-media CSS), HTML subset (simple HTML table) and PDF.

2) Once the above mod_perl service is complete, we can subclass Dojo’s
DataGrid and potentially borrow some elements from our existing AutoGrid
widget.  Specifically, we can provide row editing dialogs if we limit
the fields that can be edited to those directly on the “core” class in our
search query.

* We can also connect DataGrid features supporting sorting on multiple columns to changes in the URL used for the grid’s store, giving us more low-cost-to-implement functionality.

SC-A1:  Per-Screen Column Layout (high priority)

The goal is for Evergreen to support applying and saving distinct sets of columns for any individual screen using the column picker. In particular, distinct settings should be used for the holds view in the patron holds screen, the title holds screen, the browse holds shelf screen, and the pull list for holds request screen.

AutoGrid-based interfaces already potentially support this, and can just have attributes set to do it.

For XUL-based interfaces, any interfaces outside of those mentioned explicitly in the requirements (all relating to holds) should already support the desired functionality.

For those interfaces related to holds, the issue is that those interfaces are really a single interface with different “modes.”  The change required to save column layout individually per “mode” is for the interface Javascript to alter the id attribute of the table widget (actually called a tree by XUL) to give it a specific ID per “mode.”  This id will then be passed to the save_columns() method in the util.list class, which is already prepared to save column layouts separately by ID.

SC-B5:  Options for double clicking (high priority)

For XUL lists, we will add the ability to handle an on_dblclick key and callback to the object we pass into util.list.init, as a peer to the existing on_select.  A function thus defined will work in the same manner as on_select, and is responsible for calling util.list.retrieve_selection(); if needed and working on the rows thus selected.  Double-clicking a row effectively selects it, though it’s also possible for a double-click to work on a multi-selection.  So on_dblclick could rely on on_select for gathering the row data to affect, at least with current xulrunners, though it may be more robust to call retrieve_selection() a second time.

Once this is implemented, we will use this infrastructure to enhance two interfaces and augment their respective lists with double-click behavior:

• Patron Search

The file display.js embeds search_result.js, and passes through an on_select when the search interface is loaded.  We will pass through an on_dblclick as a peer to on_select and modify search_result.js to make use of it.  The function passed through on_dblclick will simply fire off a virtual command event using util.widgets.dispatch for “cmd_patron_retrieve”, triggering the existing behavior for retrieving selected patrons.

• Holdings Maintenance

In copy_browser.js, we will pass an on_dblclick handler as a peer to on_select with util.list.init.  It will rely on on_select having fired and gathering the pertinent row data, and it will use util.widgets.dispatch to fire off a virtual command event, “cmd_edit_items”, to leverage the behavior for the existing Edit Items action.

SC-B8:  Improved navigation of ACQ providers and funds (high priority)

Navigating through provider and fund lists in ACQ can be unwieldy, because it may require the user to page through many pages of data before the user finds the data he is searching for.  Two changes are proposed to make it easier to navigate through large data set.

1. Only retrieve funds and providers the user has permission to use/view.  This will reduce the overall number of pages of data that may need to be scanned.
   • On the provider, fund, and funding_source administration interfaces, the main display grids will be replaced (or augmented) by grids that are capable of taking advantage of the new PCrud flattening service for improved permission handling and to speed up the display.  See requirement SC-A4.
2. Allow for filtering on fields in the grid.  This will create additional opportunities for limiting the total number of values returned.

• On the provider, fund, and funding_source interface, plug in the improved PCrudFilterDialog to, which will allow users to select multiple filter fields and apply filter values.  PCrudFilterDialog will require the addition of operator selection.

SC-E3:  Individual screens use its own settings for receipt templates (high priority)

There are several XUL-based holds list that are all implemented with the same holds.js file:

• Actions for this Record -> View Holds

• Patron Display -> Holds

• Circulation -> Browse Hold Shelf

• Circulation->Pull List for Hold Requests

• Admin->For Developers->Browse Unfilled Holds for this Pickup Lib

The main Print action (from the “Print” button next to the List Actions menu) uses the same template, “holds”, for all incarnations of the interface.  We will change this behavior in holds.js (specifically in the cmd_holds_print method) such that each interface variation will use its own template.  The new templates will be:

• holds_on_bib

• holds_for_patron

• holds_shelf

• holds_pull_list

• holds_unfulfilled

We will keep the “holds” template for backwards compatibility and as a fallback for when these new templates have not yet been configured.  These new templates will be stubbed in data.js (in the print_list_defaults method) so that they will appear in the Receipt Template Editor, but by default they will not have any defined content for their headers, footers, and line items.  Instead, we will use a new field called “inherit”, and have each new template use “holds” (referring to the original template) as the value for their inherit fields.
So, for example, the current default ‘holds’ template is defined like this:
‘holds’ : {
‘type’ : ‘holds’,
‘header’ : ‘Welcome to %LIBRARY%!<br/>\r\nYou have the following titles on hold:<hr/><ol>’,
‘line_item’ : ‘<li>%title%\r\n’,
‘footer’ : ‘</ol><hr />%SHORTNAME% %TODAY_TRIM%<br/>\r\nYou were helped by %STAFF_FIRSTNAME%<br/>\r\n<br/>\r\n’
},
The new ‘holds_for_patron’ template will be defined as a peer like this:
‘holds_for_patron’ : {
‘type’ : ‘holds’,
‘inherit’ : ‘holds’
},
We will modify the _print_tree method in list.js and the post_init method in print_list_template_editor.js such that they will react to any value in a template’s inherit field and allow it to redirect them to use the contents of the inherited template.  For this particular use-case, we only need to support one level of indirection, but we may opt to support chains of inheritance for future use.

We will modify the save_template method in print_list_template_editor.js so that the inherit field for a given template will be cleared if that specific template is saved, breaking the link and associating the displayed (and possibly edited) header, footer, and line item with the template.

SC-A5:  Line numbers for display screen (medium priority)

Evergreen list interfaces currently do not supply a row-number column, which would be useful for scanning a long list without losing ones place.

There are two main classes of lists with column pickers.  “XUL” lists are used in interfaces that have a gray background, with an icon in the upper right of the list after the last column for invoking the column picker.  “Auto-Grid” lists are used in interfaces that have a white background, and the ones with column pickers typically have a labelled button near the list for invoking the column picker.

• XUL lists

o A given XUL list is fed a list of column definitions. We will provide a column definition to each list for “Line Number” (the actual label we can tweak. Perhaps “#”, “Line”, or “LineNo”). Normally, these would work like any other column, and may be sorted, hidden, resized, and repositioned. We’ll take care to disallow sorting.
o We will make the util.list class automatically generate these columns, and thus the column definition will not need to be fed explicitly through the columns parameter during list.init. Since the column will be defined from within util.list, it will have access to util.list internals, and for a given row we could, for example, iterate and count through all of obj.treechildren.childNodes and compare against params.my_node to determine our ordinal positioning and render that as the value for that cell. This is a simplistic algorithm, and we may come up with a better alternative.
o We will modify the click handler for column headers to ignore the Line Number column based on an attribute within the column definition.
o Optional: We can give the cells for this column a CSS addressable property, so that we may style them differently than the rest of the list (to, for example, make them look less like data and more like UI).

* Auto-Grid lists

o Add a new virtual column to AutoGrid for “Line Number” (though perhaps with an icon instead of the space-gobbling text), next to the virtual checkboxes/selector column. The new field’s get() function will simply return the row number (already available) for display.

* PCrudFlattener-Aware-Grid (ESI-WIDGET-01)

o Process will be the same as Auto-Grid (or directly inherited).

SC-B1:  Customizable toolbar (medium priority)

There are currently two toolbars for the Evergreen staff client, one geared for Circulation staff and one for Catalogers.  One or the other may be set as the default for a given workstation, and the actual buttons on the toolbars are hard-coded in a XUL file.

We will re-implement these with a single XUL toolbar element, where the actual buttons in use are cloned from a common pool of toolbarbuttons hidden in a non-rendered XUL element.  The order and set of buttons (and any separators or spacers) thus cloned is data-driven, depending on the selections made in the Admin -> Workstation Administration -> Toolbars menu.

Actual layout data will be whitespace separated ordered lists of identifiers corresponding with the values stored in “picker_id” attributes on the available set of toolbarbuttons (and spacers and separators).

We will add the following toolbarbuttons to the existing set:

Home (return to portal page)

Copy buckets

Record buckets

Renew items

Pull list for holds requests

Browse holds shelf

Retrieve last patron

Acquisitions search

Acquisitions My Selection Lists

Acquisitions New Brief Record

Acquisitions Patron Requests

Acquisitions MARC Federated Search

Acquisitions Load MARC Order Records

Acquisitions Create Purchase Order

Acquisitions Claim Ready Items

Acquisitions Create Invoice

Booking Create Reservations

Booking Pull List

Booking Capture Resources

Booking Pick Up Reservations

Booking Return Reservations

We will make the list of available toolbars configurable.  A given toolbar will have a name and an implementation (specifying how/where to get the layout data).

For example, we could do such things as the following:

Circ toolbar -> preference: oils.toolbar.circ.layout
Cat toolbar -> file: http://hostname/xul/server/skin/toolbars/oils.toolbar.cat.layout
Acquisitions toolbar -> org unit setting: ui.toolbar.acq.layout
Serials toolbar -> file: chrome://open_ils_staff_client/skin/toolbars/serials.layout
User toolbar -> user setting: ui.toolbar.user.layout

Considering the chrome:// example we see with the Serials toolbar above, we could iterate through a toolbars directory much like we do for hotkeys and .keyset files, and inform the staff client of one set of available toolbars in that manner, with toolbar names being derived from the corresponding filenames.

The list of available toolbars could be augmented by other mechanisms, such as JSON data structures coming from mozilla preferences, org and/or user settings, and from the global OpenILS.data datastore (which could be modified, along with preferences, for example, by server/skin/custom.js)

We will re-implement the existing Circ and Cat toolbars as .layout files in the chrome skin directory, and will hard-code a user setting for a User toolbar and an org unit setting for a Library toolbar.  And Workstation toolbar powered by a preference.  That should cover our bases.

For preference and user setting and org unit setting powered toolbars that get included in this list of available toolbars (such as the User, Library, and Workstation toolbars proposed above), we will provide a dedicated interface for manipulating the layout data, under Admin -> Workstation Administration -> Toolbars -> Edit Layout.  This interface will consist of a drop-down menu containing the list of available toolbars/settings to manipulate, and two lists, one containing identifiers or labels for the buttons “in use”, and the other for the buttons “available”.  Labels may be provided by “picker_label” attributes on the actual toolbarbuttons, and we can default to the normal label attribute in the absence of picker_label.

If one or more rows are selected in the Available list, a button with an arrow pointing to the In Use list may be pressed to either move or copy the rows to the In Use list (I’m leaning toward Move except for Separators and Spacers).  Similarly, there will be a button for moving rows back to the Available list (and for removing Separators and Spacers from the In Use list).  For the In Use list, there will be Up/Down buttons for re-ordering a selected row in the list.

When this interface is active, the toolbar being edited will temporarily be made the active toolbar (if not already active), so that changes made to the toolbar can be previewed in real-time.

A Save button will make the change permanent, and will be subject to normal permission handling for org unit and user settings, and in the case of the Workstation toolbar powered by a preference, maybe no specific permission for the time being (if you’re logged in and using the workstation, have at it).

SC-E1:  Automatic reload (medium priority)

In some instances, when data is changed within the Evergreen staff client, the new data is not reflected in the interface.  The following interfaces should be modified to support real-time display of updated data.

Catalog bib record display: reload detail page after items added, issues received.

• Create a function in the OPAC wrapper that can reload the detail page (or any OPAC page).
• Expose said function via xulG to child interfaces (particularly Holdings Maintenance, Bib Brief Summary, and the Serials interface)
• Modify said interfaces to call the function as appropriate when the OPAC needs to be reloaded.

Checkout: update checkout count as checkouts occur; update items-out list in real time.

• There is deprecated code which attempted to do this, so this will be revived and improved.  In particular, we should rename the callback “on_list_change_old” that gets shoved into the Checkout Interface via xulG, and make sure that it gets called as a peer to the current “on_list_change” (which handles updating the summary).
• “on_list_change_old” attempts to feed the id for the circ just handled to the Items Out list, where that list’s own retrieve_item function will handle the fleshing (redundantly, in this case).
  • we can reduce redundancy and bring parity to the shape of the data between the checkout list and items out list
  • the function currently only works if the Items Out interface has already been open.  This is probably okay, since when the Items Out interface is actually opened for the first time, it fetches all the circs.
• There are more summary points that “on_list_change” may need to update, so we need to extend that function and verify that it’s working generally.

Import Records: Update import selector widgets to reflect new entries

• Any time a user returns to the main import screen (“Import Records”) or the “Inspect Queue” tab, ensure that the screen is freshly loaded.  We’ll do this by making the “Import Records” and “Inspect Queue” tab actions be links instead of the div-swapping actions currently in place.  The remaining Vandelay displays are Grid-based and should be updated automatically as new data is added.

Purchase Order: Allow activation after entering prices w/o manual reload

• After a price is entered for a lineitem, if all other lineitems in the list also have prices, automatically recalculate whether the PO can be activated with the new price data and update the Activatable action in the display accordingly.  This re-calculation will be performed in “authoritative” mode.

SC-A4 (Circ06 is dependent on):  Multiple-level sorting (low priority)

MassLNC’s requirement: “Any interface using column pickers can be sorted by multiple columns.”

In any XUL based interface, each column header should get a context menu with two entries.  The first should be “sort by this column (first)” while the second should be “sort by this coulmn next.”  The second entry should be greyed out until the something is already selected for “first” sorting (whether through the context menu entry or through the usual single left click).  The effect of “sort by this column next” should be cumulative, so that a user can keep adding columns by which to sort.  Control+click should be a shortcut for “sort by this column next”.

Where existing AutoGrid-based interfaces don’t otherwise need the new pcrud flesh and flatten features, AutoGrid can simply be improved to sort on multiple columns, making a call to the server with an adjusted order_by clause in a similar way to how it currently implements paging.  The user will select columns for sorting via either context menus in a fashion analogous to that described for XUL-based interfaces, or via a menu accessed by hyperlinks, whichever is more readily integrated with the existing fundamental widgets at implementation time.

New grid interfaces based ESI Widget 01 will be using a service that inherently supports sorting by any number of columns, and the column selection interface will be made in a way that’s consistent with that described above for XUL-based and AutoGrid interfaces.

SC-B4:  Easy return to search results from MARC record view (low priority)

We need to add a “back to search results” button in the part of the staff client’s chrome that currently displays the start, previous, next and end buttons, to be displayed when the user is in lower-pane displays other than the OPAC View.

By adding a new “event” in the old runEvt/attachEvt system to be run from the search results page, and to be “caught” in opac.js, we can keep our notion of the most recently viewed search results page up to date, and we can offer a button for “Back to search results” in the desired situations.  The same “event” system is used to prepare the start/previous/next/end buttons already.

The rest of the time we keep that button hidden, as at least in the OPAC view, it’s redundant with an in-browser link.

The TPac detects when the referer to a results page is the same record detail page that it wants to redirect (on 1 hit) to and skips the automatic redirect.  Because of this, adding the ‘back to search results’ button on a single-hit-redirect record page will simply take the user to the results page for that one record, and doesn’t have to be treated as special.

SC-C1:  New tab button (low priority)

Currently in Evergreen one can open a new tab through a hot-key or File menu option.  However, in addition to these, there is a dedicated close-tab button on the tab bar for closing tabs.

In menu_frame_overlay.xul, we will create a <toolbarbutton command=”cmd_new_tab”> as a peer to the one for closing tabs.  Experimentation will be required to find a working icon for the button and a useful and aesthetic location for it.  Current versions of Firefox have a single newtab button flush against the last tab, and one closetab button per tab.  Our current staff client has a single closetab button to the far right of the tab bar.  We would place the <toolbarbutton> to the left of the <arrowscrollbox> and have the button be the first thing on the tab bar to the far left, but that wouldn’t match the scheme of any major browser using tabs.

SC-D3:  Removal of unused fields from copy editor (low priority)

Currently in Evergreen there is no generalized infrastructure for customized hiding of interface elements.  Several interfaces do implement independent, one-off versions of this, such as the user registration interface.  We will leverage the general pattern of Org Unit settings to inform general infrastructure about what should be hidden based on local policy.

We will create a JSAN util/hide.js function library in the vein of widgets.js.  It will contain the following functions:

• util.hide.generate_dialog
• util.hide.generate_css

Both functions will take two arguments, the database name for an org unit setting, and an org id for the context org.  If an org id is not specified, we will default to the workstation library.

generate_dialog will create a node list from the window where it is invoked, gathering all nodes that have an attribute of “hideable”.  It will remember the values of the hideable attribute and link them to the value of any “label” attribute.  Only one label will be associated with a given hideable value.  The function will then pop open a modal dialog window containing a list of checkboxes for each hideable value, and using the remembered label to show to the user.  There will also be a Save button and a Cancel button.  If Save is pressed, then all the checked holdeable values will be saved to the specified org unit setting as a space delimited string of values, the dialog will close, and generate_css will be invoked.  Cancel will close the dialog and do nothing else.

generate_css will read the values from the specified org unit setting, and will then manipulate CSS to hide any nodes containing hideable attributes on the page where it is invoked that contain the values from the org unit setting.  We will investigate having the function inject actual CSS into the page, but as a fallback strategy we can have the function append CSS classnames derived from the holdeable values onto the document.documentElement, in the same vein as patron.util.set_penalty_css.  We will then rely on stock CSS to hide the elements based on the presence of the documentElement CSS classnames and the holdeable attribute values.

For example, let’s say that the <groupbox> in the Item Attribute Editor for “Deposit” has a hideable attribute with the value “Deposit”, and the <caption> element for that groupbox also has the same hideable attribute.  Then generate_dialog will find the “Deposit” value, and associate with the label attribute from the <caption> (since the <groupbox> won’t have a label), and present a dialog with a checkbox for the “Deposit” value, but with a label matching the caption label (which in this case, also happens to be “Deposit”).

If the user selects this checkbox and Saves, then generate_css could append a CSS classname to the document.documentElement called “Hide_Deposit”.  Assuming we can’t figure out a good way to do dynamic inline CSS, then we can rely on the server/skin/cat.css file containing a line like this:

.Hide_Deposit [hideable^=Deposit] { display: none; }

This will hide the entire groupbox for Deposit if all the conditions are met.  If we’re able to do live injection of CSS, we could simply inject something like:

[hideable^=Deposit] { display: none; }

We also need to ensure that the generate_css function will clear out any previous Hide_ classnames (or injected CSS) that were made prior to the latest invocation of generate_dialog.  For classnames, we can do this by looping through and pruning the classnames on document.documentElement.

Given this infrastructure (which we could re-use in the future with other interfaces), we can then do the following for the Item Attribute Editor:

• Create an org unit setting for ui.hide_copy_editor_fields with a datatype of string

• Modify the g.render function in copy_editor.js such that we apply hideable attributes to the dynamically generated groupbox and caption elements, using the same label we feed to the caption as the hideable value.  Very technical note: this is using an existing bad design decision in the item editor, where we’re using localized values as identifiers within the editor itself.  This means our setting (like item templates) will be locale dependent, and if a user switches locale (say, from en-US to en-CA), the fields will be identified with different names, and any settings we’ve configured will not work unless the labels happen to coincidentally match.  If we have time, we can refactor the Item Editor to use hard-coded id’s that are independent of locale (we’d likely use the actual property names we’re using to key our localizations).  Our org unit setting will then also use these id’s for designating which fields to hide.  A disadvantage here is that the org setting becomes difficult to use outside of generate_dialog, if staff try to configure it through Admin -> Local Administration -> Library Settings.  It may be worth adding functionality to hide org settings from that interface if they have their own dedicated interfaces.  Backwards compatibility with existing item attribute templates would also be a complicating factor with fixing the design.
• Modify the my_init function in copy_editor.js to invoke generate_css with the ui.hjide_copy_editor_fields setting prior to any fields being rendered.  This will minimize screen flicker.
• Modify copy_editor_overlay.xul and copy_editor.js to add a “Column Picker” button to the interface.  Clicking this button will invoke the generate_dialog function with the ui.hide_copy_editor_fields setting.
• Assuming we use the document.documentElement/classname strategy, modify cat.css and add entries for every hideable field like the .Hide_Deposit example given earlier.

Acq-02:  Record matching and overlay in Acquisitions (no assigned priority)

The Evergreen acquisitions infrastructure provides no automated method for matching incoming acquisitions records to existing catalog records.  Unless done manually, records in the ACQ system will not be “linked” to existing catalog records.   During the order phase, where records are typically loaded into the system, the system only allows record import of un-linked records by creating new catalog record, potentially resulting in (multiple) duplicate catalog records or import failures due to non-override-able import collisions.

A preferred work flow is one that allows Evergreen to automatically match ACQ records to existing catalog records with flexible control over which MARC fields to consider for matching and field weighting.  These are problems Evergreen’s Vandelay batch MARC import/export system was designed to solve.  The goal, then, is to leverage the full power of Vandelay seamlessly within acquisitions to support automated record matching and record merge/overlay.

ACQ User Interface Changes

• Interfaces that support “loading bibs and items” (i.e. creating catalog records) will now give the user the ability to select additional Vandelay import options.  These interfaces include the general lineitem list page (selection lists, purchase orders, including implicit loading via PO activation) and the vendor order record import interface.  New Vandelay options will include:
  • Record Match Set
  • Merge Profile
  • Import Non-Matching Records
  • Merge On Exact Match (901c)
  • Merge On Single Match
  • Merge On Best Match
  • Best/Single Match Minimum Quality Ratio
  • Insufficient Quality Fall-Through Profile
  • Recaculate Best Match option (for lineitems that have already been manually linked)
• The generic Lineitem list page page will provide links to view the Vandelay queue responsible for importing each lineitem, allowing staff to resolve merge/overlay conflicts directly from within the Vandelay interface.
• Within the Vandelay queue grid interface, it will be clearly marked when a user is viewing an ACQ queue vs a regular bib or authority import queue.

ACQ/Vandelay API changes

• During record import, when any Vandelay options are chosen, all incoming records will be passed off to Vandelay for queue creation and record match detection.
• For each queued bib record created within Vandelay, a link is made from the ACQ lineitem to the queued bib record resonsible for its import.
• Within Vandelay, any time a record from an ACQ queue is successfully imported, it will update the ACQ lineitem to point to the matched record (acq.lineitem.eg_bib_id).  Note, we want Vandelay to be responsible for updating the lineitem to support the case where ACQ Vandelay records are imported directly from within Vandelay.
• In Vandelay, if a lineitem is already linked to a bib record, use the linked bib record as the merge/overlay target instead of searching for the best match, unless the user selects the “recalculate best match” option during import.

DB

• Create a new vandelay.queue.queue_type option for “ACQ” to differentiate between regular bib imports and ACQ bib imports.
• acq.lineitem gets a new column which points to the vandelay.queued_bib_record responsible for its import

Pac-04:  Flexible scopes (no assigned priority)

Currently in Evergreen, one can limit search to one or more shelving locations via the advanced search interface if one is searching at an org type where can_have_vols is true.

The desired, new functionality would present a pre-defined group of shelving locations as a single entity in the library hierarchy, under the owning library.

Overview
To facilitate this functionality, Evergreen will gain the ability to track and use shelving location groups.  Each group would have a owner-distinct, translatable label and a visibility flag denoting whether it should be included in generated search range selection interfaces available within the OPAC, and a primary ordering column for sorting within a set of owned groups.  Sort will fall back on the display label of the groups.

QueryParser will gain a new filter, called location_groups(), which will accept a list of identifiers for these shelving location groups.  This will be used to gather the distinct set of shelving locations provided to the search stored procedure.

The TPAC will append the sorted groups owned by an org unit indented under it in a tree or dropdown, as appropriate, after any child org units.

The availability of copy locations when building a location group will be based on the context OU full path.

DB Sketch
CREATE TABLE asset.copy_location_group (
id    SERIAL     PRIMARY KEY,
name    TEXT        NOT NULL, — i18n
owner    INT        NOT NULL REFERENCES actor.org_unit (id),
pos    INT        NOT NULL DEFAULT 0,
opac_visible    BOOL    NOT NULL DEFAULT TRUE,
CONSTRAINT lgroup_once_per_owner UNIQUE (owner,name)
);

CREATE TABLE asset.copy_location_group_map (
id         SERIAL    PRIMARY KEY,
location    INT        NOT NULL REFERENCES asset.copy_location (id),
lgroup        INT        NOT NULL REFERENCES asset.copy_location_group (id),
CONSTRAINT    lgroup_once_per_group UNIQUE (lgroup,location)
);

TPAC: Template code and mod_perl handler code

opac/parts/org_selector.tt2 implements the selector widget for org units as it’s used today, and will be extended to take advantage of the model described above.

Since that widget is used in contexts other than search, it will need to retain a “mode” where it works as it does now (showing only org units, and not copy location groups), but also grow a new mode where it shows both org units and the copy location groups that belong to them.

In the new “mode” the widget should use a distinct CGI parameter (rather than “loc,” used throughout TPAC to specify an org unit exactly) which we’ll call “loc_or_lgroup” here until an appropriate name is determined.

The value of “loc_or_lgroup” should be either an integer or two integers delimited by a colon (or some other fixed punctuation).  The first integer refers to the org unit. The second, if present, refers to a copy location group.  Although knowing the id of the latter technically is enough information to find out the owning org_unit, it should be less disruptive to existing TPAC code if we still have the org unit id readily available.

Pac-05:  Greater flexibility in library selector display (no assigned priority)

Problem 1:
The library selector for TPac (Open-ILS/src/templates/opac/parts/org_selector.tt2) currently evaluates the OPAC-visibility of each org unit for which it provides an <option> tag before doing so, and a negative evaluation not only means that there will be no option for that org unit, but also that the org unit’s descendants will not even be considered.  This is the traditional behavior apparently desired by users in general.

MassLNC’s new requirement is that they be able to specify OPAC-visibility with more granularity, with the option of hiding an org unit but not its descendants.

* In existing org unit selectors today, and I assume under the new behavior that’s desired here, if an interface is wrapped in the staff client we show all org units, regardless of OPAC-visibility.  For the purpose of this document, I assume that won’t change, but this point should be clarified with MassLNC.  MassLNC confirms keeping the current meaning is desired.

Solution 1:
Create a new global flag (config.global_flag), named “opac.org_unit.non_inheritied_visibility”.

Consult this flag at the beginning of the build_org_selector_options block in the aforementioned tt2 file, using the generated context method for accessing cgf objects (the class is field_safe in the IDL, and thus helper methods are available automatically).

If the flag is enabled, do not recurse to display org unit children in the existing for loop, but rather in a new for loop outside the current IF block that tests visibility.  If the flag is not enabled, use the existing for loop, but not the new one.

Once this global flag is enabled, libraries will be able to pick and choose which specific systems will be visible.

Problem 2:
Evergreen will need the ability to accept user specifications for org unit sort order (among nodes with the same parent).

Solution 2:
Add an integer column to the actor.org_unit called “sibling_order”, not nullable with a default value of 0.

Add logic in the build_org_selector_options block to sort any list of org unit children by the value of this new sibling_order column, followed by the displayed label (name or shortname) before making iterative recursive calls over them.

Add a minimal interface, requiring the UPDATE_ORG_UNIT permission to be effective, depicting an org tree with a dijit.Tree.

Make nodes in the tree draggable only among their siblings.  A save operation will reset the sibling_order column on all org units nodes at which the UPDATE_ORG_UNIT permission is held to values determined by evaluating the final order of sibling nodes in branches of the dijit.Tree.

This interface may need to offer warnings to the user about the need to run autogen.sh or to restart services after making such changes, if that will be necessary (testing will tell).

Sys-07:  Last date of patron authentication (no assigned priority)

Note:  Some aspects of this design have been superseded.

Evergreen currently does not track authentication requests, successful authentication, or otherwise record persistent data regarding use actions.  This makes identifying inactive users difficult, and troubleshooting problems requires log scraping.

To remedy this, it is proposed that certain actions cause addition of a timestamped marker to a central logging table for the purpose of identifying activity, particularly authentication and authorization.  Each trackable action will be associated with an action group, one of authen, authz, circ, hold or search, to start.  This list of action groups can be extended in future versions.  Each action can be individually enabled or disabled, and a trigger on the logging table will arbitrate inclusion or exclusion.

For added privacy, it will also be possible to configure actions to only retain the most recent event for a given action.

The specific use-case is to track the last time at which each patron successfully authenticates in the system, with support for tracking this value during OPAC login, SIP login, and remoteauth.cgi login.  This design extends the use-case coverage to nearly any action that can be performed by a user.

DB Components

1. Add a new config.usr_activity_type table for configuring types of actions with seed data.  This table will track the action type, whether it’s enabled, and whether its events should be transient.
2. Add a new actor.usr_activity table for tracking actions performed by users.  This will track the user, action performed, and timestamp.

API / Middle Layer

1. Add a new default auth type to opensrf.xml for remoteauth.cgi
2. Augment oils_auth to insert rows in actor.usr_activity based on the login type
3. Augment SIP to insert rows into actor.usr_activity for each patron authorization-check based “login” (i.e. call to check_passwd)
4. Add virtual patron fields for last_activity and flesh the values by reading from actor.usr_activity during fleshed patron retrieval.  Flesh values in  open-ils.actor.user.fleshed.retrieve[.by_barcode] for display in the staff client.
5. Add support for open-ils.auth.authenticate.verify to support checking username/barcode without logging in (faster authz).
6. Insert activity event rows as necessary, starting with authentication-based events.

UI

1. Display last_auth_time and last_auth_type in patron summary display.
2. AutoGrid-based UI for managing event types, particularly disabling or enabling specific event logging.

User Activity Additions:  Last Date of Patron Authentication Part II*

(*This development is being donated by Equinox Software, Inc. to the Evergreen OS ILS Community.)

The goal for these additions to the user activity development is to collect additional data about each event.  Instead of simply tracking a generic event type, the code will now capture and track the “who” (e.g. vendor, UI agent), “how” (sip2, gateway, remoteauth, etc.), and the “what” (action) for each event.

The “who” value will be an optional parameter for authentication requests, set by the caller.  For interfaces within Evergreen that provide authentication (e.g. opac, staff client), the “who” value will simply indicate the interface in question.  For all other 3rd-party client applications that leverage Evergreen for authentication (e.g. remoteauth, xmlrpc, gateways) the client should coordinate with the Evergreen system on a name to use as the “who” value for the 3rd-party and add that value as the “who” parameter to all authentication requests.

For capturing the “how” value, we’re proposing an addition to OpenSRF, called ingress tracking.  This addition adds support for an OpenSRF message-level variable called “ingress” which is a free-text tag used to indicate the entry point for the client into the OpenSRF network.  Examples include gateway-v1, translator-v1, srfsh.  Additional examples specific to Evergreen include sip2, xmlrpc, remoteauth, etc.

For maximum control, each user activity type will support a combination of any three of these values.  Only 1 value will be required for each type.  When events are logged, the most specific applicable event type will be used.

Initial suggestion for tracking who (vendor) and how (via) courtesy of Thomas Berezansky:
https://docs.google.com/document/d/1WBqsx7OlCPvzqGvqsNLQysiYag64yRke9tiQ08X8xFc/edit

Circ-06 (depends on ESI Widget-01, SC-A1 & SC-A4):  Configurable holds pull list (no assigned priority)

Requirements:

1. Staff should have the ability to identify fields that display in the holds pull list.
2. Staff should also have the ability to identify the order in which these fields display.
3. These settings can either be an organizational unit setting or a workstation setting.
4. Staff should have the ability to filter the holds pull list by one or more copy locations at the time they are generating the list.
5. This functionality should be available for a pull list that sorts first by copy location and then by call number. The copy location order should be determined by the Copy Location Order interface in the Local Settings menu.

We’ll build a new interface for the holds pull list based on ESI Widget 01 (the grid), with a broad set of predefined fields for holds in the “map” as defined in ESI Service 02 (the flattener).  Make sure the widget described in ESI Widget 01 (the grid) either generally offers the ability to hide unwanted fields, or make sure we can create a special purpose subclass of it that will answer for this interface.  Make sure that columns are re-arrangeable.

ESI Widget 01 (the grid) as designed should satisfy req 1, 2, 3 almost automatically.

We address req 4 by connecting PCrudFilterDialog, or something like it.

Req 5 means we may need to extend ESI Service 02 (the flattener) and/or ESI Widget 01 (the grid) to allow us to customize how we sort per field.  I.e. by default, to sort by a field means a straightforward “order_by”: {“hint”: [“field”]} kind of clause, but we’ll want a mechanism through which we can optionally order by more complex expressions, so that when we sort (at the interface level) by copy location, we actually sort by acqplo.position and and all that.

This means we must make Evergreen work with current releases of Dojo now.

For printing, we need a simple button or link that will take the current parameters used to build the URL to feed the store to populate the DataGrid, optionally taking out any paging parameters (so as to print an entire pull list).  We then use those parameters to make a streaming API call to the middle layer, rendering a simple HTML-based table (presuming that Dijits like the DataGrid are no good for printing) in an iframe for printing.  We won’t use A/T for printing like some existing interfaces do since the form of the data (which columns, in what order, etc) and what to label each column is a dynamic matter, not one to be hardcoded in a template.

END OF DOCUMENT