Release note fixes

[evergreen-equinox.git] / docs / RELEASE_NOTES_3_11.adoc

= Evergreen 3_11 Release Notes =
:toc:
:numbered:
:toclevels: 3

== Upgrade notes ==

== New Features ==

:leveloffset: +2


= Acquisitions =


== Lineitem Detail Audit Table ==

The default schema has added an audit table for the
`acq.lineitem_detail` table.  The audit table is NOT created during
database upgrade.  If you wish to add the audit table to your
Evergreen installation, you can run the following SQL in your
database:

[source, sql]
----
SELECT acq.create_acq_auditor ( 'acq', 'lineitem_detail' );
CREATE INDEX acq_lineitem_detail_hist_id_idx ON acq.acq_lineitem_detail_history( id );
CREATE INDEX acq_lineitem_detail_hist_lineitem_idx ON acq.acq_lineitem_detail_history( lineitem );
CREATE INDEX acq_lineitem_detail_hist_fund_debit_idx ON acq.acq_lineitem_detail_history( fund_debit );
----



= Administration =


== Enhanced Concerto dataset ==

An alternative sample dataset called "Enhanced Concerto" is now
available. This dataset includes a more realistic organizational
unit structure and additional data with which to test Evergreen
functionality.

The "Enhanced Concerto" dataset supports human-driven testing
and community demonstrations. However, at present, automated
tests, particularly the ones found in `Open-ILS/src/sql/Pg/live_t`
and `Open-ILS/src/perlmods/live_t`, are not guaranteed to pass
with the "Enhanced Concerto" dataset; instead, they remain targeted
to working with the original "Concerto" dataset.

=== Installing the dataset ===

The dataset can be loaded when installing Evergreen by using the
`--load-concerto-enhanced` option of `eg_db_config`. For example:

[source,bash]
----
perl Open-ILS/src/support-scripts/eg_db_config --update-config \
       --service all --create-database --create-schema --create-offline \
       --user <user> --password <password> --hostname <hostname> --port <port> \
       --database <dbname> --admin-user <admin-user> --admin-pass <admin-pass> \
       --load-concerto-enhanced
----

The `--load-concerto-enhanced` switch is mutually exclusive with the
`--load-all-sample` switch to load the full original "Concerto" dataset
and the `--load-concerto` switch to load just the bibs, items, and
authorities from the original "Concerto" dataset.

=== Updating the dataset ===

An experimental script called `make_concerto_from_evergreen_db.pl` is
included to support

* updating the dataset as Evergreen's schema evolves
* incorporating changes to the dataset from a running Evergreen system

The script compares data between different versions of an
Evergreen database depending on how it is invoked. This script has known
bugs and its output *must* be manually reviewed before being committed.


== --pipe option for marc_export ==

Add a --pipe option to marc_export to force reading of record ids from
standard input when one or more of the --library, or
--descendants options are used.  This permits more flexibility when
exporting records with holdings for given libraries or subsets of a
collection.

It is an error to specify the --all or --since  and --pipe options in the same
command line.


[source,sh]
--------
        --pipe

Some examples:

        --pipe --library BR1   [Filter the list of bib ids by those 
                                with BR1 as the Owning Library]

        --pipe --descendants BR1 [Filters the list of bib ids by those 
                                  with BR1 or descendants as the Owning Library]

--------


== --strip option for marc_export ==

The --strip option is used to suppress generation of specified elements
from the marc_export output. The option, which can be specified more than
once, is in one if these forms:

[source,sh]
--------
        --strip <field RE>/<subfield RE>
        --strip /<subfield RE>
        --strip <field RE>

Some examples:

        --strip 856/0   [Delete subfield 0's in fields with tag 856.]

Regular expressions are accepted:

        --strip 8../0   [Delete subfield 0's in fields with tag 800-899.]

If the field is omitted, it is as if you specified "..." for the field RE.

    --strip /0      [Delete subfield 0's in all fields.]
    --strip /[abc]  [Delete subfield a, b or c in all fields.]

If the slash and subfield are omitted, it means to delete the given fields.

    -strip 856     [Delete fields with tag 856]

If the slash is present, but the subfield is omitted, it means "all subfields"

    --strip 856/    [Delete all subfields with tag 856]
--------


== Applying Missing Database Upgrades ==

This patch fixes a situation where an Evergreen database that had
been been upgraded to 3.6.0 at some point in its past using the
3.5.1-3.6.0 DB update script may be missing some DB revisions.

The following bugfixes are affected:

    - https://bugs.launchpad.net/evergreen/+bug/1788260
    - https://bugs.launchpad.net/evergreen/+bug/1908727
    - https://bugs.launchpad.net/evergreen/+bug/1835127
    - https://bugs.launchpad.net/evergreen/+bug/1910891
    - https://bugs.launchpad.net/evergreen/+bug/1882825
    - https://bugs.launchpad.net/evergreen/+bug/1096209


== Improvements to Mapping Permissions to Groups ==

The dialog to add permission mappings to a group in the Permission
Groups administrative interface now allows multiple permissions
to be added in one invocation. This reduces the number of clicks
and keyboard interactions required when adding a large number of
permission mappings.



= Architecture =


== Malicious Search Protection ==

Evergreen sometimes sees some "novel" query strings in the wild that
cause the search backend to time out or worse.  These are sometimes
malicious and sometimes accidental, but the effect on users is the
same.

The changes here improve query compilation in several respects in order
to reduce the chances of an overly complex query causing problems for
the search subsystem.

More work is done up front to simplify and combine parts of the
resulting SQL, allowing more work to be done closer to the data.
This change allows Evergreen to handle many more tested or chained
boolean expressions, and negated terms are now handled directly in
line with other adjacent terms. Phrases (exact matches) are now
searched for using Postgres' adjacency tsearch operator.

All of these changes work together to improve performance by getting
more search work done in fewer database operations while protecting
against certain query constructs that have caused problems in the
past.



== Angular and Bootstrap Updates for Staff Interface ==

The Angular staff interface now uses Angular 15 and Bootstrap 5. This
introduces various changes to how the staff interface styling is
done; developers should take note.


== Fix SQL Injection Vulnernability ==

An SQL injection vulnernability related to the implementation of
search term highlights is now closed.


== Queued Ingest ==

This feature allows for the separation of bib and authority record
updates and the search (and other) indexing that occurs when a record is
modified in some way. Prior to this feature, bib and authority records
would be indexed immediately upon an update.

While individual record ingest has not become a problem with regard to
system performance or interface usability, there exist several batch
operations which aggregate many inserts or updates and whose aggregate
ingest time cost can be significant.  These include, but are not
limited to, reingest caused by authority control propagation, reingest
required by the addition or modification of indexing configuration,
cataloging and acquisitions record import and overlay from the staff
interface, and upgrade-time reingest required by structural changes
to the underlying indexing and search system.

=== New Utility ===

When Queued Ingest is enabled, a new control script, `ingest_ctl`, is
available to perform several functions:

* Run in the background to process the queues of indexing requests
* Display statistics of queued ingest activity
* Specify that a set of records should be reindexed.

Here are some examples of how it is used:

[source,bash]
------------------------------------------------------------------------------------
# Enqueue records 1-500000 for reingest later, just one worker for the queue
/openils/bin/ingest_ctl --queue-threads 1
    --queue-type biblio
    --queue-run-at tomorrow
    --queue-owner admin
    --queue-name "slowly updating records due to new RDA attributes"
    --start-id 1 --end-id 500000

# Start the background worker
/openils/bin/ingest_ctl --coordinator --max-child 20

# Stop the background worker
/openils/bin/ingest_ctl --coordinator --stop

# Process whatever you can Right Now
/openils/bin/ingest_ctl --max-child 20

# Process a single queue Right Now
/openils/bin/ingest_ctl --queue 1234 --max-child 20

# Stats on Queued Ingest processing so far today
/openils/bin/ingest_ctl --stats --since today --totals-only
------------------------------------------------------------------------------------

This script also requires the following switches (or environment
variables) in order to connect to the database:

* `--db_user` (or environment variable `PGUSER`)
* `--db` (or environment variable `PGDATABASE`)
* `--dbpw` (or environment variable `PGPASSWORD`)
* `--db_port` (or environment variable `PGPORT`)

=== New Settings ===

This feature adds several new global flags:


|===
| Global Flag | Enabled

| Queued Ingest: Abort transaction on ingest error rather than simply logging an error | no
| Queued Ingest: Queue all bib record updates on authority change propagation, even if bib queuing is not generally enabled | no
| Queued Ingest: Use Queued Ingest for bib record ingest on insert and undelete | no
| Queued Ingest: Use Queued Ingest for authority record ingest on insert and undelete | no
| Queued Ingest: Use Queued Ingest for bib record ingest on update | no
| Queued Ingest: Use Queued Ingest for authority record ingest on update | no
| Queued Ingest: Use Queued Ingest for bib record ingest on delete | no
| Queued Ingest: Use Queued Ingest for authority record ingest on delete | no
| Queued Ingest: Maximum number of database workers allowed for queued ingest processes | yes; default value 20
| Queued Ingest: Use Queued Ingest for all bib record ingest | no
| Queued Ingest: Use Queued Ingest for all bib and authority record ingest | no
| Queued Ingest: Do NOT use Queued Ingest when creating a new bib, or undeleting a bib, via the MARC editor | yes
| Queued Ingest: Use Queued Ingest for all authority record ingest | no
| Queued Ingest: Do NOT Use Queued Ingest when editing bib records via the MARC Editor | yes
|===

This feature does not add any new library settings or permissions.

=== Upgrade Notes ===

Queued Ingest is not automatically turned on upon upgrade. to enable it,
at minimum the following actions should be taken:

* Enable at least the "Use Queued Ingest for all bib and authority record
  ingest" global flag
* Ensure that `ingest_ctl` is running with the `--coordinator` flag.



= Cataloging =


== Catalog Staff View ==

Adds a Staff View tab to the record details page in the staff catalog.
In addition to showing various fields from the bib record, it also shows
a count of hold requests and a break-down of available items versus
total items, and it shows a breakdown of similar records by formats and
language editions, based on the associated metarecord.




= Circulation =


== Adjust Hold Cancel Reasons ==

The dialog to confirm canceling a hold in the staff interface
now lists only hold cancel reasons that can be manually applied,
excluding the ones that can only be automatically applied.

In addition, two more hold cancel reasons are added:

 * Patron via email
 * Patron via SMS


== New Permission for Viewing Holds Pull List ==

Access to the holds pull list is now restricted to users with the
VIEW_HOLD_PULL_LIST permission.  By default, this new permission is granted to
all users with the VIEW_HOLDS permission.




= Client =


== Angular navbar keyboard navigation ==

Adds basic keyboard navigation support to the Angular web client navbar as follows:

* Use the Tab key to move between the top-level menu items.
* Press Enter, Space, or the down arrow on a top-level item to open its submenu. Pressing Esc will close it.
* Shift-tab from the first submenu item back to the top level, and use Enter or Space to toggle the button again and close the submenu.
* Within the submenu, use either Tab and shift-Tab to move up and down, or the up/down arrow keys.



= OPAC =


== Multi-term Did You Mean search suggestions ==

Expanding on the previous single-class, single-term search suggestion
development, this feature provides suggestions for single-class searches
with multiple terms.

 * The Library Settings that were previously used to control the global
behavior of search suggestions have been moved to search class
configuration fields.  This was done because the data in each search
class benefits from different setting values. If the Library Settings
had been set, they will be used to update the corresponding search
class configuration fields. Regardless, Evergreen administrators are
recommended to review the settings.

 * If a patron's search matches a variant or non-preferred heading from
an authority record, if the main heading of that authority is linked
to at least bibliographic record, the system will provide that main
heading as a suggestion as well, along with spelling-corrected suggestions.

 * Quoted phrases in user input require strict term order and adjacency
for the phrase portion of the suggestion generated for the phrase(s),
whereas unquoted input (or the portion that is not quoted) does not.

=== MARC Search/Facet Class field additions ===

 * variant_authority_suggestion   Whether this class should attempt variant authority suggestions based on search-class/browse-axis mapping
 * symspell_transfer_case         Whether suggestions should retain user-supplied letter case
 * symspell_skip_correct          Only supply suggestions to misspelled words
 * symspell_suggestion_verbosity  Setting that controls the amount of effort, and therefore time, spent on suggestion generation
 * max_phrase_edit_distance       Maximum average per-word edit distance when evaluating suggestions
 * suggestion_word_option_count   Maximum alternate suggestions per word
 * max_suggestions                Maximum suggstions to present
 * low_result_threshold           Maximum hit count beyond which suggestions are not provided
 * min_suggestion_use_threshold   Minimum number of times a suggestion must exist in the corpus
 * pg_trgm_weight                 Weight of the trigram similarity metric; 0 avoids calculation costs
 * soundex_weight                 Weight of the soundex similarity metric; 0 avoids calculation costs
 * keyboard_distance_weight       Weight of the keyboard distance similarity metric; 0 avoids calculation costs

=== Upgrade ===

If the databse has authority records that are linked to bilbiographic
records, a reingest of the search suggestion dictionary is recommended.

Instructions for performing that reingest are included in the database
update scripts and will be output to the log when those scripts are
run.


== Restrict login redirect ==

As a security best-practice, Evergreen should not allow arbitrary
redirection on successful login, but instead limit redirection to
local links or configured domains and schemes.

This feature is controlled by a new global flag called *opac.login_redirect_domains*
which must contain a comma-separated list of domains.  All hostnames
under each domain is allowed for redirect, and the scheme of the
redirect URL must be one of http, https, ftp, or ftps.



== Accepting payments with SmartPAY ==
SmartPAY is a payment processing service that lets
sites take credit card payments without payment card information ever
touching the sites' own servers.

=== Library Settings ===
The following settings need to be set at the appropriate org level for
sites wanting to use SmartPAY.

 * "Allow Credit Card Payments" (should be 'true')

   credit.payments.allow

 * "Enable SmartPAY payments" (should be 'true')

   credit.processor.smartpay.enabled

 * "SmartPAY location ID" (value provided by Comprise)

   credit.processor.smartpay.location_id

 * "SmartPAY customer ID" (value provided by Comprise)

   credit.processor.smartpay.customer_id

 * "SmartPAY login name" (value provided by Comprise)

   credit.processor.smartpay.login

 * "SmartPAY password" (value provided by Comprise)

   credit.processor.smartpay.password

 * "SmartPAY API key" (value provided by Comprise)

   credit.processor.smartpay.api_key

 * "SmartPAY server name" (value provided by Comprise)

   credit.processor.smartpay.server

 * "SmartPAY server port" (value provided by Comprise)

   credit.processor.smartpay.port

 * "Name default credit processor" (should be 'SmartPAY')

   credit.processor.default



= Miscellaneous =

* Add patron home library code as a column to the View Holds grid in the staff catalog record details page (LP#1991726)
* Include template ID in the template table in the Reporter (LP#1998386)
* Remove the `pub` flag from the `biblio.record_note` table (LP#1978978)
* Add the publication date to the Staff Catalog's Shelf Browse (LP#1999432)
* Resolve search performance degradation with PostgreSQL version 12 and up (LP#1999274)
* Improved styling of paid line items in acquisitions screens (LP#1999270)
* Improved styling of the keyboard shortcut info modal (LP#1999955)
* (Developer) Add Emacs mode to `fm_IDL.xml` (LP#1914625)
* `autogen.sh` can now accept a `-c` switch to specify the location of `opensrf_core.xml`. This is useful for certain multi-tenant setups of Evergreen. (LP#2003707)
* Better organization of acquisitions line item alert fields (LP#2002977)
* Prevent templates from applying or changing magical status in angular holdings editor (LP#1999401)
* Prevent directly editing the shelving location deleted field in the Shelving Locations Editor (LP#2002435)
* The "Strict Barcode" checkbox is now closer to the barcode input on the Check Out, Check In, and Renew Items pages (LP#1990968)
* LP1929593 UPDATE_COPY_BARCODE permission

    This adds the permission UPDATE_COPY_BARCODE and a new API call,

      open-ils.cat.update_copy_barcode

    which explicitly tests for both UPDATE_COPY_BARCODE and UPDATE_COPY,
    with either being sufficient for allowing a barcode change.  Existing
    Replace Barcode UI's in both Angular and AngularJS have been modified
    to use this API call instead of the pcrud service.  One side-effect of
    this has been better surfacing of errors, as errors in pcrud were
    uncaught and bypassing the normal error handling.  This addresses
    LP1951469.

    The upgrade script gives any permission groups that already have the
    UPDATE_COPY permission the new UPDATE_COPY_BARCODE permission at the
    same depth, though it's technically not needed.
* Patron and staff login forms now include a button to reveal the password input. (LP#1977554)
* Adds new Local Administration entries for Item Statistical Categories Editor and Patron Statistical Categories Editor, which are angularized interfaces.
* Tweaks eg-grids to underline hyperlinks within cells.  This potentially affects multiple interfaces.
* eg-org-family-select now supports persistKey
* LP1965446 Option to Disable Title-Level Holds on Bib Records with Parts

    This feature adds one global flag and one library setting, respectively:

        * circ.holds.api_require_monographic_part_when_present
          Holds: Require Monographic Part When Present for hold check.
        * circ.holds.ui_require_monographic_part_when_present
          Require Monographic Part when Present

    Normally the selection of a monographic part during hold placement is optional if there is at least one copy
    on the bib without a monographic part.  A true value for this setting for any involved owning library for the
    bib or for the global flag will require part selection even under this condition.  This essentially removes
    the All/Any Parts option from the part selection drop-down, for both versions of the public catalog (TPAC and
    BOOPAC), and for the Angular staff catalog interface. It should be noted that if the library setting is set
    below the consortium level, Title level holds may be allowed for some libraries and not others.

    At the API level, we consider just the global flag and will throw a TITLE_HOLD_WHEN_MONOGRAPHIC_PART_REQUIRED
    event for a title hold request when there are items with monographic parts on the bib.  It is possible for
    the library settings and the global flag to differ, but the global flag will catch every instance of hold
    placement including those by third party callers, SIP, etc.
* Links the lineitem id link in Acquisitions search results to the lineitem detail page rather than the purchase order. (LP#2003946)
== Angular navbar keyboard navigation ==

Adds basic keyboard navigation support to the Angular web client navbar as follows:

* Use the Tab key to move between the top-level menu items.
* Press Enter, Space, or the down arrow on a top-level item to open its submenu. Pressing Esc will close it.
* Shift-tab from the first submenu item back to the top level, and use Enter or Space to toggle the button again and close the submenu.
* Within the submenu, use either Tab and shift-Tab to move up and down, or the up/down arrow keys.

:leveloffset: 0


== Acknowledgments ==
The Evergreen project would like to acknowledge the following
organizations that commissioned developments in this release of
Evergreen:

* CW MARS
* Evergreen Community Development Initiative
* Evergreen Indiana
* King County Library System
* Pennsylvania Integrated Library System
* Westchester Library System

We would also like to thank the following individuals who contributed
code, translations, documentations patches and tests to this release of
Evergreen:

* John Amundson
* Scott Angel
* Jason Boyer
* Dan Briem
* Andrea Buntz Neiman
* Eva Cerninakova
* Galen Charlton
* Garry Collum
* Elizabeth Davis
* Jeff Davis
* Britta Dorsey
* Bill Erickson
* Jason Etheridge
* Ruth Frasur
* Jeff Godin
* Blake Graham-Henderson
* Rogan Hamby
* Elaine Hardy
* Stephanie Leary
* Clayton Liddell
* Shula Link
* Tiffany Little
* Mary Llewellyn
* Debbie Luchenbill
* Llewellyn Marshall
* Terran McCanna
* Chrystal Messam
* Gina Monti
* Christine Morgan
* Michele Morgan
* Susan Morrison
* Susasn Morrison
* Dan Pearl
* Jennifer Pringle
* Mike Risher
* Mike Rylander
* Jane Sandberg
* Chris Sharp
* Ben Shum
* Jason Stephenson
* Josh Stompro
* Elizabeth Thomsen
* Jennifer Weston
* Beth Willis
* Carol Witt
* Adam Woolford
* Jessica Woolford

We also thank the following organizations whose employees contributed
patches:

* BC Libraries Coop
* Bibliomation
* Catalyte
* CW MARS
* Equinox Open Library Initiative
* Evergreen Indiana
* Georgia Public Library Service
* Kenton County Library
* King County Library System
* Lake Agassiz Regional Library
* LibraryMarket
* Linn Benton Community College
* MOBIUS
* NC Cardinal
* NOBLE
* Princeton University
* Sigio
* Westchester Library System

We regret any omissions.  If a contributor has been inadvertently
missed, please open a bug at http://bugs.launchpad.net/evergreen/
with a correction.