Revert "quicksheet experiment, do not omit tabs based on number of unique values"

This reverts commit 90c43c93714655ffad0bce2b34934b4c0e367f9e.

Merge branch 'master' of git.esilibrary.com:migration-tools

adding some basic copy tag and serial tables to the mig init function

get_marc_leader()

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

function for setting passwords in actor.passwd quickly

better regexp for grabbing field names, redux

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

better regexp for grabbing field names

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

quicksheet experiment, do not omit tabs based on number of unique values

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

update unicorn holdings example

example cleanup

example holdings map for Unicorn/Symphony

overzealous heuristic

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

toward better name parsing (handles names like "Van Horn, Fiona")

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

toward better name parsing (handles names like "Van Horn, Fiona")

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

comment

push bib sequence and return starting value for reserved range

Merge branch 'master' of git.esilibrary.com:migration-tools

fix mig-init

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

adding actor.usr_setting to production tables

migration_tools.attempt_sierra_timestamp

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

more address parsing helpers

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

psql variable for :migschema

mig-sql already puts the migration schema in the search path, but
this can be useful for things like

    creator = (SELECT id FROM actor_usr WHERE usrname = :'migschema' || '_admin')

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

multiple file support for mig-quicksheet

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

mig-init: loop through sorted mig-sql/ directory

mig-init will look for .sql files in mig-sql to run

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

mig-sql: silence warning if no $BIBSTART

mig-env: set PGOPTIONS for search_path

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

extract_holdings: --suffix support

helps when you need to extract tag data from multiple MARC sources in a given migration

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

adding deletion of extra permissions and groups for users in remove ou scripts

adding basic program to convert from Apollo LDF to staging tables

adding deletion of surveys to remove ou scripts

fix -fs, and more output when fixing stuff

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

some marc mappings for III Sierra

mig-sql: pass through the BIBSTART env variable

available as :bibstart

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

mig-sql

Mig wrapper for psql.  Plan is to start injecting some useful mig-tracked
variables into psql sessions thus invoked.

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

some stuff I had lying around

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

ASSERT not supported until PG 9.5

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

some useful wrappers for loading bre's

"mig" tool

Wraps some new and existing tools in a git-like command system,
currently great for quickly getting files into staging tables,
has aspirations for one day tracking mapping relationships and
doing code generation for mapping.

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

third variant of migration_tools.assert with success message as 3rd argument

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

--force-utf8 in leader for marc_cleanup

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

assert tool for use with \set ON_ERROR_STOP on

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

use UNLOGGED tables for the base staging tables

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

add note to readme

add vacuum analyze script for database maintenance in remove_ou scripts

add a script to aid in generating the delete file for patrons in remove_ou scripts

add disable/renable rule so that copy locations can be deleted in remove_ou scripts

add removing usr circ history to remove_ou scripts

add removing call number suffixes and prefixes to remove_ou scripts

removed unncessary schema declared on column and fixed issue where value needed was ID and org shortname was given

removed moving call numbers where items and call numbers were on incompatible orgs to be deleted

added check to make sure holds were fulfilled when deleting by fulfillment library

removed nulling of parent circs where child circ is at non-deleted library

Remove stat cats owned by to be deleted org unit, stat cat entries and entry user mappings by those stat cats or patrons of those org units.

move call numbers from to be deleted org to undeleted org when the copy is at an undeleted org

add deleting holds by request lib

remove auditor history edits

remove holds based on fulfillment library

update the noncat circ code to note delete non cat circs when the staff member is registered at the wrong library

deletions of non cat circs

remove albermale references by putting in a variable and deleting usr messages

added line to handle child checkouts where parent checkout is at the org being deleted but child is not by nulling parent circ, unfortunately no way to indicate this

OU data removal script

This is a set of SQL scripts to be run, in order, to completely
remove data belonging to an OU and its descendants in an Evergreen
database.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

filter out linefeeds and carriage returns from holding tag values

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

filter out tabs from holding tag values

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

minor whitespace correction

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

Make --wait optional in eg_staged_bib_overlay.

This commit only causes load bibs to wait if the --wait option is
specified with a non-zero value.  It also changes the default for the
global $wait variable that the option is read into to zero.

Signed-off-by: Jason Stephenson <jason@sigio.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>

Perl eval improvements in eg_staged_bib_overlay.

Two changes to improve the eval conditions added in the previous
commit:

1. Both error handlers now warn with the same message:

   Record $i is bad: $@; skipping.

Thereby reporting the original message in case the user wants to try
and recover the records later.

2. If a bib record's 901$c is either missing or not numeric, die() is
called with a message to that effect.  This eliminates the Perl
warnings about exiting an eval block with next.

Signed-off-by: Jason Stephenson <jason@sigio.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>

Use eval in eg_staged_bib_overlay.

Wrap the parsing of MARC from input files in eval blocks with an error
handler to skip to the next record.  I have encounterd bad records
that have stopped the processing of the rest of the file.  This
occurred the very first time I used this script.

Signed-off-by: Jason Stephenson <jason@sigio.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>

convert DestinyCustomReport XML to TSV

add --fix-subfield

and tweak some comments and command-line help

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

example Follet/Destiny 852 map

syntax in INHERIT example

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

eg_staged_bib_overlay: add --link-skipped

The new --link-skipped switch modifies the link_auth_bibs
action to say that bibs that were skipped due to having been
edited after the cutoff should be linked to authorities (rather
than linking imported bibs).

The use case for this: the library decides that they don't
want to scribble over the skipped bibs, but still want to
take advantage of any new authorities that correspond to them.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

eg_staged_bib_overlay: add actions to export skipped records

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

eg_staged_bib_overlay: fix help

Corrected the description of the --overlay_auths_stage1
and --overlay_auths_stage2 switches.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

eg_staged_bib_overlay: add link_auth_bib action

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

staged record loader: add various authority record actions

--action stage_auths          - load MARC authorities into staging
                                table
--action match_auths          - identify matches with authority
                                records already present in the
                                Evergreen database; matching is
                                based on LCCN, cancelled LCCN, and
                                main heading.
--action load_new_auths       - load new (unmatched) authorities
--action overlay_auths_stage1 - overlay based on LCCN where
                                heading has not change; this step
                                disables propagation to bib records
--action overlay_auths_stage2 - overlay based on LCCN where heading
                                has NOT changed; propagates changes
                                to bib records
--action overlay_auths_stage3 - overlay for records where a cancelled
                                LCCN is replaced with a new one
--action link_auth_auth       - run authority_authority_linker.pl for
                                the authorities that were overlaid
                                or added in this batch.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

tweaks to name parser

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

staged bib overlay: tweak bib filtering

Filtering now always ignores staged bibs that are
imported or already marked as not to be imported.

Also, bibs whose XML is not well-formed are now
excluded.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

update usage information

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

new utility to overlay bib records in an Evergreen database

Utility to stage and overlay bib records in an Evergreen database. This
expects that the incoming records will have been previously exported
from that Evergreen database and modified in some fashion (e.g., for
authority record processing) and that the bib ID can be found in the
901$c subfield.

This program has several modes controlled by the --action switch:

  --action stage_bibs  - load MARC bib records into a staging table
  --action filter_bibs - mark previously staged bibs that should
                         be excluded from a subsequent load, either
                         because the target bib is deleted in Evergreen
                         or the record was modified after a date
                         specified by the --cutoff switch
  --action load_bibs   - overlay bib records using a previously staged
                         batch, one at a time. After each bib, it will
                         wait the number of seconds specified by the
                         --wait switch.

Several switches are used regardless of the specified action:

  --schema  - Pg schema in which staging table will live; should be
              created beforehand
  --batch   - name of bib batch; will also be used as the name
              of the staging table
  --db      - database name
  --dbuser  - database user
  --dbpw    - database password
  --dbhost  - database host

Examples:

./eg_staged_bib_overlay --schema bib_load --batch bibs_2016_01 --db evergreen \
   --dbuser evergreen --dbpw evergreen --dbhost localhost \
   --action stage_bibs -- file1.mrc file2.mrc [...]

./eg_staged_bib_overlay --schema bib_load --batch bibs_2016_01 --db evergreen \
   --dbuser evergreen --dbpw evergreen --dbhost localhost \
   --action filter_bibs --cutoff 2016-01-02

./eg_staged_bib_overlay --schema bib_load --batch bibs_2016_01 --db evergreen \
   --dbuser evergreen --dbpw evergreen --dbhost localhost \
   --action load_bibs

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

teach export_evergreen_library how to emit serials data

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

acq.fund_tag_map

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

add acq.provider_contact and acq.provider_contact_address to base staging table

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

add action.transit_copy and action.hold_transit_copy to base tables

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

add action.hold_request_note to the list of base tables

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

migration_tools.attempt_hz_date

tweak for case when just last name

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

migration_tools.name_parse_out_last_first_middle_and_random_affix for LAST FIRST MIDDLE with no comma

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

migration_tools.find_hold_matrix_matchpoint2

another convenience function

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

remove config fig.circ_matrix_circ_mod_test

and _map from the list of base tables so that .build() will work with newer
versions of EG

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

more money converters

migration_tools.attempt_money_from_pennies
and migration_tools.attempt_money_from_pennies6

Avoids dividing by 100 when doing the conversion

2nd argument, the fail value, should not be pennies

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

duplicate_template_but_change_delay

Let's us get around duplicate key issues in some cases by tweaking the delay as
part of the cloning process.

Example:

    ubuntudeskfx=# SELECT * FROM migration_tools.duplicate_template(5,'{3,4}','00:30:00'::INTERVAL);

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

Minor typo fix in export_evergreen_library

Signed-off-by: Mike Rylander <mrylander@gmail.com>

tweak migration_tools.insert_856_9

rename original to migration_tools.insert_856_9_conditional, since it has
behavior based on 856$u content.  New version simply adds a $9 if one does
not already exist (regardless of org unit referenced)

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

handle dates like 051980

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

add some in-db matchpoint tables, for potential ease of staging policy rules

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

convenience wrappers for finding in-db rules

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

make migration_tools.apply_circ_matrix_to_specific_circ use the more recent apply_circ_matrix code

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

add a delimiter paramater, and a variant that returns an array of tags

Signed-off-by: Jason Etheridge <jason@esilibrary.com>