migration-tools.git
4 years agomoving config into else clause for cleanliness
Rogan Hamby [Mon, 12 Feb 2018 19:17:33 +0000]
moving config into else clause for cleanliness

4 years agofixing typo and moving grab file into else clause
Rogan Hamby [Mon, 12 Feb 2018 19:16:37 +0000]
fixing typo and moving grab file into else clause

4 years agomade csv2sql mig aware
Rogan Hamby [Mon, 12 Feb 2018 16:50:21 +0000]
made csv2sql mig aware

4 years agoadding a report package for an overview of system data like that done in migrations
Rogan Hamby [Mon, 12 Feb 2018 16:02:31 +0000]
adding a report package for an overview of system data like that done in migrations

4 years agofix typo and add some security to output format
Rogan Hamby [Sun, 11 Feb 2018 23:07:46 +0000]
fix typo and add some security to output format

4 years agofixing doc
Rogan Hamby [Fri, 9 Feb 2018 22:04:16 +0000]
fixing doc

4 years agominor fixes to gsheet
Rogan Hamby [Fri, 9 Feb 2018 20:59:45 +0000]
minor fixes to gsheet

4 years agoadding the mig gsheet main file
Rogan Hamby [Fri, 9 Feb 2018 20:46:26 +0000]
adding the mig gsheet main file

4 years agoadding mig gsheet
Rogan Hamby [Fri, 9 Feb 2018 20:46:05 +0000]
adding mig gsheet

4 years agoclean up reporter share withs in remove ou scripts
Rogan Hamby [Fri, 9 Feb 2018 15:09:03 +0000]
clean up reporter share withs in remove ou scripts

4 years agostart of serial script in remove ou
Rogan Hamby [Fri, 9 Feb 2018 14:44:19 +0000]
start of serial script in remove ou

4 years agostart of distinct acq removal file
Rogan Hamby [Fri, 9 Feb 2018 13:44:59 +0000]
start of distinct acq removal file

4 years agoadmin user sanity check added to remove ou scripts
Rogan Hamby [Fri, 9 Feb 2018 13:23:52 +0000]
admin user sanity check added to remove ou scripts

4 years agomaking the user delete information a bit more robust
Rogan Hamby [Fri, 9 Feb 2018 13:18:14 +0000]
making the user delete information a bit more robust

4 years agomoving some cross saved templates and output folders
Rogan Hamby [Fri, 9 Feb 2018 13:14:37 +0000]
moving some cross saved templates and output folders

4 years agoadded location for added page file to be miggitdir
Rogan Hamby [Thu, 8 Feb 2018 19:51:49 +0000]
added location for added page file to be miggitdir

4 years agoadded reporter documentation to mig
Rogan Hamby [Thu, 8 Feb 2018 19:49:25 +0000]
added reporter documentation to mig

4 years agoinitial working branch for mig reporter
Rogan Hamby [Thu, 8 Feb 2018 19:39:22 +0000]
initial working branch for mig reporter

4 years agoadd removal of copy tag stuff from remove ou scripts
Rogan Hamby [Wed, 31 Jan 2018 13:47:22 +0000]
add removal of copy tag stuff from remove ou scripts

4 years agoadding last pushed timestamp to gsheet tracked table
Rogan Hamby [Tue, 30 Jan 2018 14:24:58 +0000]
adding last pushed timestamp to gsheet tracked table

4 years agoadd unique constraint for worksheet and tab name in ghseet tracked table
Rogan Hamby [Tue, 30 Jan 2018 14:23:34 +0000]
add unique constraint for worksheet and tab name in ghseet tracked table

4 years agoadding primary key to gsheet_tracked_table
Rogan Hamby [Tue, 30 Jan 2018 14:15:17 +0000]
adding primary key to gsheet_tracked_table

4 years agoMerge branch 'master' of git.esilibrary.com:migration-tools
Rogan Hamby [Mon, 29 Jan 2018 20:54:13 +0000]
Merge branch 'master' of git.esilibrary.com:migration-tools

4 years agogsheet blobs for mig init directory
Rogan Hamby [Mon, 29 Jan 2018 20:53:54 +0000]
gsheet blobs for mig init directory

4 years agosimplify arguments for mig-skip-iconv and mig-skip-clean
Jason Etheridge [Mon, 29 Jan 2018 20:51:29 +0000]
simplify arguments for mig-skip-iconv and mig-skip-clean

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agochanging sequence of loading bibs so that newest bibs are prioritized which helps...
Rogan Hamby [Fri, 19 Jan 2018 16:13:27 +0000]
changing sequence of loading bibs so that newest bibs are prioritized which helps the catalogers

4 years agoremove check for wait since it is defined by default now and add it to loading auths...
Rogan Hamby [Tue, 16 Jan 2018 22:08:15 +0000]
remove check for wait since it is defined by default now and add it to loading auths as well

4 years agosummary table and subtables as an output option for quicksheet
Jason Etheridge [Fri, 12 Jan 2018 21:39:00 +0000]
summary table and subtables as an output option for quicksheet

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agoinit/ subdirectory for mig-sql/, so we can more conveniently store sql for other...
Jason Etheridge [Fri, 12 Jan 2018 15:58:44 +0000]
init/ subdirectory for mig-sql/, so we can more conveniently store sql for other mig commands

4 years agoRevert "quicksheet experiment, do not omit tabs based on number of unique values"
Jason Etheridge [Fri, 5 Jan 2018 20:37:44 +0000]
Revert "quicksheet experiment, do not omit tabs based on number of unique values"

This reverts commit 90c43c93714655ffad0bce2b34934b4c0e367f9e.

4 years agoMerge branch 'master' of git.esilibrary.com:migration-tools
Rogan Hamby [Fri, 15 Dec 2017 18:34:48 +0000]
Merge branch 'master' of git.esilibrary.com:migration-tools

4 years agoadding some basic copy tag and serial tables to the mig init function
Rogan Hamby [Fri, 15 Dec 2017 18:34:26 +0000]
adding some basic copy tag and serial tables to the mig init function

4 years agoget_marc_leader()
Jason Etheridge [Wed, 6 Dec 2017 04:54:41 +0000]
get_marc_leader()

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agofunction for setting passwords in actor.passwd quickly
Rogan Hamby [Wed, 29 Nov 2017 14:52:49 +0000]
function for setting passwords in actor.passwd quickly

4 years agobetter regexp for grabbing field names, redux
Jason Etheridge [Mon, 27 Nov 2017 22:42:16 +0000]
better regexp for grabbing field names, redux

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agobetter regexp for grabbing field names
Jason Etheridge [Mon, 27 Nov 2017 22:35:57 +0000]
better regexp for grabbing field names

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agoquicksheet experiment, do not omit tabs based on number of unique values
Jason Etheridge [Mon, 27 Nov 2017 22:21:48 +0000]
quicksheet experiment, do not omit tabs based on number of unique values

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agoupdate unicorn holdings example
Jason Etheridge [Mon, 27 Nov 2017 20:41:04 +0000]
update unicorn holdings example

4 years agoexample cleanup
Jason Etheridge [Mon, 27 Nov 2017 20:19:30 +0000]
example cleanup

4 years agoexample holdings map for Unicorn/Symphony
Jason Etheridge [Mon, 27 Nov 2017 20:15:31 +0000]
example holdings map for Unicorn/Symphony

4 years agooverzealous heuristic
Jason Etheridge [Wed, 8 Nov 2017 02:06:55 +0000]
overzealous heuristic

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agotoward better name parsing (handles names like "Van Horn, Fiona")
Jason Etheridge [Wed, 11 Oct 2017 08:40:44 +0000]
toward better name parsing (handles names like "Van Horn, Fiona")

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agotoward better name parsing (handles names like "Van Horn, Fiona")
Jason Etheridge [Wed, 11 Oct 2017 07:55:36 +0000]
toward better name parsing (handles names like "Van Horn, Fiona")

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agocomment
Jason Etheridge [Tue, 3 Oct 2017 19:07:14 +0000]
comment

4 years agopush bib sequence and return starting value for reserved range
Jason Etheridge [Tue, 3 Oct 2017 19:05:35 +0000]
push bib sequence and return starting value for reserved range

4 years agoMerge branch 'master' of git.esilibrary.com:migration-tools
Jason Etheridge [Tue, 3 Oct 2017 18:33:58 +0000]
Merge branch 'master' of git.esilibrary.com:migration-tools

4 years agofix mig-init
Jason Etheridge [Tue, 3 Oct 2017 18:33:47 +0000]
fix mig-init

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agoadding actor.usr_setting to production tables
Rogan Hamby [Tue, 26 Sep 2017 13:03:59 +0000]
adding actor.usr_setting to production tables

4 years agomigration_tools.attempt_sierra_timestamp
Jason Etheridge [Mon, 11 Sep 2017 07:34:18 +0000]
migration_tools.attempt_sierra_timestamp

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agomore address parsing helpers
Jason Etheridge [Mon, 11 Sep 2017 02:58:51 +0000]
more address parsing helpers

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agopsql variable for :migschema
Jason Etheridge [Sat, 9 Sep 2017 02:51:38 +0000]
psql variable for :migschema

mig-sql already puts the migration schema in the search path, but
this can be useful for things like

    creator = (SELECT id FROM actor_usr WHERE usrname = :'migschema' || '_admin')

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agomultiple file support for mig-quicksheet
Jason Etheridge [Mon, 28 Aug 2017 19:22:49 +0000]
multiple file support for mig-quicksheet

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agomig-init: loop through sorted mig-sql/ directory
Jason Etheridge [Wed, 23 Aug 2017 20:45:15 +0000]
mig-init: loop through sorted mig-sql/ directory

mig-init will look for .sql files in mig-sql to run

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agomig-sql: silence warning if no $BIBSTART
Jason Etheridge [Wed, 23 Aug 2017 20:42:32 +0000]
mig-sql: silence warning if no $BIBSTART

4 years agomig-env: set PGOPTIONS for search_path
Jason Etheridge [Tue, 8 Aug 2017 14:29:04 +0000]
mig-env: set PGOPTIONS for search_path

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agoextract_holdings: --suffix support
Jason Etheridge [Tue, 8 Aug 2017 13:42:28 +0000]
extract_holdings: --suffix support

helps when you need to extract tag data from multiple MARC sources in a given migration

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

4 years agoadding deletion of extra permissions and groups for users in remove ou scripts
Rogan Hamby [Thu, 3 Aug 2017 19:57:46 +0000]
adding deletion of extra permissions and groups for users in remove ou scripts

4 years agoadding basic program to convert from Apollo LDF to staging tables
Rogan Hamby [Thu, 3 Aug 2017 19:28:57 +0000]
adding basic program to convert from Apollo LDF to staging tables

4 years agoadding deletion of surveys to remove ou scripts
Rogan Hamby [Thu, 3 Aug 2017 19:23:38 +0000]
adding deletion of surveys to remove ou scripts

5 years agofix -fs, and more output when fixing stuff
Jason Etheridge [Thu, 29 Jun 2017 18:50:07 +0000]
fix -fs, and more output when fixing stuff

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

5 years agosome marc mappings for III Sierra
Jason Etheridge [Thu, 29 Jun 2017 17:01:44 +0000]
some marc mappings for III Sierra

5 years agomig-sql: pass through the BIBSTART env variable
Jason Etheridge [Thu, 29 Jun 2017 15:57:45 +0000]
mig-sql: pass through the BIBSTART env variable

available as :bibstart

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

5 years agomig-sql
Jason Etheridge [Wed, 3 May 2017 19:21:20 +0000]
mig-sql

Mig wrapper for psql.  Plan is to start injecting some useful mig-tracked
variables into psql sessions thus invoked.

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

5 years agosome stuff I had lying around
Jason Etheridge [Wed, 3 May 2017 19:02:43 +0000]
some stuff I had lying around

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

5 years agoASSERT not supported until PG 9.5
Jason Etheridge [Tue, 11 Apr 2017 20:04:25 +0000]
ASSERT not supported until PG 9.5

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

5 years agosome useful wrappers for loading bre's
Jason Etheridge [Fri, 7 Apr 2017 14:53:13 +0000]
some useful wrappers for loading bre's

5 years ago"mig" tool
Jason Etheridge [Fri, 7 Apr 2017 12:31:31 +0000]
"mig" tool

Wraps some new and existing tools in a git-like command system,
currently great for quickly getting files into staging tables,
has aspirations for one day tracking mapping relationships and
doing code generation for mapping.

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

5 years agothird variant of migration_tools.assert with success message as 3rd argument
Jason Etheridge [Thu, 30 Mar 2017 17:32:44 +0000]
third variant of migration_tools.assert with success message as 3rd argument

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

5 years ago--force-utf8 in leader for marc_cleanup
Jason Etheridge [Thu, 30 Mar 2017 17:19:08 +0000]
--force-utf8 in leader for marc_cleanup

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

5 years agoassert tool for use with \set ON_ERROR_STOP on
Jason Etheridge [Thu, 30 Mar 2017 17:18:25 +0000]
assert tool for use with \set ON_ERROR_STOP on

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

5 years agouse UNLOGGED tables for the base staging tables
Jason Etheridge [Tue, 14 Mar 2017 17:22:51 +0000]
use UNLOGGED tables for the base staging tables

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

5 years agoadd note to readme
Rogan Hamby [Mon, 27 Feb 2017 13:38:29 +0000]
add note to readme

5 years agoadd vacuum analyze script for database maintenance in remove_ou scripts
Rogan Hamby [Mon, 27 Feb 2017 13:37:56 +0000]
add vacuum analyze script for database maintenance in remove_ou scripts

5 years agoadd a script to aid in generating the delete file for patrons in remove_ou scripts
Rogan Hamby [Mon, 27 Feb 2017 13:37:02 +0000]
add a script to aid in generating the delete file for patrons in remove_ou scripts

5 years agoadd disable/renable rule so that copy locations can be deleted in remove_ou scripts
Rogan Hamby [Mon, 27 Feb 2017 13:35:55 +0000]
add disable/renable rule so that copy locations can be deleted in remove_ou scripts

5 years agoadd removing usr circ history to remove_ou scripts
Rogan Hamby [Mon, 27 Feb 2017 13:34:45 +0000]
add removing usr circ history to remove_ou scripts

5 years agoadd removing call number suffixes and prefixes to remove_ou scripts
Rogan Hamby [Mon, 27 Feb 2017 13:33:59 +0000]
add removing call number suffixes and prefixes to remove_ou scripts

5 years agoremoved unncessary schema declared on column and fixed issue where value needed was...
Rogan Hamby [Thu, 9 Feb 2017 16:36:13 +0000]
removed unncessary schema declared on column and fixed issue where value needed was ID and org shortname was given

5 years agoremoved moving call numbers where items and call numbers were on incompatible orgs...
Rogan Hamby [Mon, 17 Oct 2016 18:50:31 +0000]
removed moving call numbers where items and call numbers were on incompatible orgs to be deleted

5 years agoadded check to make sure holds were fulfilled when deleting by fulfillment library
Rogan Hamby [Mon, 17 Oct 2016 18:47:19 +0000]
added check to make sure holds were fulfilled when deleting by fulfillment library

5 years agoremoved nulling of parent circs where child circ is at non-deleted library
Rogan Hamby [Mon, 17 Oct 2016 18:44:56 +0000]
removed nulling of parent circs where child circ is at non-deleted library

5 years agoRemove stat cats owned by to be deleted org unit, stat cat entries and entry user...
Rogan Hamby [Mon, 17 Oct 2016 16:25:44 +0000]
Remove stat cats owned by to be deleted org unit, stat cat entries and entry user mappings by those stat cats or patrons of those org units.

5 years agomove call numbers from to be deleted org to undeleted org when the copy is at an...
Rogan Hamby [Mon, 17 Oct 2016 16:03:20 +0000]
move call numbers from to be deleted org to undeleted org when the copy is at an undeleted org

5 years agoadd deleting holds by request lib
Rogan Hamby [Mon, 17 Oct 2016 15:03:02 +0000]
add deleting holds by request lib

5 years agoremove auditor history edits
Rogan Hamby [Mon, 17 Oct 2016 14:53:07 +0000]
remove auditor history edits

5 years agoremove holds based on fulfillment library
Rogan Hamby [Mon, 17 Oct 2016 14:34:18 +0000]
remove holds based on fulfillment library

5 years agoupdate the noncat circ code to note delete non cat circs when the staff member is...
Rogan Hamby [Mon, 17 Oct 2016 14:17:49 +0000]
update the noncat circ code to note delete non cat circs when the staff member is registered at the wrong library

5 years agodeletions of non cat circs
Rogan Hamby [Mon, 17 Oct 2016 14:08:39 +0000]
deletions of non cat circs

5 years agoremove albermale references by putting in a variable and deleting usr messages
Rogan Hamby [Mon, 17 Oct 2016 14:03:24 +0000]
remove albermale references by putting in a variable and deleting usr messages

5 years agoadded line to handle child checkouts where parent checkout is at the org being delete...
Rogan Hamby [Mon, 17 Oct 2016 12:48:24 +0000]
added line to handle child checkouts where parent checkout is at the org being deleted but child is not by nulling parent circ, unfortunately no way to indicate this

5 years agoOU data removal script
Galen Charlton [Fri, 14 Oct 2016 15:17:15 +0000]
OU data removal script

This is a set of SQL scripts to be run, in order, to completely
remove data belonging to an OU and its descendants in an Evergreen
database.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

5 years agofilter out linefeeds and carriage returns from holding tag values
Jason Etheridge [Wed, 14 Sep 2016 13:59:42 +0000]
filter out linefeeds and carriage returns from holding tag values

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

5 years agofilter out tabs from holding tag values
Jason Etheridge [Wed, 14 Sep 2016 13:33:54 +0000]
filter out tabs from holding tag values

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

5 years agominor whitespace correction
Galen Charlton [Mon, 8 Aug 2016 18:25:17 +0000]
minor whitespace correction

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

5 years agoMake --wait optional in eg_staged_bib_overlay.
Jason Stephenson [Sat, 6 Aug 2016 21:28:47 +0000]
Make --wait optional in eg_staged_bib_overlay.

This commit only causes load bibs to wait if the --wait option is
specified with a non-zero value.  It also changes the default for the
global $wait variable that the option is read into to zero.

Signed-off-by: Jason Stephenson <jason@sigio.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>

5 years agoPerl eval improvements in eg_staged_bib_overlay.
Jason Stephenson [Sun, 7 Aug 2016 14:23:42 +0000]
Perl eval improvements in eg_staged_bib_overlay.

Two changes to improve the eval conditions added in the previous
commit:

1. Both error handlers now warn with the same message:

   Record $i is bad: $@; skipping.

Thereby reporting the original message in case the user wants to try
and recover the records later.

2. If a bib record's 901$c is either missing or not numeric, die() is
called with a message to that effect.  This eliminates the Perl
warnings about exiting an eval block with next.

Signed-off-by: Jason Stephenson <jason@sigio.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>

5 years agoUse eval in eg_staged_bib_overlay.
Jason Stephenson [Sat, 6 Aug 2016 21:10:12 +0000]
Use eval in eg_staged_bib_overlay.

Wrap the parsing of MARC from input files in eval blocks with an error
handler to skip to the next record.  I have encounterd bad records
that have stopped the processing of the rest of the file.  This
occurred the very first time I used this script.

Signed-off-by: Jason Stephenson <jason@sigio.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>

5 years agoconvert DestinyCustomReport XML to TSV
Jason Etheridge [Tue, 19 Jul 2016 20:17:49 +0000]
convert DestinyCustomReport XML to TSV

6 years agoadd --fix-subfield
Jason Etheridge [Mon, 6 Jun 2016 19:59:44 +0000]
add --fix-subfield

and tweak some comments and command-line help

Signed-off-by: Jason Etheridge <jason@esilibrary.com>

6 years agoexample Follet/Destiny 852 map
Jason Etheridge [Fri, 27 May 2016 19:17:59 +0000]
example Follet/Destiny 852 map