Ben Ostrowsky [Mon, 2 Nov 2009 21:12:03 +0000]
Unicorn charge.data to TSV parser
git-svn-id: svn://nox.esilibrary.com/migration-tools@634
eee7cc8d-164e-4af6-8e1b-
092a69004917
Ben Ostrowsky [Mon, 2 Nov 2009 15:17:22 +0000]
Unicorn bill parser finished
git-svn-id: svn://nox.esilibrary.com/migration-tools@633
eee7cc8d-164e-4af6-8e1b-
092a69004917
Ben Ostrowsky [Fri, 30 Oct 2009 19:40:47 +0000]
Unfinished draft of Unicorn bill converter
git-svn-id: svn://nox.esilibrary.com/migration-tools@632
eee7cc8d-164e-4af6-8e1b-
092a69004917
Ben Ostrowsky [Fri, 30 Oct 2009 18:55:39 +0000]
Utter pedantry? Perhaps, but misleading comments are bugs too.
git-svn-id: svn://nox.esilibrary.com/migration-tools@630
eee7cc8d-164e-4af6-8e1b-
092a69004917
Ben Ostrowsky [Tue, 27 Oct 2009 20:18:57 +0000]
First draft mark two (with syntax errors fixed)
git-svn-id: svn://nox.esilibrary.com/migration-tools@628
eee7cc8d-164e-4af6-8e1b-
092a69004917
Ben Ostrowsky [Tue, 27 Oct 2009 20:10:40 +0000]
First finished draft of new unicorn_patrons_to_tsv.pl
git-svn-id: svn://nox.esilibrary.com/migration-tools@627
eee7cc8d-164e-4af6-8e1b-
092a69004917
Ben Ostrowsky [Tue, 27 Oct 2009 19:57:22 +0000]
Finished the bit that processes the records. Next up: actual output.
git-svn-id: svn://nox.esilibrary.com/migration-tools@626
eee7cc8d-164e-4af6-8e1b-
092a69004917
Ben Ostrowsky [Tue, 27 Oct 2009 17:39:05 +0000]
Reforking unicorn importer; this is an unfinished version of a patron importer
git-svn-id: svn://nox.esilibrary.com/migration-tools@624
eee7cc8d-164e-4af6-8e1b-
092a69004917
Ben Ostrowsky [Mon, 26 Oct 2009 18:11:29 +0000]
Cleaning up older copy of unicorn_to_tsv.pl
git-svn-id: svn://nox.esilibrary.com/migration-tools@623
eee7cc8d-164e-4af6-8e1b-
092a69004917
Ben Ostrowsky [Mon, 26 Oct 2009 18:05:56 +0000]
Unicorn TSV-maker can now handle bill.data and charge.data too.
git-svn-id: svn://nox.esilibrary.com/migration-tools@621
eee7cc8d-164e-4af6-8e1b-
092a69004917
Jason Etheridge [Wed, 9 Sep 2009 22:00:28 +0000]
change \r\n to \n if they occur
git-svn-id: svn://nox.esilibrary.com/migration-tools@617
eee7cc8d-164e-4af6-8e1b-
092a69004917
Ben Ostrowsky [Tue, 11 Aug 2009 12:20:22 +0000]
First draft of Unicorn patron-file converter. No known bugs, but memory linear (at least) with number of patrons.
git-svn-id: svn://nox.esilibrary.com/migration-tools@606
eee7cc8d-164e-4af6-8e1b-
092a69004917
Ben Ostrowsky [Wed, 16 Dec 2009 21:32:00 +0000]
And this turns off all the STDERR verbosity. Sheesh; sorry.
git-svn-id: svn://nox.esilibrary.com/migration-tools@652
eee7cc8d-164e-4af6-8e1b-
092a69004917
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Ben Ostrowsky [Wed, 16 Dec 2009 21:30:45 +0000]
Regex fix was a bug; this undoes it
git-svn-id: svn://nox.esilibrary.com/migration-tools@651
eee7cc8d-164e-4af6-8e1b-
092a69004917
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Ben Ostrowsky [Wed, 16 Dec 2009 21:02:26 +0000]
Fixed regex to handle lines that begin with a dot and something other than a capital letter (i.e., not a field label)
git-svn-id: svn://nox.esilibrary.com/migration-tools@650
eee7cc8d-164e-4af6-8e1b-
092a69004917
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Ben Ostrowsky [Fri, 30 Oct 2009 19:18:51 +0000]
FORM=foo should not be ignored
git-svn-id: svn://nox.esilibrary.com/migration-tools@631
eee7cc8d-164e-4af6-8e1b-
092a69004917
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Ben Ostrowsky [Mon, 26 Oct 2009 18:10:32 +0000]
unicorn_to_tsv.pl has been generalized to use non-user-specific wording
git-svn-id: svn://nox.esilibrary.com/migration-tools@622
eee7cc8d-164e-4af6-8e1b-
092a69004917
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Galen Charlton [Mon, 16 Jul 2012 15:53:35 +0000]
add copyright statement and GPL2+ license statement
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Galen Charlton [Mon, 16 Jul 2012 15:43:53 +0000]
make cleanup_merge_map.pl executable
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Galen Charlton [Mon, 16 Jul 2012 15:43:06 +0000]
remove .csvignore
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Ben Ostrowsky [Wed, 22 Feb 2012 20:30:10 +0000]
Three variations on apply_circ_matrix: _before_20 spells it recuring instead of recurring, _after_20 uses the matchpoint structure, and the unmakred case (for now, at least) does neither.
Ben Ostrowsky [Tue, 24 Jan 2012 17:30:40 +0000]
Added ContentReserve to the list of providers in insert_856_9
Ben Ostrowsky [Mon, 16 Jan 2012 18:25:29 +0000]
This time for sure!
Ben Ostrowsky [Mon, 16 Jan 2012 18:22:59 +0000]
Now handles subfield codes A-Z in addition to a-z (this is strictly speaking not allowed in data, but it exists anyway)
Ben Ostrowsky [Fri, 6 Jan 2012 17:32:30 +0000]
Added two related functions for updating a copy's owning_lib, both called
migration_tools.change_owning_lib. The difference is that if you call it with
the arguments (BIGINT, TEXT), it will try to look it up as a shortname and
pass the resulting actor.org_unit ID to change_owning_lib(BIGINT, INTEGER).
Ben Ostrowsky [Thu, 5 Jan 2012 14:11:06 +0000]
Avoid collisions while refreshing OPAC visibility
Ben Ostrowsky [Wed, 4 Jan 2012 19:39:39 +0000]
Replaced TRUNCATE with DELETE FROM in migration_tools.create_cards
Ben Ostrowsky [Thu, 1 Dec 2011 12:53:11 +0000]
Replaced TRUNCATE with DELETE FROM
Ben Ostrowsky [Mon, 21 Nov 2011 15:56:04 +0000]
Horizon-specific functions, beginning with a date conversion routine
Ben Ostrowsky [Tue, 18 Oct 2011 15:19:20 +0000]
Added new helper utility: migration_tools.refresh_opac_visible_copies
Ben Ostrowsky [Fri, 16 Sep 2011 13:39:42 +0000]
Avoiding collisions with existing permission groups and existing permissions
Ben Ostrowsky [Tue, 13 Sep 2011 18:20:58 +0000]
Improved check_ou_type
Ben Ostrowsky [Tue, 13 Sep 2011 17:55:19 +0000]
Added new function: check_ou_depth
Ben Ostrowsky [Fri, 9 Sep 2011 19:16:39 +0000]
insert_856_9: more accurate detection of incumbent subfield-9 data
Ben Ostrowsky [Mon, 8 Aug 2011 12:53:52 +0000]
Added migration_tools.zip_to_city_state_county
Ben Ostrowsky [Mon, 25 Jul 2011 16:28:50 +0000]
When Geonames has no data for a ZIP, use patron data even if --believegn
Galen Charlton [Fri, 22 Jul 2011 16:05:38 +0000]
fix bug introduced in r739
When replacing named captures for Perl 5.8 compatibility,
need to make sure regex groups that shouldn't capture
are marked as such.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Ben Ostrowsky [Wed, 20 Jul 2011 13:06:29 +0000]
Added --believegn flag to believe Geonames over patron data where they conflict
Ben Ostrowsky [Wed, 20 Jul 2011 13:05:17 +0000]
Added --belivegn flag to believe Geonames over patron data where they conflict
Ben Ostrowsky [Tue, 19 Jul 2011 18:15:29 +0000]
Handle percents in apply_circ_matrix
Ben Ostrowsky [Tue, 28 Jun 2011 12:09:05 +0000]
Added attempt_timestamptz
Ben Ostrowsky [Thu, 23 Jun 2011 17:10:17 +0000]
Utilities to nuke data
Ben Ostrowsky [Fri, 10 Jun 2011 17:07:40 +0000]
Munging ind2 in insert_856_9
Ben Ostrowsky [Wed, 11 May 2011 13:25:01 +0000]
New utility. Give it city-state-zip data in tab-separated format and it will give you the winning city/state for each zip in the same format.
Ben Ostrowsky [Tue, 10 May 2011 18:46:59 +0000]
Script by Galen, used in dedupe/merge processing.
Ben Ostrowsky [Thu, 5 May 2011 16:45:50 +0000]
Avoiding error message about non-numeric comparison; reporting on tags that are not strictly numeric
Ben Ostrowsky [Thu, 5 May 2011 16:33:18 +0000]
Call attention to subfields appearing more than once within a single tag
Ben Ostrowsky [Wed, 6 Apr 2011 13:48:09 +0000]
New function: migration_tools.change_call_number (copy_id BIGINT, new_label TEXT).
If the old or new label is ##URI##, the label will not be changed.
If the old label's volume is vacated by the change, it will be marked deleted.
Ben Ostrowsky [Wed, 23 Mar 2011 14:21:17 +0000]
Updated field names (recuring->recurring) in one function and added another to insert 856 $9 as needed
Ben Ostrowsky [Wed, 9 Mar 2011 18:11:33 +0000]
Routine to create cards in a migration schema
Ben Ostrowsky [Tue, 15 Feb 2011 13:08:36 +0000]
Function for assigning standing penalties
Ben Ostrowsky [Tue, 15 Feb 2011 13:07:48 +0000]
New options: --quiet, --ignoresubtitle
Ben Ostrowsky [Thu, 13 Jan 2011 16:27:18 +0000]
Adding metarecord spells as functions for convenience
Ben Ostrowsky [Mon, 22 Nov 2010 19:26:45 +0000]
Added --ignore-indexes option to grab more data, much of which will be junk to filter out downstream
Ben Ostrowsky [Thu, 18 Nov 2010 21:09:03 +0000]
A few adjustments
Ben Ostrowsky [Thu, 18 Nov 2010 20:33:38 +0000]
Handles items as well -- row length is a two-byte field
Ben Ostrowsky [Thu, 18 Nov 2010 19:29:55 +0000]
Works fine on TITLE.DB, but misses the boat on other files.
Ben Ostrowsky [Thu, 18 Nov 2010 16:10:20 +0000]
(cosmetic) Adjust digits in column numbers to fit data
Ben Ostrowsky [Thu, 18 Nov 2010 16:06:18 +0000]
Tools for handling Bibliofile *.DB/*.DBD files
Galen Charlton [Fri, 12 Nov 2010 15:18:55 +0000]
die if cannot open output or exception files
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Ben Ostrowsky [Mon, 8 Nov 2010 21:16:32 +0000]
A few refinements
Ben Ostrowsky [Mon, 8 Nov 2010 16:51:27 +0000]
A tool for generating Apache vhost configs from a table modeled after actor.org_unit_setting.
USAGE: ./opac_settings.pl --bootstrap /openils/conf/opensrf_core.xml --schema m_sage
Input will look like this:
evergreen=# select * from m_sage.opac_settings where org_unit = 107;
id | org_unit | name | value
-----+----------+-------------------+-----------------------------------
36 | 107 | opac.server_name | "conhs.sage.eou.edu"
93 | 107 | opac.server_alias | "www.conhs.sage.eou.edu"
147 | 107 | opac.directory | "condonhs"
251 | 107 | opac.ssl_cert | "ssl/wildcard.sage.eou.edu.crt"
301 | 107 | opac.ssl_key | "ssl/wildcard.sage.eou.edu.key"
Multiple libraries may share an 'opac.directory' (branches of a system with the same OPAC look and feel).
Multiple 'opac.server_alias' rows may be defined.
Ben Ostrowsky [Fri, 22 Oct 2010 16:28:37 +0000]
migration_tools.stage_not_applicable_asset_stat_cats
Ben Ostrowsky [Sun, 19 Sep 2010 01:11:42 +0000]
Better output-file renaming
Ben Ostrowsky [Fri, 10 Sep 2010 17:48:17 +0000]
Generates a TSV with egid and MARCXML snippets to preserve for post-deduping reinsertion (insert_tags).
Ben Ostrowsky [Fri, 10 Sep 2010 14:56:56 +0000]
Added feedback to remind the user where to find the results (since it doesn't go to STDOUT).
Ben Ostrowsky [Wed, 8 Sep 2010 16:07:48 +0000]
New function: apply_circ_matrix('m_foo.action_circulation').
Sets duration, fine, and max_fine names and values, and renewal_remaining,
based on a circ matrix (and patron profiles, and circ modifiers) already in place.
Ben Ostrowsky [Fri, 20 Aug 2010 16:27:50 +0000]
More precise deduping for insert_tags
Ben Ostrowsky [Fri, 20 Aug 2010 16:25:32 +0000]
Added function: insert_tags (marc, '<datafield tag="999" ... </datafield>')
Inserts into the proper numerical order, ignoring fields that are duplicates of each other.
Ben Ostrowsky [Thu, 10 Jun 2010 19:47:35 +0000]
attempt_phone can take '' as second argument to refrain from adding an area code; added is_blank(TEXT) for convenience
Ben Ostrowsky [Thu, 3 Jun 2010 15:44:09 +0000]
New function: migration_tools.is_staff_profile(actor.usr.profile), returns boolean
Ben Ostrowsky [Thu, 27 May 2010 15:15:24 +0000]
Built-in manual gave wrong command-line arguments. Fixed.
Ben Ostrowsky [Thu, 6 May 2010 16:29:04 +0000]
Added two MARC-manipulating functions from Galen
Ben Ostrowsky [Fri, 20 May 2011 19:52:18 +0000]
Eliminating one line of Perl 5.10 dependency
Ben Ostrowsky [Fri, 20 May 2011 16:16:41 +0000]
Fixed typo
Ben Ostrowsky [Fri, 20 May 2011 16:13:09 +0000]
Rewrote some regular expressions to make them compatible with Perl versions before 5.10
Ben Ostrowsky [Fri, 13 May 2011 15:54:46 +0000]
To be run immediately after normal EG installation
Galen Charlton [Fri, 17 Jun 2011 03:10:16 +0000]
add license statement
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Galen Charlton [Fri, 17 Jun 2011 03:06:00 +0000]
SCLENDS bibliographic dedupe routine
This implements a bibliographic deduplication routine based
on criteria and an algorithm specified by the South Carolina
State Library on behalf of the SC LENDS consortium. This work
was sponsored by SC LENDS, whose impetus is gratefully
acknowledged. Portions of this script were subseqently expanded
based on the advice of the Indiana State Library on the behalf
of the Evergreen Indiana project.
This is meant to run in an Evergreen database. Usage would
be something like:
psql -U evergreen < sclends_dedupe.sql
However, in practice it is highly recommend to edit the script
first; any bibliographic deduplication requires close attention
to the particular data found in the Evergreen database.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Ben Ostrowsky [Wed, 11 May 2011 17:07:35 +0000]
Two new utilities that, together, can generate a zips.txt file customized for your patron database.
Galen Charlton [Thu, 6 May 2010 18:12:35 +0000]
update test case
Thanks to Sue Hopkins for the catch.
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Ben Ostrowsky [Thu, 22 Apr 2010 17:29:59 +0000]
Added note about non-ESI tools' licensing
Signed-off-by: Equinox Git Owner <koha@yeti.esilibrary.com>
Galen Charlton [Thu, 22 Apr 2010 04:53:33 +0000]
added COPYING, LICENSE, and README for release
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Galen Charlton [Tue, 13 Apr 2010 19:23:51 +0000]
skip over records that cannot be parsed
Galen Charlton [Tue, 13 Apr 2010 19:22:01 +0000]
revert previous commit (swept too much in)
Galen Charlton [Tue, 13 Apr 2010 19:18:52 +0000]
recover gracefully after an XML parsing error
Galen Charlton [Tue, 13 Apr 2010 00:19:06 +0000]
NAME
xlsx2tab
SUMMARY
Quick-and-dirty filter lifted from the Spreadsheet::XSLX POD to convert the first sheet of an Excel .xlsx file to TSV
USAGE
xlsx2tab foo.xlsx > foo.tsv
Ben Ostrowsky [Mon, 12 Apr 2010 22:08:34 +0000]
Will leave 'unlisted' alone
Ben Ostrowsky [Mon, 12 Apr 2010 22:06:41 +0000]
Handling a few more exceptions...
Ben Ostrowsky [Mon, 12 Apr 2010 21:54:27 +0000]
Added logic to attempt_phone for handling a few more variants correctly
Ben Ostrowsky [Mon, 12 Apr 2010 21:47:43 +0000]
Added function attempt_phone (TEXT,TEXT) RETURNS TEXT. First argument is legacy phone data, second is area code to prepend.
Ben Ostrowsky [Tue, 6 Apr 2010 14:55:30 +0000]
debugged a comment ;)
Ben Ostrowsky [Tue, 6 Apr 2010 14:50:54 +0000]
Renamed utility, added --help and perldoc documentation
Ben Ostrowsky [Tue, 6 Apr 2010 14:31:16 +0000]
Moving DBF-to-TSV utility (not Athena-specific)
Ben Ostrowsky [Tue, 6 Apr 2010 14:23:16 +0000]
First version of a utility to convert Athena DBF to TSV suitable for \copy.
Suitable includes escaping backslash, tab, newline, and carriage return,
and converting output to UTF-8.
Galen Charlton [Wed, 17 Mar 2010 19:20:33 +0000]
normalize output by escaping backslashes
This allows psql to import the data correctly; use
the new --disable-pg-normalization option to disable
this behavior if the output is not meant to go
directly into a Pg database via copy or \copy
Galen Charlton [Mon, 15 Mar 2010 16:34:57 +0000]
fix retrieval of correct column headers and data from non-holdings tags
This work was sponsored by the Indiana State Library.
Galen Charlton [Fri, 12 Mar 2010 20:32:45 +0000]
improved handling for drawing data from fields other than the holdings tag
Suppose the 852 is the main holdings tag, but information is needed
from the 590$a (e.g., as an item note). The extract_holdings configuration file
can now work as follows:
# to map the subfield contents to the item record
barcode 852 p
note 590 a
# to map, but have the 590$a information be associated
# only with the first item in the bib; if m:first is
# not supplied, the 590$a note will be supplied
# to all items associated with the bib
barcode 852 p
note 590 a m:first
# to put all occurrences of the 590$a into a separate
# file for linking with the appropriate items. m:multi
# *must* be supplied if the 590 tag occurs more than once
# in the bib records
barcode 852 p
note 590 a m:multi
# to concatenate repeats of $a in the 590 together
barcode 852 p
note 590 a m:concatenate
# to concatenate with a custom separator
barcode 852 p
note 590 a m:concatenate c:' / '
Development sponsored by the Indiana State Library
Galen Charlton [Mon, 8 Mar 2010 21:14:08 +0000]
fix error in Codabar checkdigit algorithm
Ben Ostrowsky [Tue, 2 Feb 2010 15:55:35 +0000]
asset.copy.price is NUMERIC(8,2),
not NUMERIC(6,2).