1 Migration Data Work HOWTO / Toolkit
2 ========================================================================
3 The following is for migrating into an existing system like PINES:
5 Get the incoming bib data, and translate to UTF-8 MARCXML. It may
6 contain holdings. It may contain XML or MARC errors that you have to
7 sanitize before your tools will work. This is one way to translate
8 MARC-8 MARC21 to UTF-8 MARCXML:
10 yaz-marcdump -f MARC-8 -t UTF-8 -o marcxml incoming.mrc > incoming.mrc.xml
12 If you need to trim the bibs to a subset based on the presence of a
13 certain value in a specific tag/subfield (for example, if you have the
14 bibs for all libraries in a foreign system and only need bibs
15 belonging to a specific migrating library, you might filter based on
18 trim_marc_based_on_tag_subfield_value.pl 999 m BRANCH_CODE \
19 incoming.mrc.xml > incoming.filtered.mrc.xml
21 Embed potential native record ids into the incumbent records
23 set_record_ids.pl 100000 903 a incoming.mrc.xml > incoming.renumbered.mrc.xml
25 Get primary fingerprints for incoming data and get a bib dump of
26 matching records from the incumbent system
28 fingerprinter -r primary -t 903 -s a -o incoming.primary.fp \
29 -x incoming.primary.fp_err incoming.renumbered.mrc.xml
31 Edit the query_for_primary_matching_incumbent_record.pl script to
32 point to the correct Evergreen database and table holding the
33 incumbent primary fingerprints (FIXME add in how to create such a
36 query_for_primary_matching_incumbent_record.pl incoming.primary.fp | \
37 sort | uniq > primary_matching_incumbent.record_ids
39 In a postgres shell, you create a temporary table to hold these id's:
41 CREATE TABLE primary_matching_incumbent_records_for_incoming_library
43 COPY primary_matching_incumbent_records_for_incoming_library
44 FROM 'primary_matching_incumbent.record_ids';
46 To dump the matching incumbent records to a file, in a postgres shell
49 matching_incumbent_records.dump SELECT b.id, b.tcn_source, b.tcn_value,
50 regexp_replace(b.marc,E'\n','','g')
51 FROM biblio.record_entry AS b
52 JOIN primary_matching_incumbent_records_for_incoming_library
55 Now to turn that dump into a MARCXML file with record numbers and TCN
56 embedded in tag 901, do:
58 marc_add_ids -f id -f tcn_source -f tcn_value -f marc < \
59 matching_incumbent_records.dump » matching_incumbent_records.mrc.xml
61 It's possible that this file may need to be itself sanitized some.
62 This will transform code=""" into code="&x0022;", for example:
64 cat matching_incumbent_records.mrc.xml | \
65 sed 's/code=\"\"\"/code=\"\"\"/' > \
66 matching_incumbent_records.escaped.mrc.xml
68 Get full fingerprints for both datasets and match them.
70 fingerprinter -r full -t 901 -s c -o incumbent.fp -x incumbent.fp_err \
71 matching_incumbent_records.mrc.xml
72 fingerprinter -r full -t 903 -s a -o incoming.fp -x incoming.fp_err \
73 incoming.renumbered.mrc.xml
75 The script below will produce matched groupings, and can optionally
76 take a 4th and 5th parameter providing scoring information for
77 determining lead records. In the past, this would consider certain
78 metrics for MARC quality, but in the latest incarnation, it assumes an
79 incumbent record will be the lead record, and looks at # of holdings
80 and possible matching of tag 245 subfield b for determining which of
81 the incumbent records would be the lead record. The example
82 invocation below does not use scoring.
84 match_fingerprints.pl "name of dataset for dedup interface" \
85 incumbent.fp incoming.fp
87 This will produce two files, match.groupings and match.record_ids.
88 The format for match.groupings is suitable for insertion into the db
89 for the dedup interface.
91 Import these matches and records into the legacy dedup interface for viewing:
93 Now to tar up the specific MARC records involved for the dedup interface:
95 cat match.groupings | cut -d^ -f3 » incumbent.record_ids
96 cat match.groupings | cut -d^ -f5 | cut -d, -f2- | sed 's/,/\n/g' > \
98 mkdir dataset ; cd dataset
99 select_marc.pl ../incumbent.record_ids 901 c \
100 ../matching_incumbent_records.mrc.xml
101 select_marc.pl ../incoming.record_ids 903 a \
102 ../incoming.renumbered.mrc.xml
104 tar cvf dataset.tar dataset
106 In a mysql shell for the database used with the dedup interface:
108 LOAD DATA LOCAL INFILE 'match.groupings' INTO TABLE record_group
109 FIELDS TERMINATED BY '^'
110 ( status, dataset, best_record,records,original_records );
112 Create a pretty printed text dump of the non-matching incoming records:
114 dump_inverse_select_marc.pl incoming.record_ids 903 a \
115 incoming.renumbered.mrc.xml > non_matching_incoming.mrc.txt 2> \
116 non_matching_incoming.mrc.txt.err