Bug fix for issue #136

added created.dow (day of week) to template

CHANGELOG.md

@@ -4,6 +4,68 @@ All notable changes to this project will be documented in this file. Dates are d
 
 Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 
+#### [v0.29.12](https://github.com/RhetTbull/osxphotos/compare/v0.29.9...v0.29.12)
+
+> 7 June 2020
+
+- Refactoring with sourceryAI [`5c7a0c3`](https://github.com/RhetTbull/osxphotos/commit/5c7a0c3a246cd5fec329b4fd4979d2b77352f916)
+- Partial fix for #155 [`2271d89`](https://github.com/RhetTbull/osxphotos/commit/2271d8935507ecc27e6227b11b4796f2f4d2f10d)
+- Partial fix for #155 [`62d096b`](https://github.com/RhetTbull/osxphotos/commit/62d096b5a1a7e960195ec5c48fc9cffbebf2c735)
+
+#### [v0.29.9](https://github.com/RhetTbull/osxphotos/compare/v0.29.8...v0.29.9)
+
+> 31 May 2020
+
+- Added --filename to CLI, closes #89 [`#89`](https://github.com/RhetTbull/osxphotos/issues/89)
+- Updated CHANGELOG.md [`d47fd46`](https://github.com/RhetTbull/osxphotos/commit/d47fd46a21881bea86d1bc624c6027e2cbe08d9c)
+
+#### [v0.29.8](https://github.com/RhetTbull/osxphotos/compare/v0.29.5...v0.29.8)
+
+> 31 May 2020
+
+- Added --edited-suffix to CLI, closes #145 [`#145`](https://github.com/RhetTbull/osxphotos/issues/145)
+- refactored render_template, closes #149 [`#149`](https://github.com/RhetTbull/osxphotos/issues/149)
+- Added test for Photos 5 on 10.15.5 [`2243395`](https://github.com/RhetTbull/osxphotos/commit/2243395bff9e1cc379626cc5007e44e6e63b95e0)
+- Refactored template code out of PhotoInfo into PhotoTemplate [`16f802b`](https://github.com/RhetTbull/osxphotos/commit/16f802bf717610e13712b8aa477d05d94b14d294)
+- Added test for SearchInfo on 10.15.5 [`3a8bef1`](https://github.com/RhetTbull/osxphotos/commit/3a8bef1572e4d83b1e0a4b85c8f06e329cc7e8de)
+- performance improvements for update and export_db [`42b89d3`](https://github.com/RhetTbull/osxphotos/commit/42b89d34f3d14818daefbd3bfabc1be9344d2e1a)
+- More refactoring in PhotoTemplate [`f35ea70`](https://github.com/RhetTbull/osxphotos/commit/f35ea70b72e8c6743b1f6009466d2a15d40338ac)
+
+#### [v0.29.5](https://github.com/RhetTbull/osxphotos/compare/v0.29.2...v0.29.5)
+
+> 25 May 2020
+
+- added created.dow (day of week) to template [`#147`](https://github.com/RhetTbull/osxphotos/pull/147)
+- Added --dry-run option to CLI export, closes #91 [`#91`](https://github.com/RhetTbull/osxphotos/issues/91)
+- added created.dd and modified.dd to template system, closes #135 [`#135`](https://github.com/RhetTbull/osxphotos/issues/135)
+- Catch exception in folder processing to address #148 [`46fdc94`](https://github.com/RhetTbull/osxphotos/commit/46fdc94398c80b157048649434c7312074ce5c58)
+- Updated CHANGELOG.md [`af750dd`](https://github.com/RhetTbull/osxphotos/commit/af750dd2e392be1a7163cf32497526405665ea70)
+- Added test for DateTimeFormatter.dow [`09c7d18`](https://github.com/RhetTbull/osxphotos/commit/09c7d18901b61669d8b9242babd82eba6987c89a)
+
+#### [v0.29.2](https://github.com/RhetTbull/osxphotos/compare/v0.29.1...v0.29.2)
+
+> 24 May 2020
+
+- Added try/except for bad datettime values [`1d095d7`](https://github.com/RhetTbull/osxphotos/commit/1d095d7284bae57037b8b200c8b3422835c611b2)
+
+#### [v0.29.1](https://github.com/RhetTbull/osxphotos/compare/v0.29.0...v0.29.1)
+
+> 23 May 2020
+
+- Catch illegal timestamp value [`#146`](https://github.com/RhetTbull/osxphotos/pull/146)
+- Updated CHANGELOG.md [`1450b3c`](https://github.com/RhetTbull/osxphotos/commit/1450b3ccace326fe1c0ed810a1b40e781709acb3)
+
+#### [v0.29.0](https://github.com/RhetTbull/osxphotos/compare/v0.28.19...v0.29.0)
+
+> 23 May 2020
+
+- Made --exiftool and --export-as-hardlink incompatible in CLI to fix #132 [`#132`](https://github.com/RhetTbull/osxphotos/issues/132)
+- Added --update to CLI export; reference issue #100 [`b1171e9`](https://github.com/RhetTbull/osxphotos/commit/b1171e96cc06362555725995bb311317eb163e49)
+- Added as_dict to PlaceInfo [`8c4fe40`](https://github.com/RhetTbull/osxphotos/commit/8c4fe40aa6850f166e526cffaa088550884399af)
+- Updated README.md [`11d368a`](https://github.com/RhetTbull/osxphotos/commit/11d368a69cbe67e909e64b020f0334fc09dd3ac4)
+- Updated CHANGELOG.md [`cafa483`](https://github.com/RhetTbull/osxphotos/commit/cafa483cfc228c651a03d3361d6d48a35deab1e8)
+- version bump [`c06c230`](https://github.com/RhetTbull/osxphotos/commit/c06c230a469754691d11fff1034fb02daeeba649)
+
 #### [v0.28.19](https://github.com/RhetTbull/osxphotos/compare/v0.28.18...v0.28.19)
 
 > 15 May 2020
@@ -12,6 +74,8 @@ Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 - Revert "test library updates" [`48e9c32`](https://github.com/RhetTbull/osxphotos/commit/48e9c32add549e66c3ef8c65f8821f5033b55b11)
 - test library updates [`d125854`](https://github.com/RhetTbull/osxphotos/commit/d125854f2a04e37747af3e0796370a565c1c9bd0)
 - Updated CHANGELOG.md [`e228cfa`](https://github.com/RhetTbull/osxphotos/commit/e228cfab746055c8d6df428aebe0ed001fb6d4d0)
+- version bump [`bd9d5a2`](https://github.com/RhetTbull/osxphotos/commit/bd9d5a26f3bfcbb33896a139fa86cdab46768103)
+- Update README.md [`85760dc`](https://github.com/RhetTbull/osxphotos/commit/85760dc4fe2274d826ed80494fd4e66866398609)
 
 #### [v0.28.18](https://github.com/RhetTbull/osxphotos/compare/v0.28.17...v0.28.18)
 
@@ -37,6 +101,8 @@ Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 - Refactored photosdb and photoinfo to add SearchInfo and labels [`98b3f63`](https://github.com/RhetTbull/osxphotos/commit/98b3f63a92aa2105f8fa97af992fc6fe2d78b973)
 - Added additional test for --export-as-hardlink [`57315d4`](https://github.com/RhetTbull/osxphotos/commit/57315d44497fde977956f76f667470208f11aa2d)
 - added CHANGELOG.md [`00e1661`](https://github.com/RhetTbull/osxphotos/commit/00e16611fc86c05fb090d036084db9eb42444071)
+- Updated a couple of tests to use pytest-mock [`397db0d`](https://github.com/RhetTbull/osxphotos/commit/397db0d72fb218669a9ecbff134fa9b392a14661)
+- added test for export using hardlinks, fixed a test that failed if users locale settings were different to en_US [`b0ec6c6`](https://github.com/RhetTbull/osxphotos/commit/b0ec6c6b36d8cfe05723d47b210d9d7c5aabdfe5)
 
 #### [v0.28.13](https://github.com/RhetTbull/osxphotos/compare/v0.28.10...v0.28.13)
 
@@ -61,6 +127,7 @@ Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 - Updated CHANGELOG.md [`072a8d7`](https://github.com/RhetTbull/osxphotos/commit/072a8d795e5e15fa8ca8d8872aecf4cddd7837f7)
 - Update README.md [`5cc98c3`](https://github.com/RhetTbull/osxphotos/commit/5cc98c338bcc19fd05bf293eb3afe24c07c8b380)
 - Updated README.md [`a800711`](https://github.com/RhetTbull/osxphotos/commit/a80071111f810a1d7d6e2d735839e85499091ea4)
+- Update README.md [`1c9d4f2`](https://github.com/RhetTbull/osxphotos/commit/1c9d4f282beea2ac12273c8d0f9453bad1255c2c)
 
 #### [v0.28.7](https://github.com/RhetTbull/osxphotos/compare/v0.28.6...v0.28.7)
 
@@ -89,6 +156,8 @@ Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 - Updated tests and test library with RAW images [`9b9b54e`](https://github.com/RhetTbull/osxphotos/commit/9b9b54e590e43ae49fb3ae41d493a1f8faec4181)
 - Updated setup.py to resolve issue with bpylist2 on python < 3.8 [`8e4b88a`](https://github.com/RhetTbull/osxphotos/commit/8e4b88ad1fc18438f941e045bfc8aeac878914f9)
 - Added cli.py for use with pyinstaller [`cf28cb6`](https://github.com/RhetTbull/osxphotos/commit/cf28cb6452de17f2ef8d80435386e8d5a1aabd34)
+- added raw_is_original handling [`a337e79`](https://github.com/RhetTbull/osxphotos/commit/a337e79e13802b4824c2f088ce9db1c027d6f3c5)
+- Updated CHANGELOG.md [`22f1e8f`](https://github.com/RhetTbull/osxphotos/commit/22f1e8f2a6478e0576f6bff53e348aad8680ae69)
 
 #### [0.28.2](https://github.com/RhetTbull/osxphotos/compare/v0.28.1...0.28.2)
 
@@ -98,6 +167,7 @@ Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 - cleaned up SQL statements in _process_database4 [`6f28171`](https://github.com/RhetTbull/osxphotos/commit/6f281711e2001a63ffad076d7b9835272d5d09da)
 - Updated CHANGELOG.md [`1fa9583`](https://github.com/RhetTbull/osxphotos/commit/1fa9583ea689d54d2613a064f1ade25bcdfbf043)
 - Fixed suffix check on export to be case insensitive [`4b30b3b`](https://github.com/RhetTbull/osxphotos/commit/4b30b3b4260e2c7409e18825e5b626efe646db16)
+- test library update [`3bac106`](https://github.com/RhetTbull/osxphotos/commit/3bac106eb7a180e9e39643a89087d92bf2a437d0)
 
 #### [v0.28.1](https://github.com/RhetTbull/osxphotos/compare/v0.27.4...v0.28.1)
 
@@ -137,6 +207,7 @@ Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 - Added tests and README for AlbumInfo and FolderInfo [`d6a22b7`](https://github.com/RhetTbull/osxphotos/commit/d6a22b765ab17f6ef1ba8c50b77946f090979968)
 - Added albuminfo.py for AlbumInfo and FolderInfo classes [`9636572`](https://github.com/RhetTbull/osxphotos/commit/96365728c2ff42abfb6828872ffac53b4c3c8024)
 - Updated CHANGELOG.md [`cde56e9`](https://github.com/RhetTbull/osxphotos/commit/cde56e9d13baf3098ec85839cf1aaa33b4915ac9)
+- Update README.md TOC [`8544667`](https://github.com/RhetTbull/osxphotos/commit/8544667c729ea0d7fe39671d909e09cda519e250)
 
 #### [v0.26.1](https://github.com/RhetTbull/osxphotos/compare/v0.26.0...v0.26.1)
 
@@ -170,6 +241,8 @@ Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 - Updated render_filepath_template to support multiple values [`6a89888`](https://github.com/RhetTbull/osxphotos/commit/6a898886ddadc9d5bc9dbad6ee7365270dd0a26d)
 - Added {album}, {keyword}, and {person} to template system [`507c4a3`](https://github.com/RhetTbull/osxphotos/commit/507c4a374014f999ca19789bce0df0c14332e021)
 - Added places command to CLI [`fd5e748`](https://github.com/RhetTbull/osxphotos/commit/fd5e748dca759ea1c3a7329d447f363afe8418b7)
+- Updated export example [`01cd7fe`](https://github.com/RhetTbull/osxphotos/commit/01cd7fed6d7fc0c61c171a05319c211eb0a9f7c1)
+- Updated CHANGELOG.md [`daea30f`](https://github.com/RhetTbull/osxphotos/commit/daea30f1626a208209ab6854cbd3b12f4b0a3405)
 
 #### [v0.24.2](https://github.com/RhetTbull/osxphotos/compare/v0.24.1...v0.24.2)
 
@@ -248,6 +321,8 @@ Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 - Added query/export options for special media types [`2b7d84a`](https://github.com/RhetTbull/osxphotos/commit/2b7d84a4d103982ad874d875bafbc34d654d539a)
 - README.md update [`a27ce33`](https://github.com/RhetTbull/osxphotos/commit/a27ce33473df3260dfb7ed26e28295cbf87d1e78)
 - Test library updates [`2d7d0b8`](https://github.com/RhetTbull/osxphotos/commit/2d7d0b86e0008cae043e314937504f36ad882990)
+- Fixed bug in --download-missing related to burst images [`1f13ba8`](https://github.com/RhetTbull/osxphotos/commit/1f13ba837fe36ff4eeb48cca02f5312a88a0a765)
+- test library update [`acb6b9e`](https://github.com/RhetTbull/osxphotos/commit/acb6b9e72f7f6b8f4f1d64b46f270a4d3e984fef)
 
 #### [v0.22.13](https://github.com/RhetTbull/osxphotos/compare/v0.22.12...v0.22.13)
 
@@ -282,6 +357,8 @@ Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 - Slight refactor to PhotosDB.photos() [`91d5729`](https://github.com/RhetTbull/osxphotos/commit/91d5729beaa0f0c2583e6320b18d958429e66075)
 - Test library updates [`6e563e2`](https://github.com/RhetTbull/osxphotos/commit/6e563e214c569ba7838f7464de9258c3bba5db23)
 - Removed _tmp_file code that's no longer needed [`27994c9`](https://github.com/RhetTbull/osxphotos/commit/27994c9fd372303833a5794f1de9815f425c762e)
+- Updated photos_repl.py [`fdf636a`](https://github.com/RhetTbull/osxphotos/commit/fdf636ac8864ebb2cc324b1f9d3c6c82ee3910f9)
+- Updated CHANGELOG.md [`f910124`](https://github.com/RhetTbull/osxphotos/commit/f910124fe1fbf75d44c09c79607374bf000733a1)
 
 #### [v0.22.7](https://github.com/RhetTbull/osxphotos/compare/v0.22.4...v0.22.7)
 
@@ -294,6 +371,8 @@ Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 - Added XMP sidecar to export [`4dfb131`](https://github.com/RhetTbull/osxphotos/commit/4dfb131a21b1b1efefe3b918ecb06fc6fcb03f2c)
 - Added date_modified to PhotoInfo [`67b0ae0`](https://github.com/RhetTbull/osxphotos/commit/67b0ae0bf679815372d415c3064e21d46a5b8718)
 - Added date_modified to PhotoInfo [`4d36b3b`](https://github.com/RhetTbull/osxphotos/commit/4d36b3b31f3e0e74d9d111b6b691771e19f94086)
+- Updated CLI options with more descriptive metavar names [`e79cb92`](https://github.com/RhetTbull/osxphotos/commit/e79cb92693758c984dc789d5fa5d2e87e381e921)
+- CLI now looks for photos library to use if non specified by user [`50b7e69`](https://github.com/RhetTbull/osxphotos/commit/50b7e6920a694aa45f478d1131868525c9147919)
 
 #### [v0.22.4](https://github.com/RhetTbull/osxphotos/compare/v0.22.0...v0.22.4)
 
@@ -304,6 +383,8 @@ Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 - Refactor cli: singular --db, --json and query options. [`e214746`](https://github.com/RhetTbull/osxphotos/commit/e214746063271e6f9f586286103ed051ada49d85)
 - Implement from_date and to_date in PhotosDB as well as query and export command. Some refactoring of CLI as well. [`cfa2b4a`](https://github.com/RhetTbull/osxphotos/commit/cfa2b4a828facf0aff5bc19f777457ad776c4a05)
 - Refactored _query.  Still hairy, but less so. [`b9dee49`](https://github.com/RhetTbull/osxphotos/commit/b9dee4995c6d89fadb3d2482374b7098f2ab5ed9)
+- Updated README.md [`0aff83f`](https://github.com/RhetTbull/osxphotos/commit/0aff83ff21c20e293c0b75bacf2863090a0fb725)
+- Started adding tests for CLI [`f0b18c3`](https://github.com/RhetTbull/osxphotos/commit/f0b18c3d29b2141d348be0495013c51c072c6251)
 
 #### [v0.22.0](https://github.com/RhetTbull/osxphotos/compare/v0.21.5...v0.22.0)
 
@@ -354,12 +435,10 @@ Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 - removed old applescript code and files [`1839593`](https://github.com/RhetTbull/osxphotos/commit/18395933a583314d5d992492713752003852e75c)
 - Added test cases and documentation for shared photos and shared albums [`6d20e9e`](https://github.com/RhetTbull/osxphotos/commit/6d20e9e36185aa027d82237cadfe3b55614ba96f)
 - Refactored PhotoInfo to use properties instead of methods--major update [`1ddd90c`](https://github.com/RhetTbull/osxphotos/commit/1ddd90cbdc824afc5df9d2347e730bd9f86350ee)
+- Moved PhotosDB attributes to properties instead of methods [`d95acdf`](https://github.com/RhetTbull/osxphotos/commit/d95acdf9f8764a1720bcba71a6dad29bf668eaf9)
+- changed interface for export, prepped for exiftool_json_sidecar [`1fe8859`](https://github.com/RhetTbull/osxphotos/commit/1fe885962e8a9a420e776bdd3dc640ca143224b2)
 
-#### [v0.15.1](https://github.com/RhetTbull/osxphotos/compare/v0.15.0...v0.15.1)
-
-> 15 May 2020
-
-#### [v0.15.0](https://github.com/RhetTbull/osxphotos/compare/v0.14.21...v0.15.0)
+#### [v0.15.1](https://github.com/RhetTbull/osxphotos/compare/v0.14.21...v0.15.1)
 
 > 14 December 2019
 
@@ -377,6 +456,8 @@ Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 - Added get_db_path and get_library_path to PhotosDB [`1d006a4`](https://github.com/RhetTbull/osxphotos/commit/1d006a4b50ed58b01c6116734bef5f740655a063)
 - Updated PhotosDB.__init__() to accept positional or named arg for dbfile and added associated tests [`9118043`](https://github.com/RhetTbull/osxphotos/commit/911804317b98bf485a39b8588c772be14314aa51)
 - Updated album code in process_database4 and process_database5 to use album uuid [`1cf3e4b`](https://github.com/RhetTbull/osxphotos/commit/1cf3e4b9540c15f8bda2545deb183912bcda40a7)
+- Updated get_db_version and associated tests [`eb563ad`](https://github.com/RhetTbull/osxphotos/commit/eb563ad29738f29f3514ebfb4747baa2dc5356be)
+- Added external_edit for Photos 5 [`42baa29`](https://github.com/RhetTbull/osxphotos/commit/42baa29c18fe2ff16e4d684f87ef7a85993898c1)
 
 #### [v0.14.8](https://github.com/RhetTbull/osxphotos/compare/v0.14.6...v0.14.8)
 

README.md

@@ -203,6 +203,8 @@ Options:
                                   w/o TZ).
   --update                        Only export new or updated files. See notes
                                   below on export and --update.
+  --dry-run                       Dry run (test) the export but don't actually
+                                  export any files; most useful with --verbose
   --export-as-hardlink            Hardlink files instead of copying them.
                                   Cannot be used with --exiftool which creates
                                   copies of the files with embedded EXIF data.
@@ -280,6 +282,19 @@ Options:
                                   output directory in the form
                                   '{name,DEFAULT}'. See below for additional
                                   details on templating system.
+  --filename FILENAME             Optional template for specifying name of
+                                  output file in the form '{name,DEFAULT}'.
+                                  File extension will be added automatically--
+                                  do not include an extension in the FILENAME
+                                  template. See below for additional details
+                                  on templating system.
+  --edited-suffix SUFFIX          Optional suffix for naming edited photos.
+                                  Default name for edited photos is in form
+                                  'photoname_edited.ext'. For example, with '
+                                  --edited-suffix _bearbeiten', the edited
+                                  photo would be named
+                                  'photoname_bearbeiten.ext'.  The default
+                                  suffix is '_edited'.
   --no-extended-attributes        Don't copy extended attributes when
                                   exporting.  You only need this if exporting
                                   to a filesystem that doesn't support Mac OS
@@ -317,13 +332,16 @@ option to re-export the entire library thus rebuilding the
 
 ** Templating System **
 
-With the --directory option you may specify a template for the export
-directory.  This directory will be appended to the export path specified in
-the export DEST argument to export.  For example, if template is
-'{created.year}/{created.month}', and export desitnation DEST is
-'/Users/maria/Pictures/export', the actual export directory for a photo would
-be '/Users/maria/Pictures/export/2020/March' if the photo was created in March
-2020.
+With the --directory and --filename options you may specify a template for the
+export directory or filename, respectively. The directory will be appended to
+the export path specified in the export DEST argument to export.  For example,
+if template is '{created.year}/{created.month}', and export desitnation DEST
+is '/Users/maria/Pictures/export', the actual export directory for a photo
+would be '/Users/maria/Pictures/export/2020/March' if the photo was created in
+March 2020. Some template substitutions may result in more than one value, for
+example '{album}' if photo is in more than one album or '{keyword}' if photo
+has more than one keyword. In this case, more than one copy of the photo will
+be exported, each in a separate directory or with a different filename.
 
 The templating system may also be used with the --keyword-template option to
 set keywords on export (with --exiftool or --sidecar), for example, to set a
@@ -349,9 +367,7 @@ contain a brace symbol ('{' or '}').
 
 If you do not specify a default value and the template substitution has no
 value, '_' (underscore) will be used as the default value. For example, in the
-above example, this would result in '2020/_/photoname.jpg' if address was null
-I plan to eventually extend the templating system to the exported filename so
-you can specify the filename using a template.
+above example, this would result in '2020/_/photoname.jpg' if address was null.
 
 Substitution                    Description
 {name}                          Current filename of the photo
@@ -369,6 +385,10 @@ Substitution                    Description
                                 creation time
 {created.mon}                   Month abbreviation in the user's locale of
                                 the file creation time
+{created.dd}                    2-digit day of the month (zero padded) of
+                                file creation time
+{created.dow}                   Day of week in user's locale of the file
+                                creation time
 {created.doy}                   3-digit day of year (e.g Julian day) of file
                                 creation time, starting from 1 (zero padded)
 {modified.date}                 Photo's modification date in ISO format,
@@ -381,6 +401,8 @@ Substitution                    Description
                                 modification time
 {modified.mon}                  Month abbreviation in the user's locale of
                                 the file modification time
+{modified.dd}                   2-digit day of the month (zero padded) of
+                                the file modification time
 {modified.doy}                  3-digit day of year (e.g Julian day) of file
                                 modification time, starting from 1 (zero
                                 padded)
@@ -1153,9 +1175,9 @@ If overwrite=False and increment=False, export will fail if destination file alr
 
 #### <a name="rendertemplate">`render_template()`</a>
 
-`render_template(template, none_str = "_", path_sep = None)`
+`render_template(template_str, none_str = "_", path_sep = None)`
 Render template string for photo.  none_str is used if template substitution results in None value and no default specified. 
-- `template`: str in form "{name,DEFAULT}" where name is one of the values in table below. The "," and default value that follows are optional. If specified, "DEFAULT" will be used if "name" is None.  This is useful for values which are not always present, for example reverse geolocation data.
+- `template_str`: str in form "{name,DEFAULT}" where name is one of the values in table below. The "," and default value that follows are optional. If specified, "DEFAULT" will be used if "name" is None.  This is useful for values which are not always present, for example reverse geolocation data.
 - `none_str`: optional str to use as substitution when template value is None and no default specified in the template string.  default is "_".
 - `path_sep`: optional character to use as path separator, default is os.path.sep
 
@@ -1369,6 +1391,7 @@ The following substitutions are availabe for use with `PhotoInfo.render_template
 |{created.mm}|2-digit month of the file creation time (zero padded)|
 |{created.month}|Month name in user's locale of the file creation time|
 |{created.mon}|Month abbreviation in the user's locale of the file creation time|
+|{created.dd}|2-digit day of the month (zero padded) of file creation time|
 |{created.doy}|3-digit day of year (e.g Julian day) of file creation time, starting from 1 (zero padded)|
 |{modified.date}|Photo's modification date in ISO format, e.g. '2020-03-22'|
 |{modified.year}|4-digit year of file modification time|
@@ -1376,6 +1399,7 @@ The following substitutions are availabe for use with `PhotoInfo.render_template
 |{modified.mm}|2-digit month of the file modification time (zero padded)|
 |{modified.month}|Month name in user's locale of the file modification time|
 |{modified.mon}|Month abbreviation in the user's locale of the file modification time|
+|{modified.dd}|2-digit day of the month (zero padded) of the file modification time|
 |{modified.doy}|3-digit day of year (e.g Julian day) of file modification time, starting from 1 (zero padded)|
 |{place.name}|Place name from the photo's reverse geolocation data, as displayed in Photos|
 |{place.country_code}|The ISO country code from the photo's reverse geolocation data|
@@ -1421,11 +1445,6 @@ Convert latitude, longitude in degrees to degrees, minutes, seconds as string.
 returns: string tuple in format ("51 deg 30' 12.86\\" N", "0 deg 7' 54.50\\" W")
 This is the same format used by exiftool's json format.
 
-#### `create_path_by_date(dest, dt)`
-Creates a path in dest folder in form dest/YYYY/MM/DD/
-- `dest`: valid path as str
-- `dt`: datetime.timetuple() object
-Checks to see if path exists, if it does, do nothing and return path. If path does not exist, creates it and returns path.  Useful for exporting photos to a date-based folder structure.
 
 ## Examples
 

examples/photos_repl.py

@@ -20,11 +20,7 @@ from osxphotos.__main__ import get_photos_db, _list_libraries
 def main():
     db = None
 
-    if len(sys.argv) > 1:
-        db = sys.argv[1]
-    else:
-        db = get_photos_db()
-
+    db = sys.argv[1] if len(sys.argv) > 1 else get_photos_db()
     if db:
         print("loading database")
         tic = time.perf_counter()

osxphotos/__main__.py

@@ -1,10 +1,13 @@
+""" command line interface for osxphotos """
 import csv
 import datetime
+import functools
 import json
 import logging
 import os
 import os.path
 import pathlib
+import pprint
 import sys
 import time
 
@@ -13,22 +16,20 @@ import yaml
 from pathvalidate import (
     is_valid_filename,
     is_valid_filepath,
-    sanitize_filepath,
     sanitize_filename,
+    sanitize_filepath,
 )
 
 import osxphotos
 
 from ._constants import _EXIF_TOOL_URL, _PHOTOS_4_VERSION, _UNKNOWN_PLACE
+from ._export_db import ExportDB, ExportDBInMemory
 from ._version import __version__
+from .datetime_formatter import DateTimeFormatter
 from .exiftool import get_exiftool_path
+from .fileutil import FileUtil, FileUtilNoOp
 from .photoinfo import ExportResults
-from .photoinfo.template import (
-    TEMPLATE_SUBSTITUTIONS,
-    TEMPLATE_SUBSTITUTIONS_MULTI_VALUED,
-)
-from .utils import _copy_file, create_path_by_date
-from ._export_db import ExportDB
+from .phototemplate import TEMPLATE_SUBSTITUTIONS, TEMPLATE_SUBSTITUTIONS_MULTI_VALUED
 
 # global variable to control verbose output
 # set via --verbose/-V
@@ -111,9 +112,11 @@ class ExportCommand(click.Command):
             + "should be treated as a backup, not a working copy where you intend to make changes. "
         )
         formatter.write("\n")
-        formatter.write_text("Note: The number of files reported for export and the number actually exported "
-            +"may differ due to live photos, associated RAW images, and edited photos which are reported "
-            +"in the total photos exported.")
+        formatter.write_text(
+            "Note: The number of files reported for export and the number actually exported "
+            + "may differ due to live photos, associated RAW images, and edited photos which are reported "
+            + "in the total photos exported."
+        )
         formatter.write("\n")
         formatter.write_text(
             "Implementation note: To determine which files need to be updated, "
@@ -129,13 +132,18 @@ class ExportCommand(click.Command):
         formatter.write_text("** Templating System **")
         formatter.write("\n")
         formatter.write_text(
-            "With the --directory option you may specify a template for the "
-            + "export directory.  This directory will be appended to the export path specified "
+            "With the --directory and --filename options you may specify a template for the "
+            + "export directory or filename, respectively. "
+            + "The directory will be appended to the export path specified "
             + "in the export DEST argument to export.  For example, if template is "
             + "'{created.year}/{created.month}', and export desitnation DEST is "
             + "'/Users/maria/Pictures/export', "
             + "the actual export directory for a photo would be '/Users/maria/Pictures/export/2020/March' "
             + "if the photo was created in March 2020. "
+            + "Some template substitutions may result in more than one value, for example '{album}' if "
+            + "photo is in more than one album or '{keyword}' if photo has more than one keyword. "
+            + "In this case, more than one copy of the photo will be exported, each in a separate directory "
+            + "or with a different filename."
         )
         formatter.write("\n")
         formatter.write_text(
@@ -171,11 +179,7 @@ class ExportCommand(click.Command):
         formatter.write_text(
             "If you do not specify a default value and the template substitution "
             + "has no value, '_' (underscore) will be used as the default value. For example, in the "
-            + "above example, this would result in '2020/_/photoname.jpg' if address was null"
-        )
-        formatter.write_text(
-            "I plan to eventually extend the templating system "
-            + "to the exported filename so you can specify the filename using a template."
+            + "above example, this would result in '2020/_/photoname.jpg' if address was null."
         )
 
         formatter.write("\n")
@@ -439,6 +443,81 @@ def cli(ctx, db, json_, debug):
     ctx.obj = CLI_Obj(db=db, json=json_, debug=debug)
 
 
+@cli.command(hidden=True)
+@DB_OPTION
+@DB_ARGUMENT
+@click.option(
+    "--dump",
+    metavar="ATTR",
+    help="Name of PhotosDB attribute to print; "
+    + "can also use albums, persons, keywords, photos to dump related attributes.",
+    multiple=True,
+)
+@click.option(
+    "--uuid",
+    metavar="UUID",
+    help="Use with '--dump photos' to dump only certain UUIDs",
+    multiple=True,
+)
+@click.pass_obj
+@click.pass_context
+def debug_dump(ctx, cli_obj, db, photos_library, dump, uuid):
+    """ Print out debug info """
+    db = get_photos_db(*photos_library, db, cli_obj.db)
+    if db is None:
+        click.echo(cli.commands["debug-dump"].get_help(ctx), err=True)
+        click.echo("\n\nLocated the following Photos library databases: ", err=True)
+        _list_libraries()
+        return
+
+    start_t = time.perf_counter()
+    print(f"Opening database: {db}")
+    photosdb = osxphotos.PhotosDB(dbfile=db)
+    stop_t = time.perf_counter()
+    print(f"Done; took {(stop_t-start_t):.2f} seconds")
+
+    for attr in dump:
+        if attr == "albums":
+            print("_dbalbums_album:")
+            pprint.pprint(photosdb._dbalbums_album)
+            print("_dbalbums_uuid:")
+            pprint.pprint(photosdb._dbalbums_uuid)
+            print("_dbalbum_details:")
+            pprint.pprint(photosdb._dbalbum_details)
+            print("_dbalbum_folders:")
+            pprint.pprint(photosdb._dbalbum_folders)
+            print("_dbfolder_details:")
+            pprint.pprint(photosdb._dbfolder_details)
+        elif attr == "keywords":
+            print("_dbkeywords_keyword:")
+            pprint.pprint(photosdb._dbkeywords_keyword)
+            print("_dbkeywords_uuid:")
+            pprint.pprint(photosdb._dbkeywords_uuid)
+        elif attr == "persons":
+            print("_dbfaces_person:")
+            pprint.pprint(photosdb._dbfaces_person)
+            print("_dbfaces_uuid:")
+            pprint.pprint(photosdb._dbfaces_uuid)
+        elif attr == "photos":
+            if uuid:
+                for uuid_ in uuid:
+                    print(f"_dbphotos['{uuid_}']:")
+                    try:
+                        pprint.pprint(photosdb._dbphotos[uuid_])
+                    except KeyError:
+                        print(f"Did not find uuid {uuid_} in _dbphotos")
+            else:
+                print("_dbphotos:")
+                pprint.pprint(photosdb._dbphotos)
+        else:
+            try:
+                val = getattr(photosdb, attr)
+                print(f"{attr}:")
+                pprint.pprint(val)
+            except:
+                print(f"Did not find attribute {attr} in PhotosDB")
+
+
 @cli.command()
 @DB_OPTION
 @JSON_OPTION
@@ -530,43 +609,40 @@ def info(ctx, cli_obj, db, json_, photos_library):
         _list_libraries()
         return
 
-    pdb = osxphotos.PhotosDB(dbfile=db)
-    info = {}
-    info["database_path"] = pdb.db_path
-    info["database_version"] = pdb.db_version
-
-    photos = pdb.photos()
+    photosdb = osxphotos.PhotosDB(dbfile=db)
+    info = {"database_path": photosdb.db_path, "database_version": photosdb.db_version}
+    photos = photosdb.photos()
     not_shared_photos = [p for p in photos if not p.shared]
     info["photo_count"] = len(not_shared_photos)
 
     hidden = [p for p in photos if p.hidden]
     info["hidden_photo_count"] = len(hidden)
 
-    movies = pdb.photos(images=False, movies=True)
+    movies = photosdb.photos(images=False, movies=True)
     not_shared_movies = [p for p in movies if not p.shared]
     info["movie_count"] = len(not_shared_movies)
 
-    if pdb.db_version > _PHOTOS_4_VERSION:
+    if photosdb.db_version > _PHOTOS_4_VERSION:
         shared_photos = [p for p in photos if p.shared]
         info["shared_photo_count"] = len(shared_photos)
 
         shared_movies = [p for p in movies if p.shared]
         info["shared_movie_count"] = len(shared_movies)
 
-    keywords = pdb.keywords_as_dict
+    keywords = photosdb.keywords_as_dict
     info["keywords_count"] = len(keywords)
     info["keywords"] = keywords
 
-    albums = pdb.albums_as_dict
+    albums = photosdb.albums_as_dict
     info["albums_count"] = len(albums)
     info["albums"] = albums
 
-    if pdb.db_version > _PHOTOS_4_VERSION:
-        albums_shared = pdb.albums_shared_as_dict
+    if photosdb.db_version > _PHOTOS_4_VERSION:
+        albums_shared = photosdb.albums_shared_as_dict
         info["shared_albums_count"] = len(albums_shared)
         info["shared_albums"] = albums_shared
 
-    persons = pdb.persons_as_dict
+    persons = photosdb.persons_as_dict
 
     info["persons_count"] = len(persons)
     info["persons"] = persons
@@ -643,8 +719,8 @@ def dump(ctx, cli_obj, db, json_, photos_library):
         _list_libraries()
         return
 
-    pdb = osxphotos.PhotosDB(dbfile=db)
-    photos = pdb.photos(movies=True)
+    photosdb = osxphotos.PhotosDB(dbfile=db)
+    photos = photosdb.photos(movies=True)
     print_photo_info(photos, json_ or cli_obj.json)
 
 
@@ -916,6 +992,11 @@ def query(
     is_flag=True,
     help="Only export new or updated files. See notes below on export and --update.",
 )
+@click.option(
+    "--dry-run",
+    is_flag=True,
+    help="Dry run (test) the export but don't actually export any files; most useful with --verbose",
+)
 @click.option(
     "--export-as-hardlink",
     is_flag=True,
@@ -1028,6 +1109,23 @@ def query(
     help="Optional template for specifying name of output directory in the form '{name,DEFAULT}'. "
     "See below for additional details on templating system.",
 )
+@click.option(
+    "--filename",
+    "filename_template",
+    metavar="FILENAME",
+    default=None,
+    help="Optional template for specifying name of output file in the form '{name,DEFAULT}'. "
+    "File extension will be added automatically--do not include an extension in the FILENAME template. "
+    "See below for additional details on templating system.",
+)
+@click.option(
+    "--edited-suffix",
+    metavar="SUFFIX",
+    default="_edited",
+    help="Optional suffix for naming edited photos.  Default name for edited photos is in form "
+    "'photoname_edited.ext'. For example, with '--edited-suffix _bearbeiten', the edited photo "
+    "would be named 'photoname_bearbeiten.ext'.  The default suffix is '_edited'.",
+)
 @click.option(
     "--no-extended-attributes",
     is_flag=True,
@@ -1068,6 +1166,7 @@ def export(
     to_date,
     verbose_,
     update,
+    dry_run,
     export_as_hardlink,
     overwrite,
     export_by_date,
@@ -1105,6 +1204,8 @@ def export(
     not_panorama,
     has_raw,
     directory,
+    filename_template,
+    edited_suffix,
     place,
     no_place,
     no_extended_attributes,
@@ -1147,7 +1248,7 @@ def export(
         (export_as_hardlink, exiftool),
         (any(place), no_place),
     ]
-    if any([all(bb) for bb in exclusive]):
+    if any(all(bb) for bb in exclusive):
         click.echo("Incompatible export options", err=True)
         click.echo(cli.commands["export"].get_help(ctx), err=True)
         return
@@ -1158,11 +1259,6 @@ def export(
         not x for x in [skip_edited, skip_bursts, skip_live, skip_raw]
     ]
 
-    # though the command line option is current_name, internally all processing
-    # logic uses original_name which is the boolean inverse of current_name
-    # because the original code used --original-name as an option
-    original_name = not current_name
-
     # verify exiftool installed an in path
     if exiftool:
         try:
@@ -1190,8 +1286,15 @@ def export(
         _list_libraries()
         return
 
-    # open export database
-    export_db = ExportDB(os.path.join(dest, OSXPHOTOS_EXPORT_DB))
+    # open export database and assign copy/link/unlink functions
+    if dry_run:
+        export_db = ExportDBInMemory(os.path.join(dest, OSXPHOTOS_EXPORT_DB))
+        # echo = functools.partial(click.echo, err=True)
+        # fileutil = FileUtilNoOp(verbose=echo)
+        fileutil = FileUtilNoOp
+    else:
+        export_db = ExportDB(os.path.join(dest, OSXPHOTOS_EXPORT_DB))
+        fileutil = FileUtil
 
     photos = _query(
         db=db,
@@ -1248,10 +1351,6 @@ def export(
     )
 
     results_exported = []
-    results_new = []
-    results_updated = []
-    results_skipped = []
-    results_exif_updated = []
     if photos:
         if export_bursts:
             # add the burst_photos to the export set
@@ -1261,63 +1360,46 @@ def export(
                 photos.extend(burst_set)
 
         num_photos = len(photos)
+        # TODO: photos or photo appears several times, pull into a separate function
         photo_str = "photos" if num_photos > 1 else "photo"
         click.echo(f"Exporting {num_photos} {photo_str} to {dest}...")
         start_time = time.perf_counter()
-        if not verbose_:
-            # show progress bar
-            with click.progressbar(photos) as bar:
-                for p in bar:
-                    results = export_photo(
-                        p,
-                        dest,
-                        verbose_,
-                        export_by_date,
-                        sidecar,
-                        update,
-                        export_as_hardlink,
-                        overwrite,
-                        export_edited,
-                        original_name,
-                        export_live,
-                        download_missing,
-                        exiftool,
-                        directory,
-                        no_extended_attributes,
-                        export_raw,
-                        album_keyword,
-                        person_keyword,
-                        keyword_template,
-                        export_db,
-                    )
-                    results_exported.extend(results.exported)
-                    results_new.extend(results.new)
-                    results_updated.extend(results.updated)
-                    results_skipped.extend(results.skipped)
-                    results_exif_updated.extend(results.exif_updated)
-        else:
+        # though the command line option is current_name, internally all processing
+        # logic uses original_name which is the boolean inverse of current_name
+        # because the original code used --original-name as an option
+        original_name = not current_name
+
+        results_new = []
+        results_updated = []
+        results_skipped = []
+        results_exif_updated = []
+        if verbose_:
             for p in photos:
                 results = export_photo(
-                    p,
-                    dest,
-                    verbose_,
-                    export_by_date,
-                    sidecar,
-                    update,
-                    export_as_hardlink,
-                    overwrite,
-                    export_edited,
-                    original_name,
-                    export_live,
-                    download_missing,
-                    exiftool,
-                    directory,
-                    no_extended_attributes,
-                    export_raw,
-                    album_keyword,
-                    person_keyword,
-                    keyword_template,
-                    export_db,
+                    photo=p,
+                    dest=dest,
+                    verbose_=verbose_,
+                    export_by_date=export_by_date,
+                    sidecar=sidecar,
+                    update=update,
+                    export_as_hardlink=export_as_hardlink,
+                    overwrite=overwrite,
+                    export_edited=export_edited,
+                    original_name=original_name,
+                    export_live=export_live,
+                    download_missing=download_missing,
+                    exiftool=exiftool,
+                    directory=directory,
+                    filename_template=filename_template,
+                    no_extended_attributes=no_extended_attributes,
+                    export_raw=export_raw,
+                    album_keyword=album_keyword,
+                    person_keyword=person_keyword,
+                    keyword_template=keyword_template,
+                    export_db=export_db,
+                    fileutil=fileutil,
+                    dry_run=dry_run,
+                    edited_suffix=edited_suffix,
                 )
                 results_exported.extend(results.exported)
                 results_new.extend(results.new)
@@ -1325,13 +1407,44 @@ def export(
                 results_skipped.extend(results.skipped)
                 results_exif_updated.extend(results.exif_updated)
 
+        else:
+            # show progress bar
+            with click.progressbar(photos) as bar:
+                for p in bar:
+                    results = export_photo(
+                        photo=p,
+                        dest=dest,
+                        verbose_=verbose_,
+                        export_by_date=export_by_date,
+                        sidecar=sidecar,
+                        update=update,
+                        export_as_hardlink=export_as_hardlink,
+                        overwrite=overwrite,
+                        export_edited=export_edited,
+                        original_name=original_name,
+                        export_live=export_live,
+                        download_missing=download_missing,
+                        exiftool=exiftool,
+                        directory=directory,
+                        filename_template=filename_template,
+                        no_extended_attributes=no_extended_attributes,
+                        export_raw=export_raw,
+                        album_keyword=album_keyword,
+                        person_keyword=person_keyword,
+                        keyword_template=keyword_template,
+                        export_db=export_db,
+                        fileutil=fileutil,
+                        dry_run=dry_run,
+                        edited_suffix=edited_suffix,
+                    )
+                    results_exported.extend(results.exported)
+                    results_new.extend(results.new)
+                    results_updated.extend(results.updated)
+                    results_skipped.extend(results.skipped)
+                    results_exif_updated.extend(results.exif_updated)
         stop_time = time.perf_counter()
         # print summary results
-        if not update:
-            photo_str = "photos" if len(results_exported) != 1 else "photo"
-            click.echo(f"Exported: {len(results_exported)} {photo_str}")
-            click.echo(f"Elapsed time: {stop_time-start_time} seconds")
-        else:
+        if update:
             photo_str_new = "photos" if len(results_new) != 1 else "photo"
             photo_str_updated = "photos" if len(results_new) != 1 else "photo"
             photo_str_skipped = "photos" if len(results_skipped) != 1 else "photo"
@@ -1344,8 +1457,10 @@ def export(
                 + f"skipped: {len(results_skipped)} {photo_str_skipped}, "
                 + f"updated EXIF data: {len(results_exif_updated)} {photo_str_exif_updated}"
             )
-            click.echo(f"Elapsed time: {stop_time-start_time} seconds")
-
+        else:
+            photo_str = "photos" if len(results_exported) != 1 else "photo"
+            click.echo(f"Exported: {len(results_exported)} {photo_str}")
+        click.echo(f"Elapsed time: {stop_time-start_time} seconds")
     else:
         click.echo("Did not find any photos to export")
 
@@ -1368,8 +1483,8 @@ def help(ctx, topic, **kw):
 
 
 def print_photo_info(photos, json=False):
+    dump = []
     if json:
-        dump = []
         for p in photos:
             dump.append(p.json())
         click.echo(f"[{', '.join(dump)}]")
@@ -1378,7 +1493,6 @@ def print_photo_info(photos, json=False):
         csv_writer = csv.writer(
             sys.stdout, delimiter=",", quotechar='"', quoting=csv.QUOTE_MINIMAL
         )
-        dump = []
         # add headers
         dump.append(
             [
@@ -1715,28 +1829,34 @@ def _query(
 
 
 def export_photo(
-    photo,
-    dest,
-    verbose_,
-    export_by_date,
-    sidecar,
-    update,
-    export_as_hardlink,
-    overwrite,
-    export_edited,
-    original_name,
-    export_live,
-    download_missing,
-    exiftool,
-    directory,
-    no_extended_attributes,
-    export_raw,
-    album_keyword,
-    person_keyword,
-    keyword_template,
-    export_db,
+    photo=None,
+    dest=None,
+    verbose_=None,
+    export_by_date=None,
+    sidecar=None,
+    update=None,
+    export_as_hardlink=None,
+    overwrite=None,
+    export_edited=None,
+    original_name=None,
+    export_live=None,
+    download_missing=None,
+    exiftool=None,
+    directory=None,
+    filename_template=None,
+    no_extended_attributes=None,
+    export_raw=None,
+    album_keyword=None,
+    person_keyword=None,
+    keyword_template=None,
+    export_db=None,
+    fileutil=FileUtil,
+    dry_run=None,
+    edited_suffix="_edited",
 ):
     """ Helper function for export that does the actual export
+
+    Args:
         photo: PhotoInfo object
         dest: destination path as string
         verbose_: boolean; print verbose output
@@ -1750,20 +1870,25 @@ def export_photo(
         download_missing: attempt download of missing iCloud photos
         exiftool: use exiftool to write EXIF metadata directly to exported photo
         directory: template used to determine output directory
+        filename_template: template use to determine output file
         no_extended_attributes: boolean; if True, exports photo without preserving extended attributes
         export_raw: boolean; if True exports RAW image associate with the photo
         album_keyword: boolean; if True, exports album names as keywords in metadata
         person_keyword: boolean; if True, exports person names as keywords in metadata
         keyword_template: list of strings; if provided use rendered template strings as keywords
-        returns list of path(s) of exported photo or None if photo was missing 
+        export_db: export database instance compatible with ExportDB_ABC
+        fileutil: file util class compatible with FileUtilABC
+        dry_run: boolean; if True, doesn't actually export or update any files
+
+    Returns:
+        list of path(s) of exported photo or None if photo was missing
+    
+    Raises:
+        ValueError on invalid filename_template
     """
     global VERBOSE
     VERBOSE = True if verbose_ else False
 
-    # Can export to multiple paths
-    # Start with single path [dest] but direcotry and export_by_date will modify dest_paths
-    dest_paths = [dest]
-
     if not download_missing:
         if photo.ismissing:
             space = " " if not verbose_ else ""
@@ -1782,143 +1907,125 @@ def export_photo(
         )
         return ExportResults([], [], [], [], [])
 
-    filename = None
-    if original_name:
-        filename = photo.original_filename
-    else:
-        filename = photo.filename
+    filenames = get_filenames_from_template(photo, filename_template, original_name)
+    for filename in filenames:
+        verbose(f"Exporting {photo.filename} as {filename}")
 
-    verbose(f"Exporting {photo.filename} as {filename}")
-
-    if export_by_date:
-        date_created = photo.date.timetuple()
-        dest_path = create_path_by_date(dest, date_created)
-        dest_paths = [dest_path]
-    elif directory:
-        # got a directory template, render it and check results are valid
-        dirnames, unmatched = photo.render_template(directory)
-        if unmatched:
-            raise click.BadOptionUsage(
-                "directory",
-                f"Invalid substitution in template '{directory}': {unmatched}",
-            )
-        dest_paths = []
-        for dirname in dirnames:
-            dirname = sanitize_filepath(dirname, platform="auto")
-            dest_path = os.path.join(dest, dirname)
-            if not is_valid_filepath(dest_path, platform="auto"):
-                raise ValueError(f"Invalid file path: '{dest_path}'")
-            if not os.path.isdir(dest_path):
-                os.makedirs(dest_path)
-            dest_paths.append(dest_path)
-
-    sidecar = [s.lower() for s in sidecar]
-    sidecar_json = sidecar_xmp = False
-    if "json" in sidecar:
-        sidecar_json = True
-    if "xmp" in sidecar:
-        sidecar_xmp = True
-
-    # if download_missing and the photo is missing or path doesn't exist,
-    # try to download with Photos
-    use_photos_export = download_missing and (
-        photo.ismissing or not os.path.exists(photo.path)
-    )
-
-    # export the photo to each path in dest_paths
-    results_exported = []
-    results_new = []
-    results_updated = []
-    results_skipped = []
-    results_exif_updated = []
-    for dest_path in dest_paths:
-        export_results = photo.export2(
-            dest_path,
-            filename,
-            sidecar_json=sidecar_json,
-            sidecar_xmp=sidecar_xmp,
-            live_photo=export_live,
-            raw_photo=export_raw,
-            export_as_hardlink=export_as_hardlink,
-            overwrite=overwrite,
-            use_photos_export=use_photos_export,
-            exiftool=exiftool,
-            no_xattr=no_extended_attributes,
-            use_albums_as_keywords=album_keyword,
-            use_persons_as_keywords=person_keyword,
-            keyword_template=keyword_template,
-            update=update,
-            export_db=export_db,
+        dest_paths = get_dirnames_from_template(
+            photo, directory, export_by_date, dest, dry_run
         )
 
-        results_exported.extend(export_results.exported)
-        results_new.extend(export_results.new)
-        results_updated.extend(export_results.updated)
-        results_skipped.extend(export_results.skipped)
-        results_exif_updated.extend(export_results.exif_updated)
+        sidecar = [s.lower() for s in sidecar]
+        sidecar_json = sidecar_xmp = False
+        if "json" in sidecar:
+            sidecar_json = True
+        if "xmp" in sidecar:
+            sidecar_xmp = True
 
-        if verbose_:
-            for exported in export_results.exported:
-                verbose(f"Exported {exported}")
-            for new in export_results.new:
-                verbose(f"Exported new file {new}")
-            for updated in export_results.updated:
-                verbose(f"Exported updated file {updated}")
-            for skipped in export_results.skipped:
-                verbose(f"Skipped up to date file {skipped}")
+        # if download_missing and the photo is missing or path doesn't exist,
+        # try to download with Photos
+        use_photos_export = download_missing and (
+            photo.ismissing or not os.path.exists(photo.path)
+        )
 
-        # if export-edited, also export the edited version
-        # verify the photo has adjustments and valid path to avoid raising an exception
-        if export_edited and photo.hasadjustments:
-            # if download_missing and the photo is missing or path doesn't exist,
-            # try to download with Photos
-            use_photos_export = download_missing and photo.path_edited is None
-            if not download_missing and photo.path_edited is None:
-                verbose(f"Skipping missing edited photo for {filename}")
-            else:
-                edited_name = pathlib.Path(filename)
-                # check for correct edited suffix
-                if photo.path_edited is not None:
-                    edited_suffix = pathlib.Path(photo.path_edited).suffix
+        # export the photo to each path in dest_paths
+        results_exported = []
+        results_new = []
+        results_updated = []
+        results_skipped = []
+        results_exif_updated = []
+        for dest_path in dest_paths:
+            export_results = photo.export2(
+                dest_path,
+                filename,
+                sidecar_json=sidecar_json,
+                sidecar_xmp=sidecar_xmp,
+                live_photo=export_live,
+                raw_photo=export_raw,
+                export_as_hardlink=export_as_hardlink,
+                overwrite=overwrite,
+                use_photos_export=use_photos_export,
+                exiftool=exiftool,
+                no_xattr=no_extended_attributes,
+                use_albums_as_keywords=album_keyword,
+                use_persons_as_keywords=person_keyword,
+                keyword_template=keyword_template,
+                update=update,
+                export_db=export_db,
+                fileutil=fileutil,
+                dry_run=dry_run,
+            )
+
+            results_exported.extend(export_results.exported)
+            results_new.extend(export_results.new)
+            results_updated.extend(export_results.updated)
+            results_skipped.extend(export_results.skipped)
+            results_exif_updated.extend(export_results.exif_updated)
+
+            if verbose_:
+                for exported in export_results.exported:
+                    verbose(f"Exported {exported}")
+                for new in export_results.new:
+                    verbose(f"Exported new file {new}")
+                for updated in export_results.updated:
+                    verbose(f"Exported updated file {updated}")
+                for skipped in export_results.skipped:
+                    verbose(f"Skipped up to date file {skipped}")
+
+            # if export-edited, also export the edited version
+            # verify the photo has adjustments and valid path to avoid raising an exception
+            if export_edited and photo.hasadjustments:
+                # if download_missing and the photo is missing or path doesn't exist,
+                # try to download with Photos
+                use_photos_export = download_missing and photo.path_edited is None
+                if not download_missing and photo.path_edited is None:
+                    verbose(f"Skipping missing edited photo for {filename}")
                 else:
-                    # use filename suffix which might be wrong,
-                    # will be corrected by use_photos_export
-                    edited_suffix = pathlib.Path(photo.filename).suffix
-                edited_name = f"{edited_name.stem}_edited{edited_suffix}"
-                verbose(f"Exporting edited version of {filename} as {edited_name}")
-                export_results_edited = photo.export2(
-                    dest_path,
-                    edited_name,
-                    sidecar_json=sidecar_json,
-                    sidecar_xmp=sidecar_xmp,
-                    export_as_hardlink=export_as_hardlink,
-                    overwrite=overwrite,
-                    edited=True,
-                    use_photos_export=use_photos_export,
-                    exiftool=exiftool,
-                    no_xattr=no_extended_attributes,
-                    use_albums_as_keywords=album_keyword,
-                    use_persons_as_keywords=person_keyword,
-                    keyword_template=keyword_template,
-                    update=update,
-                    export_db=export_db,
-                )
+                    edited_name = pathlib.Path(filename)
+                    # check for correct edited suffix
+                    if photo.path_edited is not None:
+                        edited_ext = pathlib.Path(photo.path_edited).suffix
+                    else:
+                        # use filename suffix which might be wrong,
+                        # will be corrected by use_photos_export
+                        edited_ext = pathlib.Path(photo.filename).suffix
+                    edited_name = f"{edited_name.stem}{edited_suffix}{edited_ext}"
+                    verbose(f"Exporting edited version of {filename} as {edited_name}")
+                    export_results_edited = photo.export2(
+                        dest_path,
+                        edited_name,
+                        sidecar_json=sidecar_json,
+                        sidecar_xmp=sidecar_xmp,
+                        export_as_hardlink=export_as_hardlink,
+                        overwrite=overwrite,
+                        edited=True,
+                        use_photos_export=use_photos_export,
+                        exiftool=exiftool,
+                        no_xattr=no_extended_attributes,
+                        use_albums_as_keywords=album_keyword,
+                        use_persons_as_keywords=person_keyword,
+                        keyword_template=keyword_template,
+                        update=update,
+                        export_db=export_db,
+                        fileutil=fileutil,
+                        dry_run=dry_run,
+                    )
 
-                results_exported.extend(export_results_edited.exported)
-                results_new.extend(export_results_edited.new)
-                results_updated.extend(export_results_edited.updated)
-                results_skipped.extend(export_results_edited.skipped)
-                results_exif_updated.extend(export_results_edited.exif_updated)
+                    results_exported.extend(export_results_edited.exported)
+                    results_new.extend(export_results_edited.new)
+                    results_updated.extend(export_results_edited.updated)
+                    results_skipped.extend(export_results_edited.skipped)
+                    results_exif_updated.extend(export_results_edited.exif_updated)
 
-                if verbose_:
-                    for exported in export_results_edited.exported:
-                        verbose(f"Exported {exported}")
-                    for new in export_results_edited.new:
-                        verbose(f"Exported new file {new}")
-                    for updated in export_results_edited.updated:
-                        verbose(f"Exported updated file {updated}")
-                    for skipped in export_results_edited.skipped:
-                        verbose(f"Skipped up to date file {skipped}")
+                    if verbose_:
+                        for exported in export_results_edited.exported:
+                            verbose(f"Exported {exported}")
+                        for new in export_results_edited.new:
+                            verbose(f"Exported new file {new}")
+                        for updated in export_results_edited.updated:
+                            verbose(f"Exported updated file {updated}")
+                        for skipped in export_results_edited.skipped:
+                            verbose(f"Skipped up to date file {skipped}")
 
     return ExportResults(
         results_exported,
@@ -1929,5 +2036,85 @@ def export_photo(
     )
 
 
+def get_filenames_from_template(photo, filename_template, original_name):
+    """ get list of export filenames for a photo
+
+    Args:
+        photo: a PhotoInfo instance
+        filename_template: a PhotoTemplate template string, may be None
+        original_name: boolean; if True, use photo's original filename instead of current filename
+    
+    Returns:
+        list of filenames
+    
+    Raises:
+        click.BadOptionUsage if template is invalid
+    """
+    if filename_template:
+        photo_ext = pathlib.Path(photo.original_filename).suffix
+        filenames, unmatched = photo.render_template(filename_template, path_sep="_")
+        if not filenames or unmatched:
+            raise click.BadOptionUsage(
+                "filename_template",
+                f"Invalid template '{filename_template}': results={filenames} unmatched={unmatched}",
+            )
+        filenames = [f"{file_}{photo_ext}" for file_ in filenames]
+    else:
+        filenames = [photo.original_filename] if original_name else [photo.filename]
+    return filenames
+
+
+def get_dirnames_from_template(photo, directory, export_by_date, dest, dry_run):
+    """ get list of directories to export a photo into, creates directories if they don't exist
+
+    Args:
+        photo: a PhotoInstance object
+        directory: a PhotoTemplate template string, may be None
+        export_by_date: boolean; if True, creates output directories in form YYYY-MM-DD
+        dest: top-level destination directory
+        dry_run: boolean; if True, runs in dry-run mode and does not create output directories
+
+    Returns:
+        list of export directories
+
+    Raises:
+        click.BadOptionUsage if template is invalid
+    """
+
+    if export_by_date:
+        date_created = DateTimeFormatter(photo.date)
+        dest_path = os.path.join(
+            dest, date_created.year, date_created.mm, date_created.dd
+        )
+        if not (dry_run or os.path.isdir(dest_path)):
+            os.makedirs(dest_path)
+        dest_paths = [dest_path]
+    elif directory:
+        # got a directory template, render it and check results are valid
+        dirnames, unmatched = photo.render_template(directory)
+        if not dirnames:
+            raise click.BadOptionUsage(
+                "directory",
+                f"Invalid template '{directory}': results={dirnames} unmatched={unmatched}",
+            )
+        elif unmatched:
+            raise click.BadOptionUsage(
+                "directory",
+                f"Invalid template '{directory}': results={dirnames} unmatched={unmatched}",
+            )
+        dest_paths = []
+        for dirname in dirnames:
+            dirname = sanitize_filepath(dirname, platform="auto")
+            dest_path = os.path.join(dest, dirname)
+            if not is_valid_filepath(dest_path, platform="auto"):
+                raise ValueError(f"Invalid file path: '{dest_path}'")
+            if not (dry_run or os.path.isdir(dest_path)):
+                os.makedirs(dest_path)
+            dest_paths.append(dest_path)
+    else:
+        dest_paths = [dest]
+    return dest_paths
+
+
 if __name__ == "__main__":
     cli()  # pylint: disable=no-value-for-parameter

osxphotos/_constants.py

@@ -18,7 +18,7 @@ _PHOTOS_3_VERSION = "3301"
 
 # versions 5.0 and later have a different database structure
 _PHOTOS_4_VERSION = "4025"  # latest Mojove version on 10.14.6
-_PHOTOS_5_VERSION = "6000"  # seems to be current on 10.15.1 through 10.15.4
+_PHOTOS_5_VERSION = "6000"  # seems to be current on 10.15.1 through 10.15.5
 
 # which major version operating systems have been tested
 _TESTED_OS_VERSIONS = ["12", "13", "14", "15"]

osxphotos/_export_db.py

@@ -9,6 +9,7 @@ import pathlib
 import sqlite3
 import sys
 from abc import ABC, abstractmethod
+from io import StringIO
 from sqlite3 import Error
 
 from ._version import __version__
@@ -17,6 +18,8 @@ OSXPHOTOS_EXPORTDB_VERSION = "1.0"
 
 
 class ExportDB_ABC(ABC):
+    """ abstract base class for ExportDB """
+
     @abstractmethod
     def get_uuid_for_file(self, filename):
         pass
@@ -58,7 +61,7 @@ class ExportDB_ABC(ABC):
         pass
 
     @abstractmethod
-    def set_data(self, file, uuid, orig_stat, exif_stat, info_json, exif_json):
+    def set_data(self, filename, uuid, orig_stat, exif_stat, info_json, exif_json):
         pass
 
 
@@ -95,7 +98,7 @@ class ExportDBNoOp(ExportDB_ABC):
     def set_exifdata_for_file(self, uuid, exifdata):
         pass
 
-    def set_data(self, file, uuid, orig_stat, exif_stat, info_json, exif_json):
+    def set_data(self, filename, uuid, orig_stat, exif_stat, info_json, exif_json):
         pass
 
 
@@ -302,21 +305,41 @@ class ExportDB(ExportDB_ABC):
 
         logging.debug(f"set_exifdata: {filename}, {exifdata}")
 
-    def set_data(self, file, uuid, orig_stat, exif_stat, info_json, exif_json):
+    def set_data(self, filename, uuid, orig_stat, exif_stat, info_json, exif_json):
         """ sets all the data for file and uuid at once 
-            calls set_uuid_for_file
-                  set_info_for_uuid
-                  set_stat_orig_for_file
-                  set_stat_exif_for_file
-                  set_exifdata_for_file        
         """
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
-
-        self.set_uuid_for_file(filename, uuid)
-        self.set_info_for_uuid(uuid, info_json)
-        self.set_stat_orig_for_file(filename, orig_stat)
-        self.set_stat_exif_for_file(filename, exif_stat)
-        self.set_exifdata_for_file(filename, exif_json)
+        filename = str(pathlib.Path(filename).relative_to(self._path))
+        filename_normalized = filename.lower()
+        conn = self._conn
+        try:
+            c = conn.cursor()
+            c.execute(
+                f"INSERT OR REPLACE INTO files(filepath, filepath_normalized, uuid) VALUES (?, ?, ?);",
+                (filename, filename_normalized, uuid),
+            )
+            c.execute(
+                "UPDATE files "
+                + "SET orig_mode = ?, orig_size = ?, orig_mtime = ? "
+                + "WHERE filepath_normalized = ?;",
+                (*orig_stat, filename_normalized),
+            )
+            c.execute(
+                "UPDATE files "
+                + "SET exif_mode = ?, exif_size = ?, exif_mtime = ? "
+                + "WHERE filepath_normalized = ?;",
+                (*exif_stat, filename_normalized),
+            )
+            c.execute(
+                "INSERT OR REPLACE INTO info(uuid, json_info) VALUES (?, ?);",
+                (uuid, info_json),
+            )
+            c.execute(
+                "INSERT OR REPLACE INTO exifdata(filepath_normalized, json_exifdata) VALUES (?, ?);",
+                (filename_normalized, exif_json),
+            )
+            conn.commit()
+        except Error as e:
+            logging.warning(e)
 
     def close(self):
         """ close the database connection """
@@ -422,10 +445,7 @@ class ExportDB(ExportDB_ABC):
         dt = datetime.datetime.utcnow().isoformat()
         python_path = sys.executable
         cmd = sys.argv[0]
-        if len(sys.argv) > 1:
-            args = " ".join(sys.argv[1:])
-        else:
-            args = ""
+        args = " ".join(sys.argv[1:]) if len(sys.argv) > 1 else ""
         cwd = os.getcwd()
         conn = self._conn
         try:
@@ -437,3 +457,64 @@ class ExportDB(ExportDB_ABC):
             conn.commit()
         except Error as e:
             logging.warning(e)
+
+
+class ExportDBInMemory(ExportDB):
+    """ In memory version of ExportDB
+        Copies the on-disk database into memory so it may be operated on without 
+        modifying the on-disk verison 
+    """
+
+    def init(self, dbfile):
+        self._dbfile = dbfile
+        # _path is parent of the database
+        # all files referenced by get_/set_uuid_for_file will be converted to
+        # relative paths to this parent _path
+        # this allows the entire export tree to be moved to a new disk/location
+        # whilst preserving the UUID to filename mappping
+        self._path = pathlib.Path(dbfile).parent
+        self._conn = self._open_export_db(dbfile)
+        self._insert_run_info()
+
+    def _open_export_db(self, dbfile):
+        """ open export database and return a db connection
+            if dbfile does not exist, will create and initialize the database 
+            returns: connection to the database 
+        """
+        if not os.path.isfile(dbfile):
+            logging.debug(f"dbfile {dbfile} doesn't exist, creating in memory version")
+            conn = self._get_db_connection()
+            if conn:
+                self._create_db_tables(conn)
+            else:
+                raise Exception("Error getting connection to in-memory database")
+        else:
+            logging.debug(f"dbfile {dbfile} exists, opening it and copying to memory")
+            try:
+                conn = sqlite3.connect(dbfile)
+            except Error as e:
+                logging.warning(e)
+                raise e
+
+            tempfile = StringIO()
+            for line in conn.iterdump():
+                tempfile.write("%s\n" % line)
+            conn.close()
+            tempfile.seek(0)
+
+            # Create a database in memory and import from tempfile
+            conn = sqlite3.connect(":memory:")
+            conn.cursor().executescript(tempfile.read())
+            conn.commit()
+
+        return conn
+
+    def _get_db_connection(self):
+        """ return db connection to in memory database """
+        try:
+            conn = sqlite3.connect(":memory:")
+        except Error as e:
+            logging.warning(e)
+            conn = None
+
+        return conn

osxphotos/_filecmp.py

@@ -1,54 +0,0 @@
-"""Utilities for comparing files 
-
-Modified from CPython/Lib/filecmp.py
-
-Functions:
-    cmp_file(f1, s2) -> int
-    file_sig(f1) -> Tuple[int, int, float]
-
-"""
-
-import os
-import stat
-
-__all__ = ["cmp", "sig"]
-
-
-def cmp_file(f1, s2):
-    """Compare file f1 to signature s2.
-
-    Arguments:
-
-    f1 --  File name
-
-    s2 -- stats as returned by sig
-
-    Return value:
-
-    True if the files are the same, False otherwise.
-
-    This function uses a cache for past comparisons and the results,
-    with cache entries invalidated if their stat information
-    changes.  The cache may be cleared by calling clear_cache().
-
-    """
-
-    if not s2:
-        return False
-
-    s1 = _sig(os.stat(f1))
-
-    if s1[0] != stat.S_IFREG or s2[0] != stat.S_IFREG:
-        return False
-    if s1 == s2:
-        return True
-    return False
-
-
-def _sig(st):
-    return (stat.S_IFMT(st.st_mode), st.st_size, st.st_mtime)
-
-
-def file_sig(f1):
-    """ return os.stat signature for file f1 """
-    return _sig(os.stat(f1))

osxphotos/_version.py

@@ -1,3 +1,3 @@
 """ version info """
 
-__version__ = "0.29.0"
+__version__ = "0.29.14"

osxphotos/albuminfo.py

@@ -62,10 +62,7 @@ class AlbumInfo:
         try:
             return self._folder_names
         except AttributeError:
-            if self._db._db_version <= _PHOTOS_4_VERSION:
-                self._folder_names = self._db._album_folder_hierarchy_list(self._uuid)
-            else:
-                self._folder_names = self._db._album_folder_hierarchy_list(self._uuid)
+            self._folder_names = self._db._album_folder_hierarchy_list(self._uuid)
             return self._folder_names
 
     @property

osxphotos/datetime_formatter.py

@@ -12,41 +12,59 @@ class DateTimeFormatter:
     @property
     def date(self):
         """ ISO date in form 2020-03-22 """
-        date = self.dt.date().isoformat()
-        return date
+        return self.dt.date().isoformat()
 
     @property
     def year(self):
         """ 4 digit year """
-        year = f"{self.dt.year}"
-        return year
+        return f"{self.dt.year}"
 
     @property
     def yy(self):
         """ 2 digit year """
-        yy = f"{self.dt.strftime('%y')}"
-        return yy
+        return f"{self.dt.strftime('%y')}"
 
     @property
     def mm(self):
         """ 2 digit month """
-        mm = f"{self.dt.strftime('%m')}"
-        return mm
+        return f"{self.dt.strftime('%m')}"
 
     @property
     def month(self):
         """ Month as locale's full name """
-        month = f"{self.dt.strftime('%B')}"
-        return month
+        return f"{self.dt.strftime('%B')}"
 
     @property
     def mon(self):
         """ Month as locale's abbreviated name """
-        mon = f"{self.dt.strftime('%b')}"
-        return mon
+        return f"{self.dt.strftime('%b')}"
+
+    @property
+    def dd(self):
+        """ 2-digit day of the month """
+        return f"{self.dt.strftime('%d')}"
+
+    @property
+    def dow(self):
+        """ Day of week as locale's name """
+        return f"{self.dt.strftime('%A')}"
 
     @property
     def doy(self):
         """ Julian day of year starting from 001 """
-        doy = f"{self.dt.strftime('%j')}"
-        return doy
+        return f"{self.dt.strftime('%j')}"
+
+    @property
+    def hour(self):
+        """ 2-digit hour """
+        return f"{self.dt.strftime('%H')}"
+
+    @property
+    def min(self):
+        """ 2-digit minute """
+        return f"{self.dt.strftime('%M')}"
+
+    @property
+    def sec(self):
+        """ 2-digit second """
+        return f"{self.dt.strftime('%S')}"

osxphotos/exiftool.py

@@ -59,11 +59,7 @@ class _ExifToolProc:
                 )
             return
 
-        if exiftool:
-            self._exiftool = exiftool
-        else:
-            self._exiftool = get_exiftool_path()
-
+        self._exiftool = exiftool if exiftool else get_exiftool_path()
         self._process_running = False
         self._start_proc()
 
@@ -156,8 +152,7 @@ class ExifTool:
 
         if value is None:
             value = ""
-        command = []
-        command.append(f"-{tag}={value}")
+        command = [f"-{tag}={value}"]
         if self.overwrite:
             command.append("-overwrite_original")
         self.run_commands(*command)
@@ -193,7 +188,7 @@ class ExifTool:
             no_file: (bool) do not pass the filename to exiftool (default=False)
                      by default, all commands will be run against self.file
                      use no_file=True to run a command without passing the filename """
-        if not hasattr(self, "_process") or not self._process:
+        if not (hasattr(self, "_process") and self._process):
             raise ValueError("exiftool process is not running")
 
         if not commands:
@@ -245,8 +240,7 @@ class ExifTool:
 
     def json(self):
         """ returns JSON string containing all EXIF tags and values from exiftool """
-        json_str = self.run_commands("-json")
-        return json_str
+        return self.run_commands("-json")
 
     def _read_exif(self):
         """ read exif data from file """
@@ -254,5 +248,4 @@ class ExifTool:
         self.data = {k: v for k, v in data.items()}
 
     def __str__(self):
-        str_ = f"file: {self.file}\nexiftool: {self._exiftoolproc._exiftool}"
-        return str_
+        return f"file: {self.file}\nexiftool: {self._exiftoolproc._exiftool}"

osxphotos/fileutil.py

@@ -0,0 +1,178 @@
+""" FileUtil class with methods for copy, hardlink, unlink, etc. """
+
+import logging
+import os
+import pathlib
+import stat
+import subprocess
+import sys
+from abc import ABC, abstractmethod
+
+
+class FileUtilABC(ABC):
+    """ Abstract base class for FileUtil """
+
+    @classmethod
+    @abstractmethod
+    def hardlink(cls, src, dest):
+        pass
+
+    @classmethod
+    @abstractmethod
+    def copy(cls, src, dest, norsrc=False):
+        pass
+
+    @classmethod
+    @abstractmethod
+    def unlink(cls, dest):
+        pass
+
+    @classmethod
+    @abstractmethod
+    def cmp_sig(cls, file1, file2):
+        pass
+
+    @classmethod
+    @abstractmethod
+    def file_sig(cls, file1):
+        pass
+
+
+class FileUtilMacOS(FileUtilABC):
+    """ Various file utilities """
+
+    @classmethod
+    def hardlink(cls, src, dest):
+        """ Hardlinks a file from src path to dest path 
+            src: source path as string 
+            dest: destination path as string
+            Raises exception if linking fails or either path is None """
+
+        if src is None or dest is None:
+            raise ValueError("src and dest must not be None", src, dest)
+
+        if not os.path.isfile(src):
+            raise FileNotFoundError("src file does not appear to exist", src)
+
+        # if error on copy, subprocess will raise CalledProcessError
+        try:
+            os.link(src, dest)
+        except Exception as e:
+            logging.critical(f"os.link returned error: {e}")
+            raise e
+
+    @classmethod
+    def copy(cls, src, dest, norsrc=False):
+        """ Copies a file from src path to dest path 
+            src: source path as string 
+            dest: destination path as string
+            norsrc: (bool) if True, uses --norsrc flag with ditto so it will not copy
+                    resource fork or extended attributes.  May be useful on volumes that
+                    don't work with extended attributes (likely only certain SMB mounts)
+                    default is False
+            Uses ditto to perform copy; will silently overwrite dest if it exists
+            Raises exception if copy fails or either path is None """
+
+        if src is None or dest is None:
+            raise ValueError("src and dest must not be None", src, dest)
+
+        if not os.path.isfile(src):
+            raise FileNotFoundError("src file does not appear to exist", src)
+
+        if norsrc:
+            command = ["/usr/bin/ditto", "--norsrc", src, dest]
+        else:
+            command = ["/usr/bin/ditto", src, dest]
+
+        # if error on copy, subprocess will raise CalledProcessError
+        try:
+            result = subprocess.run(command, check=True, stderr=subprocess.PIPE)
+        except subprocess.CalledProcessError as e:
+            logging.critical(
+                f"ditto returned error: {e.returncode} {e.stderr.decode(sys.getfilesystemencoding()).rstrip()}"
+            )
+            raise e
+
+        return result.returncode
+
+    @classmethod
+    def unlink(cls, filepath):
+        """ unlink filepath; if it's pathlib.Path, use Path.unlink, otherwise use os.unlink """
+        if isinstance(filepath, pathlib.Path):
+            filepath.unlink()
+        else:
+            os.unlink(filepath)
+
+    @classmethod
+    def cmp_sig(cls, f1, s2):
+        """Compare file f1 to signature s2.
+        Arguments:
+        f1 --  File name
+        s2 -- stats as returned by sig
+
+        Return value:
+        True if the files are the same, False otherwise.
+        """
+
+        if not s2:
+            return False
+
+        s1 = cls._sig(os.stat(f1))
+
+        if s1[0] != stat.S_IFREG or s2[0] != stat.S_IFREG:
+            return False
+        return s1 == s2
+
+    @classmethod
+    def file_sig(cls, f1):
+        """ return os.stat signature for file f1 """
+        return cls._sig(os.stat(f1))
+
+    @staticmethod
+    def _sig(st):
+        return (stat.S_IFMT(st.st_mode), st.st_size, st.st_mtime)
+
+
+class FileUtil(FileUtilMacOS):
+    """ Various file utilities """
+
+    pass
+
+
+class FileUtilNoOp(FileUtil):
+    """ No-Op implementation of FileUtil for testing / dry-run mode
+        all methods with exception of cmp_sig and file_cmp are no-op
+        cmp_sig functions as FileUtil.cmp_sig does
+        file_cmp returns mock data
+    """
+
+    @staticmethod
+    def noop(*args):
+        pass
+
+    verbose = noop
+
+    def __new__(cls, verbose=None):
+        if verbose:
+            if callable(verbose):
+                cls.verbose = verbose
+            else:
+                raise ValueError(f"verbose {verbose} not callable")
+        return super(FileUtilNoOp, cls).__new__(cls)
+
+    @classmethod
+    def hardlink(cls, src, dest):
+        cls.verbose(f"hardlink: {src} {dest}")
+
+    @classmethod
+    def copy(cls, src, dest, norsrc=False):
+        cls.verbose(f"copy: {src} {dest}")
+
+    @classmethod
+    def unlink(cls, dest):
+        cls.verbose(f"unlink: {dest}")
+
+    @classmethod
+    def file_sig(cls, file1):
+        cls.verbose(f"file_sig: {file1}")
+        return (42, 42, 42)

osxphotos/photoinfo/_photoinfo_export.py

@@ -1,4 +1,13 @@
-""" export methods for PhotoInfo """
+""" Export methods for PhotoInfo 
+    The following methods are defined and must be imported into PhotoInfo as instance methods:
+    export
+    export2
+    _export_photo
+    _write_exif_data
+    _exiftool_json_sidecar
+    _xmp_sidecar
+    _write_sidecar
+    """
 
 # TODO: should this be its own PhotoExporter class?
 
@@ -9,10 +18,12 @@ import logging
 import os
 import pathlib
 import re
+import tempfile
 from collections import namedtuple  # pylint: disable=syntax-error
 
 from mako.template import Template
 
+from .._applescript import AppleScript
 from .._constants import (
     _MAX_IPTC_KEYWORD_LEN,
     _OSXPHOTOS_NONE_SENTINEL,
@@ -20,21 +31,171 @@ from .._constants import (
     _UNKNOWN_PERSON,
     _XMP_TEMPLATE_NAME,
 )
-from ..exiftool import ExifTool
 from .._export_db import ExportDBNoOp
-from .._filecmp import cmp_file, file_sig
-from ..utils import (
-    _copy_file,
-    _export_photo_uuid_applescript,
-    _hardlink_file,
-    dd_to_dms_str,
-)
+from ..exiftool import ExifTool
+from ..fileutil import FileUtil
+from ..utils import dd_to_dms_str
 
 ExportResults = namedtuple(
     "ExportResults", ["exported", "new", "updated", "skipped", "exif_updated"]
 )
 
 
+# _export_photo_uuid_applescript is not a class method, don't import this into PhotoInfo
+def _export_photo_uuid_applescript(
+    uuid,
+    dest,
+    filestem=None,
+    original=True,
+    edited=False,
+    live_photo=False,
+    timeout=120,
+    burst=False,
+    dry_run=False,
+):
+    """ Export photo to dest path using applescript to control Photos
+        If photo is a live photo, exports both the photo and associated .mov file
+        uuid: UUID of photo to export
+        dest: destination path to export to
+        filestem: (string) if provided, exported filename will be named stem.ext 
+                  where ext is extension of the file exported by photos (e.g. .jpeg, .mov, etc)
+                  If not provided, file will be named with whatever name Photos uses
+                  If filestem.ext exists, it wil be overwritten
+        original: (boolean) if True, export original image; default = True
+        edited: (boolean) if True, export edited photo; default = False
+                If photo not edited and edited=True, will still export the original image
+                caller must verify image has been edited
+        *Note*: must be called with either edited or original but not both, 
+                will raise error if called with both edited and original = True
+        live_photo: (boolean) if True, export associated .mov live photo; default = False
+        timeout: timeout value in seconds; export will fail if applescript run time exceeds timeout
+        burst: (boolean) set to True if file is a burst image to avoid Photos export error
+        dry_run: (boolean) set to True to run in "dry run" mode which will download file but not actually copy to destination
+        Returns: list of paths to exported file(s) or None if export failed
+        Note: For Live Photos, if edited=True, will export a jpeg but not the movie, even if photo
+              has not been edited. This is due to how Photos Applescript interface works.
+    """
+
+    # setup the applescript to do the export
+    export_scpt = AppleScript(
+        """ 
+		on export_by_uuid(theUUID, thePath, original, edited, theTimeOut)
+			tell application "Photos"
+				set thePath to thePath
+				set theItem to media item id theUUID
+				set theFilename to filename of theItem
+				set itemList to {theItem}
+				
+				if original then
+					with timeout of theTimeOut seconds
+						export itemList to POSIX file thePath with using originals
+					end timeout
+				end if
+				
+				if edited then
+					with timeout of theTimeOut seconds
+						export itemList to POSIX file thePath
+					end timeout
+				end if
+				
+				return theFilename
+			end tell
+
+		end export_by_uuid
+		"""
+    )
+
+    dest = pathlib.Path(dest)
+    if not dest.is_dir:
+        raise ValueError(f"dest {dest} must be a directory")
+
+    if not original ^ edited:
+        raise ValueError(f"edited or original must be True but not both")
+
+    tmpdir = tempfile.TemporaryDirectory(prefix="osxphotos_")
+
+    # export original
+    filename = None
+    try:
+        filename = export_scpt.call(
+            "export_by_uuid", uuid, tmpdir.name, original, edited, timeout
+        )
+    except Exception as e:
+        logging.warning(f"Error exporting uuid {uuid}: {e}")
+        return None
+
+    if filename is not None:
+        # need to find actual filename as sometimes Photos renames JPG to jpeg on export
+        # may be more than one file exported (e.g. if Live Photo, Photos exports both .jpeg and .mov)
+        # TemporaryDirectory will cleanup on return
+        filename_stem = pathlib.Path(filename).stem
+        files = glob.glob(os.path.join(tmpdir.name, "*"))
+        exported_paths = []
+        for fname in files:
+            path = pathlib.Path(fname)
+            if len(files) > 1 and not live_photo and path.suffix.lower() == ".mov":
+                # it's the .mov part of live photo but not requested, so don't export
+                logging.debug(f"Skipping live photo file {path}")
+                continue
+            if len(files) > 1 and burst and path.stem != filename_stem:
+                # skip any burst photo that's not the one we asked for
+                logging.debug(f"Skipping burst photo file {path}")
+                continue
+            if filestem:
+                # rename the file based on filestem, keeping original extension
+                dest_new = dest / f"{filestem}{path.suffix}"
+            else:
+                # use the name Photos provided
+                dest_new = dest / path.name
+            logging.debug(f"exporting {path} to dest_new: {dest_new}")
+            if not dry_run:
+                FileUtil.copy(str(path), str(dest_new))
+            exported_paths.append(str(dest_new))
+        return exported_paths
+    else:
+        return None
+
+
+# _check_export_suffix is not a class method, don't import this into PhotoInfo
+def _check_export_suffix(src, dest, edited):
+    """Helper function for exporting photos to check file extensions of destination path.
+    
+    Checks that dst file extension is appropriate for the src.
+    If edited=True, will use src file extension of ".jpeg" if None provided for src.
+    
+    Args:
+        src: path to source file or None.
+        dest: path to destination file.
+        edited: set to True if exporting an edited photo.
+
+    Returns:
+        True if src and dest extensions are OK, else False.
+
+    Raises:
+        ValueError if edited is False and src is None
+    """
+
+    # check extension of destination
+    if src is not None:
+        # use suffix from edited file
+        actual_suffix = pathlib.Path(src).suffix
+    elif edited:
+        # use .jpeg as that's probably correct
+        actual_suffix = ".jpeg"
+    else:
+        raise ValueError("src must not be None if edited=False")
+
+    # Photo's often converts .JPG to .jpeg or .tif to .tiff on import
+    dest_ext = dest.suffix.lower()
+    actual_ext = actual_suffix.lower()
+    suffixes = sorted([dest_ext, actual_ext])
+    return (
+        dest_ext == actual_ext
+        or suffixes == [".jpeg", ".jpg"]
+        or suffixes == [".tif", ".tiff"]
+    )
+
+
 def export(
     self,
     dest,
@@ -61,8 +222,11 @@ def export(
                     **NOTE**: if provided, user must ensure file extension (suffix) is correct. 
                     For example, if photo is .CR2 file, edited image may be .jpeg.  
                     If you provide an extension different than what the actual file is, 
-                    export will print a warning but will happily export the photo using the 
-                    incorrect file extension.  e.g. to get the extension of the edited photo, 
+                    export will print a warning but will export the photo using the 
+                    incorrect file extension (unless use_photos_export is true, in which case export will
+                    use the extension provided by Photos upon export; in this case, an incorrect extension is
+                    silently ignored).
+                    e.g. to get the extension of the edited photo, 
                     reference PhotoInfo.path_edited
         edited: (boolean, default=False); if True will export the edited version of the photo 
                 (or raise exception if no edited version) 
@@ -135,15 +299,20 @@ def export2(
     keyword_template=None,
     update=False,
     export_db=None,
+    fileutil=FileUtil,
+    dry_run=False,
 ):
-    """ export photo 
-        dest: must be valid destination path (or exception raised) 
+    """ export photo, like export but with update and dry_run options
+        dest: must be valid destination path or exception raised 
         filename: (optional): name of exported picture; if not provided, will use current filename 
                     **NOTE**: if provided, user must ensure file extension (suffix) is correct. 
                     For example, if photo is .CR2 file, edited image may be .jpeg.  
                     If you provide an extension different than what the actual file is, 
-                    export will print a warning but will happily export the photo using the 
-                    incorrect file extension.  e.g. to get the extension of the edited photo, 
+                    export will print a warning but will export the photo using the 
+                    incorrect file extension (unless use_photos_export is true, in which case export will
+                    use the extension provided by Photos upon export; in this case, an incorrect extension is
+                    silently ignored).
+                    e.g. to get the extension of the edited photo, 
                     reference PhotoInfo.path_edited
         edited: (boolean, default=False); if True will export the edited version of the photo 
                 (or raise exception if no edited version) 
@@ -171,12 +340,17 @@ def export2(
                 not export the photo if the current version already exists in the destination
         export_db: (ExportDB_ABC); instance of a class that conforms to ExportDB_ABC with methods
                 for getting/setting data related to exported files to compare update state
+        fileutil: (FileUtilABC); class that conforms to FileUtilABC with various file utilities
+        dry_run: (boolean, default=False); set to True to run in "dry run" mode
+
         Returns: ExportResults namedtuple with fields: exported, new, updated, skipped 
                     where each field is a list of file paths
+        
+        Note: to use dry run mode, you must set dry_run=True and also pass in memory version of export_db,
+              and no-op fileutil (e.g. ExportDBInMemory and FileUtilNoOp)
             """
 
-    # if update, caller may pass function refs to get/set uuid for file being exported
-    # and for setting/getting the PhotoInfo json info for an exported file
+    # when called from export(), won't get an export_db, so use no-op version
     if export_db is None:
         export_db = ExportDBNoOp()
 
@@ -212,7 +386,7 @@ def export2(
         # verify destination is a valid path
         if dest is None:
             raise ValueError("Destination must not be None")
-        elif not os.path.isdir(dest):
+        elif not dry_run and not os.path.isdir(dest):
             raise FileNotFoundError("Invalid path passed to export")
 
         if filename and len(filename) == 1:
@@ -242,27 +416,6 @@ def export2(
     fname = pathlib.Path(fname)
     dest = dest / fname
 
-    # check extension of destination
-    if edited and self.path_edited is not None:
-        # use suffix from edited file
-        actual_suffix = pathlib.Path(self.path_edited).suffix
-    elif edited:
-        # use .jpeg as that's probably correct
-        # if edited and path_edited is None, will raise FileNotFoundError below
-        # unless use_photos_export is True
-        actual_suffix = ".jpeg"
-    else:
-        # use suffix from the non-edited file
-        actual_suffix = pathlib.Path(self.filename).suffix
-
-    # warn if suffixes don't match but ignore .JPG / .jpeg as
-    # Photo's often converts .JPG to .jpeg
-    suffixes = sorted([x.lower() for x in [dest.suffix, actual_suffix]])
-    if dest.suffix.lower() != actual_suffix.lower() and suffixes != [".jpeg", ".jpg"]:
-        logging.warning(
-            f"Invalid destination suffix: {dest.suffix}, should be {actual_suffix}"
-        )
-
     # check to see if file exists and if so, add (1), (2), etc until we find one that works
     # Photos checks the stem and adds (1), (2), etc which avoids collision with sidecars
     # e.g. exporting sidecar for file1.png and file1.jpeg
@@ -279,10 +432,6 @@ def export2(
             count += 1
         dest = dest.parent / f"{dest_new}{dest.suffix}"
 
-    # TODO: need way to check if DB is missing, try to find the right photo anyway by seeing if they're the same and then updating
-    # move the checks into "if not use_photos_export" block below
-    # if use_photos_export is True then we'll export wether destination exists or not
-
     # if overwrite==False and #increment==False, export should fail if file exists
     if dest.exists() and not update and not overwrite and not increment:
         raise FileExistsError(
@@ -314,6 +463,13 @@ def export2(
         if not os.path.isfile(src):
             raise FileNotFoundError(f"{src} does not appear to exist")
 
+        if not _check_export_suffix(src, dest, edited):
+            logging.warning(
+                f"Invalid destination suffix: {dest.suffix} for {self.path}, "
+                + f"edited={edited}, path_edited={self.path_edited}, "
+                + f"original_filename={self.original_filename}, filename={self.filename}"
+            )
+
         logging.debug(
             f"exporting {src} to {dest}, overwrite={overwrite}, increment={increment}, dest exists: {dest.exists()}"
         )
@@ -328,11 +484,14 @@ def export2(
                     f"Found matching file with blank uuid: {self.uuid}, {dest}"
                 )
                 dest_uuid = self.uuid
-                export_db.set_uuid_for_file(dest, self.uuid)
-                export_db.set_info_for_uuid(self.uuid, self.json())
-                export_db.set_stat_orig_for_file(dest, file_sig(dest))
-                export_db.set_stat_exif_for_file(dest, (None, None, None))
-                export_db.set_exifdata_for_file(dest, None)
+                export_db.set_data(
+                    dest,
+                    self.uuid,
+                    fileutil.file_sig(dest),
+                    (None, None, None),
+                    self.json(),
+                    None,
+                )
             if dest_uuid != self.uuid:
                 # not the right file, find the right one
                 logging.debug(
@@ -358,11 +517,14 @@ def export2(
                         )
                         dest = pathlib.Path(file_)
                         found_match = True
-                        export_db.set_uuid_for_file(file_, self.uuid)
-                        export_db.set_info_for_uuid(self.uuid, self.json())
-                        export_db.set_stat_orig_for_file(dest, file_sig(dest))
-                        export_db.set_stat_exif_for_file(dest, (None, None, None))
-                        export_db.set_exifdata_for_file(dest, None)
+                        export_db.set_data(
+                            dest,
+                            self.uuid,
+                            fileutil.file_sig(dest),
+                            (None, None, None),
+                            self.json(),
+                            None,
+                        )
                         break
 
                 if not found_match:
@@ -392,6 +554,7 @@ def export2(
             no_xattr,
             export_as_hardlink,
             exiftool,
+            fileutil,
         )
         exported_files = results.exported
         update_new_files = results.new
@@ -416,6 +579,7 @@ def export2(
                     no_xattr,
                     export_as_hardlink,
                     exiftool,
+                    fileutil,
                 )
                 exported_files.extend(results.exported)
                 update_new_files.extend(results.new)
@@ -440,6 +604,7 @@ def export2(
                     no_xattr,
                     export_as_hardlink,
                     exiftool,
+                    fileutil,
                 )
                 exported_files.extend(results.exported)
                 update_new_files.extend(results.new)
@@ -471,6 +636,7 @@ def export2(
                 live_photo=live_photo,
                 timeout=timeout,
                 burst=self.burst,
+                dry_run=dry_run,
             )
         else:
             # export original version and not edited
@@ -484,6 +650,7 @@ def export2(
                 live_photo=live_photo,
                 timeout=timeout,
                 burst=self.burst,
+                dry_run=dry_run,
             )
 
         if exported is not None:
@@ -504,11 +671,12 @@ def export2(
             use_persons_as_keywords=use_persons_as_keywords,
             keyword_template=keyword_template,
         )
-        try:
-            self._write_sidecar(sidecar_filename, sidecar_str)
-        except Exception as e:
-            logging.warning(f"Error writing json sidecar to {sidecar_filename}")
-            raise e
+        if not dry_run:
+            try:
+                self._write_sidecar(sidecar_filename, sidecar_str)
+            except Exception as e:
+                logging.warning(f"Error writing json sidecar to {sidecar_filename}")
+                raise e
 
     if sidecar_xmp:
         logging.debug("writing xmp_sidecar")
@@ -518,11 +686,12 @@ def export2(
             use_persons_as_keywords=use_persons_as_keywords,
             keyword_template=keyword_template,
         )
-        try:
-            self._write_sidecar(sidecar_filename, sidecar_str)
-        except Exception as e:
-            logging.warning(f"Error writing xmp sidecar to {sidecar_filename}")
-            raise e
+        if not dry_run:
+            try:
+                self._write_sidecar(sidecar_filename, sidecar_str)
+            except Exception as e:
+                logging.warning(f"Error writing xmp sidecar to {sidecar_filename}")
+                raise e
 
     # if exiftool, write the metadata
     if update:
@@ -552,12 +721,13 @@ def export2(
                 # didn't have old data, assume we need to write it
                 # or files were different
                 logging.debug(f"No exifdata for {exported_file}, writing it")
-                self._write_exif_data(
-                    exported_file,
-                    use_albums_as_keywords=use_albums_as_keywords,
-                    use_persons_as_keywords=use_persons_as_keywords,
-                    keyword_template=keyword_template,
-                )
+                if not dry_run:
+                    self._write_exif_data(
+                        exported_file,
+                        use_albums_as_keywords=use_albums_as_keywords,
+                        use_persons_as_keywords=use_persons_as_keywords,
+                        keyword_template=keyword_template,
+                    )
                 export_db.set_exifdata_for_file(
                     exported_file,
                     self._exiftool_json_sidecar(
@@ -566,17 +736,20 @@ def export2(
                         keyword_template=keyword_template,
                     ),
                 )
-                export_db.set_stat_exif_for_file(exported_file, file_sig(exported_file))
+                export_db.set_stat_exif_for_file(
+                    exported_file, fileutil.file_sig(exported_file)
+                )
                 exif_files_updated.append(exported_file)
     elif exiftool and exif_files:
         for exported_file in exif_files:
             logging.debug(f"Writing exif data to {exported_file}")
-            self._write_exif_data(
-                exported_file,
-                use_albums_as_keywords=use_albums_as_keywords,
-                use_persons_as_keywords=use_persons_as_keywords,
-                keyword_template=keyword_template,
-            )
+            if not dry_run:
+                self._write_exif_data(
+                    exported_file,
+                    use_albums_as_keywords=use_albums_as_keywords,
+                    use_persons_as_keywords=use_persons_as_keywords,
+                    keyword_template=keyword_template,
+                )
             export_db.set_exifdata_for_file(
                 exported_file,
                 self._exiftool_json_sidecar(
@@ -585,7 +758,9 @@ def export2(
                     keyword_template=keyword_template,
                 ),
             )
-            export_db.set_stat_exif_for_file(exported_file, file_sig(exported_file))
+            export_db.set_stat_exif_for_file(
+                exported_file, fileutil.file_sig(exported_file)
+            )
             exif_files_updated.append(exported_file)
 
     return ExportResults(
@@ -607,12 +782,15 @@ def _export_photo(
     no_xattr,
     export_as_hardlink,
     exiftool,
+    fileutil=FileUtil,
 ):
     """ Helper function for export()
         Does the actual copy or hardlink taking the appropriate 
         action depending on update, overwrite
         Assumes destination is the right destination (e.g. UUID matches)
         sets UUID and JSON info foo exported file using set_uuid_for_file, set_inf_for_uuido
+    
+    Args:
         src: src path (string)
         dest: dest path (pathlib.Path)
         update: bool
@@ -621,7 +799,10 @@ def _export_photo(
         no_xattr: don't copy extended attributes
         export_as_hardlink: bool
         exiftool: bool
-        Returns: ExportResults
+        fileutil: FileUtil class that conforms to fileutil.FileUtilABC
+
+    Returns:
+        ExportResults
     """
 
     exported_files = []
@@ -637,14 +818,18 @@ def _export_photo(
             # not update, do the the hardlink
             if overwrite and dest.exists():
                 # need to remove the destination first
-                dest.unlink()
+                # dest.unlink()
+                fileutil.unlink(dest)
             logging.debug(f"Not update: export_as_hardlink linking file {src} {dest}")
-            _hardlink_file(src, dest)
-            export_db.set_uuid_for_file(dest_str, self.uuid)
-            export_db.set_info_for_uuid(self.uuid, self.json())
-            export_db.set_stat_orig_for_file(dest_str, file_sig(dest_str))
-            export_db.set_stat_exif_for_file(dest_str, (None, None, None))
-            export_db.set_exifdata_for_file(dest_str, None)
+            fileutil.hardlink(src, dest)
+            export_db.set_data(
+                dest_str,
+                self.uuid,
+                fileutil.file_sig(dest_str),
+                (None, None, None),
+                self.json(),
+                None,
+            )
             exported_files.append(dest_str)
         elif dest_exists and dest.samefile(src):
             # update, hardlink and it already points to the right file, do nothing
@@ -657,13 +842,17 @@ def _export_photo(
             logging.debug(
                 f"Update: removing existing file prior to export_as_hardlink {src} {dest}"
             )
-            dest.unlink()
-            _hardlink_file(src, dest)
-            export_db.set_uuid_for_file(dest_str, self.uuid)
-            export_db.set_info_for_uuid(self.uuid, self.json())
-            export_db.set_stat_orig_for_file(dest_str, file_sig(dest_str))
-            export_db.set_stat_exif_for_file(dest_str, (None, None, None))
-            export_db.set_exifdata_for_file(dest_str, None)
+            # dest.unlink()
+            fileutil.unlink(dest)
+            fileutil.hardlink(src, dest)
+            export_db.set_data(
+                dest_str,
+                self.uuid,
+                fileutil.file_sig(dest_str),
+                (None, None, None),
+                self.json(),
+                None,
+            )
             update_updated_files.append(dest_str)
             exported_files.append(dest_str)
         else:
@@ -671,12 +860,15 @@ def _export_photo(
             logging.debug(
                 f"Update: exporting new file with export_as_hardlink {src} {dest}"
             )
-            _hardlink_file(src, dest)
-            export_db.set_uuid_for_file(dest_str, self.uuid)
-            export_db.set_info_for_uuid(self.uuid, self.json())
-            export_db.set_stat_orig_for_file(dest_str, file_sig(dest_str))
-            export_db.set_stat_exif_for_file(dest_str, (None, None, None))
-            export_db.set_exifdata_for_file(dest_str, None)
+            fileutil.hardlink(src, dest)
+            export_db.set_data(
+                dest_str,
+                self.uuid,
+                fileutil.file_sig(dest_str),
+                (None, None, None),
+                self.json(),
+                None,
+            )
             exported_files.append(dest_str)
             update_new_files.append(dest_str)
     else:
@@ -684,15 +876,19 @@ def _export_photo(
             # not update, do the the copy
             if overwrite and dest.exists():
                 # need to remove the destination first
-                dest.unlink()
+                # dest.unlink()
+                fileutil.unlink(dest)
             logging.debug(f"Not update: copying file {src} {dest}")
-            _copy_file(src, dest_str, norsrc=no_xattr)
-            export_db.set_uuid_for_file(dest_str, self.uuid)
-            export_db.set_info_for_uuid(self.uuid, self.json())
-            export_db.set_stat_orig_for_file(dest_str, file_sig(dest_str))
-            export_db.set_stat_exif_for_file(dest_str, (None, None, None))
-            export_db.set_exifdata_for_file(dest_str, None)
+            fileutil.copy(src, dest_str, norsrc=no_xattr)
             exported_files.append(dest_str)
+            export_db.set_data(
+                dest_str,
+                self.uuid,
+                fileutil.file_sig(dest_str),
+                (None, None, None),
+                self.json(),
+                None,
+            )
         # elif dest_exists and not exiftool and cmp_file(dest_str, export_db.get_stat_orig_for_file(dest_str)):
         elif (
             dest_exists
@@ -704,12 +900,12 @@ def _export_photo(
             logging.debug(f"Update: skipping identifical original files {src} {dest}")
             # call set_stat because code can reach this spot if no export DB but exporting a RAW or live photo
             # potentially re-writes the data in the database but ensures database is complete
-            export_db.set_stat_orig_for_file(dest_str, file_sig(dest_str))
+            export_db.set_stat_orig_for_file(dest_str, fileutil.file_sig(dest_str))
             update_skipped_files.append(dest_str)
         elif (
             dest_exists
             and exiftool
-            and cmp_file(dest_str, export_db.get_stat_exif_for_file(dest_str))
+            and fileutil.cmp_sig(dest_str, export_db.get_stat_exif_for_file(dest_str))
             and not dest.samefile(src)
         ):
             # destination exists but is identical
@@ -720,24 +916,31 @@ def _export_photo(
             logging.debug(f"Update: removing existing file prior to copy {src} {dest}")
             stat_src = os.stat(src)
             stat_dest = os.stat(dest)
-            dest.unlink()
-            _copy_file(src, dest_str, norsrc=no_xattr)
-            export_db.set_uuid_for_file(dest_str, self.uuid)
-            export_db.set_info_for_uuid(self.uuid, self.json())
-            export_db.set_stat_orig_for_file(dest_str, file_sig(dest_str))
-            export_db.set_stat_exif_for_file(dest_str, (None, None, None))
-            export_db.set_exifdata_for_file(dest_str, None)
+            # dest.unlink()
+            fileutil.unlink(dest)
+            fileutil.copy(src, dest_str, norsrc=no_xattr)
+            export_db.set_data(
+                dest_str,
+                self.uuid,
+                fileutil.file_sig(dest_str),
+                (None, None, None),
+                self.json(),
+                None,
+            )
             exported_files.append(dest_str)
             update_updated_files.append(dest_str)
         else:
             # destination doesn't exist, copy the file
             logging.debug(f"Update: copying new file {src} {dest}")
-            _copy_file(src, dest_str, norsrc=no_xattr)
-            export_db.set_uuid_for_file(dest_str, self.uuid)
-            export_db.set_info_for_uuid(self.uuid, self.json())
-            export_db.set_stat_orig_for_file(dest_str, file_sig(dest_str))
-            export_db.set_stat_exif_for_file(dest_str, (None, None, None))
-            export_db.set_exifdata_for_file(dest_str, None)
+            fileutil.copy(src, dest_str, norsrc=no_xattr)
+            export_db.set_data(
+                dest_str,
+                self.uuid,
+                fileutil.file_sig(dest_str),
+                (None, None, None),
+                self.json(),
+                None,
+            )
             exported_files.append(dest_str)
             update_new_files.append(dest_str)
 
@@ -986,7 +1189,7 @@ def _xmp_sidecar(
 def _write_sidecar(self, filename, sidecar_str):
     """ write sidecar_str to filename
         used for exporting sidecar info """
-    if not filename and not sidecar_str:
+    if not (filename or sidecar_str):
         raise (
             ValueError(
                 f"filename {filename} and sidecar_str {sidecar_str} must not be None"

osxphotos/photoinfo/photoinfo.py

@@ -11,7 +11,6 @@ import logging
 import os
 import os.path
 import pathlib
-import re
 import subprocess
 import sys
 from datetime import timedelta, timezone
@@ -24,17 +23,11 @@ from .._constants import (
     _PHOTO_TYPE,
     _PHOTOS_4_VERSION,
     _PHOTOS_5_SHARED_PHOTO_PATH,
-    _UNKNOWN_PERSON,
 )
 from ..albuminfo import AlbumInfo
-from ..datetime_formatter import DateTimeFormatter
 from ..placeinfo import PlaceInfo4, PlaceInfo5
+from ..phototemplate import PhotoTemplate
 from ..utils import _debug, _get_resource_loc, findfiles, get_preferred_uti_extension
-from .template import (
-    MULTI_VALUE_SUBSTITUTIONS,
-    TEMPLATE_SUBSTITUTIONS,
-    TEMPLATE_SUBSTITUTIONS_MULTI_VALUED,
-)
 
 
 class PhotoInfo:
@@ -636,339 +629,17 @@ class PhotoInfo:
             otherwise returns False """
         return self._info["raw_is_original"]
 
-    def render_template(self, template, none_str="_", path_sep=None):
-        """ render a filename or directory template 
-            template: str template 
-            none_str: str to use default for None values, default is '_' 
-            path_sep: optional character to use as path separator, default is os.path.sep """
+    def render_template(self, template_str, none_str="_", path_sep=None):
+        """Renders a template string for PhotoInfo instance using PhotoTemplate
 
-        if path_sep is None:
-            path_sep = os.path.sep
-        elif path_sep is not None and len(path_sep) != 1:
-            raise ValueError(f"path_sep must be single character: {path_sep}")
-
-        # the rendering happens in two phases:
-        # phase 1: handle all the single-value template substitutions
-        #          results in a single string with all the template fields replaced
-        # phase 2: loop through all the multi-value template substitutions
-        #          could result in multiple strings
-        #          e.g. if template is "{album}/{person}" and there are 2 albums and 3 persons in the photo
-        #          there would be 6 possible renderings (2 albums x 3 persons)
-
-        # regex to find {template_field,optional_default} in strings
-        # for explanation of regex see https://regex101.com/r/4JJg42/1
-        # pylint: disable=anomalous-backslash-in-string
-        regex = r"(?<!\{)\{([^\\,}]+)(,{0,1}(([\w\-. ]+))?)(?=\}(?!\}))\}"
-        if type(template) is not str:
-            raise TypeError(f"template must be type str, not {type(template)}")
-
-        def make_subst_function(self, none_str, get_func=self.get_template_value):
-            """ returns: substitution function for use in re.sub 
-                none_str: value to use if substitution lookup is None and no default provided
-                get_func: function that gets the substitution value for a given template field
-                        default is get_template_value which handles the single-value fields """
-
-            # closure to capture photo, none_str in subst
-            def subst(matchobj):
-                groups = len(matchobj.groups())
-                if groups == 4:
-                    try:
-                        val = get_func(matchobj.group(1))
-                    except KeyError:
-                        return matchobj.group(0)
-
-                    if val is None:
-                        return (
-                            matchobj.group(3)
-                            if matchobj.group(3) is not None
-                            else none_str
-                        )
-                    else:
-                        return val
-                else:
-                    raise ValueError(
-                        f"Unexpected number of groups: expected 4, got {groups}"
-                    )
-
-            return subst
-
-        subst_func = make_subst_function(self, none_str)
-
-        # do the replacements
-        rendered = re.sub(regex, subst_func, template)
-
-        # do multi-valued placements
-        # start with the single string from phase 1 above then loop through all
-        # multi-valued fields and all values for each of those fields
-        # rendered_strings will be updated as each field is processed
-        # for example: if two albums, two keywords, and one person and template is:
-        # "{created.year}/{album}/{keyword}/{person}"
-        # rendered strings would do the following:
-        # start (created.year filled in phase 1)
-        #   ['2011/{album}/{keyword}/{person}']
-        # after processing albums:
-        #   ['2011/Album1/{keyword}/{person}',
-        #    '2011/Album2/{keyword}/{person}',]
-        # after processing keywords:
-        #   ['2011/Album1/keyword1/{person}',
-        #    '2011/Album1/keyword2/{person}',
-        #    '2011/Album2/keyword1/{person}',
-        #    '2011/Album2/keyword2/{person}',]
-        # after processing person:
-        #   ['2011/Album1/keyword1/person1',
-        #    '2011/Album1/keyword2/person1',
-        #    '2011/Album2/keyword1/person1',
-        #    '2011/Album2/keyword2/person1',]
-
-        rendered_strings = set([rendered])
-        for field in MULTI_VALUE_SUBSTITUTIONS:
-            if field == "album":
-                values = self.albums
-            elif field == "keyword":
-                values = self.keywords
-            elif field == "person":
-                values = self.persons
-                # remove any _UNKNOWN_PERSON values
-                values = [val for val in values if val != _UNKNOWN_PERSON]
-            elif field == "label":
-                values = self.labels
-            elif field == "label_normalized":
-                values = self.labels_normalized
-            elif field == "folder_album":
-                values = []
-                # photos must be in an album to be in a folder
-                for album in self.album_info:
-                    if album.folder_names:
-                        # album in folder
-                        folder = path_sep.join(album.folder_names)
-                        folder += path_sep + album.title
-                        values.append(folder)
-                    else:
-                        # album not in folder
-                        values.append(album.title)
-            else:
-                raise ValueError(f"Unhandleded template value: {field}")
-
-            # If no values, insert None so code below will substite none_str for None
-            values = values or [None]
-
-            # Build a regex that matches only the field being processed
-            re_str = r"(?<!\\)\{(" + field + r")(,(([\w\-. ]{0,})))?\}"
-            regex_multi = re.compile(re_str)
-
-            # holds each of the new rendered_strings, set() to avoid duplicates
-            new_strings = set()
-
-            for str_template in rendered_strings:
-                for val in values:
-
-                    def get_template_value_multi(lookup_value):
-                        """ Closure passed to make_subst_function get_func 
-                            Capture val and field in the closure 
-                            Allows make_subst_function to be re-used w/o modification """
-                        if lookup_value == field:
-                            return val
-                        else:
-                            raise KeyError(f"Unexpected value: {lookup_value}")
-
-                    subst = make_subst_function(
-                        self, none_str, get_func=get_template_value_multi
-                    )
-                    new_string = regex_multi.sub(subst, str_template)
-                    new_strings.add(new_string)
-
-            # update rendered_strings for the next field to process
-            rendered_strings = new_strings
-
-        # find any {fields} that weren't replaced
-        unmatched = []
-        for rendered_str in rendered_strings:
-            unmatched.extend(
-                [
-                    no_match[0]
-                    for no_match in re.findall(regex, rendered_str)
-                    if no_match[0] not in unmatched
-                ]
-            )
-
-        # fix any escaped curly braces
-        rendered_strings = [
-            rendered_str.replace("{{", "{").replace("}}", "}")
-            for rendered_str in rendered_strings
-        ]
-
-        return rendered_strings, unmatched
-
-    def get_template_value(self, lookup):
-        """ lookup template value (single-value template substitutions) for use in make_subst_function
-            lookup: value to find a match for
-            returns: either the matching template value (which may be None)
-            raises: KeyError if no rule exists for lookup """
-
-        # must be a valid keyword
-        if lookup == "name":
-            return pathlib.Path(self.filename).stem
-
-        if lookup == "original_name":
-            return pathlib.Path(self.original_filename).stem
-
-        if lookup == "title":
-            return self.title
-
-        if lookup == "descr":
-            return self.description
-
-        if lookup == "created.date":
-            return DateTimeFormatter(self.date).date
-
-        if lookup == "created.year":
-            return DateTimeFormatter(self.date).year
-
-        if lookup == "created.yy":
-            return DateTimeFormatter(self.date).yy
-
-        if lookup == "created.mm":
-            return DateTimeFormatter(self.date).mm
-
-        if lookup == "created.month":
-            return DateTimeFormatter(self.date).month
-
-        if lookup == "created.mon":
-            return DateTimeFormatter(self.date).mon
-
-        if lookup == "created.doy":
-            return DateTimeFormatter(self.date).doy
-
-        if lookup == "modified.date":
-            return (
-                DateTimeFormatter(self.date_modified).date
-                if self.date_modified
-                else None
-            )
-
-        if lookup == "modified.year":
-            return (
-                DateTimeFormatter(self.date_modified).year
-                if self.date_modified
-                else None
-            )
-
-        if lookup == "modified.yy":
-            return (
-                DateTimeFormatter(self.date_modified).yy if self.date_modified else None
-            )
-
-        if lookup == "modified.mm":
-            return (
-                DateTimeFormatter(self.date_modified).mm if self.date_modified else None
-            )
-
-        if lookup == "modified.month":
-            return (
-                DateTimeFormatter(self.date_modified).month
-                if self.date_modified
-                else None
-            )
-
-        if lookup == "modified.mon":
-            return (
-                DateTimeFormatter(self.date_modified).mon
-                if self.date_modified
-                else None
-            )
-
-        if lookup == "modified.doy":
-            return (
-                DateTimeFormatter(self.date_modified).doy
-                if self.date_modified
-                else None
-            )
-
-        if lookup == "place.name":
-            return self.place.name if self.place else None
-
-        if lookup == "place.country_code":
-            return self.place.country_code if self.place else None
-
-        if lookup == "place.name.country":
-            return (
-                self.place.names.country[0]
-                if self.place and self.place.names.country
-                else None
-            )
-
-        if lookup == "place.name.state_province":
-            return (
-                self.place.names.state_province[0]
-                if self.place and self.place.names.state_province
-                else None
-            )
-
-        if lookup == "place.name.city":
-            return (
-                self.place.names.city[0]
-                if self.place and self.place.names.city
-                else None
-            )
-
-        if lookup == "place.name.area_of_interest":
-            return (
-                self.place.names.area_of_interest[0]
-                if self.place and self.place.names.area_of_interest
-                else None
-            )
-
-        if lookup == "place.address":
-            return (
-                self.place.address_str
-                if self.place and self.place.address_str
-                else None
-            )
-
-        if lookup == "place.address.street":
-            return (
-                self.place.address.street
-                if self.place and self.place.address.street
-                else None
-            )
-
-        if lookup == "place.address.city":
-            return (
-                self.place.address.city
-                if self.place and self.place.address.city
-                else None
-            )
-
-        if lookup == "place.address.state_province":
-            return (
-                self.place.address.state_province
-                if self.place and self.place.address.state_province
-                else None
-            )
-
-        if lookup == "place.address.postal_code":
-            return (
-                self.place.address.postal_code
-                if self.place and self.place.address.postal_code
-                else None
-            )
-
-        if lookup == "place.address.country":
-            return (
-                self.place.address.country
-                if self.place and self.place.address.country
-                else None
-            )
-
-        if lookup == "place.address.country_code":
-            return (
-                self.place.address.iso_country_code
-                if self.place and self.place.address.iso_country_code
-                else None
-            )
-
-        # if here, didn't get a match
-        raise KeyError(f"No rule for processing {lookup}")
+        Args:
+            template_str: a template string with fields to render
+            none_str: a str to use if template field renders to None, default is "_".
+            path_sep: a single character str to use as path separator when joining 
+                fields like folder_album; if not provided, defaults to os.path.sep
+        """
+        template = PhotoTemplate(self)
+        return template.render(template_str, none_str, path_sep)
 
     @property
     def _longitude(self):

osxphotos/photoinfo/template.py

@@ -1,66 +0,0 @@
-""" Custom template system for osxphotos (implemented in PhotoInfo.render_template) """
-
-# Rolled my own template system because:
-# 1. Needed to handle multiple values (e.g. album, keyword)
-# 2. Needed to handle default values if template not found
-# 3. Didn't want user to need to know python (e.g. by using Mako which is
-#    already used elsewhere in this project)
-# 4. Couldn't figure out how to do #1 and #2 with str.format()
-#
-# This code isn't elegant but it seems to work well.  PRs gladly accepted.
-
-import locale
-
-# ensure locale set to user's locale
-locale.setlocale(locale.LC_ALL, "")
-
-# Permitted substitutions (each of these returns a single value or None)
-TEMPLATE_SUBSTITUTIONS = {
-    "{name}": "Current filename of the photo",
-    "{original_name}": "Photo's original filename when imported to Photos",
-    "{title}": "Title of the photo",
-    "{descr}": "Description of the photo",
-    "{created.date}": "Photo's creation date in ISO format, e.g. '2020-03-22'",
-    "{created.year}": "4-digit year of file creation time",
-    "{created.yy}": "2-digit year of file creation time",
-    "{created.mm}": "2-digit month of the file creation time (zero padded)",
-    "{created.month}": "Month name in user's locale of the file creation time",
-    "{created.mon}": "Month abbreviation in the user's locale of the file creation time",
-    "{created.doy}": "3-digit day of year (e.g Julian day) of file creation time, starting from 1 (zero padded)",
-    "{modified.date}": "Photo's modification date in ISO format, e.g. '2020-03-22'",
-    "{modified.year}": "4-digit year of file modification time",
-    "{modified.yy}": "2-digit year of file modification time",
-    "{modified.mm}": "2-digit month of the file modification time (zero padded)",
-    "{modified.month}": "Month name in user's locale of the file modification time",
-    "{modified.mon}": "Month abbreviation in the user's locale of the file modification time",
-    "{modified.doy}": "3-digit day of year (e.g Julian day) of file modification time, starting from 1 (zero padded)",
-    "{place.name}": "Place name from the photo's reverse geolocation data, as displayed in Photos",
-    "{place.country_code}": "The ISO country code from the photo's reverse geolocation data",
-    "{place.name.country}": "Country name from the photo's reverse geolocation data",
-    "{place.name.state_province}": "State or province name from the photo's reverse geolocation data",
-    "{place.name.city}": "City or locality name from the photo's reverse geolocation data",
-    "{place.name.area_of_interest}": "Area of interest name (e.g. landmark or public place) from the photo's reverse geolocation data",
-    "{place.address}": "Postal address from the photo's reverse geolocation data, e.g. '2007 18th St NW, Washington, DC 20009, United States'",
-    "{place.address.street}": "Street part of the postal address, e.g. '2007 18th St NW'",
-    "{place.address.city}": "City part of the postal address, e.g. 'Washington'",
-    "{place.address.state_province}": "State/province part of the postal address, e.g. 'DC'",
-    "{place.address.postal_code}": "Postal code part of the postal address, e.g. '20009'",
-    "{place.address.country}": "Country name of the postal address, e.g. 'United States'",
-    "{place.address.country_code}": "ISO country code of the postal address, e.g. 'US'",
-}
-
-# Permitted multi-value substitutions (each of these returns None or 1 or more values)
-TEMPLATE_SUBSTITUTIONS_MULTI_VALUED = {
-    "{album}": "Album(s) photo is contained in",
-    "{folder_album}": "Folder path + album photo is contained in. e.g. 'Folder/Subfolder/Album' or just 'Album' if no enclosing folder",
-    "{keyword}": "Keyword(s) assigned to photo",
-    "{person}": "Person(s) / face(s) in a photo",
-    "{label}": "Image categorization label associated with a photo (Photos 5 only)",
-    "{label_normalized}": "All lower case version of 'label' (Photos 5 only)",
-}
-
-# Just the multi-valued substitution names without the braces
-MULTI_VALUE_SUBSTITUTIONS = [
-    field.replace("{", "").replace("}", "")
-    for field in TEMPLATE_SUBSTITUTIONS_MULTI_VALUED.keys()
-]

osxphotos/photosdb/_photosdb_process_searchinfo.py

@@ -3,6 +3,7 @@
     ref: https://github.com/dogsheep/photos-to-sqlite/issues/16
 """
 
+from functools import lru_cache
 import logging
 import pathlib
 import uuid as uuidlib
@@ -39,6 +40,19 @@ def _process_searchinfo(self):
         search terms.  In Photos 5, this is called psi.sqlite
         Note: Only works on Photos version == 5.0 """
 
+    # _db_searchinfo_uuid is dict in form {uuid : [list of associated search info records]
+    self._db_searchinfo_uuid = _db_searchinfo_uuid = {}
+
+    # _db_searchinfo_categories is dict in form {search info category id: list normalized strings for the category
+    # right now, this is mostly for debugging to easily see which search terms are in the library
+    self._db_searchinfo_categories = _db_searchinfo_categories = {}
+
+    # _db_searchinfo_labels is dict in form {normalized label: [list of photo uuids]}
+    # this serves as a reverse index from label to photos containing the label
+    # _db_searchinfo_labels_normalized is the same but with normalized (lower case) version of the label
+    self._db_searchinfo_labels = _db_searchinfo_labels = {}
+    self._db_searchinfo_labels_normalized = _db_searchinfo_labels_normalized = {}
+
     if self._db_version <= _PHOTOS_4_VERSION:
         raise NotImplementedError(
             f"search info not implemented for this database version"
@@ -46,7 +60,8 @@ def _process_searchinfo(self):
 
     search_db_path = pathlib.Path(self._dbfile).parent / "search" / "psi.sqlite"
     if not search_db_path.exists():
-        raise FileNotFoundError(f"could not find search db: {search_db_path}")
+        logging.warning(f"could not find search db: {search_db_path}")
+        return None
 
     if _db_is_locked(search_db_path):
         search_db = self._copy_db_file(search_db_path)
@@ -55,7 +70,7 @@ def _process_searchinfo(self):
 
     (conn, c) = _open_sql_file(search_db)
 
-    result = conn.execute(
+    result = c.execute(
         """
         select
         ga.rowid,
@@ -76,27 +91,31 @@ def _process_searchinfo(self):
         """
     )
 
-    # _db_searchinfo_uuid is dict in form {uuid : [list of associated search info records]
-    _db_searchinfo_uuid = {}
+    # 0: ga.rowid,
+    # 1: assets.uuid_0,
+    # 2: assets.uuid_1,
+    # 3: groups.rowid as groupid,
+    # 4: groups.category,
+    # 5: groups.owning_groupid,
+    # 6: groups.content_string,
+    # 7: groups.normalized_string,
+    # 8: groups.lookup_identifier
 
-    # _db_searchinfo_categories is dict in form {search info category id: list normalized strings for the category
-    # right now, this is mostly for debugging to easily see which search terms are in the library
-    _db_searchinfo_categories = {}
-
-    # _db_searchinfo_labels is dict in form {normalized label: [list of photo uuids]}
-    # this serves as a reverse index from label to photos containing the label
-    # _db_searchinfo_labels_normalized is the same but with normalized (lower case) version of the label
-    _db_searchinfo_labels = {}
-    _db_searchinfo_labels_normalized = {}
-
-    cols = [c[0] for c in result.description]
-    for row in result.fetchall():
-        record = dict(zip(cols, row))
-        uuid = ints_to_uuid(record["uuid_0"], record["uuid_1"])
+    for row in c:
+        uuid = ints_to_uuid(row[1],row[2])
         # strings have null character appended, so strip it
-        for key in record:
-            if isinstance(record[key], str):
-                record[key] = record[key].replace("\x00", "")
+        record = {}
+        record["uuid"] = uuid
+        record["rowid"] = row[0]
+        record["uuid_0"] = row[1]
+        record["uuid_1"] = row[2]
+        record["groupid"] = row[3]
+        record["category"] = row[4]
+        record["owning_groupid"] = row[5]
+        record["content_string"] = row[6].replace("\x00", "")
+        record["normalized_string"] = row[7].replace("\x00", "")
+        record["lookup_identifier"] = row[8]
+
         try:
             _db_searchinfo_uuid[uuid].append(record)
         except KeyError:
@@ -104,15 +123,15 @@ def _process_searchinfo(self):
 
         category = record["category"]
         try:
-            _db_searchinfo_categories[record["category"]].append(
+            _db_searchinfo_categories[category].append(
                 record["normalized_string"]
             )
         except KeyError:
-            _db_searchinfo_categories[record["category"]] = [
+            _db_searchinfo_categories[category] = [
                 record["normalized_string"]
             ]
 
-        if record["category"] == SEARCH_CATEGORY_LABEL:
+        if category == SEARCH_CATEGORY_LABEL:
             label = record["content_string"]
             label_norm = record["normalized_string"]
             try:
@@ -122,11 +141,6 @@ def _process_searchinfo(self):
                 _db_searchinfo_labels[label] = [uuid]
                 _db_searchinfo_labels_normalized[label_norm] = [uuid]
 
-    self._db_searchinfo_categories = _db_searchinfo_categories
-    self._db_searchinfo_uuid = _db_searchinfo_uuid
-    self._db_searchinfo_labels = _db_searchinfo_labels
-    self._db_searchinfo_labels_normalized = _db_searchinfo_labels_normalized
-
     if _debug():
         logging.debug(
             "_db_searchinfo_categories: \n" + pformat(self._db_searchinfo_categories)
@@ -184,7 +198,7 @@ def labels_normalized_as_dict(self):
 
 # The following method is not imported into PhotosDB
 
-
+@lru_cache(maxsize=128)
 def ints_to_uuid(uuid_0, uuid_1):
     """ convert two signed ints into a UUID strings
         uuid_0, uuid_1: the two int components of an RFC 4122 UUID """

osxphotos/photosdb/photosdb.py

@@ -294,18 +294,17 @@ class PhotosDB:
     @property
     def keywords_as_dict(self):
         """ return keywords as dict of keyword, count in reverse sorted order (descending) """
-        keywords = {}
-        for k in self._dbkeywords_keyword.keys():
-            keywords[k] = len(self._dbkeywords_keyword[k])
+        keywords = {
+            k: len(self._dbkeywords_keyword[k]) for k in self._dbkeywords_keyword.keys()
+        }
+
         keywords = dict(sorted(keywords.items(), key=lambda kv: kv[1], reverse=True))
         return keywords
 
     @property
     def persons_as_dict(self):
         """ return persons as dict of person, count in reverse sorted order (descending) """
-        persons = {}
-        for k in self._dbfaces_person.keys():
-            persons[k] = len(self._dbfaces_person[k])
+        persons = {k: len(self._dbfaces_person[k]) for k in self._dbfaces_person.keys()}
         persons = dict(sorted(persons.items(), key=lambda kv: kv[1], reverse=True))
         return persons
 
@@ -413,13 +412,12 @@ class PhotosDB:
     def album_info(self):
         """ return list of AlbumInfo objects for each album in the photos database """
 
-        albums = [
+        return [
             AlbumInfo(db=self, uuid=album)
             for album in self._dbalbums_album.keys()
             if self._dbalbum_details[album]["cloudownerhashedpersonid"] is None
             and not self._dbalbum_details[album]["intrash"]
         ]
-        return albums
 
     @property
     def album_info_shared(self):
@@ -433,13 +431,12 @@ class PhotosDB:
             )
             return []
 
-        albums_shared = [
+        return [
             AlbumInfo(db=self, uuid=album)
             for album in self._dbalbums_album.keys()
             if self._dbalbum_details[album]["cloudownerhashedpersonid"] is not None
             and not self._dbalbum_details[album]["intrash"]
         ]
-        return albums_shared
 
     @property
     def albums(self):
@@ -798,14 +795,20 @@ class PhotosDB:
             # There are sometimes negative values for lastmodifieddate in the database
             # I don't know what these mean but they will raise exception in datetime if
             # not accounted for
-            if row[4] is not None and row[4] >= 0:
+            try:
                 self._dbphotos[uuid]["lastmodifieddate"] = datetime.fromtimestamp(
                     row[4] + td
                 )
-            else:
+            except ValueError:
+                self._dbphotos[uuid]["lastmodifieddate"] = None
+            except TypeError:
                 self._dbphotos[uuid]["lastmodifieddate"] = None
 
-            self._dbphotos[uuid]["imageDate"] = datetime.fromtimestamp(row[5] + td)
+            try:
+                self._dbphotos[uuid]["imageDate"] = datetime.fromtimestamp(row[5] + td)
+            except ValueError:
+                self._dbphotos[uuid]["imageDate"] = datetime.date(1970, 1, 1)
+
             self._dbphotos[uuid]["mainRating"] = row[6]
             self._dbphotos[uuid]["hasAdjustments"] = row[7]
             self._dbphotos[uuid]["hasKeywords"] = row[8]
@@ -1512,12 +1515,18 @@ class PhotosDB:
             # There are sometimes negative values for lastmodifieddate in the database
             # I don't know what these mean but they will raise exception in datetime if
             # not accounted for
-            if row[4] is not None and row[4] >= 0:
+            try:
                 info["lastmodifieddate"] = datetime.fromtimestamp(row[4] + td)
-            else:
+            except ValueError:
+                info["lastmodifieddate"] = None
+            except TypeError:
                 info["lastmodifieddate"] = None
 
-            info["imageDate"] = datetime.fromtimestamp(row[5] + td)
+            try:
+                info["imageDate"] = datetime.fromtimestamp(row[5] + td)
+            except ValueError:
+                info["imageDate"] = datetime.date(1970, 1, 1)
+
             info["imageTimeZoneOffsetSeconds"] = row[6]
             info["hidden"] = row[9]
             info["favorite"] = row[10]
@@ -1712,8 +1721,6 @@ class PhotosDB:
         # determine if a photo is missing in Photos 5
 
         # Get info on remote/local availability for photos in shared albums
-        # Shared photos have a null fingerprint (and some other photos do too)
-        # TODO: There may be a bug here, perhaps ZDATASTORESUBTYPE should be 1 --> it's the longest ZDATALENGTH (is this the original)
         c.execute(
             """ SELECT 
                 ZGENERICASSET.ZUUID, 
@@ -1722,10 +1729,7 @@ class PhotosDB:
                 FROM ZGENERICASSET
                 JOIN ZADDITIONALASSETATTRIBUTES ON ZADDITIONALASSETATTRIBUTES.ZASSET = ZGENERICASSET.Z_PK 
                 JOIN ZINTERNALRESOURCE ON ZINTERNALRESOURCE.ZASSET = ZADDITIONALASSETATTRIBUTES.ZASSET 
-                WHERE  ZDATASTORESUBTYPE = 0 OR ZDATASTORESUBTYPE = 3 """
-            # WHERE  ZDATASTORESUBTYPE = 1 OR ZDATASTORESUBTYPE = 3 """
-            # WHERE  ZDATASTORESUBTYPE = 0 OR ZDATASTORESUBTYPE = 3 """
-            # WHERE ZINTERNALRESOURCE.ZFINGERPRINT IS NULL AND ZINTERNALRESOURCE.ZDATASTORESUBTYPE = 3 """
+                WHERE  ZDATASTORESUBTYPE = 1 OR ZDATASTORESUBTYPE = 3 """
         )
 
         for row in c:
@@ -1926,6 +1930,7 @@ class PhotosDB:
         return folders
 
     def _album_folder_hierarchy_list(self, album_uuid):
+        """ return appropriate album_folder_hierarchy_list for the _db_version """
         if self._db_version <= _PHOTOS_4_VERSION:
             return self._album_folder_hierarchy_list_4(album_uuid)
         else:
@@ -1936,8 +1941,11 @@ class PhotosDB:
             the folder list is in form:
             ["Top level folder", "sub folder 1", "sub folder 2"]
             returns empty list of album is not in any folders """
-        # title = photosdb._dbalbum_details[album_uuid]["title"]
-        folders = self._dbalbum_folders[album_uuid]
+        try:
+            folders = self._dbalbum_folders[album_uuid]
+        except KeyError:
+            logging.debug(f"Caught _dbalbum_folders KeyError for album: {album_uuid}")
+            return []
 
         def _recurse_folder_hierarchy(folders, hierarchy=[]):
             """ recursively walk the folders dict to build list of folder hierarchy """
@@ -1970,8 +1978,11 @@ class PhotosDB:
             the folder list is in form:
             ["Top level folder", "sub folder 1", "sub folder 2"]
             returns empty list of album is not in any folders """
-        # title = photosdb._dbalbum_details[album_uuid]["title"]
-        folders = self._dbalbum_folders[album_uuid]
+        try:
+            folders = self._dbalbum_folders[album_uuid]
+        except KeyError:
+            logging.debug(f"Caught _dbalbum_folders KeyError for album: {album_uuid}")
+            return []
 
         def _recurse_folder_hierarchy(folders, hierarchy=[]):
             """ recursively walk the folders dict to build list of folder hierarchy """
@@ -2117,7 +2128,11 @@ class PhotosDB:
                     if album in self._dbalbum_titles:
                         title_set = set()
                         for album_id in self._dbalbum_titles[album]:
-                            title_set.update(self._dbalbums_album[album_id])
+                            try:
+                                title_set.update(self._dbalbums_album[album_id])
+                            except KeyError:
+                                # an empty album will be in _dbalbum_titles but not _dbalbums_album
+                                pass
                         album_set.update(title_set)
                     else:
                         logging.debug(f"Could not find album '{album}' in database")

osxphotos/phototemplate.py

@@ -0,0 +1,462 @@
+""" Custom template system for osxphotos (implemented in PhotoInfo.render_template) """
+
+
+# Rolled my own template system because:
+# 1. Needed to handle multiple values (e.g. album, keyword)
+# 2. Needed to handle default values if template not found
+# 3. Didn't want user to need to know python (e.g. by using Mako which is
+#    already used elsewhere in this project)
+# 4. Couldn't figure out how to do #1 and #2 with str.format()
+#
+# This code isn't elegant but it seems to work well.  PRs gladly accepted.
+import locale
+import os
+import re
+import pathlib
+
+from ._constants import _UNKNOWN_PERSON
+from .datetime_formatter import DateTimeFormatter
+
+# ensure locale set to user's locale
+locale.setlocale(locale.LC_ALL, "")
+
+# Permitted substitutions (each of these returns a single value or None)
+TEMPLATE_SUBSTITUTIONS = {
+    "{name}": "Current filename of the photo",
+    "{original_name}": "Photo's original filename when imported to Photos",
+    "{title}": "Title of the photo",
+    "{descr}": "Description of the photo",
+    "{created.date}": "Photo's creation date in ISO format, e.g. '2020-03-22'",
+    "{created.year}": "4-digit year of file creation time",
+    "{created.yy}": "2-digit year of file creation time",
+    "{created.mm}": "2-digit month of the file creation time (zero padded)",
+    "{created.month}": "Month name in user's locale of the file creation time",
+    "{created.mon}": "Month abbreviation in the user's locale of the file creation time",
+    "{created.dd}": "2-digit day of the month (zero padded) of file creation time",
+    "{created.dow}": "Day of week in user's locale of the file creation time",
+    "{created.doy}": "3-digit day of year (e.g Julian day) of file creation time, starting from 1 (zero padded)",
+    "{modified.date}": "Photo's modification date in ISO format, e.g. '2020-03-22'",
+    "{modified.year}": "4-digit year of file modification time",
+    "{modified.yy}": "2-digit year of file modification time",
+    "{modified.mm}": "2-digit month of the file modification time (zero padded)",
+    "{modified.month}": "Month name in user's locale of the file modification time",
+    "{modified.mon}": "Month abbreviation in the user's locale of the file modification time",
+    "{modified.dd}": "2-digit day of the month (zero padded) of the file modification time",
+    "{modified.doy}": "3-digit day of year (e.g Julian day) of file modification time, starting from 1 (zero padded)",
+    "{place.name}": "Place name from the photo's reverse geolocation data, as displayed in Photos",
+    "{place.country_code}": "The ISO country code from the photo's reverse geolocation data",
+    "{place.name.country}": "Country name from the photo's reverse geolocation data",
+    "{place.name.state_province}": "State or province name from the photo's reverse geolocation data",
+    "{place.name.city}": "City or locality name from the photo's reverse geolocation data",
+    "{place.name.area_of_interest}": "Area of interest name (e.g. landmark or public place) from the photo's reverse geolocation data",
+    "{place.address}": "Postal address from the photo's reverse geolocation data, e.g. '2007 18th St NW, Washington, DC 20009, United States'",
+    "{place.address.street}": "Street part of the postal address, e.g. '2007 18th St NW'",
+    "{place.address.city}": "City part of the postal address, e.g. 'Washington'",
+    "{place.address.state_province}": "State/province part of the postal address, e.g. 'DC'",
+    "{place.address.postal_code}": "Postal code part of the postal address, e.g. '20009'",
+    "{place.address.country}": "Country name of the postal address, e.g. 'United States'",
+    "{place.address.country_code}": "ISO country code of the postal address, e.g. 'US'",
+}
+
+# Permitted multi-value substitutions (each of these returns None or 1 or more values)
+TEMPLATE_SUBSTITUTIONS_MULTI_VALUED = {
+    "{album}": "Album(s) photo is contained in",
+    "{folder_album}": "Folder path + album photo is contained in. e.g. 'Folder/Subfolder/Album' or just 'Album' if no enclosing folder",
+    "{keyword}": "Keyword(s) assigned to photo",
+    "{person}": "Person(s) / face(s) in a photo",
+    "{label}": "Image categorization label associated with a photo (Photos 5 only)",
+    "{label_normalized}": "All lower case version of 'label' (Photos 5 only)",
+}
+
+# Just the multi-valued substitution names without the braces
+MULTI_VALUE_SUBSTITUTIONS = [
+    field.replace("{", "").replace("}", "")
+    for field in TEMPLATE_SUBSTITUTIONS_MULTI_VALUED
+]
+
+
+class PhotoTemplate:
+    """ PhotoTemplate class to render a template string from a PhotoInfo object """
+
+    def __init__(self, photo):
+        """ Inits PhotoTemplate class with photo, non_str, and path_sep
+
+        Args:
+            photo: a PhotoInfo instance.
+        """
+        self.photo = photo
+
+    def render(self, template, none_str="_", path_sep=None):
+        """ render a filename or directory template 
+            template: str template 
+            none_str: str to use default for None values, default is '_' 
+            path_sep: optional character to use as path separator, default is os.path.sep """
+        if path_sep is None:
+            path_sep = os.path.sep
+        elif path_sep is not None and len(path_sep) != 1:
+            raise ValueError(f"path_sep must be single character: {path_sep}")
+
+        # the rendering happens in two phases:
+        # phase 1: handle all the single-value template substitutions
+        #          results in a single string with all the template fields replaced
+        # phase 2: loop through all the multi-value template substitutions
+        #          could result in multiple strings
+        #          e.g. if template is "{album}/{person}" and there are 2 albums and 3 persons in the photo
+        #          there would be 6 possible renderings (2 albums x 3 persons)
+
+        # regex to find {template_field,optional_default} in strings
+        # for explanation of regex see https://regex101.com/r/4JJg42/1
+        # pylint: disable=anomalous-backslash-in-string
+        regex = r"(?<!\{)\{([^\\,}]+)(,{0,1}(([\w\-. ]+))?)(?=\}(?!\}))\}"
+        if type(template) is not str:
+            raise TypeError(f"template must be type str, not {type(template)}")
+
+        def make_subst_function(self, none_str, get_func=self.get_template_value):
+            """ returns: substitution function for use in re.sub 
+                none_str: value to use if substitution lookup is None and no default provided
+                get_func: function that gets the substitution value for a given template field
+                        default is get_template_value which handles the single-value fields """
+
+            # closure to capture photo, none_str in subst
+            def subst(matchobj):
+                groups = len(matchobj.groups())
+                if groups == 4:
+                    try:
+                        val = get_func(matchobj.group(1))
+                    except ValueError:
+                        return matchobj.group(0)
+
+                    if val is None:
+                        return (
+                            matchobj.group(3)
+                            if matchobj.group(3) is not None
+                            else none_str
+                        )
+                    else:
+                        return val
+                else:
+                    raise ValueError(
+                        f"Unexpected number of groups: expected 4, got {groups}"
+                    )
+
+            return subst
+
+        subst_func = make_subst_function(self, none_str)
+
+        # do the replacements
+        rendered = re.sub(regex, subst_func, template)
+
+        # do multi-valued placements
+        # start with the single string from phase 1 above then loop through all
+        # multi-valued fields and all values for each of those fields
+        # rendered_strings will be updated as each field is processed
+        # for example: if two albums, two keywords, and one person and template is:
+        # "{created.year}/{album}/{keyword}/{person}"
+        # rendered strings would do the following:
+        # start (created.year filled in phase 1)
+        #   ['2011/{album}/{keyword}/{person}']
+        # after processing albums:
+        #   ['2011/Album1/{keyword}/{person}',
+        #    '2011/Album2/{keyword}/{person}',]
+        # after processing keywords:
+        #   ['2011/Album1/keyword1/{person}',
+        #    '2011/Album1/keyword2/{person}',
+        #    '2011/Album2/keyword1/{person}',
+        #    '2011/Album2/keyword2/{person}',]
+        # after processing person:
+        #   ['2011/Album1/keyword1/person1',
+        #    '2011/Album1/keyword2/person1',
+        #    '2011/Album2/keyword1/person1',
+        #    '2011/Album2/keyword2/person1',]
+
+        rendered_strings = set([rendered])
+        for field in MULTI_VALUE_SUBSTITUTIONS:
+            # Build a regex that matches only the field being processed
+            re_str = r"(?<!\\)\{(" + field + r")(,(([\w\-. ]{0,})))?\}"
+            regex_multi = re.compile(re_str)
+
+            # holds each of the new rendered_strings, set() to avoid duplicates
+            new_strings = set()
+
+            for str_template in rendered_strings:
+                if regex_multi.search(str_template):
+                    values = self.get_template_value_multi(field, path_sep)
+                    for val in values:
+
+                        def lookup_template_value_multi(lookup_value):
+                            """ Closure passed to make_subst_function get_func 
+                                Capture val and field in the closure 
+                                Allows make_subst_function to be re-used w/o modification """
+                            if lookup_value == field:
+                                return val
+                            else:
+                                raise ValueError(f"Unexpected value: {lookup_value}")
+
+                        subst = make_subst_function(
+                            self, none_str, get_func=lookup_template_value_multi
+                        )
+                        new_string = regex_multi.sub(subst, str_template)
+                        new_strings.add(new_string)
+
+                    # update rendered_strings for the next field to process
+                    rendered_strings = new_strings
+
+        # find any {fields} that weren't replaced
+        unmatched = []
+        for rendered_str in rendered_strings:
+            unmatched.extend(
+                [
+                    no_match[0]
+                    for no_match in re.findall(regex, rendered_str)
+                    if no_match[0] not in unmatched
+                ]
+            )
+
+        # fix any escaped curly braces
+        rendered_strings = [
+            rendered_str.replace("{{", "{").replace("}}", "}")
+            for rendered_str in rendered_strings
+        ]
+
+        return rendered_strings, unmatched
+
+    def get_template_value(self, field):
+        """lookup value for template field (single-value template substitutions)
+
+        Args:
+            field: template field to find value for.
+        
+        Returns:
+            The matching template value (which may be None).
+
+        Raises:
+            ValueError if no rule exists for field.
+        """
+
+        # must be a valid keyword
+        if field == "name":
+            return pathlib.Path(self.photo.filename).stem
+
+        if field == "original_name":
+            return pathlib.Path(self.photo.original_filename).stem
+
+        if field == "title":
+            return self.photo.title
+
+        if field == "descr":
+            return self.photo.description
+
+        if field == "created.date":
+            return DateTimeFormatter(self.photo.date).date
+
+        if field == "created.year":
+            return DateTimeFormatter(self.photo.date).year
+
+        if field == "created.yy":
+            return DateTimeFormatter(self.photo.date).yy
+
+        if field == "created.mm":
+            return DateTimeFormatter(self.photo.date).mm
+
+        if field == "created.month":
+            return DateTimeFormatter(self.photo.date).month
+
+        if field == "created.mon":
+            return DateTimeFormatter(self.photo.date).mon
+
+        if field == "created.dd":
+            return DateTimeFormatter(self.photo.date).dd
+
+        if field == "created.dow":
+            return DateTimeFormatter(self.photo.date).dow
+
+        if field == "created.doy":
+            return DateTimeFormatter(self.photo.date).doy
+
+        if field == "modified.date":
+            return (
+                DateTimeFormatter(self.photo.date_modified).date
+                if self.photo.date_modified
+                else None
+            )
+
+        if field == "modified.year":
+            return (
+                DateTimeFormatter(self.photo.date_modified).year
+                if self.photo.date_modified
+                else None
+            )
+
+        if field == "modified.yy":
+            return (
+                DateTimeFormatter(self.photo.date_modified).yy
+                if self.photo.date_modified
+                else None
+            )
+
+        if field == "modified.mm":
+            return (
+                DateTimeFormatter(self.photo.date_modified).mm
+                if self.photo.date_modified
+                else None
+            )
+
+        if field == "modified.month":
+            return (
+                DateTimeFormatter(self.photo.date_modified).month
+                if self.photo.date_modified
+                else None
+            )
+
+        if field == "modified.mon":
+            return (
+                DateTimeFormatter(self.photo.date_modified).mon
+                if self.photo.date_modified
+                else None
+            )
+
+        if field == "modified.dd":
+            return (
+                DateTimeFormatter(self.photo.date_modified).dd
+                if self.photo.date_modified
+                else None
+            )
+
+        if field == "modified.doy":
+            return (
+                DateTimeFormatter(self.photo.date_modified).doy
+                if self.photo.date_modified
+                else None
+            )
+
+        if field == "place.name":
+            return self.photo.place.name if self.photo.place else None
+
+        if field == "place.country_code":
+            return self.photo.place.country_code if self.photo.place else None
+
+        if field == "place.name.country":
+            return (
+                self.photo.place.names.country[0]
+                if self.photo.place and self.photo.place.names.country
+                else None
+            )
+
+        if field == "place.name.state_province":
+            return (
+                self.photo.place.names.state_province[0]
+                if self.photo.place and self.photo.place.names.state_province
+                else None
+            )
+
+        if field == "place.name.city":
+            return (
+                self.photo.place.names.city[0]
+                if self.photo.place and self.photo.place.names.city
+                else None
+            )
+
+        if field == "place.name.area_of_interest":
+            return (
+                self.photo.place.names.area_of_interest[0]
+                if self.photo.place and self.photo.place.names.area_of_interest
+                else None
+            )
+
+        if field == "place.address":
+            return (
+                self.photo.place.address_str
+                if self.photo.place and self.photo.place.address_str
+                else None
+            )
+
+        if field == "place.address.street":
+            return (
+                self.photo.place.address.street
+                if self.photo.place and self.photo.place.address.street
+                else None
+            )
+
+        if field == "place.address.city":
+            return (
+                self.photo.place.address.city
+                if self.photo.place and self.photo.place.address.city
+                else None
+            )
+
+        if field == "place.address.state_province":
+            return (
+                self.photo.place.address.state_province
+                if self.photo.place and self.photo.place.address.state_province
+                else None
+            )
+
+        if field == "place.address.postal_code":
+            return (
+                self.photo.place.address.postal_code
+                if self.photo.place and self.photo.place.address.postal_code
+                else None
+            )
+
+        if field == "place.address.country":
+            return (
+                self.photo.place.address.country
+                if self.photo.place and self.photo.place.address.country
+                else None
+            )
+
+        if field == "place.address.country_code":
+            return (
+                self.photo.place.address.iso_country_code
+                if self.photo.place and self.photo.place.address.iso_country_code
+                else None
+            )
+
+        # if here, didn't get a match
+        raise ValueError(f"Unhandled template value: {field}")
+
+    def get_template_value_multi(self, field, path_sep):
+        """lookup value for template field (multi-value template substitutions)
+
+        Args:
+            field: template field to find value for.
+            path_sep: path separator to use for folder_album field
+        
+        Returns:
+            List of the matching template values or [None].
+
+        Raises:
+            ValueError if no rule exists for field.
+        """
+
+        """ return list of values for a multi-valued template field """
+        if field == "album":
+            values = self.photo.albums
+        elif field == "keyword":
+            values = self.photo.keywords
+        elif field == "person":
+            values = self.photo.persons
+            # remove any _UNKNOWN_PERSON values
+            values = [val for val in values if val != _UNKNOWN_PERSON]
+        elif field == "label":
+            values = self.photo.labels
+        elif field == "label_normalized":
+            values = self.photo.labels_normalized
+        elif field == "folder_album":
+            values = []
+            # photos must be in an album to be in a folder
+            for album in self.photo.album_info:
+                if album.folder_names:
+                    # album in folder
+                    folder = path_sep.join(album.folder_names)
+                    folder += path_sep + album.title
+                    values.append(folder)
+                else:
+                    # album not in folder
+                    values.append(album.title)
+        else:
+            raise ValueError(f"Unhandleded template value: {field}")
+
+        # If no values, insert None so code below will substite none_str for None
+        values = values or [None]
+        return values

osxphotos/placeinfo.py

@@ -87,19 +87,19 @@ class PLRevGeoLocationInfo:
         self.postalAddress = postalAddress
 
     def __eq__(self, other):
-        for field in [
-            "addressString",
-            "countryCode",
-            "isHome",
-            "compoundNames",
-            "compoundSecondaryNames",
-            "version",
-            "geoServiceProvider",
-            "postalAddress",
-        ]:
-            if getattr(self, field) != getattr(other, field):
-                return False
-        return True
+        return all(
+            getattr(self, field) == getattr(other, field)
+            for field in [
+                "addressString",
+                "countryCode",
+                "isHome",
+                "compoundNames",
+                "compoundSecondaryNames",
+                "version",
+                "geoServiceProvider",
+                "postalAddress",
+            ]
+        )
 
     def __ne__(self, other):
         return not self.__eq__(other)
@@ -151,21 +151,17 @@ class PLRevGeoMapItem:
         self.finalPlaceInfos = finalPlaceInfos
 
     def __eq__(self, other):
-        for field in ["sortedPlaceInfos", "finalPlaceInfos"]:
-            if getattr(self, field) != getattr(other, field):
-                return False
-        return True
+        return all(
+            getattr(self, field) == getattr(other, field)
+            for field in ["sortedPlaceInfos", "finalPlaceInfos"]
+        )
 
     def __ne__(self, other):
         return not self.__eq__(other)
 
     def __str__(self):
-        sortedPlaceInfos = []
-        finalPlaceInfos = []
-        for place in self.sortedPlaceInfos:
-            sortedPlaceInfos.append(str(place))
-        for place in self.finalPlaceInfos:
-            finalPlaceInfos.append(str(place))
+        sortedPlaceInfos = [str(place) for place in self.sortedPlaceInfos]
+        finalPlaceInfos = [str(place) for place in self.finalPlaceInfos]
         return (
             f"finalPlaceInfos: {finalPlaceInfos}, sortedPlaceInfos: {sortedPlaceInfos}"
         )
@@ -192,10 +188,10 @@ class PLRevGeoMapItemAdditionalPlaceInfo:
         self.dominantOrderType = dominantOrderType
 
     def __eq__(self, other):
-        for field in ["area", "name", "placeType", "dominantOrderType"]:
-            if getattr(self, field) != getattr(other, field):
-                return False
-        return True
+        return all(
+            getattr(self, field) == getattr(other, field)
+            for field in ["area", "name", "placeType", "dominantOrderType"]
+        )
 
     def __ne__(self, other):
         return not self.__eq__(other)
@@ -245,19 +241,19 @@ class CNPostalAddress:
         self._subLocality = _subLocality
 
     def __eq__(self, other):
-        for field in [
-            "_ISOCountryCode",
-            "_city",
-            "_country",
-            "_postalCode",
-            "_state",
-            "_street",
-            "_subAdministrativeArea",
-            "_subLocality",
-        ]:
-            if getattr(self, field) != getattr(other, field):
-                return False
-        return True
+        return all(
+            getattr(self, field) == getattr(other, field)
+            for field in [
+                "_ISOCountryCode",
+                "_city",
+                "_country",
+                "_postalCode",
+                "_state",
+                "_street",
+                "_subAdministrativeArea",
+                "_subLocality",
+            ]
+        )
 
     def __ne__(self, other):
         return not self.__eq__(other)
@@ -490,16 +486,14 @@ class PlaceInfo4(PlaceInfo):
             "names": self.names,
             "country_code": self.country_code,
         }
-        strval = "PlaceInfo(" + ", ".join([f"{k}='{v}'" for k, v in info.items()]) + ")"
-        return strval
+        return "PlaceInfo(" + ", ".join([f"{k}='{v}'" for k, v in info.items()]) + ")"
 
     def as_dict(self):
-        info = {
+        return {
             "name": self.name,
             "names": self.names._asdict(),
             "country_code": self.country_code,
         }
-        return info
 
 
 class PlaceInfo5(PlaceInfo):
@@ -541,7 +535,7 @@ class PlaceInfo5(PlaceInfo):
     @property
     def address(self):
         addr = self._plrevgeoloc.postalAddress
-        address = PostalAddress(
+        return PostalAddress(
             street=addr._street,
             sub_locality=addr._subLocality,
             city=addr._city,
@@ -551,7 +545,6 @@ class PlaceInfo5(PlaceInfo):
             country=addr._country,
             iso_country_code=addr._ISOCountryCode,
         )
-        return address
 
     def _process_place_info(self):
         """ Process sortedPlaceInfos to set self._name and self._names """
@@ -630,11 +623,10 @@ class PlaceInfo5(PlaceInfo):
             "address_str": self.address_str,
             "address": str(self.address),
         }
-        strval = "PlaceInfo(" + ", ".join([f"{k}='{v}'" for k, v in info.items()]) + ")"
-        return strval
+        return "PlaceInfo(" + ", ".join([f"{k}='{v}'" for k, v in info.items()]) + ")"
 
     def as_dict(self):
-        info = {
+        return {
             "name": self.name,
             "names": self.names._asdict(),
             "country_code": self.country_code,
@@ -642,4 +634,3 @@ class PlaceInfo5(PlaceInfo):
             "address_str": self.address_str,
             "address": self.address._asdict(),
         }
-        return info

osxphotos/utils.py

@@ -18,7 +18,7 @@ import CoreServices
 import objc
 from Foundation import *
 
-from osxphotos._applescript import AppleScript
+from .fileutil import FileUtil
 
 _DEBUG = False
 
@@ -118,60 +118,6 @@ def _dd_to_dms(dd):
     return int(deg_), int(min_), sec_
 
 
-def _hardlink_file(src, dest):
-    """ Hardlinks a file from src path to dest path 
-        src: source path as string 
-        dest: destination path as string
-        Raises exception if linking fails or either path is None """
-
-    if src is None or dest is None:
-        raise ValueError("src and dest must not be None", src, dest)
-
-    if not os.path.isfile(src):
-        raise FileNotFoundError("src file does not appear to exist", src)
-
-    # if error on copy, subprocess will raise CalledProcessError
-    try:
-        os.link(src, dest)
-    except Exception as e:
-        logging.critical(f"os.link returned error: {e}")
-        raise e
-
-
-def _copy_file(src, dest, norsrc=False):
-    """ Copies a file from src path to dest path 
-        src: source path as string 
-        dest: destination path as string
-        norsrc: (bool) if True, uses --norsrc flag with ditto so it will not copy
-                resource fork or extended attributes.  May be useful on volumes that
-                don't work with extended attributes (likely only certain SMB mounts)
-                default is False
-        Uses ditto to perform copy; will silently overwrite dest if it exists
-        Raises exception if copy fails or either path is None """
-
-    if src is None or dest is None:
-        raise ValueError("src and dest must not be None", src, dest)
-
-    if not os.path.isfile(src):
-        raise FileNotFoundError("src file does not appear to exist", src)
-
-    if norsrc:
-        command = ["/usr/bin/ditto", "--norsrc", src, dest]
-    else:
-        command = ["/usr/bin/ditto", src, dest]
-
-    # if error on copy, subprocess will raise CalledProcessError
-    try:
-        result = subprocess.run(command, check=True, stderr=subprocess.PIPE)
-    except subprocess.CalledProcessError as e:
-        logging.critical(
-            f"ditto returned error: {e.returncode} {e.stderr.decode(sys.getfilesystemencoding()).rstrip()}"
-        )
-        raise e
-
-    return result.returncode
-
-
 def dd_to_dms_str(lat, lon):
     """ convert latitude, longitude in degrees to degrees, minutes, seconds as string """
     """ lat: latitude in degrees  """
@@ -308,24 +254,6 @@ def list_photo_libraries():
     return lib_list
 
 
-def create_path_by_date(dest, dt):
-    """ Creates a path in dest folder in form dest/YYYY/MM/DD/
-        dest: valid path as str
-        dt: datetime.timetuple() object
-        Checks to see if path exists, if it does, do nothing and return path
-        If path does not exist, creates it and returns path"""
-    if not os.path.isdir(dest):
-        raise FileNotFoundError(f"dest {dest} must be valid path")
-    yyyy, mm, dd = dt[0:3]
-    yyyy = str(yyyy).zfill(4)
-    mm = str(mm).zfill(2)
-    dd = str(dd).zfill(2)
-    new_dest = os.path.join(dest, yyyy, mm, dd)
-    if not os.path.isdir(new_dest):
-        os.makedirs(new_dest)
-    return new_dest
-
-
 def get_preferred_uti_extension(uti):
     """ get preferred extension for a UTI type
         uti: UTI str, e.g. 'public.jpeg'
@@ -333,10 +261,9 @@ def get_preferred_uti_extension(uti):
 
     # reference: https://developer.apple.com/documentation/coreservices/1442744-uttypecopypreferredtagwithclass?language=objc
 
-    ext = CoreServices.UTTypeCopyPreferredTagWithClass(
+    return CoreServices.UTTypeCopyPreferredTagWithClass(
         uti, CoreServices.kUTTagClassFilenameExtension
     )
-    return ext
 
 
 def findfiles(pattern, path_):
@@ -365,117 +292,6 @@ def findfiles(pattern, path_):
 #     open_scpt.run()
 
 
-def _export_photo_uuid_applescript(
-    uuid,
-    dest,
-    filestem=None,
-    original=True,
-    edited=False,
-    live_photo=False,
-    timeout=120,
-    burst=False,
-):
-    """ Export photo to dest path using applescript to control Photos
-        If photo is a live photo, exports both the photo and associated .mov file
-        uuid: UUID of photo to export
-        dest: destination path to export to
-        filestem: (string) if provided, exported filename will be named stem.ext 
-                  where ext is extension of the file exported by photos (e.g. .jpeg, .mov, etc)
-                  If not provided, file will be named with whatever name Photos uses
-                  If filestem.ext exists, it wil be overwritten
-        original: (boolean) if True, export original image; default = True
-        edited: (boolean) if True, export edited photo; default = False
-                If photo not edited and edited=True, will still export the original image
-                caller must verify image has been edited
-        *Note*: must be called with either edited or original but not both, 
-                will raise error if called with both edited and original = True
-        live_photo: (boolean) if True, export associated .mov live photo; default = False
-        timeout: timeout value in seconds; export will fail if applescript run time exceeds timeout
-        burst: (boolean) set to True if file is a burst image to avoid Photos export error
-        Returns: list of paths to exported file(s) or None if export failed
-        Note: For Live Photos, if edited=True, will export a jpeg but not the movie, even if photo
-              has not been edited. This is due to how Photos Applescript interface works.
-    """
-
-    # setup the applescript to do the export
-    export_scpt = AppleScript(
-        """ 
-		on export_by_uuid(theUUID, thePath, original, edited, theTimeOut)
-			tell application "Photos"
-				set thePath to thePath
-				set theItem to media item id theUUID
-				set theFilename to filename of theItem
-				set itemList to {theItem}
-				
-				if original then
-					with timeout of theTimeOut seconds
-						export itemList to POSIX file thePath with using originals
-					end timeout
-				end if
-				
-				if edited then
-					with timeout of theTimeOut seconds
-						export itemList to POSIX file thePath
-					end timeout
-				end if
-				
-				return theFilename
-			end tell
-
-		end export_by_uuid
-		"""
-    )
-
-    dest = pathlib.Path(dest)
-    if not dest.is_dir:
-        raise ValueError(f"dest {dest} must be a directory")
-
-    if not original ^ edited:
-        raise ValueError(f"edited or original must be True but not both")
-
-    tmpdir = tempfile.TemporaryDirectory(prefix="osxphotos_")
-
-    # export original
-    filename = None
-    try:
-        filename = export_scpt.call(
-            "export_by_uuid", uuid, tmpdir.name, original, edited, timeout
-        )
-    except Exception as e:
-        logging.warning("Error exporting uuid %s: %s" % (uuid, str(e)))
-        return None
-
-    if filename is not None:
-        # need to find actual filename as sometimes Photos renames JPG to jpeg on export
-        # may be more than one file exported (e.g. if Live Photo, Photos exports both .jpeg and .mov)
-        # TemporaryDirectory will cleanup on return
-        filename_stem = pathlib.Path(filename).stem
-        files = glob.glob(os.path.join(tmpdir.name, "*"))
-        exported_paths = []
-        for fname in files:
-            path = pathlib.Path(fname)
-            if len(files) > 1 and not live_photo and path.suffix.lower() == ".mov":
-                # it's the .mov part of live photo but not requested, so don't export
-                logging.debug(f"Skipping live photo file {path}")
-                continue
-            if len(files) > 1 and burst and path.stem != filename_stem:
-                # skip any burst photo that's not the one we asked for
-                logging.debug(f"Skipping burst photo file {path}")
-                continue
-            if filestem:
-                # rename the file based on filestem, keeping original extension
-                dest_new = dest / f"{filestem}{path.suffix}"
-            else:
-                # use the name Photos provided
-                dest_new = dest / path.name
-            logging.debug(f"exporting {path} to dest_new: {dest_new}")
-            _copy_file(str(path), str(dest_new))
-            exported_paths.append(str(dest_new))
-        return exported_paths
-    else:
-        return None
-
-
 def _open_sql_file(dbname):
     """ opens sqlite file dbname in read-only mode
         returns tuple of (connection, cursor) """
@@ -527,7 +343,7 @@ def _db_is_locked(dbname):
 #     except KeyError:
 #         uuid_str = None
 #     return uuid_str
-    
+
 # def set_uuid_for_file(filepath, uuid):
 #     """ sets the UUID associated with an exported file
 #         filepath: path to exported photo

requirements.txt

@@ -29,8 +29,6 @@ pluggy==0.12.0
 py==1.8.0
 py2app==0.21
 Pygments==2.4.2
-PyInstaller==3.6
-pyinstaller-setuptools==2019.3
 pylint==2.3.1
 pyobjc==6.0.1
 pyobjc-core==6.0.1

tests/Test-10.15.1.photoslibrary/database/Photos.sqlite.lock

@@ -7,7 +7,7 @@
 	<key>hostuuid</key>
 	<string>9575E48B-8D5F-5654-ABAC-4431B1167324</string>
 	<key>pid</key>
-	<integer>725</integer>
+	<integer>721</integer>
 	<key>processname</key>
 	<string>photolibraryd</string>
 	<key>uid</key>

tests/Test-10.15.1.photoslibrary/private/com.apple.photoanalysisd/caches/graph/PhotoAnalysisServicePreferences.plist

@@ -3,24 +3,24 @@
 <plist version="1.0">
 <dict>
 	<key>BackgroundHighlightCollection</key>
-	<date>2020-05-01T23:03:12Z</date>
+	<date>2020-05-25T17:08:24Z</date>
 	<key>BackgroundHighlightEnrichment</key>
-	<date>2020-05-01T23:03:11Z</date>
+	<date>2020-05-25T17:08:24Z</date>
 	<key>BackgroundJobAssetRevGeocode</key>
-	<date>2020-05-02T01:35:19Z</date>
+	<date>2020-05-25T17:08:25Z</date>
 	<key>BackgroundJobSearch</key>
-	<date>2020-05-01T23:03:12Z</date>
+	<date>2020-05-25T17:08:25Z</date>
 	<key>BackgroundPeopleSuggestion</key>
-	<date>2020-05-01T23:03:11Z</date>
+	<date>2020-05-25T17:08:24Z</date>
 	<key>BackgroundUserBehaviorProcessor</key>
-	<date>2020-05-01T23:03:13Z</date>
+	<date>2020-05-25T17:08:25Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundGraphConsistencyUpdateJobDateKey</key>
-	<date>2020-05-02T01:35:36Z</date>
+	<date>2020-05-25T17:12:44Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundGraphRebuildJobDate</key>
-	<date>2020-05-01T23:03:11Z</date>
+	<date>2020-05-25T17:08:24Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundMemoryGenerationJobDate</key>
-	<date>2020-05-02T01:35:19Z</date>
+	<date>2020-05-25T17:08:25Z</date>
 	<key>SiriPortraitDonation</key>
-	<date>2020-05-01T23:03:13Z</date>
+	<date>2020-05-25T17:08:25Z</date>
 </dict>
 </plist>

tests/Test-10.15.1.photoslibrary/private/com.apple.photoanalysisd/caches/vision/PhotoAnalysisServicePreferences.plist

@@ -3,8 +3,8 @@
 <plist version="1.0">
 <dict>
 	<key>FaceIDModelLastGenerationKey</key>
-	<date>2020-05-01T23:03:14Z</date>
+	<date>2020-05-25T17:08:25Z</date>
 	<key>LastContactClassificationKey</key>
-	<date>2020-05-01T23:03:18Z</date>
+	<date>2020-05-25T17:08:27Z</date>
 </dict>
 </plist>

tests/Test-10.15.5.photoslibrary/database/DataModelVersion.plist

@@ -0,0 +1,10 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>LibrarySchemaVersion</key>
+	<integer>5001</integer>
+	<key>MetaSchemaVersion</key>
+	<integer>3</integer>
+</dict>
+</plist>

tests/Test-10.15.5.photoslibrary/database/Photos.sqlite.lock

@@ -0,0 +1,16 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>hostname</key>
+	<string>Rhets-MacBook-Pro.local</string>
+	<key>hostuuid</key>
+	<string>9575E48B-8D5F-5654-ABAC-4431B1167324</string>
+	<key>pid</key>
+	<integer>703</integer>
+	<key>processname</key>
+	<string>photolibraryd</string>
+	<key>uid</key>
+	<integer>501</integer>
+</dict>
+</plist>

tests/Test-10.15.5.photoslibrary/database/search/searchMetadata.plist

@@ -0,0 +1,188 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>BlacklistedMeaningsByMeaning</key>
+	<dict/>
+	<key>MePersonUUID</key>
+	<string>39488755-78C0-40B2-B378-EDA280E1823C</string>
+	<key>SceneWhitelist</key>
+	<array>
+		<string>Graduation</string>
+		<string>Aquarium</string>
+		<string>Food</string>
+		<string>Ice Skating</string>
+		<string>Mountain</string>
+		<string>Cliff</string>
+		<string>Basketball</string>
+		<string>Tennis</string>
+		<string>Jewelry</string>
+		<string>Cheese</string>
+		<string>Softball</string>
+		<string>Football</string>
+		<string>Circus</string>
+		<string>Jet Ski</string>
+		<string>Playground</string>
+		<string>Carousel</string>
+		<string>Paint Ball</string>
+		<string>Windsurfing</string>
+		<string>Sailboat</string>
+		<string>Sunbathing</string>
+		<string>Dam</string>
+		<string>Fireplace</string>
+		<string>Flower</string>
+		<string>Scuba</string>
+		<string>Hiking</string>
+		<string>Cetacean</string>
+		<string>Pier</string>
+		<string>Bowling</string>
+		<string>Snowboarding</string>
+		<string>Zoo</string>
+		<string>Snowmobile</string>
+		<string>Theater</string>
+		<string>Boat</string>
+		<string>Casino</string>
+		<string>Car</string>
+		<string>Diving</string>
+		<string>Cycling</string>
+		<string>Musical Instrument</string>
+		<string>Board Game</string>
+		<string>Castle</string>
+		<string>Sunset Sunrise</string>
+		<string>Martial Arts</string>
+		<string>Motocross</string>
+		<string>Submarine</string>
+		<string>Cat</string>
+		<string>Snow</string>
+		<string>Kiteboarding</string>
+		<string>Squash</string>
+		<string>Geyser</string>
+		<string>Music</string>
+		<string>Archery</string>
+		<string>Desert</string>
+		<string>Blackjack</string>
+		<string>Fireworks</string>
+		<string>Sportscar</string>
+		<string>Feline</string>
+		<string>Soccer</string>
+		<string>Museum</string>
+		<string>Baby</string>
+		<string>Fencing</string>
+		<string>Railroad</string>
+		<string>Nascar</string>
+		<string>Sky Surfing</string>
+		<string>Bird</string>
+		<string>Games</string>
+		<string>Baseball</string>
+		<string>Dressage</string>
+		<string>Snorkeling</string>
+		<string>Pyramid</string>
+		<string>Kite</string>
+		<string>Rowboat</string>
+		<string>Golf</string>
+		<string>Watersports</string>
+		<string>Lightning</string>
+		<string>Canyon</string>
+		<string>Auditorium</string>
+		<string>Night Sky</string>
+		<string>Karaoke</string>
+		<string>Skiing</string>
+		<string>Parade</string>
+		<string>Forest</string>
+		<string>Hot Air Balloon</string>
+		<string>Dragon Parade</string>
+		<string>Easter Egg</string>
+		<string>Monument</string>
+		<string>Jungle</string>
+		<string>Thanksgiving</string>
+		<string>Jockey Horse</string>
+		<string>Stadium</string>
+		<string>Airplane</string>
+		<string>Ballet</string>
+		<string>Yoga</string>
+		<string>Coral Reef</string>
+		<string>Skating</string>
+		<string>Wrestling</string>
+		<string>Bicycle</string>
+		<string>Tattoo</string>
+		<string>Amusement Park</string>
+		<string>Canoe</string>
+		<string>Cheerleading</string>
+		<string>Ping Pong</string>
+		<string>Fishing</string>
+		<string>Magic</string>
+		<string>Reptile</string>
+		<string>Winter Sport</string>
+		<string>Waterfall</string>
+		<string>Train</string>
+		<string>Bonsai</string>
+		<string>Surfing</string>
+		<string>Dog</string>
+		<string>Cake</string>
+		<string>Sledding</string>
+		<string>Sandcastle</string>
+		<string>Glacier</string>
+		<string>Lighthouse</string>
+		<string>Equestrian</string>
+		<string>Rafting</string>
+		<string>Shore</string>
+		<string>Hockey</string>
+		<string>Santa Claus</string>
+		<string>Formula One Car</string>
+		<string>Sport</string>
+		<string>Vehicle</string>
+		<string>Boxing</string>
+		<string>Rollerskating</string>
+		<string>Underwater</string>
+		<string>Orchestra</string>
+		<string>Carnival</string>
+		<string>Rocket</string>
+		<string>Skateboarding</string>
+		<string>Helicopter</string>
+		<string>Performance</string>
+		<string>Oktoberfest</string>
+		<string>Water Polo</string>
+		<string>Skate Park</string>
+		<string>Animal</string>
+		<string>Nightclub</string>
+		<string>String Instrument</string>
+		<string>Dinosaur</string>
+		<string>Gymnastics</string>
+		<string>Cricket</string>
+		<string>Volcano</string>
+		<string>Lake</string>
+		<string>Aurora</string>
+		<string>Dancing</string>
+		<string>Concert</string>
+		<string>Rock Climbing</string>
+		<string>Hang Glider</string>
+		<string>Rodeo</string>
+		<string>Fish</string>
+		<string>Art</string>
+		<string>Motorcycle</string>
+		<string>Volleyball</string>
+		<string>Wake Boarding</string>
+		<string>Badminton</string>
+		<string>Motor Sport</string>
+		<string>Sumo</string>
+		<string>Parasailing</string>
+		<string>Skydiving</string>
+		<string>Kickboxing</string>
+		<string>Pinata</string>
+		<string>Foosball</string>
+		<string>Go Kart</string>
+		<string>Poker</string>
+		<string>Kayak</string>
+		<string>Swimming</string>
+		<string>Atv</string>
+		<string>Beach</string>
+		<string>Dartboard</string>
+		<string>Athletics</string>
+		<string>Camping</string>
+		<string>Tornado</string>
+		<string>Billiards</string>
+		<string>Rugby</string>
+		<string>Airshow</string>
+	</array>
+</dict>
+</plist>

tests/Test-10.15.5.photoslibrary/database/search/searchProgress.plist

@@ -0,0 +1,26 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>insertAlbum</key>
+	<array/>
+	<key>insertAsset</key>
+	<array/>
+	<key>insertHighlight</key>
+	<array/>
+	<key>insertMemory</key>
+	<array/>
+	<key>insertMoment</key>
+	<array/>
+	<key>removeAlbum</key>
+	<array/>
+	<key>removeAsset</key>
+	<array/>
+	<key>removeHighlight</key>
+	<array/>
+	<key>removeMemory</key>
+	<array/>
+	<key>removeMoment</key>
+	<array/>
+</dict>
+</plist>

tests/Test-10.15.5.photoslibrary/database/search/searchSystemInfo.plist

@@ -0,0 +1,14 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>embeddingVersion</key>
+	<string>1</string>
+	<key>localeIdentifier</key>
+	<string>en_US</string>
+	<key>sceneTaxonomySHA</key>
+	<string>87914a047c69fbe8013fad2c70fa70c6c03b08b56190fe4054c880e6b9f57cc3</string>
+	<key>searchIndexVersion</key>
+	<string>10</string>
+</dict>
+</plist>

tests/Test-10.15.5.photoslibrary/private/com.apple.Photos.Migration/appPrivateData.plist

@@ -0,0 +1,26 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>MigrationService</key>
+	<dict>
+		<key>State</key>
+		<integer>4</integer>
+	</dict>
+	<key>MigrationService.LastCompletedTask</key>
+	<integer>12</integer>
+	<key>MigrationService.ValidationCounts</key>
+	<dict>
+		<key>MigrationDetectedFaceprint</key>
+		<integer>6</integer>
+		<key>MigrationManagedAsset</key>
+		<integer>0</integer>
+		<key>MigrationSceneClassification</key>
+		<integer>44</integer>
+		<key>MigrationUnmanagedAdjustment</key>
+		<integer>0</integer>
+		<key>RDVersion.cloudLocalState.CPLIsNotPushed</key>
+		<integer>7</integer>
+	</dict>
+</dict>
+</plist>

tests/Test-10.15.5.photoslibrary/private/com.apple.Photos/appPrivateData.plist

@@ -0,0 +1,52 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>CollapsedSidebarSectionIdentifiers</key>
+	<array/>
+	<key>ExpandedSidebarItemIdentifiers</key>
+	<array>
+		<string>92D68107-B6C7-453B-96D2-97B0F26D5B8B/L0/020</string>
+		<string>88A5F8B8-5B9A-43C7-BB85-3952B81580EB/L0/020</string>
+		<string>29EF7A97-7E76-4D5F-A5E0-CC0A93E8524C/L0/020</string>
+		<string>2C2AF115-BD1D-4434-A747-D1C8BD8E2045/L0/020</string>
+	</array>
+	<key>Photos</key>
+	<dict>
+		<key>CollapsedSidebarSectionIdentifiers</key>
+		<array/>
+		<key>ExpandedSidebarItemIdentifiers</key>
+		<array>
+			<string>TopLevelAlbums</string>
+			<string>TopLevelSlideshows</string>
+		</array>
+		<key>IPXWorkspaceControllerZoomLevelsKey</key>
+		<dict>
+			<key>kZoomLevelIdentifierAlbums</key>
+			<integer>7</integer>
+			<key>kZoomLevelIdentifierVersions</key>
+			<integer>7</integer>
+		</dict>
+		<key>lastAddToDestination</key>
+		<dict>
+			<key>key</key>
+			<integer>1</integer>
+			<key>lastKnownDisplayName</key>
+			<string>September 28, 2018</string>
+			<key>type</key>
+			<string>album</string>
+			<key>uuid</key>
+			<string>DFFKmHt3Tk+AGzZLe2Xq+g</string>
+		</dict>
+		<key>lastKnownItemCounts</key>
+		<dict>
+			<key>other</key>
+			<integer>0</integer>
+			<key>photos</key>
+			<integer>7</integer>
+			<key>videos</key>
+			<integer>0</integer>
+		</dict>
+	</dict>
+</dict>
+</plist>