Added --missing to export, see issue #277

Add XML escaping to XMP sidecar export, thanks to @jstrine for the fix!

.all-contributorsrc

@@ -100,6 +100,15 @@
       "contributions": [
         "code"
       ]
+    },
+    {
+      "login": "jstrine",
+      "name": "Jonathan Strine",
+      "avatar_url": "https://avatars1.githubusercontent.com/u/33943447?v=4",
+      "profile": "https://github.com/jstrine",
+      "contributions": [
+        "code"
+      ]
     }
   ],
   "contributorsPerLine": 7

CHANGELOG.md

@@ -4,6 +4,147 @@ All notable changes to this project will be documented in this file. Dates are d
 
 Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 
+#### [v0.37.1](https://github.com/RhetTbull/osxphotos/compare/v0.37.0...v0.37.1)
+
+> 28 November 2020
+
+- Added --report option to CLI, implements #253 [`d22eaf3`](https://github.com/RhetTbull/osxphotos/commit/d22eaf39edc8b0b489b011d6d21345dcedcc8dff)
+- Updated template values [`af827d7`](https://github.com/RhetTbull/osxphotos/commit/af827d7a5769f41579d300a7cc511251d86b7eed)
+
+#### [v0.37.0](https://github.com/RhetTbull/osxphotos/compare/v0.36.25...v0.37.0)
+
+> 28 November 2020
+
+- Added {exiftool} template, implements issue #259 [`48acb42`](https://github.com/RhetTbull/osxphotos/commit/48acb42631226a71bfc636eea2d3151f1b7165f4)
+
+#### [v0.36.25](https://github.com/RhetTbull/osxphotos/compare/v0.36.24...v0.36.25)
+
+> 27 November 2020
+
+- Added --original-suffix for issue #263 [`399d432`](https://github.com/RhetTbull/osxphotos/commit/399d432a66354b9c235f30d10c6985fbde1b7e4f)
+
+#### [v0.36.24](https://github.com/RhetTbull/osxphotos/compare/v0.36.23...v0.36.24)
+
+> 26 November 2020
+
+- Initial implementation for issue #265 [`382fca3`](https://github.com/RhetTbull/osxphotos/commit/382fca3f92a3c251c12426dd0dc6d7dc21b691cf)
+- More work on issue #265 [`d5a9f76`](https://github.com/RhetTbull/osxphotos/commit/d5a9f767199d25ebd9d5925d05ee39ea7e51ac26)
+- Simplified sidecar table in export_db [`0632a97`](https://github.com/RhetTbull/osxphotos/commit/0632a97f55af67c7e5265b0d3283155c7c087e89)
+
+#### [v0.36.23](https://github.com/RhetTbull/osxphotos/compare/v0.36.22...v0.36.23)
+
+> 26 November 2020
+
+- Fix for missing original_filename, issue #267 [`fa33218`](https://github.com/RhetTbull/osxphotos/commit/fa332186ab3cdbe1bfd6496ff29b652ef984a5f8)
+- version bump [`b5195f9`](https://github.com/RhetTbull/osxphotos/commit/b5195f9d2b81cf6737b65e3cd3793ea9b0da13eb)
+- Updated test [`aa2ebf5`](https://github.com/RhetTbull/osxphotos/commit/aa2ebf55bb50eec14f86a532334b376e407f4bbc)
+
+#### [v0.36.22](https://github.com/RhetTbull/osxphotos/compare/v0.36.21...v0.36.22)
+
+> 26 November 2020
+
+- Add XML escaping to XMP sidecar export, thanks to @jstrine for the fix! [`#272`](https://github.com/RhetTbull/osxphotos/pull/272)
+- Fix EXIF GPS format for XMP sidecar, thanks to @jstrine for the fix! [`#270`](https://github.com/RhetTbull/osxphotos/pull/270)
+- Continue even if the original filename is None, thanks to @jstrine for the fix! [`#268`](https://github.com/RhetTbull/osxphotos/pull/268)
+- Added test for missing original_filename [`116cb66`](https://github.com/RhetTbull/osxphotos/commit/116cb662fbddf9153f6858c6ea97dc7f65c77705)
+-  Add @jstrine as a contributor [`7460bc8`](https://github.com/RhetTbull/osxphotos/commit/7460bc88fcc5e1e7435c9b9bcdf7ec9c7c5e39ea)
+- Escape characters which cause XML parsing issues [`c42050a`](https://github.com/RhetTbull/osxphotos/commit/c42050a10cac40b0b5ac70c587e07f257a9b50dd)
+- Fix tests for apostrophe [`d0d2e80`](https://github.com/RhetTbull/osxphotos/commit/d0d2e8080096bf66f93a830386800ce713680c51)
+- Fix test for XMP sidecar with GPS info [`c27cfb1`](https://github.com/RhetTbull/osxphotos/commit/c27cfb1223fa82b9e5549b93c283e9444693270a)
+
+#### [v0.36.21](https://github.com/RhetTbull/osxphotos/compare/v0.36.20...v0.36.21)
+
+> 25 November 2020
+
+- Exposed --use-photos-export and --use-photokit [`e951e53`](https://github.com/RhetTbull/osxphotos/commit/e951e5361e59060229787bb1ea3fc4e088ffff99)
+
+#### [v0.36.20](https://github.com/RhetTbull/osxphotos/compare/v0.36.19...v0.36.20)
+
+> 23 November 2020
+
+- Added photokit export as hidden --use-photokit option [`26f96d5`](https://github.com/RhetTbull/osxphotos/commit/26f96d582c01ce9816b1f54f0e74c8570f133f7c)
+
+#### [v0.36.19](https://github.com/RhetTbull/osxphotos/compare/v0.36.18...v0.36.19)
+
+> 19 November 2020
+
+- Removed debug statement in _photoinfo_export [`8cb15d1`](https://github.com/RhetTbull/osxphotos/commit/8cb15d15551094dcaf1b0ef32d6ac0273be7fd37)
+
+#### [v0.36.18](https://github.com/RhetTbull/osxphotos/compare/v0.36.17...v0.36.18)
+
+> 14 November 2020
+
+- Moved AppleScript to photoscript [`3c85f26`](https://github.com/RhetTbull/osxphotos/commit/3c85f26f901645ce297685ccd639792757fbc995)
+- Fixed missing data file for photoscript [`2d9429c`](https://github.com/RhetTbull/osxphotos/commit/2d9429c8eefabe6233fc580f65511c48ee6c01e5)
+- Version bump, updated requirements [`3b6dd08`](https://github.com/RhetTbull/osxphotos/commit/3b6dd08d2bb2b20a55064bf24fe7ce788e7268ef)
+
+#### [v0.36.17](https://github.com/RhetTbull/osxphotos/compare/v0.36.15...v0.36.17)
+
+> 12 November 2020
+
+- Fixed path for photos actually missing off disk [`5d4d7d7`](https://github.com/RhetTbull/osxphotos/commit/5d4d7d7db7ca1109b6230803fe777d7a30882efe)
+- Fixed erroneous attempt to export edited with --download-missing [`8dc59cb`](https://github.com/RhetTbull/osxphotos/commit/8dc59cbc35c33e71d0d912f4139e855180ac4fbd)
+- version bump [`802e2f0`](https://github.com/RhetTbull/osxphotos/commit/802e2f069a5f8b37ddc6b3b8ba07519ce10f88a7)
+
+#### [v0.36.15](https://github.com/RhetTbull/osxphotos/compare/v0.36.14...v0.36.15)
+
+> 11 November 2020
+
+- Avoid copying db files if not necessary [`ea9b41b`](https://github.com/RhetTbull/osxphotos/commit/ea9b41bae41a05aad53454f67871c5e6c9a49f79)
+
+#### [v0.36.14](https://github.com/RhetTbull/osxphotos/compare/v0.36.13...v0.36.14)
+
+> 9 November 2020
+
+- Fix for issue #247 [`38397b5`](https://github.com/RhetTbull/osxphotos/commit/38397b507b456169cf3be2d2dc6743ec8653feb3)
+
+#### [v0.36.13](https://github.com/RhetTbull/osxphotos/compare/v0.36.11...v0.36.13)
+
+> 9 November 2020
+
+- Refactored phototemplate.py to add PATH_SEP option [`3636fcb`](https://github.com/RhetTbull/osxphotos/commit/3636fcbc76100d9898a59f24ed6e9b1965cc6022)
+- More work on phototemplate.py to add inline expansion [`a6231e2`](https://github.com/RhetTbull/osxphotos/commit/a6231e29ff28b2c7dc3239445f41afcb35926a7a)
+
+#### [v0.36.11](https://github.com/RhetTbull/osxphotos/compare/v0.36.10...v0.36.11)
+
+> 8 November 2020
+
+- Implemented boolean type template fields [`7fa3704`](https://github.com/RhetTbull/osxphotos/commit/7fa3704840f7800689b4ac5f8edee8210eb3e8db)
+- Bug fix in handling missing edited photos [`e829212`](https://github.com/RhetTbull/osxphotos/commit/e829212987bbc1a88f845922abcffef70c159883)
+- Fixed message in CLI [`df37a01`](https://github.com/RhetTbull/osxphotos/commit/df37a017a8efdc8d0b9bc8d00a4452dc4cb892b3)
+
+#### [v0.36.10](https://github.com/RhetTbull/osxphotos/compare/v0.36.9...v0.36.10)
+
+> 8 November 2020
+
+- Implemented issue #255 [`ae2fd2e`](https://github.com/RhetTbull/osxphotos/commit/ae2fd2e3db984756e6cc3f7b3338b8ba819ce28c)
+
+#### [v0.36.9](https://github.com/RhetTbull/osxphotos/compare/v0.36.8...v0.36.9)
+
+> 7 November 2020
+
+- Refactored regex in phototemplate [`653b7e6`](https://github.com/RhetTbull/osxphotos/commit/653b7e6600e0738ecd00f74d510a893e0d447ca4)
+- Fix for exporting slow mo videos, issue #252 [`9d38885`](https://github.com/RhetTbull/osxphotos/commit/9d38885416b528bd8c91bb09120be85a8b109f29)
+
+#### [v0.36.8](https://github.com/RhetTbull/osxphotos/compare/v0.36.7...v0.36.8)
+
+> 5 November 2020
+
+- Refactored exiftool.py [`2202f1b`](https://github.com/RhetTbull/osxphotos/commit/2202f1b1e9c4f83558ef48e58cb94af6b3a38cdd)
+- README.md update [`a509ef1`](https://github.com/RhetTbull/osxphotos/commit/a509ef18d3db2ac15a661e763a7254974cf8d84a)
+
+#### [v0.36.7](https://github.com/RhetTbull/osxphotos/compare/v0.36.6...v0.36.7)
+
+> 4 November 2020
+
+- Implemented context manager for ExifTool, closes #250 [`#250`](https://github.com/RhetTbull/osxphotos/issues/250)
+
+#### [v0.36.6](https://github.com/RhetTbull/osxphotos/compare/v0.36.5...v0.36.6)
+
+> 2 November 2020
+
+- Fix for issue #39 [`c7c5320`](https://github.com/RhetTbull/osxphotos/commit/c7c5320587e31070b55cc8c7e74f30b0f9e61379)
+
 #### [v0.36.5](https://github.com/RhetTbull/osxphotos/compare/v0.36.4...v0.36.5)
 
 > 1 November 2020

README.md

@@ -3,7 +3,7 @@
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![Python package](https://github.com/RhetTbull/osxphotos/workflows/Python%20package/badge.svg)](https://github.com/RhetTbull/osxphotos/workflows/Python%20package/badge.svg)
 <!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->
-[![All Contributors](https://img.shields.io/badge/all_contributors-10-orange.svg?style=flat-square)](#contributors-)
+[![All Contributors](https://img.shields.io/badge/all_contributors-11-orange.svg?style=flat-square)](#contributors-)
 <!-- ALL-CONTRIBUTORS-BADGE:END -->
 
 - [OSXPhotos](#osxphotos)
@@ -227,6 +227,9 @@ Options:
   --no-comment                    Search for photos with no comments.
   --has-likes                     Search for photos that have likes.
   --no-likes                      Search for photos with no likes.
+  --missing                       Export only photos missing from the Photos
+                                  library; must be used with --download-
+                                  missing.
   --deleted                       Include photos from the 'Recently Deleted'
                                   folder.
   --deleted-only                  Include only photos from the 'Recently
@@ -234,7 +237,8 @@ Options:
   --update                        Only export new or updated files. See notes
                                   below on export and --update.
   --dry-run                       Dry run (test) the export but don't actually
-                                  export any files; most useful with --verbose
+                                  export any files; most useful with
+                                  --verbose.
   --export-as-hardlink            Hardlink files instead of copying them.
                                   Cannot be used with --exiftool which creates
                                   copies of the files with embedded EXIF data.
@@ -351,11 +355,30 @@ Options:
                                   photo would be named
                                   'photoname_bearbeiten.ext'.  The default
                                   suffix is '_edited'.
+  --original-suffix SUFFIX        Optional suffix for naming original photos.
+                                  Default name for original photos is in form
+                                  'filename.ext'. For example, with '--
+                                  original-suffix _original', the original
+                                  photo would be named
+                                  'filename_original.ext'.  The default suffix
+                                  is '' (no suffix).
   --no-extended-attributes        Don't copy extended attributes when
                                   exporting.  You only need this if exporting
                                   to a filesystem that doesn't support Mac OS
                                   extended attributes.  Only use this if you
                                   get an error while exporting.
+  --use-photos-export             Force the use of AppleScript or PhotoKit to
+                                  export even if not missing (see also '--
+                                  download-missing' and '--use-photokit').
+  --use-photokit                  Use with '--download-missing' or '--use-
+                                  photos-export' to use direct Photos
+                                  interface instead of AppleScript to export.
+                                  Highly experimental alpha feature; does not
+                                  work with iTerm2 (use with Terminal.app).
+                                  This is faster and more reliable than the
+                                  default AppleScript interface.
+  --report REPORTNAME.CSV         Write a CSV formatted report of all files
+                                  that were exported.
   -h, --help                      Show this message and exit.
 
 ** Export **
@@ -388,16 +411,72 @@ option to re-export the entire library thus rebuilding the
 
 ** Templating System **
 
+Several options, such as --directory, allow you to specify a template  which
+will be rendered to substitute template fields with values from the photo.
+For example, '{created.month}' would be replaced with the month name of the
+photo creation date.  e.g. 'November'.
+
+The general format for a template is '{TEMPLATE_FIELD[,[DEFAULT]]}'.  Some
+templates have optional modifiers in form
+'{[[DELIM]+]TEMPLATE_FIELD[(PATH_SEP)][?VALUE_IF_TRUE][,[DEFAULT]]}'
+
+The ',' and DEFAULT value are optional.  If TEMPLATE_FIELD results in a null
+(empty) value, the default is '_'.   You may specify an alternate default
+value by appending ',DEFAULT' after template_field.  e.g. '{title,no_title}'
+would result in 'no_title' if the photo had no title.  You may include other
+text in the template string outside the {} and use more than  one template
+field, e.g. '{created.year} - {created.month}' (e.g. '2020 - November').
+
+Some template fields such as 'hdr' are boolean and resolve to True or False.
+These take the form: '{TEMPLATE_FIELD?VALUE_IF_TRUE,VALUE_IF_FALSE}', e.g.
+{hdr?is_hdr,not_hdr} which would result in 'is_hdr' if photo is an HDR  image
+and 'not_hdr' otherwise.
+
+Some template fields such as 'folder_template' are "path-like" in that they
+join  multiple elements into a single path-like string.  For example, if photo
+is in  album Album1 in folder Folder1, '{folder_album}` results in
+'Folder1/Album1'.  This is so these template fields may be used as paths in
+--directory.  If you intend to use such a field as a string, e.g. in the
+filename, you may specify  a different path separator using the form:
+'{TEMPLATE_FIELD(PATH_SEP)}'.  For example, using the example above,
+'{folder_album(-)}' would result in  'Folder1-Album1' and '{folder_album()}'
+would result in  'Folder1Album1'.
+
+Some templates may resolve to more than one value.  For example, a photo can
+have  multiple keywords so '{keyword}' can result in multiple values.  If used
+in a filename  or directory, these templates may result in more than one copy
+of the photo being exported.  For example, if photo has keywords "foo" and
+"bar", --directory '{keyword}' will result in  copies of the photo being
+exported to 'foo/image_name.jpeg' and 'bar/image_name.jpeg'.
+
+Multi-value template fields such as '{keyword}' may be expanded 'in place'
+with an optional delimiter using the template form '{DELIM+TEMPLATE_FIELD}'.
+For example, a photo with  keywords 'foo' and 'bar':
+
+'{keyword}' renders to 'foo' and 'bar'
+
+'{,+keyword}' renders to: 'foo,bar'
+
+'{; +keyword}' renders to: 'foo; bar'
+
+'{+keyword}' renders to 'foobar'
+
+Some template fields such as '{media_type}' use the 'DEFAULT' value to allow
+customization  of the output. For example, '{media_type}' resolves to the
+special media type of the  photo such as 'panorama' or 'selfie'.  You may use
+the 'DEFAULT' value to override  these in form:
+'{media_type,video=vidéo;time_lapse=vidéo_accélérée}'.  In this example, if
+photo is a time_lapse photo, 'media_type' would resolve to  'vidéo_accélérée'
+instead of 'time_lapse' and video would resolve to 'vidéo' if photo is an
+ordinary video.
+
 With the --directory and --filename options you may specify a template for the
 export directory or filename, respectively. The directory will be appended to
 the export path specified in the export DEST argument to export.  For example,
-if template is '{created.year}/{created.month}', and export desitnation DEST
+if template is '{created.year}/{created.month}', and export destination DEST
 is '/Users/maria/Pictures/export', the actual export directory for a photo
 would be '/Users/maria/Pictures/export/2020/March' if the photo was created in
-March 2020. Some template substitutions may result in more than one value, for
-example '{album}' if photo is in more than one album or '{keyword}' if photo
-has more than one keyword. In this case, more than one copy of the photo will
-be exported, each in a separate directory or with a different filename.
+March 2020.
 
 The templating system may also be used with the --keyword-template option to
 set keywords on export (with --exiftool or --sidecar), for example, to set a
@@ -426,12 +505,28 @@ value, '_' (underscore) will be used as the default value. For example, in the
 above example, this would result in '2020/_/photoname.jpg' if address was
 null.
 
+You may specify a null default (e.g. "" or empty string) by omitting the value
+after the comma, e.g. {title,} which would render to "" if title had no value.
+
 Substitution                    Description
 {name}                          Current filename of the photo
 {original_name}                 Photo's original filename when imported to
                                 Photos
 {title}                         Title of the photo
 {descr}                         Description of the photo
+{media_type}                    Special media type resolved in this
+                                precedence: selfie, time_lapse, panorama,
+                                slow_mo, screenshot, portrait, live_photo,
+                                burst, photo, video. Defaults to 'photo' or
+                                'video' if no special type. Customize one or
+                                more media types using format: '{media_type,
+                                video=vidéo;time_lapse=vidéo_accélérée}'
+{photo_or_video}                'photo' or 'video' depending on what type
+                                the image is. To customize, use default
+                                value as in
+                                '{photo_or_video,photo=fotos;video=videos}'
+{hdr}                           Photo is HDR?; True/False value, use in
+                                format '{hdr?VALUE_IF_TRUE,VALUE_IF_FALSE}'
 {created.date}                  Photo's creation date in ISO format, e.g.
                                 '2020-03-22'
 {created.year}                  4-digit year of photo creation time
@@ -547,18 +642,26 @@ exported, one to each directory.  For example: --directory
 of the following directories if the photos were created in 2019 and were in
 albums 'Vacation' and 'Family': 2019/Vacation, 2019/Family
 
-Substitution        Description
-{album}             Album(s) photo is contained in
-{folder_album}      Folder path + album photo is contained in. e.g.
-                    'Folder/Subfolder/Album' or just 'Album' if no enclosing
-                    folder
-{keyword}           Keyword(s) assigned to photo
-{person}            Person(s) / face(s) in a photo
-{label}             Image categorization label associated with a photo
-                    (Photos 5 only)
-{label_normalized}  All lower case version of 'label' (Photos 5 only)
-{comment}           Comment(s) on shared Photos; format is 'Person name:
-                    comment text' (Photos 5 only)
+Substitution              Description
+{album}                   Album(s) photo is contained in
+{folder_album}            Folder path + album photo is contained in. e.g.
+                          'Folder/Subfolder/Album' or just 'Album' if no
+                          enclosing folder
+{keyword}                 Keyword(s) assigned to photo
+{person}                  Person(s) / face(s) in a photo
+{label}                   Image categorization label associated with a photo
+                          (Photos 5 only)
+{label_normalized}        All lower case version of 'label' (Photos 5 only)
+{comment}                 Comment(s) on shared Photos; format is 'Person
+                          name: comment text' (Photos 5 only)
+{exiftool:GROUP:TAGNAME}  Use exiftool (https://exiftool.org) to extract
+                          metadata, in form GROUP:TAGNAME, from image.  E.g.
+                          '{exiftool:EXIF:Make}' to get camera make, or
+                          {exiftool:IPTC:Keywords} to extract keywords. See
+                          https://exiftool.org/TagNames/ for list of valid
+                          tag names.  You must specify group (e.g. EXIF,
+                          IPTC, etc) as used in `exiftool -G`. exiftool must
+                          be installed in the path to use this template.
 ```
 
 Example: export all photos to ~/Desktop/export group in folders by date created
@@ -1269,6 +1372,9 @@ Returns True if photo is a panorama, otherwise False.
 
 **Note**: The result of `PhotoInfo.panorama` will differ from the "Panoramas" Media Types smart album in that it will also identify panorama photos from older phones that Photos does not recognize as panoramas. 
 
+#### `slow_mo`
+Returns True if photo is a slow motion video, otherwise False
+
 #### `labels`
 Returns image categorization labels associated with the photo as list of str.
 
@@ -1391,11 +1497,13 @@ If overwrite=False and increment=False, export will fail if destination file alr
 
 #### <a name="rendertemplate">`render_template()`</a>
 
-`render_template(template_str, none_str = "_", path_sep = None, expand_inplace = False, inplace_sep = None,         filename=False, dirname=False, replacement=":",)`
-Render template string for photo.  none_str is used if template substitution results in None value and no default specified. 
-- `template_str`: str in form "{name,DEFAULT}" where name is one of the values in table below. The "," and default value that follows are optional. If specified, "DEFAULT" will be used if "name" is None.  This is useful for values which are not always present, for example reverse geolocation data.
+`render_template(template_str, none_str = "_", path_sep = None, expand_inplace = False, inplace_sep = None, filename=False, dirname=False, replacement=":",)`
+
+Render template string for photo.  none_str is used if template substitution results in None value and no default specified.
+
+- `template_str`: str in format "{[[DELIM]+]name[(PATH_SEP)][?TRUE_VALUE][,[DEFAULT]]}" where name is one of the values in the [Template Substitutions](#template-substitutions) table. See notes below regarding specific details of the syntax.
 - `none_str`: optional str to use as substitution when template value is None and no default specified in the template string.  default is "_".
-- `path_sep`: optional character to use as path separator, default is os.path.sep
+- `path_sep`: optional character to use as path separator, default is `os.path.sep`
 - `expand_inplace`: expand multi-valued substitutions in-place as a single string instead of returning individual strings
 - `inplace_sep`: optional string to use as separator between multi-valued keywords with expand_inplace; default is ','
 - `filename`: if True, template output will be sanitized to produce valid file name
@@ -1412,6 +1520,56 @@ e.g. `render_template("{created.year}/{{foo}}", photo)` would return `(["2020/{f
 
 Some substitutions, notably `album`, `keyword`, and `person` could return multiple values, hence a new string will be return for each possible substitution (hence why a list of rendered strings is returned).  For example, a photo in 2 albums: 'Vacation' and 'Family' would result in the following rendered values if template was "{created.year}/{album}" and created.year == 2020: `["2020/Vacation","2020/Family"]` 
 
+The template field format contains optional modifiers:
+
+`"{[[DELIM]+]name[(PATH_SEP)][?TRUE_VALUE][,[DEFAULT]]}"`
+
+`DELIM`: optional delimiter string to use when expanding multi-valued template values in-place
+
+`+`: If present before template `name`, expands the template in place.  If `DELIM` not provided, values are joined with no delimiter.
+
+e.g. if Photo keywords are `["foo","bar"]`:
+
+- `"{keyword}"` renders to `["foo", "bar"]`
+- `"{,+keyword}"` renders to: `["foo,bar"]`
+- `"{; +keyword}"` renders to: `["foo; bar"]`
+- `"{+keyword}"` renders to `["foobar"]`
+
+`PATH_SEP`: optional path separator to use when joining path like fields, for example `{folder_album}`.  May also be provided as `path_sep` argument in `render_template()`.  If provided both in the call to `render_template()` and in the template itself, the value in the template string takes precedence.  If not provided in either the template string or in `path_sep` argument, defaults to `os.path.sep`.
+
+e.g. If Photo is in `Album1` in `Folder1`:
+
+- `"{folder_album}"` renders to `["Folder1/Album1"]`
+- `"{folder_album(:)}"` renders to `["Folder1:Album1"]`
+- `"{folder_album()}"` renders to `["Folder1Album1"]`
+
+`?TRUE_VALUE`: optional value to use if name is boolean-type field which evaluates to true.  For example `"{hdr}"` evaluates to True if photo is an high dynamic range (HDR) image and False otherwise. In these types of fields, use `?TRUE_VALUE` to provide the value if True and `,DEFAULT` to provide the value of False.  
+
+e.g. if photo is an HDR image,
+
+- `"{hdr?ISHDR,NOTHDR}"` renders to `["ISHDR"]`
+
+and if it is not an HDR image,
+
+- `"{hdr?ISHDR,NOTHDR}"` renders to `["NOTHDR"]`
+
+Either or both `TRUE_VALUE` or `DEFAULT` (False value) may be empty which would result in empty string `[""]` when rendered.
+
+`,DEFAULT`: optional default value to use if the template name has no value.  This modifier is also used for the value if False for boolean-type fields (see above) as well as to hold a sub-template for values like `{created.strftime}`.  If no default value provided, "_" is used. May also be provided in the `none_str` argument to `render_template()`.  If provided both in the template string and in `none_str`, the value in the template string takes precedence.
+
+e.g., if photo has no title set,
+
+- `"{title}"` renders to ["_"]
+- `"{title,I have no title}"` renders to `["I have no title"]`
+
+Template fields such as `created.strftime` use the DEFAULT value to pass the template to use for `strftime`.  
+
+e.g., if photo date is 4 February 2020, 19:07:38,
+
+- `"{created.strftime,%Y-%m-%d-%H%M%S}"` renders to `["2020-02-04-190738"]`
+
+Some template fields such as `"{media_type}"` use the `DEFAULT` value to allow customization of the output. For example, `"{media_type}"` resolves to the special media type of the photo such as `panorama` or `selfie`.  You may use the `DEFAULT` value to override these in form: `"{media_type,video=vidéo;time_lapse=vidéo_accélérée}"`. In this example, if photo was a time_lapse photo, `media_type` would resolve to `vidéo_accélérée` instead of `time_lapse`. 
+
 See [Template Substitutions](#template-substitutions) for additional details.
 
 ### ExifInfo
@@ -1832,37 +1990,42 @@ To get the path of every raw photo, whether it's a single raw photo or a raw+JPE
 
 ### Template Substitutions
 
-The following substitutions are availabe for use with `PhotoInfo.render_template()` 
+The following template field substitutions are availabe for use with `PhotoInfo.render_template()`
+
 | Substitution | Description |
 |--------------|-------------|
 |{name}|Current filename of the photo|
 |{original_name}|Photo's original filename when imported to Photos|
 |{title}|Title of the photo|
 |{descr}|Description of the photo|
+|{media_type}|Special media type resolved in this precedence: selfie, time_lapse, panorama, slow_mo, screenshot, portrait, live_photo, burst, photo, video. Defaults to 'photo' or 'video' if no special type. Customize one or more media types using format: '{media_type,video=vidéo;time_lapse=vidéo_accélérée}'|
+|{photo_or_video}|'photo' or 'video' depending on what type the image is. To customize, use default value as in '{photo_or_video,photo=fotos;video=videos}'|
+|{hdr}|Photo is HDR?; True/False value, use in format '{hdr?VALUE_IF_TRUE,VALUE_IF_FALSE}'|
 |{created.date}|Photo's creation date in ISO format, e.g. '2020-03-22'|
-|{created.year}|4-digit year of file creation time|
-|{created.yy}|2-digit year of file creation time|
-|{created.mm}|2-digit month of the file creation time (zero padded)|
-|{created.month}|Month name in user's locale of the file creation time|
-|{created.mon}|Month abbreviation in the user's locale of the file creation time|
-|{created.dd}|2-digit day of the month (zero padded) of file creation time|
-|{created.dow}|Day of week in user's locale of the file creation time|
-|{created.doy}|3-digit day of year (e.g Julian day) of file creation time, starting from 1 (zero padded)|
-|{created.hour}|2-digit hour of the file creation time|
-|{created.min}|2-digit minute of the file creation time|
-|{created.sec}|2-digit second of the file creation time|
+|{created.year}|4-digit year of photo creation time|
+|{created.yy}|2-digit year of photo creation time|
+|{created.mm}|2-digit month of the photo creation time (zero padded)|
+|{created.month}|Month name in user's locale of the photo creation time|
+|{created.mon}|Month abbreviation in the user's locale of the photo creation time|
+|{created.dd}|2-digit day of the month (zero padded) of photo creation time|
+|{created.dow}|Day of week in user's locale of the photo creation time|
+|{created.doy}|3-digit day of year (e.g Julian day) of photo creation time, starting from 1 (zero padded)|
+|{created.hour}|2-digit hour of the photo creation time|
+|{created.min}|2-digit minute of the photo creation time|
+|{created.sec}|2-digit second of the photo creation time|
 |{created.strftime}|Apply strftime template to file creation date/time. Should be used in form {created.strftime,TEMPLATE} where TEMPLATE is a valid strftime template, e.g. {created.strftime,%Y-%U} would result in year-week number of year: '2020-23'. If used with no template will return null value. See https://strftime.org/ for help on strftime templates.|
 |{modified.date}|Photo's modification date in ISO format, e.g. '2020-03-22'|
-|{modified.year}|4-digit year of file modification time|
-|{modified.yy}|2-digit year of file modification time|
-|{modified.mm}|2-digit month of the file modification time (zero padded)|
-|{modified.month}|Month name in user's locale of the file modification time|
-|{modified.mon}|Month abbreviation in the user's locale of the file modification time|
-|{modified.dd}|2-digit day of the month (zero padded) of the file modification time|
-|{modified.doy}|3-digit day of year (e.g Julian day) of file modification time, starting from 1 (zero padded)|
-|{modified.hour}|2-digit hour of the file modification time|
-|{modified.min}|2-digit minute of the file modification time|
-|{modified.sec}|2-digit second of the file modification time|
+|{modified.year}|4-digit year of photo modification time|
+|{modified.yy}|2-digit year of photo modification time|
+|{modified.mm}|2-digit month of the photo modification time (zero padded)|
+|{modified.month}|Month name in user's locale of the photo modification time|
+|{modified.mon}|Month abbreviation in the user's locale of the photo modification time|
+|{modified.dd}|2-digit day of the month (zero padded) of the photo modification time|
+|{modified.dow}|Day of week in user's locale of the photo modification time|
+|{modified.doy}|3-digit day of year (e.g Julian day) of photo modification time, starting from 1 (zero padded)|
+|{modified.hour}|2-digit hour of the photo modification time|
+|{modified.min}|2-digit minute of the photo modification time|
+|{modified.sec}|2-digit second of the photo modification time|
 |{today.date}|Current date in iso format, e.g. '2020-03-22'|
 |{today.year}|4-digit year of current date|
 |{today.yy}|2-digit year of current date|
@@ -1896,6 +2059,7 @@ The following substitutions are availabe for use with `PhotoInfo.render_template
 |{label}|Image categorization label associated with a photo (Photos 5 only)|
 |{label_normalized}|All lower case version of 'label' (Photos 5 only)|
 |{comment}|Comment(s) on shared Photos; format is 'Person name: comment text' (Photos 5 only)|
+|{exiftool:GROUP:TAGNAME}|Use exiftool (https://exiftool.org) to extract metadata, in form GROUP:TAGNAME, from image.  E.g. '{exiftool:EXIF:Make}' to get camera make, or {exiftool:IPTC:Keywords} to extract keywords. See https://exiftool.org/TagNames/ for list of valid tag names.  You must specify group (e.g. EXIF, IPTC, etc) as used in `exiftool -G`. exiftool must be installed in the path to use this template.|
 
 ### Utility Functions
 
@@ -2017,6 +2181,7 @@ Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/d
     <td align="center"><a href="https://github.com/grundsch"><img src="https://avatars0.githubusercontent.com/u/3874928?v=4?s=100" width="100px;" alt=""/><br /><sub><b>grundsch</b></sub></a><br /><a href="https://github.com/RhetTbull/osxphotos/commits?author=grundsch" title="Code">💻</a></td>
     <td align="center"><a href="https://github.com/agprimatic"><img src="https://avatars1.githubusercontent.com/u/4685054?v=4?s=100" width="100px;" alt=""/><br /><sub><b>Ag Primatic</b></sub></a><br /><a href="https://github.com/RhetTbull/osxphotos/commits?author=agprimatic" title="Code">💻</a></td>
     <td align="center"><a href="https://github.com/hhoeck"><img src="https://avatars1.githubusercontent.com/u/6313998?v=4?s=100" width="100px;" alt=""/><br /><sub><b>Horst Höck</b></sub></a><br /><a href="https://github.com/RhetTbull/osxphotos/commits?author=hhoeck" title="Code">💻</a></td>
+    <td align="center"><a href="https://github.com/jstrine"><img src="https://avatars1.githubusercontent.com/u/33943447?v=4?s=100" width="100px;" alt=""/><br /><sub><b>Jonathan Strine</b></sub></a><br /><a href="https://github.com/RhetTbull/osxphotos/commits?author=jstrine" title="Code">💻</a></td>
   </tr>
 </table>
 

make_cli_exe.sh

@@ -5,4 +5,4 @@
 # If you need to install pyinstaller:
 # python3 -m pip install --upgrade pyinstaller
 
-pyinstaller --onefile --hidden-import="pkg_resources.py2_warn" --name osxphotos --add-data osxphotos/templates/xmp_sidecar.mako:osxphotos/templates cli.py
+pyinstaller osxphotos.spec

osxphotos.spec

@@ -0,0 +1,48 @@
+# -*- mode: python ; coding: utf-8 -*-
+# spec file for pyinstaller
+# run `pyinstaller osxphotos.spec`
+
+import os
+import importlib
+
+pathex = os.getcwd()
+
+# include necessary data files
+datas=[('osxphotos/templates/xmp_sidecar.mako', 'osxphotos/templates')]
+package_imports = [['photoscript', ['photoscript.applescript']]]
+for package, files in package_imports:
+    proot = os.path.dirname(importlib.import_module(package).__file__)
+    datas.extend((os.path.join(proot, f), package) for f in files)
+
+block_cipher = None
+
+a = Analysis(['cli.py'],
+             pathex=[pathex],
+             binaries=[],
+             datas=datas,
+             hiddenimports=['pkg_resources.py2_warn'],
+             hookspath=[],
+             runtime_hooks=[],
+             excludes=[],
+             win_no_prefer_redirects=False,
+             win_private_assemblies=False,
+             cipher=block_cipher,
+             noarchive=False)
+
+pyz = PYZ(a.pure, a.zipped_data,
+             cipher=block_cipher)
+
+exe = EXE(pyz,
+          a.scripts,
+          a.binaries,
+          a.zipfiles,
+          a.datas,
+          [],
+          name='osxphotos',
+          debug=False,
+          bootloader_ignore_signals=False,
+          strip=False,
+          upx=True,
+          upx_exclude=[],
+          runtime_tmpdir=None,
+          console=True )

osxphotos/__main__.py

@@ -28,6 +28,7 @@ from .export_db import ExportDB, ExportDBInMemory
 from .fileutil import FileUtil, FileUtilNoOp
 from .path_utils import is_valid_filepath, sanitize_filename, sanitize_filepath
 from .photoinfo import ExportResults
+from .photokit import check_photokit_authorization, request_photokit_authorization
 from .phototemplate import TEMPLATE_SUBSTITUTIONS, TEMPLATE_SUBSTITUTIONS_MULTI_VALUED
 
 # global variable to control verbose output
@@ -97,7 +98,7 @@ class DateTimeISO8601(click.ParamType):
     def convert(self, value, param, ctx):
         try:
             return datetime.datetime.fromisoformat(value)
-        except:
+        except Exception:
             self.fail(
                 f"Invalid value for --{param.name}: invalid datetime format {value}. "
                 "Valid format: YYYY-MM-DD[*HH[:MM[:SS[.fff[fff]]]][+HH:MM[:SS[.ffffff]]]]"
@@ -158,19 +159,76 @@ class ExportCommand(click.Command):
         formatter.write("\n\n")
         formatter.write_text("** Templating System **")
         formatter.write("\n")
+        formatter.write_text(
+            """
+Several options, such as --directory, allow you to specify a template 
+which will be rendered to substitute template fields with values from the photo. 
+For example, '{created.month}' would be replaced with the month name of the photo creation date. 
+e.g. 'November'. 
+\n
+The general format for a template is '{TEMPLATE_FIELD[,[DEFAULT]]}'. 
+Some templates have optional modifiers in form 
+'{[[DELIM]+]TEMPLATE_FIELD[(PATH_SEP)][?VALUE_IF_TRUE][,[DEFAULT]]}'
+\n
+The ',' and DEFAULT value are optional. 
+If TEMPLATE_FIELD results in a null (empty) value, the default is '_'.  
+You may specify an alternate default value by appending ',DEFAULT' after template_field. 
+e.g. '{title,no_title}' would result in 'no_title' if the photo had no title. 
+You may include other text in the template string outside the {} and use more than 
+one template field, e.g. '{created.year} - {created.month}' (e.g. '2020 - November'). 
+\n
+Some template fields such as 'hdr' are boolean and resolve to True or False. 
+These take the form: '{TEMPLATE_FIELD?VALUE_IF_TRUE,VALUE_IF_FALSE}', e.g. 
+{hdr?is_hdr,not_hdr} which would result in 'is_hdr' if photo is an HDR 
+image and 'not_hdr' otherwise.
+\n
+Some template fields such as 'folder_template' are "path-like" in that they join 
+multiple elements into a single path-like string.  For example, if photo is in 
+album Album1 in folder Folder1, '{folder_album}` results in 'Folder1/Album1'. 
+This is so these template fields may be used as paths in --directory. 
+If you intend to use such a field as a string, e.g. in the filename, you may specify 
+a different path separator using the form: '{TEMPLATE_FIELD(PATH_SEP)}'. 
+For example, using the example above, '{folder_album(-)}' would result in 
+'Folder1-Album1' and '{folder_album()}' would result in 
+'Folder1Album1'.
+\n
+Some templates may resolve to more than one value.  For example, a photo can have 
+multiple keywords so '{keyword}' can result in multiple values.  If used in a filename 
+or directory, these templates may result in more than one copy of the photo being exported. 
+For example, if photo has keywords "foo" and "bar", --directory '{keyword}' will result in 
+copies of the photo being exported to 'foo/image_name.jpeg' and 'bar/image_name.jpeg'.  
+\n
+Multi-value template fields such as '{keyword}' may be expanded 'in place' with an optional
+delimiter using the template form '{DELIM+TEMPLATE_FIELD}'.  For example, a photo with 
+keywords 'foo' and 'bar':
+\n
+'{keyword}' renders to 'foo' and 'bar'
+\n
+'{,+keyword}' renders to: 'foo,bar'
+\n
+'{; +keyword}' renders to: 'foo; bar'
+\n
+'{+keyword}' renders to 'foobar'
+\n
+Some template fields such as '{media_type}' use the 'DEFAULT' value to allow customization 
+of the output. For example, '{media_type}' resolves to the special media type of the 
+photo such as 'panorama' or 'selfie'.  You may use the 'DEFAULT' value to override 
+these in form: '{media_type,video=vidéo;time_lapse=vidéo_accélérée}'. 
+In this example, if photo is a time_lapse photo, 'media_type' would resolve to 
+'vidéo_accélérée' instead of 'time_lapse' and video would resolve to 'vidéo' if photo
+is an ordinary video. 
+"""
+        )
+        formatter.write("\n")
         formatter.write_text(
             "With the --directory and --filename options you may specify a template for the "
             + "export directory or filename, respectively. "
             + "The directory will be appended to the export path specified "
             + "in the export DEST argument to export.  For example, if template is "
-            + "'{created.year}/{created.month}', and export desitnation DEST is "
+            + "'{created.year}/{created.month}', and export destination DEST is "
             + "'/Users/maria/Pictures/export', "
             + "the actual export directory for a photo would be '/Users/maria/Pictures/export/2020/March' "
             + "if the photo was created in March 2020. "
-            + "Some template substitutions may result in more than one value, for example '{album}' if "
-            + "photo is in more than one album or '{keyword}' if photo has more than one keyword. "
-            + "In this case, more than one copy of the photo will be exported, each in a separate directory "
-            + "or with a different filename."
         )
         formatter.write("\n")
         formatter.write_text(
@@ -208,7 +266,11 @@ class ExportCommand(click.Command):
             + "has no value, '_' (underscore) will be used as the default value. For example, in the "
             + "above example, this would result in '2020/_/photoname.jpg' if address was null."
         )
-
+        formatter.write("\n")
+        formatter.write_text(
+            'You may specify a null default (e.g. "" or empty string) by omitting the value after '
+            + 'the comma, e.g. {title,} which would render to "" if title had no value.'
+        )
         formatter.write("\n")
         templ_tuples = [("Substitution", "Description")]
         templ_tuples.extend((k, v) for k, v in TEMPLATE_SUBSTITUTIONS.items())
@@ -589,7 +651,7 @@ def debug_dump(ctx, cli_obj, db, photos_library, dump, uuid, verbose_):
                 val = getattr(photosdb, attr)
                 print(f"{attr}:")
                 pprint.pprint(val)
-            except:
+            except Exception:
                 print(f"Did not find attribute {attr} in PhotosDB")
 
 
@@ -784,12 +846,12 @@ def places(ctx, cli_obj, db, json_, photos_library):
         if photo.place:
             try:
                 place_names[photo.place.name] += 1
-            except:
+            except Exception:
                 place_names[photo.place.name] = 1
         else:
             try:
                 place_names[_UNKNOWN_PLACE] += 1
-            except:
+            except Exception:
                 place_names[_UNKNOWN_PLACE] = 1
 
     # sort by place count
@@ -1137,6 +1199,11 @@ def query(
 @DB_OPTION
 @click.option("--verbose", "-V", "verbose_", is_flag=True, help="Print verbose output.")
 @query_options
+@click.option(
+    "--missing",
+    is_flag=True,
+    help="Export only photos missing from the Photos library; must be used with --download-missing.",
+)
 @deleted_options
 @click.option(
     "--update",
@@ -1146,7 +1213,7 @@ def query(
 @click.option(
     "--dry-run",
     is_flag=True,
-    help="Dry run (test) the export but don't actually export any files; most useful with --verbose",
+    help="Dry run (test) the export but don't actually export any files; most useful with --verbose.",
 )
 @click.option(
     "--export-as-hardlink",
@@ -1321,6 +1388,14 @@ def query(
     "'photoname_edited.ext'. For example, with '--edited-suffix _bearbeiten', the edited photo "
     "would be named 'photoname_bearbeiten.ext'.  The default suffix is '_edited'.",
 )
+@click.option(
+    "--original-suffix",
+    metavar="SUFFIX",
+    default="",
+    help="Optional suffix for naming original photos.  Default name for original photos is in form "
+    "'filename.ext'. For example, with '--original-suffix _original', the original photo "
+    "would be named 'filename_original.ext'.  The default suffix is '' (no suffix).",
+)
 @click.option(
     "--no-extended-attributes",
     is_flag=True,
@@ -1333,8 +1408,21 @@ def query(
     "--use-photos-export",
     is_flag=True,
     default=False,
-    hidden=True,
-    help="Force the use of AppleScript to export even if not missing (see also --download-missing).",
+    help="Force the use of AppleScript or PhotoKit to export even if not missing (see also '--download-missing' and '--use-photokit').",
+)
+@click.option(
+    "--use-photokit",
+    is_flag=True,
+    default=False,
+    help="Use with '--download-missing' or '--use-photos-export' to use direct Photos interface instead of AppleScript to export. "
+    "Highly experimental alpha feature; does not work with iTerm2 (use with Terminal.app). "
+    "This is faster and more reliable than the default AppleScript interface.",
+)
+@click.option(
+    "--report",
+    metavar="REPORTNAME.CSV",
+    help="Write a CSV formatted report of all files that were exported.",
+    type=click.Path(),
 )
 @DB_ARGUMENT
 @click.argument("dest", nargs=1, type=click.Path(exists=True))
@@ -1368,6 +1456,7 @@ def export(
     from_date,
     to_date,
     verbose_,
+    missing,
     update,
     dry_run,
     export_as_hardlink,
@@ -1415,6 +1504,7 @@ def export(
     directory,
     filename_template,
     edited_suffix,
+    original_suffix,
     place,
     no_place,
     has_comment,
@@ -1426,6 +1516,8 @@ def export(
     deleted,
     deleted_only,
     use_photos_export,
+    use_photokit,
+    report,
 ):
     """ Export photos from the Photos database.
         Export path DEST is required.
@@ -1443,7 +1535,12 @@ def export(
     VERBOSE = bool(verbose_)
 
     if not os.path.isdir(dest):
-        sys.exit(f"DEST {dest} must be valid path")
+        click.echo(f"DEST {dest} must be valid path", err=True)
+        raise click.Abort()
+
+    if report and os.path.isdir(report):
+        click.echo(f"report is a directory, must be file name", err=True)
+        raise click.Abort()
 
     # sanity check input args
     exclusive = [
@@ -1476,6 +1573,32 @@ def export(
         click.echo(cli.commands["export"].get_help(ctx), err=True)
         return
 
+    if export_as_hardlink and download_missing:
+        click.echo(
+            "Incompatible export options: --export-as-hardlink is not compatible with --download-missing",
+            err=True,
+        )
+        raise click.Abort()
+
+    if missing and not download_missing:
+        click.echo(
+            "Incompatible export options: --missing must be used with --download-missing",
+            err=True,
+        )
+        raise click.Abort()
+
+    if use_photokit and not check_photokit_authorization():
+        click.echo(
+            "Requesting access to use your Photos library. Click 'OK' on the dialog box to grant access."
+        )
+        request_photokit_authorization()
+        click.confirm("Have you granted access?")
+        if not check_photokit_authorization():
+            click.echo(
+                "Failed to get access to the Photos library which is needed with `--use-photokit`."
+            )
+            return
+
     # initialize export flags
     # by default, will export all versions of photos unless skip flag is set
     (export_edited, export_bursts, export_live, export_raw) = [
@@ -1571,7 +1694,7 @@ def export(
         not_favorite=not_favorite,
         hidden=hidden,
         not_hidden=not_hidden,
-        missing=None,  # missing -- won't export these but will warn user
+        missing=missing,
         not_missing=None,
         shared=shared,
         not_shared=not_shared,
@@ -1638,6 +1761,11 @@ def export(
         results_skipped = []
         results_exif_updated = []
         results_touched = []
+        results_converted = []
+        results_sidecar_json_written = []
+        results_sidecar_json_skipped = []
+        results_sidecar_xmp_written = []
+        results_sidecar_xmp_skipped = []
         if verbose_:
             for p in photos:
                 results = export_photo(
@@ -1668,10 +1796,12 @@ def export(
                     dry_run=dry_run,
                     touch_file=touch_file,
                     edited_suffix=edited_suffix,
+                    original_suffix=original_suffix,
                     use_photos_export=use_photos_export,
                     convert_to_jpeg=convert_to_jpeg,
                     jpeg_quality=jpeg_quality,
                     ignore_date_modified=ignore_date_modified,
+                    use_photokit=use_photokit,
                 )
                 results_exported.extend(results.exported)
                 results_new.extend(results.new)
@@ -1679,6 +1809,11 @@ def export(
                 results_skipped.extend(results.skipped)
                 results_exif_updated.extend(results.exif_updated)
                 results_touched.extend(results.touched)
+                results_converted.extend(results.converted_to_jpeg)
+                results_sidecar_json_written.extend(results.sidecar_json_written)
+                results_sidecar_json_skipped.extend(results.sidecar_json_skipped)
+                results_sidecar_xmp_written.extend(results.sidecar_xmp_written)
+                results_sidecar_xmp_skipped.extend(results.sidecar_xmp_skipped)
 
                 # if convert_to_jpeg and p.isphoto and p.uti != "public.jpeg":
                 #     for photo_file in set(
@@ -1718,10 +1853,12 @@ def export(
                         dry_run=dry_run,
                         touch_file=touch_file,
                         edited_suffix=edited_suffix,
+                        original_suffix=original_suffix,
                         use_photos_export=use_photos_export,
                         convert_to_jpeg=convert_to_jpeg,
                         jpeg_quality=jpeg_quality,
                         ignore_date_modified=ignore_date_modified,
+                        use_photokit=use_photokit,
                     )
                     results_exported.extend(results.exported)
                     results_new.extend(results.new)
@@ -1729,9 +1866,41 @@ def export(
                     results_skipped.extend(results.skipped)
                     results_exif_updated.extend(results.exif_updated)
                     results_touched.extend(results.touched)
+                    results_converted.extend(results.converted_to_jpeg)
+                    results_sidecar_json_written.extend(results.sidecar_json_written)
+                    results_sidecar_json_skipped.extend(results.sidecar_json_skipped)
+                    results_sidecar_xmp_written.extend(results.sidecar_xmp_written)
+                    results_sidecar_xmp_skipped.extend(results.sidecar_xmp_skipped)
 
         stop_time = time.perf_counter()
         # print summary results
+        # print(f"results_exported: {results_exported}")
+        # print(f"results_new: {results_new}")
+        # print(f"results_updated: {results_updated}")
+        # print(f"results_skipped: {results_skipped}")
+        # print(f"results_exif_updated: {results_exif_updated}")
+        # print(f"results_touched: {results_touched}")
+        # print(f"results_converted: {results_converted}")
+        # print(f"results_sidecar_json: {results_sidecar_json}")
+        # print(f"results_sidecar_xmp: {results_sidecar_xmp}")
+
+        if report:
+            verbose(f"Writing export report to {report}")
+            write_export_report(
+                report,
+                results_exported=results_exported,
+                results_new=results_new,
+                results_updated=results_updated,
+                results_skipped=results_skipped,
+                results_exif_updated=results_exif_updated,
+                results_touched=results_touched,
+                results_converted=results_converted,
+                results_sidecar_json_written=results_sidecar_json_written,
+                results_sidecar_json_skipped=results_sidecar_json_skipped,
+                results_sidecar_xmp_written=results_sidecar_xmp_written,
+                results_sidecar_xmp_skipped=results_sidecar_xmp_skipped,
+            )
+
         if update:
             photo_str_new = "photos" if len(results_new) != 1 else "photo"
             photo_str_updated = "photos" if len(results_updated) != 1 else "photo"
@@ -2225,10 +2394,12 @@ def export_photo(
     dry_run=None,
     touch_file=None,
     edited_suffix="_edited",
+    original_suffix="",
     use_photos_export=False,
     convert_to_jpeg=False,
     jpeg_quality=1.0,
     ignore_date_modified=False,
+    use_photokit=False,
 ):
     """ Helper function for export that does the actual export
 
@@ -2273,23 +2444,25 @@ def export_photo(
     global VERBOSE
     VERBOSE = bool(verbose_)
 
+    # TODO: if --skip-original-if-edited, it's possible edited version is on disk but
+    # original is missing, in which case we should download the edited version
     if not download_missing:
         if photo.ismissing:
             space = " " if not verbose_ else ""
             verbose(f"{space}Skipping missing photo {photo.original_filename}")
-            return ExportResults([], [], [], [], [], [])
-        elif not os.path.exists(photo.path):
+            return ExportResults([], [], [], [], [], [], [], [], [], [], [])
+        elif photo.path is None:
             space = " " if not verbose_ else ""
             verbose(
-                f"{space}WARNING: file {photo.path} is missing but ismissing=False, "
+                f"{space}WARNING: photo {photo.original_filename} ({photo.uuid}) is missing but ismissing=False, "
                 f"skipping {photo.original_filename}"
             )
-            return ExportResults([], [], [], [], [], [])
+            return ExportResults([], [], [], [], [], [], [], [], [], [], [])
     elif photo.ismissing and not photo.iscloudasset and not photo.incloud:
         verbose(
             f"Skipping missing {photo.original_filename}: not iCloud asset or missing from cloud"
         )
-        return ExportResults([], [], [], [], [], [])
+        return ExportResults([], [], [], [], [], [], [], [], [], [], [])
 
     results_exported = []
     results_new = []
@@ -2297,12 +2470,45 @@ def export_photo(
     results_skipped = []
     results_exif_updated = []
     results_touched = []
+    results_converted = []
+    results_sidecar_json_written = []
+    results_sidecar_json_skipped = []
+    results_sidecar_xmp_written = []
+    results_sidecar_xmp_skipped = []
 
     export_original = not (skip_original_if_edited and photo.hasadjustments)
 
+    # can't export edited if photo doesn't have edited versions
+    export_edited = export_edited if photo.hasadjustments else False
+
+    # slow_mo photos will always have hasadjustments=True even if not edited
+    if photo.hasadjustments and photo.path_edited is None:
+        if photo.slow_mo:
+            export_original = True
+            export_edited = False
+        elif not download_missing:
+            # requested edited version but it's missing, download original
+            export_original = True
+            export_edited = False
+            verbose(
+                f"Edited file for {photo.original_filename} is missing, exporting original"
+            )
+
     filenames = get_filenames_from_template(photo, filename_template, original_name)
     for filename in filenames:
-        verbose(f"Exporting {photo.original_filename} ({photo.filename}) as {filename}")
+        if original_suffix:
+            original_filename = pathlib.Path(filename)
+            original_filename = (
+                original_filename.parent
+                / f"{original_filename.stem}{original_suffix}{original_filename.suffix}"
+            )
+            original_filename = str(original_filename)
+        else:
+            original_filename = filename
+
+        verbose(
+            f"Exporting {photo.original_filename} ({photo.filename}) as {original_filename}"
+        )
 
         dest_paths = get_dirnames_from_template(
             photo, directory, export_by_date, dest, dry_run
@@ -2321,87 +2527,24 @@ def export_photo(
             download_missing
             and (
                 photo.ismissing
-                or not os.path.exists(photo.path)
+                or photo.path is None
                 or (export_edited and photo.path_edited is None)
             )
         )
 
         # export the photo to each path in dest_paths
         for dest_path in dest_paths:
-            if not export_original:
-                verbose(f"Skipping original version of {photo.original_filename}")
-            else:
-                export_results = photo.export2(
-                    dest_path,
-                    filename,
-                    sidecar_json=sidecar_json,
-                    sidecar_xmp=sidecar_xmp,
-                    live_photo=export_live,
-                    raw_photo=export_raw,
-                    export_as_hardlink=export_as_hardlink,
-                    overwrite=overwrite,
-                    use_photos_export=use_photos_export,
-                    exiftool=exiftool,
-                    no_xattr=no_extended_attributes,
-                    use_albums_as_keywords=album_keyword,
-                    use_persons_as_keywords=person_keyword,
-                    keyword_template=keyword_template,
-                    description_template=description_template,
-                    update=update,
-                    export_db=export_db,
-                    fileutil=fileutil,
-                    dry_run=dry_run,
-                    touch_file=touch_file,
-                    convert_to_jpeg=convert_to_jpeg,
-                    jpeg_quality=jpeg_quality,
-                    ignore_date_modified=ignore_date_modified,
-                )
-
-                results_exported.extend(export_results.exported)
-                results_new.extend(export_results.new)
-                results_updated.extend(export_results.updated)
-                results_skipped.extend(export_results.skipped)
-                results_exif_updated.extend(export_results.exif_updated)
-                results_touched.extend(export_results.touched)
-
-                if verbose_:
-                    for exported in export_results.exported:
-                        verbose(f"Exported {exported}")
-                    for new in export_results.new:
-                        verbose(f"Exported new file {new}")
-                    for updated in export_results.updated:
-                        verbose(f"Exported updated file {updated}")
-                    for skipped in export_results.skipped:
-                        verbose(f"Skipped up to date file {skipped}")
-                    for touched in export_results.touched:
-                        verbose(f"Touched date on file {touched}")
-
-            # if export-edited, also export the edited version
-            # verify the photo has adjustments and valid path to avoid raising an exception
-            if export_edited and photo.hasadjustments:
-                # if download_missing and the photo is missing or path doesn't exist,
-                # try to download with Photos
-                if not download_missing and photo.path_edited is None:
-                    verbose(f"Skipping missing edited photo for {filename}")
-                else:
-                    edited_name = pathlib.Path(filename)
-                    # check for correct edited suffix
-                    if photo.path_edited is not None:
-                        edited_ext = pathlib.Path(photo.path_edited).suffix
-                    else:
-                        # use filename suffix which might be wrong,
-                        # will be corrected by use_photos_export
-                        edited_ext = pathlib.Path(photo.filename).suffix
-                    edited_name = f"{edited_name.stem}{edited_suffix}{edited_ext}"
-                    verbose(f"Exporting edited version of {filename} as {edited_name}")
-                    export_results_edited = photo.export2(
+            if export_original:
+                try:
+                    export_results = photo.export2(
                         dest_path,
-                        edited_name,
+                        original_filename,
                         sidecar_json=sidecar_json,
                         sidecar_xmp=sidecar_xmp,
+                        live_photo=export_live,
+                        raw_photo=export_raw,
                         export_as_hardlink=export_as_hardlink,
                         overwrite=overwrite,
-                        edited=True,
                         use_photos_export=use_photos_export,
                         exiftool=exiftool,
                         no_xattr=no_extended_attributes,
@@ -2417,26 +2560,137 @@ def export_photo(
                         convert_to_jpeg=convert_to_jpeg,
                         jpeg_quality=jpeg_quality,
                         ignore_date_modified=ignore_date_modified,
+                        use_photokit=use_photokit,
+                        verbose=verbose,
                     )
 
-                    results_exported.extend(export_results_edited.exported)
-                    results_new.extend(export_results_edited.new)
-                    results_updated.extend(export_results_edited.updated)
-                    results_skipped.extend(export_results_edited.skipped)
-                    results_exif_updated.extend(export_results_edited.exif_updated)
-                    results_touched.extend(export_results_edited.touched)
+                    results_exported.extend(export_results.exported)
+                    results_new.extend(export_results.new)
+                    results_updated.extend(export_results.updated)
+                    results_skipped.extend(export_results.skipped)
+                    results_exif_updated.extend(export_results.exif_updated)
+                    results_touched.extend(export_results.touched)
+                    results_converted.extend(export_results.converted_to_jpeg)
+                    results_sidecar_json_written.extend(
+                        export_results.sidecar_json_written
+                    )
+                    results_sidecar_json_skipped.extend(
+                        export_results.sidecar_json_skipped
+                    )
+                    results_sidecar_xmp_written.extend(
+                        export_results.sidecar_xmp_written
+                    )
+                    results_sidecar_xmp_skipped.extend(
+                        export_results.sidecar_xmp_skipped
+                    )
 
                     if verbose_:
-                        for exported in export_results_edited.exported:
+                        for exported in export_results.exported:
                             verbose(f"Exported {exported}")
-                        for new in export_results_edited.new:
+                        for new in export_results.new:
                             verbose(f"Exported new file {new}")
-                        for updated in export_results_edited.updated:
+                        for updated in export_results.updated:
                             verbose(f"Exported updated file {updated}")
-                        for skipped in export_results_edited.skipped:
+                        for skipped in export_results.skipped:
                             verbose(f"Skipped up to date file {skipped}")
-                        for touched in export_results_edited.touched:
+                        for touched in export_results.touched:
                             verbose(f"Touched date on file {touched}")
+                except Exception:
+                    click.echo(
+                        f"Error exporting photo {photo.original_filename} ({photo.filename}) as {original_filename}",
+                        err=True,
+                    )
+            else:
+                verbose(f"Skipping original version of {photo.original_filename}")
+
+            # if export-edited, also export the edited version
+            # verify the photo has adjustments and valid path to avoid raising an exception
+            if export_edited and photo.hasadjustments:
+                # if download_missing and the photo is missing or path doesn't exist,
+                # try to download with Photos
+                if not download_missing and photo.path_edited is None:
+                    verbose(f"Skipping missing edited photo for {filename}")
+                else:
+                    edited_filename = pathlib.Path(filename)
+                    # check for correct edited suffix
+                    if photo.path_edited is not None:
+                        edited_ext = pathlib.Path(photo.path_edited).suffix
+                    else:
+                        # use filename suffix which might be wrong,
+                        # will be corrected by use_photos_export
+                        edited_ext = pathlib.Path(photo.filename).suffix
+                    edited_filename = (
+                        f"{edited_filename.stem}{edited_suffix}{edited_ext}"
+                    )
+                    verbose(
+                        f"Exporting edited version of {photo.original_filename} ({photo.filename}) as {edited_filename}"
+                    )
+                    try:
+                        export_results_edited = photo.export2(
+                            dest_path,
+                            edited_filename,
+                            sidecar_json=sidecar_json,
+                            sidecar_xmp=sidecar_xmp,
+                            export_as_hardlink=export_as_hardlink,
+                            overwrite=overwrite,
+                            edited=True,
+                            use_photos_export=use_photos_export,
+                            exiftool=exiftool,
+                            no_xattr=no_extended_attributes,
+                            use_albums_as_keywords=album_keyword,
+                            use_persons_as_keywords=person_keyword,
+                            keyword_template=keyword_template,
+                            description_template=description_template,
+                            update=update,
+                            export_db=export_db,
+                            fileutil=fileutil,
+                            dry_run=dry_run,
+                            touch_file=touch_file,
+                            convert_to_jpeg=convert_to_jpeg,
+                            jpeg_quality=jpeg_quality,
+                            ignore_date_modified=ignore_date_modified,
+                            use_photokit=use_photokit,
+                            verbose=verbose,
+                        )
+
+                        results_exported.extend(export_results_edited.exported)
+                        results_new.extend(export_results_edited.new)
+                        results_updated.extend(export_results_edited.updated)
+                        results_skipped.extend(export_results_edited.skipped)
+                        results_exif_updated.extend(export_results_edited.exif_updated)
+                        results_touched.extend(export_results_edited.touched)
+                        results_converted.extend(
+                            export_results_edited.converted_to_jpeg
+                        )
+                        results_sidecar_json_written.extend(
+                            export_results_edited.sidecar_json_written
+                        )
+                        results_sidecar_json_skipped.extend(
+                            export_results_edited.sidecar_json_skipped
+                        )
+                        results_sidecar_xmp_written.extend(
+                            export_results_edited.sidecar_xmp_written
+                        )
+                        results_sidecar_xmp_skipped.extend(
+                            export_results_edited.sidecar_xmp_skipped
+                        )
+
+                        if verbose_:
+                            for exported in export_results_edited.exported:
+                                verbose(f"Exported {exported}")
+                            for new in export_results_edited.new:
+                                verbose(f"Exported new file {new}")
+                            for updated in export_results_edited.updated:
+                                verbose(f"Exported updated file {updated}")
+                            for skipped in export_results_edited.skipped:
+                                verbose(f"Skipped up to date file {skipped}")
+                            for touched in export_results_edited.touched:
+                                verbose(f"Touched date on file {touched}")
+                    except Exception:
+                        click.echo(
+                            f"Error exporting photo {filename} as {edited_filename}",
+                            err=True,
+                        )
 
     return ExportResults(
         results_exported,
@@ -2445,6 +2699,11 @@ def export_photo(
         results_skipped,
         results_exif_updated,
         results_touched,
+        results_converted,
+        results_sidecar_json_written,
+        results_sidecar_json_skipped,
+        results_sidecar_xmp_written,
+        results_sidecar_xmp_skipped,
     )
 
 
@@ -2474,7 +2733,11 @@ def get_filenames_from_template(photo, filename_template, original_name):
             )
         filenames = [f"{file_}{photo_ext}" for file_ in filenames]
     else:
-        filenames = [photo.original_filename] if original_name else [photo.filename]
+        filenames = (
+            [photo.original_filename]
+            if (original_name and (photo.original_filename is not None))
+            else [photo.filename]
+        )
 
     filenames = [sanitize_filename(filename) for filename in filenames]
     return filenames
@@ -2592,5 +2855,111 @@ def load_uuid_from_file(filename):
     return uuid
 
 
+def write_export_report(
+    report_file,
+    results_exported,
+    results_new,
+    results_updated,
+    results_skipped,
+    results_exif_updated,
+    results_touched,
+    results_converted,
+    results_sidecar_json_written,
+    results_sidecar_json_skipped,
+    results_sidecar_xmp_written,
+    results_sidecar_xmp_skipped,
+):
+
+    """ write CSV report with results from export """
+
+    # Collect results for reporting
+    # TODO: pull this in a separate write_report function
+    all_results = {
+        result: {
+            "filename": result,
+            "exported": 0,
+            "new": 0,
+            "updated": 0,
+            "skipped": 0,
+            "exif_updated": 0,
+            "touched": 0,
+            "converted_to_jpeg": 0,
+            "sidecar_xmp": 0,
+            "sidecar_json": 0,
+        }
+        for result in results_exported
+        + results_new
+        + results_updated
+        + results_skipped
+        + results_exif_updated
+        + results_touched
+        + results_converted
+        + results_sidecar_json_written
+        + results_sidecar_json_skipped
+        + results_sidecar_xmp_written
+        + results_sidecar_xmp_skipped
+    }
+
+    for result in results_exported:
+        all_results[result]["exported"] = 1
+
+    for result in results_new:
+        all_results[result]["new"] = 1
+
+    for result in results_updated:
+        all_results[result]["updated"] = 1
+
+    for result in results_skipped:
+        all_results[result]["skipped"] = 1
+
+    for result in results_exif_updated:
+        all_results[result]["exif_updated"] = 1
+
+    for result in results_touched:
+        all_results[result]["touched"] = 1
+
+    for result in results_converted:
+        all_results[result]["converted_to_jpeg"] = 1
+
+    for result in results_sidecar_xmp_written:
+        all_results[result]["sidecar_xmp"] = 1
+        all_results[result]["exported"] = 1
+
+    for result in results_sidecar_xmp_skipped:
+        all_results[result]["sidecar_xmp"] = 1
+        all_results[result]["skipped"] = 1
+
+    for result in results_sidecar_json_written:
+        all_results[result]["sidecar_json"] = 1
+        all_results[result]["exported"] = 1
+
+    for result in results_sidecar_json_skipped:
+        all_results[result]["sidecar_json"] = 1
+        all_results[result]["skipped"] = 1
+
+    report_columns = [
+        "filename",
+        "exported",
+        "new",
+        "updated",
+        "skipped",
+        "exif_updated",
+        "touched",
+        "converted_to_jpeg",
+        "sidecar_xmp",
+        "sidecar_json",
+    ]
+
+    try:
+        with open(report_file, "w") as csvfile:
+            writer = csv.DictWriter(csvfile, fieldnames=report_columns)
+            writer.writeheader()
+            for data in [result for result in all_results.values()]:
+                writer.writerow(data)
+    except IOError:
+        click.echo("Could not open output file for writing", err=True)
+        raise click.Abort()
+
+
 if __name__ == "__main__":
     cli()  # pylint: disable=no-value-for-parameter

osxphotos/_version.py

@@ -1,4 +1,4 @@
 """ version info """
 
-__version__ = "0.36.7"
+__version__ = "0.37.2"
 

osxphotos/exiftool.py

@@ -100,7 +100,7 @@ class _ExifToolProc:
             ],
             stdin=subprocess.PIPE,
             stdout=subprocess.PIPE,
-            stderr=subprocess.DEVNULL,
+            stderr=subprocess.STDOUT,
         )
         self._process_running = True
 
@@ -133,13 +133,19 @@ class ExifTool:
     """ Basic exiftool interface for reading and writing EXIF tags """
 
     def __init__(self, filepath, exiftool=None, overwrite=True):
-        """ Return ExifTool object
-            file: path to image file
-            exiftool: path to exiftool, if not specified will look in path
-            overwrite: if True, will overwrite image file without creating backup, default=False """
+        """ Create ExifTool object
+
+        Args:
+                file: path to image file
+                exiftool: path to exiftool, if not specified will look in path
+                overwrite: if True, will overwrite image file without creating backup, default=False
+        Returns:
+            ExifTool instance
+        """
         self.file = filepath
         self.overwrite = overwrite
         self.data = {}
+        self.error = None
         # if running as a context manager, self._context_mgr will be True
         self._context_mgr = False
         self._exiftoolproc = _ExifToolProc(exiftool=exiftool)
@@ -147,8 +153,18 @@ class ExifTool:
         self._read_exif()
 
     def setvalue(self, tag, value):
-        """ Set tag to value(s) 
-            if value is None, will delete tag """
+        """ Set tag to value(s); if value is None, will delete tag
+
+        Args:
+            tag: str; name of tag to set
+            value: str; value to set tag to
+        
+        Returns:
+            True if success otherwise False
+            
+            If error generated by exiftool, returns False and sets self.error to error string
+            If called in context manager, returns True (execution is delayed until exiting context manager)
+        """
 
         if value is None:
             value = ""
@@ -157,19 +173,32 @@ class ExifTool:
             command.append("-overwrite_original")
         if self._context_mgr:
             self._commands.extend(command)
+            return True
         else:
-            self.run_commands(*command)
+            _, self.error = self.run_commands(*command)
+            return self.error is None
 
     def addvalues(self, tag, *values):
         """ Add one or more value(s) to tag
             If more than one value is passed, each value will be added to the tag
-            Notes: exiftool may add duplicate values for some tags so the caller must ensure
-                   the values being added are not already in the EXIF data
-                   For some tags, such as IPTC:Keywords, this will add a new value to the list of keywords, 
-                   but for others, such as EXIF:ISO, this will literally add a value to the existing value.
-                   It's up to the caller to know what exiftool will do for each tag
-                   If setvalue called before addvalues, exiftool does not appear to add duplicates, 
-                   but if addvalues called without first calling setvalue, exiftool will add duplicate values
+
+        Args:
+            tag: str; tag to set
+            *values: str; one or more values to set
+        
+        Returns:
+            True if success otherwise False
+            
+            If error generated by exiftool, returns False and sets self.error to error string
+            If called in context manager, returns True (execution is delayed until exiting context manager)
+            
+        Notes: exiftool may add duplicate values for some tags so the caller must ensure
+               the values being added are not already in the EXIF data
+               For some tags, such as IPTC:Keywords, this will add a new value to the list of keywords, 
+               but for others, such as EXIF:ISO, this will literally add a value to the existing value.
+               It's up to the caller to know what exiftool will do for each tag
+               If setvalue called before addvalues, exiftool does not appear to add duplicates, 
+               but if addvalues called without first calling setvalue, exiftool will add duplicate values
         """
         if not values:
             raise ValueError("Must pass at least one value")
@@ -185,14 +214,26 @@ class ExifTool:
 
         if self._context_mgr:
             self._commands.extend(command)
+            return True
         else:
-            self.run_commands(*command)
+            _, self.error = self.run_commands(*command)
+            return self.error is None
 
     def run_commands(self, *commands, no_file=False):
-        """ run commands in the exiftool process and return result 
-            no_file: (bool) do not pass the filename to exiftool (default=False)
-                     by default, all commands will be run against self.file
-                     use no_file=True to run a command without passing the filename """
+        """ Run commands in the exiftool process and return result.
+
+        Args:
+                *commands: exiftool commands to run
+                no_file: (bool) do not pass the filename to exiftool (default=False)
+                        by default, all commands will be run against self.file
+                        use no_file=True to run a command without passing the filename
+        Returns:
+            (output, errror)
+            output: bytes is containing output of exiftool commands
+            error: if exiftool generated an error, bytes containing error string otherwise None
+
+        Note: Also sets self.error if error generated.
+        """
         if not (hasattr(self, "_process") and self._process):
             raise ValueError("exiftool process is not running")
 
@@ -218,9 +259,16 @@ class ExifTool:
 
         # read the output
         output = b""
+        error = b""
         while EXIFTOOL_STAYOPEN_EOF not in str(output):
-            output += self._process.stdout.readline().strip()
-        return output[:-EXIFTOOL_STAYOPEN_EOF_LEN]
+            line = self._process.stdout.readline()
+            if line.startswith(b"Warning"):
+                error += line
+            else:
+                output += line.strip()
+        error = None if error == b"" else error
+        self.error = error
+        return output[:-EXIFTOOL_STAYOPEN_EOF_LEN], error
 
     @property
     def pid(self):
@@ -230,14 +278,14 @@ class ExifTool:
     @property
     def version(self):
         """ returns exiftool version """
-        ver = self.run_commands("-ver", no_file=True)
+        ver, _ = self.run_commands("-ver", no_file=True)
         return ver.decode("utf-8")
 
     def asdict(self):
         """ return dictionary of all EXIF tags and values from exiftool 
             returns empty dict if no tags
         """
-        json_str = self.run_commands("-json")
+        json_str, _ = self.run_commands("-json")
         if json_str:
             exifdict = json.loads(json_str)
             return exifdict[0]
@@ -246,7 +294,8 @@ class ExifTool:
 
     def json(self):
         """ returns JSON string containing all EXIF tags and values from exiftool """
-        return self.run_commands("-json")
+        json, _ = self.run_commands("-json")
+        return json
 
     def _read_exif(self):
         """ read exif data from file """
@@ -265,4 +314,4 @@ class ExifTool:
         if exc_type:
             return False
         elif self._commands:
-            self.run_commands(*self._commands)
+            _, self.error = self.run_commands(*self._commands)

osxphotos/export_db.py

@@ -14,7 +14,7 @@ from sqlite3 import Error
 
 from ._version import __version__
 
-OSXPHOTOS_EXPORTDB_VERSION = "2.0"
+OSXPHOTOS_EXPORTDB_VERSION = "3.2"
 
 
 class ExportDB_ABC(ABC):
@@ -76,6 +76,14 @@ class ExportDB_ABC(ABC):
     def set_exifdata_for_file(self, uuid, exifdata):
         pass
 
+    @abstractmethod
+    def set_sidecar_for_file(self, filename, sidecar_data, sidecar_sig):
+        pass
+
+    @abstractmethod
+    def get_sidecar_for_file(self, filename):
+        pass
+
     @abstractmethod
     def set_data(
         self,
@@ -141,6 +149,12 @@ class ExportDBNoOp(ExportDB_ABC):
     def set_exifdata_for_file(self, uuid, exifdata):
         pass
 
+    def set_sidecar_for_file(self, filename, sidecar_data, sidecar_sig):
+        pass
+
+    def get_sidecar_for_file(self, filename):
+        return None, (None, None, None)
+
     def set_data(
         self,
         filename,
@@ -379,6 +393,48 @@ class ExportDB(ExportDB_ABC):
         except Error as e:
             logging.warning(e)
 
+    def get_sidecar_for_file(self, filename):
+        """ returns the sidecar data and signature for a file """
+        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        conn = self._conn
+        try:
+            c = conn.cursor()
+            c.execute(
+                "SELECT sidecar_data, mode, size, mtime FROM sidecar WHERE filepath_normalized = ?",
+                (filename,),
+            )
+            results = c.fetchone()
+            if results:
+                sidecar_data = results[0]
+                sidecar_sig = (
+                    results[1],
+                    results[2],
+                    int(results[3]) if results[3] is not None else None,
+                )
+            else:
+                sidecar_data = None
+                sidecar_sig = (None, None, None)
+        except Error as e:
+            logging.warning(e)
+            sidecar_data = None
+            sidecar_sig = (None, None, None)
+
+        return sidecar_data, sidecar_sig
+
+    def set_sidecar_for_file(self, filename, sidecar_data, sidecar_sig):
+        """ sets the sidecar data and signature for a file """
+        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        conn = self._conn
+        try:
+            c = conn.cursor()
+            c.execute(
+                "INSERT OR REPLACE INTO sidecar(filepath_normalized, sidecar_data, mode, size, mtime) VALUES (?, ?, ?, ?, ?);",
+                (filename, sidecar_data, *sidecar_sig),
+            )
+            conn.commit()
+        except Error as e:
+            logging.warning(e)
+
     def set_data(
         self,
         filename,
@@ -479,13 +535,11 @@ class ExportDB(ExportDB_ABC):
 
         if not os.path.isfile(dbfile):
             conn = self._get_db_connection(dbfile)
-            if conn:
-                self._create_db_tables(conn)
-                self.was_created = True
-                self.was_upgraded = ()
-                self.version = OSXPHOTOS_EXPORTDB_VERSION
-            else:
+            if not conn:
                 raise Exception("Error getting connection to database {dbfile}")
+            self._create_db_tables(conn)
+            self.was_created = True
+            self.was_upgraded = ()
         else:
             conn = self._get_db_connection(dbfile)
             self.was_created = False
@@ -495,8 +549,7 @@ class ExportDB(ExportDB_ABC):
                 self.was_upgraded = (version_info[1], OSXPHOTOS_EXPORTDB_VERSION)
             else:
                 self.was_upgraded = ()
-            self.version = OSXPHOTOS_EXPORTDB_VERSION
-
+        self.version = OSXPHOTOS_EXPORTDB_VERSION
         return conn
 
     def _get_db_connection(self, dbfile):
@@ -570,11 +623,20 @@ class ExportDB(ExportDB_ABC):
                               size INTEGER,
                               mtime REAL
                               ); """,
+            "sql_sidecar_table": """ CREATE TABLE IF NOT EXISTS sidecar (
+                              id INTEGER PRIMARY KEY,
+                              filepath_normalized TEXT NOT NULL,
+                              sidecar_data TEXT,
+                              mode INTEGER,
+                              size INTEGER,
+                              mtime REAL
+                              ); """,
             "sql_files_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_files_filepath_normalized on files (filepath_normalized); """,
             "sql_info_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_info_uuid on info (uuid); """,
             "sql_exifdata_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_exifdata_filename on exifdata (filepath_normalized); """,
             "sql_edited_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_edited_filename on edited (filepath_normalized);""",
             "sql_converted_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_converted_filename on converted (filepath_normalized);""",
+            "sql_sidecar_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_sidecar_filename on sidecar (filepath_normalized);""",
         }
         try:
             c = conn.cursor()

osxphotos/fileutil.py

@@ -1,6 +1,5 @@
 """ FileUtil class with methods for copy, hardlink, unlink, etc. """
 
-import logging
 import os
 import pathlib
 import stat
@@ -74,7 +73,6 @@ class FileUtilMacOS(FileUtilABC):
         try:
             os.link(src, dest)
         except Exception as e:
-            logging.critical(f"os.link returned error: {e}")
             raise e
 
     @classmethod
@@ -92,7 +90,7 @@ class FileUtilMacOS(FileUtilABC):
         if src is None or dest is None:
             raise ValueError("src and dest must not be None", src, dest)
 
-        if not os.path.isfile(src):
+        if not os.path.exists(src):
             raise FileNotFoundError("src file does not appear to exist", src)
 
         if norsrc:
@@ -104,9 +102,6 @@ class FileUtilMacOS(FileUtilABC):
         try:
             result = subprocess.run(command, check=True, stderr=subprocess.PIPE)
         except subprocess.CalledProcessError as e:
-            logging.critical(
-                f"ditto returned error: {e.returncode} {e.stderr.decode(sys.getfilesystemencoding()).rstrip()}"
-            )
             raise e
 
         return result.returncode

osxphotos/path_utils.py

@@ -1,8 +1,9 @@
 """ utility functions for validating/sanitizing path components """
 
-from ._constants import MAX_DIRNAME_LEN, MAX_FILENAME_LEN
 import pathvalidate
 
+from ._constants import MAX_DIRNAME_LEN, MAX_FILENAME_LEN
+
 
 def sanitize_filepath(filepath):
     """ sanitize a filepath """

osxphotos/photoinfo/_photoinfo_export.py

@@ -13,6 +13,7 @@
 # TODO: should this be its own PhotoExporter class?
 
 import glob
+import hashlib
 import json
 import logging
 import os
@@ -21,9 +22,10 @@ import re
 import tempfile
 from collections import namedtuple  # pylint: disable=syntax-error
 
+import photoscript
 from mako.template import Template
 
-from .._applescript import AppleScript
+# from .._applescript import AppleScript
 from .._constants import (
     _MAX_IPTC_KEYWORD_LEN,
     _OSXPHOTOS_NONE_SENTINEL,
@@ -31,17 +33,43 @@ from .._constants import (
     _UNKNOWN_PERSON,
     _XMP_TEMPLATE_NAME,
 )
-from ..export_db import ExportDBNoOp
 from ..exiftool import ExifTool
+from ..export_db import ExportDBNoOp
 from ..fileutil import FileUtil
-from ..utils import dd_to_dms_str, findfiles
+from ..photokit import (
+    PHOTOS_VERSION_CURRENT,
+    PHOTOS_VERSION_ORIGINAL,
+    PhotoLibrary,
+    PhotoKitFetchFailed,
+)
+from ..utils import dd_to_dms_str, findfiles, noop
 
 ExportResults = namedtuple(
     "ExportResults",
-    ["exported", "new", "updated", "skipped", "exif_updated", "touched"],
+    [
+        "exported",
+        "new",
+        "updated",
+        "skipped",
+        "exif_updated",
+        "touched",
+        "converted_to_jpeg",
+        "sidecar_json_written",
+        "sidecar_json_skipped",
+        "sidecar_xmp_written",
+        "sidecar_xmp_skipped",
+    ],
 )
 
 
+# hexdigest is not a class method, don't import this into PhotoInfo
+def hexdigest(strval):
+    """ hexdigest of a string, using blake2b """
+    h = hashlib.blake2b(digest_size=20)
+    h.update(bytes(strval, "utf-8"))
+    return h.hexdigest()
+
+
 # _export_photo_uuid_applescript is not a class method, don't import this into PhotoInfo
 def _export_photo_uuid_applescript(
     uuid,
@@ -78,33 +106,33 @@ def _export_photo_uuid_applescript(
     """
 
     # setup the applescript to do the export
-    export_scpt = AppleScript(
-        """ 
-		on export_by_uuid(theUUID, thePath, original, edited, theTimeOut)
-			tell application "Photos"
-				set thePath to thePath
-				set theItem to media item id theUUID
-				set theFilename to filename of theItem
-				set itemList to {theItem}
-				
-				if original then
-					with timeout of theTimeOut seconds
-						export itemList to POSIX file thePath with using originals
-					end timeout
-				end if
-				
-				if edited then
-					with timeout of theTimeOut seconds
-						export itemList to POSIX file thePath
-					end timeout
-				end if
-				
-				return theFilename
-			end tell
+    # export_scpt = AppleScript(
+    #     """
+    # 	on export_by_uuid(theUUID, thePath, original, edited, theTimeOut)
+    # 		tell application "Photos"
+    # 			set thePath to thePath
+    # 			set theItem to media item id theUUID
+    # 			set theFilename to filename of theItem
+    # 			set itemList to {theItem}
 
-		end export_by_uuid
-		"""
-    )
+    # 			if original then
+    # 				with timeout of theTimeOut seconds
+    # 					export itemList to POSIX file thePath with using originals
+    # 				end timeout
+    # 			end if
+
+    # 			if edited then
+    # 				with timeout of theTimeOut seconds
+    # 					export itemList to POSIX file thePath
+    # 				end timeout
+    # 			end if
+
+    # 			return theFilename
+    # 		end tell
+
+    # 	end export_by_uuid
+    # 	"""
+    # )
 
     dest = pathlib.Path(dest)
     if not dest.is_dir():
@@ -115,32 +143,36 @@ def _export_photo_uuid_applescript(
 
     tmpdir = tempfile.TemporaryDirectory(prefix="osxphotos_")
 
-    # export original
+    exported_files = []
     filename = None
     try:
-        filename = export_scpt.call(
-            "export_by_uuid", uuid, tmpdir.name, original, edited, timeout
-        )
+        photo = photoscript.Photo(uuid)
+        filename = photo.filename
+        exported_files = photo.export(tmpdir.name, original=original, timeout=timeout)
+        # filename = export_scpt.call(
+        #     "export_by_uuid", uuid, tmpdir.name, original, edited, timeout
+        # )
     except Exception as e:
         logging.warning(f"Error exporting uuid {uuid}: {e}")
         return None
 
-    if filename is not None:
+    if exported_files and filename:
         # need to find actual filename as sometimes Photos renames JPG to jpeg on export
         # may be more than one file exported (e.g. if Live Photo, Photos exports both .jpeg and .mov)
         # TemporaryDirectory will cleanup on return
         filename_stem = pathlib.Path(filename).stem
-        files = glob.glob(os.path.join(tmpdir.name, "*"))
         exported_paths = []
-        for fname in files:
-            path = pathlib.Path(fname)
-            if len(files) > 1 and not live_photo and path.suffix.lower() == ".mov":
+        for fname in exported_files:
+            path = pathlib.Path(tmpdir.name) / fname
+            if (
+                len(exported_files) > 1
+                and not live_photo
+                and path.suffix.lower() == ".mov"
+            ):
                 # it's the .mov part of live photo but not requested, so don't export
-                logging.debug(f"Skipping live photo file {path}")
                 continue
-            if len(files) > 1 and burst and path.stem != filename_stem:
+            if len(exported_files) > 1 and burst and path.stem != filename_stem:
                 # skip any burst photo that's not the one we asked for
-                logging.debug(f"Skipping burst photo file {path}")
                 continue
             if filestem:
                 # rename the file based on filestem, keeping original extension
@@ -148,7 +180,6 @@ def _export_photo_uuid_applescript(
             else:
                 # use the name Photos provided
                 dest_new = dest / path.name
-            logging.debug(f"exporting {path} to dest_new: {dest_new}")
             if not dry_run:
                 FileUtil.copy(str(path), str(dest_new))
             exported_paths.append(str(dest_new))
@@ -310,6 +341,8 @@ def export2(
     convert_to_jpeg=False,
     jpeg_quality=1.0,
     ignore_date_modified=False,
+    use_photokit=False,
+    verbose=None,
 ):
     """ export photo, like export but with update and dry_run options
         dest: must be valid destination path or exception raised 
@@ -353,6 +386,7 @@ def export2(
         convert_to_jpeg: boolean; if True, converts non-jpeg images to jpeg
         jpeg_quality: float in range 0.0 <= jpeg_quality <= 1.0.  A value of 1.0 specifies use best quality, a value of 0.0 specifies use maximum compression.
         ignore_date_modified: for use with sidecar and exiftool; if True, sets EXIF:ModifyDate to EXIF:DateTimeOriginal even if date_modified is set
+        verbose: optional callable function to use for printing verbose text during processing; if None (default), does not print output.
 
         Returns: ExportResults namedtuple with fields: exported, new, updated, skipped 
                     where each field is a list of file paths
@@ -369,6 +403,12 @@ def export2(
     if export_db is None:
         export_db = ExportDBNoOp()
 
+    if verbose is None:
+        verbose = noop
+    elif not callable(verbose):
+        raise TypeError("verbose must be callable")
+    self._verbose = verbose
+
     # suffix to add to edited files
     # e.g. name will be filename_edited.jpg
     edited_identifier = "_edited"
@@ -388,6 +428,9 @@ def export2(
     # list of all files with utime touched (touch_file = True)
     touched_files = []
 
+    # list of all files convereted to jpeg
+    converted_to_jpeg_files = []
+
     # check edited and raise exception trying to export edited version of
     # photo that hasn't been edited
     if edited and not self.hasadjustments:
@@ -474,11 +517,6 @@ def export2(
                     f"Cannot export edited photo if path_edited is None"
                 )
         else:
-            if self.ismissing:
-                logging.debug(
-                    f"Attempting to export photo with ismissing=True: path = {self.path}"
-                )
-
             if self.path is not None:
                 src = self.path
             else:
@@ -487,32 +525,22 @@ def export2(
         if not os.path.isfile(src):
             raise FileNotFoundError(f"{src} does not appear to exist")
 
-        if not _check_export_suffix(src, dest, edited):
-            logging.debug(
-                f"Invalid destination suffix: {dest.suffix} for {self.path}, "
-                + f"edited={edited}, path_edited={self.path_edited}, "
-                + f"original_filename={self.original_filename}, filename={self.filename}"
-            )
-
         # found source now try to find right destination
         if update and dest.exists():
             # destination exists, check to see if destination is the right UUID
             dest_uuid = export_db.get_uuid_for_file(dest)
             if dest_uuid is None and fileutil.cmp(src, dest):
                 # might be exporting into a pre-ExportDB folder or the DB got deleted
-                logging.debug(
-                    f"Found matching file with blank uuid: {self.uuid}, {dest}"
-                )
                 dest_uuid = self.uuid
                 export_db.set_data(
-                    dest,
-                    self.uuid,
-                    fileutil.file_sig(dest),
-                    (None, None, None),
-                    (None, None, None),
-                    (None, None, None),
-                    self.json(),
-                    None,
+                    filename=dest,
+                    uuid=self.uuid,
+                    orig_stat=fileutil.file_sig(dest),
+                    exif_stat=(None, None, None),
+                    converted_stat=(None, None, None),
+                    edited_stat=(None, None, None),
+                    info_json=self.json(),
+                    exif_json=None,
                 )
             if dest_uuid != self.uuid:
                 # not the right file, find the right one
@@ -531,14 +559,14 @@ def export2(
                         dest = pathlib.Path(file_)
                         found_match = True
                         export_db.set_data(
-                            dest,
-                            self.uuid,
-                            fileutil.file_sig(dest),
-                            (None, None, None),
-                            (None, None, None),
-                            (None, None, None),
-                            self.json(),
-                            None,
+                            filename=dest,
+                            uuid=self.uuid,
+                            orig_stat=fileutil.file_sig(dest),
+                            exif_stat=(None, None, None),
+                            converted_stat=(None, None, None),
+                            edited_stat=(None, None, None),
+                            info_json=self.json(),
+                            exif_json=None,
                         )
                         break
 
@@ -575,6 +603,7 @@ def export2(
         update_updated_files = results.updated
         update_skipped_files = results.skipped
         touched_files = results.touched
+        converted_to_jpeg_files = results.converted_to_jpeg
 
         # copy live photo associated .mov if requested
         if live_photo and self.live_photo:
@@ -582,9 +611,6 @@ def export2(
             src_live = self.path_live_photo
 
             if src_live is not None:
-                logging.debug(
-                    f"Exporting live photo video of {filename} as {live_name.name}"
-                )
                 results = self._export_photo(
                     src_live,
                     live_name,
@@ -603,8 +629,7 @@ def export2(
                 update_updated_files.extend(results.updated)
                 update_skipped_files.extend(results.skipped)
                 touched_files.extend(results.touched)
-            else:
-                logging.debug(f"Skipping missing live movie for {filename}")
+                converted_to_jpeg_files.extend(results.converted_to_jpeg)
 
         # copy associated RAW image if requested
         if raw_photo and self.has_raw:
@@ -612,7 +637,6 @@ def export2(
             raw_ext = raw_path.suffix
             raw_name = dest.parent / f"{dest.stem}{raw_ext}"
             if raw_path is not None:
-                logging.debug(f"Exporting RAW photo of {filename} as {raw_name.name}")
                 results = self._export_photo(
                     raw_path,
                     raw_name,
@@ -632,8 +656,7 @@ def export2(
                 update_updated_files.extend(results.updated)
                 update_skipped_files.extend(results.skipped)
                 touched_files.extend(results.touched)
-            else:
-                logging.debug(f"Skipping missing RAW photo for {filename}")
+                converted_to_jpeg_files.extend(results.converted_to_jpeg)
     else:
         # use_photo_export
         exported = []
@@ -651,32 +674,73 @@ def export2(
                 # didn't get passed a filename, add _edited
                 filestem = f"{dest.stem}{edited_identifier}"
                 dest = dest.parent / f"{filestem}.jpeg"
-
-            exported = _export_photo_uuid_applescript(
-                self.uuid,
-                dest.parent,
-                filestem=filestem,
-                original=False,
-                edited=True,
-                live_photo=live_photo,
-                timeout=timeout,
-                burst=self.burst,
-                dry_run=dry_run,
-            )
+            if use_photokit:
+                photolib = PhotoLibrary()
+                photo = None
+                try:
+                    photo = photolib.fetch_uuid(self.uuid)
+                except PhotoKitFetchFailed:
+                    # if failed to find UUID, might be a burst photo
+                    if self.burst and self._info["burstUUID"]:
+                        bursts = photolib.fetch_burst_uuid(
+                            self._info["burstUUID"], all=True
+                        )
+                        # PhotoKit UUIDs may contain "/L0/001" so only look at beginning
+                        photo = [p for p in bursts if p.uuid.startswith(self.uuid)]
+                        photo = photo[0] if photo else None
+                if photo:
+                    exported = photo.export(
+                        dest.parent, dest.name, version=PHOTOS_VERSION_CURRENT
+                    )
+                else:
+                    exported = []
+            else:
+                exported = _export_photo_uuid_applescript(
+                    self.uuid,
+                    dest.parent,
+                    filestem=filestem,
+                    original=False,
+                    edited=True,
+                    live_photo=live_photo,
+                    timeout=timeout,
+                    burst=self.burst,
+                    dry_run=dry_run,
+                )
         else:
             # export original version and not edited
             filestem = dest.stem
-            exported = _export_photo_uuid_applescript(
-                self.uuid,
-                dest.parent,
-                filestem=filestem,
-                original=True,
-                edited=False,
-                live_photo=live_photo,
-                timeout=timeout,
-                burst=self.burst,
-                dry_run=dry_run,
-            )
+            if use_photokit:
+                photolib = PhotoLibrary()
+                photo = None
+                try:
+                    photo = photolib.fetch_uuid(self.uuid)
+                except PhotoKitFetchFailed:
+                    # if failed to find UUID, might be a burst photo
+                    if self.burst and self._info["burstUUID"]:
+                        bursts = photolib.fetch_burst_uuid(
+                            self._info["burstUUID"], all=True
+                        )
+                        # PhotoKit UUIDs may contain "/L0/001" so only look at beginning
+                        photo = [p for p in bursts if p.uuid.startswith(self.uuid)]
+                        photo = photo[0] if photo else None
+                if photo:
+                    exported = photo.export(
+                        dest.parent, dest.name, version=PHOTOS_VERSION_ORIGINAL
+                    )
+                else:
+                    exported = []
+            else:
+                exported = _export_photo_uuid_applescript(
+                    self.uuid,
+                    dest.parent,
+                    filestem=filestem,
+                    original=True,
+                    edited=False,
+                    live_photo=live_photo,
+                    timeout=timeout,
+                    burst=self.burst,
+                    dry_run=dry_run,
+                )
         if exported:
             if touch_file:
                 for exported_file in exported:
@@ -693,8 +757,9 @@ def export2(
             )
 
     # export metadata
+    sidecar_json_files_skipped = []
+    sidecar_json_files_written = []
     if sidecar_json:
-        logging.debug("writing exiftool_json_sidecar")
         sidecar_filename = dest.parent / pathlib.Path(f"{dest.stem}{dest.suffix}.json")
         sidecar_str = self._exiftool_json_sidecar(
             use_albums_as_keywords=use_albums_as_keywords,
@@ -703,15 +768,36 @@ def export2(
             description_template=description_template,
             ignore_date_modified=ignore_date_modified,
         )
-        if not dry_run:
-            try:
+        sidecar_digest = hexdigest(sidecar_str)
+        old_sidecar_digest, sidecar_sig = export_db.get_sidecar_for_file(
+            sidecar_filename
+        )
+        write_sidecar = (
+            not update
+            or (update and not sidecar_filename.exists())
+            or (
+                update
+                and (sidecar_digest != old_sidecar_digest)
+                or not fileutil.cmp_file_sig(sidecar_filename, sidecar_sig)
+            )
+        )
+        if write_sidecar:
+            verbose(f"Writing exiftool JSON sidecar {sidecar_filename}")
+            sidecar_json_files_written.append(str(sidecar_filename))
+            if not dry_run:
                 self._write_sidecar(sidecar_filename, sidecar_str)
-            except Exception as e:
-                logging.warning(f"Error writing json sidecar to {sidecar_filename}")
-                raise e
+                export_db.set_sidecar_for_file(
+                    sidecar_filename,
+                    sidecar_digest,
+                    fileutil.file_sig(sidecar_filename),
+                )
+        else:
+            verbose(f"Skipped up to date exiftool JSON sidecar {sidecar_filename}")
+            sidecar_json_files_skipped.append(str(sidecar_filename))
 
+    sidecar_xmp_files_skipped = []
+    sidecar_xmp_files_written = []
     if sidecar_xmp:
-        logging.debug("writing xmp_sidecar")
         sidecar_filename = dest.parent / pathlib.Path(f"{dest.stem}{dest.suffix}.xmp")
         sidecar_str = self._xmp_sidecar(
             use_albums_as_keywords=use_albums_as_keywords,
@@ -720,12 +806,32 @@ def export2(
             description_template=description_template,
             extension=dest.suffix[1:] if dest.suffix else None,
         )
-        if not dry_run:
-            try:
+        sidecar_digest = hexdigest(sidecar_str)
+        old_sidecar_digest, sidecar_sig = export_db.get_sidecar_for_file(
+            sidecar_filename
+        )
+        write_sidecar = (
+            not update
+            or (update and not sidecar_filename.exists())
+            or (
+                update
+                and (sidecar_digest != old_sidecar_digest)
+                or not fileutil.cmp_file_sig(sidecar_filename, sidecar_sig)
+            )
+        )
+        if write_sidecar:
+            verbose(f"Writing XMP sidecar {sidecar_filename}")
+            sidecar_xmp_files_written.append(str(sidecar_filename))
+            if not dry_run:
                 self._write_sidecar(sidecar_filename, sidecar_str)
-            except Exception as e:
-                logging.warning(f"Error writing xmp sidecar to {sidecar_filename}")
-                raise e
+                export_db.set_sidecar_for_file(
+                    sidecar_filename,
+                    sidecar_digest,
+                    fileutil.file_sig(sidecar_filename),
+                )
+        else:
+            verbose(f"Skipped up to date XMP sidecar {sidecar_filename}")
+            sidecar_xmp_files_skipped.append(str(sidecar_filename))
 
     # if exiftool, write the metadata
     if update:
@@ -736,7 +842,6 @@ def export2(
     exif_files_updated = []
     if exiftool and update and exif_files:
         for exported_file in exif_files:
-            logging.debug(f"checking exif for {exported_file}")
             files_are_different = False
             old_data = export_db.get_exifdata_for_file(exported_file)
             if old_data is not None:
@@ -756,6 +861,7 @@ def export2(
             if old_data is None or files_are_different:
                 # didn't have old data, assume we need to write it
                 # or files were different
+                verbose(f"Writing metadata with exiftool for {exported_file}")
                 if not dry_run:
                     self._write_exif_data(
                         exported_file,
@@ -779,8 +885,11 @@ def export2(
                     exported_file, fileutil.file_sig(exported_file)
                 )
                 exif_files_updated.append(exported_file)
+            else:
+                verbose(f"Skipped up to date exiftool metadata for {exported_file}")
     elif exiftool and exif_files:
         for exported_file in exif_files:
+            verbose(f"Writing metadata with exiftool for {exported_file}")
             if not dry_run:
                 self._write_exif_data(
                     exported_file,
@@ -808,6 +917,7 @@ def export2(
 
     if touch_file:
         for exif_file in exif_files_updated:
+            verbose(f"Updating file modification time for {exif_file}")
             touched_files.append(exif_file)
             ts = int(self.date.timestamp())
             fileutil.utime(exif_file, (ts, ts))
@@ -821,6 +931,11 @@ def export2(
         update_skipped_files,
         exif_files_updated,
         touched_files,
+        converted_to_jpeg_files,
+        sidecar_json_files_written,
+        sidecar_json_files_skipped,
+        sidecar_xmp_files_written,
+        sidecar_xmp_files_skipped,
     )
     return results
 
@@ -877,6 +992,7 @@ def _export_photo(
     update_new_files = []
     update_skipped_files = []
     touched_files = []
+    converted_to_jpeg_files = []
 
     dest_str = str(dest)
     dest_exists = dest.exists()
@@ -943,13 +1059,11 @@ def _export_photo(
 
         else:
             # update, destination doesn't exist (new file)
-            logging.debug(f"Update: exporting new file with {op_desc} {src} {dest}")
             update_new_files.append(dest_str)
             if touch_file:
                 touched_files.append(dest_str)
     else:
         # not update, export the file
-        logging.debug(f"Exporting file with {op_desc} {src} {dest}")
         exported_files.append(dest_str)
         if touch_file:
             sig = fileutil.file_sig(src)
@@ -961,9 +1075,6 @@ def _export_photo(
         edited_stat = fileutil.file_sig(src) if edited else (None, None, None)
         if dest_exists and (update or overwrite):
             # need to remove the destination first
-            logging.debug(
-                f"Update: removing existing file prior to {op_desc} {src} {dest}"
-            )
             fileutil.unlink(dest)
         if export_as_hardlink:
             fileutil.hardlink(src, dest)
@@ -971,18 +1082,19 @@ def _export_photo(
             # use convert_to_jpeg to export the file
             fileutil.convert_to_jpeg(src, dest_str, compression_quality=jpeg_quality)
             converted_stat = fileutil.file_sig(dest_str)
+            converted_to_jpeg_files.append(dest_str)
         else:
             fileutil.copy(src, dest_str, norsrc=no_xattr)
 
         export_db.set_data(
-            dest_str,
-            self.uuid,
-            fileutil.file_sig(dest_str),
-            (None, None, None),
-            converted_stat,
-            edited_stat,
-            self.json(),
-            None,
+            filename=dest_str,
+            uuid=self.uuid,
+            orig_stat=fileutil.file_sig(dest_str),
+            exif_stat=(None, None, None),
+            converted_stat=converted_stat,
+            edited_stat=edited_stat,
+            info_json=self.json(),
+            exif_json=None,
         )
 
     if touched_files:
@@ -996,6 +1108,11 @@ def _export_photo(
         update_skipped_files,
         [],
         touched_files,
+        converted_to_jpeg_files,
+        [],
+        [],
+        [],
+        [],
     )
 
 
@@ -1066,14 +1183,14 @@ def _exiftool_dict(
         IPTC:Keywords (may include album name, person name, or template)
         XMP:Subject
         XMP:PersonInImage
+        EXIF:GPSLatitudeRef, EXIF:GPSLongitudeRef
         EXIF:GPSLatitude, EXIF:GPSLongitude
         EXIF:GPSPosition
-        EXIF:GPSLatitudeRef, EXIF:GPSLongitudeRef
         EXIF:DateTimeOriginal
         EXIF:OffsetTimeOriginal
         EXIF:ModifyDate
-        IPTC:DigitalCreationDate
         IPTC:DateCreated
+        IPTC:TimeCreated
     """
 
     exif = {}
@@ -1187,9 +1304,11 @@ def _exiftool_dict(
     exif["EXIF:OffsetTimeOriginal"] = offsettime
 
     dateoriginal = date.strftime("%Y:%m:%d")
-    exif["IPTC:DigitalCreationDate"] = dateoriginal
     exif["IPTC:DateCreated"] = dateoriginal
 
+    timeoriginal = date.strftime(f"%H:%M:%S{offsettime}")
+    exif["IPTC:TimeCreated"] = timeoriginal
+
     if self.date_modified is not None and not ignore_date_modified:
         exif["EXIF:ModifyDate"] = self.date_modified.strftime("%Y:%m:%d %H:%M:%S")
     else:
@@ -1226,9 +1345,9 @@ def _exiftool_json_sidecar(
         IPTC:Keywords (may include album name, person name, or template)
         XMP:Subject
         XMP:PersonInImage
+        EXIF:GPSLatitudeRef, EXIF:GPSLongitudeRef
         EXIF:GPSLatitude, EXIF:GPSLongitude
         EXIF:GPSPosition
-        EXIF:GPSLatitudeRef, EXIF:GPSLongitudeRef
         EXIF:DateTimeOriginal
         EXIF:OffsetTimeOriginal
         EXIF:ModifyDate

osxphotos/photoinfo/photoinfo.py

@@ -91,9 +91,10 @@ class PhotoInfo:
             and self.raw_original
         ):
             # return the JPEG version as that's what Photos 5+ does
-            return self._info["raw_pair_info"]["originalFilename"]
+            original_name = self._info["raw_pair_info"]["originalFilename"]
         else:
-            return self._info["originalFilename"]
+            original_name = self._info["originalFilename"]
+        return original_name or self.filename
 
     @property
     def date(self):
@@ -164,6 +165,8 @@ class PhotoInfo:
                         photopath = os.path.join(
                             self._db._masters_path, self._info["imagePath"]
                         )
+                if not os.path.isfile(photopath):
+                    photopath = None
                 self._path = photopath
                 return photopath
 
@@ -175,6 +178,8 @@ class PhotoInfo:
                     self._info["directory"],
                     self._info["filename"],
                 )
+                if not os.path.isfile(photopath):
+                    photopath = None
                 self._path = photopath
                 return photopath
 
@@ -188,6 +193,8 @@ class PhotoInfo:
                     self._info["directory"],
                     self._info["filename"],
                 )
+            if not os.path.isfile(photopath):
+                photopath = None
             self._path = photopath
             return photopath
 

osxphotos/photosdb/photosdb.py

@@ -35,6 +35,7 @@ from .._constants import (
 from .._version import __version__
 from ..albuminfo import AlbumInfo, FolderInfo, ImportInfo
 from ..datetime_utils import datetime_has_tz, datetime_naive_to_local
+from ..fileutil import FileUtil
 from ..personinfo import PersonInfo
 from ..photoinfo import PhotoInfo
 from ..utils import (
@@ -546,6 +547,33 @@ class PhotosDB:
 
         return dest_path
 
+    # NOTE: This method seems to cause problems with applescript
+    # Bummer...would'be been nice to avoid copying the DB
+    # def _link_db_file(self, fname):
+    #     """ links the sqlite database file to a temp file """
+    #     """ returns the name of the temp file """
+    #     """ If sqlite shared memory and write-ahead log files exist, those are copied too """
+    #     # required because python's sqlite3 implementation can't read a locked file
+    #     # _, suffix = os.path.splitext(fname)
+    #     dest_name = dest_path = ""
+    #     try:
+    #         dest_name = pathlib.Path(fname).name
+    #         dest_path = os.path.join(self._tempdir_name, dest_name)
+    #         FileUtil.hardlink(fname, dest_path)
+    #         # link write-ahead log and shared memory files (-wal and -shm) files if they exist
+    #         if os.path.exists(f"{fname}-wal"):
+    #             FileUtil.hardlink(f"{fname}-wal", f"{dest_path}-wal")
+    #         if os.path.exists(f"{fname}-shm"):
+    #             FileUtil.hardlink(f"{fname}-shm", f"{dest_path}-shm")
+    #     except:
+    #         print("Error linking " + fname + " to " + dest_path, file=sys.stderr)
+    #         raise Exception
+
+    #     if _debug():
+    #         logging.debug(dest_path)
+
+    #     return dest_path
+
     def _process_database4(self):
         """ process the Photos database to extract info
             works on Photos version <= 4.0 """

osxphotos/phototemplate.py

@@ -18,17 +18,40 @@ from functools import partial
 
 from ._constants import _UNKNOWN_PERSON
 from .datetime_formatter import DateTimeFormatter
+from .exiftool import ExifTool
 from .path_utils import sanitize_dirname, sanitize_filename, sanitize_pathpart
 
 # ensure locale set to user's locale
 locale.setlocale(locale.LC_ALL, "")
 
+PHOTO_VIDEO_TYPE_DEFAULTS = {"photo": "photo", "video": "video"}
+
+MEDIA_TYPE_DEFAULTS = {
+    "selfie": "selfie",
+    "time_lapse": "time_lapse",
+    "panorama": "panorama",
+    "slow_mo": "slow_mo",
+    "screenshot": "screenshot",
+    "portrait": "portrait",
+    "live_photo": "live_photo",
+    "burst": "burst",
+    "photo": "photo",
+    "video": "video",
+}
+
 # Permitted substitutions (each of these returns a single value or None)
 TEMPLATE_SUBSTITUTIONS = {
     "{name}": "Current filename of the photo",
     "{original_name}": "Photo's original filename when imported to Photos",
     "{title}": "Title of the photo",
     "{descr}": "Description of the photo",
+    "{media_type}": (
+        f"Special media type resolved in this precedence: {', '.join(t for t in MEDIA_TYPE_DEFAULTS)}. "
+        "Defaults to 'photo' or 'video' if no special type. "
+        "Customize one or more media types using format: '{media_type,video=vidéo;time_lapse=vidéo_accélérée}'"
+    ),
+    "{photo_or_video}": "'photo' or 'video' depending on what type the image is. To customize, use default value as in '{photo_or_video,photo=fotos;video=videos}'",
+    "{hdr}": "Photo is HDR?; True/False value, use in format '{hdr?VALUE_IF_TRUE,VALUE_IF_FALSE}'",
     "{created.date}": "Photo's creation date in ISO format, e.g. '2020-03-22'",
     "{created.year}": "4-digit year of photo creation time",
     "{created.yy}": "2-digit year of photo creation time",
@@ -104,6 +127,10 @@ TEMPLATE_SUBSTITUTIONS_MULTI_VALUED = {
     "{label}": "Image categorization label associated with a photo (Photos 5 only)",
     "{label_normalized}": "All lower case version of 'label' (Photos 5 only)",
     "{comment}": "Comment(s) on shared Photos; format is 'Person name: comment text' (Photos 5 only)",
+    "{exiftool:GROUP:TAGNAME}": "Use exiftool (https://exiftool.org) to extract metadata, in form GROUP:TAGNAME, from image.  "
+    "E.g. '{exiftool:EXIF:Make}' to get camera make, or {exiftool:IPTC:Keywords} to extract keywords. "
+    "See https://exiftool.org/TagNames/ for list of valid tag names.  You must specify group (e.g. EXIF, IPTC, etc) "
+    "as used in `exiftool -G`. exiftool must be installed in the path to use this template.",
 }
 
 # Just the multi-valued substitution names without the braces
@@ -128,6 +155,62 @@ class PhotoTemplate:
         # gets initialized in get_template_value
         self.today = None
 
+    def make_subst_function(
+        self, none_str, filename, dirname, replacement, get_func=None
+    ):
+        """ returns: substitution function for use in re.sub 
+            none_str: value to use if substitution lookup is None and no default provided
+            get_func: function that gets the substitution value for a given template field
+                    default is get_template_value which handles the single-value fields """
+
+        if get_func is None:
+            # used by make_subst_function to get the value for a template substitution
+            get_func = partial(
+                self.get_template_value,
+                filename=filename,
+                dirname=dirname,
+                replacement=replacement,
+            )
+
+        # closure to capture photo, none_str, filename, dirname in subst
+        def subst(matchobj):
+            groups = len(matchobj.groups())
+            if groups != 5:
+                raise ValueError(
+                    f"Unexpected number of groups: expected 4, got {groups}"
+                )
+
+            delim = matchobj.group(1)
+            field = matchobj.group(2)
+            path_sep = matchobj.group(3)
+            bool_val = matchobj.group(4)
+            default = matchobj.group(5)
+
+            # drop the '+' on delim
+            delim = delim[:-1] if delim is not None else None
+            # drop () from path_sep
+            path_sep = path_sep.strip("()") if path_sep is not None else None
+            # drop the ? on bool_val
+            bool_val = bool_val[1:] if bool_val is not None else None
+            # drop the comma on default
+            default_val = default[1:] if default is not None else None
+
+            try:
+                val = get_func(field, default_val, bool_val, delim, path_sep)
+            except ValueError:
+                return matchobj.group(0)
+
+            if val is None:
+                # field valid but didn't match a value
+                if default == ",":
+                    val = ""
+                else:
+                    val = default_val if default_val is not None else none_str
+
+            return val
+
+        return subst
+
     def render(
         self,
         template,
@@ -144,7 +227,7 @@ class PhotoTemplate:
         Args:
             template: str template 
             none_str: str to use default for None values, default is '_' 
-            path_sep: optional character to use as path separator, default is os.path.sep
+            path_sep: optional string to use as path separator, default is os.path.sep
             expand_inplace: expand multi-valued substitutions in-place as a single string 
                 instead of returning individual strings
             inplace_sep: optional string to use as separator between multi-valued keywords
@@ -159,8 +242,6 @@ class PhotoTemplate:
 
         if path_sep is None:
             path_sep = os.path.sep
-        elif path_sep is not None and len(path_sep) != 1:
-            raise ValueError(f"path_sep must be single character: {path_sep}")
 
         if inplace_sep is None:
             inplace_sep = ","
@@ -174,51 +255,21 @@ class PhotoTemplate:
         #          there would be 6 possible renderings (2 albums x 3 persons)
 
         # regex to find {template_field,optional_default} in strings
-        # for explanation of regex see https://regex101.com/r/4JJg42/1
         # pylint: disable=anomalous-backslash-in-string
-        regex = r"(?<!\{)\{([^\\,}]+)(,{0,1}(([\w\-\%. ]+))?)(?=\}(?!\}))\}"
+        regex = (
+            r"(?<!\{)\{"  # match { but not {{
+            + r"([^}]*\+)?"  # group 1: optional DELIM+
+            + r"([^\\,}+\?]+)"  # group 2: field name
+            + r"(\([^{}\)]*\))?"  # group 3: optional (PATH_SEP)
+            + r"(\?[^\\,}]*)?"  # group 4: optional ?TRUE_VALUE for boolean fields
+            + r"(,[\w\=\;\-\%. ]*)?"  # group 5: optional ,DEFAULT
+            + r"(?=\}(?!\}))\}"  # match } but not }}
+        )
+
         if type(template) is not str:
             raise TypeError(f"template must be type str, not {type(template)}")
 
-        # used by make_subst_function to get the value for a template substitution
-        get_func = partial(
-            self.get_template_value,
-            filename=filename,
-            dirname=dirname,
-            replacement=replacement,
-        )
-
-        def make_subst_function(self, none_str, get_func=get_func):
-            """ returns: substitution function for use in re.sub 
-                none_str: value to use if substitution lookup is None and no default provided
-                get_func: function that gets the substitution value for a given template field
-                        default is get_template_value which handles the single-value fields """
-
-            # closure to capture photo, none_str, filename, dirname in subst
-            def subst(matchobj):
-                groups = len(matchobj.groups())
-                if groups == 4:
-                    try:
-                        val = get_func(matchobj.group(1), matchobj.group(3))
-                    except ValueError:
-                        return matchobj.group(0)
-
-                    if val is None:
-                        val = (
-                            matchobj.group(3)
-                            if matchobj.group(3) is not None
-                            else none_str
-                        )
-
-                    return val
-                else:
-                    raise ValueError(
-                        f"Unexpected number of groups: expected 4, got {groups}"
-                    )
-
-            return subst
-
-        subst_func = make_subst_function(self, none_str)
+        subst_func = self.make_subst_function(none_str, filename, dirname, replacement)
 
         # do the replacements
         rendered = re.sub(regex, subst_func, template)
@@ -246,82 +297,37 @@ class PhotoTemplate:
         #    '2011/Album2/keyword1/person1',
         #    '2011/Album2/keyword2/person1',]
 
-        rendered_strings = [rendered]
-        for field in MULTI_VALUE_SUBSTITUTIONS:
-            # Build a regex that matches only the field being processed
-            re_str = r"(?<!\\)\{(" + field + r")(,(([\w\-\%. ]{0,})))?\}"
-            regex_multi = re.compile(re_str)
+        rendered_strings = self._render_multi_valued_templates(
+            rendered,
+            none_str,
+            path_sep,
+            expand_inplace,
+            inplace_sep,
+            filename,
+            dirname,
+            replacement,
+        )
 
-            # holds each of the new rendered_strings, dict to avoid repeats (dict.keys())
-            new_strings = {}
-
-            for str_template in rendered_strings:
-                if regex_multi.search(str_template):
-                    values = self.get_template_value_multi(
-                        field,
-                        path_sep,
-                        filename=filename,
-                        dirname=dirname,
-                        replacement=replacement,
-                    )
-                    if expand_inplace:
-                        # instead of returning multiple strings, join values into a single string
-                        val = (
-                            inplace_sep.join(sorted(values))
-                            if values and values[0]
-                            else None
-                        )
-
-                        def lookup_template_value_multi(lookup_value, _):
-                            """ Closure passed to make_subst_function get_func 
-                                    Capture val and field in the closure 
-                                    Allows make_subst_function to be re-used w/o modification
-                                    _ is not used but required so signature matches get_template_value """
-                            if lookup_value == field:
-                                return val
-                            else:
-                                raise ValueError(f"Unexpected value: {lookup_value}")
-
-                        subst = make_subst_function(
-                            self, none_str, get_func=lookup_template_value_multi
-                        )
-                        new_string = regex_multi.sub(subst, str_template)
-
-                        # update rendered_strings for the next field to process
-                        rendered_strings = {new_string}
-                    else:
-                        # create a new template string for each value
-                        for val in values:
-
-                            def lookup_template_value_multi(lookup_value, _):
-                                """ Closure passed to make_subst_function get_func 
-                                    Capture val and field in the closure 
-                                    Allows make_subst_function to be re-used w/o modification
-                                    _ is not used but required so signature matches get_template_value """
-                                if lookup_value == field:
-                                    return val
-                                else:
-                                    raise ValueError(
-                                        f"Unexpected value: {lookup_value}"
-                                    )
-
-                            subst = make_subst_function(
-                                self, none_str, get_func=lookup_template_value_multi
-                            )
-                            new_string = regex_multi.sub(subst, str_template)
-                            new_strings[new_string] = 1
-
-                        # update rendered_strings for the next field to process
-                        rendered_strings = list(new_strings.keys())
+        # process exiftool: templates
+        rendered_strings = self._render_exiftool_template(
+            rendered_strings,
+            none_str,
+            path_sep,
+            expand_inplace,
+            inplace_sep,
+            filename,
+            dirname,
+            replacement,
+        )
 
         # find any {fields} that weren't replaced
         unmatched = []
         for rendered_str in rendered_strings:
             unmatched.extend(
                 [
-                    no_match[0]
+                    no_match[1]
                     for no_match in re.findall(regex, rendered_str)
-                    if no_match[0] not in unmatched
+                    if no_match[1] not in unmatched
                 ]
             )
 
@@ -338,14 +344,263 @@ class PhotoTemplate:
 
         return rendered_strings, unmatched
 
+    def _render_multi_valued_templates(
+        self,
+        rendered,
+        none_str,
+        path_sep,
+        expand_inplace,
+        inplace_sep,
+        filename,
+        dirname,
+        replacement,
+    ):
+        rendered_strings = [rendered]
+        new_rendered_strings = []
+        while new_rendered_strings != rendered_strings:
+            new_rendered_strings = rendered_strings
+            for field in MULTI_VALUE_SUBSTITUTIONS:
+                # Build a regex that matches only the field being processed
+                re_str = (
+                    r"(?<!\{)\{"  # match { but not {{
+                    + r"([^}]*\+)?"  # group 1: optional DELIM+
+                    + r"("
+                    + field  # group 2: field name
+                    + r")"
+                    + r"(\([^{}\)]*\))?"  # group 3: optional (PATH_SEP)
+                    + r"(\?[^\\,}]*)?"  # group 4: optional ?TRUE_VALUE for boolean fields
+                    + r"(,[\w\=\;\-\%. ]*)?"  # group 5: optional ,DEFAULT
+                    + r"(?=\}(?!\}))\}"  # match } but not }}
+                )
+                regex_multi = re.compile(re_str)
+
+                # holds each of the new rendered_strings, dict to avoid repeats (dict.keys())
+                new_strings = {}
+
+                for str_template in rendered_strings:
+                    matches = regex_multi.search(str_template)
+                    if matches:
+                        path_sep = (
+                            matches.group(3).strip("()")
+                            if matches.group(3) is not None
+                            else path_sep
+                        )
+                        values = self.get_template_value_multi(
+                            field,
+                            path_sep,
+                            filename=filename,
+                            dirname=dirname,
+                            replacement=replacement,
+                        )
+                        if expand_inplace or matches.group(1) is not None:
+                            delim = (
+                                matches.group(1)[:-1]
+                                if matches.group(1) is not None
+                                else inplace_sep
+                            )
+                            # instead of returning multiple strings, join values into a single string
+                            val = (
+                                delim.join(sorted(values))
+                                if values and values[0]
+                                else None
+                            )
+
+                            def lookup_template_value_multi(lookup_value, *_):
+                                """ Closure passed to make_subst_function get_func 
+                                        Capture val and field in the closure 
+                                        Allows make_subst_function to be re-used w/o modification
+                                        _ is not used but required so signature matches get_template_value """
+                                if lookup_value == field:
+                                    return val
+                                else:
+                                    raise ValueError(
+                                        f"Unexpected value: {lookup_value}"
+                                    )
+
+                            subst = self.make_subst_function(
+                                none_str,
+                                filename,
+                                dirname,
+                                replacement,
+                                get_func=lookup_template_value_multi,
+                            )
+                            new_string = regex_multi.sub(subst, str_template)
+
+                            # update rendered_strings for the next field to process
+                            rendered_strings = list({new_string})
+                        else:
+                            # create a new template string for each value
+                            for val in values:
+
+                                def lookup_template_value_multi(lookup_value, *_):
+                                    """ Closure passed to make_subst_function get_func 
+                                        Capture val and field in the closure 
+                                        Allows make_subst_function to be re-used w/o modification
+                                        _ is not used but required so signature matches get_template_value """
+                                    if lookup_value == field:
+                                        return val
+                                    else:
+                                        raise ValueError(
+                                            f"Unexpected value: {lookup_value}"
+                                        )
+
+                                subst = self.make_subst_function(
+                                    none_str,
+                                    filename,
+                                    dirname,
+                                    replacement,
+                                    get_func=lookup_template_value_multi,
+                                )
+                                new_string = regex_multi.sub(subst, str_template)
+                                new_strings[new_string] = 1
+
+                            # update rendered_strings for the next field to process
+                            rendered_strings = sorted(list(new_strings.keys()))
+        return rendered_strings
+
+    def _render_exiftool_template(
+        self,
+        rendered_strings,
+        none_str,
+        path_sep,
+        expand_inplace,
+        inplace_sep,
+        filename,
+        dirname,
+        replacement,
+    ):
+        # TODO: lots of code commonality with render_multi_valued_templates -- combine or pull out
+        # TODO: put these in globals
+        if path_sep is None:
+            path_sep = os.path.sep
+
+        if inplace_sep is None:
+            inplace_sep = ","
+
+        # Build a regex that matches only the field being processed
+        # todo: pull out regexes into globals?
+        re_str = (
+            r"(?<!\{)\{"  # match { but not {{
+            + r"([^}]*\+)?"  # group 1: optional DELIM+
+            + r"(exiftool:[^\\,}+\?]+)"  # group 3 field name
+            + r"(\([^{}\)]*\))?"  # group 3: optional (PATH_SEP)
+            + r"(\?[^\\,}]*)?"  # group 4: optional ?TRUE_VALUE for boolean fields
+            + r"(,[\w\=\;\-\%. ]*)?"  # group 5: optional ,DEFAULT
+            + r"(?=\}(?!\}))\}"  # match } but not }}
+        )
+        regex_multi = re.compile(re_str)
+
+        # holds each of the new rendered_strings, dict to avoid repeats (dict.keys())
+        new_rendered_strings = []
+        while new_rendered_strings != rendered_strings:
+            new_rendered_strings = rendered_strings
+            new_strings = {}
+            for str_template in rendered_strings:
+                matches = regex_multi.search(str_template)
+                if matches:
+                    # allmatches = regex_multi.finditer(str_template)
+                    # for matches in allmatches:
+                    path_sep = (
+                        matches.group(3).strip("()")
+                        if matches.group(3) is not None
+                        else path_sep
+                    )
+                    field = matches.group(2)
+                    subfield = field[9:]
+
+                    if not self.photo.path:
+                        values = []
+                    else:
+                        exif = ExifTool(self.photo.path)
+                        exifdict = exif.asdict()
+                        exifdict = {k.lower(): v for (k, v) in exifdict.items()}
+                        subfield = subfield.lower()
+                        if subfield in exifdict:
+                            values = exifdict[subfield]
+                            values = (
+                                [values] if not isinstance(values, list) else values
+                            )
+                        else:
+                            values = [None]
+                    if expand_inplace or matches.group(1) is not None:
+                        delim = (
+                            matches.group(1)[:-1]
+                            if matches.group(1) is not None
+                            else inplace_sep
+                        )
+                        # instead of returning multiple strings, join values into a single string
+                        val = (
+                            delim.join(sorted(values)) if values and values[0] else None
+                        )
+
+                        def lookup_template_value_exif(lookup_value, *_):
+                            """ Closure passed to make_subst_function get_func 
+                                    Capture val and field in the closure 
+                                    Allows make_subst_function to be re-used w/o modification
+                                    _ is not used but required so signature matches get_template_value """
+                            if lookup_value == field:
+                                return val
+                            else:
+                                raise ValueError(f"Unexpected value: {lookup_value}")
+
+                        subst = self.make_subst_function(
+                            none_str,
+                            filename,
+                            dirname,
+                            replacement,
+                            get_func=lookup_template_value_exif,
+                        )
+                        new_string = regex_multi.sub(subst, str_template)
+                        # update rendered_strings for the next field to process
+                        rendered_strings = list({new_string})
+                    else:
+                        # create a new template string for each value
+                        for val in values:
+
+                            def lookup_template_value_exif(lookup_value, *_):
+                                """ Closure passed to make_subst_function get_func 
+                                    Capture val and field in the closure 
+                                    Allows make_subst_function to be re-used w/o modification
+                                    _ is not used but required so signature matches get_template_value """
+                                if lookup_value == field:
+                                    return val
+                                else:
+                                    raise ValueError(
+                                        f"Unexpected value: {lookup_value}"
+                                    )
+
+                            subst = self.make_subst_function(
+                                none_str,
+                                filename,
+                                dirname,
+                                replacement,
+                                get_func=lookup_template_value_exif,
+                            )
+                            new_string = regex_multi.sub(subst, str_template)
+                            new_strings[new_string] = 1
+                        # update rendered_strings for the next field to process
+                        rendered_strings = sorted(list(new_strings.keys()))
+        return rendered_strings
+
     def get_template_value(
-        self, field, default, filename=False, dirname=False, replacement=":"
+        self,
+        field,
+        default,
+        bool_val=None,
+        delim=None,
+        path_sep=None,
+        filename=False,
+        dirname=False,
+        replacement=":",
     ):
         """lookup value for template field (single-value template substitutions)
 
         Args:
             field: template field to find value for.
             default: the default value provided by the user
+            bool_val: True value if expression is boolean 
+            delim: delimiter for expand in place
+            path_sep: path separator for fields that are path-like
             filename: if True, template output will be sanitized to produce valid file name
             dirname: if True, template output will be sanitized to produce valid directory name 
             replacement: str, value to replace any illegal file path characters with; default = ":"
@@ -372,6 +627,12 @@ class PhotoTemplate:
             value = self.photo.title
         elif field == "descr":
             value = self.photo.description
+        elif field == "media_type":
+            value = self.get_media_type(default)
+        elif field == "photo_or_video":
+            value = self.get_photo_video_type(default)
+        elif field == "hdr":
+            value = self.get_photo_hdr(default, bool_val)
         elif field == "created.date":
             value = DateTimeFormatter(self.photo.date).date
         elif field == "created.year":
@@ -606,6 +867,7 @@ class PhotoTemplate:
         """
 
         """ return list of values for a multi-valued template field """
+        values = [] 
         if field == "album":
             values = self.photo.albums
         elif field == "keyword":
@@ -649,7 +911,7 @@ class PhotoTemplate:
             values = [
                 f"{comment.user}: {comment.text}" for comment in self.photo.comments
             ]
-        else:
+        elif not field.startswith("exiftool:"):
             raise ValueError(f"Unhandled template value: {field}")
 
         # sanitize directory names if needed, folder_album handled differently above
@@ -667,3 +929,66 @@ class PhotoTemplate:
         values = values or [None]
         return values
 
+    def get_photo_video_type(self, default):
+        """ return media type, e.g. photo or video """
+        default_dict = parse_default_kv(default, PHOTO_VIDEO_TYPE_DEFAULTS)
+        if self.photo.isphoto:
+            return default_dict["photo"]
+        else:
+            return default_dict["video"]
+
+    def get_media_type(self, default):
+        """ return special media type, e.g. slow_mo, panorama, etc., defaults to photo or video if no special type """
+        default_dict = parse_default_kv(default, MEDIA_TYPE_DEFAULTS)
+        p = self.photo
+        if p.selfie:
+            return default_dict["selfie"]
+        elif p.time_lapse:
+            return default_dict["time_lapse"]
+        elif p.panorama:
+            return default_dict["panorama"]
+        elif p.slow_mo:
+            return default_dict["slow_mo"]
+        elif p.screenshot:
+            return default_dict["screenshot"]
+        elif p.portrait:
+            return default_dict["portrait"]
+        elif p.live_photo:
+            return default_dict["live_photo"]
+        elif p.burst:
+            return default_dict["burst"]
+        elif p.ismovie:
+            return default_dict["video"]
+        else:
+            return default_dict["photo"]
+
+    def get_photo_hdr(self, default, bool_val):
+        if self.photo.hdr:
+            return bool_val
+        else:
+            return default
+
+
+def parse_default_kv(default, default_dict):
+    """ parse a string in form key1=value1;key2=value2,... as used for some template fields
+
+    Args:
+        default: str, in form 'photo=foto;video=vidéo'
+        default_dict: dict, in form {"photo": "fotos", "video": "vidéos"} with default values
+
+    Returns:
+        dict in form {"photo": "fotos", "video": "vidéos"}
+    """
+
+    default_dict_ = default_dict.copy()
+    if default:
+        defaults = default.split(";")
+        for kv in defaults:
+            try:
+                k, v = kv.split("=")
+                k = k.strip()
+                v = v.strip()
+                default_dict_[k] = v
+            except ValueError:
+                pass
+    return default_dict_

osxphotos/templates/xmp_sidecar.mako

@@ -12,7 +12,7 @@
     % if desc is None:
         <dc:description></dc:description>
     % else:
-        <dc:description>${desc}</dc:description>
+        <dc:description>${desc | x}</dc:description>
     % endif
 </%def>
 
@@ -20,7 +20,7 @@
     % if title is None:
         <dc:title></dc:title>
     % else:
-        <dc:title>${title}</dc:title>
+        <dc:title>${title | x}</dc:title>
     % endif
 </%def>
 
@@ -30,7 +30,7 @@
         <dc:subject>
             <rdf:Seq>
             % for subj in subject:
-                <rdf:li>${subj}</rdf:li>
+                <rdf:li>${subj | x}</rdf:li>
             % endfor
             </rdf:Seq>
         </dc:subject>
@@ -48,7 +48,7 @@
         <Iptc4xmpExt:PersonInImage>
             <rdf:Bag>
                 % for person in persons:
-                    <rdf:li>${person}</rdf:li>
+                    <rdf:li>${person | x}</rdf:li>
                 % endfor
             </rdf:Bag>
         </Iptc4xmpExt:PersonInImage>
@@ -60,7 +60,7 @@
         <digiKam:TagsList>
             <rdf:Seq>
             % for keyword in keywords:
-                <rdf:li>${keyword}</rdf:li>
+                <rdf:li>${keyword | x}</rdf:li>
             % endfor
             </rdf:Seq>
         </digiKam:TagsList>
@@ -81,10 +81,8 @@
 
 <%def name="gps_info(latitude, longitude)">
     % if latitude is not None and longitude is not None:
-        <exif:GPSLongitudeRef>${"E" if longitude >= 0 else "W"}</exif:GPSLongitudeRef>
-        <exif:GPSLongitude>${abs(longitude)}</exif:GPSLongitude>
-        <exif:GPSLatitude>${abs(latitude)}</exif:GPSLatitude>
-        <exif:GPSLatitudeRef>${"N" if latitude >= 0 else "S"}</exif:GPSLatitudeRef>
+        <exif:GPSLongitude>${int(abs(longitude))},${(abs(longitude) % 1) * 60}${"E" if longitude >= 0 else "W"}</exif:GPSLongitude>
+        <exif:GPSLatitude>${int(abs(latitude))},${(abs(latitude) % 1) * 60}${"N" if latitude >= 0 else "S"}</exif:GPSLatitude>
     % endif
 </%def>
 

osxphotos/utils.py

@@ -17,7 +17,6 @@ from plistlib import load as plistload
 import CoreFoundation
 import CoreServices
 import objc
-from Foundation import *
 
 from ._constants import UNICODE_FORMAT
 from .fileutil import FileUtil
@@ -57,10 +56,12 @@ def _debug():
     """ returns True if debugging turned on (via _set_debug), otherwise, false """
     return _DEBUG
 
+
 def noop(*args, **kwargs):
     """ do nothing (no operation) """
     pass
 
+
 def _get_os_version():
     # returns tuple containing OS version
     # e.g. 10.13.6 = (10, 13, 6)
@@ -200,7 +201,7 @@ def get_last_library_path():
         # pylint: disable=no-member
         # pylint: disable=undefined-variable
         photosurl = CoreFoundation.CFURLCreateByResolvingBookmarkData(
-            kCFAllocatorDefault, photosurlref, 0, None, None, None, None
+            CoreFoundation.kCFAllocatorDefault, photosurlref, 0, None, None, None, None
         )
 
         # the CFURLRef we got is a sruct that python treats as an array
@@ -361,9 +362,35 @@ def _db_is_locked(dbname):
 
 def normalize_unicode(value):
     """ normalize unicode data """
-    if value is not None:
-        if not isinstance(value, str):
-            raise ValueError("value must be str")
-        return unicodedata.normalize(UNICODE_FORMAT, value)
-    else:
+    if value is None:
         return None
+
+    if not isinstance(value, str):
+        raise ValueError("value must be str")
+    return unicodedata.normalize(UNICODE_FORMAT, value)
+
+
+def increment_filename(filepath):
+    """ Return filename (1).ext, etc if filename.ext exists
+
+        If file exists in filename's parent folder with same stem as filename, 
+        add (1), (2), etc. until a non-existing filename is found.
+
+    Args:
+        filepath: str; full path, including file name
+
+    Returns:
+        new filepath (or same if not incremented)
+
+    Note: This obviously is subject to race condition so using with caution.
+    """
+    dest = pathlib.Path(str(filepath))
+    count = 1
+    dest_files = findfiles(f"{dest.stem}*", str(dest.parent))
+    dest_files = [pathlib.Path(f).stem.lower() for f in dest_files]
+    dest_new = dest.stem
+    while dest_new.lower() in dest_files:
+        dest_new = f"{dest.stem} ({count})"
+        count += 1
+    dest = dest.parent / f"{dest_new}{dest.suffix}"
+    return str(dest)

requirements.txt

@@ -47,6 +47,7 @@ parso==0.6.2
 pathspec==0.7.0
 pathvalidate==2.2.1
 pexpect==4.8.0
+photoscript==0.1.0
 pickleshare==0.7.5
 Pillow==7.2.0
 pkginfo==1.5.0.1

setup.py

@@ -79,6 +79,7 @@ setup(
         "pathvalidate==2.2.1",
         "dataclasses==0.7;python_version<'3.7'",
         "wurlitzer>=2.0.1",
+        "photoscript>=0.1.0",
     ],
     entry_points={"console_scripts": ["osxphotos=osxphotos.__main__:cli"]},
     include_package_data=True,

tests/tempdiskimage.py

@@ -0,0 +1,114 @@
+""" Create a temporary disk image on MacOS """
+
+import pathlib
+import platform
+import subprocess
+import tempfile
+import time
+
+
+class TempDiskImage:
+    """ Create and mount a temporary disk image """
+
+    def __init__(self, size=100, prefix=None):
+        """ Create and mount a temporary disk image.
+        
+        Args:
+            size: int; size in MB of disk image, default = 100
+            prefix: str; optional prefix to prepend to name of the temporary disk image
+            name: str; name of the mounted volume, default = "TemporaryDiskImage"
+
+        Raises:
+            TypeError if size is not int
+            RunTimeError if not on MacOS
+        """
+        if type(size) != int:
+            raise TypeError("size must be int")
+
+        system = platform.system()
+        if system != "Darwin":
+            raise RuntimeError("TempDiskImage only runs on MacOS")
+
+        self._tempdir = tempfile.TemporaryDirectory()
+        # hacky mktemp: this could create a race condition but unlikely given it's created in a TemporaryDirectory
+        prefix = "TemporaryDiskImage" if prefix is None else prefix
+        volume_name = f"{prefix}_{str(time.time()).replace('.','_')}_{str(time.perf_counter()).replace('.','_')}"
+        image_name = f"{volume_name}.dmg"
+        image_path = pathlib.Path(self._tempdir.name) / image_name
+        hdiutil = subprocess.run(
+            [
+                "/usr/bin/hdiutil",
+                "create",
+                "-size",
+                f"{size}m",
+                "-fs",
+                "HFS+",
+                "-volname",
+                volume_name,
+                image_path,
+            ],
+            check=True,
+            text=True,
+            capture_output=True,
+        )
+
+        if "created" not in hdiutil.stdout:
+            raise OSError(f"Could not create DMG {image_path}")
+
+        self.path = image_path
+        self._mount_point, self.name = self._mount_image(self.path)
+
+    def _mount_image(self, image_path):
+        """ mount a DMG file and return path, returns (mount_point, path) """
+        hdiutil = subprocess.run(
+            ["/usr/bin/hdiutil", "attach", image_path],
+            check=True,
+            text=True,
+            capture_output=True,
+        )
+        mount_point, path = None, None
+        for line in hdiutil.stdout.split("\n"):
+            line = line.strip()
+            if "Apple_HFS" not in line:
+                continue
+            output = line.split()
+            if len(output) < 3:
+                raise ValueError(f"Error mounting disk image {image_path}")
+            mount_point = output[0]
+            path = output[2]
+            break
+        return (mount_point, path)
+
+    def unmount(self):
+        try:
+            if self._mount_point:
+                hdiutil = subprocess.run(
+                    ["/usr/bin/hdiutil", "detach", self._mount_point],
+                    check=True,
+                    text=True,
+                    capture_output=True,
+                )
+                self._mount_point = None
+        except AttributeError:
+            pass
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        self.unmount()
+        if exc_type:
+            return False
+
+
+if __name__ == "__main__":
+    # Create a temporary disk image, 50mb in size
+    img = TempDiskImage(size=50, prefix="MyDiskImage")
+    # Be sure to unmount it, image will be cleaned up automatically
+    img.unmount()
+
+    # Or use it as a context handler
+    # Default values are 100mb and prefix = "TemporaryDiskImage"
+    with TempDiskImage() as img:
+        print(f"image: {img.path}")
+        print(f"mounted at: {img.name}")

tests/test_catalina_10_15_6.py

@@ -177,6 +177,12 @@ RAW_DICT = {
     ),
 }
 
+ORIGINAL_FILENAME_DICT = {
+    "uuid": "D79B8D77-BFFC-460B-9312-034F2877D35B",
+    "filename": "D79B8D77-BFFC-460B-9312-034F2877D35B.jpeg",
+    "original_filename": "Pumkins2.jpg",
+}
+
 
 @pytest.fixture(scope="module")
 def photosdb():
@@ -864,6 +870,27 @@ def test_export_14(photosdb, caplog):
     assert "Invalid destination suffix" not in caplog.text
 
 
+def test_export_no_original_filename(photosdb):
+    # test export OK if original filename is null
+    # issue #267
+
+    tempdir = tempfile.TemporaryDirectory(prefix="osxphotos_")
+    dest = tempdir.name
+    photos = photosdb.photos(uuid=[UUID_DICT["export"]])
+
+    # monkey patch original_filename for testing
+    original_filename = photos[0]._info["originalFilename"]
+    photos[0]._info["originalFilename"] = None
+    filename = f"{photos[0].uuid}.jpeg"
+    expected_dest = os.path.join(dest, filename)
+    got_dest = photos[0].export(dest)[0]
+
+    assert got_dest == expected_dest
+    assert os.path.isfile(got_dest)
+
+    photos[0]._info["originalFilename"] = original_filename
+
+
 def test_eq():
     """ Test equality of two PhotoInfo objects """
 
@@ -1070,3 +1097,18 @@ def test_verbose(capsys):
     photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB, verbose=print)
     captured = capsys.readouterr()
     assert "Processing database" in captured.out
+
+
+def test_original_filename(photosdb):
+    """ test original filename """
+    uuid = ORIGINAL_FILENAME_DICT["uuid"]
+    photo = photosdb.get_photo(uuid)
+    assert photo.original_filename == ORIGINAL_FILENAME_DICT["original_filename"]
+    assert photo.filename == ORIGINAL_FILENAME_DICT["filename"]
+
+    # monkey patch
+    original_filename = photo._info["originalFilename"]
+    photo._info["originalFilename"] = None
+    assert photo.original_filename == ORIGINAL_FILENAME_DICT["filename"]
+    photo._info["originalFilename"] = original_filename
+

tests/test_cli.py

@@ -17,6 +17,7 @@ PLACES_PHOTOS_DB_13 = "tests/Test-Places-High-Sierra-10.13.6.photoslibrary"
 PHOTOS_DB_15_4 = "tests/Test-10.15.4.photoslibrary"
 PHOTOS_DB_15_5 = "tests/Test-10.15.5.photoslibrary"
 PHOTOS_DB_15_6 = "tests/Test-10.15.6.photoslibrary"
+PHOTOS_DB_15_7 = "tests/Test-10.15.7.photoslibrary"
 PHOTOS_DB_TOUCH = PHOTOS_DB_15_6
 PHOTOS_DB_14_6 = "tests/Test-10.14.6.photoslibrary"
 
@@ -65,6 +66,7 @@ CLI_EXPORT_FILENAMES_ALBUM_UNICODE = ["IMG_4547.jpg"]
 CLI_EXPORT_FILENAMES_DELETED_TWIN = ["wedding.jpg", "wedding_edited.jpeg"]
 
 CLI_EXPORT_EDITED_SUFFIX = "_bearbeiten"
+CLI_EXPORT_ORIGINAL_SUFFIX = "_original"
 
 CLI_EXPORT_FILENAMES_EDITED_SUFFIX = [
     "Pumkins1.jpg",
@@ -77,6 +79,16 @@ CLI_EXPORT_FILENAMES_EDITED_SUFFIX = [
     "wedding_bearbeiten.jpeg",
 ]
 
+CLI_EXPORT_FILENAMES_ORIGINAL_SUFFIX = [
+    "Pumkins1_original.jpg",
+    "Pumkins2_original.jpg",
+    "Pumpkins3_original.jpg",
+    "St James Park_original.jpg",
+    "St James Park_edited.jpeg",
+    "Tulips_original.jpg",
+    "wedding_original.jpg",
+    "wedding_edited.jpeg",
+]
 
 CLI_EXPORT_FILENAMES_CURRENT = [
     "1EB2B765-0765-43BA-A90C-0D0580E6172C.jpeg",
@@ -976,7 +988,9 @@ def test_export_exiftool_ignore_date_modified():
             )
             assert result.exit_code == 0
 
-            exif = ExifTool(CLI_EXIFTOOL_IGNORE_DATE_MODIFIED[uuid]["File:FileName"]).asdict()
+            exif = ExifTool(
+                CLI_EXIFTOOL_IGNORE_DATE_MODIFIED[uuid]["File:FileName"]
+            ).asdict()
             for key in CLI_EXIFTOOL_IGNORE_DATE_MODIFIED[uuid]:
                 assert exif[key] == CLI_EXIFTOOL_IGNORE_DATE_MODIFIED[uuid][key]
 
@@ -1008,6 +1022,33 @@ def test_export_edited_suffix():
         assert sorted(files) == sorted(CLI_EXPORT_FILENAMES_EDITED_SUFFIX)
 
 
+def test_export_original_suffix():
+    """ test export with --original-suffix """
+    import glob
+    import os
+    import os.path
+    import osxphotos
+    from osxphotos.__main__ import export
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with runner.isolated_filesystem():
+        result = runner.invoke(
+            export,
+            [
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "--original-suffix",
+                CLI_EXPORT_ORIGINAL_SUFFIX,
+                "-V",
+            ],
+        )
+        assert result.exit_code == 0
+        files = glob.glob("*")
+        assert sorted(files) == sorted(CLI_EXPORT_FILENAMES_ORIGINAL_SUFFIX)
+
+
 @pytest.mark.skipif(
     "OSXPHOTOS_TEST_CONVERT" not in os.environ,
     reason="Skip if running in Github actions, no GPU.",
@@ -1734,6 +1775,142 @@ def test_export_sidecar_templates():
         )
 
 
+def test_export_sidecar_update():
+    """ test sidecar don't update if not changed and do update if changed """
+    import datetime
+    import glob
+    import os
+    import os.path
+    import osxphotos
+    from osxphotos.fileutil import FileUtil
+
+    from osxphotos.__main__ import cli
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with runner.isolated_filesystem():
+        result = runner.invoke(
+            cli,
+            [
+                "export",
+                "--db",
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "--sidecar=json",
+                "--sidecar=xmp",
+                f"--uuid={CLI_EXPORT_UUID}",
+                "-V",
+            ],
+        )
+        assert result.exit_code == 0
+        assert "Writing XMP sidecar" in result.output
+        assert "Writing exiftool JSON sidecar" in result.output
+
+        # delete a sidecar file and run update
+        fileutil = FileUtil()
+        fileutil.unlink(CLI_EXPORT_SIDECAR_FILENAMES[1])
+
+        result = runner.invoke(
+            cli,
+            [
+                "export",
+                "--db",
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "--sidecar=json",
+                "--sidecar=xmp",
+                f"--uuid={CLI_EXPORT_UUID}",
+                "-V",
+                "--update",
+            ],
+        )
+        assert result.exit_code == 0
+        assert "Skipped up to date XMP sidecar" in result.output
+        assert "Writing exiftool JSON sidecar" in result.output
+
+        # run update again, no sidecar files should update
+        result = runner.invoke(
+            cli,
+            [
+                "export",
+                "--db",
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "--sidecar=json",
+                "--sidecar=xmp",
+                f"--uuid={CLI_EXPORT_UUID}",
+                "-V",
+                "--update",
+            ],
+        )
+        assert result.exit_code == 0
+        assert "Skipped up to date XMP sidecar" in result.output
+        assert "Skipped up to date exiftool JSON sidecar" in result.output
+
+        # touch a file and export again
+        ts = datetime.datetime.now().timestamp() + 1000
+        fileutil.utime(CLI_EXPORT_SIDECAR_FILENAMES[2], (ts, ts))
+
+        result = runner.invoke(
+            cli,
+            [
+                "export",
+                "--db",
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "--sidecar=json",
+                "--sidecar=xmp",
+                f"--uuid={CLI_EXPORT_UUID}",
+                "-V",
+                "--update",
+            ],
+        )
+        assert result.exit_code == 0
+        assert "Writing XMP sidecar" in result.output
+        assert "Skipped up to date exiftool JSON sidecar" in result.output
+
+        # run update again, no sidecar files should update
+        result = runner.invoke(
+            cli,
+            [
+                "export",
+                "--db",
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "--sidecar=json",
+                "--sidecar=xmp",
+                f"--uuid={CLI_EXPORT_UUID}",
+                "-V",
+                "--update",
+            ],
+        )
+        assert result.exit_code == 0
+        assert "Skipped up to date XMP sidecar" in result.output
+        assert "Skipped up to date exiftool JSON sidecar" in result.output
+
+        # run update again with updated metadata, forcing update
+        result = runner.invoke(
+            cli,
+            [
+                "export",
+                "--db",
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "--sidecar=json",
+                "--sidecar=xmp",
+                f"--uuid={CLI_EXPORT_UUID}",
+                "-V",
+                "--update",
+                "--keyword-template",
+                "foo",
+            ],
+        )
+        assert result.exit_code == 0
+        assert "Writing XMP sidecar" in result.output
+        assert "Writing exiftool JSON sidecar" in result.output
+
+
 def test_export_live():
     import glob
     import os
@@ -2693,8 +2870,8 @@ def test_export_sidecar_keyword_template():
             "EXIF:DateTimeOriginal": "2018:09:28 16:07:07", 
             "EXIF:CreateDate": "2018:09:28 16:07:07", 
             "EXIF:OffsetTimeOriginal": "-04:00", 
-            "IPTC:DigitalCreationDate": "2018:09:28", 
-            "IPTC:DateCreated": "2018:09:28", 
+            "IPTC:DateCreated": "2018:09:28",
+            "IPTC:TimeCreated": "16:07:07-04:00",
             "EXIF:ModifyDate": "2018:09:28 16:07:07"}]
             """
         )[0]
@@ -3587,3 +3764,116 @@ def test_persons():
 
     json_got = json.loads(result.output)
     assert json_got == PERSONS_JSON
+
+
+def test_export_report():
+    """ test export with --report option """
+    import glob
+    import os
+    import os.path
+    import osxphotos
+    from osxphotos.__main__ import export
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with runner.isolated_filesystem():
+        result = runner.invoke(
+            export,
+            [os.path.join(cwd, CLI_PHOTOS_DB), ".", "-V", "--report", "report.csv"],
+        )
+        assert result.exit_code == 0
+        assert "Writing export report" in result.output
+        assert os.path.exists("report.csv")
+
+
+def test_export_report_not_a_file():
+    """ test export with --report option and bad report value """
+    import glob
+    import os
+    import os.path
+    import osxphotos
+    from osxphotos.__main__ import export
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with runner.isolated_filesystem():
+        result = runner.invoke(
+            export, [os.path.join(cwd, CLI_PHOTOS_DB), ".", "-V", "--report", "."]
+        )
+        assert result.exit_code != 0
+        assert "Aborted!" in result.output
+
+
+def test_export_as_hardlink_download_missing():
+    """ test export with incompatible export options """
+    import glob
+    import os
+    import os.path
+    import osxphotos
+    from osxphotos.__main__ import export
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with runner.isolated_filesystem():
+        result = runner.invoke(
+            export,
+            [
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "-V",
+                "--download-missing",
+                "--export-as-hardlink",
+                ".",
+            ],
+        )
+        assert result.exit_code != 0
+        assert "Aborted!" in result.output
+
+
+def test_export_missing():
+    """ test export with --missing """
+    import glob
+    import os
+    import os.path
+    import osxphotos
+    from osxphotos.__main__ import export
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with runner.isolated_filesystem():
+        result = runner.invoke(
+            export,
+            [
+                os.path.join(cwd, PHOTOS_DB_15_7),
+                ".",
+                "-V",
+                "--missing",
+                "--download-missing",
+                ".",
+            ],
+        )
+        assert result.exit_code == 0
+        assert "Exporting 2 photos" in result.output
+
+
+def test_export_missing_not_download_missing():
+    """ test export with incompatible export options """
+    import glob
+    import os
+    import os.path
+    import osxphotos
+    from osxphotos.__main__ import export
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with runner.isolated_filesystem():
+        result = runner.invoke(
+            export, [os.path.join(cwd, CLI_PHOTOS_DB), ".", "-V", "--missing", "."]
+        )
+        assert result.exit_code != 0
+        assert "Aborted!" in result.output

tests/test_exiftool.py

@@ -103,10 +103,28 @@ def test_setvalue_1():
 
     exif = osxphotos.exiftool.ExifTool(tempfile)
     exif.setvalue("IPTC:Keywords", "test")
+    assert not exif.error
+
     exif._read_exif()
     assert exif.data["IPTC:Keywords"] == "test"
 
 
+def test_setvalue_error():
+    # test setting illegal tag value generates error
+    import os.path
+    import tempfile
+    import osxphotos.exiftool
+    from osxphotos.fileutil import FileUtil
+
+    tempdir = tempfile.TemporaryDirectory(prefix="osxphotos_")
+    tempfile = os.path.join(tempdir.name, os.path.basename(TEST_FILE_ONE_KEYWORD))
+    FileUtil.copy(TEST_FILE_ONE_KEYWORD, tempfile)
+
+    exif = osxphotos.exiftool.ExifTool(tempfile)
+    exif.setvalue("IPTC:Foo", "test")
+    assert exif.error
+
+
 def test_setvalue_context_manager():
     # test setting a tag value as context manager
     import os.path
@@ -124,6 +142,8 @@ def test_setvalue_context_manager():
         exif.setvalue("XMP:Title", "title")
         exif.setvalue("XMP:Subject", "subject")
 
+    assert exif.error is None
+
     exif2 = osxphotos.exiftool.ExifTool(tempfile)
     exif2._read_exif()
     assert sorted(exif2.data["IPTC:Keywords"]) == ["test1", "test2"]
@@ -131,6 +151,22 @@ def test_setvalue_context_manager():
     assert exif2.data["XMP:Subject"] == "subject"
 
 
+def test_setvalue_context_manager_error():
+    # test setting a tag value as context manager when error generated
+    import os.path
+    import tempfile
+    import osxphotos.exiftool
+    from osxphotos.fileutil import FileUtil
+
+    tempdir = tempfile.TemporaryDirectory(prefix="osxphotos_")
+    tempfile = os.path.join(tempdir.name, os.path.basename(TEST_FILE_ONE_KEYWORD))
+    FileUtil.copy(TEST_FILE_ONE_KEYWORD, tempfile)
+
+    with osxphotos.exiftool.ExifTool(tempfile) as exif:
+        exif.setvalue("Foo:Bar", "test1")
+    assert exif.error
+
+
 def test_clear_value():
     # test clearing a tag value
     import os.path

tests/test_export_catalina_10_15_6_use_photos_export.py

@@ -11,9 +11,9 @@ pytestmark = pytest.mark.skipif(
 PHOTOS_DB = "/Users/rhet/Pictures/Photos Library.photoslibrary"
 
 UUID_DICT = {
-    "has_adjustments": "A8111956-E900-4DEC-9191-A04A87C07BC5",
-    "no_adjustments": "EA7BB55F-92F1-4818-94E3-E8DEDC6B2E31",
-    "live": "9032C168-9319-40C0-8210-5ADC42F4C603",
+    "has_adjustments": "2B2D5434-6D31-49E2-BF47-B973D34A317B",
+    "no_adjustments": "A8D646C3-89A9-4D74-8001-4EB46BA55B94",
+    "live": "BFF29EBD-22DF-4FCF-9817-317E7104EA50",
 }
 
 

tests/test_export_catalina_10_15_7.py

@@ -78,8 +78,8 @@ EXIF_JSON_EXPECTED = """
     "EXIF:DateTimeOriginal": "2019:04:15 14:40:24", 
     "EXIF:CreateDate": "2019:04:15 14:40:24", 
     "EXIF:OffsetTimeOriginal": "-04:00", 
-    "IPTC:DigitalCreationDate": "2019:04:15", 
     "IPTC:DateCreated": "2019:04:15", 
+    "IPTC:TimeCreated": "14:40:24-04:00", 
     "EXIF:ModifyDate": "2019:07:27 17:33:28"}]
     """
 
@@ -94,8 +94,8 @@ EXIF_JSON_EXPECTED_IGNORE_DATE_MODIFIED = """
     "EXIF:DateTimeOriginal": "2019:04:15 14:40:24", 
     "EXIF:CreateDate": "2019:04:15 14:40:24", 
     "EXIF:OffsetTimeOriginal": "-04:00", 
-    "IPTC:DigitalCreationDate": "2019:04:15", 
     "IPTC:DateCreated": "2019:04:15", 
+    "IPTC:TimeCreated": "14:40:24-04:00", 
     "EXIF:ModifyDate": "2019:04:15 14:40:24"}]
     """
 
@@ -554,8 +554,8 @@ def test_exiftool_json_sidecar_keyword_template_long(caplog):
         "EXIF:DateTimeOriginal": "2019:04:15 14:40:24", 
         "EXIF:CreateDate": "2019:04:15 14:40:24", 
         "EXIF:OffsetTimeOriginal": "-04:00", 
-        "IPTC:DigitalCreationDate": "2019:04:15", 
         "IPTC:DateCreated": "2019:04:15", 
+        "IPTC:TimeCreated": "14:40:24-04:00", 
         "EXIF:ModifyDate": "2019:07:27 17:33:28"}]
         """
     )[0]
@@ -604,8 +604,8 @@ def test_exiftool_json_sidecar_keyword_template():
         "EXIF:DateTimeOriginal": "2019:04:15 14:40:24", 
         "EXIF:CreateDate": "2019:04:15 14:40:24", 
         "EXIF:OffsetTimeOriginal": "-04:00", 
-        "IPTC:DigitalCreationDate": "2019:04:15", 
         "IPTC:DateCreated": "2019:04:15", 
+        "IPTC:TimeCreated": "14:40:24-04:00", 
         "EXIF:ModifyDate": "2019:07:27 17:33:28"}]
         """
     )[0]
@@ -666,8 +666,8 @@ def test_exiftool_json_sidecar_use_persons_keyword():
         "EXIF:DateTimeOriginal": "2018:09:28 15:35:49", 
         "EXIF:CreateDate": "2018:09:28 15:35:49", 
         "EXIF:OffsetTimeOriginal": "-04:00", 
-        "IPTC:DigitalCreationDate": "2018:09:28", 
         "IPTC:DateCreated": "2018:09:28", 
+        "IPTC:TimeCreated": "15:35:49-04:00", 
         "EXIF:ModifyDate": "2018:09:28 15:35:49"}]
         """
     )[0]
@@ -709,8 +709,8 @@ def test_exiftool_json_sidecar_use_albums_keyword():
         "EXIF:DateTimeOriginal": "2018:09:28 15:35:49", 
         "EXIF:CreateDate": "2018:09:28 15:35:49", 
         "EXIF:OffsetTimeOriginal": "-04:00", 
-        "IPTC:DigitalCreationDate": "2018:09:28", 
         "IPTC:DateCreated": "2018:09:28", 
+        "IPTC:TimeCreated": "15:35:49-04:00", 
         "EXIF:ModifyDate": "2018:09:28 15:35:49"}]
         """
     )[0]
@@ -744,7 +744,7 @@ def test_xmp_sidecar_is_valid(tmp_path):
     xmp_file = tmp_path / XMP_FILENAME
     assert xmp_file.is_file()
     exiftool = ExifTool(str(xmp_file))
-    output = exiftool.run_commands("-validate", "-warning")
+    output, _ = exiftool.run_commands("-validate", "-warning")
     assert output == b"[ExifTool]      Validate                        : 0 0 0"
 
 
@@ -1029,7 +1029,7 @@ def test_xmp_sidecar_gps():
             xmlns:photoshop="http://ns.adobe.com/photoshop/1.0/">
         <photoshop:SidecarForExtension>jpg</photoshop:SidecarForExtension>
         <dc:description></dc:description>
-        <dc:title>St. James's Park</dc:title>
+        <dc:title>St. James&#39;s Park</dc:title>
         <!-- keywords and persons listed in <dc:subject> as Photos does -->
         <dc:subject>
             <rdf:Seq>
@@ -1038,7 +1038,7 @@ def test_xmp_sidecar_gps():
                 <rdf:li>London</rdf:li>
                 <rdf:li>United Kingdom</rdf:li>
                 <rdf:li>London 2018</rdf:li>
-                <rdf:li>St. James's Park</rdf:li>
+                <rdf:li>St. James&#39;s Park</rdf:li>
             </rdf:Seq>
         </dc:subject>
         <photoshop:DateCreated>2018-10-13T09:18:12.501000-04:00</photoshop:DateCreated>
@@ -1055,7 +1055,7 @@ def test_xmp_sidecar_gps():
                 <rdf:li>London</rdf:li>
                 <rdf:li>United Kingdom</rdf:li>
                 <rdf:li>London 2018</rdf:li>
-                <rdf:li>St. James's Park</rdf:li>
+                <rdf:li>St. James&#39;s Park</rdf:li>
             </rdf:Seq>
         </digiKam:TagsList>
         </rdf:Description>
@@ -1066,10 +1066,8 @@ def test_xmp_sidecar_gps():
         </rdf:Description>
         <rdf:Description rdf:about=""
             xmlns:exif='http://ns.adobe.com/exif/1.0/'>
-        <exif:GPSLongitudeRef>W</exif:GPSLongitudeRef>
-        <exif:GPSLongitude>0.1318055</exif:GPSLongitude>
-        <exif:GPSLatitude>51.50357167</exif:GPSLatitude>
-        <exif:GPSLatitudeRef>N</exif:GPSLatitudeRef>
+        <exif:GPSLongitude>0,7.908329999999999W</exif:GPSLongitude>
+        <exif:GPSLatitude>51,30.21430019999997N</exif:GPSLatitude>
         </rdf:Description>
    </rdf:RDF>
 </x:xmpmeta>"""

tests/test_export_convert_to_jpeg.py

@@ -20,10 +20,10 @@ NAMES_DICT = {
     "heic": "7783E8E6-9CAC-40F3-BE22-81FB7051C266.jpeg",
 }
 
-UUID_LIVE_HEIC = "1337F3F6-5C9F-4FC7-80CC-BD9A5B928F72"
+UUID_LIVE_HEIC = "612CE30B-3D8F-417A-9B14-EC42CBA10ACC"
 NAMES_LIVE_HEIC = [
-    "1337F3F6-5C9F-4FC7-80CC-BD9A5B928F72.jpeg",
-    "1337F3F6-5C9F-4FC7-80CC-BD9A5B928F72.mov",
+    "612CE30B-3D8F-417A-9B14-EC42CBA10ACC.jpeg",
+    "612CE30B-3D8F-417A-9B14-EC42CBA10ACC.mov",
 ]
 
 

tests/test_export_db.py

@@ -4,6 +4,8 @@ import pytest
 
 EXIF_DATA = """[{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", "EXIF:ImageDescription": "\u2068Elder Park\u2069, \u2068Adelaide\u2069, \u2068Australia\u2069", "XMP:Description": "\u2068Elder Park\u2069, \u2068Adelaide\u2069, \u2068Australia\u2069", "XMP:Title": "Elder Park", "EXIF:GPSLatitude": "34 deg 55' 8.01\" S", "EXIF:GPSLongitude": "138 deg 35' 48.70\" E", "Composite:GPSPosition": "34 deg 55' 8.01\" S, 138 deg 35' 48.70\" E", "EXIF:GPSLatitudeRef": "South", "EXIF:GPSLongitudeRef": "East", "EXIF:DateTimeOriginal": "2017:06:20 17:18:56", "EXIF:OffsetTimeOriginal": "+09:30", "EXIF:ModifyDate": "2020:05:18 14:42:04"}]"""
 INFO_DATA = """{"uuid": "3DD2C897-F19E-4CA6-8C22-B027D5A71907", "filename": "3DD2C897-F19E-4CA6-8C22-B027D5A71907.jpeg", "original_filename": "IMG_4547.jpg", "date": "2017-06-20T17:18:56.518000+09:30", "description": "\u2068Elder Park\u2069, \u2068Adelaide\u2069, \u2068Australia\u2069", "title": "Elder Park", "keywords": [], "labels": ["Statue", "Art"], "albums": ["AlbumInFolder"], "folders": {"AlbumInFolder": ["Folder1", "SubFolder2"]}, "persons": [], "path": "/Users/rhet/Pictures/Test-10.15.4.photoslibrary/originals/3/3DD2C897-F19E-4CA6-8C22-B027D5A71907.jpeg", "ismissing": false, "hasadjustments": true, "external_edit": false, "favorite": false, "hidden": false, "latitude": -34.91889167000001, "longitude": 138.59686167, "path_edited": "/Users/rhet/Pictures/Test-10.15.4.photoslibrary/resources/renders/3/3DD2C897-F19E-4CA6-8C22-B027D5A71907_1_201_a.jpeg", "shared": false, "isphoto": true, "ismovie": false, "uti": "public.jpeg", "burst": false, "live_photo": false, "path_live_photo": null, "iscloudasset": false, "incloud": null, "date_modified": "2020-05-18T14:42:04.608664+09:30", "portrait": false, "screenshot": false, "slow_mo": false, "time_lapse": false, "hdr": false, "selfie": false, "panorama": false, "has_raw": false, "uti_raw": null, "path_raw": null, "place": {"name": "Elder Park, Adelaide, South Australia, Australia, River Torrens", "names": {"field0": [], "country": ["Australia"], "state_province": ["South Australia"], "sub_administrative_area": ["Adelaide"], "city": ["Adelaide", "Adelaide"], "field5": [], "additional_city_info": ["Adelaide CBD", "Tarndanya"], "ocean": [], "area_of_interest": ["Elder Park", ""], "inland_water": ["River Torrens", "River Torrens"], "field10": [], "region": [], "sub_throughfare": [], "field13": [], "postal_code": [], "field15": [], "field16": [], "street_address": [], "body_of_water": ["River Torrens", "River Torrens"]}, "country_code": "AU", "ishome": false, "address_str": "River Torrens, Adelaide SA, Australia", "address": {"street": null, "sub_locality": "Tarndanya", "city": "Adelaide", "sub_administrative_area": "Adelaide", "state_province": "SA", "postal_code": null, "country": "Australia", "iso_country_code": "AU"}}, "exif": {"flash_fired": false, "iso": 320, "metering_mode": 3, "sample_rate": null, "track_format": null, "white_balance": 0, "aperture": 2.2, "bit_rate": null, "duration": null, "exposure_bias": 0.0, "focal_length": 4.15, "fps": null, "latitude": null, "longitude": null, "shutter_speed": 0.058823529411764705, "camera_make": "Apple", "camera_model": "iPhone 6s", "codec": null, "lens_model": "iPhone 6s back camera 4.15mm f/2.2"}}"""
+SIDECAR_DATA = """FOO_BAR"""
+
 EXIF_DATA2 = """[{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", "XMP:Title": "St. James's Park", "XMP:TagsList": ["London 2018", "St. James's Park", "England", "United Kingdom", "UK", "London"], "IPTC:Keywords": ["London 2018", "St. James's Park", "England", "United Kingdom", "UK", "London"], "XMP:Subject": ["London 2018", "St. James's Park", "England", "United Kingdom", "UK", "London"], "EXIF:GPSLatitude": "51 deg 30' 12.86\" N", "EXIF:GPSLongitude": "0 deg 7' 54.50\" W", "Composite:GPSPosition": "51 deg 30' 12.86\" N, 0 deg 7' 54.50\" W", "EXIF:GPSLatitudeRef": "North", "EXIF:GPSLongitudeRef": "West", "EXIF:DateTimeOriginal": "2018:10:13 09:18:12", "EXIF:OffsetTimeOriginal": "-04:00", "EXIF:ModifyDate": "2019:12:08 14:06:44"}]"""
 INFO_DATA2 = """{"uuid": "F2BB3F98-90F0-4E4C-A09B-25C6822A4529", "filename": "F2BB3F98-90F0-4E4C-A09B-25C6822A4529.jpeg", "original_filename": "IMG_8440.JPG", "date": "2019-06-11T11:42:06.711805-07:00", "description": null, "title": null, "keywords": [], "labels": ["Sky", "Cloudy", "Fence", "Land", "Outdoor", "Park", "Amusement Park", "Roller Coaster"], "albums": [], "folders": {}, "persons": [], "path": "/Volumes/MacBook Catalina - Data/Users/rhet/Pictures/Photos Library.photoslibrary/originals/F/F2BB3F98-90F0-4E4C-A09B-25C6822A4529.jpeg", "ismissing": false, "hasadjustments": false, "external_edit": false, "favorite": false, "hidden": false, "latitude": 33.81558666666667, "longitude": -117.99298, "path_edited": null, "shared": false, "isphoto": true, "ismovie": false, "uti": "public.jpeg", "burst": false, "live_photo": false, "path_live_photo": null, "iscloudasset": true, "incloud": true, "date_modified": "2019-10-14T00:51:47.141950-07:00", "portrait": false, "screenshot": false, "slow_mo": false, "time_lapse": false, "hdr": false, "selfie": false, "panorama": false, "has_raw": false, "uti_raw": null, "path_raw": null, "place": {"name": "Adventure City, Stanton, California, United States", "names": {"field0": [], "country": ["United States"], "state_province": ["California"], "sub_administrative_area": ["Orange"], "city": ["Stanton", "Anaheim", "Anaheim"], "field5": [], "additional_city_info": ["West Anaheim"], "ocean": [], "area_of_interest": ["Adventure City", "Adventure City"], "inland_water": [], "field10": [], "region": [], "sub_throughfare": [], "field13": [], "postal_code": [], "field15": [], "field16": [], "street_address": [], "body_of_water": []}, "country_code": "US", "ishome": false, "address_str": "Adventure City, 1240 S Beach Blvd, Anaheim, CA  92804, United States", "address": {"street": "1240 S Beach Blvd", "sub_locality": "West Anaheim", "city": "Stanton", "sub_administrative_area": "Orange", "state_province": "CA", "postal_code": "92804", "country": "United States", "iso_country_code": "US"}}, "exif": {"flash_fired": false, "iso": 25, "metering_mode": 5, "sample_rate": null, "track_format": null, "white_balance": 0, "aperture": 2.2, "bit_rate": null, "duration": null, "exposure_bias": 0.0, "focal_length": 4.15, "fps": null, "latitude": null, "longitude": null, "shutter_speed": 0.0004940711462450593, "camera_make": "Apple", "camera_model": "iPhone 6s", "codec": null, "lens_model": "iPhone 6s back camera 4.15mm f/2.2"}}"""
 DATABASE_VERSION1 = "tests/export_db_version1.db"
@@ -41,6 +43,8 @@ def test_export_db():
     assert db.get_stat_edited_for_file(filepath) == (10, 11, 12)
     db.set_stat_converted_for_file(filepath, (7, 8, 9))
     assert db.get_stat_converted_for_file(filepath) == (7, 8, 9)
+    db.set_sidecar_for_file(filepath, SIDECAR_DATA, (13, 14, 15))
+    assert db.get_sidecar_for_file(filepath) == (SIDECAR_DATA, (13, 14, 15))
 
     # test set_data which sets all at the same time
     filepath2 = os.path.join(tempdir.name, "test2.jpg")
@@ -109,6 +113,8 @@ def test_export_db_no_op():
     assert db.get_stat_converted_for_file(filepath) is None
     db.set_stat_edited_for_file(filepath, (10, 11, 12))
     assert db.get_stat_edited_for_file(filepath) is None
+    db.set_sidecar_for_file(filepath, SIDECAR_DATA, (13, 14, 15))
+    assert db.get_sidecar_for_file(filepath) == (None, (None, None, None))
 
     # test set_data which sets all at the same time
     filepath2 = os.path.join(tempdir.name, "test2.jpg")
@@ -160,6 +166,7 @@ def test_export_db_in_memory():
     db.set_stat_exif_for_file(filepath, (4, 5, 6))
     db.set_stat_converted_for_file(filepath, (7, 8, 9))
     db.set_stat_edited_for_file(filepath, (10, 11, 12))
+    db.set_sidecar_for_file(filepath, SIDECAR_DATA, (13, 14, 15))
 
     db.close()
 
@@ -176,6 +183,7 @@ def test_export_db_in_memory():
     assert dbram.get_stat_exif_for_file(filepath) == (4, 5, 6)
     assert dbram.get_stat_converted_for_file(filepath) == (7, 8, 9)
     assert dbram.get_stat_edited_for_file(filepath) == (10, 11, 12)
+    assert dbram.get_sidecar_for_file(filepath) == (SIDECAR_DATA, (13, 14, 15))
 
     # change a value
     dbram.set_uuid_for_file(filepath, "FUBAR")
@@ -185,6 +193,7 @@ def test_export_db_in_memory():
     dbram.set_stat_exif_for_file(filepath, (10, 11, 12))
     dbram.set_stat_converted_for_file(filepath, (1, 2, 3))
     dbram.set_stat_edited_for_file(filepath, (4, 5, 6))
+    dbram.set_sidecar_for_file(filepath, "FUBAR", (20, 21, 22))
 
     assert dbram.get_uuid_for_file(filepath_lower) == "FUBAR"
     assert dbram.get_info_for_uuid("FUBAR") == INFO_DATA2
@@ -193,6 +202,7 @@ def test_export_db_in_memory():
     assert dbram.get_stat_exif_for_file(filepath) == (10, 11, 12)
     assert dbram.get_stat_converted_for_file(filepath) == (1, 2, 3)
     assert dbram.get_stat_edited_for_file(filepath) == (4, 5, 6)
+    assert dbram.get_sidecar_for_file(filepath) == ("FUBAR", (20, 21, 22))
 
     dbram.close()
 
@@ -205,6 +215,7 @@ def test_export_db_in_memory():
     assert db.get_stat_exif_for_file(filepath) == (4, 5, 6)
     assert db.get_stat_converted_for_file(filepath) == (7, 8, 9)
     assert db.get_stat_edited_for_file(filepath) == (10, 11, 12)
+    assert db.get_sidecar_for_file(filepath) == (SIDECAR_DATA, (13, 14, 15))
 
     assert db.get_info_for_uuid("FUBAR") is None
 
@@ -232,6 +243,7 @@ def test_export_db_in_memory_nofile():
     dbram.set_stat_exif_for_file(filepath, (10, 11, 12))
     dbram.set_stat_converted_for_file(filepath, (1, 2, 3))
     dbram.set_stat_edited_for_file(filepath, (4, 5, 6))
+    dbram.set_sidecar_for_file(filepath, "FUBAR", (20, 21, 22))
 
     assert dbram.get_uuid_for_file(filepath_lower) == "FUBAR"
     assert dbram.get_info_for_uuid("FUBAR") == INFO_DATA2
@@ -240,5 +252,6 @@ def test_export_db_in_memory_nofile():
     assert dbram.get_stat_exif_for_file(filepath) == (10, 11, 12)
     assert dbram.get_stat_converted_for_file(filepath) == (1, 2, 3)
     assert dbram.get_stat_edited_for_file(filepath) == (4, 5, 6)
+    assert dbram.get_sidecar_for_file(filepath) == ("FUBAR", (20, 21, 22))
 
     dbram.close()

tests/test_export_mojave_10_14_6.py

@@ -58,8 +58,8 @@ EXIF_JSON_EXPECTED = """
     "EXIF:DateTimeOriginal": "2018:10:13 09:18:12", 
     "EXIF:CreateDate": "2018:10:13 09:18:12", 
     "EXIF:OffsetTimeOriginal": "-04:00", 
-    "IPTC:DigitalCreationDate": "2018:10:13", 
     "IPTC:DateCreated": "2018:10:13", 
+    "IPTC:TimeCreated": "09:18:12-04:00", 
     "EXIF:ModifyDate": "2019:12:01 11:43:45"}]
     """
 

tests/test_photokit.py

@@ -0,0 +1,387 @@
+""" test photokit.py methods """
+
+import os
+import pathlib
+import tempfile
+
+import pytest
+
+from osxphotos.photokit import (
+    LivePhotoAsset,
+    PhotoAsset,
+    PhotoLibrary,
+    VideoAsset,
+    PHOTOS_VERSION_CURRENT,
+    PHOTOS_VERSION_ORIGINAL,
+    PHOTOS_VERSION_UNADJUSTED,
+)
+
+skip_test = "OSXPHOTOS_TEST_EXPORT" not in os.environ
+pytestmark = pytest.mark.skipif(
+    skip_test, reason="Skip if not running with author's personal library."
+)
+
+
+UUID_DICT = {
+    "plain_photo": {
+        "uuid": "A8D646C3-89A9-4D74-8001-4EB46BA55B94",
+        "filename": "IMG_8844.JPG",
+    },
+    "hdr": {"uuid": "DA87C6FF-60E8-4DCB-A21D-9C57595667F1", "filename": "IMG_6162.JPG"},
+    "selfie": {
+        "uuid": "316AEBE0-971D-4A33-833C-6BDBFF83469B",
+        "filename": "IMG_1929.JPG",
+    },
+    "video": {
+        "uuid": "5814D9DE-FAB6-473A-9C9A-5A73C6DD1AF5",
+        "filename": "IMG_9411.TRIM.MOV",
+    },
+    "hasadjustments": {
+        "uuid": "2B2D5434-6D31-49E2-BF47-B973D34A317B",
+        "filename": "IMG_2860.JPG",
+        "adjusted_size": 3012634,
+        "unadjusted_size": 2580058,
+    },
+    "slow_mo": {
+        "uuid": "DAABC6D9-1FBA-4485-AA39-0A2B100300B1",
+        "filename": "IMG_4055.MOV",
+    },
+    "live_photo": {
+        "uuid": "612CE30B-3D8F-417A-9B14-EC42CBA10ACC",
+        "filename": "IMG_3259.HEIC",
+        "filename_video": "IMG_3259.mov",
+    },
+    "burst": {
+        "uuid": "CD97EC84-71F0-40C6-BAC1-2BABEE305CAC",
+        "filename": "IMG_8196.JPG",
+        "burst_selected": 3,
+        "burst_all": 5,
+    },
+}
+
+
+def test_fetch_uuid():
+    """ test fetch_uuid """
+    uuid = UUID_DICT["plain_photo"]["uuid"]
+    filename = UUID_DICT["plain_photo"]["filename"]
+
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+    assert isinstance(photo, PhotoAsset)
+
+
+def test_plain_photo():
+    """ test plain_photo """
+    uuid = UUID_DICT["plain_photo"]["uuid"]
+    filename = UUID_DICT["plain_photo"]["filename"]
+
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+    assert photo.original_filename == filename
+    assert photo.isphoto
+    assert not photo.ismovie
+
+
+def test_hdr():
+    """ test hdr """
+    uuid = UUID_DICT["hdr"]["uuid"]
+    filename = UUID_DICT["hdr"]["filename"]
+
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+    assert photo.original_filename == filename
+    assert photo.hdr
+
+
+def test_burst():
+    """ test burst and burstid """
+    test_dict = UUID_DICT["burst"]
+    uuid = test_dict["uuid"]
+    filename = test_dict["filename"]
+
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+    assert photo.original_filename == filename
+    assert photo.burst
+    assert photo.burstid
+
+
+
+# def test_selfie():
+#     """ test selfie """
+#     uuid = UUID_DICT["selfie"]["uuid"]
+#     filename = UUID_DICT["selfie"]["filename"]
+
+#     lib = PhotoLibrary()
+#     photo = lib.fetch_uuid(uuid)
+#     assert photo.original_filename == filename
+#     assert photo.selfie
+
+
+def test_video():
+    """ test ismovie """
+    uuid = UUID_DICT["video"]["uuid"]
+    filename = UUID_DICT["video"]["filename"]
+
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+    assert isinstance(photo, VideoAsset)
+    assert photo.original_filename == filename
+    assert photo.ismovie
+    assert not photo.isphoto
+
+
+def test_slow_mo():
+    """ test slow_mo """
+    test_dict = UUID_DICT["slow_mo"]
+    uuid = test_dict["uuid"]
+    filename = test_dict["filename"]
+
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+    assert isinstance(photo, VideoAsset)
+    assert photo.original_filename == filename
+    assert photo.ismovie
+    assert photo.slow_mo
+    assert not photo.isphoto
+
+
+### PhotoAsset
+
+
+def test_export_photo_original():
+    """ test PhotoAsset.export """
+    test_dict = UUID_DICT["hasadjustments"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, version=PHOTOS_VERSION_ORIGINAL)
+        export_path = pathlib.Path(export_path[0])
+        assert export_path.is_file()
+        filename = test_dict["filename"]
+        assert export_path.stem == pathlib.Path(filename).stem
+        assert export_path.stat().st_size == test_dict["unadjusted_size"]
+
+
+def test_export_photo_unadjusted():
+    """ test PhotoAsset.export """
+    test_dict = UUID_DICT["hasadjustments"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, version=PHOTOS_VERSION_UNADJUSTED)
+        export_path = pathlib.Path(export_path[0])
+        assert export_path.is_file()
+        filename = test_dict["filename"]
+        assert export_path.stem == pathlib.Path(filename).stem
+        assert export_path.stat().st_size == test_dict["unadjusted_size"]
+
+
+def test_export_photo_current():
+    """ test PhotoAsset.export """
+    test_dict = UUID_DICT["hasadjustments"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir)
+        export_path = pathlib.Path(export_path[0])
+        assert export_path.is_file()
+        filename = test_dict["filename"]
+        assert export_path.stem == pathlib.Path(filename).stem
+        assert export_path.stat().st_size == test_dict["adjusted_size"]
+
+
+### VideoAsset
+
+
+def test_export_video_original():
+    """ test VideoAsset.export """
+    test_dict = UUID_DICT["video"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, version=PHOTOS_VERSION_ORIGINAL)
+        export_path = pathlib.Path(export_path[0])
+        assert export_path.is_file()
+        filename = test_dict["filename"]
+        assert export_path.stem == pathlib.Path(filename).stem
+
+
+def test_export_video_unadjusted():
+    """ test VideoAsset.export """
+    test_dict = UUID_DICT["video"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, version=PHOTOS_VERSION_UNADJUSTED)
+        export_path = pathlib.Path(export_path[0])
+        assert export_path.is_file()
+        filename = test_dict["filename"]
+        assert export_path.stem == pathlib.Path(filename).stem
+
+
+def test_export_video_current():
+    """ test VideoAsset.export """
+    test_dict = UUID_DICT["video"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, version=PHOTOS_VERSION_CURRENT)
+        export_path = pathlib.Path(export_path[0])
+        assert export_path.is_file()
+        filename = test_dict["filename"]
+        assert export_path.stem == pathlib.Path(filename).stem
+
+
+### Slow-Mo VideoAsset
+
+
+def test_export_slow_mo_original():
+    """ test VideoAsset.export for slow mo video"""
+    test_dict = UUID_DICT["slow_mo"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, version=PHOTOS_VERSION_ORIGINAL)
+        export_path = pathlib.Path(export_path[0])
+        assert export_path.is_file()
+        filename = test_dict["filename"]
+        assert export_path.stem == pathlib.Path(filename).stem
+
+
+def test_export_slow_mo_unadjusted():
+    """ test VideoAsset.export for slow mo video"""
+    test_dict = UUID_DICT["slow_mo"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, version=PHOTOS_VERSION_UNADJUSTED)
+        export_path = pathlib.Path(export_path[0])
+        assert export_path.is_file()
+        filename = test_dict["filename"]
+        assert export_path.stem == pathlib.Path(filename).stem
+
+
+def test_export_slow_mo_current():
+    """ test VideoAsset.export for slow mo video"""
+    test_dict = UUID_DICT["slow_mo"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, version=PHOTOS_VERSION_CURRENT)
+        export_path = pathlib.Path(export_path[0])
+        assert export_path.is_file()
+        filename = test_dict["filename"]
+        assert export_path.stem == pathlib.Path(filename).stem
+
+
+### LivePhotoAsset
+
+
+def test_export_live_original():
+    """ test LivePhotoAsset.export """
+    test_dict = UUID_DICT["live_photo"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, version=PHOTOS_VERSION_ORIGINAL)
+        for f in export_path:
+            filepath = pathlib.Path(f)
+            assert filepath.is_file()
+            filename = test_dict["filename"]
+            assert filepath.stem == pathlib.Path(filename).stem
+
+
+def test_export_live_unadjusted():
+    """ test LivePhotoAsset.export """
+    test_dict = UUID_DICT["live_photo"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, version=PHOTOS_VERSION_UNADJUSTED)
+        for file in export_path:
+            filepath = pathlib.Path(file)
+            assert filepath.is_file()
+            filename = test_dict["filename"]
+            assert filepath.stem == pathlib.Path(filename).stem
+
+
+def test_export_live_current():
+    """ test LivePhotAsset.export """
+    test_dict = UUID_DICT["live_photo"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, version=PHOTOS_VERSION_CURRENT)
+        for file in export_path:
+            filepath = pathlib.Path(file)
+            assert filepath.is_file()
+            filename = test_dict["filename"]
+            assert filepath.stem == pathlib.Path(filename).stem
+
+
+def test_export_live_current_just_photo():
+    """ test LivePhotAsset.export """
+    test_dict = UUID_DICT["live_photo"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, photo=True, video=False)
+        assert len(export_path) == 1
+        assert export_path[0].lower().endswith(".heic")
+
+
+def test_export_live_current_just_video():
+    """ test LivePhotAsset.export """
+    test_dict = UUID_DICT["live_photo"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, photo=False, video=True)
+        assert len(export_path) == 1
+        assert export_path[0].lower().endswith(".mov")
+
+
+def test_fetch_burst_uuid():
+    """ test fetch_burst_uuid """
+    test_dict = UUID_DICT["burst"]
+    uuid = test_dict["uuid"]
+    filename = test_dict["filename"]
+
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+    bursts_selected = lib.fetch_burst_uuid(photo.burstid)
+    assert len(bursts_selected) == test_dict["burst_selected"]
+    assert isinstance(bursts_selected[0], PhotoAsset)
+
+    bursts_all = lib.fetch_burst_uuid(photo.burstid, all=True)
+    assert len(bursts_all) == test_dict["burst_all"]
+    assert isinstance(bursts_all[0], PhotoAsset)

tests/test_template.py

@@ -1,14 +1,20 @@
 """ Test template.py """
 import pytest
 
+from osxphotos.exiftool import get_exiftool_path
+
+try:
+    exiftool = get_exiftool_path()
+except:
+    exiftool = None
+
 PHOTOS_DB_PLACES = (
     "./tests/Test-Places-Catalina-10_15_7.photoslibrary/database/photos.db"
 )
-PHOTOS_DB_15_1 = "./tests/Test-10.15.1.photoslibrary/database/photos.db"
-PHOTOS_DB_15_4 = "./tests/Test-10.15.4.photoslibrary/database/photos.db"
+PHOTOS_DB_15_7 = "./tests/Test-10.15.7.photoslibrary/database/photos.db"
 PHOTOS_DB_14_6 = "./tests/Test-10.14.6.photoslibrary/database/photos.db"
-
 PHOTOS_DB_COMMENTS = "tests/Test-Cloud-10.15.6.photoslibrary"
+PHOTOS_DB_CLOUD = "./tests/Test-Cloud-10.15.6.photoslibrary/database/photos.db"
 
 UUID_DICT = {
     "place_dc": "128FB4C6-0B16-4E7D-9108-FB2E90DA1546",
@@ -22,6 +28,65 @@ UUID_DICT = {
     "date_not_modified": "128FB4C6-0B16-4E7D-9108-FB2E90DA1546",
 }
 
+UUID_MEDIA_TYPE = {
+    "photo": "C2BBC7A4-5333-46EE-BAF0-093E72111B39",
+    "video": "45099D34-A414-464F-94A2-60D6823679C8",
+    "selfie": "080525C4-1F05-48E5-A3F4-0C53127BB39C",
+    "time_lapse": "4614086E-C797-4876-B3B9-3057E8D757C9",
+    "panorama": "1C1C8F1F-826B-4A24-B1CB-56628946A834",
+    "slow_mo": None,
+    "screenshot": None,
+    "portrait": "7CDA5F84-AA16-4D28-9AA6-A49E1DF8A332",
+    "live_photo": "51F2BEF7-431A-4D31-8AC1-3284A57826AE",
+    "burst": None,
+}
+
+# multi keywords
+UUID_MULTI_KEYWORDS = "6191423D-8DB8-4D4C-92BE-9BBBA308AAC4"
+TEMPLATE_VALUES_MULTI_KEYWORDS = {
+    "{keyword}": ["flowers", "wedding"],
+    "{+keyword}": ["flowerswedding"],
+    "{;+keyword}": ["flowers;wedding"],
+    "{; +keyword}": ["flowers; wedding"],
+}
+
+UUID_TITLE = "6191423D-8DB8-4D4C-92BE-9BBBA308AAC4"
+TEMPLATE_VALUES_TITLE = {
+    "{title}": ["Tulips tied together at a flower shop"],
+    "{+title}": ["Tulips tied together at a flower shop"],
+    "{,+title}": ["Tulips tied together at a flower shop"],
+    "{, +title}": ["Tulips tied together at a flower shop"],
+}
+
+# Boolean type values that render to True
+UUID_BOOL_VALUES = {"hdr": "D11D25FF-5F31-47D2-ABA9-58418878DC15"}
+
+# Boolean type values that render to False
+UUID_BOOL_VALUES_NOT = {"hdr": "51F2BEF7-431A-4D31-8AC1-3284A57826AE"}
+
+# for exiftool template
+UUID_EXIFTOOL = {
+    "A92D9C26-3A50-4197-9388-CB5F7DB9FA91": {
+        "{exiftool:EXIF:Make}": ["Canon"],
+        "{exiftool:EXIF:Model}": ["Canon PowerShot G10"],
+        "{exiftool:EXIF:Make}/{exiftool:EXIF:Model}": ["Canon/Canon PowerShot G10"],
+        "{exiftool:IPTC:Keywords,foo}": ["foo"],
+    },
+    "DC99FBDD-7A52-4100-A5BB-344131646C30": {
+        "{exiftool:IPTC:Keywords}": [
+            "England",
+            "London",
+            "London 2018",
+            "St. James's Park",
+            "UK",
+            "United Kingdom",
+        ],
+        "{,+exiftool:IPTC:Keywords}": [
+            "England,London,London 2018,St. James's Park,UK,United Kingdom"
+        ],
+    },
+}
+
 TEMPLATE_VALUES = {
     "{name}": "128FB4C6-0B16-4E7D-9108-FB2E90DA1546",
     "{original_name}": "IMG_1064",
@@ -267,7 +332,7 @@ def test_subst_default_val_2():
 
     template = "{place.name.area_of_interest,}"
     rendered, _ = photo.render_template(template)
-    assert rendered[0] == "_"
+    assert rendered[0] == ""
 
 
 def test_subst_unknown_val():
@@ -284,10 +349,6 @@ def test_subst_unknown_val():
     assert rendered[0] == "2020/{foo}"
     assert unknown == ["foo"]
 
-    template = "{place.name.area_of_interest,}"
-    rendered, _ = photo.render_template(template)
-    assert rendered[0] == "_"
-
 
 def test_subst_double_brace():
     """ Test substitution with double brace {{ which should be ignored """
@@ -322,7 +383,7 @@ def test_subst_multi_1_1_2():
     # one album, one keyword, two persons
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_1)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
     photo = photosdb.photos(uuid=[UUID_DICT["1_1_2"]])[0]
 
     template = "{created.year}/{album}/{keyword}/{person}"
@@ -336,16 +397,12 @@ def test_subst_multi_2_1_1():
     # 2 albums, 1 keyword, 1 person
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_1)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
     # one album, one keyword, two persons
     photo = photosdb.photos(uuid=[UUID_DICT["2_1_1"]])[0]
 
     template = "{created.year}/{album}/{keyword}/{person}"
-    expected = [
-        "2018/Pumpkin Farm/Kids/Katie",
-        "2018/Test Album/Kids/Katie",
-        "2018/Multi Keyword/Kids/Katie",
-    ]
+    expected = ["2018/Pumpkin Farm/Kids/Katie", "2018/Test Album/Kids/Katie"]
     rendered, _ = photo.render_template(template)
     assert sorted(rendered) == sorted(expected)
 
@@ -355,7 +412,7 @@ def test_subst_multi_2_1_1_single():
     # 2 albums, 1 keyword, 1 person but only do keywords
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_1)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
     # one album, one keyword, two persons
     photo = photosdb.photos(uuid=[UUID_DICT["2_1_1"]])[0]
 
@@ -370,7 +427,7 @@ def test_subst_multi_0_2_0():
     # 0 albums, 2 keywords, 0 persons
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_1)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
     # one album, one keyword, two persons
     photo = photosdb.photos(uuid=[UUID_DICT["0_2_0"]])[0]
 
@@ -385,7 +442,7 @@ def test_subst_multi_0_2_0_single():
     # 0 albums, 2 keywords, 0 persons, but only do albums
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_1)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
     # one album, one keyword, two persons
     photo = photosdb.photos(uuid=[UUID_DICT["0_2_0"]])[0]
 
@@ -400,7 +457,7 @@ def test_subst_multi_0_2_0_default_val():
     # 0 albums, 2 keywords, 0 persons, default vals provided
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_1)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
     # one album, one keyword, two persons
     photo = photosdb.photos(uuid=[UUID_DICT["0_2_0"]])[0]
 
@@ -415,7 +472,7 @@ def test_subst_multi_0_2_0_default_val_unknown_val():
     # 0 albums, 2 keywords, 0 persons, default vals provided, unknown val in template
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_1)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
     # one album, one keyword, two persons
     photo = photosdb.photos(uuid=[UUID_DICT["0_2_0"]])[0]
 
@@ -436,7 +493,7 @@ def test_subst_multi_0_2_0_default_val_unknown_val_2():
     # 0 albums, 2 keywords, 0 persons, default vals provided, unknown val in template
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_1)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
     # one album, one keyword, two persons
     photo = photosdb.photos(uuid=[UUID_DICT["0_2_0"]])[0]
 
@@ -454,12 +511,35 @@ def test_subst_multi_folder_albums_1():
     """ Test substitutions for folder_album are correct """
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_4)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
 
     # photo in an album in a folder
     photo = photosdb.photos(uuid=[UUID_DICT["folder_album_1"]])[0]
     template = "{folder_album}"
-    expected = ["Folder1/SubFolder2/AlbumInFolder"]
+    expected = [
+        "2018-10 - Sponsion, Museum, Frühstück, Römermuseum",
+        "2019-10/11 Paris Clermont",
+        "Folder1/SubFolder2/AlbumInFolder",
+    ]
+    rendered, unknown = photo.render_template(template)
+    assert sorted(rendered) == sorted(expected)
+    assert unknown == []
+
+
+def test_subst_multi_folder_albums_1_path_sep():
+    """ Test substitutions for folder_album are correct with custom PATH_SEP """
+    import osxphotos
+
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
+
+    # photo in an album in a folder
+    photo = photosdb.photos(uuid=[UUID_DICT["folder_album_1"]])[0]
+    template = "{folder_album(:)}"
+    expected = [
+        "2018-10 - Sponsion, Museum, Frühstück, Römermuseum",
+        "2019-10/11 Paris Clermont",
+        "Folder1:SubFolder2:AlbumInFolder",
+    ]
     rendered, unknown = photo.render_template(template)
     assert sorted(rendered) == sorted(expected)
     assert unknown == []
@@ -469,7 +549,7 @@ def test_subst_multi_folder_albums_2():
     """ Test substitutions for folder_album are correct """
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_4)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
 
     # photo in an album in a folder
     photo = photosdb.photos(uuid=[UUID_DICT["folder_album_no_folder"]])[0]
@@ -480,6 +560,21 @@ def test_subst_multi_folder_albums_2():
     assert unknown == []
 
 
+def test_subst_multi_folder_albums_2_path_sep():
+    """ Test substitutions for folder_album are correct with custom PATH_SEP """
+    import osxphotos
+
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
+
+    # photo in an album in a folder
+    photo = photosdb.photos(uuid=[UUID_DICT["folder_album_no_folder"]])[0]
+    template = "{folder_album(:)}"
+    expected = ["Pumpkin Farm", "Test Album"]
+    rendered, unknown = photo.render_template(template)
+    assert sorted(rendered) == sorted(expected)
+    assert unknown == []
+
+
 def test_subst_multi_folder_albums_3():
     """ Test substitutions for folder_album on < Photos 5 """
     import osxphotos
@@ -495,6 +590,21 @@ def test_subst_multi_folder_albums_3():
     assert unknown == []
 
 
+def test_subst_multi_folder_albums_3_path_sep():
+    """ Test substitutions for folder_album on < Photos 5 with custom PATH_SEP """
+    import osxphotos
+
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_14_6)
+
+    # photo in an album in a folder
+    photo = photosdb.photos(uuid=[UUID_DICT["mojave_album_1"]])[0]
+    template = "{folder_album(:)}"
+    expected = ["Folder1:SubFolder2:AlbumInFolder", "Pumpkin Farm", "Test Album (1)"]
+    rendered, unknown = photo.render_template(template)
+    assert sorted(rendered) == sorted(expected)
+    assert unknown == []
+
+
 def test_subst_strftime():
     """ Test that strftime substitutions are correct """
     import locale
@@ -515,7 +625,7 @@ def test_subst_expand_inplace_1():
     """ Test that substitutions are correct when expand_inplace=True """
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_1)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
     # one album, one keyword, two persons
     photo = photosdb.photos(uuid=[UUID_DICT["1_1_2"]])[0]
 
@@ -529,7 +639,7 @@ def test_subst_expand_inplace_2():
     """ Test that substitutions are correct when expand_inplace=True """
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_1)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
     # one album, one keyword, two persons
     photo = photosdb.photos(uuid=[UUID_DICT["1_1_2"]])[0]
 
@@ -543,7 +653,7 @@ def test_subst_expand_inplace_3():
     """ Test that substitutions are correct when expand_inplace=True and inplace_sep specified"""
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_1)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
     # one album, one keyword, two persons
     photo = photosdb.photos(uuid=[UUID_DICT["1_1_2"]])[0]
 
@@ -563,3 +673,109 @@ def test_comment():
         photo = photosdb.get_photo(uuid)
         comments = photo.render_template("{comment}")
         assert comments[0] == COMMENT_UUID_DICT[uuid]
+
+
+def test_media_type():
+    """ test {media_type} template """
+    import osxphotos
+
+    photosdb = osxphotos.PhotosDB(PHOTOS_DB_CLOUD)
+
+    for field, uuid in UUID_MEDIA_TYPE.items():
+        if uuid is not None:
+            photo = photosdb.get_photo(uuid)
+            rendered, _ = photo.render_template("{media_type}")
+            assert rendered[0] == osxphotos.phototemplate.MEDIA_TYPE_DEFAULTS[field]
+
+
+def test_media_type_default():
+    """ test {media_type,photo=foo} template style """
+    import osxphotos
+
+    photosdb = osxphotos.PhotosDB(PHOTOS_DB_CLOUD)
+
+    for field, uuid in UUID_MEDIA_TYPE.items():
+        if uuid is not None:
+            photo = photosdb.get_photo(uuid)
+            rendered, _ = photo.render_template("{media_type," + f"{field}" + "=foo}")
+            assert rendered[0] == "foo"
+
+
+def test_bool_values():
+    """ test {bool?TRUE,FALSE} template values """
+    import osxphotos
+
+    photosdb = osxphotos.PhotosDB(PHOTOS_DB_CLOUD)
+
+    for field, uuid in UUID_BOOL_VALUES.items():
+        if uuid is not None:
+            photo = photosdb.get_photo(uuid)
+            rendered, _ = photo.render_template("{" + f"{field}" + "?True,False}")
+            assert rendered[0] == "True"
+
+
+def test_bool_values_not():
+    """ test {bool?TRUE,FALSE} template values for FALSE values """
+    import osxphotos
+
+    photosdb = osxphotos.PhotosDB(PHOTOS_DB_CLOUD)
+
+    for field, uuid in UUID_BOOL_VALUES_NOT.items():
+        if uuid is not None:
+            photo = photosdb.get_photo(uuid)
+            rendered, _ = photo.render_template("{" + f"{field}" + "?True,False}")
+            assert rendered[0] == "False"
+
+
+def test_partial_match():
+    """ test that template successfully rejects a field that is superset of valid field """
+    import osxphotos
+
+    photosdb = osxphotos.PhotosDB(PHOTOS_DB_CLOUD)
+
+    for uuid in COMMENT_UUID_DICT:
+        photo = photosdb.get_photo(uuid)
+        rendered, notmatched = photo.render_template("{keywords}")
+        assert [rendered, notmatched] == [["{keywords}"], ["keywords"]]
+        rendered, notmatched = photo.render_template("{keywords,}")
+        assert [rendered, notmatched] == [["{keywords,}"], ["keywords"]]
+        rendered, notmatched = photo.render_template("{keywords,foo}")
+        assert [rendered, notmatched] == [["{keywords,foo}"], ["keywords"]]
+        rendered, notmatched = photo.render_template("{,+keywords,foo}")
+        assert [rendered, notmatched] == [["{,+keywords,foo}"], ["keywords"]]
+
+
+def test_expand_in_place_with_delim():
+    """ Test that substitutions are correct when {DELIM+FIELD} format used """
+    import osxphotos
+
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
+    photo = photosdb.get_photo(UUID_MULTI_KEYWORDS)
+
+    for template in TEMPLATE_VALUES_MULTI_KEYWORDS:
+        rendered, _ = photo.render_template(template)
+        assert sorted(rendered) == sorted(TEMPLATE_VALUES_MULTI_KEYWORDS[template])
+
+
+def test_expand_in_place_with_delim_single_value():
+    """ Test that single-value substitutions are correct when {DELIM+FIELD} format used """
+    import osxphotos
+
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
+    photo = photosdb.get_photo(UUID_TITLE)
+
+    for template in TEMPLATE_VALUES_TITLE:
+        rendered, _ = photo.render_template(template)
+        assert sorted(rendered) == sorted(TEMPLATE_VALUES_TITLE[template])
+
+
+@pytest.mark.skipif(exiftool is None, reason="exiftool not installed")
+def test_exiftool_template():
+    import osxphotos
+
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
+    for uuid in UUID_EXIFTOOL:
+        photo = photosdb.get_photo(uuid)
+        for template in UUID_EXIFTOOL[uuid]:
+            rendered, _ = photo.render_template(template)
+            assert sorted(rendered) == sorted(UUID_EXIFTOOL[uuid][template])