Fixed erroneous attempt to export edited with --download-missing

CHANGELOG.md

@@ -4,6 +4,34 @@ All notable changes to this project will be documented in this file. Dates are d
 
 Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 
+#### [v0.36.11](https://github.com/RhetTbull/osxphotos/compare/v0.36.10...v0.36.11)
+
+> 8 November 2020
+
+- Implemented boolean type template fields [`7fa3704`](https://github.com/RhetTbull/osxphotos/commit/7fa3704840f7800689b4ac5f8edee8210eb3e8db)
+- Bug fix in handling missing edited photos [`e829212`](https://github.com/RhetTbull/osxphotos/commit/e829212987bbc1a88f845922abcffef70c159883)
+- Fixed message in CLI [`df37a01`](https://github.com/RhetTbull/osxphotos/commit/df37a017a8efdc8d0b9bc8d00a4452dc4cb892b3)
+
+#### [v0.36.10](https://github.com/RhetTbull/osxphotos/compare/v0.36.9...v0.36.10)
+
+> 8 November 2020
+
+- Implemented issue #255 [`ae2fd2e`](https://github.com/RhetTbull/osxphotos/commit/ae2fd2e3db984756e6cc3f7b3338b8ba819ce28c)
+
+#### [v0.36.9](https://github.com/RhetTbull/osxphotos/compare/v0.36.8...v0.36.9)
+
+> 7 November 2020
+
+- Refactored regex in phototemplate [`653b7e6`](https://github.com/RhetTbull/osxphotos/commit/653b7e6600e0738ecd00f74d510a893e0d447ca4)
+- Fix for exporting slow mo videos, issue #252 [`9d38885`](https://github.com/RhetTbull/osxphotos/commit/9d38885416b528bd8c91bb09120be85a8b109f29)
+
+#### [v0.36.8](https://github.com/RhetTbull/osxphotos/compare/v0.36.7...v0.36.8)
+
+> 5 November 2020
+
+- Refactored exiftool.py [`2202f1b`](https://github.com/RhetTbull/osxphotos/commit/2202f1b1e9c4f83558ef48e58cb94af6b3a38cdd)
+- README.md update [`a509ef1`](https://github.com/RhetTbull/osxphotos/commit/a509ef18d3db2ac15a661e763a7254974cf8d84a)
+
 #### [v0.36.7](https://github.com/RhetTbull/osxphotos/compare/v0.36.6...v0.36.7)
 
 > 4 November 2020

README.md

@@ -388,16 +388,72 @@ option to re-export the entire library thus rebuilding the
 
 ** Templating System **
 
+Several options, such as --directory, allow you to specify a template  which
+will be rendered to substitute template fields with values from the photo.
+For example, '{created.month}' would be replaced with the month name of the
+photo creation date.  e.g. 'November'.
+
+The general format for a template is '{TEMPLATE_FIELD[,[DEFAULT]]}'.  Some
+templates have optional modifiers in form
+'{[[DELIM]+]TEMPLATE_FIELD[(PATH_SEP)][?VALUE_IF_TRUE][,[DEFAULT]]}'
+
+The ',' and DEFAULT value are optional.  If TEMPLATE_FIELD results in a null
+(empty) value, the default is '_'.   You may specify an alternate default
+value by appending ',DEFAULT' after template_field.  e.g. '{title,no_title}'
+would result in 'no_title' if the photo had no title.  You may include other
+text in the template string outside the {} and use more than  one template
+field, e.g. '{created.year} - {created.month}' (e.g. '2020 - November').
+
+Some template fields such as 'hdr' are boolean and resolve to True or False.
+These take the form: '{TEMPLATE_FIELD?VALUE_IF_TRUE,VALUE_IF_FALSE}', e.g.
+{hdr?is_hdr,not_hdr} which would result in 'is_hdr' if photo is an HDR  image
+and 'not_hdr' otherwise.
+
+Some template fields such as 'folder_template' are "path-like" in that they
+join  multiple elements into a single path-like string.  For example, if photo
+is in  album Album1 in folder Folder1, '{folder_album}` results in
+'Folder1/Album1'.  This is so these template fields may be used as paths in
+--directory.  If you intend to use such a field as a string, e.g. in the
+filename, you may specify  a different path separator using the form:
+'{TEMPLATE_FIELD(PATH_SEP)}'.  For example, using the example above,
+'{folder_album(-)}' would result in  'Folder1-Album1' and '{folder_album()}'
+would result in  'Folder1Album1'.
+
+Some templates may resolve to more than one value.  For example, a photo can
+have  multiple keywords so '{keyword}' can result in multiple values.  If used
+in a filename  or directory, these templates may result in more than one copy
+of the photo being exported.  For example, if photo has keywords "foo" and
+"bar", --directory '{keyword}' will result in  copies of the photo being
+exported to 'foo/image_name.jpeg' and 'bar/image_name.jpeg'.
+
+Multi-value template fields such as '{keyword}' may be expanded 'in place'
+with an optional delimiter using the template form '{DELIM+TEMPLATE_FIELD}'.
+For example, a photo with  keywords 'foo' and 'bar':
+
+'{keyword}' renders to 'foo' and 'bar'
+
+'{,+keyword}' renders to: 'foo,bar'
+
+'{; +keyword}' renders to: 'foo; bar'
+
+'{+keyword}' renders to 'foobar'
+
+Some template fields such as '{media_type}' use the 'DEFAULT' value to allow
+customization  of the output. For example, '{media_type}' resolves to the
+special media type of the  photo such as 'panorama' or 'selfie'.  You may use
+the 'DEFAULT' value to override  these in form:
+'{media_type,video=vidéo;time_lapse=vidéo_accélérée}'.  In this example, if
+photo is a time_lapse photo, 'media_type' would resolve to  'vidéo_accélérée'
+instead of 'time_lapse' and video would resolve to 'vidéo' if photo is an
+ordinary video.
+
 With the --directory and --filename options you may specify a template for the
 export directory or filename, respectively. The directory will be appended to
 the export path specified in the export DEST argument to export.  For example,
-if template is '{created.year}/{created.month}', and export desitnation DEST
+if template is '{created.year}/{created.month}', and export destination DEST
 is '/Users/maria/Pictures/export', the actual export directory for a photo
 would be '/Users/maria/Pictures/export/2020/March' if the photo was created in
-March 2020. Some template substitutions may result in more than one value, for
-example '{album}' if photo is in more than one album or '{keyword}' if photo
-has more than one keyword. In this case, more than one copy of the photo will
-be exported, each in a separate directory or with a different filename.
+March 2020.
 
 The templating system may also be used with the --keyword-template option to
 set keywords on export (with --exiftool or --sidecar), for example, to set a
@@ -426,12 +482,28 @@ value, '_' (underscore) will be used as the default value. For example, in the
 above example, this would result in '2020/_/photoname.jpg' if address was
 null.
 
+You may specify a null default (e.g. "" or empty string) by omitting the value
+after the comma, e.g. {title,} which would render to "" if title had no value.
+
 Substitution                    Description
 {name}                          Current filename of the photo
 {original_name}                 Photo's original filename when imported to
                                 Photos
 {title}                         Title of the photo
 {descr}                         Description of the photo
+{media_type}                    Special media type resolved in this
+                                precedence: selfie, time_lapse, panorama,
+                                slow_mo, screenshot, portrait, live_photo,
+                                burst, photo, video. Defaults to 'photo' or
+                                'video' if no special type. Customize one or
+                                more media types using format: '{media_type,
+                                video=vidéo;time_lapse=vidéo_accélérée}'
+{photo_or_video}                'photo' or 'video' depending on what type
+                                the image is. To customize, use default
+                                value as in
+                                '{photo_or_video,photo=fotos;video=videos}'
+{hdr}                           Photo is HDR?; True/False value, use in
+                                format '{hdr?VALUE_IF_TRUE,VALUE_IF_FALSE}'
 {created.date}                  Photo's creation date in ISO format, e.g.
                                 '2020-03-22'
 {created.year}                  4-digit year of photo creation time
@@ -1269,6 +1341,9 @@ Returns True if photo is a panorama, otherwise False.
 
 **Note**: The result of `PhotoInfo.panorama` will differ from the "Panoramas" Media Types smart album in that it will also identify panorama photos from older phones that Photos does not recognize as panoramas. 
 
+#### `slow_mo`
+Returns True if photo is a slow motion video, otherwise False
+
 #### `labels`
 Returns image categorization labels associated with the photo as list of str.
 
@@ -1391,11 +1466,13 @@ If overwrite=False and increment=False, export will fail if destination file alr
 
 #### <a name="rendertemplate">`render_template()`</a>
 
-`render_template(template_str, none_str = "_", path_sep = None, expand_inplace = False, inplace_sep = None,         filename=False, dirname=False, replacement=":",)`
-Render template string for photo.  none_str is used if template substitution results in None value and no default specified. 
-- `template_str`: str in form "{name,DEFAULT}" where name is one of the values in table below. The "," and default value that follows are optional. If specified, "DEFAULT" will be used if "name" is None.  This is useful for values which are not always present, for example reverse geolocation data.
+`render_template(template_str, none_str = "_", path_sep = None, expand_inplace = False, inplace_sep = None, filename=False, dirname=False, replacement=":",)`
+
+Render template string for photo.  none_str is used if template substitution results in None value and no default specified.
+
+- `template_str`: str in format "{[[DELIM]+]name[(PATH_SEP)][?TRUE_VALUE][,[DEFAULT]]}" where name is one of the values in the [Template Substitutions](#template-substitutions) table. See notes below regarding specific details of the syntax.
 - `none_str`: optional str to use as substitution when template value is None and no default specified in the template string.  default is "_".
-- `path_sep`: optional character to use as path separator, default is os.path.sep
+- `path_sep`: optional character to use as path separator, default is `os.path.sep`
 - `expand_inplace`: expand multi-valued substitutions in-place as a single string instead of returning individual strings
 - `inplace_sep`: optional string to use as separator between multi-valued keywords with expand_inplace; default is ','
 - `filename`: if True, template output will be sanitized to produce valid file name
@@ -1412,6 +1489,56 @@ e.g. `render_template("{created.year}/{{foo}}", photo)` would return `(["2020/{f
 
 Some substitutions, notably `album`, `keyword`, and `person` could return multiple values, hence a new string will be return for each possible substitution (hence why a list of rendered strings is returned).  For example, a photo in 2 albums: 'Vacation' and 'Family' would result in the following rendered values if template was "{created.year}/{album}" and created.year == 2020: `["2020/Vacation","2020/Family"]` 
 
+The template field format contains optional modifiers:
+
+`"{[[DELIM]+]name[(PATH_SEP)][?TRUE_VALUE][,[DEFAULT]]}"`
+
+`DELIM`: optional delimiter string to use when expanding multi-valued template values in-place
+
+`+`: If present before template `name`, expands the template in place.  If `DELIM` not provided, values are joined with no delimiter.
+
+e.g. if Photo keywords are `["foo","bar"]`:
+
+- `"{keyword}"` renders to `["foo", "bar"]`
+- `"{,+keyword}"` renders to: `["foo,bar"]`
+- `"{; +keyword}"` renders to: `["foo; bar"]`
+- `"{+keyword}"` renders to `["foobar"]`
+
+`PATH_SEP`: optional path separator to use when joining path like fields, for example `{folder_album}`.  May also be provided as `path_sep` argument in `render_template()`.  If provided both in the call to `render_template()` and in the template itself, the value in the template string takes precedence.  If not provided in either the template string or in `path_sep` argument, defaults to `os.path.sep`.
+
+e.g. If Photo is in `Album1` in `Folder1`:
+
+- `"{folder_album}"` renders to `["Folder1/Album1"]`
+- `"{folder_album(:)}"` renders to `["Folder1:Album1"]`
+- `"{folder_album()}"` renders to `["Folder1Album1"]`
+
+`?TRUE_VALUE`: optional value to use if name is boolean-type field which evaluates to true.  For example `"{hdr}"` evaluates to True if photo is an high dynamic range (HDR) image and False otherwise. In these types of fields, use `?TRUE_VALUE` to provide the value if True and `,DEFAULT` to provide the value of False.  
+
+e.g. if photo is an HDR image,
+
+- `"{hdr?ISHDR,NOTHDR}"` renders to `["ISHDR"]`
+
+and if it is not an HDR image,
+
+- `"{hdr?ISHDR,NOTHDR}"` renders to `["NOTHDR"]`
+
+Either or both `TRUE_VALUE` or `DEFAULT` (False value) may be empty which would result in empty string `[""]` when rendered.
+
+`,DEFAULT`: optional default value to use if the template name has no value.  This modifier is also used for the value if False for boolean-type fields (see above) as well as to hold a sub-template for values like `{created.strftime}`.  If no default value provided, "_" is used. May also be provided in the `none_str` argument to `render_template()`.  If provided both in the template string and in `none_str`, the value in the template string takes precedence.
+
+e.g., if photo has no title set,
+
+- `"{title}"` renders to ["_"]
+- `"{title,I have no title}"` renders to `["I have no title"]`
+
+Template fields such as `created.strftime` use the DEFAULT value to pass the template to use for `strftime`.  
+
+e.g., if photo date is 4 February 2020, 19:07:38,
+
+- `"{created.strftime,%Y-%m-%d-%H%M%S}"` renders to `["2020-02-04-190738"]`
+
+Some template fields such as `"{media_type}"` use the `DEFAULT` value to allow customization of the output. For example, `"{media_type}"` resolves to the special media type of the photo such as `panorama` or `selfie`.  You may use the `DEFAULT` value to override these in form: `"{media_type,video=vidéo;time_lapse=vidéo_accélérée}"`. In this example, if photo was a time_lapse photo, `media_type` would resolve to `vidéo_accélérée` instead of `time_lapse`. 
+
 See [Template Substitutions](#template-substitutions) for additional details.
 
 ### ExifInfo
@@ -1832,38 +1959,42 @@ To get the path of every raw photo, whether it's a single raw photo or a raw+JPE
 
 ### Template Substitutions
 
-The following substitutions are availabe for use with `PhotoInfo.render_template()` 
+The following template field substitutions are availabe for use with `PhotoInfo.render_template()`
+
 | Substitution | Description |
 |--------------|-------------|
 |{name}|Current filename of the photo|
 |{original_name}|Photo's original filename when imported to Photos|
 |{title}|Title of the photo|
 |{descr}|Description of the photo|
+|{media_type}|Special media type resolved in this precedence: selfie, time_lapse, panorama, slow_mo, screenshot, portrait, live_photo, burst, photo, video. Defaults to 'photo' or 'video' if no special type. Customize one or more media types using format: '{media_type,video=vidéo;time_lapse=vidéo_accélérée}'|
+|{photo_or_video}|'photo' or 'video' depending on what type the image is. To customize, use default value as in '{photo_or_video,photo=fotos;video=videos}'|
+|{hdr}|Photo is HDR?; True/False value, use in format '{hdr?VALUE_IF_TRUE,VALUE_IF_FALSE}'|
 |{created.date}|Photo's creation date in ISO format, e.g. '2020-03-22'|
-|{created.year}|4-digit year of file creation time|
-|{created.yy}|2-digit year of file creation time|
-|{created.mm}|2-digit month of the file creation time (zero padded)|
-|{created.month}|Month name in user's locale of the file creation time|
-|{created.mon}|Month abbreviation in the user's locale of the file creation time|
-|{created.dd}|2-digit day of the month (zero padded) of file creation time|
-|{created.dow}|Day of week in user's locale of the file creation time|
-|{created.doy}|3-digit day of year (e.g Julian day) of file creation time, starting from 1 (zero padded)|
-|{created.hour}|2-digit hour of the file creation time|
-|{created.min}|2-digit minute of the file creation time|
-|{created.sec}|2-digit second of the file creation time|
+|{created.year}|4-digit year of photo creation time|
+|{created.yy}|2-digit year of photo creation time|
+|{created.mm}|2-digit month of the photo creation time (zero padded)|
+|{created.month}|Month name in user's locale of the photo creation time|
+|{created.mon}|Month abbreviation in the user's locale of the photo creation time|
+|{created.dd}|2-digit day of the month (zero padded) of photo creation time|
+|{created.dow}|Day of week in user's locale of the photo creation time|
+|{created.doy}|3-digit day of year (e.g Julian day) of photo creation time, starting from 1 (zero padded)|
+|{created.hour}|2-digit hour of the photo creation time|
+|{created.min}|2-digit minute of the photo creation time|
+|{created.sec}|2-digit second of the photo creation time|
 |{created.strftime}|Apply strftime template to file creation date/time. Should be used in form {created.strftime,TEMPLATE} where TEMPLATE is a valid strftime template, e.g. {created.strftime,%Y-%U} would result in year-week number of year: '2020-23'. If used with no template will return null value. See https://strftime.org/ for help on strftime templates.|
 |{modified.date}|Photo's modification date in ISO format, e.g. '2020-03-22'|
-|{modified.year}|4-digit year of file modification time|
-|{modified.yy}|2-digit year of file modification time|
-|{modified.mm}|2-digit month of the file modification time (zero padded)|
-|{modified.month}|Month name in user's locale of the file modification time|
-|{modified.mon}|Month abbreviation in the user's locale of the file modification time|
-|{modified.dd}|2-digit day of the month (zero padded) of the file modification time|
+|{modified.year}|4-digit year of photo modification time|
+|{modified.yy}|2-digit year of photo modification time|
+|{modified.mm}|2-digit month of the photo modification time (zero padded)|
+|{modified.month}|Month name in user's locale of the photo modification time|
+|{modified.mon}|Month abbreviation in the user's locale of the photo modification time|
+|{modified.dd}|2-digit day of the month (zero padded) of the photo modification time|
 |{modified.dow}|Day of week in user's locale of the photo modification time|
-|{modified.doy}|3-digit day of year (e.g Julian day) of file modification time, starting from 1 (zero padded)|
-|{modified.hour}|2-digit hour of the file modification time|
-|{modified.min}|2-digit minute of the file modification time|
-|{modified.sec}|2-digit second of the file modification time|
+|{modified.doy}|3-digit day of year (e.g Julian day) of photo modification time, starting from 1 (zero padded)|
+|{modified.hour}|2-digit hour of the photo modification time|
+|{modified.min}|2-digit minute of the photo modification time|
+|{modified.sec}|2-digit second of the photo modification time|
 |{today.date}|Current date in iso format, e.g. '2020-03-22'|
 |{today.year}|4-digit year of current date|
 |{today.yy}|2-digit year of current date|

osxphotos/__main__.py

@@ -158,19 +158,76 @@ class ExportCommand(click.Command):
         formatter.write("\n\n")
         formatter.write_text("** Templating System **")
         formatter.write("\n")
+        formatter.write_text(
+            """
+Several options, such as --directory, allow you to specify a template 
+which will be rendered to substitute template fields with values from the photo. 
+For example, '{created.month}' would be replaced with the month name of the photo creation date. 
+e.g. 'November'. 
+\n
+The general format for a template is '{TEMPLATE_FIELD[,[DEFAULT]]}'. 
+Some templates have optional modifiers in form 
+'{[[DELIM]+]TEMPLATE_FIELD[(PATH_SEP)][?VALUE_IF_TRUE][,[DEFAULT]]}'
+\n
+The ',' and DEFAULT value are optional. 
+If TEMPLATE_FIELD results in a null (empty) value, the default is '_'.  
+You may specify an alternate default value by appending ',DEFAULT' after template_field. 
+e.g. '{title,no_title}' would result in 'no_title' if the photo had no title. 
+You may include other text in the template string outside the {} and use more than 
+one template field, e.g. '{created.year} - {created.month}' (e.g. '2020 - November'). 
+\n
+Some template fields such as 'hdr' are boolean and resolve to True or False. 
+These take the form: '{TEMPLATE_FIELD?VALUE_IF_TRUE,VALUE_IF_FALSE}', e.g. 
+{hdr?is_hdr,not_hdr} which would result in 'is_hdr' if photo is an HDR 
+image and 'not_hdr' otherwise.
+\n
+Some template fields such as 'folder_template' are "path-like" in that they join 
+multiple elements into a single path-like string.  For example, if photo is in 
+album Album1 in folder Folder1, '{folder_album}` results in 'Folder1/Album1'. 
+This is so these template fields may be used as paths in --directory. 
+If you intend to use such a field as a string, e.g. in the filename, you may specify 
+a different path separator using the form: '{TEMPLATE_FIELD(PATH_SEP)}'. 
+For example, using the example above, '{folder_album(-)}' would result in 
+'Folder1-Album1' and '{folder_album()}' would result in 
+'Folder1Album1'.
+\n
+Some templates may resolve to more than one value.  For example, a photo can have 
+multiple keywords so '{keyword}' can result in multiple values.  If used in a filename 
+or directory, these templates may result in more than one copy of the photo being exported. 
+For example, if photo has keywords "foo" and "bar", --directory '{keyword}' will result in 
+copies of the photo being exported to 'foo/image_name.jpeg' and 'bar/image_name.jpeg'.  
+\n
+Multi-value template fields such as '{keyword}' may be expanded 'in place' with an optional
+delimiter using the template form '{DELIM+TEMPLATE_FIELD}'.  For example, a photo with 
+keywords 'foo' and 'bar':
+\n
+'{keyword}' renders to 'foo' and 'bar'
+\n
+'{,+keyword}' renders to: 'foo,bar'
+\n
+'{; +keyword}' renders to: 'foo; bar'
+\n
+'{+keyword}' renders to 'foobar'
+\n
+Some template fields such as '{media_type}' use the 'DEFAULT' value to allow customization 
+of the output. For example, '{media_type}' resolves to the special media type of the 
+photo such as 'panorama' or 'selfie'.  You may use the 'DEFAULT' value to override 
+these in form: '{media_type,video=vidéo;time_lapse=vidéo_accélérée}'. 
+In this example, if photo is a time_lapse photo, 'media_type' would resolve to 
+'vidéo_accélérée' instead of 'time_lapse' and video would resolve to 'vidéo' if photo
+is an ordinary video. 
+"""
+        )
+        formatter.write("\n")
         formatter.write_text(
             "With the --directory and --filename options you may specify a template for the "
             + "export directory or filename, respectively. "
             + "The directory will be appended to the export path specified "
             + "in the export DEST argument to export.  For example, if template is "
-            + "'{created.year}/{created.month}', and export desitnation DEST is "
+            + "'{created.year}/{created.month}', and export destination DEST is "
             + "'/Users/maria/Pictures/export', "
             + "the actual export directory for a photo would be '/Users/maria/Pictures/export/2020/March' "
             + "if the photo was created in March 2020. "
-            + "Some template substitutions may result in more than one value, for example '{album}' if "
-            + "photo is in more than one album or '{keyword}' if photo has more than one keyword. "
-            + "In this case, more than one copy of the photo will be exported, each in a separate directory "
-            + "or with a different filename."
         )
         formatter.write("\n")
         formatter.write_text(
@@ -208,7 +265,11 @@ class ExportCommand(click.Command):
             + "has no value, '_' (underscore) will be used as the default value. For example, in the "
             + "above example, this would result in '2020/_/photoname.jpg' if address was null."
         )
-
+        formatter.write("\n")
+        formatter.write_text(
+            'You may specify a null default (e.g. "" or empty string) by omitting the value after '
+            + 'the comma, e.g. {title,} which would render to "" if title had no value.'
+        )
         formatter.write("\n")
         templ_tuples = [("Substitution", "Description")]
         templ_tuples.extend((k, v) for k, v in TEMPLATE_SUBSTITUTIONS.items())
@@ -2273,15 +2334,17 @@ def export_photo(
     global VERBOSE
     VERBOSE = bool(verbose_)
 
+    # TODO: if --skip-original-if-edited, it's possible edited version is on disk but
+    # original is missing, in which case we should download the edited version
     if not download_missing:
         if photo.ismissing:
             space = " " if not verbose_ else ""
             verbose(f"{space}Skipping missing photo {photo.original_filename}")
             return ExportResults([], [], [], [], [], [])
-        elif not os.path.exists(photo.path):
+        elif photo.path is None:
             space = " " if not verbose_ else ""
             verbose(
-                f"{space}WARNING: file {photo.path} is missing but ismissing=False, "
+                f"{space}WARNING: photo {photo.original_filename} ({photo.uuid}) is missing but ismissing=False, "
                 f"skipping {photo.original_filename}"
             )
             return ExportResults([], [], [], [], [], [])
@@ -2300,6 +2363,22 @@ def export_photo(
 
     export_original = not (skip_original_if_edited and photo.hasadjustments)
 
+    # can't export edited if photo doesn't have edited versions
+    export_edited = export_edited if photo.hasadjustments else False
+
+    # slow_mo photos will always have hasadjustments=True even if not edited
+    if photo.hasadjustments and photo.path_edited is None:
+        if photo.slow_mo:
+            export_original = True
+            export_edited = False
+        elif not download_missing:
+            # requested edited version but it's missing, download original
+            export_original = True
+            export_edited = False
+            verbose(
+                f"Edited file for {photo.original_filename} is missing, exporting original"
+            )
+
     filenames = get_filenames_from_template(photo, filename_template, original_name)
     for filename in filenames:
         verbose(f"Exporting {photo.original_filename} ({photo.filename}) as {filename}")
@@ -2321,7 +2400,7 @@ def export_photo(
             download_missing
             and (
                 photo.ismissing
-                or not os.path.exists(photo.path)
+                or photo.path is None
                 or (export_edited and photo.path_edited is None)
             )
         )

osxphotos/_version.py

@@ -1,4 +1,4 @@
 """ version info """
 
-__version__ = "0.36.8"
+__version__ = "0.36.17"
 

osxphotos/fileutil.py

@@ -1,6 +1,5 @@
 """ FileUtil class with methods for copy, hardlink, unlink, etc. """
 
-import logging
 import os
 import pathlib
 import stat
@@ -74,7 +73,6 @@ class FileUtilMacOS(FileUtilABC):
         try:
             os.link(src, dest)
         except Exception as e:
-            logging.critical(f"os.link returned error: {e}")
             raise e
 
     @classmethod
@@ -92,7 +90,7 @@ class FileUtilMacOS(FileUtilABC):
         if src is None or dest is None:
             raise ValueError("src and dest must not be None", src, dest)
 
-        if not os.path.isfile(src):
+        if not os.path.exists(src):
             raise FileNotFoundError("src file does not appear to exist", src)
 
         if norsrc:
@@ -104,9 +102,6 @@ class FileUtilMacOS(FileUtilABC):
         try:
             result = subprocess.run(command, check=True, stderr=subprocess.PIPE)
         except subprocess.CalledProcessError as e:
-            logging.critical(
-                f"ditto returned error: {e.returncode} {e.stderr.decode(sys.getfilesystemencoding()).rstrip()}"
-            )
             raise e
 
         return result.returncode

osxphotos/photoinfo/_photoinfo_export.py

@@ -1072,8 +1072,8 @@ def _exiftool_dict(
         EXIF:DateTimeOriginal
         EXIF:OffsetTimeOriginal
         EXIF:ModifyDate
-        IPTC:DigitalCreationDate
         IPTC:DateCreated
+        IPTC:TimeCreated
     """
 
     exif = {}
@@ -1187,9 +1187,12 @@ def _exiftool_dict(
     exif["EXIF:OffsetTimeOriginal"] = offsettime
 
     dateoriginal = date.strftime("%Y:%m:%d")
-    exif["IPTC:DigitalCreationDate"] = dateoriginal
     exif["IPTC:DateCreated"] = dateoriginal
 
+    timeoriginal = date.strftime(f"%H:%M:%S{offsettime}")
+    exif["IPTC:TimeCreated"] = timeoriginal
+    print(f"time = {timeoriginal}")
+
     if self.date_modified is not None and not ignore_date_modified:
         exif["EXIF:ModifyDate"] = self.date_modified.strftime("%Y:%m:%d %H:%M:%S")
     else:

osxphotos/photoinfo/photoinfo.py

@@ -164,6 +164,8 @@ class PhotoInfo:
                         photopath = os.path.join(
                             self._db._masters_path, self._info["imagePath"]
                         )
+                if not os.path.isfile(photopath):
+                    photopath = None
                 self._path = photopath
                 return photopath
 
@@ -175,6 +177,8 @@ class PhotoInfo:
                     self._info["directory"],
                     self._info["filename"],
                 )
+                if not os.path.isfile(photopath):
+                    photopath = None
                 self._path = photopath
                 return photopath
 
@@ -188,6 +192,8 @@ class PhotoInfo:
                     self._info["directory"],
                     self._info["filename"],
                 )
+            if not os.path.isfile(photopath):
+                photopath = None
             self._path = photopath
             return photopath
 

osxphotos/photosdb/photosdb.py

@@ -35,6 +35,7 @@ from .._constants import (
 from .._version import __version__
 from ..albuminfo import AlbumInfo, FolderInfo, ImportInfo
 from ..datetime_utils import datetime_has_tz, datetime_naive_to_local
+from ..fileutil import FileUtil
 from ..personinfo import PersonInfo
 from ..photoinfo import PhotoInfo
 from ..utils import (
@@ -102,7 +103,7 @@ class PhotosDB:
         # tempfile.TemporaryDirectory gets cleaned up when the object does
         self._tempdir = tempfile.TemporaryDirectory(prefix="osxphotos_")
         self._tempdir_name = self._tempdir.name
-
+        
         # set up the data structures used to store all the Photo database info
 
         # TODO: I don't think these keywords flags are actually used
@@ -265,8 +266,11 @@ class PhotosDB:
         # photoanalysisd sometimes maintains this lock even after Photos is closed
         # In those cases, make a temp copy of the file for sqlite3 to read
         if _db_is_locked(self._dbfile):
-            verbose(f"Database locked, creating temporary copy.")
-            self._tmp_db = self._copy_db_file(self._dbfile)
+            try:
+                self._tmp_db = self._link_db_file(self._dbfile)
+            except:
+                verbose(f"Database locked, creating temporary copy.")
+                self._tmp_db = self._copy_db_file(self._dbfile)
 
         self._db_version = get_db_version(self._tmp_db)
 
@@ -281,8 +285,11 @@ class PhotosDB:
                 verbose(f"Processing database {self._dbfile_actual}")
                 # if database is exclusively locked, make a copy of it and use the copy
                 if _db_is_locked(self._dbfile_actual):
-                    verbose(f"Database locked, creating temporary copy.")
-                    self._tmp_db = self._copy_db_file(self._dbfile_actual)
+                    try:
+                        self._tmp_db = self._link_db_file(self._dbfile_actual)
+                    except:
+                        verbose(f"Database locked, creating temporary copy.")
+                        self._tmp_db = self._copy_db_file(self._dbfile_actual)
 
             if _debug():
                 logging.debug(
@@ -546,6 +553,32 @@ class PhotosDB:
 
         return dest_path
 
+    def _link_db_file(self, fname):
+        """ links the sqlite database file to a temp file """
+        """ returns the name of the temp file """
+        """ If sqlite shared memory and write-ahead log files exist, those are copied too """
+        # required because python's sqlite3 implementation can't read a locked file
+        # _, suffix = os.path.splitext(fname)
+        dest_name = dest_path = ""
+        try:
+            dest_name = pathlib.Path(fname).name
+            dest_path = os.path.join(self._tempdir_name, dest_name)
+            FileUtil.hardlink(fname, dest_path)
+            # link write-ahead log and shared memory files (-wal and -shm) files if they exist
+            if os.path.exists(f"{fname}-wal"):
+                FileUtil.hardlink(f"{fname}-wal", f"{dest_path}-wal")
+            if os.path.exists(f"{fname}-shm"):
+                FileUtil.hardlink(f"{fname}-shm", f"{dest_path}-shm")
+        except:
+            print("Error linking " + fname + " to " + dest_path, file=sys.stderr)
+            raise Exception
+
+        if _debug():
+            logging.debug(dest_path)
+
+        return dest_path
+
+
     def _process_database4(self):
         """ process the Photos database to extract info
             works on Photos version <= 4.0 """

osxphotos/phototemplate.py

@@ -23,12 +23,34 @@ from .path_utils import sanitize_dirname, sanitize_filename, sanitize_pathpart
 # ensure locale set to user's locale
 locale.setlocale(locale.LC_ALL, "")
 
+PHOTO_VIDEO_TYPE_DEFAULTS = {"photo": "photo", "video": "video"}
+
+MEDIA_TYPE_DEFAULTS = {
+    "selfie": "selfie",
+    "time_lapse": "time_lapse",
+    "panorama": "panorama",
+    "slow_mo": "slow_mo",
+    "screenshot": "screenshot",
+    "portrait": "portrait",
+    "live_photo": "live_photo",
+    "burst": "burst",
+    "photo": "photo",
+    "video": "video",
+}
+
 # Permitted substitutions (each of these returns a single value or None)
 TEMPLATE_SUBSTITUTIONS = {
     "{name}": "Current filename of the photo",
     "{original_name}": "Photo's original filename when imported to Photos",
     "{title}": "Title of the photo",
     "{descr}": "Description of the photo",
+    "{media_type}": (
+        f"Special media type resolved in this precedence: {', '.join(t for t in MEDIA_TYPE_DEFAULTS)}. "
+        "Defaults to 'photo' or 'video' if no special type. "
+        "Customize one or more media types using format: '{media_type,video=vidéo;time_lapse=vidéo_accélérée}'"
+    ),
+    "{photo_or_video}": "'photo' or 'video' depending on what type the image is. To customize, use default value as in '{photo_or_video,photo=fotos;video=videos}'",
+    "{hdr}": "Photo is HDR?; True/False value, use in format '{hdr?VALUE_IF_TRUE,VALUE_IF_FALSE}'",
     "{created.date}": "Photo's creation date in ISO format, e.g. '2020-03-22'",
     "{created.year}": "4-digit year of photo creation time",
     "{created.yy}": "2-digit year of photo creation time",
@@ -144,7 +166,7 @@ class PhotoTemplate:
         Args:
             template: str template 
             none_str: str to use default for None values, default is '_' 
-            path_sep: optional character to use as path separator, default is os.path.sep
+            path_sep: optional string to use as path separator, default is os.path.sep
             expand_inplace: expand multi-valued substitutions in-place as a single string 
                 instead of returning individual strings
             inplace_sep: optional string to use as separator between multi-valued keywords
@@ -159,8 +181,6 @@ class PhotoTemplate:
 
         if path_sep is None:
             path_sep = os.path.sep
-        elif path_sep is not None and len(path_sep) != 1:
-            raise ValueError(f"path_sep must be single character: {path_sep}")
 
         if inplace_sep is None:
             inplace_sep = ","
@@ -174,9 +194,17 @@ class PhotoTemplate:
         #          there would be 6 possible renderings (2 albums x 3 persons)
 
         # regex to find {template_field,optional_default} in strings
-        # for explanation of regex see https://regex101.com/r/4JJg42/1
         # pylint: disable=anomalous-backslash-in-string
-        regex = r"(?<!\{)\{([^\\,}]+)(,{0,1}(([\w\-\%. ]+))?)(?=\}(?!\}))\}"
+        regex = (
+            r"(?<!\{)\{"  # match { but not {{
+            + r"([^}]*\+)?"  # group 1: optional DELIM+
+            + r"([^\\,}+\?]+)"  # group 2: field name
+            + r"(\([^{}\)]*\))?"  # group 3: optional (PATH_SEP)
+            + r"(\?[^\\,}]*)?"  # group 4: optional ?TRUE_VALUE for boolean fields
+            + r"(,[\w\=\;\-\%. ]*)?"  # group 5: optional ,DEFAULT
+            + r"(?=\}(?!\}))\}"  # match } but not }}
+        )
+
         if type(template) is not str:
             raise TypeError(f"template must be type str, not {type(template)}")
 
@@ -197,18 +225,33 @@ class PhotoTemplate:
             # closure to capture photo, none_str, filename, dirname in subst
             def subst(matchobj):
                 groups = len(matchobj.groups())
-                if groups == 4:
+                if groups == 5:
+                    delim = matchobj.group(1)
+                    field = matchobj.group(2)
+                    path_sep = matchobj.group(3)
+                    bool_val = matchobj.group(4)
+                    default = matchobj.group(5)
+
+                    # drop the '+' on delim
+                    delim = delim[:-1] if delim is not None else None
+                    # drop () from path_sep
+                    path_sep = path_sep.strip("()") if path_sep is not None else None
+                    # drop the ? on bool_val
+                    bool_val = bool_val[1:] if bool_val is not None else None
+                    # drop the comma on default
+                    default_val = default[1:] if default is not None else None
+
                     try:
-                        val = get_func(matchobj.group(1), matchobj.group(3))
+                        val = get_func(field, default_val, bool_val, delim, path_sep)
                     except ValueError:
                         return matchobj.group(0)
 
                     if val is None:
-                        val = (
-                            matchobj.group(3)
-                            if matchobj.group(3) is not None
-                            else none_str
-                        )
+                        # field valid but didn't match a value
+                        if default == ",":
+                            val = ""
+                        else:
+                            val = default_val if default_val is not None else none_str
 
                     return val
                 else:
@@ -249,14 +292,30 @@ class PhotoTemplate:
         rendered_strings = [rendered]
         for field in MULTI_VALUE_SUBSTITUTIONS:
             # Build a regex that matches only the field being processed
-            re_str = r"(?<!\\)\{(" + field + r")(,(([\w\-\%. ]{0,})))?\}"
+            re_str = (
+                r"(?<!\{)\{"  # match { but not {{
+                + r"([^}]*\+)?"  # group 1: optional DELIM+
+                + r"("
+                + field  # group 2: field name
+                + r")"
+                + r"(\([^{}\)]*\))?"  # group 3: optional (PATH_SEP)
+                + r"(\?[^\\,}]*)?"  # group 4: optional ?TRUE_VALUE for boolean fields
+                + r"(,[\w\=\;\-\%. ]*)?"  # group 5: optional ,DEFAULT
+                + r"(?=\}(?!\}))\}"  # match } but not }}
+            )
             regex_multi = re.compile(re_str)
 
             # holds each of the new rendered_strings, dict to avoid repeats (dict.keys())
             new_strings = {}
 
             for str_template in rendered_strings:
-                if regex_multi.search(str_template):
+                matches = regex_multi.search(str_template)
+                if matches:
+                    path_sep = (
+                        matches.group(3).strip("()")
+                        if matches.group(3) is not None
+                        else path_sep
+                    )
                     values = self.get_template_value_multi(
                         field,
                         path_sep,
@@ -264,15 +323,14 @@ class PhotoTemplate:
                         dirname=dirname,
                         replacement=replacement,
                     )
-                    if expand_inplace:
-                        # instead of returning multiple strings, join values into a single string
-                        val = (
-                            inplace_sep.join(sorted(values))
-                            if values and values[0]
-                            else None
+                    if expand_inplace or matches.group(1) is not None:
+                        delim = (
+                            matches.group(1)[:-1] if matches.group(1) is not None else inplace_sep 
                         )
+                        # instead of returning multiple strings, join values into a single string
+                        val = delim.join(sorted(values)) if values and values[0] else None
 
-                        def lookup_template_value_multi(lookup_value, _):
+                        def lookup_template_value_multi(lookup_value, *_):
                             """ Closure passed to make_subst_function get_func 
                                     Capture val and field in the closure 
                                     Allows make_subst_function to be re-used w/o modification
@@ -293,7 +351,7 @@ class PhotoTemplate:
                         # create a new template string for each value
                         for val in values:
 
-                            def lookup_template_value_multi(lookup_value, _):
+                            def lookup_template_value_multi(lookup_value, *_):
                                 """ Closure passed to make_subst_function get_func 
                                     Capture val and field in the closure 
                                     Allows make_subst_function to be re-used w/o modification
@@ -319,9 +377,9 @@ class PhotoTemplate:
         for rendered_str in rendered_strings:
             unmatched.extend(
                 [
-                    no_match[0]
+                    no_match[1]
                     for no_match in re.findall(regex, rendered_str)
-                    if no_match[0] not in unmatched
+                    if no_match[1] not in unmatched
                 ]
             )
 
@@ -339,13 +397,24 @@ class PhotoTemplate:
         return rendered_strings, unmatched
 
     def get_template_value(
-        self, field, default, filename=False, dirname=False, replacement=":"
+        self,
+        field,
+        default,
+        bool_val=None,
+        delim=None,
+        path_sep=None,
+        filename=False,
+        dirname=False,
+        replacement=":",
     ):
         """lookup value for template field (single-value template substitutions)
 
         Args:
             field: template field to find value for.
             default: the default value provided by the user
+            bool_val: True value if expression is boolean 
+            delim: delimiter for expand in place
+            path_sep: path separator for fields that are path-like
             filename: if True, template output will be sanitized to produce valid file name
             dirname: if True, template output will be sanitized to produce valid directory name 
             replacement: str, value to replace any illegal file path characters with; default = ":"
@@ -372,6 +441,12 @@ class PhotoTemplate:
             value = self.photo.title
         elif field == "descr":
             value = self.photo.description
+        elif field == "media_type":
+            value = self.get_media_type(default)
+        elif field == "photo_or_video":
+            value = self.get_photo_video_type(default)
+        elif field == "hdr":
+            value = self.get_photo_hdr(default, bool_val)
         elif field == "created.date":
             value = DateTimeFormatter(self.photo.date).date
         elif field == "created.year":
@@ -667,3 +742,66 @@ class PhotoTemplate:
         values = values or [None]
         return values
 
+    def get_photo_video_type(self, default):
+        """ return media type, e.g. photo or video """
+        default_dict = parse_default_kv(default, PHOTO_VIDEO_TYPE_DEFAULTS)
+        if self.photo.isphoto:
+            return default_dict["photo"]
+        else:
+            return default_dict["video"]
+
+    def get_media_type(self, default):
+        """ return special media type, e.g. slow_mo, panorama, etc., defaults to photo or video if no special type """
+        default_dict = parse_default_kv(default, MEDIA_TYPE_DEFAULTS)
+        p = self.photo
+        if p.selfie:
+            return default_dict["selfie"]
+        elif p.time_lapse:
+            return default_dict["time_lapse"]
+        elif p.panorama:
+            return default_dict["panorama"]
+        elif p.slow_mo:
+            return default_dict["slow_mo"]
+        elif p.screenshot:
+            return default_dict["screenshot"]
+        elif p.portrait:
+            return default_dict["portrait"]
+        elif p.live_photo:
+            return default_dict["live_photo"]
+        elif p.burst:
+            return default_dict["burst"]
+        elif p.ismovie:
+            return default_dict["video"]
+        else:
+            return default_dict["photo"]
+
+    def get_photo_hdr(self, default, bool_val):
+        if self.photo.hdr:
+            return bool_val
+        else:
+            return default
+
+
+def parse_default_kv(default, default_dict):
+    """ parse a string in form key1=value1;key2=value2,... as used for some template fields
+
+    Args:
+        default: str, in form 'photo=foto;video=vidéo'
+        default_dict: dict, in form {"photo": "fotos", "video": "vidéos"} with default values
+
+    Returns:
+        dict in form {"photo": "fotos", "video": "vidéos"}
+    """
+
+    default_dict_ = default_dict.copy()
+    if default:
+        defaults = default.split(";")
+        for kv in defaults:
+            try:
+                k, v = kv.split("=")
+                k = k.strip()
+                v = v.strip()
+                default_dict_[k] = v
+            except ValueError:
+                pass
+    return default_dict_

tests/tempdiskimage.py

@@ -0,0 +1,114 @@
+""" Create a temporary disk image on MacOS """
+
+import pathlib
+import platform
+import subprocess
+import tempfile
+import time
+
+
+class TempDiskImage:
+    """ Create and mount a temporary disk image """
+
+    def __init__(self, size=100, prefix=None):
+        """ Create and mount a temporary disk image.
+        
+        Args:
+            size: int; size in MB of disk image, default = 100
+            prefix: str; optional prefix to prepend to name of the temporary disk image
+            name: str; name of the mounted volume, default = "TemporaryDiskImage"
+
+        Raises:
+            TypeError if size is not int
+            RunTimeError if not on MacOS
+        """
+        if type(size) != int:
+            raise TypeError("size must be int")
+
+        system = platform.system()
+        if system != "Darwin":
+            raise RuntimeError("TempDiskImage only runs on MacOS")
+
+        self._tempdir = tempfile.TemporaryDirectory()
+        # hacky mktemp: this could create a race condition but unlikely given it's created in a TemporaryDirectory
+        prefix = "TemporaryDiskImage" if prefix is None else prefix
+        volume_name = f"{prefix}_{str(time.time()).replace('.','_')}_{str(time.perf_counter()).replace('.','_')}"
+        image_name = f"{volume_name}.dmg"
+        image_path = pathlib.Path(self._tempdir.name) / image_name
+        hdiutil = subprocess.run(
+            [
+                "/usr/bin/hdiutil",
+                "create",
+                "-size",
+                f"{size}m",
+                "-fs",
+                "HFS+",
+                "-volname",
+                volume_name,
+                image_path,
+            ],
+            check=True,
+            text=True,
+            capture_output=True,
+        )
+
+        if "created" not in hdiutil.stdout:
+            raise OSError(f"Could not create DMG {image_path}")
+
+        self.path = image_path
+        self._mount_point, self.name = self._mount_image(self.path)
+
+    def _mount_image(self, image_path):
+        """ mount a DMG file and return path, returns (mount_point, path) """
+        hdiutil = subprocess.run(
+            ["/usr/bin/hdiutil", "attach", image_path],
+            check=True,
+            text=True,
+            capture_output=True,
+        )
+        mount_point, path = None, None
+        for line in hdiutil.stdout.split("\n"):
+            line = line.strip()
+            if "Apple_HFS" not in line:
+                continue
+            output = line.split()
+            if len(output) < 3:
+                raise ValueError(f"Error mounting disk image {image_path}")
+            mount_point = output[0]
+            path = output[2]
+            break
+        return (mount_point, path)
+
+    def unmount(self):
+        try:
+            if self._mount_point:
+                hdiutil = subprocess.run(
+                    ["/usr/bin/hdiutil", "detach", self._mount_point],
+                    check=True,
+                    text=True,
+                    capture_output=True,
+                )
+                self._mount_point = None
+        except AttributeError:
+            pass
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        self.unmount()
+        if exc_type:
+            return False
+
+
+if __name__ == "__main__":
+    # Create a temporary disk image, 50mb in size
+    img = TempDiskImage(size=50, prefix="MyDiskImage")
+    # Be sure to unmount it, image will be cleaned up automatically
+    img.unmount()
+
+    # Or use it as a context handler
+    # Default values are 100mb and prefix = "TemporaryDiskImage"
+    with TempDiskImage() as img:
+        print(f"image: {img.path}")
+        print(f"mounted at: {img.name}")

tests/test_cli.py

@@ -2693,8 +2693,8 @@ def test_export_sidecar_keyword_template():
             "EXIF:DateTimeOriginal": "2018:09:28 16:07:07", 
             "EXIF:CreateDate": "2018:09:28 16:07:07", 
             "EXIF:OffsetTimeOriginal": "-04:00", 
-            "IPTC:DigitalCreationDate": "2018:09:28", 
-            "IPTC:DateCreated": "2018:09:28", 
+            "IPTC:DateCreated": "2018:09:28",
+            "IPTC:TimeCreated": "16:07:07-04:00",
             "EXIF:ModifyDate": "2018:09:28 16:07:07"}]
             """
         )[0]

tests/test_export_catalina_10_15_7.py

@@ -78,8 +78,8 @@ EXIF_JSON_EXPECTED = """
     "EXIF:DateTimeOriginal": "2019:04:15 14:40:24", 
     "EXIF:CreateDate": "2019:04:15 14:40:24", 
     "EXIF:OffsetTimeOriginal": "-04:00", 
-    "IPTC:DigitalCreationDate": "2019:04:15", 
     "IPTC:DateCreated": "2019:04:15", 
+    "IPTC:TimeCreated": "14:40:24-04:00", 
     "EXIF:ModifyDate": "2019:07:27 17:33:28"}]
     """
 
@@ -94,8 +94,8 @@ EXIF_JSON_EXPECTED_IGNORE_DATE_MODIFIED = """
     "EXIF:DateTimeOriginal": "2019:04:15 14:40:24", 
     "EXIF:CreateDate": "2019:04:15 14:40:24", 
     "EXIF:OffsetTimeOriginal": "-04:00", 
-    "IPTC:DigitalCreationDate": "2019:04:15", 
     "IPTC:DateCreated": "2019:04:15", 
+    "IPTC:TimeCreated": "14:40:24-04:00", 
     "EXIF:ModifyDate": "2019:04:15 14:40:24"}]
     """
 
@@ -554,8 +554,8 @@ def test_exiftool_json_sidecar_keyword_template_long(caplog):
         "EXIF:DateTimeOriginal": "2019:04:15 14:40:24", 
         "EXIF:CreateDate": "2019:04:15 14:40:24", 
         "EXIF:OffsetTimeOriginal": "-04:00", 
-        "IPTC:DigitalCreationDate": "2019:04:15", 
         "IPTC:DateCreated": "2019:04:15", 
+        "IPTC:TimeCreated": "14:40:24-04:00", 
         "EXIF:ModifyDate": "2019:07:27 17:33:28"}]
         """
     )[0]
@@ -604,8 +604,8 @@ def test_exiftool_json_sidecar_keyword_template():
         "EXIF:DateTimeOriginal": "2019:04:15 14:40:24", 
         "EXIF:CreateDate": "2019:04:15 14:40:24", 
         "EXIF:OffsetTimeOriginal": "-04:00", 
-        "IPTC:DigitalCreationDate": "2019:04:15", 
         "IPTC:DateCreated": "2019:04:15", 
+        "IPTC:TimeCreated": "14:40:24-04:00", 
         "EXIF:ModifyDate": "2019:07:27 17:33:28"}]
         """
     )[0]
@@ -666,8 +666,8 @@ def test_exiftool_json_sidecar_use_persons_keyword():
         "EXIF:DateTimeOriginal": "2018:09:28 15:35:49", 
         "EXIF:CreateDate": "2018:09:28 15:35:49", 
         "EXIF:OffsetTimeOriginal": "-04:00", 
-        "IPTC:DigitalCreationDate": "2018:09:28", 
         "IPTC:DateCreated": "2018:09:28", 
+        "IPTC:TimeCreated": "15:35:49-04:00", 
         "EXIF:ModifyDate": "2018:09:28 15:35:49"}]
         """
     )[0]
@@ -709,8 +709,8 @@ def test_exiftool_json_sidecar_use_albums_keyword():
         "EXIF:DateTimeOriginal": "2018:09:28 15:35:49", 
         "EXIF:CreateDate": "2018:09:28 15:35:49", 
         "EXIF:OffsetTimeOriginal": "-04:00", 
-        "IPTC:DigitalCreationDate": "2018:09:28", 
         "IPTC:DateCreated": "2018:09:28", 
+        "IPTC:TimeCreated": "15:35:49-04:00", 
         "EXIF:ModifyDate": "2018:09:28 15:35:49"}]
         """
     )[0]

tests/test_export_mojave_10_14_6.py

@@ -58,8 +58,8 @@ EXIF_JSON_EXPECTED = """
     "EXIF:DateTimeOriginal": "2018:10:13 09:18:12", 
     "EXIF:CreateDate": "2018:10:13 09:18:12", 
     "EXIF:OffsetTimeOriginal": "-04:00", 
-    "IPTC:DigitalCreationDate": "2018:10:13", 
     "IPTC:DateCreated": "2018:10:13", 
+    "IPTC:TimeCreated": "09:18:12-04:00", 
     "EXIF:ModifyDate": "2019:12:01 11:43:45"}]
     """
 

tests/test_link_db.py

@@ -0,0 +1,29 @@
+""" Test PhotosDB._link_db_file """
+
+import pytest
+
+from tempdiskimage import TempDiskImage
+
+PHOTOS_DB = "tests/Test-Movie-5_0.photoslibrary"
+
+def test_link_db(capsys):
+    """ Test that database doesn't get copied when opened """
+    import osxphotos
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB, verbose=print)
+    captured = capsys.readouterr()
+    assert "creating temporary copy" not in captured.out
+
+def test_copy_db(capsys):
+    """ Test that database does get copied if on different filesystem """
+    import pathlib
+    import tempfile
+    import osxphotos
+    
+    from osxphotos.fileutil import FileUtil
+    
+    with TempDiskImage(prefix="osxphotos") as tmpimg:
+        newdb = pathlib.Path(tmpimg.name) / pathlib.Path(PHOTOS_DB).name
+        FileUtil.copy(PHOTOS_DB,newdb)
+        photosdb = osxphotos.PhotosDB(dbfile=newdb, verbose=print)
+        captured = capsys.readouterr()
+        assert "creating temporary copy" in captured.out

tests/test_template.py

@@ -4,11 +4,10 @@ import pytest
 PHOTOS_DB_PLACES = (
     "./tests/Test-Places-Catalina-10_15_7.photoslibrary/database/photos.db"
 )
-PHOTOS_DB_15_1 = "./tests/Test-10.15.1.photoslibrary/database/photos.db"
-PHOTOS_DB_15_4 = "./tests/Test-10.15.4.photoslibrary/database/photos.db"
+PHOTOS_DB_15_7 = "./tests/Test-10.15.7.photoslibrary/database/photos.db"
 PHOTOS_DB_14_6 = "./tests/Test-10.14.6.photoslibrary/database/photos.db"
-
 PHOTOS_DB_COMMENTS = "tests/Test-Cloud-10.15.6.photoslibrary"
+PHOTOS_DB_CLOUD = "./tests/Test-Cloud-10.15.6.photoslibrary/database/photos.db"
 
 UUID_DICT = {
     "place_dc": "128FB4C6-0B16-4E7D-9108-FB2E90DA1546",
@@ -22,6 +21,42 @@ UUID_DICT = {
     "date_not_modified": "128FB4C6-0B16-4E7D-9108-FB2E90DA1546",
 }
 
+UUID_MEDIA_TYPE = {
+    "photo": "C2BBC7A4-5333-46EE-BAF0-093E72111B39",
+    "video": "45099D34-A414-464F-94A2-60D6823679C8",
+    "selfie": "080525C4-1F05-48E5-A3F4-0C53127BB39C",
+    "time_lapse": "4614086E-C797-4876-B3B9-3057E8D757C9",
+    "panorama": "1C1C8F1F-826B-4A24-B1CB-56628946A834",
+    "slow_mo": None,
+    "screenshot": None,
+    "portrait": "7CDA5F84-AA16-4D28-9AA6-A49E1DF8A332",
+    "live_photo": "51F2BEF7-431A-4D31-8AC1-3284A57826AE",
+    "burst": None,
+}
+
+# multi keywords
+UUID_MULTI_KEYWORDS = "6191423D-8DB8-4D4C-92BE-9BBBA308AAC4"
+TEMPLATE_VALUES_MULTI_KEYWORDS = {
+    "{keyword}": ["flowers", "wedding"],
+    "{+keyword}": ["flowerswedding"],
+    "{;+keyword}": ["flowers;wedding"],
+    "{; +keyword}": ["flowers; wedding"],
+}
+
+UUID_TITLE = "6191423D-8DB8-4D4C-92BE-9BBBA308AAC4"
+TEMPLATE_VALUES_TITLE = {
+    "{title}": ["Tulips tied together at a flower shop"],
+    "{+title}": ["Tulips tied together at a flower shop"],
+    "{,+title}": ["Tulips tied together at a flower shop"],
+    "{, +title}": ["Tulips tied together at a flower shop"],
+}
+
+# Boolean type values that render to True
+UUID_BOOL_VALUES = {"hdr": "D11D25FF-5F31-47D2-ABA9-58418878DC15"}
+
+# Boolean type values that render to False
+UUID_BOOL_VALUES_NOT = {"hdr": "51F2BEF7-431A-4D31-8AC1-3284A57826AE"}
+
 TEMPLATE_VALUES = {
     "{name}": "128FB4C6-0B16-4E7D-9108-FB2E90DA1546",
     "{original_name}": "IMG_1064",
@@ -267,7 +302,7 @@ def test_subst_default_val_2():
 
     template = "{place.name.area_of_interest,}"
     rendered, _ = photo.render_template(template)
-    assert rendered[0] == "_"
+    assert rendered[0] == ""
 
 
 def test_subst_unknown_val():
@@ -284,10 +319,6 @@ def test_subst_unknown_val():
     assert rendered[0] == "2020/{foo}"
     assert unknown == ["foo"]
 
-    template = "{place.name.area_of_interest,}"
-    rendered, _ = photo.render_template(template)
-    assert rendered[0] == "_"
-
 
 def test_subst_double_brace():
     """ Test substitution with double brace {{ which should be ignored """
@@ -322,7 +353,7 @@ def test_subst_multi_1_1_2():
     # one album, one keyword, two persons
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_1)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
     photo = photosdb.photos(uuid=[UUID_DICT["1_1_2"]])[0]
 
     template = "{created.year}/{album}/{keyword}/{person}"
@@ -336,16 +367,12 @@ def test_subst_multi_2_1_1():
     # 2 albums, 1 keyword, 1 person
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_1)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
     # one album, one keyword, two persons
     photo = photosdb.photos(uuid=[UUID_DICT["2_1_1"]])[0]
 
     template = "{created.year}/{album}/{keyword}/{person}"
-    expected = [
-        "2018/Pumpkin Farm/Kids/Katie",
-        "2018/Test Album/Kids/Katie",
-        "2018/Multi Keyword/Kids/Katie",
-    ]
+    expected = ["2018/Pumpkin Farm/Kids/Katie", "2018/Test Album/Kids/Katie"]
     rendered, _ = photo.render_template(template)
     assert sorted(rendered) == sorted(expected)
 
@@ -355,7 +382,7 @@ def test_subst_multi_2_1_1_single():
     # 2 albums, 1 keyword, 1 person but only do keywords
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_1)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
     # one album, one keyword, two persons
     photo = photosdb.photos(uuid=[UUID_DICT["2_1_1"]])[0]
 
@@ -370,7 +397,7 @@ def test_subst_multi_0_2_0():
     # 0 albums, 2 keywords, 0 persons
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_1)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
     # one album, one keyword, two persons
     photo = photosdb.photos(uuid=[UUID_DICT["0_2_0"]])[0]
 
@@ -385,7 +412,7 @@ def test_subst_multi_0_2_0_single():
     # 0 albums, 2 keywords, 0 persons, but only do albums
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_1)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
     # one album, one keyword, two persons
     photo = photosdb.photos(uuid=[UUID_DICT["0_2_0"]])[0]
 
@@ -400,7 +427,7 @@ def test_subst_multi_0_2_0_default_val():
     # 0 albums, 2 keywords, 0 persons, default vals provided
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_1)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
     # one album, one keyword, two persons
     photo = photosdb.photos(uuid=[UUID_DICT["0_2_0"]])[0]
 
@@ -415,7 +442,7 @@ def test_subst_multi_0_2_0_default_val_unknown_val():
     # 0 albums, 2 keywords, 0 persons, default vals provided, unknown val in template
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_1)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
     # one album, one keyword, two persons
     photo = photosdb.photos(uuid=[UUID_DICT["0_2_0"]])[0]
 
@@ -436,7 +463,7 @@ def test_subst_multi_0_2_0_default_val_unknown_val_2():
     # 0 albums, 2 keywords, 0 persons, default vals provided, unknown val in template
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_1)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
     # one album, one keyword, two persons
     photo = photosdb.photos(uuid=[UUID_DICT["0_2_0"]])[0]
 
@@ -454,12 +481,35 @@ def test_subst_multi_folder_albums_1():
     """ Test substitutions for folder_album are correct """
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_4)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
 
     # photo in an album in a folder
     photo = photosdb.photos(uuid=[UUID_DICT["folder_album_1"]])[0]
     template = "{folder_album}"
-    expected = ["Folder1/SubFolder2/AlbumInFolder"]
+    expected = [
+        "2018-10 - Sponsion, Museum, Frühstück, Römermuseum",
+        "2019-10/11 Paris Clermont",
+        "Folder1/SubFolder2/AlbumInFolder",
+    ]
+    rendered, unknown = photo.render_template(template)
+    assert sorted(rendered) == sorted(expected)
+    assert unknown == []
+
+
+def test_subst_multi_folder_albums_1_path_sep():
+    """ Test substitutions for folder_album are correct with custom PATH_SEP """
+    import osxphotos
+
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
+
+    # photo in an album in a folder
+    photo = photosdb.photos(uuid=[UUID_DICT["folder_album_1"]])[0]
+    template = "{folder_album(:)}"
+    expected = [
+        "2018-10 - Sponsion, Museum, Frühstück, Römermuseum",
+        "2019-10/11 Paris Clermont",
+        "Folder1:SubFolder2:AlbumInFolder",
+    ]
     rendered, unknown = photo.render_template(template)
     assert sorted(rendered) == sorted(expected)
     assert unknown == []
@@ -469,7 +519,7 @@ def test_subst_multi_folder_albums_2():
     """ Test substitutions for folder_album are correct """
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_4)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
 
     # photo in an album in a folder
     photo = photosdb.photos(uuid=[UUID_DICT["folder_album_no_folder"]])[0]
@@ -480,6 +530,21 @@ def test_subst_multi_folder_albums_2():
     assert unknown == []
 
 
+def test_subst_multi_folder_albums_2_path_sep():
+    """ Test substitutions for folder_album are correct with custom PATH_SEP """
+    import osxphotos
+
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
+
+    # photo in an album in a folder
+    photo = photosdb.photos(uuid=[UUID_DICT["folder_album_no_folder"]])[0]
+    template = "{folder_album(:)}"
+    expected = ["Pumpkin Farm", "Test Album"]
+    rendered, unknown = photo.render_template(template)
+    assert sorted(rendered) == sorted(expected)
+    assert unknown == []
+
+
 def test_subst_multi_folder_albums_3():
     """ Test substitutions for folder_album on < Photos 5 """
     import osxphotos
@@ -495,6 +560,21 @@ def test_subst_multi_folder_albums_3():
     assert unknown == []
 
 
+def test_subst_multi_folder_albums_3_path_sep():
+    """ Test substitutions for folder_album on < Photos 5 with custom PATH_SEP """
+    import osxphotos
+
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_14_6)
+
+    # photo in an album in a folder
+    photo = photosdb.photos(uuid=[UUID_DICT["mojave_album_1"]])[0]
+    template = "{folder_album(:)}"
+    expected = ["Folder1:SubFolder2:AlbumInFolder", "Pumpkin Farm", "Test Album (1)"]
+    rendered, unknown = photo.render_template(template)
+    assert sorted(rendered) == sorted(expected)
+    assert unknown == []
+
+
 def test_subst_strftime():
     """ Test that strftime substitutions are correct """
     import locale
@@ -515,7 +595,7 @@ def test_subst_expand_inplace_1():
     """ Test that substitutions are correct when expand_inplace=True """
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_1)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
     # one album, one keyword, two persons
     photo = photosdb.photos(uuid=[UUID_DICT["1_1_2"]])[0]
 
@@ -529,7 +609,7 @@ def test_subst_expand_inplace_2():
     """ Test that substitutions are correct when expand_inplace=True """
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_1)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
     # one album, one keyword, two persons
     photo = photosdb.photos(uuid=[UUID_DICT["1_1_2"]])[0]
 
@@ -543,7 +623,7 @@ def test_subst_expand_inplace_3():
     """ Test that substitutions are correct when expand_inplace=True and inplace_sep specified"""
     import osxphotos
 
-    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_1)
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
     # one album, one keyword, two persons
     photo = photosdb.photos(uuid=[UUID_DICT["1_1_2"]])[0]
 
@@ -563,3 +643,97 @@ def test_comment():
         photo = photosdb.get_photo(uuid)
         comments = photo.render_template("{comment}")
         assert comments[0] == COMMENT_UUID_DICT[uuid]
+
+
+def test_media_type():
+    """ test {media_type} template """
+    import osxphotos
+
+    photosdb = osxphotos.PhotosDB(PHOTOS_DB_CLOUD)
+
+    for field, uuid in UUID_MEDIA_TYPE.items():
+        if uuid is not None:
+            photo = photosdb.get_photo(uuid)
+            rendered, _ = photo.render_template("{media_type}")
+            assert rendered[0] == osxphotos.phototemplate.MEDIA_TYPE_DEFAULTS[field]
+
+
+def test_media_type_default():
+    """ test {media_type,photo=foo} template style """
+    import osxphotos
+
+    photosdb = osxphotos.PhotosDB(PHOTOS_DB_CLOUD)
+
+    for field, uuid in UUID_MEDIA_TYPE.items():
+        if uuid is not None:
+            photo = photosdb.get_photo(uuid)
+            rendered, _ = photo.render_template("{media_type," + f"{field}" + "=foo}")
+            assert rendered[0] == "foo"
+
+
+def test_bool_values():
+    """ test {bool?TRUE,FALSE} template values """
+    import osxphotos
+
+    photosdb = osxphotos.PhotosDB(PHOTOS_DB_CLOUD)
+
+    for field, uuid in UUID_BOOL_VALUES.items():
+        if uuid is not None:
+            photo = photosdb.get_photo(uuid)
+            rendered, _ = photo.render_template("{" + f"{field}" + "?True,False}")
+            assert rendered[0] == "True"
+
+
+def test_bool_values_not():
+    """ test {bool?TRUE,FALSE} template values for FALSE values """
+    import osxphotos
+
+    photosdb = osxphotos.PhotosDB(PHOTOS_DB_CLOUD)
+
+    for field, uuid in UUID_BOOL_VALUES_NOT.items():
+        if uuid is not None:
+            photo = photosdb.get_photo(uuid)
+            rendered, _ = photo.render_template("{" + f"{field}" + "?True,False}")
+            assert rendered[0] == "False"
+
+
+def test_partial_match():
+    """ test that template successfully rejects a field that is superset of valid field """
+    import osxphotos
+
+    photosdb = osxphotos.PhotosDB(PHOTOS_DB_CLOUD)
+
+    for uuid in COMMENT_UUID_DICT:
+        photo = photosdb.get_photo(uuid)
+        rendered, notmatched = photo.render_template("{keywords}")
+        assert [rendered, notmatched] == [["{keywords}"], ["keywords"]]
+        rendered, notmatched = photo.render_template("{keywords,}")
+        assert [rendered, notmatched] == [["{keywords,}"], ["keywords"]]
+        rendered, notmatched = photo.render_template("{keywords,foo}")
+        assert [rendered, notmatched] == [["{keywords,foo}"], ["keywords"]]
+        rendered, notmatched = photo.render_template("{,+keywords,foo}")
+        assert [rendered, notmatched] == [["{,+keywords,foo}"], ["keywords"]]
+
+
+def test_expand_in_place_with_delim():
+    """ Test that substitutions are correct when {DELIM+FIELD} format used """
+    import osxphotos
+
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
+    photo = photosdb.get_photo(UUID_MULTI_KEYWORDS)
+
+    for template in TEMPLATE_VALUES_MULTI_KEYWORDS:
+        rendered, _ = photo.render_template(template)
+        assert sorted(rendered) == sorted(TEMPLATE_VALUES_MULTI_KEYWORDS[template])
+
+
+def test_expand_in_place_with_delim_single_value():
+    """ Test that single-value substitutions are correct when {DELIM+FIELD} format used """
+    import osxphotos
+
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
+    photo = photosdb.get_photo(UUID_TITLE)
+
+    for template in TEMPLATE_VALUES_TITLE:
+        rendered, _ = photo.render_template(template)
+        assert sorted(rendered) == sorted(TEMPLATE_VALUES_TITLE[template])