Changed PhotosDB albums interface as prep for adding folders

CHANGELOG.md

@@ -4,6 +4,47 @@ All notable changes to this project will be documented in this file. Dates are d
 
 Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 
+#### [v0.25.1](https://github.com/RhetTbull/osxphotos/compare/v0.25.0...v0.25.1)
+
+> 5 April 2020
+
+- Added --no-extended-attributes option to CLI, closes #85 [`#85`](https://github.com/RhetTbull/osxphotos/issues/85)
+- Fixed CLI help for invalid topic, closes #76 [`#76`](https://github.com/RhetTbull/osxphotos/issues/76)
+- Updated test library [`bae0283`](https://github.com/RhetTbull/osxphotos/commit/bae0283441f04d71aa78dbd1cf014f376ef1f91a)
+
+#### [v0.25.0](https://github.com/RhetTbull/osxphotos/compare/v0.24.2...v0.25.0)
+
+> 4 April 2020
+
+- Added places, --place, --no-place to CLI, closes #87, #88 [`#87`](https://github.com/RhetTbull/osxphotos/issues/87)
+- Updated render_filepath_template to support multiple values [`6a89888`](https://github.com/RhetTbull/osxphotos/commit/6a898886ddadc9d5bc9dbad6ee7365270dd0a26d)
+- Added {album}, {keyword}, and {person} to template system [`507c4a3`](https://github.com/RhetTbull/osxphotos/commit/507c4a374014f999ca19789bce0df0c14332e021)
+- Added places command to CLI [`fd5e748`](https://github.com/RhetTbull/osxphotos/commit/fd5e748dca759ea1c3a7329d447f363afe8418b7)
+- Updated export example [`01cd7fe`](https://github.com/RhetTbull/osxphotos/commit/01cd7fed6d7fc0c61c171a05319c211eb0a9f7c1)
+- Updated CHANGELOG.md [`daea30f`](https://github.com/RhetTbull/osxphotos/commit/daea30f1626a208209ab6854cbd3b12f4b0a3405)
+
+#### [v0.24.2](https://github.com/RhetTbull/osxphotos/compare/v0.24.1...v0.24.2)
+
+> 28 March 2020
+
+- added {place.country_code} to template system [`be2e167`](https://github.com/RhetTbull/osxphotos/commit/be2e16769d5d2c75af6d7792f1311f5a65c3bc67)
+
+#### [v0.24.1](https://github.com/RhetTbull/osxphotos/compare/v0.23.4...v0.24.1)
+
+> 28 March 2020
+
+- Added detailed place data in PlaceInfo.names [`c06dd42`](https://github.com/RhetTbull/osxphotos/commit/c06dd4233f917f068c087f5604013d371b0a826a)
+- Template system now supports default values [`67a9a9e`](https://github.com/RhetTbull/osxphotos/commit/67a9a9e21bd05d01a3202b0a1279487f5d04c9d9)
+- Replaced template renderer with regex-based renderer [`427c4c0`](https://github.com/RhetTbull/osxphotos/commit/427c4c0bc49f671477866d30eee74834c67d7bc5)
+
+#### [v0.23.4](https://github.com/RhetTbull/osxphotos/compare/v0.23.3...v0.23.4)
+
+> 22 March 2020
+
+- Added export_by_album.py to examples [`908fead`](https://github.com/RhetTbull/osxphotos/commit/908fead8a2fbcef3b4a387f34d83d88c507c5939)
+- Updated CHANGELOG.md [`072e894`](https://github.com/RhetTbull/osxphotos/commit/072e894e56c4dfe5522d073b202933fed0204ef5)
+- Updated pathvalidate calls [`d066435`](https://github.com/RhetTbull/osxphotos/commit/d066435e3df4062be6a0a3d5fa7308f293e764d5)
+
 #### [v0.23.3](https://github.com/RhetTbull/osxphotos/compare/v0.23.1...v0.23.3)
 
 > 22 March 2020

README.md

@@ -57,7 +57,14 @@ Then you should be able to run `osxphotos` on the command line:
 Usage: osxphotos [OPTIONS] COMMAND [ARGS]...
 
 Options:
-  --db <Photos database path>  Specify database file.
+  --db <Photos database path>  Specify Photos database path. Path to Photos
+                               library/database can be specified using either
+                               --db or directly as PHOTOS_LIBRARY positional
+                               argument. If neither --db or PHOTOS_LIBRARY
+                               provided, will attempt to find the library to
+                               use in the following order: 1. last opened
+                               library, 2. system library, 3.
+                               ~/Pictures/Photos Library.photoslibrary
   --json                       Print output in JSON format.
   -v, --version                Show the version and exit.
   -h, --help                   Show this message and exit.
@@ -71,6 +78,7 @@ Commands:
   keywords  Print out keywords found in the Photos library.
   list      Print list of Photos libraries found on the system.
   persons   Print out persons (faces) found in the Photos library.
+  places    Print out places found in the Photos library.
   query     Query the Photos database using 1 or more search options; if...
 ```
 
@@ -111,11 +119,15 @@ Options:
   --no-title                      Search for photos with no title.
   --description DESC              Search for DESC in description of photo.
   --no-description                Search for photos with no description.
+  --place PLACE                   Search for PLACE in photo's reverse
+                                  geolocation info
+  --no-place                      Search for photos with no associated place
+                                  name info (no reverse geolocation info)
   --uti UTI                       Search for photos whose uniform type
                                   identifier (UTI) matches UTI
-  -i, --ignore-case               Case insensitive search for title or
-                                  description. Does not apply to keyword,
-                                  person, or album.
+  -i, --ignore-case               Case insensitive search for title,
+                                  description, or place. Does not apply to
+                                  keyword, person, or album.
   --edited                        Search for photos that have been edited.
   --external-edit                 Search for photos edited in external editor.
   --favorite                      Search for photos marked favorite.
@@ -232,13 +244,11 @@ be '/Users/maria/Pictures/export/2020/March' if the photo was created in March
 
 In the template, valid template substitutions will be replaced by the
 corresponding value from the table below.  Invalid substitutions will result
-in a warning but will be left unchanged. e.g. if you put '{foo}' in your
-template, e.g. '{created.year}/{foo}', the resulting output directory would
-look like '/Users/maria/Pictures/export/2020/{foo}'
+in a an error and the script will abort.
 
 If you want the actual text of the template substition to appear in the
-rendered name, escape the curly braces with \, for example, using
-'{created.year}/\{name\}' for --directory would result in output of
+rendered name, use double braces, e.g. '{{' or '}}', thus using
+'{created.year}/{{name}}' for --directory would result in output of
 2020/{name}/photoname.jpg
 
 You may specify an optional default value to use if the substitution does not
@@ -288,6 +298,8 @@ Substitution                    Description
                                 padded)
 {place.name}                    Place name from the photo's reverse
                                 geolocation data, as displayed in Photos
+{place.country_code}            The ISO country code from the photo's
+                                reverse geolocation data
 {place.name.country}            Country name from the photo's reverse
                                 geolocation data
 {place.name.state_province}     State or province name from the photo's
@@ -312,6 +324,18 @@ Substitution                    Description
                                 'United States'
 {place.address.country_code}    ISO country code of the postal address, e.g.
                                 'US'
+
+The following substitutions may result in multiple values. Thus if specified
+for --directory these could result in multiple copies of a photo being being
+exported, one to each directory.  For example: --directory
+'{created.year}/{album}' could result in the same photo being exported to each
+of the following directories if the photos were created in 2019 and were in
+albums 'Vacation' and 'Family': 2019/Vacation, 2019/Family
+
+Substitution  Description
+{album}       Album(s) photo is contained in
+{keyword}     Keyword(s) assigned to photo
+{person}      Person(s) / face(s) in a photo
 ```
 
 Example: export all photos to ~/Desktop/export, including edited versions and live photo movies, group in folders by date created
@@ -344,7 +368,7 @@ def main():
     photosdb = osxphotos.PhotosDB(db)
     print(photosdb.keywords)
     print(photosdb.persons)
-    print(photosdb.albums)
+    print(photosdb.album_names)
 
     print(photosdb.keywords_as_dict)
     print(photosdb.persons_as_dict)
@@ -530,17 +554,17 @@ keywords = photosdb.keywords
 
 Returns a list of the keywords found in the Photos library
 
-#### `albums`
+#### `album_names`
 ```python
 # assumes photosdb is a PhotosDB object (see above)
-albums = photosdb.albums
+albums = photosdb.album_names
 ```
 
 Returns a list of the albums found in the Photos library.  
 
 **Note**: In Photos 5.0 (MacOS 10.15/Catalina), It is possible to have more than one album with the same name in Photos.  Albums with duplicate names are treated as a single album and the photos in each are combined.  For example, if you have two albums named "Wedding" and each has 2 photos, osxphotos will treat this as a single album named "Wedding" with 4 photos in it.
 
-#### `albums_shared`
+#### `album_names_shared`
 
 Returns list of shared albums found in photos database (e.g. albums shared via iCloud photo sharing)
 
@@ -868,7 +892,7 @@ Returns True if photo is a panorama, otherwise False.
 #### `json()`
 Returns a JSON representation of all photo info 
 
-#### `export(dest, *filename, edited=False, live_photo=False, overwrite=False, increment=True, sidecar_json=False, sidecar_xmp=False, use_photos_export=False, timeout=120, exiftool=False)`
+#### `export(dest, *filename, edited=False, live_photo=False, overwrite=False, increment=True, sidecar_json=False, sidecar_xmp=False, use_photos_export=False, timeout=120, exiftool=False, no_xattr=False)`
 
 Export photo from the Photos library to another destination on disk.  
 - dest: must be valid destination path as str (or exception raised).
@@ -882,6 +906,7 @@ Export photo from the Photos library to another destination on disk.
 - use_photos_export: boolean; (default=False), if True will attempt to export photo via applescript interaction with Photos; useful for forcing download of missing photos.  This only works if the Photos library being used is the default library (last opened by Photos) as applescript will directly interact with whichever library Photos is currently using.
 - timeout: (int, default=120) timeout in seconds used with use_photos_export
 - exiftool: (boolean, default = False) if True, will use [exiftool](https://exiftool.org/) to write metadata directly to the exported photo; exiftool must be installed and in the system path
+- no_xattr: (boolean, default = False); if True, exports file without preserving extended attributes
 
 Returns: list of paths to exported files. More than one file could be exported, for example if live_photo=True, both the original imaage and the associated .mov file will be exported
 
@@ -982,9 +1007,16 @@ Render template string for photo.  none_str is used if template substitution res
 - `photo`: a [PhotoInfo](#photoinfo) object
 - `none_str`: optional str to use as substitution when template value is None and no default specified in the template string.  default is "_".
 
-Returns a tuple of (rendered, unmatched) where rendered is the rendered template string with all substitutions made and unmatched is a list of any strings that resembled a template substitution but did not match a known substitution. E.g. strings in the form "{foo}".
+Returns a tuple of (rendered, unmatched) where rendered is a list of rendered strings with all substitutions made and unmatched is a list of any strings that resembled a template substitution but did not match a known substitution. E.g. if template contained "{foo}", unmatched would be ["foo"].
+
+e.g. `render_filepath_template("{created.year}/{foo}", photo)` would return `(["2020/{foo}"],["foo"])`
+
+If you want to include "{" or "}" in the output, use "{{" or "}}"
+
+e.g. `render_filepath_template("{created.year}/{{foo}}", photo)` would return `(["2020/{foo}"],[])`
+
+Some substitutions, notably `album`, `keyword`, and `person` could return multiple values, hence a new string will be return for each possible substitution (hence why a list of rendered strings is returned).  For example, a photo in 2 albums: 'Vacation' and 'Family' would result in the following rendered values if template was "{created.year}/{album}" and created.year == 2020: `["2020/Vacation","2020/Family"]` 
 
-e.g. `render_filepath_template("{created.year}/{foo}", photo)` would return `("2020/{foo}",["{foo}"])`
 
 | Substitution | Description |
 |--------------|-------------|
@@ -1007,7 +1039,7 @@ e.g. `render_filepath_template("{created.year}/{foo}", photo)` would return `("2
 |{modified.mon}|Month abbreviation in the user's locale of the file modification time|
 |{modified.doy}|3-digit day of year (e.g Julian day) of file modification time, starting from 1 (zero padded)|
 |{place.name}|Place name from the photo's reverse geolocation data, as displayed in Photos|
-|{place.country_code}|The ISO country code from the photo's reverse geolocationo data|
+|{place.country_code}|The ISO country code from the photo's reverse geolocation data|
 |{place.name.country}|Country name from the photo's reverse geolocation data|
 |{place.name.state_province}|State or province name from the photo's reverse geolocation data|
 |{place.name.city}|City or locality name from the photo's reverse geolocation data|
@@ -1019,6 +1051,10 @@ e.g. `render_filepath_template("{created.year}/{foo}", photo)` would return `("2
 |{place.address.postal_code}|Postal code part of the postal address, e.g. '20009'|
 |{place.address.country}|Country name of the postal address, e.g. 'United States'|
 |{place.address.country_code}|ISO country code of the postal address, e.g. 'US'|
+|{album}|Album(s) photo is contained in|
+|{keyword}|Keyword(s) assigned to photo|
+|{person}|Person(s) / face(s) in a photo|
+
 
 #### `DateTimeFormatter(dt)`
 Class that provides easy access to formatted datetime values.
@@ -1077,7 +1113,7 @@ def main():
 
     print(photosdb.keywords)
     print(photosdb.persons)
-    print(photosdb.albums)
+    print(photosdb.album_names)
 
     print(photosdb.keywords_as_dict)
     print(photosdb.persons_as_dict)

examples/examples.py

@@ -14,7 +14,7 @@ def main():
 
     print(photosdb.keywords)
     print(photosdb.persons)
-    print(photosdb.albums)
+    print(photosdb.album_names)
 
     print(photosdb.keywords_as_dict)
     print(photosdb.persons_as_dict)

examples/export_by_album.py

@@ -25,7 +25,14 @@ import osxphotos
     help="Path to Photos library, default to last used library",
     default=None,
 )
-def export(export_path, default_album, library_path):
+@click.option(
+    "--edited",
+    help="Also export edited versions of photos (default is originals only)",
+    is_flag=True,
+    default=False,
+)
+def export(export_path, default_album, library_path, edited):
+    """ Export all photos, organized by album """
     export_path = os.path.expanduser(export_path)
     library_path = os.path.expanduser(library_path) if library_path else None
 
@@ -42,7 +49,7 @@ def export(export_path, default_album, library_path):
             if not albums:
                 albums = [default_album]
             for album in albums:
-                click.echo(f"exporting {p.filename} in album {album}")
+                click.echo(f"exporting {p.original_filename} in album {album}")
 
                 # make sure no invalid characters in destination path (could be in album name)
                 album_name = sanitize_filepath(album, platform="auto")
@@ -58,17 +65,21 @@ def export(export_path, default_album, library_path):
                 if not os.path.isdir(dest_dir):
                     os.makedirs(dest_dir)
 
-                # export the photo
-                if p.hasadjustments:
+                filename = p.original_filename
+                # export the photo but only if --edited, photo has adjustments, and 
+                # path_edited is not None (can be None if edited photo is missing)
+                if edited and p.hasadjustments and p.path_edited:
                     # export edited version
-                    exported = p.export(dest_dir, edited=True)
-                    edited_name = pathlib.Path(p.path_edited).name
-                    click.echo(f"Exported {edited_name} to {exported}")
+                    # use original filename with _edited appended but make sure suffix is 
+                    # same as edited file
+                    edited_filename = f"{pathlib.Path(filename).stem}_edited{pathlib.Path(p.path_edited).suffix}"
+                    exported = p.export(dest_dir, edited_filename, edited=True)
+                    click.echo(f"Exported {edited_filename} to {exported}")
                 # export unedited version
-                exported = p.export(dest_dir)
-                click.echo(f"Exported {p.filename} to {exported}")
+                exported = p.export(dest_dir, filename)
+                click.echo(f"Exported {filename} to {exported}")
         else:
-            click.echo(f"Skipping missing photo: {p.filename}")
+            click.echo(f"Skipping missing photo: {p.original_filename} in album {album}")
 
 
 if __name__ == "__main__":

osxphotos/__main__.py

@@ -18,10 +18,14 @@ from pathvalidate import (
 
 import osxphotos
 
-from ._constants import _EXIF_TOOL_URL, _PHOTOS_5_VERSION
+from ._constants import _EXIF_TOOL_URL, _PHOTOS_5_VERSION, _UNKNOWN_PLACE
 from ._version import __version__
 from .exiftool import get_exiftool_path
-from .template import render_filepath_template, TEMPLATE_SUBSTITUTIONS
+from .template import (
+    render_filepath_template,
+    TEMPLATE_SUBSTITUTIONS,
+    TEMPLATE_SUBSTITUTIONS_MULTI_VALUED,
+)
 from .utils import _copy_file, create_path_by_date
 
 
@@ -91,15 +95,13 @@ class ExportCommand(click.Command):
         formatter.write_text(
             "In the template, valid template substitutions will be replaced by "
             + "the corresponding value from the table below.  Invalid substitutions will result in a "
-            + "warning but will be left unchanged. e.g. if you put '{foo}' in your template, "
-            + "e.g. '{created.year}/{foo}', the resulting output directory would look like "
-            + "'/Users/maria/Pictures/export/2020/{foo}' "
+            + "an error and the script will abort."
         )
         formatter.write("\n")
         formatter.write_text(
             "If you want the actual text of the template substition to appear "
-            + "in the rendered name, escape the curly braces with \\, for example, "
-            + "using '{created.year}/\\{name\\}' for --directory "
+            + "in the rendered name, use double braces, e.g. '{{' or '}}', thus "
+            + "using '{created.year}/{{name}}' for --directory "
             + "would result in output of 2020/{name}/photoname.jpg"
         )
         formatter.write("\n")
@@ -126,6 +128,23 @@ class ExportCommand(click.Command):
         formatter.write("\n")
         templ_tuples = [("Substitution", "Description")]
         templ_tuples.extend((k, v) for k, v in TEMPLATE_SUBSTITUTIONS.items())
+        formatter.write_dl(templ_tuples)
+
+        formatter.write("\n")
+        formatter.write_text(
+            "The following substitutions may result in multiple values. Thus "
+            + "if specified for --directory these could result in multiple copies of a photo being "
+            + "being exported, one to each directory.  For example: "
+            + "--directory '{created.year}/{album}' could result in the same photo being exported "
+            + "to each of the following directories if the photos were created in 2019 "
+            + "and were in albums 'Vacation' and 'Family': "
+            + "2019/Vacation, 2019/Family"
+        )
+        formatter.write("\n")
+        templ_tuples = [("Substitution", "Description")]
+        templ_tuples.extend(
+            (k, v) for k, v in TEMPLATE_SUBSTITUTIONS_MULTI_VALUED.items()
+        )
 
         formatter.write_dl(templ_tuples)
         help_text += formatter.getvalue()
@@ -214,6 +233,18 @@ def query_options(f):
             is_flag=True,
             help="Search for photos with no description.",
         ),
+        o(
+            "--place",
+            metavar="PLACE",
+            default=None,
+            multiple=True,
+            help="Search for PLACE in photo's reverse geolocation info",
+        ),
+        o(
+            "--no-place",
+            is_flag=True,
+            help="Search for photos with no associated place name info (no reverse geolocation info)",
+        ),
         o(
             "--uti",
             metavar="UTI",
@@ -225,7 +256,7 @@ def query_options(f):
             "-i",
             "--ignore-case",
             is_flag=True,
-            help="Case insensitive search for title or description. Does not apply to keyword, person, or album.",
+            help="Case insensitive search for title, description, or place. Does not apply to keyword, person, or album.",
         ),
         o("--edited", is_flag=True, help="Search for photos that have been edited."),
         o(
@@ -479,6 +510,56 @@ def info(ctx, cli_obj, db, json_, photos_library):
         click.echo(yaml.dump(info, sort_keys=False))
 
 
+@cli.command()
+@DB_OPTION
+@JSON_OPTION
+@DB_ARGUMENT
+@click.pass_obj
+@click.pass_context
+def places(ctx, cli_obj, db, json_, photos_library):
+    """ Print out places found in the Photos library. """
+
+    # below needed for to make CliRunner work for testing
+    cli_db = cli_obj.db if cli_obj is not None else None
+    db = get_photos_db(*photos_library, db, cli_db)
+    if db is None:
+        click.echo(cli.commands["places"].get_help(ctx), err=True)
+        click.echo("\n\nLocated the following Photos library databases: ", err=True)
+        _list_libraries()
+        return
+
+    photosdb = osxphotos.PhotosDB(dbfile=db)
+    place_names = {}
+    for photo in photosdb.photos(movies=True):
+        if photo.place:
+            try:
+                place_names[photo.place.name] += 1
+            except:
+                place_names[photo.place.name] = 1
+        else:
+            try:
+                place_names[_UNKNOWN_PLACE] += 1
+            except:
+                place_names[_UNKNOWN_PLACE] = 1
+
+    # sort by place count
+    places = {
+        "places": {
+            name: place_names[name]
+            for name in sorted(
+                place_names.keys(), key=lambda key: place_names[key], reverse=True
+            )
+        }
+    }
+
+    # below needed for to make CliRunner work for testing
+    cli_json = cli_obj.json if cli_obj is not None else None
+    if json_ or cli_json:
+        click.echo(json.dumps(places))
+    else:
+        click.echo(yaml.dump(places, sort_keys=False))
+
+
 @cli.command()
 @DB_OPTION
 @JSON_OPTION
@@ -633,6 +714,8 @@ def query(
     not_selfie,
     panorama,
     not_panorama,
+    place,
+    no_place,
 ):
     """ Query the Photos database using 1 or more search options; 
         if more than one option is provided, they are treated as "AND" 
@@ -670,6 +753,7 @@ def query(
         (hdr, not_hdr),
         (selfie, not_selfie),
         (panorama, not_panorama),
+        (any(place), no_place),
     ]
     # print help if no non-exclusive term or a double exclusive term is given
     if not any(nonexclusive + [b ^ n for b, n in exclusive]):
@@ -740,6 +824,8 @@ def query(
         not_selfie=not_selfie,
         panorama=panorama,
         not_panorama=not_panorama,
+        place=place,
+        no_place=no_place,
     )
 
     # below needed for to make CliRunner work for testing
@@ -826,6 +912,14 @@ def query(
     help="Optional template for specifying name of output directory in the form '{name,DEFAULT}'. "
     "See below for additional details on templating system.",
 )
+@click.option(
+    "--no-extended-attributes",
+    is_flag=True,
+    default=False,
+    help="Don't copy extended attributes when exporting.  You only need this if exporting "
+    "to a filesystem that doesn't support Mac OS extended attributes.  Only use this if you get "
+    "an error while exporting.",
+)
 @DB_ARGUMENT
 @click.argument("dest", nargs=1, type=click.Path(exists=True))
 @click.pass_obj
@@ -887,6 +981,9 @@ def export(
     panorama,
     not_panorama,
     directory,
+    place,
+    no_place,
+    no_extended_attributes,
 ):
     """ Export photos from the Photos database.
         Export path DEST is required.
@@ -916,6 +1013,7 @@ def export(
         (selfie, not_selfie),
         (panorama, not_panorama),
         (export_by_date, directory),
+        (any(place), no_place),
     ]
     if any([all(bb) for bb in exclusive]):
         click.echo(cli.commands["export"].get_help(ctx), err=True)
@@ -996,6 +1094,8 @@ def export(
         not_selfie=not_selfie,
         panorama=panorama,
         not_panorama=not_panorama,
+        place=place,
+        no_place=no_place,
     )
 
     if photos:
@@ -1026,10 +1126,11 @@ def export(
                         download_missing,
                         exiftool,
                         directory,
+                        no_extended_attributes,
                     )
         else:
             for p in photos:
-                export_path = export_photo(
+                export_paths = export_photo(
                     p,
                     dest,
                     verbose,
@@ -1042,9 +1143,10 @@ def export(
                     download_missing,
                     exiftool,
                     directory,
+                    no_extended_attributes,
                 )
-                if export_path:
-                    click.echo(f"Exported {p.filename} to {export_path}")
+                if export_paths:
+                    click.echo(f"Exported {p.filename} to {export_paths}")
                 else:
                     click.echo(f"Did not export missing file {p.filename}")
     else:
@@ -1058,9 +1160,12 @@ def help(ctx, topic, **kw):
     """ Print help; for help on commands: help <command>. """
     if topic is None:
         click.echo(ctx.parent.get_help())
-    else:
+    elif topic in cli.commands:
         ctx.info_name = topic
-        click.echo(cli.commands[topic].get_help(ctx))
+        click.echo_via_pager(cli.commands[topic].get_help(ctx))
+    else:
+        click.echo(f"Invalid command: {topic}", err=True)
+        click.echo(ctx.parent.get_help())
 
 
 def print_photo_info(photos, json=False):
@@ -1208,6 +1313,8 @@ def _query(
     not_selfie=None,
     panorama=None,
     not_panorama=None,
+    place=None,
+    no_place=None,
 ):
     """ run a query against PhotosDB to extract the photos based on user supply criteria """
     """ used by query and export commands """
@@ -1242,7 +1349,7 @@ def _query(
 
     if description:
         # search description field for text
-        # if more than one, find photos with all name values in description
+        # if more than one, find photos with all description values in description
         if ignore_case:
             # case-insensitive
             for d in description:
@@ -1256,6 +1363,40 @@ def _query(
     elif no_description:
         photos = [p for p in photos if not p.description]
 
+    if place:
+        # search place.names for text matching place
+        # if more than one place, find photos with all place values in description
+        if ignore_case:
+            # case-insensitive
+            for place_name in place:
+                place_name = place_name.lower()
+                photos = [
+                    p
+                    for p in photos
+                    if p.place
+                    and any(
+                        pname
+                        for pname in p.place.names
+                        if any(
+                            pvalue for pvalue in pname if place_name in pvalue.lower()
+                        )
+                    )
+                ]
+        else:
+            for place_name in place:
+                photos = [
+                    p
+                    for p in photos
+                    if p.place
+                    and any(
+                        pname
+                        for pname in p.place.names
+                        if any(pvalue for pvalue in pname if place_name in pvalue)
+                    )
+                ]
+    elif no_place:
+        photos = [p for p in photos if not p.place]
+
     if edited:
         photos = [p for p in photos if p.hasadjustments]
 
@@ -1361,6 +1502,7 @@ def export_photo(
     download_missing,
     exiftool,
     directory,
+    no_extended_attributes,
 ):
     """ Helper function for export that does the actual export
         photo: PhotoInfo object
@@ -1375,9 +1517,14 @@ def export_photo(
         download_missing: attempt download of missing iCloud photos
         exiftool: use exiftool to write EXIF metadata directly to exported photo
         directory: template used to determine output directory
-        returns destination path of exported photo or None if photo was missing 
+        no_extended_attributes: boolean; if True, exports photo without preserving extended attributes
+        returns list of path(s) of exported photo or None if photo was missing 
     """
 
+    # Can export to multiple paths
+    # Start with single path [dest] but direcotry and export_by_date will modify dest_paths
+    dest_paths = [dest]
+
     if not download_missing:
         if photo.ismissing:
             space = " " if not verbose else ""
@@ -1407,19 +1554,25 @@ def export_photo(
 
     if export_by_date:
         date_created = photo.date.timetuple()
-        dest = create_path_by_date(dest, date_created)
+        dest_path = create_path_by_date(dest, date_created)
+        dest_paths = [dest_path]
     elif directory:
-        dirname, unmatched = render_filepath_template(directory, photo)
+        # got a directory template, render it and check results are valid
+        dirnames, unmatched = render_filepath_template(directory, photo)
         if unmatched:
-            click.echo(
-                f"Possible unmatched substitution in template: {unmatched}", err=True
+            raise click.BadOptionUsage(
+                "directory",
+                f"Invalid substitution in template '{directory}': {unmatched}",
             )
-        dirname = sanitize_filepath(dirname, platform="auto")
-        if not is_valid_filepath(dirname, platform="auto"):
-            raise ValueError(f"Invalid file path: {dirname}")
-        dest = os.path.join(dest, dirname)
-        if not os.path.isdir(dest):
-            os.makedirs(dest)
+        dest_paths = []
+        for dirname in dirnames:
+            dirname = sanitize_filepath(dirname, platform="auto")
+            if not is_valid_filepath(dirname, platform="auto"):
+                raise ValueError(f"Invalid file path: {dirname}")
+            dest_path = os.path.join(dest, dirname)
+            if not os.path.isdir(dest_path):
+                os.makedirs(dest_path)
+            dest_paths.append(dest_path)
 
     sidecar = [s.lower() for s in sidecar]
     sidecar_json = sidecar_xmp = False
@@ -1433,49 +1586,58 @@ def export_photo(
     use_photos_export = download_missing and (
         photo.ismissing or not os.path.exists(photo.path)
     )
-    photo_path = photo.export(
-        dest,
-        filename,
-        sidecar_json=sidecar_json,
-        sidecar_xmp=sidecar_xmp,
-        live_photo=export_live,
-        overwrite=overwrite,
-        use_photos_export=use_photos_export,
-        exiftool=exiftool,
-    )[0]
 
-    # if export-edited, also export the edited version
-    # verify the photo has adjustments and valid path to avoid raising an exception
-    if export_edited and photo.hasadjustments:
-        # if download_missing and the photo is missing or path doesn't exist,
-        # try to download with Photos
-        use_photos_export = download_missing and photo.path_edited is None
-        if not download_missing and photo.path_edited is None:
-            click.echo(f"Skipping missing edited photo for {filename}")
-        else:
-            edited_name = pathlib.Path(filename)
-            # check for correct edited suffix
-            if photo.path_edited is not None:
-                edited_suffix = pathlib.Path(photo.path_edited).suffix
+    # export the photo to each path in dest_paths
+    photo_paths = []
+    for dest_path in dest_paths:
+        photo_path = photo.export(
+            dest_path,
+            filename,
+            sidecar_json=sidecar_json,
+            sidecar_xmp=sidecar_xmp,
+            live_photo=export_live,
+            overwrite=overwrite,
+            use_photos_export=use_photos_export,
+            exiftool=exiftool,
+            no_xattr=no_extended_attributes,
+        )[0]
+        photo_paths.append(photo_path)
+
+        # if export-edited, also export the edited version
+        # verify the photo has adjustments and valid path to avoid raising an exception
+        if export_edited and photo.hasadjustments:
+            # if download_missing and the photo is missing or path doesn't exist,
+            # try to download with Photos
+            use_photos_export = download_missing and photo.path_edited is None
+            if not download_missing and photo.path_edited is None:
+                click.echo(f"Skipping missing edited photo for {filename}")
             else:
-                # use filename suffix which might be wrong,
-                # will be corrected by use_photos_export
-                edited_suffix = pathlib.Path(photo.filename).suffix
-            edited_name = f"{edited_name.stem}_edited{edited_suffix}"
-            if verbose:
-                click.echo(f"Exporting edited version of {filename} as {edited_name}")
-            photo.export(
-                dest,
-                edited_name,
-                sidecar_json=sidecar_json,
-                sidecar_xmp=sidecar_xmp,
-                overwrite=overwrite,
-                edited=True,
-                use_photos_export=use_photos_export,
-                exiftool=exiftool,
-            )
+                edited_name = pathlib.Path(filename)
+                # check for correct edited suffix
+                if photo.path_edited is not None:
+                    edited_suffix = pathlib.Path(photo.path_edited).suffix
+                else:
+                    # use filename suffix which might be wrong,
+                    # will be corrected by use_photos_export
+                    edited_suffix = pathlib.Path(photo.filename).suffix
+                edited_name = f"{edited_name.stem}_edited{edited_suffix}"
+                if verbose:
+                    click.echo(
+                        f"Exporting edited version of {filename} as {edited_name}"
+                    )
+                photo.export(
+                    dest_path,
+                    edited_name,
+                    sidecar_json=sidecar_json,
+                    sidecar_xmp=sidecar_xmp,
+                    overwrite=overwrite,
+                    edited=True,
+                    use_photos_export=use_photos_export,
+                    exiftool=exiftool,
+                    no_xattr=no_extended_attributes,
+                )
 
-    return photo_path
+    return photo_paths
 
 
 if __name__ == "__main__":

osxphotos/_constants.py

@@ -25,6 +25,9 @@ _TESTED_OS_VERSIONS = ["12", "13", "14", "15"]
 # Photos 5 has persons who are empty string if unidentified face
 _UNKNOWN_PERSON = "_UNKNOWN_"
 
+# photos with no reverse geolocation info (place)
+_UNKNOWN_PLACE = "_UNKNOWN_"
+
 _EXIF_TOOL_URL = "https://exiftool.org/"
 
 # Where are shared iCloud photos located?

osxphotos/_version.py

@@ -1,3 +1,3 @@
 """ version info """
 
-__version__ = "0.24.2"
+__version__ = "0.26.0"

osxphotos/photoinfo.py

@@ -522,6 +522,7 @@ class PhotoInfo:
         use_photos_export=False,
         timeout=120,
         exiftool=False,
+        no_xattr=False,
     ):
         """ export photo 
             dest: must be valid destination path (or exception raised) 
@@ -545,6 +546,7 @@ class PhotoInfo:
             use_photos_export: (boolean, default=False); if True will attempt to export photo via applescript interaction with Photos
             timeout: (int, default=120) timeout in seconds used with use_photos_export
             exiftool: (boolean, default = False); if True, will use exiftool to write metadata to export file
+            no_xattr: (boolean, default = False); if True, exports file without preserving extended attributes
             returns list of full paths to the exported files """
 
         # list of all files exported during this call to export
@@ -667,7 +669,7 @@ class PhotoInfo:
             )
 
             # copy the file, _copy_file uses ditto to preserve Mac extended attributes
-            _copy_file(src, dest)
+            _copy_file(src, dest, norsrc=no_xattr)
             exported_files.append(str(dest))
 
             # copy live photo associated .mov if requested
@@ -679,7 +681,7 @@ class PhotoInfo:
                     logging.debug(
                         f"Exporting live photo video of {filename} as {live_name.name}"
                     )
-                    _copy_file(src_live, str(live_name))
+                    _copy_file(src_live, str(live_name), norsrc=no_xattr)
                     exported_files.append(str(live_name))
                 else:
                     logging.warning(f"Skipping missing live movie for {filename}")

osxphotos/photosdb.py

@@ -120,6 +120,12 @@ class PhotosDB:
         # e.g.  {'1EB2B765-0765-43BA-A90C-0D0580E6172C': ['0C514A98-7B77-4E4F-801B-364B7B65EAFA']}
         self._dbalbums_uuid = {}
 
+        # Dict with information about all albums/photos by primary key in the album database
+        # key is album pk, value is album uuid
+        # e.g.  {'43': '0C514A98-7B77-4E4F-801B-364B7B65EAFA'}
+        # specific to Photos versions >= 5
+        self._dbalbums_pk = {}
+
         # Dict with information about all albums/photos by album
         # key is album UUID, value is list of photo UUIDs contained in that album
         # e.g. {'0C514A98-7B77-4E4F-801B-364B7B65EAFA': ['1EB2B765-0765-43BA-A90C-0D0580E6172C']}
@@ -264,6 +270,7 @@ class PhotosDB:
             k
             for k in self._dbalbums_album.keys()
             if self._dbalbum_details[k]["cloudownerhashedpersonid"] is None
+            and self._dbalbum_details[k]["intrash"] == 0
         ]
         for k in album_keys:
             title = self._dbalbum_details[k]["title"]
@@ -314,7 +321,7 @@ class PhotosDB:
         return list(persons)
 
     @property
-    def albums(self):
+    def album_names(self):
         """ return list of albums found in photos database """
 
         # Could be more than one album with same name
@@ -331,7 +338,7 @@ class PhotosDB:
         return list(albums)
 
     @property
-    def albums_shared(self):
+    def album_names_shared(self):
         """ return list of shared albums found in photos database
             only valid for Photos 5; on Photos <= 4, prints warning and returns empty list """
 
@@ -466,9 +473,9 @@ class PhotosDB:
             "uuid, "  # 0
             "name, "  # 1
             "cloudLibraryState, "  # 2
-            "cloudIdentifier "  # 3
+            "cloudIdentifier, "  # 3
+            "isInTrash "  # 4
             "FROM RKAlbum "
-            "WHERE isInTrash = 0"
         )
 
         for album in c:
@@ -476,6 +483,7 @@ class PhotosDB:
                 "title": album[1],
                 "cloudlibrarystate": album[2],
                 "cloudidentifier": album[3],
+                "intrash": album[4],
                 "cloudlocalstate": None,  # Photos 5
                 "cloudownerfirstname": None,  # Photos 5
                 "cloudownderlastname": None,  # Photos 5
@@ -846,12 +854,10 @@ class PhotosDB:
                 "FROM RKPlaceForVersion "
                 f"WHERE versionId = '{self._dbphotos[uuid]['modelID']}'"
             )
-            
+
             place_ids = [id[0] for id in place_ids_query.fetchall()]
-            self._dbphotos[uuid]["placeIDs"]  = place_ids
-            country_code = [
-                countries[x] for x in place_ids if x in countries
-            ]
+            self._dbphotos[uuid]["placeIDs"] = place_ids
+            country_code = [countries[x] for x in place_ids if x in countries]
             if len(country_code) > 1:
                 logging.warning(f"Found more than one country code for uuid: {uuid}")
 
@@ -860,7 +866,7 @@ class PhotosDB:
             else:
                 self._dbphotos[uuid]["countryCode"] = None
 
-            # get the place info that matches the RKPlace modelIDs for this photo 
+            # get the place info that matches the RKPlace modelIDs for this photo
             # (place_ids), sort by area (element 3 of the place_data tuple in places)
             place_names = [
                 pname
@@ -986,6 +992,7 @@ class PhotosDB:
             logging.debug(pformat(self._dbfaces_person))
             logging.debug(self._dbfaces_uuid)
 
+        # get details about albums
         c.execute(
             "SELECT ZGENERICALBUM.ZUUID, ZGENERICASSET.ZUUID "
             "FROM ZGENERICASSET "
@@ -995,12 +1002,15 @@ class PhotosDB:
         )
         for album in c:
             # store by uuid in _dbalbums_uuid and by album in _dbalbums_album
-            if not album[1] in self._dbalbums_uuid:
-                self._dbalbums_uuid[album[1]] = []
-            if not album[0] in self._dbalbums_album:
-                self._dbalbums_album[album[0]] = []
-            self._dbalbums_uuid[album[1]].append(album[0])
-            self._dbalbums_album[album[0]].append(album[1])
+            try:
+                self._dbalbums_uuid[album[1]].append(album[0])
+            except KeyError:
+                self._dbalbums_uuid[album[1]] = [album[0]]
+
+            try:
+                self._dbalbums_album[album[0]].append(album[1])
+            except KeyError:
+                self._dbalbums_album[album[0]] = [album[1]]
 
         # now get additional details about albums
         c.execute(
@@ -1010,8 +1020,12 @@ class PhotosDB:
             "ZCLOUDLOCALSTATE, "  # 2
             "ZCLOUDOWNERFIRSTNAME, "  # 3
             "ZCLOUDOWNERLASTNAME, "  # 4
-            "ZCLOUDOWNERHASHEDPERSONID "  # 5
-            "FROM ZGENERICALBUM"
+            "ZCLOUDOWNERHASHEDPERSONID, "  # 5
+            "ZKIND, "  # 6
+            "ZPARENTFOLDER, "  # 7
+            "Z_PK, "  # 8
+            "ZTRASHEDSTATE "  # 9
+            "FROM ZGENERICALBUM "
         )
         for album in c:
             self._dbalbum_details[album[0]] = {
@@ -1021,9 +1035,18 @@ class PhotosDB:
                 "cloudownderlastname": album[4],
                 "cloudownerhashedpersonid": album[5],
                 "cloudlibrarystate": None,  # Photos 4
-                "cloudidentifier": None,  # Photos4
+                "cloudidentifier": None,  # Photos 4
+                "kind": album[6],
+                "parentfolder": album[7],
+                "pk": album[8],
+                "intrash": album[9],
             }
 
+            # add cross-reference by pk to uuid
+            # needed to extract folder hierarchy
+            # in Photos >= 5, folders are special albums
+            self._dbalbums_pk[album[8]] = album[0]
+
         if _debug():
             logging.debug(f"Finished walking through albums")
             logging.debug(pformat(self._dbalbums_album))

osxphotos/template.py

@@ -1,10 +1,23 @@
+""" Custom template system for osxphotos """
+
+# Rolled my own template system because:
+# 1. Needed to handle multiple values (e.g. album, keyword)
+# 2. Needed to handle default values if template not found
+# 3. Didn't want user to need to know python (e.g. by using Mako which is
+#    already used elsewhere in this project)
+# 4. Couldn't figure out how to do #1 and #2 with str.format()
+#
+# This code isn't elegant but it seems to work well.  PRs gladly accepted.
+
 import datetime
 import pathlib
 import re
-from typing import Tuple  # pylint: disable=syntax-error
+from typing import Tuple, List  # pylint: disable=syntax-error
 
 from .photoinfo import PhotoInfo
+from ._constants import _UNKNOWN_PERSON
 
+# Permitted substitutions (each of these returns a single value or None)
 TEMPLATE_SUBSTITUTIONS = {
     "{name}": "Filename of the photo",
     "{original_name}": "Photo's original filename when imported to Photos",
@@ -25,7 +38,7 @@ TEMPLATE_SUBSTITUTIONS = {
     "{modified.mon}": "Month abbreviation in the user's locale of the file modification time",
     "{modified.doy}": "3-digit day of year (e.g Julian day) of file modification time, starting from 1 (zero padded)",
     "{place.name}": "Place name from the photo's reverse geolocation data, as displayed in Photos",
-    "{place.country_code}": "The ISO country code from the photo's reverse geolocationo data",
+    "{place.country_code}": "The ISO country code from the photo's reverse geolocation data",
     "{place.name.country}": "Country name from the photo's reverse geolocation data",
     "{place.name.state_province}": "State or province name from the photo's reverse geolocation data",
     "{place.name.city}": "City or locality name from the photo's reverse geolocation data",
@@ -39,9 +52,23 @@ TEMPLATE_SUBSTITUTIONS = {
     "{place.address.country_code}": "ISO country code of the postal address, e.g. 'US'",
 }
 
+# Permitted multi-value substitutions (each of these returns None or 1 or more values)
+TEMPLATE_SUBSTITUTIONS_MULTI_VALUED = {
+    "{album}": "Album(s) photo is contained in",
+    "{keyword}": "Keyword(s) assigned to photo",
+    "{person}": "Person(s) / face(s) in a photo",
+}
+
+# Just the multi-valued substitution names without the braces
+MULTI_VALUE_SUBSTITUTIONS = [
+    field.replace("{", "").replace("}", "")
+    for field in TEMPLATE_SUBSTITUTIONS_MULTI_VALUED.keys()
+]
+
 
 def get_template_value(lookup, photo):
-    """ lookup: value to find a match for
+    """ lookup template value (single-value template substitutions) for use in make_subst_function
+        lookup: value to find a match for
         photo: PhotoInfo object whose data will be used for value substitutions
         returns: either the matching template value (which may be None)
         raises: KeyError if no rule exists for lookup """
@@ -202,29 +229,24 @@ def get_template_value(lookup, photo):
     raise KeyError(f"No rule for processing {lookup}")
 
 
-def render_filepath_template(
-    template: str, photo: PhotoInfo, none_str: str = "_"
-) -> Tuple[str, list]:
-    """ render a filename or directory template """
+def render_filepath_template(template, photo, none_str="_"):
+    """ render a filename or directory template 
+        template: str template 
+        photo: PhotoInfo object
+        none_str: str to use default for None values, default is '_' """
 
+    # the rendering happens in two phases:
+    # phase 1: handle all the single-value template substitutions
+    #          results in a single string with all the template fields replaced
+    # phase 2: loop through all the multi-value template substitutions
+    #          could result in multiple strings
+    #          e.g. if template is "{album}/{person}" and there are 2 albums and 3 persons in the photo
+    #          there would be 6 possible renderings (2 albums x 3 persons)
+
+    # regex to find {template_field,optional_default} in strings
+    # for explanation of regex see https://regex101.com/r/4JJg42/1
     # pylint: disable=anomalous-backslash-in-string
-    regex = r"""(?<!\\)\{([^\\,}]+)(,{0,1}(([\w\-. ]+))?)\}"""
-
-    # pylint: disable=anomalous-backslash-in-string
-    unmatched_regex = r"(?<!\\)(\{[^\\,}]+\})"
-
-    # Explanation for regex:
-    # (?<!\\) Negative Lookbehind to skip escaped braces
-    #     assert regex following does not match "\" preceeding "{"
-    # \{ Match the opening brace
-    # 1st Capturing Group ([^\\,}]+)  Don't match "\", ",", or "}"
-    # 2nd Capturing Group (,?(([\w\-. ]+))?)
-    #     ,{0,1} optional ","
-    # 3rd Capturing Group (([\w\-. ]+))?
-    #     Matches the comma and any word characters after
-    # 4th Capturing Group ([\w\-. ]+)
-    #     Matches just the characters after the comma
-    # \} Matches the closing brace
+    regex = r"(?<!\{)\{([^\\,}]+)(,{0,1}(([\w\-. ]+))?)(?=\}(?!\}))\}"
 
     if type(template) is not str:
         raise TypeError(f"template must be type str, not {type(template)}")
@@ -232,14 +254,19 @@ def render_filepath_template(
     if type(photo) is not PhotoInfo:
         raise TypeError(f"photo must be type osxphotos.PhotoInfo, not {type(photo)}")
 
-    def make_subst_function(photo, none_str):
-        """ returns: substitution function for use in re.sub """
+    def make_subst_function(photo, none_str, get_func=get_template_value):
+        """ returns: substitution function for use in re.sub 
+            photo: a PhotoInfo object
+            none_str: value to use if substitution lookup is None and no default provided
+            get_func: function that gets the substitution value for a given template field
+                      default is get_template_value which handles the single-value fields """
+
         # closure to capture photo, none_str in subst
         def subst(matchobj):
             groups = len(matchobj.groups())
             if groups == 4:
                 try:
-                    val = get_template_value(matchobj.group(1), photo)
+                    val = get_func(matchobj.group(1), photo)
                 except KeyError:
                     return matchobj.group(0)
 
@@ -261,14 +288,91 @@ def render_filepath_template(
     # do the replacements
     rendered = re.sub(regex, subst_func, template)
 
-    # find any {words} that weren't replaced
-    unmatched = re.findall(unmatched_regex, rendered)
+    # do multi-valued placements
+    # start with the single string from phase 1 above then loop through all
+    # multi-valued fields and all values for each of those fields
+    # rendered_strings will be updated as each field is processed
+    # for example: if two albums, two keywords, and one person and template is:
+    # "{created.year}/{album}/{keyword}/{person}"
+    # rendered strings would do the following:
+    # start (created.year filled in phase 1)
+    #   ['2011/{album}/{keyword}/{person}']
+    # after processing albums:
+    #   ['2011/Album1/{keyword}/{person}',
+    #    '2011/Album2/{keyword}/{person}',]
+    # after processing keywords:
+    #   ['2011/Album1/keyword1/{person}',
+    #    '2011/Album1/keyword2/{person}',
+    #    '2011/Album2/keyword1/{person}',
+    #    '2011/Album2/keyword2/{person}',]
+    # after processing person:
+    #   ['2011/Album1/keyword1/person1',
+    #    '2011/Album1/keyword2/person1',
+    #    '2011/Album2/keyword1/person1',
+    #    '2011/Album2/keyword2/person1',]
+
+    rendered_strings = set([rendered])
+    for field in MULTI_VALUE_SUBSTITUTIONS:
+        if field == "album":
+            values = photo.albums
+        elif field == "keyword":
+            values = photo.keywords
+        elif field == "person":
+            values = photo.persons
+            # remove any _UNKNOWN_PERSON values
+            values = [val for val in values if val != _UNKNOWN_PERSON]
+        else:
+            raise ValueError(f"Unhandleded template value: {field}")
+
+        # If no values, insert None so code below will substite none_str for None
+        values = values or [None]
+
+        # Build a regex that matches only the field being processed
+        re_str = r"(?<!\\)\{(" + field + r")(,{0,1}(([\w\-. ]+))?)\}"
+        regex_multi = re.compile(re_str)
+
+        # holds each of the new rendered_strings, set() to avoid duplicates
+        new_strings = set()
+
+        for str_template in rendered_strings:
+            for val in values:
+
+                def get_template_value_multi(lookup_value, photo):
+                    """ Closure passed to make_subst_function get_func 
+                        Capture val and field in the closure 
+                        Allows make_subst_function to be re-used w/o modification """
+                    if lookup_value == field:
+                        return val
+                    else:
+                        raise KeyError(f"Unexpected value: {lookup_value}")
+
+                subst = make_subst_function(
+                    photo, none_str, get_func=get_template_value_multi
+                )
+                new_string = regex_multi.sub(subst, str_template)
+                new_strings.add(new_string)
+
+        # update rendered_strings for the next field to process
+        rendered_strings = new_strings
+
+    # find any {fields} that weren't replaced
+    unmatched = []
+    for rendered_str in rendered_strings:
+        unmatched.extend(
+            [
+                no_match[0]
+                for no_match in re.findall(regex, rendered_str)
+                if no_match[0] not in unmatched
+            ]
+        )
 
     # fix any escaped curly braces
-    rendered = re.sub(r"\\{", "{", rendered)
-    rendered = re.sub(r"\\}", "}", rendered)
+    rendered_strings = [
+        rendered_str.replace("{{", "{").replace("}}", "}")
+        for rendered_str in rendered_strings
+    ]
 
-    return rendered, unmatched
+    return rendered_strings, unmatched
 
 
 class DateTimeFormatter:

osxphotos/utils.py

@@ -114,10 +114,14 @@ def _dd_to_dms(dd):
     return int(deg_), int(min_), sec_
 
 
-def _copy_file(src, dest):
+def _copy_file(src, dest, norsrc=False):
     """ Copies a file from src path to dest path 
         src: source path as string 
         dest: destination path as string
+        norsrc: (bool) if True, uses --norsrc flag with ditto so it will not copy
+                resource fork or extended attributes.  May be useful on volumes that
+                don't work with extended attributes (likely only certain SMB mounts)
+                default is False
         Uses ditto to perform copy; will silently overwrite dest if it exists
         Raises exception if copy fails or either path is None """
 
@@ -125,19 +129,24 @@ def _copy_file(src, dest):
         raise ValueError("src and dest must not be None", src, dest)
 
     if not os.path.isfile(src):
-        raise ValueError("src file does not appear to exist", src)
+        raise FileNotFoundError("src file does not appear to exist", src)
+
+    if norsrc:
+        command = ["/usr/bin/ditto", "--norsrc", src, dest]
+    else:
+        command = ["/usr/bin/ditto", src, dest]
 
     # if error on copy, subprocess will raise CalledProcessError
     try:
-        subprocess.run(
-            ["/usr/bin/ditto", src, dest], check=True, stderr=subprocess.PIPE
-        )
+        result = subprocess.run(command, check=True, stderr=subprocess.PIPE)
     except subprocess.CalledProcessError as e:
         logging.critical(
             f"ditto returned error: {e.returncode} {e.stderr.decode(sys.getfilesystemencoding()).rstrip()}"
         )
         raise e
 
+    return result.returncode
+
 
 def dd_to_dms_str(lat, lon):
     """ convert latitude, longitude in degrees to degrees, minutes, seconds as string """

tests/Test-10.15.1.photoslibrary/database/Photos.sqlite.lock

@@ -7,7 +7,7 @@
 	<key>hostuuid</key>
 	<string>9575E48B-8D5F-5654-ABAC-4431B1167324</string>
 	<key>pid</key>
-	<integer>436</integer>
+	<integer>1526</integer>
 	<key>processname</key>
 	<string>photolibraryd</string>
 	<key>uid</key>

tests/Test-10.15.1.photoslibrary/private/com.apple.photoanalysisd/caches/graph/PhotoAnalysisServicePreferences.plist

@@ -3,24 +3,24 @@
 <plist version="1.0">
 <dict>
 	<key>BackgroundHighlightCollection</key>
-	<date>2020-03-28T16:32:26Z</date>
+	<date>2020-04-04T17:34:35Z</date>
 	<key>BackgroundHighlightEnrichment</key>
-	<date>2020-03-28T16:32:25Z</date>
+	<date>2020-04-04T17:34:34Z</date>
 	<key>BackgroundJobAssetRevGeocode</key>
-	<date>2020-03-28T16:32:26Z</date>
+	<date>2020-04-04T20:16:33Z</date>
 	<key>BackgroundJobSearch</key>
-	<date>2020-03-28T16:32:26Z</date>
+	<date>2020-04-04T17:34:35Z</date>
 	<key>BackgroundPeopleSuggestion</key>
-	<date>2020-03-28T16:32:25Z</date>
+	<date>2020-04-04T17:34:34Z</date>
 	<key>BackgroundUserBehaviorProcessor</key>
-	<date>2020-03-28T07:30:06Z</date>
+	<date>2020-04-04T13:56:56Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundGraphConsistencyUpdateJobDateKey</key>
-	<date>2020-03-28T16:35:58Z</date>
+	<date>2020-04-04T20:16:40Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundGraphRebuildJobDate</key>
-	<date>2020-03-28T07:30:06Z</date>
+	<date>2020-04-04T13:56:49Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundMemoryGenerationJobDate</key>
-	<date>2020-03-28T16:32:26Z</date>
+	<date>2020-04-04T20:16:33Z</date>
 	<key>SiriPortraitDonation</key>
-	<date>2020-03-28T07:30:06Z</date>
+	<date>2020-04-04T13:56:56Z</date>
 </dict>
 </plist>

tests/Test-10.15.1.photoslibrary/private/com.apple.photoanalysisd/caches/vision/PhotoAnalysisServicePreferences.plist

@@ -3,8 +3,8 @@
 <plist version="1.0">
 <dict>
 	<key>FaceIDModelLastGenerationKey</key>
-	<date>2020-03-28T07:30:06Z</date>
+	<date>2020-04-04T13:56:59Z</date>
 	<key>LastContactClassificationKey</key>
-	<date>2020-03-28T07:30:08Z</date>
+	<date>2020-04-04T13:57:02Z</date>
 </dict>
 </plist>

tests/Test-10.15.4.photoslibrary/database/DataModelVersion.plist

@@ -0,0 +1,10 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>LibrarySchemaVersion</key>
+	<integer>5001</integer>
+	<key>MetaSchemaVersion</key>
+	<integer>3</integer>
+</dict>
+</plist>

tests/Test-10.15.4.photoslibrary/database/Photos.sqlite.lock

@@ -0,0 +1,16 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>hostname</key>
+	<string>Rhets-MacBook-Pro.local</string>
+	<key>hostuuid</key>
+	<string>9575E48B-8D5F-5654-ABAC-4431B1167324</string>
+	<key>pid</key>
+	<integer>1440</integer>
+	<key>processname</key>
+	<string>photolibraryd</string>
+	<key>uid</key>
+	<integer>501</integer>
+</dict>
+</plist>

tests/Test-10.15.4.photoslibrary/database/search/searchMetadata.plist

@@ -0,0 +1,188 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>BlacklistedMeaningsByMeaning</key>
+	<dict/>
+	<key>MePersonUUID</key>
+	<string>39488755-78C0-40B2-B378-EDA280E1823C</string>
+	<key>SceneWhitelist</key>
+	<array>
+		<string>Graduation</string>
+		<string>Aquarium</string>
+		<string>Food</string>
+		<string>Ice Skating</string>
+		<string>Mountain</string>
+		<string>Cliff</string>
+		<string>Basketball</string>
+		<string>Tennis</string>
+		<string>Jewelry</string>
+		<string>Cheese</string>
+		<string>Softball</string>
+		<string>Football</string>
+		<string>Circus</string>
+		<string>Jet Ski</string>
+		<string>Playground</string>
+		<string>Carousel</string>
+		<string>Paint Ball</string>
+		<string>Windsurfing</string>
+		<string>Sailboat</string>
+		<string>Sunbathing</string>
+		<string>Dam</string>
+		<string>Fireplace</string>
+		<string>Flower</string>
+		<string>Scuba</string>
+		<string>Hiking</string>
+		<string>Cetacean</string>
+		<string>Pier</string>
+		<string>Bowling</string>
+		<string>Snowboarding</string>
+		<string>Zoo</string>
+		<string>Snowmobile</string>
+		<string>Theater</string>
+		<string>Boat</string>
+		<string>Casino</string>
+		<string>Car</string>
+		<string>Diving</string>
+		<string>Cycling</string>
+		<string>Musical Instrument</string>
+		<string>Board Game</string>
+		<string>Castle</string>
+		<string>Sunset Sunrise</string>
+		<string>Martial Arts</string>
+		<string>Motocross</string>
+		<string>Submarine</string>
+		<string>Cat</string>
+		<string>Snow</string>
+		<string>Kiteboarding</string>
+		<string>Squash</string>
+		<string>Geyser</string>
+		<string>Music</string>
+		<string>Archery</string>
+		<string>Desert</string>
+		<string>Blackjack</string>
+		<string>Fireworks</string>
+		<string>Sportscar</string>
+		<string>Feline</string>
+		<string>Soccer</string>
+		<string>Museum</string>
+		<string>Baby</string>
+		<string>Fencing</string>
+		<string>Railroad</string>
+		<string>Nascar</string>
+		<string>Sky Surfing</string>
+		<string>Bird</string>
+		<string>Games</string>
+		<string>Baseball</string>
+		<string>Dressage</string>
+		<string>Snorkeling</string>
+		<string>Pyramid</string>
+		<string>Kite</string>
+		<string>Rowboat</string>
+		<string>Golf</string>
+		<string>Watersports</string>
+		<string>Lightning</string>
+		<string>Canyon</string>
+		<string>Auditorium</string>
+		<string>Night Sky</string>
+		<string>Karaoke</string>
+		<string>Skiing</string>
+		<string>Parade</string>
+		<string>Forest</string>
+		<string>Hot Air Balloon</string>
+		<string>Dragon Parade</string>
+		<string>Easter Egg</string>
+		<string>Monument</string>
+		<string>Jungle</string>
+		<string>Thanksgiving</string>
+		<string>Jockey Horse</string>
+		<string>Stadium</string>
+		<string>Airplane</string>
+		<string>Ballet</string>
+		<string>Yoga</string>
+		<string>Coral Reef</string>
+		<string>Skating</string>
+		<string>Wrestling</string>
+		<string>Bicycle</string>
+		<string>Tattoo</string>
+		<string>Amusement Park</string>
+		<string>Canoe</string>
+		<string>Cheerleading</string>
+		<string>Ping Pong</string>
+		<string>Fishing</string>
+		<string>Magic</string>
+		<string>Reptile</string>
+		<string>Winter Sport</string>
+		<string>Waterfall</string>
+		<string>Train</string>
+		<string>Bonsai</string>
+		<string>Surfing</string>
+		<string>Dog</string>
+		<string>Cake</string>
+		<string>Sledding</string>
+		<string>Sandcastle</string>
+		<string>Glacier</string>
+		<string>Lighthouse</string>
+		<string>Equestrian</string>
+		<string>Rafting</string>
+		<string>Shore</string>
+		<string>Hockey</string>
+		<string>Santa Claus</string>
+		<string>Formula One Car</string>
+		<string>Sport</string>
+		<string>Vehicle</string>
+		<string>Boxing</string>
+		<string>Rollerskating</string>
+		<string>Underwater</string>
+		<string>Orchestra</string>
+		<string>Carnival</string>
+		<string>Rocket</string>
+		<string>Skateboarding</string>
+		<string>Helicopter</string>
+		<string>Performance</string>
+		<string>Oktoberfest</string>
+		<string>Water Polo</string>
+		<string>Skate Park</string>
+		<string>Animal</string>
+		<string>Nightclub</string>
+		<string>String Instrument</string>
+		<string>Dinosaur</string>
+		<string>Gymnastics</string>
+		<string>Cricket</string>
+		<string>Volcano</string>
+		<string>Lake</string>
+		<string>Aurora</string>
+		<string>Dancing</string>
+		<string>Concert</string>
+		<string>Rock Climbing</string>
+		<string>Hang Glider</string>
+		<string>Rodeo</string>
+		<string>Fish</string>
+		<string>Art</string>
+		<string>Motorcycle</string>
+		<string>Volleyball</string>
+		<string>Wake Boarding</string>
+		<string>Badminton</string>
+		<string>Motor Sport</string>
+		<string>Sumo</string>
+		<string>Parasailing</string>
+		<string>Skydiving</string>
+		<string>Kickboxing</string>
+		<string>Pinata</string>
+		<string>Foosball</string>
+		<string>Go Kart</string>
+		<string>Poker</string>
+		<string>Kayak</string>
+		<string>Swimming</string>
+		<string>Atv</string>
+		<string>Beach</string>
+		<string>Dartboard</string>
+		<string>Athletics</string>
+		<string>Camping</string>
+		<string>Tornado</string>
+		<string>Billiards</string>
+		<string>Rugby</string>
+		<string>Airshow</string>
+	</array>
+</dict>
+</plist>

tests/Test-10.15.4.photoslibrary/database/search/searchProgress.plist

@@ -0,0 +1,26 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>insertAlbum</key>
+	<array/>
+	<key>insertAsset</key>
+	<array/>
+	<key>insertHighlight</key>
+	<array/>
+	<key>insertMemory</key>
+	<array/>
+	<key>insertMoment</key>
+	<array/>
+	<key>removeAlbum</key>
+	<array/>
+	<key>removeAsset</key>
+	<array/>
+	<key>removeHighlight</key>
+	<array/>
+	<key>removeMemory</key>
+	<array/>
+	<key>removeMoment</key>
+	<array/>
+</dict>
+</plist>

tests/Test-10.15.4.photoslibrary/database/search/searchSystemInfo.plist

@@ -0,0 +1,14 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>embeddingVersion</key>
+	<string>1</string>
+	<key>localeIdentifier</key>
+	<string>en_US</string>
+	<key>sceneTaxonomySHA</key>
+	<string>87914a047c69fbe8013fad2c70fa70c6c03b08b56190fe4054c880e6b9f57cc3</string>
+	<key>searchIndexVersion</key>
+	<string>10</string>
+</dict>
+</plist>

tests/Test-10.15.4.photoslibrary/private/com.apple.Photos.Migration/appPrivateData.plist

@@ -0,0 +1,26 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>MigrationService</key>
+	<dict>
+		<key>State</key>
+		<integer>4</integer>
+	</dict>
+	<key>MigrationService.LastCompletedTask</key>
+	<integer>12</integer>
+	<key>MigrationService.ValidationCounts</key>
+	<dict>
+		<key>MigrationDetectedFaceprint</key>
+		<integer>6</integer>
+		<key>MigrationManagedAsset</key>
+		<integer>0</integer>
+		<key>MigrationSceneClassification</key>
+		<integer>44</integer>
+		<key>MigrationUnmanagedAdjustment</key>
+		<integer>0</integer>
+		<key>RDVersion.cloudLocalState.CPLIsNotPushed</key>
+		<integer>7</integer>
+	</dict>
+</dict>
+</plist>

tests/Test-10.15.4.photoslibrary/private/com.apple.Photos/appPrivateData.plist

@@ -0,0 +1,49 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>CollapsedSidebarSectionIdentifiers</key>
+	<array/>
+	<key>ExpandedSidebarItemIdentifiers</key>
+	<array>
+		<string>92D68107-B6C7-453B-96D2-97B0F26D5B8B/L0/020</string>
+	</array>
+	<key>Photos</key>
+	<dict>
+		<key>CollapsedSidebarSectionIdentifiers</key>
+		<array/>
+		<key>ExpandedSidebarItemIdentifiers</key>
+		<array>
+			<string>TopLevelAlbums</string>
+			<string>TopLevelSlideshows</string>
+		</array>
+		<key>IPXWorkspaceControllerZoomLevelsKey</key>
+		<dict>
+			<key>kZoomLevelIdentifierAlbums</key>
+			<integer>7</integer>
+			<key>kZoomLevelIdentifierVersions</key>
+			<integer>7</integer>
+		</dict>
+		<key>lastAddToDestination</key>
+		<dict>
+			<key>key</key>
+			<integer>1</integer>
+			<key>lastKnownDisplayName</key>
+			<string>September 28, 2018</string>
+			<key>type</key>
+			<string>album</string>
+			<key>uuid</key>
+			<string>DFFKmHt3Tk+AGzZLe2Xq+g</string>
+		</dict>
+		<key>lastKnownItemCounts</key>
+		<dict>
+			<key>other</key>
+			<integer>0</integer>
+			<key>photos</key>
+			<integer>7</integer>
+			<key>videos</key>
+			<integer>0</integer>
+		</dict>
+	</dict>
+</dict>
+</plist>

tests/Test-10.15.4.photoslibrary/private/com.apple.photoanalysisd/caches/graph/PhotoAnalysisServicePreferences.plist

@@ -0,0 +1,26 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>BackgroundHighlightCollection</key>
+	<date>2020-04-04T17:34:35Z</date>
+	<key>BackgroundHighlightEnrichment</key>
+	<date>2020-04-04T17:34:34Z</date>
+	<key>BackgroundJobAssetRevGeocode</key>
+	<date>2020-04-04T20:16:33Z</date>
+	<key>BackgroundJobSearch</key>
+	<date>2020-04-04T17:34:35Z</date>
+	<key>BackgroundPeopleSuggestion</key>
+	<date>2020-04-04T17:34:34Z</date>
+	<key>BackgroundUserBehaviorProcessor</key>
+	<date>2020-04-04T13:56:56Z</date>
+	<key>PhotoAnalysisGraphLastBackgroundGraphConsistencyUpdateJobDateKey</key>
+	<date>2020-04-04T20:16:40Z</date>
+	<key>PhotoAnalysisGraphLastBackgroundGraphRebuildJobDate</key>
+	<date>2020-04-04T13:56:49Z</date>
+	<key>PhotoAnalysisGraphLastBackgroundMemoryGenerationJobDate</key>
+	<date>2020-04-04T20:16:33Z</date>
+	<key>SiriPortraitDonation</key>
+	<date>2020-04-04T13:56:56Z</date>
+</dict>
+</plist>