added --keyword-template

.github/workflows/pythonpackage.yml

@@ -9,7 +9,7 @@ jobs:
     strategy:
       max-parallel: 4
       matrix:
-        python-version: [3.6, 3.7, 3.8]
+        python-version: [3.8]
 
     steps:
     - uses: actions/checkout@v1

CHANGELOG.md

@@ -4,6 +4,46 @@ All notable changes to this project will be documented in this file. Dates are d
 
 Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 
+#### [v0.28.7](https://github.com/RhetTbull/osxphotos/compare/v0.28.6...v0.28.7)
+
+> 28 April 2020
+
+- Added --album-keyword and --person-keyword to CLI, closes #61 [`#61`](https://github.com/RhetTbull/osxphotos/issues/61)
+- Updated test libraries [`54d5d4b`](https://github.com/RhetTbull/osxphotos/commit/54d5d4b7ba99204f58e723231309ab6e306be28c)
+- Updated CHANGELOG.md [`38137a1`](https://github.com/RhetTbull/osxphotos/commit/38137a1351cdb7ab72393ea03828933dac0b76b0)
+- Updated tests/README.md [`56a0006`](https://github.com/RhetTbull/osxphotos/commit/56a000609f2f08d0f8800fec49cada2980c3bb9d)
+
+#### [v0.28.6](https://github.com/RhetTbull/osxphotos/compare/v0.28.5...v0.28.6)
+
+> 26 April 2020
+
+- Fixed locale bug in templates, closes #113 [`#113`](https://github.com/RhetTbull/osxphotos/issues/113)
+- Updated CHANGELOG.md [`81d4e39`](https://github.com/RhetTbull/osxphotos/commit/81d4e392c39f0fe6f967a447c7d0c970bf224032)
+- Updated test to avoid issue with GitHub workflow [`9be0f84`](https://github.com/RhetTbull/osxphotos/commit/9be0f849b73061d053d30274ff3295b79c88f0b6)
+- Update pythonpackage.yml to remove older pythons [`ccb5f25`](https://github.com/RhetTbull/osxphotos/commit/ccb5f252d14e9335ae04a2e338a6d527b80c9a93)
+
+#### [v0.28.5](https://github.com/RhetTbull/osxphotos/compare/0.28.2...v0.28.5)
+
+> 21 April 2020
+
+- added __len__ to PhotosDB, closes #44 [`#44`](https://github.com/RhetTbull/osxphotos/issues/44)
+- Updated use of _PHOTOS_4_VERSION, closes #106 [`#106`](https://github.com/RhetTbull/osxphotos/issues/106)
+- Updated tests and test library with RAW images [`9b9b54e`](https://github.com/RhetTbull/osxphotos/commit/9b9b54e590e43ae49fb3ae41d493a1f8faec4181)
+- Updated setup.py to resolve issue with bpylist2 on python < 3.8 [`8e4b88a`](https://github.com/RhetTbull/osxphotos/commit/8e4b88ad1fc18438f941e045bfc8aeac878914f9)
+- Added cli.py for use with pyinstaller [`cf28cb6`](https://github.com/RhetTbull/osxphotos/commit/cf28cb6452de17f2ef8d80435386e8d5a1aabd34)
+- added raw_is_original handling [`a337e79`](https://github.com/RhetTbull/osxphotos/commit/a337e79e13802b4824c2f088ce9db1c027d6f3c5)
+- Updated CHANGELOG.md [`22f1e8f`](https://github.com/RhetTbull/osxphotos/commit/22f1e8f2a6478e0576f6bff53e348aad8680ae69)
+
+#### [0.28.2](https://github.com/RhetTbull/osxphotos/compare/v0.28.1...0.28.2)
+
+> 18 April 2020
+
+- Added folder support for Photos <= 4, closes #93 [`#93`](https://github.com/RhetTbull/osxphotos/issues/93)
+- cleaned up SQL statements in _process_database4 [`6f28171`](https://github.com/RhetTbull/osxphotos/commit/6f281711e2001a63ffad076d7b9835272d5d09da)
+- Updated CHANGELOG.md [`1fa9583`](https://github.com/RhetTbull/osxphotos/commit/1fa9583ea689d54d2613a064f1ade25bcdfbf043)
+- Fixed suffix check on export to be case insensitive [`4b30b3b`](https://github.com/RhetTbull/osxphotos/commit/4b30b3b4260e2c7409e18825e5b626efe646db16)
+- test library update [`3bac106`](https://github.com/RhetTbull/osxphotos/commit/3bac106eb7a180e9e39643a89087d92bf2a437d0)
+
 #### [v0.28.1](https://github.com/RhetTbull/osxphotos/compare/v0.27.4...v0.28.1)
 
 > 18 April 2020
@@ -273,7 +313,11 @@ Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 - Moved PhotosDB attributes to properties instead of methods [`d95acdf`](https://github.com/RhetTbull/osxphotos/commit/d95acdf9f8764a1720bcba71a6dad29bf668eaf9)
 - changed interface for export, prepped for exiftool_json_sidecar [`1fe8859`](https://github.com/RhetTbull/osxphotos/commit/1fe885962e8a9a420e776bdd3dc640ca143224b2)
 
-#### [v0.15.1](https://github.com/RhetTbull/osxphotos/compare/v0.14.21...v0.15.1)
+#### [v0.15.1](https://github.com/RhetTbull/osxphotos/compare/v0.15.0...v0.15.1)
+
+> 28 April 2020
+
+#### [v0.15.0](https://github.com/RhetTbull/osxphotos/compare/v0.14.21...v0.15.0)
 
 > 14 December 2019
 

README.md

@@ -16,7 +16,7 @@
     + [AlbumInfo](#albuminfo)
     + [FolderInfo](#folderinfo)
     + [PlaceInfo](#placeinfo)
-    + [Template Functions](#template-functions)
+    + [Template Substitutions](#template-substitutions)
     + [Utility Functions](#utility-functions)
   * [Examples](#examples)
   * [Related Projects](#related-projects)
@@ -33,9 +33,11 @@ OSXPhotos provides the ability to interact with and query Apple's Photos.app lib
 
 ## Supported operating systems
 
-Only works on MacOS (aka Mac OS X). Tested on MacOS 10.12.6 / Photos 2.0, 10.13.6 / Photos 3.0, MacOS 10.14.5, 10.14.6 / Photos 4.0, MacOS 10.15.1 / Photos 5.0. Requires python >= 3.6
+Only works on MacOS (aka Mac OS X). Tested on MacOS 10.12.6 / Photos 2.0, 10.13.6 / Photos 3.0, MacOS 10.14.5, 10.14.6 / Photos 4.0, MacOS 10.15.1 & 10.15.4 / Photos 5.0. 
 
-This package will read Photos databases for any supported version on any supported OS version.  E.g. you can read a database created with Photos 4.0 on MacOS 10.14 on a machine running MacOS 10.12
+Requires python >= 3.6 though if you use `pip` to install, you must use python >= 3.8.  See notes [below](#Installation-instructions). I highly recommend running this with python >= 3.8 as I'll eventually drop support for 3.6 and 3.7.
+
+This package will read Photos databases for any supported version on any supported OS version.  E.g. you can read a database created with Photos 4.0 on MacOS 10.14 on a machine running MacOS 10.12.
 
 
 ## Installation instructions
@@ -44,6 +46,14 @@ osxmetadata uses setuptools, thus simply run:
 
 	python3 setup.py install
 
+If you're using python 3.6 or 3.7, you'll need to do this first to get around an issue with bpylist2:
+
+	pip install -r requirements.txt
+
+You can also install directly from [pypi](https://pypi.org/) but you must use python >= 3.8 to avoid an error with bpylist2.  The package currently works fine with python 3.6 or 3.7 but I know of no way to get `pip` to install the right dependencies. 
+
+    pip install osxphotos
+	
 ## Command Line Usage
 
 This package will install a command line utility called `osxphotos` that allows you to query the Photos database.  Alternatively, you can also run the command line utility like this: `python3 -m osxphotos`
@@ -91,6 +101,7 @@ Example: `osxphotos help export`
 
 ```
 Usage: osxphotos export [OPTIONS] [PHOTOS_LIBRARY]... DEST
+Usage: __main__.py export [OPTIONS] [PHOTOS_LIBRARY]... DEST
 
   Export photos from the Photos database. Export path DEST is required.
   Optionally, query the Photos database using 1 or more search options;  if
@@ -209,6 +220,22 @@ Options:
                                   photos if the RAW photo does not have an
                                   associated jpeg image (e.g. the RAW file was
                                   imported to Photos without a jpeg preview).
+  --person-keyword                Use person in image as keyword/tag when
+                                  exporting metadata.
+  --album-keyword                 Use album name as keyword/tag when exporting
+                                  metadata.
+  --keyword-template TEMPLATE     For use with --exiftool, --sidecar; specify
+                                  a template string to use as keyword in the
+                                  form '{name,DEFAULT}' This is the same
+                                  format as --directory.  For example, if you
+                                  wanted to add the full path to the folder
+                                  and album photo is contained in as a keyword
+                                  when exporting you could specify --keyword-
+                                  template "{folder_album}" You may specify
+                                  more than one template, for example
+                                  --keyword-template "{folder_album}"
+                                  --keyword-template "{created.year}" See
+                                  Templating System below.
   --current-name                  Use photo's current filename instead of
                                   original filename for export.  Note:
                                   Starting with Photos 5, all photos are
@@ -256,7 +283,7 @@ Options:
 
 **Templating System**
 
-With the --directory option, you may specify a template for the export
+With the --directory option you may specify a template for the export
 directory.  This directory will be appended to the export path specified in
 the export DEST argument to export.  For example, if template is
 '{created.year}/{created.month}', and export desitnation DEST is
@@ -264,6 +291,11 @@ the export DEST argument to export.  For example, if template is
 be '/Users/maria/Pictures/export/2020/March' if the photo was created in March
 2020.
 
+The templating system may also be used with the --keyword-template option to
+set keywords on export (with --exiftool or --sidecar), for example, to set a
+new keyword in format 'folder/subfolder/album' to preserve the folder/album
+structure, you can use --keyword-template "{folder_album}"
+
 In the template, valid template substitutions will be replaced by the
 corresponding value from the table below.  Invalid substitutions will result
 in a an error and the script will abort.
@@ -276,7 +308,7 @@ rendered name, use double braces, e.g. '{{' or '}}', thus using
 You may specify an optional default value to use if the substitution does not
 contain a value (e.g. the value is null) by specifying the default value after
 a ',' in the template string: for example, if template is
-'{created.year}/{place.address,'NO_ADDRESS'}' but there was no address
+'{created.year}/{place.address,NO_ADDRESS}' but there was no address
 associated with the photo, the resulting output would be:
 '2020/NO_ADDRESS/photoname.jpg'. If specified, the default value may not
 contain a brace symbol ('{' or '}').
@@ -288,7 +320,7 @@ I plan to eventually extend the templating system to the exported filename so
 you can specify the filename using a template.
 
 Substitution                    Description
-{name}                          Filename of the photo
+{name}                          Current filename of the photo
 {original_name}                 Photo's original filename when imported to
                                 Photos
 {title}                         Title of the photo
@@ -806,6 +838,7 @@ For example, in my library, Photos says I have 19,386 photos and 474 movies.  Ho
 >>>
 ```
 
+
 ### PhotoInfo 
 PhotosDB.photos() returns a list of PhotoInfo objects.  Each PhotoInfo object represents a single photo in the Photos library.
 
@@ -950,7 +983,8 @@ Returns True if photo is a panorama, otherwise False.
 #### `json()`
 Returns a JSON representation of all photo info 
 
-#### `export(dest, *filename, edited=False, live_photo=False, overwrite=False, increment=True, sidecar_json=False, sidecar_xmp=False, use_photos_export=False, timeout=120, exiftool=False, no_xattr=False)`
+#### `export()`
+`export(dest, *filename, edited=False, live_photo=False, overwrite=False, increment=True, sidecar_json=False, sidecar_xmp=False, use_photos_export=False, timeout=120, exiftool=False, no_xattr=False, use_albums_as_keywords=False, use_persons_as_keywords=False)`
 
 Export photo from the Photos library to another destination on disk.  
 - dest: must be valid destination path as str (or exception raised).
@@ -965,6 +999,8 @@ Export photo from the Photos library to another destination on disk.
 - timeout: (int, default=120) timeout in seconds used with use_photos_export
 - exiftool: (boolean, default = False) if True, will use [exiftool](https://exiftool.org/) to write metadata directly to the exported photo; exiftool must be installed and in the system path
 - no_xattr: (boolean, default = False); if True, exports file without preserving extended attributes
+- use_albums_as_keywords: (boolean, default = False); if True, will use album names as keywords when exporting metadata with exiftool or sidecar
+- use_persons_as_keywords: (boolean, default = False); if True, will use person names as keywords when exporting metadata with exiftool or sidecar
 
 Returns: list of paths to exported files. More than one file could be exported, for example if live_photo=True, both the original imaage and the associated .mov file will be exported
 
@@ -986,6 +1022,26 @@ If overwrite=False and increment=False, export will fail if destination file alr
 
 **Implementation Note**: Because the usual python file copy methods don't preserve all the metadata available on MacOS, export uses `/usr/bin/ditto` to do the copy for export. ditto preserves most metadata such as extended attributes, permissions, ACLs, etc.
 
+#### <a name="rendertemplate">`render_template()`</a>
+
+`render_template(template, none_str = "_", path_sep = None)`
+Render template string for photo.  none_str is used if template substitution results in None value and no default specified. 
+- `template`: str in form "{name,DEFAULT}" where name is one of the values in table below. The "," and default value that follows are optional. If specified, "DEFAULT" will be used if "name" is None.  This is useful for values which are not always present, for example reverse geolocation data.
+- `none_str`: optional str to use as substitution when template value is None and no default specified in the template string.  default is "_".
+- `path_sep`: optional character to use as path separator, default is os.path.sep
+
+Returns a tuple of (rendered, unmatched) where rendered is a list of rendered strings with all substitutions made and unmatched is a list of any strings that resembled a template substitution but did not match a known substitution. E.g. if template contained "{foo}", unmatched would be ["foo"].
+
+e.g. `render_filepath_template("{created.year}/{foo}", photo)` would return `(["2020/{foo}"],["foo"])`
+
+If you want to include "{" or "}" in the output, use "{{" or "}}"
+
+e.g. `render_filepath_template("{created.year}/{{foo}}", photo)` would return `(["2020/{foo}"],[])`
+
+Some substitutions, notably `album`, `keyword`, and `person` could return multiple values, hence a new string will be return for each possible substitution (hence why a list of rendered strings is returned).  For example, a photo in 2 albums: 'Vacation' and 'Family' would result in the following rendered values if template was "{created.year}/{album}" and created.year == 2020: `["2020/Vacation","2020/Family"]` 
+
+See [Template Substitutions](#template-substitutions) for additional details.
+
 ### AlbumInfo
 PhotosDB.album_info and PhotoInfo.album_info return a list of AlbumInfo objects.  Each AlbumInfo object represents a single album in the Photos library.
 
@@ -1131,26 +1187,9 @@ PostalAddress(street='3700 Wailea Alanui Dr', sub_locality=None, city='Kihei', s
 '96753'
 ```
 
-### Template Functions
-
-There is a simple template system used by the command line client to specify the output directory using a template.  The following are available in `osxphotos.template`.
-
-#### `render_filepath_template(template, photo, none_str="_")`
-Render template string for photo.  none_str is used if template substitution results in None value and no default specified. 
-- `template`: str in form "{name,DEFAULT}" where name is one of the values in table below. The "," and default value that follows are optional. If specified, "DEFAULT" will be used if "name" is None.  This is useful for values which are not always present, for example reverse geolocation data.
-- `photo`: a [PhotoInfo](#photoinfo) object
-- `none_str`: optional str to use as substitution when template value is None and no default specified in the template string.  default is "_".
-
-Returns a tuple of (rendered, unmatched) where rendered is a list of rendered strings with all substitutions made and unmatched is a list of any strings that resembled a template substitution but did not match a known substitution. E.g. if template contained "{foo}", unmatched would be ["foo"].
-
-e.g. `render_filepath_template("{created.year}/{foo}", photo)` would return `(["2020/{foo}"],["foo"])`
-
-If you want to include "{" or "}" in the output, use "{{" or "}}"
-
-e.g. `render_filepath_template("{created.year}/{{foo}}", photo)` would return `(["2020/{foo}"],[])`
-
-Some substitutions, notably `album`, `keyword`, and `person` could return multiple values, hence a new string will be return for each possible substitution (hence why a list of rendered strings is returned).  For example, a photo in 2 albums: 'Vacation' and 'Family' would result in the following rendered values if template was "{created.year}/{album}" and created.year == 2020: `["2020/Vacation","2020/Family"]` 
+### Template Substitutions
 
+The following substitutions are availabe for use with `PhotoInfo.render_template()` 
 
 | Substitution | Description |
 |--------------|-------------|
@@ -1190,21 +1229,6 @@ Some substitutions, notably `album`, `keyword`, and `person` could return multip
 |{person}|Person(s) / face(s) in a photo|
 
 
-#### `DateTimeFormatter(dt)`
-Class that provides easy access to formatted datetime values.
-- `dt`: a datetime.datetime object
-
-Returnes `DateTimeFormater` class.
-
-Has the following properties:
-- `date`: Date in ISO format without timezone, e.g. "2020-03-04"
-- `year`: 4-digit year
-- `yy`: 2-digit year
-- `month`: month name in user's locale
-- `mon`: month abbreviation in user's locale
-- `mm`: 2-digit month
-- `doy`: 3-digit day of year (e.g. Julian day)
-
 ### Utility Functions
 
 The following functions are located in osxphotos.utils
@@ -1310,7 +1334,7 @@ Testing against "real world" Photos libraries would be especially helpful.  If y
 
 My goal is make osxphotos as reliable and comprehensive as possible.  The test suite currently has over 400 tests--but there are still some [bugs](https://github.com/RhetTbull/osxphotos/issues?q=is%3Aissue+is%3Aopen+label%3Abug) or incomplete features lurking.  If you find bugs please open an [issue](https://github.com/RhetTbull/osxphotos/issues).  Notable issues include:
 
-- RAW images imported to Photos with an associated jpeg preview are not handled correctly by osxphotos.  osxphotos query and export will operate on the jpeg preview instead of the RAW image as will `PhotoInfo.path`.  If the user selects "Use RAW as original" in Photos, the RAW image will be exported or operated on but the jpeg will be ignored.  See [Issue #101](https://github.com/RhetTbull/osxphotos/issues/101) Note: Alpha version of fix for this bug is implemented in the current version of osxphotos.
+- RAW images imported to Photos with an associated jpeg preview are not handled correctly by osxphotos.  osxphotos query and export will operate on the jpeg preview instead of the RAW image as will `PhotoInfo.path`.  If the user selects "Use RAW as original" in Photos, the RAW image will be exported or operated on but the jpeg will be ignored.  See [Issue #101](https://github.com/RhetTbull/osxphotos/issues/101) Note: Beta version of fix for this bug is implemented in the current version of osxphotos.
 - The `--download-missing` option for `osxphotos export` does not work correctly with burst images.  It will download the primary image but not the other burst images.  See [Issue #75](https://github.com/RhetTbull/osxphotos/issues/75)
 
 ## Implementation Notes

cli.py

@@ -0,0 +1,19 @@
+""" stand alone command line script for use with pyinstaller
+    
+    To build this into an executable:
+    - install pyinstaller:
+        python3 -m pip install pyinstaller
+    - then use build_cli_exe.sh to run pyinstaller or execute the following command:
+        pyinstaller --onefile --hidden-import="pkg_resources.py2_warn" --name osxphotos --add-data osxphotos/templates/xmp_sidecar.mako:osxphotos/templates cli.py
+
+    Resulting executable will be in "dist/osxphotos"
+
+    Note: This is *not* the cli that "python3 -m pip install osxphotos" or "python setup.py install" would install;
+    it's merely a wrapper around __main__.py to allow pyinstaller to work
+    
+"""
+
+from osxphotos.__main__ import cli
+
+if __name__ == "__main__":
+    cli()

make_cli_exe.sh

@@ -0,0 +1,8 @@
+#!/bin/sh 
+
+# This will build an stand-alone executable called 'osxphotos' in your ./dist directory
+# using pyinstaller
+# If you need to install pyinstaller:
+# python3 -m pip install --upgrade pyinstaller
+
+pyinstaller --onefile --hidden-import="pkg_resources.py2_warn" --name osxphotos --add-data osxphotos/templates/xmp_sidecar.mako:osxphotos/templates cli.py

osxphotos/__main__.py

@@ -18,14 +18,10 @@ from pathvalidate import (
 
 import osxphotos
 
-from ._constants import _EXIF_TOOL_URL, _PHOTOS_5_VERSION, _UNKNOWN_PLACE
+from ._constants import _EXIF_TOOL_URL, _PHOTOS_4_VERSION, _UNKNOWN_PLACE
 from ._version import __version__
 from .exiftool import get_exiftool_path
-from .template import (
-    render_filepath_template,
-    TEMPLATE_SUBSTITUTIONS,
-    TEMPLATE_SUBSTITUTIONS_MULTI_VALUED,
-)
+from .template import TEMPLATE_SUBSTITUTIONS, TEMPLATE_SUBSTITUTIONS_MULTI_VALUED
 from .utils import _copy_file, create_path_by_date
 
 
@@ -83,7 +79,7 @@ class ExportCommand(click.Command):
         formatter.write_text("**Templating System**")
         formatter.write("\n")
         formatter.write_text(
-            "With the --directory option, you may specify a template for the "
+            "With the --directory option you may specify a template for the "
             + "export directory.  This directory will be appended to the export path specified "
             + "in the export DEST argument to export.  For example, if template is "
             + "'{created.year}/{created.month}', and export desitnation DEST is "
@@ -92,6 +88,13 @@ class ExportCommand(click.Command):
             + "if the photo was created in March 2020. "
         )
         formatter.write("\n")
+        formatter.write_text(
+            "The templating system may also be used with the --keyword-template option "
+            + "to set keywords on export (with --exiftool or --sidecar), "
+            + "for example, to set a new keyword in format 'folder/subfolder/album' to "
+            + 'preserve the folder/album structure, you can use --keyword-template "{folder_album}"'
+        )
+        formatter.write("\n")
         formatter.write_text(
             "In the template, valid template substitutions will be replaced by "
             + "the corresponding value from the table below.  Invalid substitutions will result in a "
@@ -109,7 +112,7 @@ class ExportCommand(click.Command):
             "You may specify an optional default value to use if the substitution does not contain a value "
             + "(e.g. the value is null) "
             + "by specifying the default value after a ',' in the template string: "
-            + "for example, if template is '{created.year}/{place.address,'NO_ADDRESS'}' "
+            + "for example, if template is '{created.year}/{place.address,NO_ADDRESS}' "
             + "but there was no address associated with the photo, the resulting output would be: "
             + "'2020/NO_ADDRESS/photoname.jpg'. "
             + "If specified, the default value may not contain a brace symbol ('{' or '}')."
@@ -428,7 +431,7 @@ def albums(ctx, cli_obj, db, json_, photos_library):
 
     photosdb = osxphotos.PhotosDB(dbfile=db)
     albums = {"albums": photosdb.albums_as_dict}
-    if photosdb.db_version >= _PHOTOS_5_VERSION:
+    if photosdb.db_version > _PHOTOS_4_VERSION:
         albums["shared albums"] = photosdb.albums_shared_as_dict
 
     if json_ or cli_obj.json:
@@ -493,7 +496,7 @@ def info(ctx, cli_obj, db, json_, photos_library):
     not_shared_movies = [p for p in movies if not p.shared]
     info["movie_count"] = len(not_shared_movies)
 
-    if pdb.db_version >= _PHOTOS_5_VERSION:
+    if pdb.db_version > _PHOTOS_4_VERSION:
         shared_photos = [p for p in photos if p.shared]
         info["shared_photo_count"] = len(shared_photos)
 
@@ -508,7 +511,7 @@ def info(ctx, cli_obj, db, json_, photos_library):
     info["albums_count"] = len(albums)
     info["albums"] = albums
 
-    if pdb.db_version >= _PHOTOS_5_VERSION:
+    if pdb.db_version > _PHOTOS_4_VERSION:
         albums_shared = pdb.albums_shared_as_dict
         info["shared_albums_count"] = len(albums_shared)
         info["shared_albums"] = albums_shared
@@ -893,6 +896,30 @@ def query(
     "Note: this does not skip RAW photos if the RAW photo does not have an associated jpeg image "
     "(e.g. the RAW file was imported to Photos without a jpeg preview).",
 )
+@click.option(
+    "--person-keyword",
+    is_flag=True,
+    help="Use person in image as keyword/tag when exporting metadata.",
+)
+@click.option(
+    "--album-keyword",
+    is_flag=True,
+    help="Use album name as keyword/tag when exporting metadata.",
+)
+@click.option(
+    "--keyword-template",
+    metavar="TEMPLATE",
+    multiple=True,
+    default=None,
+    help="For use with --exiftool, --sidecar; specify a template string to use as "
+    "keyword in the form '{name,DEFAULT}' "
+    "This is the same format as --directory.  For example, if you wanted to add "
+    "the full path to the folder and album photo is contained in as a keyword when exporting "
+    'you could specify --keyword-template "{folder_album}" '
+    'You may specify more than one template, for example --keyword-template "{folder_album}" '
+    '--keyword-template "{created.year}" '
+    "See Templating System below.",
+)
 @click.option(
     "--current-name",
     is_flag=True,
@@ -983,6 +1010,9 @@ def export(
     skip_bursts,
     skip_live,
     skip_raw,
+    person_keyword,
+    album_keyword,
+    keyword_template,
     current_name,
     sidecar,
     only_photos,
@@ -1174,6 +1204,9 @@ def export(
                         directory,
                         no_extended_attributes,
                         export_raw,
+                        album_keyword,
+                        person_keyword,
+                        keyword_template,
                     )
         else:
             for p in photos:
@@ -1192,6 +1225,9 @@ def export(
                     directory,
                     no_extended_attributes,
                     export_raw,
+                    album_keyword,
+                    person_keyword,
+                    keyword_template,
                 )
                 if export_paths:
                     click.echo(f"Exported {p.filename} to {export_paths}")
@@ -1578,6 +1614,9 @@ def export_photo(
     directory,
     no_extended_attributes,
     export_raw,
+    album_keyword,
+    person_keyword,
+    keyword_template,
 ):
     """ Helper function for export that does the actual export
         photo: PhotoInfo object
@@ -1594,6 +1633,9 @@ def export_photo(
         directory: template used to determine output directory
         no_extended_attributes: boolean; if True, exports photo without preserving extended attributes
         export_raw: boolean; if True exports RAW image associate with the photo
+        album_keyword: boolean; if True, exports album names as keywords in metadata
+        person_keyword: boolean; if True, exports person names as keywords in metadata
+        keyword_template: list of strings; if provided use rendered template strings as keywords
         returns list of path(s) of exported photo or None if photo was missing 
     """
 
@@ -1634,7 +1676,7 @@ def export_photo(
         dest_paths = [dest_path]
     elif directory:
         # got a directory template, render it and check results are valid
-        dirnames, unmatched = render_filepath_template(directory, photo)
+        dirnames, unmatched = photo.render_template(directory)
         if unmatched:
             raise click.BadOptionUsage(
                 "directory",
@@ -1677,6 +1719,9 @@ def export_photo(
             use_photos_export=use_photos_export,
             exiftool=exiftool,
             no_xattr=no_extended_attributes,
+            use_albums_as_keywords=album_keyword,
+            use_persons_as_keywords=person_keyword,
+            keyword_template=keyword_template,
         )[0]
         photo_paths.append(photo_path)
 
@@ -1712,6 +1757,9 @@ def export_photo(
                     use_photos_export=use_photos_export,
                     exiftool=exiftool,
                     no_xattr=no_extended_attributes,
+                    use_albums_as_keywords=album_keyword,
+                    use_persons_as_keywords=person_keyword,
+                    keyword_template=keyword_template,
                 )
 
     return photo_paths

osxphotos/_constants.py

@@ -51,3 +51,12 @@ _PHOTOS_5_ROOT_FOLDER_KIND = 3999  # root folder
 _PHOTOS_4_ALBUM_KIND = 3  # RKAlbum.albumSubclass
 _PHOTOS_4_TOP_LEVEL_ALBUM = "TopLevelAlbums"
 _PHOTOS_4_ROOT_FOLDER = "LibraryFolder"
+
+# EXIF related constants
+# max keyword length for IPTC:Keyword, reference
+# https://www.iptc.org/std/photometadata/documentation/userguide/
+_MAX_IPTC_KEYWORD_LEN = 64
+
+# Sentinel value for detecting if a template in keyword_template doesn't match
+# If anyone has a keyword matching this, then too bad...
+_OSXPHOTOS_NONE_SENTINEL = "OSXPhotosXYZZY42_Sentinel$"

osxphotos/_version.py

@@ -1,3 +1,3 @@
 """ version info """
 
-__version__ = "0.28.2"
+__version__ = "0.28.13"

osxphotos/datetime_formatter.py

@@ -0,0 +1,52 @@
+""" Simple formatting of datetime.datetime objects """
+
+import datetime
+
+
+class DateTimeFormatter:
+    """ provides property access to formatted datetime.datetime strftime values """
+
+    def __init__(self, dt: datetime.datetime):
+        self.dt = dt
+
+    @property
+    def date(self):
+        """ ISO date in form 2020-03-22 """
+        date = self.dt.date().isoformat()
+        return date
+
+    @property
+    def year(self):
+        """ 4 digit year """
+        year = f"{self.dt.year}"
+        return year
+
+    @property
+    def yy(self):
+        """ 2 digit year """
+        yy = f"{self.dt.strftime('%y')}"
+        return yy
+
+    @property
+    def mm(self):
+        """ 2 digit month """
+        mm = f"{self.dt.strftime('%m')}"
+        return mm
+
+    @property
+    def month(self):
+        """ Month as locale's full name """
+        month = f"{self.dt.strftime('%B')}"
+        return month
+
+    @property
+    def mon(self):
+        """ Month as locale's abbreviated name """
+        mon = f"{self.dt.strftime('%b')}"
+        return mon
+
+    @property
+    def doy(self):
+        """ Julian day of year starting from 001 """
+        doy = f"{self.dt.strftime('%j')}"
+        return doy

osxphotos/photoinfo.py

@@ -7,6 +7,7 @@ PhotosDB.photos() returns a list of PhotoInfo objects
 import glob
 import json
 import logging
+import os
 import os.path
 import pathlib
 import re
@@ -19,16 +20,25 @@ import yaml
 from mako.template import Template
 
 from ._constants import (
+    _MAX_IPTC_KEYWORD_LEN,
     _MOVIE_TYPE,
+    _OSXPHOTOS_NONE_SENTINEL,
     _PHOTO_TYPE,
     _PHOTOS_4_VERSION,
     _PHOTOS_5_SHARED_PHOTO_PATH,
     _TEMPLATE_DIR,
+    _UNKNOWN_PERSON,
     _XMP_TEMPLATE_NAME,
 )
+from .albuminfo import AlbumInfo
+from .datetime_formatter import DateTimeFormatter
 from .exiftool import ExifTool
 from .placeinfo import PlaceInfo4, PlaceInfo5
-from .albuminfo import AlbumInfo
+from .template import (
+    MULTI_VALUE_SUBSTITUTIONS,
+    TEMPLATE_SUBSTITUTIONS,
+    TEMPLATE_SUBSTITUTIONS_MULTI_VALUED,
+)
 from .utils import (
     _copy_file,
     _export_photo_uuid_applescript,
@@ -53,7 +63,12 @@ class PhotoInfo:
     @property
     def filename(self):
         """ filename of the picture """
-        return self._info["filename"]
+        if self.has_raw and self.raw_original:
+            # return name of the RAW file
+            # TODO: not yet implemented
+            return self._info["filename"]
+        else:
+            return self._info["filename"]
 
     @property
     def original_filename(self):
@@ -296,9 +311,10 @@ class PhotoInfo:
             glob_str = f"{filestem}*.{raw_ext}"
             raw_file = findfiles(glob_str, filepath)
             if len(raw_file) != 1:
-                logging.warning(
-                    f"Error getting path to RAW file: {filepath}/{glob_str}"
-                )
+                # Note: In Photos Version 5.0 (141.19.150), images not copied to Photos Library
+                # that are missing do not always trigger is_missing = True as happens
+                # in earlier version so it's possible for this check to fail, if so, return None
+                logging.debug(f"Error getting path to RAW file: {filepath}/{glob_str}")
                 photopath = None
             else:
                 photopath = os.path.join(filepath, raw_file[0])
@@ -610,7 +626,7 @@ class PhotoInfo:
         """ returns True if associated RAW image and the RAW image is selected in Photos
             via "Use RAW as Original "
             otherwise returns False """
-        return True if self._info["original_resource_choice"] == 1 else False
+        return self._info["raw_is_original"]
 
     def export(
         self,
@@ -627,6 +643,9 @@ class PhotoInfo:
         timeout=120,
         exiftool=False,
         no_xattr=False,
+        use_albums_as_keywords=False,
+        use_persons_as_keywords=False,
+        keyword_template=None,
     ):
         """ export photo 
             dest: must be valid destination path (or exception raised) 
@@ -652,7 +671,13 @@ class PhotoInfo:
             timeout: (int, default=120) timeout in seconds used with use_photos_export
             exiftool: (boolean, default = False); if True, will use exiftool to write metadata to export file
             no_xattr: (boolean, default = False); if True, exports file without preserving extended attributes
-            returns list of full paths to the exported files """
+            returns list of full paths to the exported files
+            use_albums_as_keywords: (boolean, default = False); if True, will include album names in keywords
+            when exporting metadata with exiftool or sidecar
+            use_persons_as_keywords: (boolean, default = False); if True, will include person names in keywords
+            when exporting metadata with exiftool or sidecar
+            keyword_template: (list of strings); list of template strings that will be rendered as used as keywords
+             """
 
         # list of all files exported during this call to export
         exported_files = []
@@ -856,7 +881,11 @@ class PhotoInfo:
         if sidecar_json:
             logging.debug("writing exiftool_json_sidecar")
             sidecar_filename = dest.parent / pathlib.Path(f"{dest.stem}.json")
-            sidecar_str = self._exiftool_json_sidecar()
+            sidecar_str = self._exiftool_json_sidecar(
+                use_albums_as_keywords=use_albums_as_keywords,
+                use_persons_as_keywords=use_persons_as_keywords,
+                keyword_template=keyword_template,
+            )
             try:
                 self._write_sidecar(sidecar_filename, sidecar_str)
             except Exception as e:
@@ -866,7 +895,11 @@ class PhotoInfo:
         if sidecar_xmp:
             logging.debug("writing xmp_sidecar")
             sidecar_filename = dest.parent / pathlib.Path(f"{dest.stem}.xmp")
-            sidecar_str = self._xmp_sidecar()
+            sidecar_str = self._xmp_sidecar(
+                use_albums_as_keywords=use_albums_as_keywords,
+                use_persons_as_keywords=use_persons_as_keywords,
+                keyword_template=keyword_template,
+            )
             try:
                 self._write_sidecar(sidecar_filename, sidecar_str)
             except Exception as e:
@@ -876,17 +909,367 @@ class PhotoInfo:
         # if exiftool, write the metadata
         if exiftool and exported_files:
             for exported_file in exported_files:
-                self._write_exif_data(exported_file)
+                self._write_exif_data(
+                    exported_file,
+                    use_albums_as_keywords=use_albums_as_keywords,
+                    use_persons_as_keywords=use_persons_as_keywords,
+                    keyword_template=keyword_template,
+                )
 
         return exported_files
 
-    def _write_exif_data(self, filepath):
+    def render_template(self, template, none_str="_", path_sep=None):
+        """ render a filename or directory template 
+            template: str template 
+            none_str: str to use default for None values, default is '_' 
+            path_sep: optional character to use as path separator, default is os.path.sep """
+
+        if path_sep is None:
+            path_sep = os.path.sep
+        elif path_sep is not None and len(path_sep) != 1:
+            raise ValueError(f"path_sep must be single character: {path_sep}")
+
+        # the rendering happens in two phases:
+        # phase 1: handle all the single-value template substitutions
+        #          results in a single string with all the template fields replaced
+        # phase 2: loop through all the multi-value template substitutions
+        #          could result in multiple strings
+        #          e.g. if template is "{album}/{person}" and there are 2 albums and 3 persons in the photo
+        #          there would be 6 possible renderings (2 albums x 3 persons)
+
+        # regex to find {template_field,optional_default} in strings
+        # for explanation of regex see https://regex101.com/r/4JJg42/1
+        # pylint: disable=anomalous-backslash-in-string
+        regex = r"(?<!\{)\{([^\\,}]+)(,{0,1}(([\w\-. ]+))?)(?=\}(?!\}))\}"
+
+        if type(template) is not str:
+            raise TypeError(f"template must be type str, not {type(template)}")
+
+        def make_subst_function(self, none_str, get_func=self.get_template_value):
+            """ returns: substitution function for use in re.sub 
+                photo: a PhotoInfo object
+                none_str: value to use if substitution lookup is None and no default provided
+                get_func: function that gets the substitution value for a given template field
+                        default is get_template_value which handles the single-value fields """
+
+            # closure to capture photo, none_str in subst
+            def subst(matchobj):
+                groups = len(matchobj.groups())
+                if groups == 4:
+                    try:
+                        val = get_func(matchobj.group(1))
+                    except KeyError:
+                        return matchobj.group(0)
+
+                    if val is None:
+                        return (
+                            matchobj.group(3)
+                            if matchobj.group(3) is not None
+                            else none_str
+                        )
+                    else:
+                        return val
+                else:
+                    raise ValueError(
+                        f"Unexpected number of groups: expected 4, got {groups}"
+                    )
+
+            return subst
+
+        subst_func = make_subst_function(self, none_str)
+
+        # do the replacements
+        rendered = re.sub(regex, subst_func, template)
+
+        # do multi-valued placements
+        # start with the single string from phase 1 above then loop through all
+        # multi-valued fields and all values for each of those fields
+        # rendered_strings will be updated as each field is processed
+        # for example: if two albums, two keywords, and one person and template is:
+        # "{created.year}/{album}/{keyword}/{person}"
+        # rendered strings would do the following:
+        # start (created.year filled in phase 1)
+        #   ['2011/{album}/{keyword}/{person}']
+        # after processing albums:
+        #   ['2011/Album1/{keyword}/{person}',
+        #    '2011/Album2/{keyword}/{person}',]
+        # after processing keywords:
+        #   ['2011/Album1/keyword1/{person}',
+        #    '2011/Album1/keyword2/{person}',
+        #    '2011/Album2/keyword1/{person}',
+        #    '2011/Album2/keyword2/{person}',]
+        # after processing person:
+        #   ['2011/Album1/keyword1/person1',
+        #    '2011/Album1/keyword2/person1',
+        #    '2011/Album2/keyword1/person1',
+        #    '2011/Album2/keyword2/person1',]
+
+        rendered_strings = set([rendered])
+        for field in MULTI_VALUE_SUBSTITUTIONS:
+            if field == "album":
+                values = self.albums
+            elif field == "keyword":
+                values = self.keywords
+            elif field == "person":
+                values = self.persons
+                # remove any _UNKNOWN_PERSON values
+                values = [val for val in values if val != _UNKNOWN_PERSON]
+            elif field == "folder_album":
+                values = []
+                # photos must be in an album to be in a folder
+                for album in self.album_info:
+                    if album.folder_names:
+                        # album in folder
+                        folder = path_sep.join(album.folder_names)
+                        folder += path_sep + album.title
+                        values.append(folder)
+                    else:
+                        # album not in folder
+                        values.append(album.title)
+
+            else:
+                raise ValueError(f"Unhandleded template value: {field}")
+
+            # If no values, insert None so code below will substite none_str for None
+            values = values or [None]
+
+            # Build a regex that matches only the field being processed
+            re_str = r"(?<!\\)\{(" + field + r")(,{0,1}(([\w\-. ]+))?)\}"
+            regex_multi = re.compile(re_str)
+
+            # holds each of the new rendered_strings, set() to avoid duplicates
+            new_strings = set()
+
+            for str_template in rendered_strings:
+                for val in values:
+
+                    def get_template_value_multi(lookup_value):
+                        """ Closure passed to make_subst_function get_func 
+                            Capture val and field in the closure 
+                            Allows make_subst_function to be re-used w/o modification """
+                        if lookup_value == field:
+                            return val
+                        else:
+                            raise KeyError(f"Unexpected value: {lookup_value}")
+
+                    subst = make_subst_function(
+                        self, none_str, get_func=get_template_value_multi
+                    )
+                    new_string = regex_multi.sub(subst, str_template)
+                    new_strings.add(new_string)
+
+            # update rendered_strings for the next field to process
+            rendered_strings = new_strings
+
+        # find any {fields} that weren't replaced
+        unmatched = []
+        for rendered_str in rendered_strings:
+            unmatched.extend(
+                [
+                    no_match[0]
+                    for no_match in re.findall(regex, rendered_str)
+                    if no_match[0] not in unmatched
+                ]
+            )
+
+        # fix any escaped curly braces
+        rendered_strings = [
+            rendered_str.replace("{{", "{").replace("}}", "}")
+            for rendered_str in rendered_strings
+        ]
+
+        return rendered_strings, unmatched
+
+    def get_template_value(self, lookup):
+        """ lookup template value (single-value template substitutions) for use in make_subst_function
+            lookup: value to find a match for
+            returns: either the matching template value (which may be None)
+            raises: KeyError if no rule exists for lookup """
+
+        # must be a valid keyword
+        if lookup == "name":
+            return pathlib.Path(self.filename).stem
+
+        if lookup == "original_name":
+            return pathlib.Path(self.original_filename).stem
+
+        if lookup == "title":
+            return self.title
+
+        if lookup == "descr":
+            return self.description
+
+        if lookup == "created.date":
+            return DateTimeFormatter(self.date).date
+
+        if lookup == "created.year":
+            return DateTimeFormatter(self.date).year
+
+        if lookup == "created.yy":
+            return DateTimeFormatter(self.date).yy
+
+        if lookup == "created.mm":
+            return DateTimeFormatter(self.date).mm
+
+        if lookup == "created.month":
+            return DateTimeFormatter(self.date).month
+
+        if lookup == "created.mon":
+            return DateTimeFormatter(self.date).mon
+
+        if lookup == "created.doy":
+            return DateTimeFormatter(self.date).doy
+
+        if lookup == "modified.date":
+            return (
+                DateTimeFormatter(self.date_modified).date
+                if self.date_modified
+                else None
+            )
+
+        if lookup == "modified.year":
+            return (
+                DateTimeFormatter(self.date_modified).year
+                if self.date_modified
+                else None
+            )
+
+        if lookup == "modified.yy":
+            return (
+                DateTimeFormatter(self.date_modified).yy if self.date_modified else None
+            )
+
+        if lookup == "modified.mm":
+            return (
+                DateTimeFormatter(self.date_modified).mm if self.date_modified else None
+            )
+
+        if lookup == "modified.month":
+            return (
+                DateTimeFormatter(self.date_modified).month
+                if self.date_modified
+                else None
+            )
+
+        if lookup == "modified.mon":
+            return (
+                DateTimeFormatter(self.date_modified).mon
+                if self.date_modified
+                else None
+            )
+
+        if lookup == "modified.doy":
+            return (
+                DateTimeFormatter(self.date_modified).doy
+                if self.date_modified
+                else None
+            )
+
+        if lookup == "place.name":
+            return self.place.name if self.place else None
+
+        if lookup == "place.country_code":
+            return self.place.country_code if self.place else None
+
+        if lookup == "place.name.country":
+            return (
+                self.place.names.country[0]
+                if self.place and self.place.names.country
+                else None
+            )
+
+        if lookup == "place.name.state_province":
+            return (
+                self.place.names.state_province[0]
+                if self.place and self.place.names.state_province
+                else None
+            )
+
+        if lookup == "place.name.city":
+            return (
+                self.place.names.city[0]
+                if self.place and self.place.names.city
+                else None
+            )
+
+        if lookup == "place.name.area_of_interest":
+            return (
+                self.place.names.area_of_interest[0]
+                if self.place and self.place.names.area_of_interest
+                else None
+            )
+
+        if lookup == "place.address":
+            return (
+                self.place.address_str
+                if self.place and self.place.address_str
+                else None
+            )
+
+        if lookup == "place.address.street":
+            return (
+                self.place.address.street
+                if self.place and self.place.address.street
+                else None
+            )
+
+        if lookup == "place.address.city":
+            return (
+                self.place.address.city
+                if self.place and self.place.address.city
+                else None
+            )
+
+        if lookup == "place.address.state_province":
+            return (
+                self.place.address.state_province
+                if self.place and self.place.address.state_province
+                else None
+            )
+
+        if lookup == "place.address.postal_code":
+            return (
+                self.place.address.postal_code
+                if self.place and self.place.address.postal_code
+                else None
+            )
+
+        if lookup == "place.address.country":
+            return (
+                self.place.address.country
+                if self.place and self.place.address.country
+                else None
+            )
+
+        if lookup == "place.address.country_code":
+            return (
+                self.place.address.iso_country_code
+                if self.place and self.place.address.iso_country_code
+                else None
+            )
+
+        # if here, didn't get a match
+        raise KeyError(f"No rule for processing {lookup}")
+
+    def _write_exif_data(
+        self,
+        filepath,
+        use_albums_as_keywords=False,
+        use_persons_as_keywords=False,
+        keyword_template=None,
+    ):
         """ write exif data to image file at filepath
         filepath: full path to the image file """
         if not os.path.exists(filepath):
             raise FileNotFoundError(f"Could not find file {filepath}")
         exiftool = ExifTool(filepath)
-        exif_info = json.loads(self._exiftool_json_sidecar())[0]
+        exif_info = json.loads(
+            self._exiftool_json_sidecar(
+                use_albums_as_keywords=use_albums_as_keywords,
+                use_persons_as_keywords=use_persons_as_keywords,
+                keyword_template=keyword_template,
+            )
+        )[0]
         for exiftag, val in exif_info.items():
             if type(val) == list:
                 # more than one, set first value the add additional values
@@ -897,16 +1280,24 @@ class PhotoInfo:
             else:
                 exiftool.setvalue(exiftag, val)
 
-    def _exiftool_json_sidecar(self):
+    def _exiftool_json_sidecar(
+        self,
+        use_albums_as_keywords=False,
+        use_persons_as_keywords=False,
+        keyword_template=None,
+    ):
         """ return json string of EXIF details in exiftool sidecar format
             Does not include all the EXIF fields as those are likely already in the image
+            use_albums_as_keywords: treat album names as keywords
+            use_persons_as_keywords: treat person names as keywords
+            keyword_template: (list of strings); list of template strings to render as keywords
             Exports the following:
                 FileName
                 ImageDescription
                 Description
                 Title
                 TagsList
-                Keywords
+                Keywords (may include album name, person name, or template)
                 Subject
                 PersonInImage
                 GPSLatitude, GPSLongitude
@@ -926,18 +1317,64 @@ class PhotoInfo:
         if self.title:
             exif["XMP:Title"] = self.title
 
+        keyword_list = []
         if self.keywords:
-            exif["XMP:TagsList"] = exif["IPTC:Keywords"] = list(self.keywords)
-            # Photos puts both keywords and persons in Subject when using "Export IPTC as XMP"
-            exif["XMP:Subject"] = list(self.keywords)
+            keyword_list.extend(self.keywords)
 
+        person_list = []
         if self.persons:
-            exif["XMP:PersonInImage"] = self.persons
+            # filter out _UNKNOWN_PERSON
+            person_list = [p for p in self.persons if p != _UNKNOWN_PERSON]
+
+        if use_persons_as_keywords and person_list:
+            keyword_list.extend(person_list)
+
+        if use_albums_as_keywords and self.albums:
+            keyword_list.extend(self.albums)
+
+        if keyword_template:
+            rendered_keywords = []
+            for template_str in keyword_template:
+                rendered, unmatched = self.render_template(
+                    template_str, none_str=_OSXPHOTOS_NONE_SENTINEL, path_sep="/"
+                )
+                if unmatched:
+                    logging.warning(
+                        f"Unmatched template substitution for template: {template_str} {unmatched}"
+                    )
+                rendered_keywords.extend(rendered)
+
+            # filter out any template values that didn't match by looking for sentinel
+            rendered_keywords = [
+                keyword
+                for keyword in rendered_keywords
+                if _OSXPHOTOS_NONE_SENTINEL not in keyword
+            ]
+
+            # check to see if any keywords too long
+            long_keywords = [
+                long_str
+                for long_str in rendered_keywords
+                if len(long_str) > _MAX_IPTC_KEYWORD_LEN
+            ]
+            if long_keywords:
+                logging.warning(
+                    f"Some keywords exceed max IPTC Keyword length of {_MAX_IPTC_KEYWORD_LEN}: {long_keywords}"
+                )
+
+            logging.debug(f"rendered_keywords: {rendered_keywords}")
+            keyword_list.extend(rendered_keywords)
+
+        if keyword_list:
+            exif["XMP:TagsList"] = exif["IPTC:Keywords"] = keyword_list
+
+        if person_list:
+            exif["XMP:PersonInImage"] = person_list
+
+        if self.keywords or person_list:
             # Photos puts both keywords and persons in Subject when using "Export IPTC as XMP"
-            if "XMP:Subject" in exif:
-                exif["XMP:Subject"].extend(self.persons)
-            else:
-                exif["XMP:Subject"] = self.persons
+            # only use Photos' keywords for subject
+            exif["XMP:Subject"] = list(self.keywords) + person_list
 
         # if self.favorite():
         #     exif["Rating"] = 5
@@ -971,14 +1408,86 @@ class PhotoInfo:
         json_str = json.dumps([exif])
         return json_str
 
-    def _xmp_sidecar(self):
-        """ returns string for XMP sidecar """
+    def _xmp_sidecar(
+        self,
+        use_albums_as_keywords=False,
+        use_persons_as_keywords=False,
+        keyword_template=None,
+    ):
+        """ returns string for XMP sidecar 
+            use_albums_as_keywords: treat album names as keywords
+            use_persons_as_keywords: treat person names as keywords
+            keyword_template: (list of strings); list of template strings to render as keywords """
+
         # TODO: add additional fields to XMP file?
 
         xmp_template = Template(
             filename=os.path.join(_TEMPLATE_DIR, _XMP_TEMPLATE_NAME)
         )
-        xmp_str = xmp_template.render(photo=self)
+
+        keyword_list = []
+        if self.keywords:
+            keyword_list.extend(self.keywords)
+
+        # TODO: keyword handling in this and _exiftool_json_sidecar is
+        # good candidate for pulling out in a function
+
+        person_list = []
+        if self.persons:
+            # filter out _UNKNOWN_PERSON
+            person_list = [p for p in self.persons if p != _UNKNOWN_PERSON]
+
+        if use_persons_as_keywords and person_list:
+            keyword_list.extend(person_list)
+
+        if use_albums_as_keywords and self.albums:
+            keyword_list.extend(self.albums)
+
+        if keyword_template:
+            rendered_keywords = []
+            for template_str in keyword_template:
+                rendered, unmatched = self.render_template(
+                    template_str, none_str=_OSXPHOTOS_NONE_SENTINEL, path_sep="/"
+                )
+                if unmatched:
+                    logging.warning(
+                        f"Unmatched template substitution for template: {template_str} {unmatched}"
+                    )
+                rendered_keywords.extend(rendered)
+
+            # filter out any template values that didn't match by looking for sentinel
+            rendered_keywords = [
+                keyword
+                for keyword in rendered_keywords
+                if _OSXPHOTOS_NONE_SENTINEL not in keyword
+            ]
+
+            # check to see if any keywords too long
+            long_keywords = [
+                long_str
+                for long_str in rendered_keywords
+                if len(long_str) > _MAX_IPTC_KEYWORD_LEN
+            ]
+            if long_keywords:
+                logging.warning(
+                    f"Some keywords exceed max IPTC Keyword length of {_MAX_IPTC_KEYWORD_LEN}: {long_keywords}"
+                )
+
+            logging.debug(f"rendered_keywords: {rendered_keywords}")
+            keyword_list.extend(rendered_keywords)
+
+        subject_list = []
+        if self.keywords or person_list:
+            # Photos puts both keywords and persons in Subject when using "Export IPTC as XMP"
+            subject_list = list(self.keywords) + person_list
+
+        xmp_str = xmp_template.render(
+            photo=self,
+            keywords=keyword_list,
+            persons=person_list,
+            subjects=subject_list,
+        )
+
         # remove extra lines that mako inserts from template
         xmp_str = "\n".join(
             [line for line in xmp_str.split("\n") if line.strip() != ""]

osxphotos/photosdb.py

@@ -19,17 +19,18 @@ from ._constants import (
     _MOVIE_TYPE,
     _PHOTO_TYPE,
     _PHOTOS_3_VERSION,
+    _PHOTOS_4_ALBUM_KIND,
+    _PHOTOS_4_ROOT_FOLDER,
+    _PHOTOS_4_TOP_LEVEL_ALBUM,
     _PHOTOS_4_VERSION,
+    _PHOTOS_5_ALBUM_KIND,
+    _PHOTOS_5_FOLDER_KIND,
+    _PHOTOS_5_ROOT_FOLDER_KIND,
+    _PHOTOS_5_SHARED_ALBUM_KIND,
     _PHOTOS_5_VERSION,
     _TESTED_DB_VERSIONS,
     _TESTED_OS_VERSIONS,
     _UNKNOWN_PERSON,
-    _PHOTOS_4_ALBUM_KIND,
-    _PHOTOS_4_TOP_LEVEL_ALBUM,
-    _PHOTOS_5_ROOT_FOLDER_KIND,
-    _PHOTOS_5_FOLDER_KIND,
-    _PHOTOS_5_ALBUM_KIND,
-    _PHOTOS_5_SHARED_ALBUM_KIND,
 )
 from ._version import __version__
 from .albuminfo import AlbumInfo, FolderInfo
@@ -77,6 +78,12 @@ class PhotosDB:
 
         # set up the data structures used to store all the Photo database info
 
+        # if True, will treat persons as keywords when exporting metadata
+        self.use_persons_as_keywords = False
+
+        # if True, will treat albums as keywords when exporting metadata
+        self.use_albums_as_keywords = False
+
         # Path to the Photos library database file
         # photos.db in the photos library database/ directory
         self._dbfile = None
@@ -240,7 +247,7 @@ class PhotosDB:
         self._db_version = self._get_db_version()
 
         # If Photos >= 5, actual data isn't in photos.db but in Photos.sqlite
-        if int(self._db_version) >= int(_PHOTOS_5_VERSION):
+        if int(self._db_version) > int(_PHOTOS_4_VERSION):
             dbpath = pathlib.Path(self._dbfile).parent
             dbfile = dbpath / "Photos.sqlite"
             if not _check_file_exists(dbfile):
@@ -259,7 +266,7 @@ class PhotosDB:
         library_path = os.path.dirname(os.path.abspath(dbfile))
         (library_path, _) = os.path.split(library_path)  # drop /database from path
         self._library_path = library_path
-        if int(self._db_version) < int(_PHOTOS_5_VERSION):
+        if int(self._db_version) <= int(_PHOTOS_4_VERSION):
             masters_path = os.path.join(library_path, "Masters")
             self._masters_path = masters_path
         else:
@@ -528,7 +535,6 @@ class PhotosDB:
         """ process the Photos database to extract info
             works on Photos version <= 4.0 """
 
-        # TODO: Update strings to remove + (not needed)
         # Epoch is Jan 1, 2001
         td = (datetime(2001, 1, 1, 0, 0) - datetime(1970, 1, 1, 0, 0)).total_seconds()
 
@@ -656,8 +662,11 @@ class PhotosDB:
         # build folder hierarchy
         for album, details in self._dbalbum_details.items():
             parent_folder = details["folderUuid"]
-            if parent_folder != _PHOTOS_4_TOP_LEVEL_ALBUM:
-                # logging.warning(f"album = {details['title']}, parent = {parent_folder}")
+            if details[
+                "albumSubclass"
+            ] == _PHOTOS_4_ALBUM_KIND and parent_folder not in [
+                _PHOTOS_4_TOP_LEVEL_ALBUM
+            ]:
                 folder_hierarchy = self._build_album_folder_hierarchy_4(parent_folder)
                 self._dbalbum_folders[album] = folder_hierarchy
             else:
@@ -888,6 +897,7 @@ class PhotosDB:
             # TODO: NOT YET USED -- PLACEHOLDER for RAW processing (currently only in _process_database5)
             # original resource choice (e.g. RAW or jpeg)
             self._dbphotos[uuid]["original_resource_choice"] = None
+            self._dbphotos[uuid]["raw_is_original"] = None
 
             # associated RAW image info
             self._dbphotos[uuid]["has_raw"] = True if row[25] == 7 else False
@@ -903,17 +913,17 @@ class PhotosDB:
         # get additional details from RKMaster, needed for RAW processing
         c.execute(
             """ SELECT 
-	            RKMaster.uuid,
+                RKMaster.uuid,
                 RKMaster.volumeId, 
                 RKMaster.imagePath, 
-	            RKMaster.isMissing, 
+                RKMaster.isMissing, 
                 RKMaster.originalFileName, 
                 RKMaster.UTI,
-	            RKMaster.modelID, 
+                RKMaster.modelID, 
                 RKMaster.fileSize, 
                 RKMaster.isTrulyRaw,
-	            RKMaster.alternateMasterUuid
-	            FROM RKMaster
+                RKMaster.alternateMasterUuid
+                FROM RKMaster
             """
         )
 
@@ -1214,17 +1224,24 @@ class PhotosDB:
 
     def _build_album_folder_hierarchy_4(self, uuid, folders=None):
         """ recursively build folder/album hierarchy
-            uuid: uuid of the album/folder being processed
-            folders: dict holding the folder hierarchy """
+            uuid: parent uuid of the album being processed 
+                 (parent uuid is a folder in RKFolders)
+            folders: dict holding the folder hierarchy 
+            NOTE: This implementation is different than _build_album_folder_hierarchy_5 
+            which takes the uuid of the album being processed.  Here uuid is the parent uuid
+            of the parent folder album because in Photos <=4, folders are in RKFolders and 
+            albums in RKAlbums.  In Photos 5, folders are just special albums 
+            with kind = _PHOTOS_5_FOLDER_KIND """
 
         parent_uuid = self._dbfolder_details[uuid]["parentFolderUuid"]
 
-        # logging.warning(f"uuid = {uuid}, parent = {parent_uuid}, folders = {folders}")
-
         if parent_uuid is None:
             return folders
 
         if parent_uuid == _PHOTOS_4_TOP_LEVEL_ALBUM:
+            if not folders:
+                # this is a top-level folder with no sub-folders
+                folders = {uuid: None}
             # at top of hierarchy, we're done
             return folders
 
@@ -1607,7 +1624,12 @@ class PhotosDB:
             info["momentID"] = row[26]
 
             # original resource choice (e.g. RAW or jpeg)
+            # for images part of a RAW/jpeg pair,
+            # ZADDITIONALASSETATTRIBUTES.ZORIGINALRESOURCECHOICE
+            # = 0 if jpeg is selected as "original" in Photos (the default)
+            # = 1 if RAW is selected as "original" in Photos
             info["original_resource_choice"] = row[27]
+            info["raw_is_original"] = True if row[27] == 1 else False
 
             # associated RAW image info
             # will be filled in later
@@ -1878,7 +1900,7 @@ class PhotosDB:
             # folder with no parent (e.g. shared iCloud folders)
             return folders
 
-        if self._db_version >= _PHOTOS_5_VERSION and parent == self._folder_root_pk:
+        if self._db_version > _PHOTOS_4_VERSION and parent == self._folder_root_pk:
             # at the top of the folder hierarchy, we're done
             return folders
 
@@ -2156,3 +2178,7 @@ class PhotosDB:
             return self.__dict__ == other.__dict__
 
         return False
+
+    def __len__(self):
+        """ returns number of photos in the database """
+        return len(self._dbphotos)

osxphotos/template.py

@@ -1,4 +1,4 @@
-""" Custom template system for osxphotos """
+""" Custom template system for osxphotos (implemented in PhotoInfo.render_template) """
 
 # Rolled my own template system because:
 # 1. Needed to handle multiple values (e.g. album, keyword)
@@ -9,18 +9,14 @@
 #
 # This code isn't elegant but it seems to work well.  PRs gladly accepted.
 
-import datetime
-import os
-import pathlib
-import re
-from typing import Tuple, List  # pylint: disable=syntax-error
+import locale
 
-from .photoinfo import PhotoInfo
-from ._constants import _UNKNOWN_PERSON
+# ensure locale set to user's locale
+locale.setlocale(locale.LC_ALL, "")
 
 # Permitted substitutions (each of these returns a single value or None)
 TEMPLATE_SUBSTITUTIONS = {
-    "{name}": "Filename of the photo",
+    "{name}": "Current filename of the photo",
     "{original_name}": "Photo's original filename when imported to Photos",
     "{title}": "Title of the photo",
     "{descr}": "Description of the photo",
@@ -66,374 +62,3 @@ MULTI_VALUE_SUBSTITUTIONS = [
     field.replace("{", "").replace("}", "")
     for field in TEMPLATE_SUBSTITUTIONS_MULTI_VALUED.keys()
 ]
-
-
-def get_template_value(lookup, photo):
-    """ lookup template value (single-value template substitutions) for use in make_subst_function
-        lookup: value to find a match for
-        photo: PhotoInfo object whose data will be used for value substitutions
-        returns: either the matching template value (which may be None)
-        raises: KeyError if no rule exists for lookup """
-
-    # must be a valid keyword
-    if lookup == "name":
-        return pathlib.Path(photo.filename).stem
-
-    if lookup == "original_name":
-        return pathlib.Path(photo.original_filename).stem
-
-    if lookup == "title":
-        return photo.title
-
-    if lookup == "descr":
-        return photo.description
-
-    if lookup == "created.date":
-        return DateTimeFormatter(photo.date).date
-
-    if lookup == "created.year":
-        return DateTimeFormatter(photo.date).year
-
-    if lookup == "created.yy":
-        return DateTimeFormatter(photo.date).yy
-
-    if lookup == "created.mm":
-        return DateTimeFormatter(photo.date).mm
-
-    if lookup == "created.month":
-        return DateTimeFormatter(photo.date).month
-
-    if lookup == "created.mon":
-        return DateTimeFormatter(photo.date).mon
-
-    if lookup == "created.doy":
-        return DateTimeFormatter(photo.date).doy
-
-    if lookup == "modified.date":
-        return (
-            DateTimeFormatter(photo.date_modified).date if photo.date_modified else None
-        )
-
-    if lookup == "modified.year":
-        return (
-            DateTimeFormatter(photo.date_modified).year if photo.date_modified else None
-        )
-
-    if lookup == "modified.yy":
-        return (
-            DateTimeFormatter(photo.date_modified).yy if photo.date_modified else None
-        )
-
-    if lookup == "modified.mm":
-        return (
-            DateTimeFormatter(photo.date_modified).mm if photo.date_modified else None
-        )
-
-    if lookup == "modified.month":
-        return (
-            DateTimeFormatter(photo.date_modified).month
-            if photo.date_modified
-            else None
-        )
-
-    if lookup == "modified.mon":
-        return (
-            DateTimeFormatter(photo.date_modified).mon if photo.date_modified else None
-        )
-
-    if lookup == "modified.doy":
-        return (
-            DateTimeFormatter(photo.date_modified).doy if photo.date_modified else None
-        )
-
-    if lookup == "place.name":
-        return photo.place.name if photo.place else None
-
-    if lookup == "place.country_code":
-        return photo.place.country_code if photo.place else None
-
-    if lookup == "place.name.country":
-        return (
-            photo.place.names.country[0]
-            if photo.place and photo.place.names.country
-            else None
-        )
-
-    if lookup == "place.name.state_province":
-        return (
-            photo.place.names.state_province[0]
-            if photo.place and photo.place.names.state_province
-            else None
-        )
-
-    if lookup == "place.name.city":
-        return (
-            photo.place.names.city[0]
-            if photo.place and photo.place.names.city
-            else None
-        )
-
-    if lookup == "place.name.area_of_interest":
-        return (
-            photo.place.names.area_of_interest[0]
-            if photo.place and photo.place.names.area_of_interest
-            else None
-        )
-
-    if lookup == "place.address":
-        return (
-            photo.place.address_str if photo.place and photo.place.address_str else None
-        )
-
-    if lookup == "place.address.street":
-        return (
-            photo.place.address.street
-            if photo.place and photo.place.address.street
-            else None
-        )
-
-    if lookup == "place.address.city":
-        return (
-            photo.place.address.city
-            if photo.place and photo.place.address.city
-            else None
-        )
-
-    if lookup == "place.address.state_province":
-        return (
-            photo.place.address.state_province
-            if photo.place and photo.place.address.state_province
-            else None
-        )
-
-    if lookup == "place.address.postal_code":
-        return (
-            photo.place.address.postal_code
-            if photo.place and photo.place.address.postal_code
-            else None
-        )
-
-    if lookup == "place.address.country":
-        return (
-            photo.place.address.country
-            if photo.place and photo.place.address.country
-            else None
-        )
-
-    if lookup == "place.address.country_code":
-        return (
-            photo.place.address.iso_country_code
-            if photo.place and photo.place.address.iso_country_code
-            else None
-        )
-
-    # if here, didn't get a match
-    raise KeyError(f"No rule for processing {lookup}")
-
-
-def render_filepath_template(template, photo, none_str="_"):
-    """ render a filename or directory template 
-        template: str template 
-        photo: PhotoInfo object
-        none_str: str to use default for None values, default is '_' """
-
-    # the rendering happens in two phases:
-    # phase 1: handle all the single-value template substitutions
-    #          results in a single string with all the template fields replaced
-    # phase 2: loop through all the multi-value template substitutions
-    #          could result in multiple strings
-    #          e.g. if template is "{album}/{person}" and there are 2 albums and 3 persons in the photo
-    #          there would be 6 possible renderings (2 albums x 3 persons)
-
-    # regex to find {template_field,optional_default} in strings
-    # for explanation of regex see https://regex101.com/r/4JJg42/1
-    # pylint: disable=anomalous-backslash-in-string
-    regex = r"(?<!\{)\{([^\\,}]+)(,{0,1}(([\w\-. ]+))?)(?=\}(?!\}))\}"
-
-    if type(template) is not str:
-        raise TypeError(f"template must be type str, not {type(template)}")
-
-    if type(photo) is not PhotoInfo:
-        raise TypeError(f"photo must be type osxphotos.PhotoInfo, not {type(photo)}")
-
-    def make_subst_function(photo, none_str, get_func=get_template_value):
-        """ returns: substitution function for use in re.sub 
-            photo: a PhotoInfo object
-            none_str: value to use if substitution lookup is None and no default provided
-            get_func: function that gets the substitution value for a given template field
-                      default is get_template_value which handles the single-value fields """
-
-        # closure to capture photo, none_str in subst
-        def subst(matchobj):
-            groups = len(matchobj.groups())
-            if groups == 4:
-                try:
-                    val = get_func(matchobj.group(1), photo)
-                except KeyError:
-                    return matchobj.group(0)
-
-                if val is None:
-                    return (
-                        matchobj.group(3) if matchobj.group(3) is not None else none_str
-                    )
-                else:
-                    return val
-            else:
-                raise ValueError(
-                    f"Unexpected number of groups: expected 4, got {groups}"
-                )
-
-        return subst
-
-    subst_func = make_subst_function(photo, none_str)
-
-    # do the replacements
-    rendered = re.sub(regex, subst_func, template)
-
-    # do multi-valued placements
-    # start with the single string from phase 1 above then loop through all
-    # multi-valued fields and all values for each of those fields
-    # rendered_strings will be updated as each field is processed
-    # for example: if two albums, two keywords, and one person and template is:
-    # "{created.year}/{album}/{keyword}/{person}"
-    # rendered strings would do the following:
-    # start (created.year filled in phase 1)
-    #   ['2011/{album}/{keyword}/{person}']
-    # after processing albums:
-    #   ['2011/Album1/{keyword}/{person}',
-    #    '2011/Album2/{keyword}/{person}',]
-    # after processing keywords:
-    #   ['2011/Album1/keyword1/{person}',
-    #    '2011/Album1/keyword2/{person}',
-    #    '2011/Album2/keyword1/{person}',
-    #    '2011/Album2/keyword2/{person}',]
-    # after processing person:
-    #   ['2011/Album1/keyword1/person1',
-    #    '2011/Album1/keyword2/person1',
-    #    '2011/Album2/keyword1/person1',
-    #    '2011/Album2/keyword2/person1',]
-
-    rendered_strings = set([rendered])
-    for field in MULTI_VALUE_SUBSTITUTIONS:
-        if field == "album":
-            values = photo.albums
-        elif field == "keyword":
-            values = photo.keywords
-        elif field == "person":
-            values = photo.persons
-            # remove any _UNKNOWN_PERSON values
-            values = [val for val in values if val != _UNKNOWN_PERSON]
-        elif field == "folder_album":
-            values = []
-            # photos must be in an album to be in a folder
-            for album in photo.album_info:
-                if album.folder_names:
-                    # album in folder
-                    folder = os.path.sep.join(album.folder_names)
-                    folder += os.path.sep + album.title
-                    values.append(folder)
-                else:
-                    # album not in folder
-                    values.append(album.title)
-
-        else:
-            raise ValueError(f"Unhandleded template value: {field}")
-
-        # If no values, insert None so code below will substite none_str for None
-        values = values or [None]
-
-        # Build a regex that matches only the field being processed
-        re_str = r"(?<!\\)\{(" + field + r")(,{0,1}(([\w\-. ]+))?)\}"
-        regex_multi = re.compile(re_str)
-
-        # holds each of the new rendered_strings, set() to avoid duplicates
-        new_strings = set()
-
-        for str_template in rendered_strings:
-            for val in values:
-
-                def get_template_value_multi(lookup_value, photo):
-                    """ Closure passed to make_subst_function get_func 
-                        Capture val and field in the closure 
-                        Allows make_subst_function to be re-used w/o modification """
-                    if lookup_value == field:
-                        return val
-                    else:
-                        raise KeyError(f"Unexpected value: {lookup_value}")
-
-                subst = make_subst_function(
-                    photo, none_str, get_func=get_template_value_multi
-                )
-                new_string = regex_multi.sub(subst, str_template)
-                new_strings.add(new_string)
-
-        # update rendered_strings for the next field to process
-        rendered_strings = new_strings
-
-    # find any {fields} that weren't replaced
-    unmatched = []
-    for rendered_str in rendered_strings:
-        unmatched.extend(
-            [
-                no_match[0]
-                for no_match in re.findall(regex, rendered_str)
-                if no_match[0] not in unmatched
-            ]
-        )
-
-    # fix any escaped curly braces
-    rendered_strings = [
-        rendered_str.replace("{{", "{").replace("}}", "}")
-        for rendered_str in rendered_strings
-    ]
-
-    return rendered_strings, unmatched
-
-
-class DateTimeFormatter:
-    """ provides property access to formatted datetime.datetime strftime values """
-
-    def __init__(self, dt: datetime.datetime):
-        self.dt = dt
-
-    @property
-    def date(self):
-        """ ISO date in form 2020-03-22 """
-        date = self.dt.date().isoformat()
-        return date
-
-    @property
-    def year(self):
-        """ 4 digit year """
-        year = f"{self.dt.year}"
-        return year
-
-    @property
-    def yy(self):
-        """ 2 digit year """
-        yy = f"{self.dt.strftime('%y')}"
-        return yy
-
-    @property
-    def mm(self):
-        """ 2 digit month """
-        mm = f"{self.dt.strftime('%m')}"
-        return mm
-
-    @property
-    def month(self):
-        """ Month as locale's full name """
-        month = f"{self.dt.strftime('%B')}"
-        return month
-
-    @property
-    def mon(self):
-        """ Month as locale's abbreviated name """
-        mon = f"{self.dt.strftime('%b')}"
-        return mon
-
-    @property
-    def doy(self):
-        """ Julian day of year starting from 001 """
-        doy = f"{self.dt.strftime('%j')}"
-        return doy

osxphotos/templates/xmp_sidecar.mako

@@ -79,16 +79,16 @@
             xmlns:photoshop="http://ns.adobe.com/photoshop/1.0/">
             ${dc_description(photo.description)}
             ${dc_title(photo.title)}
-            ${dc_subject(photo.keywords + photo.persons)}
+            ${dc_subject(subjects)}
             ${dc_datecreated(photo.date)}
         </rdf:Description>
         <rdf:Description rdf:about='' 
             xmlns:Iptc4xmpExt='http://iptc.org/std/Iptc4xmpExt/2008-02-29/'>
-            ${iptc_personinimage(photo.persons)}
+            ${iptc_personinimage(persons)}
         </rdf:Description>
         <rdf:Description rdf:about='' 
             xmlns:digiKam='http://www.digikam.org/ns/1.0/'>
-            ${dk_tagslist(photo.keywords)}
+            ${dk_tagslist(keywords)}
         </rdf:Description>
         <rdf:Description rdf:about='' 
             xmlns:xmp='http://ns.adobe.com/xap/1.0/'>

setup.py

@@ -3,7 +3,7 @@
 #
 # setup.py script for osxphotos
 #
-# Copyright (c) 2019 Rhet Turnbull, rturnbull+git@gmail.com
+# Copyright (c) 2019, 2020 Rhet Turnbull, rturnbull+git@gmail.com
 # All rights reserved.
 #
 # Permission is hereby granted, free of charge, to any person
@@ -27,23 +27,43 @@
 # SOFTWARE.
 
 import os
+import platform
+
 from setuptools import find_packages, setup
 
-this_directory = os.path.abspath(os.path.dirname(__file__))
-with open(os.path.join(this_directory, "README.md"), encoding="utf-8") as f:
-    long_description = f.read()
+# python version as 2-digit float (e.g. 3.6)
+py_ver = float(".".join(platform.python_version_tuple()[:2]))
 
+# holds config info read from disk
 about = {}
+this_directory = os.path.abspath(os.path.dirname(__file__))
+
+# get version info from _version
 with open(
     os.path.join(this_directory, "osxphotos", "_version.py"), mode="r", encoding="utf-8"
 ) as f:
     exec(f.read(), about)
 
+# read README.md into long_description
+with open(os.path.join(this_directory, "README.md"), encoding="utf-8") as f:
+    about["long_description"] = f.read()
+
+# ugly hack to install custom version of bpylist2 needed for Python < 3.8
+# the stock version of bylist2==2.0.3 causes an error related to
+# "pkg_resources.ContextualVersionConflict: (pycodestyle 2.3.1..."
+# PEP 508 no help here as URL-based lookups not allowed in PyPI packages
+# if you know a better way, PRs welcome!
+# once I go to 3.8+ required, this won't be necessary as bpylist2 3.0+ solves this issue
+if py_ver < 3.8:
+    os.system(
+        "python3 -m pip install git+git://github.com/RhetTbull/bpylist2.git#egg=bpylist2"
+    )
+
 setup(
     name="osxphotos",
     version=about["__version__"],
     description="Manipulate (read-only) Apple's Photos app library on Mac OS X",
-    long_description=long_description,
+    long_description=about["long_description"],
     long_description_content_type="text/markdown",
     author="Rhet Turnbull",
     author_email="rturnbull+git@gmail.com",
@@ -58,7 +78,7 @@ setup(
         "Intended Audience :: Developers",
         "License :: OSI Approved :: MIT License",
         "Operating System :: MacOS :: MacOS X",
-        "Programming Language :: Python :: 3.6",
+        "Programming Language :: Python :: 3.8",
         "Topic :: Software Development :: Libraries :: Python Modules",
     ],
     install_requires=[

tests/README.md

@@ -9,6 +9,13 @@ To run the tests, do the following from the main source folder:
 
 Running the tests this way allows the library to be tested without installing it.
 
+## Skipped Tests ##
+A few tests will look for certain environment variables to determine if they should run.
+
+Some of the export tests rely on photos in my local library and will look for `OSXPHOTOS_TEST_EXPORT=1` to determine if they should run.
+
+One test for locale does not run on GitHub's automated workflow and will look for `OSXPHOTOS_TEST_LOCALE=1` to determine if it should be run.  If you want to run this test, set the environment variable.  
+
 ## Attribution ##
 These tests utilize a test Photos library. The test library is populated with photos from [flickr](https://www.flickr.com).  All images used are licensed under Creative Commons 2.0 Attribution [license](https://creativecommons.org/licenses/by/2.0/).  
 

tests/Test-10.13.6.photoslibrary/private/com.apple.Photos/appPrivateData.plist

@@ -8,8 +8,11 @@
 		<array/>
 		<key>ExpandedSidebarItemIdentifiers</key>
 		<array>
+			<string>obfeGcvoT1auxoh2Tu86OQ</string>
 			<string>TopLevelAlbums</string>
 			<string>TopLevelSlideshows</string>
+			<string>MBS8+gBrQCWQxmcav+C8HQ</string>
+			<string>cHwwVoUiQ8a2nZNXgVsnCA</string>
 		</array>
 		<key>IPXWorkspaceControllerZoomLevelsKey</key>
 		<dict>
@@ -23,11 +26,11 @@
 			<key>key</key>
 			<integer>1</integer>
 			<key>lastKnownDisplayName</key>
-			<string>September 28, 2018</string>
+			<string>Pumpkin Farm (1)</string>
 			<key>type</key>
 			<string>album</string>
 			<key>uuid</key>
-			<string>+Ep8CrNRRhea9eVA618FMg</string>
+			<string>AU8Gp8bwRlOvngZFgwXBdg</string>
 		</dict>
 		<key>lastKnownItemCounts</key>
 		<dict>

tests/Test-10.13.6.photoslibrary/private/com.apple.photoanalysisd/GraphService/PhotoAnalysisServicePreferences.plist

@@ -3,8 +3,8 @@
 <plist version="1.0">
 <dict>
 	<key>PhotoAnalysisGraphLastBackgroundGraphRebuildJobDate</key>
-	<date>2019-07-28T01:23:52Z</date>
+	<date>2020-04-30T14:09:38Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundMemoryGenerationJobDate</key>
-	<date>2019-07-28T01:23:52Z</date>
+	<date>2020-05-01T04:27:48Z</date>
 </dict>
 </plist>

tests/Test-10.13.6.photoslibrary/private/com.apple.photoanalysisd/VisionService/PersonPromoter

@@ -2,7 +2,11 @@
 <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
 <plist version="1.0">
 <dict>
+	<key>ProcessedInQuiescentState</key>
+	<true/>
 	<key>SuggestedMeIdentifier</key>
 	<string></string>
+	<key>Version</key>
+	<integer>3</integer>
 </dict>
 </plist>

tests/Test-10.13.6.photoslibrary/private/com.apple.photomodel/appPrivateData.plist

@@ -5,7 +5,7 @@
 	<key>LithiumMessageTracer</key>
 	<dict>
 		<key>LastReportedDate</key>
-		<date>2019-07-27T12:01:15Z</date>
+		<date>2020-05-01T03:34:50Z</date>
 	</dict>
 </dict>
 </plist>

tests/Test-10.13.6.photoslibrary/resources/moments/analysismetadata.plist

@@ -11,6 +11,6 @@
 	<key>PLLastRevGeoForcedProviderOutOfDateCheckVersionKey</key>
 	<integer>1</integer>
 	<key>PLLastRevGeoVerFileFetchDateKey</key>
-	<date>2019-07-26T20:15:18Z</date>
+	<date>2020-04-30T12:51:41Z</date>
 </dict>
 </plist>

tests/Test-10.13.6.photoslibrary/resources/moments/historicalmarker.plist

@@ -3,7 +3,7 @@
 <plist version="1.0">
 <dict>
 	<key>LastHistoryRowId</key>
-	<integer>615</integer>
+	<integer>670</integer>
 	<key>LibraryBuildTag</key>
 	<string>BEA5F0E8-BA6B-4462-8F73-3E53BBE4C943</string>
 	<key>LibrarySchemaVersion</key>

tests/Test-10.13.6.photoslibrary/resources/recovery/Info.plist

@@ -9,7 +9,7 @@
 	<key>HistoricalMarker</key>
 	<dict>
 		<key>LastHistoryRowId</key>
-		<integer>615</integer>
+		<integer>670</integer>
 		<key>LibraryBuildTag</key>
 		<string>BEA5F0E8-BA6B-4462-8F73-3E53BBE4C943</string>
 		<key>LibrarySchemaVersion</key>
@@ -24,7 +24,7 @@
 	<key>SnapshotCompletedDate</key>
 	<date>2019-07-26T20:15:17Z</date>
 	<key>SnapshotLastValidated</key>
-	<date>2019-07-27T12:01:15Z</date>
+	<date>2020-05-01T03:34:50Z</date>
 	<key>SnapshotTables</key>
 	<dict/>
 </dict>

tests/Test-10.14.6.photoslibrary/private/com.apple.photoanalysisd/GraphService/PhotoAnalysisServicePreferences.plist

@@ -3,8 +3,8 @@
 <plist version="1.0">
 <dict>
 	<key>PhotoAnalysisGraphLastBackgroundGraphRebuildJobDate</key>
-	<date>2020-04-18T18:01:02Z</date>
+	<date>2020-04-25T23:54:43Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundMemoryGenerationJobDate</key>
-	<date>2020-04-18T17:22:55Z</date>
+	<date>2020-04-26T06:26:10Z</date>
 </dict>
 </plist>

tests/Test-10.14.6.photoslibrary/resources/moments/analysismetadata.plist

@@ -11,6 +11,6 @@
 	<key>PLLastRevGeoForcedProviderOutOfDateCheckVersionKey</key>
 	<integer>1</integer>
 	<key>PLLastRevGeoVerFileFetchDateKey</key>
-	<date>2020-04-17T17:49:52Z</date>
+	<date>2020-04-25T23:54:29Z</date>
 </dict>
 </plist>

tests/Test-10.14.6.photoslibrary/resources/recovery/Info.plist

@@ -24,7 +24,7 @@
 	<key>SnapshotCompletedDate</key>
 	<date>2019-07-27T13:16:43Z</date>
 	<key>SnapshotLastValidated</key>
-	<date>2020-04-17T17:51:16Z</date>
+	<date>2020-04-25T23:56:35Z</date>
 	<key>SnapshotTables</key>
 	<dict/>
 </dict>

tests/Test-10.15.1.photoslibrary/database/Photos.sqlite.lock

@@ -7,7 +7,7 @@
 	<key>hostuuid</key>
 	<string>9575E48B-8D5F-5654-ABAC-4431B1167324</string>
 	<key>pid</key>
-	<integer>900</integer>
+	<integer>725</integer>
 	<key>processname</key>
 	<string>photolibraryd</string>
 	<key>uid</key>

tests/Test-10.15.1.photoslibrary/private/com.apple.photoanalysisd/caches/graph/PhotoAnalysisServicePreferences.plist

@@ -3,24 +3,24 @@
 <plist version="1.0">
 <dict>
 	<key>BackgroundHighlightCollection</key>
-	<date>2020-04-17T14:33:32Z</date>
+	<date>2020-05-01T23:03:12Z</date>
 	<key>BackgroundHighlightEnrichment</key>
-	<date>2020-04-17T14:33:32Z</date>
+	<date>2020-05-01T23:03:11Z</date>
 	<key>BackgroundJobAssetRevGeocode</key>
-	<date>2020-04-17T14:33:33Z</date>
+	<date>2020-05-02T01:35:19Z</date>
 	<key>BackgroundJobSearch</key>
-	<date>2020-04-17T14:33:33Z</date>
+	<date>2020-05-01T23:03:12Z</date>
 	<key>BackgroundPeopleSuggestion</key>
-	<date>2020-04-17T14:33:31Z</date>
+	<date>2020-05-01T23:03:11Z</date>
 	<key>BackgroundUserBehaviorProcessor</key>
-	<date>2020-04-17T07:32:04Z</date>
+	<date>2020-05-01T23:03:13Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundGraphConsistencyUpdateJobDateKey</key>
-	<date>2020-04-17T14:33:37Z</date>
+	<date>2020-05-02T01:35:36Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundGraphRebuildJobDate</key>
-	<date>2020-04-17T07:32:00Z</date>
+	<date>2020-05-01T23:03:11Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundMemoryGenerationJobDate</key>
-	<date>2020-04-17T14:33:34Z</date>
+	<date>2020-05-02T01:35:19Z</date>
 	<key>SiriPortraitDonation</key>
-	<date>2020-04-17T07:32:04Z</date>
+	<date>2020-05-01T23:03:13Z</date>
 </dict>
 </plist>

tests/Test-10.15.1.photoslibrary/private/com.apple.photoanalysisd/caches/vision/PhotoAnalysisServicePreferences.plist

@@ -3,8 +3,8 @@
 <plist version="1.0">
 <dict>
 	<key>FaceIDModelLastGenerationKey</key>
-	<date>2020-04-17T07:32:07Z</date>
+	<date>2020-05-01T23:03:14Z</date>
 	<key>LastContactClassificationKey</key>
-	<date>2020-04-17T07:32:12Z</date>
+	<date>2020-05-01T23:03:18Z</date>
 </dict>
 </plist>

tests/Test-10.15.4.photoslibrary/database/Photos.sqlite.lock

@@ -7,7 +7,7 @@
 	<key>hostuuid</key>
 	<string>9575E48B-8D5F-5654-ABAC-4431B1167324</string>
 	<key>pid</key>
-	<integer>3212</integer>
+	<integer>434</integer>
 	<key>processname</key>
 	<string>photolibraryd</string>
 	<key>uid</key>

tests/Test-10.15.4.photoslibrary/private/com.apple.photoanalysisd/caches/graph/PhotoAnalysisServicePreferences.plist

@@ -3,24 +3,24 @@
 <plist version="1.0">
 <dict>
 	<key>BackgroundHighlightCollection</key>
-	<date>2020-04-11T20:00:25Z</date>
+	<date>2020-04-19T15:27:34Z</date>
 	<key>BackgroundHighlightEnrichment</key>
-	<date>2020-04-11T20:00:25Z</date>
+	<date>2020-04-19T15:27:34Z</date>
 	<key>BackgroundJobAssetRevGeocode</key>
-	<date>2020-04-11T20:00:25Z</date>
+	<date>2020-04-19T15:27:35Z</date>
 	<key>BackgroundJobSearch</key>
-	<date>2020-04-11T20:00:25Z</date>
+	<date>2020-04-19T15:27:35Z</date>
 	<key>BackgroundPeopleSuggestion</key>
-	<date>2020-04-11T20:00:24Z</date>
+	<date>2020-04-19T15:27:34Z</date>
 	<key>BackgroundUserBehaviorProcessor</key>
-	<date>2020-04-11T20:00:25Z</date>
+	<date>2020-04-19T15:27:35Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundGraphConsistencyUpdateJobDateKey</key>
 	<date>2020-04-11T20:10:27Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundGraphRebuildJobDate</key>
 	<date>2020-04-11T20:00:24Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundMemoryGenerationJobDate</key>
-	<date>2020-04-11T20:00:25Z</date>
+	<date>2020-04-19T15:27:35Z</date>
 	<key>SiriPortraitDonation</key>
-	<date>2020-04-11T20:00:25Z</date>
+	<date>2020-04-19T15:27:35Z</date>
 </dict>
 </plist>