Fix for issue #257, #275

CHANGELOG.md

@@ -4,6 +4,62 @@ All notable changes to this project will be documented in this file. Dates are d
 
 Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 
+#### [v0.37.4](https://github.com/RhetTbull/osxphotos/compare/v0.37.3...v0.37.4)
+
+> 5 December 2020
+
+- Merge branch 'master' of github.com:RhetTbull/osxphotos [`69cd236`](https://github.com/RhetTbull/osxphotos/commit/69cd2367122a3a86044df2845e706d3510bdf2c1)
+- Implement fix for issue #282, QuickTime metadata [`4cce9d4`](https://github.com/RhetTbull/osxphotos/commit/4cce9d4939a00ad2d265a510a2c6f0c8e6a8c655)
+- Implement fix for issue #282, QuickTime metadata [`cfb07cb`](https://github.com/RhetTbull/osxphotos/commit/cfb07cbfafaac493f6221be482c432812534ddfa)
+
+#### [v0.37.3](https://github.com/RhetTbull/osxphotos/compare/v0.37.2...v0.37.3)
+
+> 30 November 2020
+
+- Removed --use-photokit authorization check, issue 278 [`ed3a971`](https://github.com/RhetTbull/osxphotos/commit/ed3a9711dc0805aed1aacc30e01eeb9c1077d9e1)
+
+#### [v0.37.2](https://github.com/RhetTbull/osxphotos/compare/v0.37.1...v0.37.2)
+
+> 29 November 2020
+
+- Catch errors in export_photo [`d9dcf09`](https://github.com/RhetTbull/osxphotos/commit/d9dcf0917a541725d1e472e7f918733e4e2613d0)
+- Added --missing to export, see issue #277 [`25eacc7`](https://github.com/RhetTbull/osxphotos/commit/25eacc7caddd6721232b3f77a02532fcd35f7836)
+
+#### [v0.37.1](https://github.com/RhetTbull/osxphotos/compare/v0.37.0...v0.37.1)
+
+> 28 November 2020
+
+- Added --report option to CLI, implements #253 [`d22eaf3`](https://github.com/RhetTbull/osxphotos/commit/d22eaf39edc8b0b489b011d6d21345dcedcc8dff)
+- Updated template values [`af827d7`](https://github.com/RhetTbull/osxphotos/commit/af827d7a5769f41579d300a7cc511251d86b7eed)
+
+#### [v0.37.0](https://github.com/RhetTbull/osxphotos/compare/v0.36.25...v0.37.0)
+
+> 28 November 2020
+
+- Added {exiftool} template, implements issue #259 [`48acb42`](https://github.com/RhetTbull/osxphotos/commit/48acb42631226a71bfc636eea2d3151f1b7165f4)
+
+#### [v0.36.25](https://github.com/RhetTbull/osxphotos/compare/v0.36.24...v0.36.25)
+
+> 27 November 2020
+
+- Added --original-suffix for issue #263 [`399d432`](https://github.com/RhetTbull/osxphotos/commit/399d432a66354b9c235f30d10c6985fbde1b7e4f)
+
+#### [v0.36.24](https://github.com/RhetTbull/osxphotos/compare/v0.36.23...v0.36.24)
+
+> 26 November 2020
+
+- Initial implementation for issue #265 [`382fca3`](https://github.com/RhetTbull/osxphotos/commit/382fca3f92a3c251c12426dd0dc6d7dc21b691cf)
+- More work on issue #265 [`d5a9f76`](https://github.com/RhetTbull/osxphotos/commit/d5a9f767199d25ebd9d5925d05ee39ea7e51ac26)
+- Simplified sidecar table in export_db [`0632a97`](https://github.com/RhetTbull/osxphotos/commit/0632a97f55af67c7e5265b0d3283155c7c087e89)
+
+#### [v0.36.23](https://github.com/RhetTbull/osxphotos/compare/v0.36.22...v0.36.23)
+
+> 26 November 2020
+
+- Fix for missing original_filename, issue #267 [`fa33218`](https://github.com/RhetTbull/osxphotos/commit/fa332186ab3cdbe1bfd6496ff29b652ef984a5f8)
+- version bump [`b5195f9`](https://github.com/RhetTbull/osxphotos/commit/b5195f9d2b81cf6737b65e3cd3793ea9b0da13eb)
+- Updated test [`aa2ebf5`](https://github.com/RhetTbull/osxphotos/commit/aa2ebf55bb50eec14f86a532334b376e407f4bbc)
+
 #### [v0.36.22](https://github.com/RhetTbull/osxphotos/compare/v0.36.21...v0.36.22)
 
 > 26 November 2020

README.md

@@ -227,6 +227,9 @@ Options:
   --no-comment                    Search for photos with no comments.
   --has-likes                     Search for photos that have likes.
   --no-likes                      Search for photos with no likes.
+  --missing                       Export only photos missing from the Photos
+                                  library; must be used with --download-
+                                  missing.
   --deleted                       Include photos from the 'Recently Deleted'
                                   folder.
   --deleted-only                  Include only photos from the 'Recently
@@ -234,7 +237,8 @@ Options:
   --update                        Only export new or updated files. See notes
                                   below on export and --update.
   --dry-run                       Dry run (test) the export but don't actually
-                                  export any files; most useful with --verbose
+                                  export any files; most useful with
+                                  --verbose.
   --export-as-hardlink            Hardlink files instead of copying them.
                                   Cannot be used with --exiftool which creates
                                   copies of the files with embedded EXIF data.
@@ -351,6 +355,13 @@ Options:
                                   photo would be named
                                   'photoname_bearbeiten.ext'.  The default
                                   suffix is '_edited'.
+  --original-suffix SUFFIX        Optional suffix for naming original photos.
+                                  Default name for original photos is in form
+                                  'filename.ext'. For example, with '--
+                                  original-suffix _original', the original
+                                  photo would be named
+                                  'filename_original.ext'.  The default suffix
+                                  is '' (no suffix).
   --no-extended-attributes        Don't copy extended attributes when
                                   exporting.  You only need this if exporting
                                   to a filesystem that doesn't support Mac OS
@@ -366,6 +377,8 @@ Options:
                                   work with iTerm2 (use with Terminal.app).
                                   This is faster and more reliable than the
                                   default AppleScript interface.
+  --report REPORTNAME.CSV         Write a CSV formatted report of all files
+                                  that were exported.
   -h, --help                      Show this message and exit.
 
 ** Export **
@@ -629,18 +642,26 @@ exported, one to each directory.  For example: --directory
 of the following directories if the photos were created in 2019 and were in
 albums 'Vacation' and 'Family': 2019/Vacation, 2019/Family
 
-Substitution        Description
-{album}             Album(s) photo is contained in
-{folder_album}      Folder path + album photo is contained in. e.g.
-                    'Folder/Subfolder/Album' or just 'Album' if no enclosing
-                    folder
-{keyword}           Keyword(s) assigned to photo
-{person}            Person(s) / face(s) in a photo
-{label}             Image categorization label associated with a photo
-                    (Photos 5 only)
-{label_normalized}  All lower case version of 'label' (Photos 5 only)
-{comment}           Comment(s) on shared Photos; format is 'Person name:
-                    comment text' (Photos 5 only)
+Substitution              Description
+{album}                   Album(s) photo is contained in
+{folder_album}            Folder path + album photo is contained in. e.g.
+                          'Folder/Subfolder/Album' or just 'Album' if no
+                          enclosing folder
+{keyword}                 Keyword(s) assigned to photo
+{person}                  Person(s) / face(s) in a photo
+{label}                   Image categorization label associated with a photo
+                          (Photos 5 only)
+{label_normalized}        All lower case version of 'label' (Photos 5 only)
+{comment}                 Comment(s) on shared Photos; format is 'Person
+                          name: comment text' (Photos 5 only)
+{exiftool:GROUP:TAGNAME}  Use exiftool (https://exiftool.org) to extract
+                          metadata, in form GROUP:TAGNAME, from image.  E.g.
+                          '{exiftool:EXIF:Make}' to get camera make, or
+                          {exiftool:IPTC:Keywords} to extract keywords. See
+                          https://exiftool.org/TagNames/ for list of valid
+                          tag names.  You must specify group (e.g. EXIF,
+                          IPTC, etc) as used in `exiftool -G`. exiftool must
+                          be installed in the path to use this template.
 ```
 
 Example: export all photos to ~/Desktop/export group in folders by date created
@@ -2038,6 +2059,7 @@ The following template field substitutions are availabe for use with `PhotoInfo.
 |{label}|Image categorization label associated with a photo (Photos 5 only)|
 |{label_normalized}|All lower case version of 'label' (Photos 5 only)|
 |{comment}|Comment(s) on shared Photos; format is 'Person name: comment text' (Photos 5 only)|
+|{exiftool:GROUP:TAGNAME}|Use exiftool (https://exiftool.org) to extract metadata, in form GROUP:TAGNAME, from image.  E.g. '{exiftool:EXIF:Make}' to get camera make, or {exiftool:IPTC:Keywords} to extract keywords. See https://exiftool.org/TagNames/ for list of valid tag names.  You must specify group (e.g. EXIF, IPTC, etc) as used in `exiftool -G`. exiftool must be installed in the path to use this template.|
 
 ### Utility Functions
 
@@ -2184,8 +2206,6 @@ This package works by creating a copy of the sqlite3 database that photos uses t
 
 If apple changes the database format this will likely break.
 
-Apple does provide a framework ([PhotoKit](https://developer.apple.com/documentation/photokit?language=objc)) for querying the user's Photos library and I attempted to create the functionality in this package using this framework but unfortunately PhotoKit does not provide access to much of the needed metadata (such as Faces/Persons) and Apple's System Integrity Protection (SIP) made the interface unreliable.  If you'd like to experiment with the PhotoKit interface, here's some sample [code](https://gist.github.com/RhetTbull/41cc85e5bdeb30f761147ce32fba5c94).  While copying the sqlite file is a bit kludgy, it allows osxphotos to provide access to all available metadata.
-
 For additional details about how osxphotos is implemented or if you would like to extend the code, see the [wiki](https://github.com/RhetTbull/osxphotos/wiki).
 
 ## Dependencies

osxphotos/_version.py

@@ -1,4 +1,4 @@
 """ version info """
 
-__version__ = "0.36.23"
+__version__ = "0.37.5"
 

osxphotos/datetime_utils.py

@@ -1,10 +1,10 @@
-""" datetime utilities """
+""" datetime.datetime helper functions for converting to/from UTC """
 
 import datetime
 
 
 def get_local_tz(dt):
-    """ return local timezone as datetime.timezone tzinfo for dt
+    """ Return local timezone as datetime.timezone tzinfo for dt
     
     Args:
         dt: datetime.datetime
@@ -21,21 +21,18 @@ def get_local_tz(dt):
         raise ValueError("dt must be naive datetime.datetime object")
 
 
-def datetime_remove_tz(dt):
-    """ remove timezone from a datetime.datetime object
-        dt: datetime.datetime object with tzinfo
-        returns: dt without any timezone info (naive datetime object) """
-
-    if type(dt) != datetime.datetime:
-        raise TypeError(f"dt must be type datetime.datetime, not {type(dt)}")
-
-    return dt.replace(tzinfo=None)
-
-
 def datetime_has_tz(dt):
-    """ return True if datetime dt has tzinfo else False
+    """ Return True if datetime dt has tzinfo else False
+
+    Args:
         dt: datetime.datetime
-        returns True if dt is timezone aware, else False """
+    
+    Returns:
+        True if dt is timezone aware, else False
+        
+    Raises:
+        TypeError if dt is not a datetime.datetime object
+    """
 
     if type(dt) != datetime.datetime:
         raise TypeError(f"dt must be type datetime.datetime, not {type(dt)}")
@@ -43,11 +40,90 @@ def datetime_has_tz(dt):
     return dt.tzinfo is not None and dt.tzinfo.utcoffset(dt) is not None
 
 
-def datetime_naive_to_local(dt):
-    """ convert naive (timezone unaware) datetime.datetime
-        to aware timezone in local timezone
+def datetime_tz_to_utc(dt):
+    """ Convert datetime.datetime object with timezone to UTC timezone 
+
+    Args:
+        dt: datetime.datetime object
+
+    Returns:
+        datetime.datetime in UTC timezone
+        
+    Raises: 
+        TypeError if dt is not datetime.datetime object
+        ValueError if dt does not have timeone information
+    """
+
+    if type(dt) != datetime.datetime:
+        raise TypeError(f"dt must be type datetime.datetime, not {type(dt)}")
+
+    if dt.tzinfo is not None and dt.tzinfo.utcoffset(dt) is not None:
+        return dt.replace(tzinfo=dt.tzinfo).astimezone(tz=datetime.timezone.utc)
+    else:
+        raise ValueError(f"dt does not have timezone info")
+
+
+def datetime_remove_tz(dt):
+    """ Remove timezone from a datetime.datetime object
+
+    Args:
+        dt: datetime.datetime object with tzinfo
+    
+    Returns:
+        dt without any timezone info (naive datetime object) 
+        
+    Raises:
+        TypeError if dt is not a datetime.datetime object
+    """
+
+    if type(dt) != datetime.datetime:
+        raise TypeError(f"dt must be type datetime.datetime, not {type(dt)}")
+
+    return dt.replace(tzinfo=None)
+
+
+def datetime_naive_to_utc(dt):
+    """ Convert naive (timezone unaware) datetime.datetime
+        to aware timezone in UTC timezone
+
+    Args:
         dt: datetime.datetime without timezone
-        returns: datetime.datetime with local timezone """
+    
+    Returns:
+        datetime.datetime with UTC timezone
+        
+    Raises:
+        TypeError if dt is not a datetime.datetime object
+        ValueError if dt is not a naive/timezone unaware object
+    """
+
+    if type(dt) != datetime.datetime:
+        raise TypeError(f"dt must be type datetime.datetime, not {type(dt)}")
+
+    if dt.tzinfo is not None and dt.tzinfo.utcoffset(dt) is not None:
+        # has timezone info
+        raise ValueError(
+            "dt must be naive/timezone unaware: "
+            f"{dt} has tzinfo {dt.tzinfo} and offset {dt.tzinfo.utcoffset(dt)}"
+        )
+
+    return dt.replace(tzinfo=datetime.timezone.utc)
+
+
+def datetime_naive_to_local(dt):
+    """ Convert naive (timezone unaware) datetime.datetime
+        to aware timezone in local timezone
+
+    Args:
+        dt: datetime.datetime without timezone
+    
+    Returns:
+        datetime.datetime with local timezone
+        
+    Raises:
+        TypeError if dt is not a datetime.datetime object
+        ValueError if dt is not a naive/timezone unaware object
+    """
 
     if type(dt) != datetime.datetime:
         raise TypeError(f"dt must be type datetime.datetime, not {type(dt)}")
@@ -60,3 +136,26 @@ def datetime_naive_to_local(dt):
         )
 
     return dt.replace(tzinfo=get_local_tz(dt))
+
+
+def datetime_utc_to_local(dt):
+    """ Convert datetime.datetime object in UTC timezone to local timezone 
+
+    Args:
+        dt: datetime.datetime object
+
+    Returns:
+        datetime.datetime in local timezone
+
+    Raises:
+        TypeError if dt is not a datetime.datetime object
+        ValueError if dt is not in UTC timezone
+    """
+
+    if type(dt) != datetime.datetime:
+        raise TypeError(f"dt must be type datetime.datetime, not {type(dt)}")
+
+    if dt.tzinfo is not datetime.timezone.utc:
+        raise ValueError(f"{dt} must be in UTC timezone: timezone = {dt.tzinfo}")
+
+    return dt.replace(tzinfo=datetime.timezone.utc).astimezone(tz=None)

osxphotos/export_db.py

@@ -14,7 +14,7 @@ from sqlite3 import Error
 
 from ._version import __version__
 
-OSXPHOTOS_EXPORTDB_VERSION = "2.0"
+OSXPHOTOS_EXPORTDB_VERSION = "3.2"
 
 
 class ExportDB_ABC(ABC):
@@ -76,6 +76,14 @@ class ExportDB_ABC(ABC):
     def set_exifdata_for_file(self, uuid, exifdata):
         pass
 
+    @abstractmethod
+    def set_sidecar_for_file(self, filename, sidecar_data, sidecar_sig):
+        pass
+
+    @abstractmethod
+    def get_sidecar_for_file(self, filename):
+        pass
+
     @abstractmethod
     def set_data(
         self,
@@ -141,6 +149,12 @@ class ExportDBNoOp(ExportDB_ABC):
     def set_exifdata_for_file(self, uuid, exifdata):
         pass
 
+    def set_sidecar_for_file(self, filename, sidecar_data, sidecar_sig):
+        pass
+
+    def get_sidecar_for_file(self, filename):
+        return None, (None, None, None)
+
     def set_data(
         self,
         filename,
@@ -379,6 +393,48 @@ class ExportDB(ExportDB_ABC):
         except Error as e:
             logging.warning(e)
 
+    def get_sidecar_for_file(self, filename):
+        """ returns the sidecar data and signature for a file """
+        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        conn = self._conn
+        try:
+            c = conn.cursor()
+            c.execute(
+                "SELECT sidecar_data, mode, size, mtime FROM sidecar WHERE filepath_normalized = ?",
+                (filename,),
+            )
+            results = c.fetchone()
+            if results:
+                sidecar_data = results[0]
+                sidecar_sig = (
+                    results[1],
+                    results[2],
+                    int(results[3]) if results[3] is not None else None,
+                )
+            else:
+                sidecar_data = None
+                sidecar_sig = (None, None, None)
+        except Error as e:
+            logging.warning(e)
+            sidecar_data = None
+            sidecar_sig = (None, None, None)
+
+        return sidecar_data, sidecar_sig
+
+    def set_sidecar_for_file(self, filename, sidecar_data, sidecar_sig):
+        """ sets the sidecar data and signature for a file """
+        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        conn = self._conn
+        try:
+            c = conn.cursor()
+            c.execute(
+                "INSERT OR REPLACE INTO sidecar(filepath_normalized, sidecar_data, mode, size, mtime) VALUES (?, ?, ?, ?, ?);",
+                (filename, sidecar_data, *sidecar_sig),
+            )
+            conn.commit()
+        except Error as e:
+            logging.warning(e)
+
     def set_data(
         self,
         filename,
@@ -479,13 +535,11 @@ class ExportDB(ExportDB_ABC):
 
         if not os.path.isfile(dbfile):
             conn = self._get_db_connection(dbfile)
-            if conn:
-                self._create_db_tables(conn)
-                self.was_created = True
-                self.was_upgraded = ()
-                self.version = OSXPHOTOS_EXPORTDB_VERSION
-            else:
+            if not conn:
                 raise Exception("Error getting connection to database {dbfile}")
+            self._create_db_tables(conn)
+            self.was_created = True
+            self.was_upgraded = ()
         else:
             conn = self._get_db_connection(dbfile)
             self.was_created = False
@@ -495,8 +549,7 @@ class ExportDB(ExportDB_ABC):
                 self.was_upgraded = (version_info[1], OSXPHOTOS_EXPORTDB_VERSION)
             else:
                 self.was_upgraded = ()
-            self.version = OSXPHOTOS_EXPORTDB_VERSION
-
+        self.version = OSXPHOTOS_EXPORTDB_VERSION
         return conn
 
     def _get_db_connection(self, dbfile):
@@ -570,11 +623,20 @@ class ExportDB(ExportDB_ABC):
                               size INTEGER,
                               mtime REAL
                               ); """,
+            "sql_sidecar_table": """ CREATE TABLE IF NOT EXISTS sidecar (
+                              id INTEGER PRIMARY KEY,
+                              filepath_normalized TEXT NOT NULL,
+                              sidecar_data TEXT,
+                              mode INTEGER,
+                              size INTEGER,
+                              mtime REAL
+                              ); """,
             "sql_files_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_files_filepath_normalized on files (filepath_normalized); """,
             "sql_info_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_info_uuid on info (uuid); """,
             "sql_exifdata_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_exifdata_filename on exifdata (filepath_normalized); """,
             "sql_edited_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_edited_filename on edited (filepath_normalized);""",
             "sql_converted_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_converted_filename on converted (filepath_normalized);""",
+            "sql_sidecar_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_sidecar_filename on sidecar (filepath_normalized);""",
         }
         try:
             c = conn.cursor()

osxphotos/photoinfo/_photoinfo_export.py

@@ -13,6 +13,7 @@
 # TODO: should this be its own PhotoExporter class?
 
 import glob
+import hashlib
 import json
 import logging
 import os
@@ -32,6 +33,7 @@ from .._constants import (
     _UNKNOWN_PERSON,
     _XMP_TEMPLATE_NAME,
 )
+from ..datetime_utils import datetime_tz_to_utc
 from ..exiftool import ExifTool
 from ..export_db import ExportDBNoOp
 from ..fileutil import FileUtil
@@ -41,14 +43,36 @@ from ..photokit import (
     PhotoLibrary,
     PhotoKitFetchFailed,
 )
-from ..utils import dd_to_dms_str, findfiles
+from ..utils import dd_to_dms_str, findfiles, noop
 
 ExportResults = namedtuple(
     "ExportResults",
-    ["exported", "new", "updated", "skipped", "exif_updated", "touched"],
+    [
+        "exported",
+        "new",
+        "updated",
+        "skipped",
+        "exif_updated",
+        "touched",
+        "converted_to_jpeg",
+        "sidecar_json_written",
+        "sidecar_json_skipped",
+        "sidecar_xmp_written",
+        "sidecar_xmp_skipped",
+        "missing",
+        "error",
+    ],
 )
 
 
+# hexdigest is not a class method, don't import this into PhotoInfo
+def hexdigest(strval):
+    """ hexdigest of a string, using blake2b """
+    h = hashlib.blake2b(digest_size=20)
+    h.update(bytes(strval, "utf-8"))
+    return h.hexdigest()
+
+
 # _export_photo_uuid_applescript is not a class method, don't import this into PhotoInfo
 def _export_photo_uuid_applescript(
     uuid,
@@ -61,27 +85,27 @@ def _export_photo_uuid_applescript(
     burst=False,
     dry_run=False,
 ):
-    """ Export photo to dest path using applescript to control Photos
-        If photo is a live photo, exports both the photo and associated .mov file
-        uuid: UUID of photo to export
-        dest: destination path to export to
-        filestem: (string) if provided, exported filename will be named stem.ext 
-                  where ext is extension of the file exported by photos (e.g. .jpeg, .mov, etc)
-                  If not provided, file will be named with whatever name Photos uses
-                  If filestem.ext exists, it wil be overwritten
-        original: (boolean) if True, export original image; default = True
-        edited: (boolean) if True, export edited photo; default = False
-                If photo not edited and edited=True, will still export the original image
-                caller must verify image has been edited
-        *Note*: must be called with either edited or original but not both, 
-                will raise error if called with both edited and original = True
-        live_photo: (boolean) if True, export associated .mov live photo; default = False
-        timeout: timeout value in seconds; export will fail if applescript run time exceeds timeout
-        burst: (boolean) set to True if file is a burst image to avoid Photos export error
-        dry_run: (boolean) set to True to run in "dry run" mode which will download file but not actually copy to destination
-        Returns: list of paths to exported file(s) or None if export failed
-        Note: For Live Photos, if edited=True, will export a jpeg but not the movie, even if photo
-              has not been edited. This is due to how Photos Applescript interface works.
+    """Export photo to dest path using applescript to control Photos
+    If photo is a live photo, exports both the photo and associated .mov file
+    uuid: UUID of photo to export
+    dest: destination path to export to
+    filestem: (string) if provided, exported filename will be named stem.ext
+              where ext is extension of the file exported by photos (e.g. .jpeg, .mov, etc)
+              If not provided, file will be named with whatever name Photos uses
+              If filestem.ext exists, it wil be overwritten
+    original: (boolean) if True, export original image; default = True
+    edited: (boolean) if True, export edited photo; default = False
+            If photo not edited and edited=True, will still export the original image
+            caller must verify image has been edited
+    *Note*: must be called with either edited or original but not both,
+            will raise error if called with both edited and original = True
+    live_photo: (boolean) if True, export associated .mov live photo; default = False
+    timeout: timeout value in seconds; export will fail if applescript run time exceeds timeout
+    burst: (boolean) set to True if file is a burst image to avoid Photos export error
+    dry_run: (boolean) set to True to run in "dry run" mode which will download file but not actually copy to destination
+    Returns: list of paths to exported file(s) or None if export failed
+    Note: For Live Photos, if edited=True, will export a jpeg but not the movie, even if photo
+          has not been edited. This is due to how Photos Applescript interface works.
     """
 
     # setup the applescript to do the export
@@ -149,11 +173,9 @@ def _export_photo_uuid_applescript(
                 and path.suffix.lower() == ".mov"
             ):
                 # it's the .mov part of live photo but not requested, so don't export
-                logging.debug(f"Skipping live photo file {path}")
                 continue
             if len(exported_files) > 1 and burst and path.stem != filename_stem:
                 # skip any burst photo that's not the one we asked for
-                logging.debug(f"Skipping burst photo file {path}")
                 continue
             if filestem:
                 # rename the file based on filestem, keeping original extension
@@ -161,7 +183,6 @@ def _export_photo_uuid_applescript(
             else:
                 # use the name Photos provided
                 dest_new = dest / path.name
-            logging.debug(f"exporting {path} to dest_new: {dest_new}")
             if not dry_run:
                 FileUtil.copy(str(path), str(dest_new))
             exported_paths.append(str(dest_new))
@@ -173,10 +194,10 @@ def _export_photo_uuid_applescript(
 # _check_export_suffix is not a class method, don't import this into PhotoInfo
 def _check_export_suffix(src, dest, edited):
     """Helper function for exporting photos to check file extensions of destination path.
-    
+
     Checks that dst file extension is appropriate for the src.
     If edited=True, will use src file extension of ".jpeg" if None provided for src.
-    
+
     Args:
         src: path to source file or None.
         dest: path to destination file.
@@ -231,43 +252,43 @@ def export(
     keyword_template=None,
     description_template=None,
 ):
-    """ export photo 
-        dest: must be valid destination path (or exception raised) 
-        filename: (optional): name of exported picture; if not provided, will use current filename 
-                    **NOTE**: if provided, user must ensure file extension (suffix) is correct. 
-                    For example, if photo is .CR2 file, edited image may be .jpeg.  
-                    If you provide an extension different than what the actual file is, 
-                    export will print a warning but will export the photo using the 
-                    incorrect file extension (unless use_photos_export is true, in which case export will
-                    use the extension provided by Photos upon export; in this case, an incorrect extension is
-                    silently ignored).
-                    e.g. to get the extension of the edited photo, 
-                    reference PhotoInfo.path_edited
-        edited: (boolean, default=False); if True will export the edited version of the photo 
-                (or raise exception if no edited version) 
-        live_photo: (boolean, default=False); if True, will also export the associted .mov for live photos
-        raw_photo: (boolean, default=False); if True, will also export the associted RAW photo
-        export_as_hardlink: (boolean, default=False); if True, will hardlink files instead of copying them
-        overwrite: (boolean, default=False); if True will overwrite files if they alreay exist 
-        increment: (boolean, default=True); if True, will increment file name until a non-existant name is found 
-                    if overwrite=False and increment=False, export will fail if destination file already exists 
-        sidecar_json: (boolean, default = False); if True will also write a json sidecar with IPTC data in format readable by exiftool
-                    sidecar filename will be dest/filename.json 
-        sidecar_xmp: (boolean, default = False); if True will also write a XMP sidecar with IPTC data 
-                    sidecar filename will be dest/filename.xmp 
-        use_photos_export: (boolean, default=False); if True will attempt to export photo via applescript interaction with Photos
-        timeout: (int, default=120) timeout in seconds used with use_photos_export
-        exiftool: (boolean, default = False); if True, will use exiftool to write metadata to export file
-        no_xattr: (boolean, default = False); if True, exports file without preserving extended attributes
-        returns list of full paths to the exported files
-        use_albums_as_keywords: (boolean, default = False); if True, will include album names in keywords
-        when exporting metadata with exiftool or sidecar
-        use_persons_as_keywords: (boolean, default = False); if True, will include person names in keywords
-        when exporting metadata with exiftool or sidecar
-        keyword_template: (list of strings); list of template strings that will be rendered as used as keywords
-        description_template: string; optional template string that will be rendered for use as photo description
-        returns: list of photos exported
-        """
+    """export photo
+    dest: must be valid destination path (or exception raised)
+    filename: (optional): name of exported picture; if not provided, will use current filename
+                **NOTE**: if provided, user must ensure file extension (suffix) is correct.
+                For example, if photo is .CR2 file, edited image may be .jpeg.
+                If you provide an extension different than what the actual file is,
+                export will print a warning but will export the photo using the
+                incorrect file extension (unless use_photos_export is true, in which case export will
+                use the extension provided by Photos upon export; in this case, an incorrect extension is
+                silently ignored).
+                e.g. to get the extension of the edited photo,
+                reference PhotoInfo.path_edited
+    edited: (boolean, default=False); if True will export the edited version of the photo
+            (or raise exception if no edited version)
+    live_photo: (boolean, default=False); if True, will also export the associted .mov for live photos
+    raw_photo: (boolean, default=False); if True, will also export the associted RAW photo
+    export_as_hardlink: (boolean, default=False); if True, will hardlink files instead of copying them
+    overwrite: (boolean, default=False); if True will overwrite files if they alreay exist
+    increment: (boolean, default=True); if True, will increment file name until a non-existant name is found
+                if overwrite=False and increment=False, export will fail if destination file already exists
+    sidecar_json: (boolean, default = False); if True will also write a json sidecar with IPTC data in format readable by exiftool
+                sidecar filename will be dest/filename.json
+    sidecar_xmp: (boolean, default = False); if True will also write a XMP sidecar with IPTC data
+                sidecar filename will be dest/filename.xmp
+    use_photos_export: (boolean, default=False); if True will attempt to export photo via applescript interaction with Photos
+    timeout: (int, default=120) timeout in seconds used with use_photos_export
+    exiftool: (boolean, default = False); if True, will use exiftool to write metadata to export file
+    no_xattr: (boolean, default = False); if True, exports file without preserving extended attributes
+    returns list of full paths to the exported files
+    use_albums_as_keywords: (boolean, default = False); if True, will include album names in keywords
+    when exporting metadata with exiftool or sidecar
+    use_persons_as_keywords: (boolean, default = False); if True, will include person names in keywords
+    when exporting metadata with exiftool or sidecar
+    keyword_template: (list of strings); list of template strings that will be rendered as used as keywords
+    description_template: string; optional template string that will be rendered for use as photo description
+    returns: list of photos exported
+    """
 
     # Implementation note: calls export2 to actually do the work
 
@@ -324,56 +345,70 @@ def export2(
     jpeg_quality=1.0,
     ignore_date_modified=False,
     use_photokit=False,
+    verbose=None,
 ):
-    """ export photo, like export but with update and dry_run options
-        dest: must be valid destination path or exception raised 
-        filename: (optional): name of exported picture; if not provided, will use current filename 
-                    **NOTE**: if provided, user must ensure file extension (suffix) is correct. 
-                    For example, if photo is .CR2 file, edited image may be .jpeg.  
-                    If you provide an extension different than what the actual file is, 
-                    will export the photo using the incorrect file extension (unless use_photos_export is true, 
-                    in which case export will use the extension provided by Photos upon export.
-                    e.g. to get the extension of the edited photo, 
-                    reference PhotoInfo.path_edited
-        edited: (boolean, default=False); if True will export the edited version of the photo 
-                (or raise exception if no edited version) 
-        live_photo: (boolean, default=False); if True, will also export the associted .mov for live photos
-        raw_photo: (boolean, default=False); if True, will also export the associted RAW photo
-        export_as_hardlink: (boolean, default=False); if True, will hardlink files instead of copying them
-        overwrite: (boolean, default=False); if True will overwrite files if they alreay exist 
-        increment: (boolean, default=True); if True, will increment file name until a non-existant name is found 
-                    if overwrite=False and increment=False, export will fail if destination file already exists 
-        sidecar_json: (boolean, default = False); if True will also write a json sidecar with IPTC data in format readable by exiftool
-                    sidecar filename will be dest/filename.json 
-        sidecar_xmp: (boolean, default = False); if True will also write a XMP sidecar with IPTC data 
-                    sidecar filename will be dest/filename.xmp 
-        use_photos_export: (boolean, default=False); if True will attempt to export photo via applescript interaction with Photos
-        timeout: (int, default=120) timeout in seconds used with use_photos_export
-        exiftool: (boolean, default = False); if True, will use exiftool to write metadata to export file
-        no_xattr: (boolean, default = False); if True, exports file without preserving extended attributes
-        use_albums_as_keywords: (boolean, default = False); if True, will include album names in keywords
-        when exporting metadata with exiftool or sidecar
-        use_persons_as_keywords: (boolean, default = False); if True, will include person names in keywords
-        when exporting metadata with exiftool or sidecar
-        keyword_template: (list of strings); list of template strings that will be rendered as used as keywords
-        description_template: string; optional template string that will be rendered for use as photo description
-        update: (boolean, default=False); if True export will run in update mode, that is, it will
-                not export the photo if the current version already exists in the destination
-        export_db: (ExportDB_ABC); instance of a class that conforms to ExportDB_ABC with methods
-                for getting/setting data related to exported files to compare update state
-        fileutil: (FileUtilABC); class that conforms to FileUtilABC with various file utilities
-        dry_run: (boolean, default=False); set to True to run in "dry run" mode
-        touch_file: (boolean, default=False); if True, sets file's modification time upon photo date
-        convert_to_jpeg: boolean; if True, converts non-jpeg images to jpeg
-        jpeg_quality: float in range 0.0 <= jpeg_quality <= 1.0.  A value of 1.0 specifies use best quality, a value of 0.0 specifies use maximum compression.
-        ignore_date_modified: for use with sidecar and exiftool; if True, sets EXIF:ModifyDate to EXIF:DateTimeOriginal even if date_modified is set
+    """export photo, like export but with update and dry_run options
+    dest: must be valid destination path or exception raised
+    filename: (optional): name of exported picture; if not provided, will use current filename
+                **NOTE**: if provided, user must ensure file extension (suffix) is correct.
+                For example, if photo is .CR2 file, edited image may be .jpeg.
+                If you provide an extension different than what the actual file is,
+                will export the photo using the incorrect file extension (unless use_photos_export is true,
+                in which case export will use the extension provided by Photos upon export.
+                e.g. to get the extension of the edited photo,
+                reference PhotoInfo.path_edited
+    edited: (boolean, default=False); if True will export the edited version of the photo
+            (or raise exception if no edited version)
+    live_photo: (boolean, default=False); if True, will also export the associted .mov for live photos
+    raw_photo: (boolean, default=False); if True, will also export the associted RAW photo
+    export_as_hardlink: (boolean, default=False); if True, will hardlink files instead of copying them
+    overwrite: (boolean, default=False); if True will overwrite files if they alreay exist
+    increment: (boolean, default=True); if True, will increment file name until a non-existant name is found
+                if overwrite=False and increment=False, export will fail if destination file already exists
+    sidecar_json: (boolean, default = False); if True will also write a json sidecar with IPTC data in format readable by exiftool
+                sidecar filename will be dest/filename.json
+    sidecar_xmp: (boolean, default = False); if True will also write a XMP sidecar with IPTC data
+                sidecar filename will be dest/filename.xmp
+    use_photos_export: (boolean, default=False); if True will attempt to export photo via applescript interaction with Photos
+    timeout: (int, default=120) timeout in seconds used with use_photos_export
+    exiftool: (boolean, default = False); if True, will use exiftool to write metadata to export file
+    no_xattr: (boolean, default = False); if True, exports file without preserving extended attributes
+    use_albums_as_keywords: (boolean, default = False); if True, will include album names in keywords
+    when exporting metadata with exiftool or sidecar
+    use_persons_as_keywords: (boolean, default = False); if True, will include person names in keywords
+    when exporting metadata with exiftool or sidecar
+    keyword_template: (list of strings); list of template strings that will be rendered as used as keywords
+    description_template: string; optional template string that will be rendered for use as photo description
+    update: (boolean, default=False); if True export will run in update mode, that is, it will
+            not export the photo if the current version already exists in the destination
+    export_db: (ExportDB_ABC); instance of a class that conforms to ExportDB_ABC with methods
+            for getting/setting data related to exported files to compare update state
+    fileutil: (FileUtilABC); class that conforms to FileUtilABC with various file utilities
+    dry_run: (boolean, default=False); set to True to run in "dry run" mode
+    touch_file: (boolean, default=False); if True, sets file's modification time upon photo date
+    convert_to_jpeg: boolean; if True, converts non-jpeg images to jpeg
+    jpeg_quality: float in range 0.0 <= jpeg_quality <= 1.0.  A value of 1.0 specifies use best quality, a value of 0.0 specifies use maximum compression.
+    ignore_date_modified: for use with sidecar and exiftool; if True, sets EXIF:ModifyDate to EXIF:DateTimeOriginal even if date_modified is set
+    verbose: optional callable function to use for printing verbose text during processing; if None (default), does not print output.
 
-        Returns: ExportResults namedtuple with fields: exported, new, updated, skipped 
-                    where each field is a list of file paths
-        
-        Note: to use dry run mode, you must set dry_run=True and also pass in memory version of export_db,
-              and no-op fileutil (e.g. ExportDBInMemory and FileUtilNoOp)
-            """
+    Returns: ExportResults namedtuple with fields: 
+        "exported",
+        "new",
+        "updated",
+        "skipped",
+        "exif_updated",
+        "touched",
+        "converted_to_jpeg",
+        "sidecar_json_written",
+        "sidecar_json_skipped",
+        "sidecar_xmp_written",
+        "sidecar_xmp_skipped",
+        "missing",
+        "error"
+ 
+    Note: to use dry run mode, you must set dry_run=True and also pass in memory version of export_db,
+          and no-op fileutil (e.g. ExportDBInMemory and FileUtilNoOp)
+    """
 
     # NOTE: This function is very complex and does a lot of things.
     # Don't modify this code if you don't fully understand everything it does.
@@ -383,6 +418,12 @@ def export2(
     if export_db is None:
         export_db = ExportDBNoOp()
 
+    if verbose is None:
+        verbose = noop
+    elif not callable(verbose):
+        raise TypeError("verbose must be callable")
+    self._verbose = verbose
+
     # suffix to add to edited files
     # e.g. name will be filename_edited.jpg
     edited_identifier = "_edited"
@@ -402,6 +443,9 @@ def export2(
     # list of all files with utime touched (touch_file = True)
     touched_files = []
 
+    # list of all files convereted to jpeg
+    converted_to_jpeg_files = []
+
     # check edited and raise exception trying to export edited version of
     # photo that hasn't been edited
     if edited and not self.hasadjustments:
@@ -488,11 +532,6 @@ def export2(
                     f"Cannot export edited photo if path_edited is None"
                 )
         else:
-            if self.ismissing:
-                logging.debug(
-                    f"Attempting to export photo with ismissing=True: path = {self.path}"
-                )
-
             if self.path is not None:
                 src = self.path
             else:
@@ -501,32 +540,22 @@ def export2(
         if not os.path.isfile(src):
             raise FileNotFoundError(f"{src} does not appear to exist")
 
-        if not _check_export_suffix(src, dest, edited):
-            logging.debug(
-                f"Invalid destination suffix: {dest.suffix} for {self.path}, "
-                + f"edited={edited}, path_edited={self.path_edited}, "
-                + f"original_filename={self.original_filename}, filename={self.filename}"
-            )
-
         # found source now try to find right destination
         if update and dest.exists():
             # destination exists, check to see if destination is the right UUID
             dest_uuid = export_db.get_uuid_for_file(dest)
             if dest_uuid is None and fileutil.cmp(src, dest):
                 # might be exporting into a pre-ExportDB folder or the DB got deleted
-                logging.debug(
-                    f"Found matching file with blank uuid: {self.uuid}, {dest}"
-                )
                 dest_uuid = self.uuid
                 export_db.set_data(
-                    dest,
-                    self.uuid,
-                    fileutil.file_sig(dest),
-                    (None, None, None),
-                    (None, None, None),
-                    (None, None, None),
-                    self.json(),
-                    None,
+                    filename=dest,
+                    uuid=self.uuid,
+                    orig_stat=fileutil.file_sig(dest),
+                    exif_stat=(None, None, None),
+                    converted_stat=(None, None, None),
+                    edited_stat=(None, None, None),
+                    info_json=self.json(),
+                    exif_json=None,
                 )
             if dest_uuid != self.uuid:
                 # not the right file, find the right one
@@ -545,14 +574,14 @@ def export2(
                         dest = pathlib.Path(file_)
                         found_match = True
                         export_db.set_data(
-                            dest,
-                            self.uuid,
-                            fileutil.file_sig(dest),
-                            (None, None, None),
-                            (None, None, None),
-                            (None, None, None),
-                            self.json(),
-                            None,
+                            filename=dest,
+                            uuid=self.uuid,
+                            orig_stat=fileutil.file_sig(dest),
+                            exif_stat=(None, None, None),
+                            converted_stat=(None, None, None),
+                            edited_stat=(None, None, None),
+                            info_json=self.json(),
+                            exif_json=None,
                         )
                         break
 
@@ -589,6 +618,7 @@ def export2(
         update_updated_files = results.updated
         update_skipped_files = results.skipped
         touched_files = results.touched
+        converted_to_jpeg_files = results.converted_to_jpeg
 
         # copy live photo associated .mov if requested
         if live_photo and self.live_photo:
@@ -596,9 +626,6 @@ def export2(
             src_live = self.path_live_photo
 
             if src_live is not None:
-                logging.debug(
-                    f"Exporting live photo video of {filename} as {live_name.name}"
-                )
                 results = self._export_photo(
                     src_live,
                     live_name,
@@ -617,8 +644,7 @@ def export2(
                 update_updated_files.extend(results.updated)
                 update_skipped_files.extend(results.skipped)
                 touched_files.extend(results.touched)
-            else:
-                logging.debug(f"Skipping missing live movie for {filename}")
+                converted_to_jpeg_files.extend(results.converted_to_jpeg)
 
         # copy associated RAW image if requested
         if raw_photo and self.has_raw:
@@ -626,7 +652,6 @@ def export2(
             raw_ext = raw_path.suffix
             raw_name = dest.parent / f"{dest.stem}{raw_ext}"
             if raw_path is not None:
-                logging.debug(f"Exporting RAW photo of {filename} as {raw_name.name}")
                 results = self._export_photo(
                     raw_path,
                     raw_name,
@@ -646,8 +671,7 @@ def export2(
                 update_updated_files.extend(results.updated)
                 update_skipped_files.extend(results.skipped)
                 touched_files.extend(results.touched)
-            else:
-                logging.debug(f"Skipping missing RAW photo for {filename}")
+                converted_to_jpeg_files.extend(results.converted_to_jpeg)
     else:
         # use_photo_export
         exported = []
@@ -748,8 +772,9 @@ def export2(
             )
 
     # export metadata
+    sidecar_json_files_skipped = []
+    sidecar_json_files_written = []
     if sidecar_json:
-        logging.debug("writing exiftool_json_sidecar")
         sidecar_filename = dest.parent / pathlib.Path(f"{dest.stem}{dest.suffix}.json")
         sidecar_str = self._exiftool_json_sidecar(
             use_albums_as_keywords=use_albums_as_keywords,
@@ -758,15 +783,36 @@ def export2(
             description_template=description_template,
             ignore_date_modified=ignore_date_modified,
         )
-        if not dry_run:
-            try:
+        sidecar_digest = hexdigest(sidecar_str)
+        old_sidecar_digest, sidecar_sig = export_db.get_sidecar_for_file(
+            sidecar_filename
+        )
+        write_sidecar = (
+            not update
+            or (update and not sidecar_filename.exists())
+            or (
+                update
+                and (sidecar_digest != old_sidecar_digest)
+                or not fileutil.cmp_file_sig(sidecar_filename, sidecar_sig)
+            )
+        )
+        if write_sidecar:
+            verbose(f"Writing exiftool JSON sidecar {sidecar_filename}")
+            sidecar_json_files_written.append(str(sidecar_filename))
+            if not dry_run:
                 self._write_sidecar(sidecar_filename, sidecar_str)
-            except Exception as e:
-                logging.warning(f"Error writing json sidecar to {sidecar_filename}")
-                raise e
+                export_db.set_sidecar_for_file(
+                    sidecar_filename,
+                    sidecar_digest,
+                    fileutil.file_sig(sidecar_filename),
+                )
+        else:
+            verbose(f"Skipped up to date exiftool JSON sidecar {sidecar_filename}")
+            sidecar_json_files_skipped.append(str(sidecar_filename))
 
+    sidecar_xmp_files_skipped = []
+    sidecar_xmp_files_written = []
     if sidecar_xmp:
-        logging.debug("writing xmp_sidecar")
         sidecar_filename = dest.parent / pathlib.Path(f"{dest.stem}{dest.suffix}.xmp")
         sidecar_str = self._xmp_sidecar(
             use_albums_as_keywords=use_albums_as_keywords,
@@ -775,12 +821,32 @@ def export2(
             description_template=description_template,
             extension=dest.suffix[1:] if dest.suffix else None,
         )
-        if not dry_run:
-            try:
+        sidecar_digest = hexdigest(sidecar_str)
+        old_sidecar_digest, sidecar_sig = export_db.get_sidecar_for_file(
+            sidecar_filename
+        )
+        write_sidecar = (
+            not update
+            or (update and not sidecar_filename.exists())
+            or (
+                update
+                and (sidecar_digest != old_sidecar_digest)
+                or not fileutil.cmp_file_sig(sidecar_filename, sidecar_sig)
+            )
+        )
+        if write_sidecar:
+            verbose(f"Writing XMP sidecar {sidecar_filename}")
+            sidecar_xmp_files_written.append(str(sidecar_filename))
+            if not dry_run:
                 self._write_sidecar(sidecar_filename, sidecar_str)
-            except Exception as e:
-                logging.warning(f"Error writing xmp sidecar to {sidecar_filename}")
-                raise e
+                export_db.set_sidecar_for_file(
+                    sidecar_filename,
+                    sidecar_digest,
+                    fileutil.file_sig(sidecar_filename),
+                )
+        else:
+            verbose(f"Skipped up to date XMP sidecar {sidecar_filename}")
+            sidecar_xmp_files_skipped.append(str(sidecar_filename))
 
     # if exiftool, write the metadata
     if update:
@@ -791,7 +857,6 @@ def export2(
     exif_files_updated = []
     if exiftool and update and exif_files:
         for exported_file in exif_files:
-            logging.debug(f"checking exif for {exported_file}")
             files_are_different = False
             old_data = export_db.get_exifdata_for_file(exported_file)
             if old_data is not None:
@@ -811,6 +876,7 @@ def export2(
             if old_data is None or files_are_different:
                 # didn't have old data, assume we need to write it
                 # or files were different
+                verbose(f"Writing metadata with exiftool for {exported_file}")
                 if not dry_run:
                     self._write_exif_data(
                         exported_file,
@@ -834,8 +900,11 @@ def export2(
                     exported_file, fileutil.file_sig(exported_file)
                 )
                 exif_files_updated.append(exported_file)
+            else:
+                verbose(f"Skipped up to date exiftool metadata for {exported_file}")
     elif exiftool and exif_files:
         for exported_file in exif_files:
+            verbose(f"Writing metadata with exiftool for {exported_file}")
             if not dry_run:
                 self._write_exif_data(
                     exported_file,
@@ -863,6 +932,7 @@ def export2(
 
     if touch_file:
         for exif_file in exif_files_updated:
+            verbose(f"Updating file modification time for {exif_file}")
             touched_files.append(exif_file)
             ts = int(self.date.timestamp())
             fileutil.utime(exif_file, (ts, ts))
@@ -870,12 +940,19 @@ def export2(
     touched_files = list(set(touched_files))
 
     results = ExportResults(
-        exported_files,
-        update_new_files,
-        update_updated_files,
-        update_skipped_files,
-        exif_files_updated,
-        touched_files,
+        exported=exported_files,
+        new=update_new_files,
+        updated=update_updated_files,
+        skipped=update_skipped_files,
+        exif_updated=exif_files_updated,
+        touched=touched_files,
+        converted_to_jpeg=converted_to_jpeg_files,
+        sidecar_json_written=sidecar_json_files_written,
+        sidecar_json_skipped=sidecar_json_files_skipped,
+        sidecar_xmp_written=sidecar_xmp_files_written,
+        sidecar_xmp_skipped=sidecar_xmp_files_skipped,
+        missing=[],
+        error=[],
     )
     return results
 
@@ -896,12 +973,12 @@ def _export_photo(
     edited=False,
     jpeg_quality=1.0,
 ):
-    """ Helper function for export()
-        Does the actual copy or hardlink taking the appropriate 
+    """Helper function for export()
+        Does the actual copy or hardlink taking the appropriate
         action depending on update, overwrite, export_as_hardlink
         Assumes destination is the right destination (e.g. UUID matches)
         sets UUID and JSON info foo exported file using set_uuid_for_file, set_inf_for_uuido
-    
+
     Args:
         src: src path (string)
         dest: dest path (pathlib.Path)
@@ -932,10 +1009,10 @@ def _export_photo(
     update_new_files = []
     update_skipped_files = []
     touched_files = []
+    converted_to_jpeg_files = []
 
     dest_str = str(dest)
     dest_exists = dest.exists()
-    op_desc = "export_as_hardlink" if export_as_hardlink else "export_by_copying"
 
     if update:  # updating
         cmp_touch, cmp_orig = False, False
@@ -998,13 +1075,11 @@ def _export_photo(
 
         else:
             # update, destination doesn't exist (new file)
-            logging.debug(f"Update: exporting new file with {op_desc} {src} {dest}")
             update_new_files.append(dest_str)
             if touch_file:
                 touched_files.append(dest_str)
     else:
         # not update, export the file
-        logging.debug(f"Exporting file with {op_desc} {src} {dest}")
         exported_files.append(dest_str)
         if touch_file:
             sig = fileutil.file_sig(src)
@@ -1016,9 +1091,6 @@ def _export_photo(
         edited_stat = fileutil.file_sig(src) if edited else (None, None, None)
         if dest_exists and (update or overwrite):
             # need to remove the destination first
-            logging.debug(
-                f"Update: removing existing file prior to {op_desc} {src} {dest}"
-            )
             fileutil.unlink(dest)
         if export_as_hardlink:
             fileutil.hardlink(src, dest)
@@ -1026,18 +1098,19 @@ def _export_photo(
             # use convert_to_jpeg to export the file
             fileutil.convert_to_jpeg(src, dest_str, compression_quality=jpeg_quality)
             converted_stat = fileutil.file_sig(dest_str)
+            converted_to_jpeg_files.append(dest_str)
         else:
             fileutil.copy(src, dest_str, norsrc=no_xattr)
 
         export_db.set_data(
-            dest_str,
-            self.uuid,
-            fileutil.file_sig(dest_str),
-            (None, None, None),
-            converted_stat,
-            edited_stat,
-            self.json(),
-            None,
+            filename=dest_str,
+            uuid=self.uuid,
+            orig_stat=fileutil.file_sig(dest_str),
+            exif_stat=(None, None, None),
+            converted_stat=converted_stat,
+            edited_stat=edited_stat,
+            info_json=self.json(),
+            exif_json=None,
         )
 
     if touched_files:
@@ -1045,12 +1118,19 @@ def _export_photo(
         fileutil.utime(dest, (ts, ts))
 
     return ExportResults(
-        exported_files + update_new_files + update_updated_files,
-        update_new_files,
-        update_updated_files,
-        update_skipped_files,
-        [],
-        touched_files,
+        exported=exported_files + update_new_files + update_updated_files,
+        new=update_new_files,
+        updated=update_updated_files,
+        skipped=update_skipped_files,
+        exif_updated=[],
+        touched=touched_files,
+        converted_to_jpeg=converted_to_jpeg_files,
+        sidecar_json_written=[],
+        sidecar_json_skipped=[],
+        sidecar_xmp_written=[],
+        sidecar_xmp_skipped=[],
+        missing=[],
+        error=[],
     )
 
 
@@ -1063,10 +1143,10 @@ def _write_exif_data(
     description_template=None,
     ignore_date_modified=False,
 ):
-    """ write exif data to image file at filepath
+    """write exif data to image file at filepath
 
     Args:
-        filepath: full path to the image file 
+        filepath: full path to the image file
         use_albums_as_keywords: treat album names as keywords
         use_persons_as_keywords: treat person names as keywords
         keyword_template: (list of strings); list of template strings to render as keywords
@@ -1084,9 +1164,7 @@ def _write_exif_data(
 
     with ExifTool(filepath) as exiftool:
         for exiftag, val in exif_info.items():
-            if exiftag == "_CreatedBy":
-                continue
-            elif type(val) == list:
+            if type(val) == list:
                 for v in val:
                     exiftool.setvalue(exiftag, v)
             else:
@@ -1101,7 +1179,7 @@ def _exiftool_dict(
     description_template=None,
     ignore_date_modified=False,
 ):
-    """ Return dict of EXIF details for building exiftool JSON sidecar or sending commands to ExifTool.
+    """Return dict of EXIF details for building exiftool JSON sidecar or sending commands to ExifTool.
         Does not include all the EXIF fields as those are likely already in the image.
 
     Args:
@@ -1114,25 +1192,28 @@ def _exiftool_dict(
     Returns: dict with exiftool tags / values
 
     Exports the following:
-        EXIF:ImageDescription
+        EXIF:ImageDescription (may include template)
         XMP:Description (may include template)
         XMP:Title
-        XMP:TagsList
+        XMP:TagsList (may include album name, person name, or template)
         IPTC:Keywords (may include album name, person name, or template)
-        XMP:Subject
+        XMP:Subject (set to keywords + persons)
         XMP:PersonInImage
+        EXIF:GPSLatitudeRef, EXIF:GPSLongitudeRef
         EXIF:GPSLatitude, EXIF:GPSLongitude
         EXIF:GPSPosition
-        EXIF:GPSLatitudeRef, EXIF:GPSLongitudeRef
         EXIF:DateTimeOriginal
         EXIF:OffsetTimeOriginal
         EXIF:ModifyDate
         IPTC:DateCreated
         IPTC:TimeCreated
+        QuickTime:CreationDate (UTC)
+        QuickTime:ModifyDate (UTC)
+        QuickTime:GPSCoordinates
+        UserData:GPSCoordinates
     """
 
     exif = {}
-    exif["_CreatedBy"] = "osxphotos, https://github.com/RhetTbull/osxphotos"
     if description_template is not None:
         description = self.render_template(
             description_template, expand_inplace=True, inplace_sep=", "
@@ -1210,12 +1291,16 @@ def _exiftool_dict(
 
     (lat, lon) = self.location
     if lat is not None and lon is not None:
-        exif["EXIF:GPSLatitude"] = lat
-        exif["EXIF:GPSLongitude"] = lon
-        lat_ref = "N" if lat >= 0 else "S"
-        lon_ref = "E" if lon >= 0 else "W"
-        exif["EXIF:GPSLatitudeRef"] = lat_ref
-        exif["EXIF:GPSLongitudeRef"] = lon_ref
+        if self.isphoto:
+            exif["EXIF:GPSLatitude"] = lat
+            exif["EXIF:GPSLongitude"] = lon
+            lat_ref = "N" if lat >= 0 else "S"
+            lon_ref = "E" if lon >= 0 else "W"
+            exif["EXIF:GPSLatitudeRef"] = lat_ref
+            exif["EXIF:GPSLongitudeRef"] = lon_ref
+        elif self.ismovie:
+            exif["Keys:GPSCoordinates"] = f"{lat} {lon}"
+            exif["UserData:GPSCoordinates"] = f"{lat} {lon}"
 
     # process date/time and timezone offset
     # Photos exports the following fields and sets modify date to creation date
@@ -1227,30 +1312,45 @@ def _exiftool_dict(
     #
     # This code deviates from Photos in one regard:
     # if photo has modification date, use it otherwise use creation date
-    date = self.date
 
-    # exiftool expects format to "2015:01:18 12:00:00"
-    datetimeoriginal = date.strftime("%Y:%m:%d %H:%M:%S")
-    exif["EXIF:DateTimeOriginal"] = datetimeoriginal
-    exif["EXIF:CreateDate"] = datetimeoriginal
+    if self.isphoto:
+        date = self.date
+        # exiftool expects format to "2015:01:18 12:00:00"
+        datetimeoriginal = date.strftime("%Y:%m:%d %H:%M:%S")
 
-    offsettime = date.strftime("%z")
-    # find timezone offset in format "-04:00"
-    offset = re.findall(r"([+-]?)([\d]{2})([\d]{2})", offsettime)
-    offset = offset[0]  # findall returns list of tuples
-    offsettime = f"{offset[0]}{offset[1]}:{offset[2]}"
-    exif["EXIF:OffsetTimeOriginal"] = offsettime
+        exif["EXIF:DateTimeOriginal"] = datetimeoriginal
+        exif["EXIF:CreateDate"] = datetimeoriginal
 
-    dateoriginal = date.strftime("%Y:%m:%d")
-    exif["IPTC:DateCreated"] = dateoriginal
+        offsettime = date.strftime("%z")
+        # find timezone offset in format "-04:00"
+        offset = re.findall(r"([+-]?)([\d]{2})([\d]{2})", offsettime)
+        offset = offset[0]  # findall returns list of tuples
+        offsettime = f"{offset[0]}{offset[1]}:{offset[2]}"
+        exif["EXIF:OffsetTimeOriginal"] = offsettime
 
-    timeoriginal = date.strftime(f"%H:%M:%S{offsettime}")
-    exif["IPTC:TimeCreated"] = timeoriginal
+        dateoriginal = date.strftime("%Y:%m:%d")
+        exif["IPTC:DateCreated"] = dateoriginal
 
-    if self.date_modified is not None and not ignore_date_modified:
-        exif["EXIF:ModifyDate"] = self.date_modified.strftime("%Y:%m:%d %H:%M:%S")
-    else:
-        exif["EXIF:ModifyDate"] = self.date.strftime("%Y:%m:%d %H:%M:%S")
+        timeoriginal = date.strftime(f"%H:%M:%S{offsettime}")
+        exif["IPTC:TimeCreated"] = timeoriginal
+
+        if self.date_modified is not None and not ignore_date_modified:
+            exif["EXIF:ModifyDate"] = self.date_modified.strftime("%Y:%m:%d %H:%M:%S")
+        else:
+            exif["EXIF:ModifyDate"] = self.date.strftime("%Y:%m:%d %H:%M:%S")
+    elif self.ismovie:
+        # QuickTime spec specifies times in UTC
+        # reference: https://exiftool.org/TagNames/QuickTime.html#Keys
+        date_utc = datetime_tz_to_utc(self.date)
+        creationdate = date_utc.strftime("%Y:%m:%d %H:%M:%S")
+        exif["QuickTime:CreationDate"] = creationdate
+        exif["QuickTime:CreateDate"] = creationdate
+        if self.date_modified is not None and not ignore_date_modified:
+            exif["QuickTime:ModifyDate"] = datetime_tz_to_utc(
+                self.date_modified
+            ).strftime("%Y:%m:%d %H:%M:%S")
+        else:
+            exif["QuickTime:ModifyDate"] = creationdate
 
     return exif
 
@@ -1263,7 +1363,7 @@ def _exiftool_json_sidecar(
     description_template=None,
     ignore_date_modified=False,
 ):
-    """ Return dict of EXIF details for building exiftool JSON sidecar or sending commands to ExifTool.
+    """Return dict of EXIF details for building exiftool JSON sidecar or sending commands to ExifTool.
         Does not include all the EXIF fields as those are likely already in the image.
 
     Args:
@@ -1281,16 +1381,20 @@ def _exiftool_json_sidecar(
         XMP:Title
         XMP:TagsList
         IPTC:Keywords (may include album name, person name, or template)
-        XMP:Subject
+        XMP:Subject (set to keywords + person)
         XMP:PersonInImage
+        EXIF:GPSLatitudeRef, EXIF:GPSLongitudeRef
         EXIF:GPSLatitude, EXIF:GPSLongitude
         EXIF:GPSPosition
-        EXIF:GPSLatitudeRef, EXIF:GPSLongitudeRef
         EXIF:DateTimeOriginal
         EXIF:OffsetTimeOriginal
         EXIF:ModifyDate
         IPTC:DigitalCreationDate
         IPTC:DateCreated
+        QuickTime:CreationDate (UTC)
+        QuickTime:ModifyDate (UTC)
+        QuickTime:GPSCoordinates
+        UserData:GPSCoordinates
     """
     exif = self._exiftool_dict(
         use_albums_as_keywords=use_albums_as_keywords,
@@ -1310,11 +1414,11 @@ def _xmp_sidecar(
     description_template=None,
     extension=None,
 ):
-    """ returns string for XMP sidecar 
-        use_albums_as_keywords: treat album names as keywords
-        use_persons_as_keywords: treat person names as keywords
-        keyword_template: (list of strings); list of template strings to render as keywords 
-        description_template: string; optional template string that will be rendered for use as photo description """
+    """returns string for XMP sidecar
+    use_albums_as_keywords: treat album names as keywords
+    use_persons_as_keywords: treat person names as keywords
+    keyword_template: (list of strings); list of template strings to render as keywords
+    description_template: string; optional template string that will be rendered for use as photo description"""
 
     xmp_template = Template(filename=os.path.join(_TEMPLATE_DIR, _XMP_TEMPLATE_NAME))
 
@@ -1399,8 +1503,8 @@ def _xmp_sidecar(
 
 
 def _write_sidecar(self, filename, sidecar_str):
-    """ write sidecar_str to filename
-        used for exporting sidecar info """
+    """write sidecar_str to filename
+    used for exporting sidecar info"""
     if not (filename or sidecar_str):
         raise (
             ValueError(

osxphotos/phototemplate.py

@@ -18,6 +18,7 @@ from functools import partial
 
 from ._constants import _UNKNOWN_PERSON
 from .datetime_formatter import DateTimeFormatter
+from .exiftool import ExifTool
 from .path_utils import sanitize_dirname, sanitize_filename, sanitize_pathpart
 
 # ensure locale set to user's locale
@@ -126,6 +127,10 @@ TEMPLATE_SUBSTITUTIONS_MULTI_VALUED = {
     "{label}": "Image categorization label associated with a photo (Photos 5 only)",
     "{label_normalized}": "All lower case version of 'label' (Photos 5 only)",
     "{comment}": "Comment(s) on shared Photos; format is 'Person name: comment text' (Photos 5 only)",
+    "{exiftool:GROUP:TAGNAME}": "Use exiftool (https://exiftool.org) to extract metadata, in form GROUP:TAGNAME, from image.  "
+    "E.g. '{exiftool:EXIF:Make}' to get camera make, or {exiftool:IPTC:Keywords} to extract keywords. "
+    "See https://exiftool.org/TagNames/ for list of valid tag names.  You must specify group (e.g. EXIF, IPTC, etc) "
+    "as used in `exiftool -G`. exiftool must be installed in the path to use this template.",
 }
 
 # Just the multi-valued substitution names without the braces
@@ -150,6 +155,62 @@ class PhotoTemplate:
         # gets initialized in get_template_value
         self.today = None
 
+    def make_subst_function(
+        self, none_str, filename, dirname, replacement, get_func=None
+    ):
+        """ returns: substitution function for use in re.sub 
+            none_str: value to use if substitution lookup is None and no default provided
+            get_func: function that gets the substitution value for a given template field
+                    default is get_template_value which handles the single-value fields """
+
+        if get_func is None:
+            # used by make_subst_function to get the value for a template substitution
+            get_func = partial(
+                self.get_template_value,
+                filename=filename,
+                dirname=dirname,
+                replacement=replacement,
+            )
+
+        # closure to capture photo, none_str, filename, dirname in subst
+        def subst(matchobj):
+            groups = len(matchobj.groups())
+            if groups != 5:
+                raise ValueError(
+                    f"Unexpected number of groups: expected 4, got {groups}"
+                )
+
+            delim = matchobj.group(1)
+            field = matchobj.group(2)
+            path_sep = matchobj.group(3)
+            bool_val = matchobj.group(4)
+            default = matchobj.group(5)
+
+            # drop the '+' on delim
+            delim = delim[:-1] if delim is not None else None
+            # drop () from path_sep
+            path_sep = path_sep.strip("()") if path_sep is not None else None
+            # drop the ? on bool_val
+            bool_val = bool_val[1:] if bool_val is not None else None
+            # drop the comma on default
+            default_val = default[1:] if default is not None else None
+
+            try:
+                val = get_func(field, default_val, bool_val, delim, path_sep)
+            except ValueError:
+                return matchobj.group(0)
+
+            if val is None:
+                # field valid but didn't match a value
+                if default == ",":
+                    val = ""
+                else:
+                    val = default_val if default_val is not None else none_str
+
+            return val
+
+        return subst
+
     def render(
         self,
         template,
@@ -208,60 +269,7 @@ class PhotoTemplate:
         if type(template) is not str:
             raise TypeError(f"template must be type str, not {type(template)}")
 
-        # used by make_subst_function to get the value for a template substitution
-        get_func = partial(
-            self.get_template_value,
-            filename=filename,
-            dirname=dirname,
-            replacement=replacement,
-        )
-
-        def make_subst_function(self, none_str, get_func=get_func):
-            """ returns: substitution function for use in re.sub 
-                none_str: value to use if substitution lookup is None and no default provided
-                get_func: function that gets the substitution value for a given template field
-                        default is get_template_value which handles the single-value fields """
-
-            # closure to capture photo, none_str, filename, dirname in subst
-            def subst(matchobj):
-                groups = len(matchobj.groups())
-                if groups == 5:
-                    delim = matchobj.group(1)
-                    field = matchobj.group(2)
-                    path_sep = matchobj.group(3)
-                    bool_val = matchobj.group(4)
-                    default = matchobj.group(5)
-
-                    # drop the '+' on delim
-                    delim = delim[:-1] if delim is not None else None
-                    # drop () from path_sep
-                    path_sep = path_sep.strip("()") if path_sep is not None else None
-                    # drop the ? on bool_val
-                    bool_val = bool_val[1:] if bool_val is not None else None
-                    # drop the comma on default
-                    default_val = default[1:] if default is not None else None
-
-                    try:
-                        val = get_func(field, default_val, bool_val, delim, path_sep)
-                    except ValueError:
-                        return matchobj.group(0)
-
-                    if val is None:
-                        # field valid but didn't match a value
-                        if default == ",":
-                            val = ""
-                        else:
-                            val = default_val if default_val is not None else none_str
-
-                    return val
-                else:
-                    raise ValueError(
-                        f"Unexpected number of groups: expected 4, got {groups}"
-                    )
-
-            return subst
-
-        subst_func = make_subst_function(self, none_str)
+        subst_func = self.make_subst_function(none_str, filename, dirname, replacement)
 
         # do the replacements
         rendered = re.sub(regex, subst_func, template)
@@ -289,88 +297,28 @@ class PhotoTemplate:
         #    '2011/Album2/keyword1/person1',
         #    '2011/Album2/keyword2/person1',]
 
-        rendered_strings = [rendered]
-        for field in MULTI_VALUE_SUBSTITUTIONS:
-            # Build a regex that matches only the field being processed
-            re_str = (
-                r"(?<!\{)\{"  # match { but not {{
-                + r"([^}]*\+)?"  # group 1: optional DELIM+
-                + r"("
-                + field  # group 2: field name
-                + r")"
-                + r"(\([^{}\)]*\))?"  # group 3: optional (PATH_SEP)
-                + r"(\?[^\\,}]*)?"  # group 4: optional ?TRUE_VALUE for boolean fields
-                + r"(,[\w\=\;\-\%. ]*)?"  # group 5: optional ,DEFAULT
-                + r"(?=\}(?!\}))\}"  # match } but not }}
-            )
-            regex_multi = re.compile(re_str)
+        rendered_strings = self._render_multi_valued_templates(
+            rendered,
+            none_str,
+            path_sep,
+            expand_inplace,
+            inplace_sep,
+            filename,
+            dirname,
+            replacement,
+        )
 
-            # holds each of the new rendered_strings, dict to avoid repeats (dict.keys())
-            new_strings = {}
-
-            for str_template in rendered_strings:
-                matches = regex_multi.search(str_template)
-                if matches:
-                    path_sep = (
-                        matches.group(3).strip("()")
-                        if matches.group(3) is not None
-                        else path_sep
-                    )
-                    values = self.get_template_value_multi(
-                        field,
-                        path_sep,
-                        filename=filename,
-                        dirname=dirname,
-                        replacement=replacement,
-                    )
-                    if expand_inplace or matches.group(1) is not None:
-                        delim = (
-                            matches.group(1)[:-1] if matches.group(1) is not None else inplace_sep 
-                        )
-                        # instead of returning multiple strings, join values into a single string
-                        val = delim.join(sorted(values)) if values and values[0] else None
-
-                        def lookup_template_value_multi(lookup_value, *_):
-                            """ Closure passed to make_subst_function get_func 
-                                    Capture val and field in the closure 
-                                    Allows make_subst_function to be re-used w/o modification
-                                    _ is not used but required so signature matches get_template_value """
-                            if lookup_value == field:
-                                return val
-                            else:
-                                raise ValueError(f"Unexpected value: {lookup_value}")
-
-                        subst = make_subst_function(
-                            self, none_str, get_func=lookup_template_value_multi
-                        )
-                        new_string = regex_multi.sub(subst, str_template)
-
-                        # update rendered_strings for the next field to process
-                        rendered_strings = {new_string}
-                    else:
-                        # create a new template string for each value
-                        for val in values:
-
-                            def lookup_template_value_multi(lookup_value, *_):
-                                """ Closure passed to make_subst_function get_func 
-                                    Capture val and field in the closure 
-                                    Allows make_subst_function to be re-used w/o modification
-                                    _ is not used but required so signature matches get_template_value """
-                                if lookup_value == field:
-                                    return val
-                                else:
-                                    raise ValueError(
-                                        f"Unexpected value: {lookup_value}"
-                                    )
-
-                            subst = make_subst_function(
-                                self, none_str, get_func=lookup_template_value_multi
-                            )
-                            new_string = regex_multi.sub(subst, str_template)
-                            new_strings[new_string] = 1
-
-                        # update rendered_strings for the next field to process
-                        rendered_strings = list(new_strings.keys())
+        # process exiftool: templates
+        rendered_strings = self._render_exiftool_template(
+            rendered_strings,
+            none_str,
+            path_sep,
+            expand_inplace,
+            inplace_sep,
+            filename,
+            dirname,
+            replacement,
+        )
 
         # find any {fields} that weren't replaced
         unmatched = []
@@ -396,6 +344,244 @@ class PhotoTemplate:
 
         return rendered_strings, unmatched
 
+    def _render_multi_valued_templates(
+        self,
+        rendered,
+        none_str,
+        path_sep,
+        expand_inplace,
+        inplace_sep,
+        filename,
+        dirname,
+        replacement,
+    ):
+        rendered_strings = [rendered]
+        new_rendered_strings = []
+        while new_rendered_strings != rendered_strings:
+            new_rendered_strings = rendered_strings
+            for field in MULTI_VALUE_SUBSTITUTIONS:
+                # Build a regex that matches only the field being processed
+                re_str = (
+                    r"(?<!\{)\{"  # match { but not {{
+                    + r"([^}]*\+)?"  # group 1: optional DELIM+
+                    + r"("
+                    + field  # group 2: field name
+                    + r")"
+                    + r"(\([^{}\)]*\))?"  # group 3: optional (PATH_SEP)
+                    + r"(\?[^\\,}]*)?"  # group 4: optional ?TRUE_VALUE for boolean fields
+                    + r"(,[\w\=\;\-\%. ]*)?"  # group 5: optional ,DEFAULT
+                    + r"(?=\}(?!\}))\}"  # match } but not }}
+                )
+                regex_multi = re.compile(re_str)
+
+                # holds each of the new rendered_strings, dict to avoid repeats (dict.keys())
+                new_strings = {}
+
+                for str_template in rendered_strings:
+                    matches = regex_multi.search(str_template)
+                    if matches:
+                        path_sep = (
+                            matches.group(3).strip("()")
+                            if matches.group(3) is not None
+                            else path_sep
+                        )
+                        values = self.get_template_value_multi(
+                            field,
+                            path_sep,
+                            filename=filename,
+                            dirname=dirname,
+                            replacement=replacement,
+                        )
+                        if expand_inplace or matches.group(1) is not None:
+                            delim = (
+                                matches.group(1)[:-1]
+                                if matches.group(1) is not None
+                                else inplace_sep
+                            )
+                            # instead of returning multiple strings, join values into a single string
+                            val = (
+                                delim.join(sorted(values))
+                                if values and values[0]
+                                else None
+                            )
+
+                            def lookup_template_value_multi(lookup_value, *_):
+                                """ Closure passed to make_subst_function get_func 
+                                        Capture val and field in the closure 
+                                        Allows make_subst_function to be re-used w/o modification
+                                        _ is not used but required so signature matches get_template_value """
+                                if lookup_value == field:
+                                    return val
+                                else:
+                                    raise ValueError(
+                                        f"Unexpected value: {lookup_value}"
+                                    )
+
+                            subst = self.make_subst_function(
+                                none_str,
+                                filename,
+                                dirname,
+                                replacement,
+                                get_func=lookup_template_value_multi,
+                            )
+                            new_string = regex_multi.sub(subst, str_template)
+
+                            # update rendered_strings for the next field to process
+                            rendered_strings = list({new_string})
+                        else:
+                            # create a new template string for each value
+                            for val in values:
+
+                                def lookup_template_value_multi(lookup_value, *_):
+                                    """ Closure passed to make_subst_function get_func 
+                                        Capture val and field in the closure 
+                                        Allows make_subst_function to be re-used w/o modification
+                                        _ is not used but required so signature matches get_template_value """
+                                    if lookup_value == field:
+                                        return val
+                                    else:
+                                        raise ValueError(
+                                            f"Unexpected value: {lookup_value}"
+                                        )
+
+                                subst = self.make_subst_function(
+                                    none_str,
+                                    filename,
+                                    dirname,
+                                    replacement,
+                                    get_func=lookup_template_value_multi,
+                                )
+                                new_string = regex_multi.sub(subst, str_template)
+                                new_strings[new_string] = 1
+
+                            # update rendered_strings for the next field to process
+                            rendered_strings = sorted(list(new_strings.keys()))
+        return rendered_strings
+
+    def _render_exiftool_template(
+        self,
+        rendered_strings,
+        none_str,
+        path_sep,
+        expand_inplace,
+        inplace_sep,
+        filename,
+        dirname,
+        replacement,
+    ):
+        # TODO: lots of code commonality with render_multi_valued_templates -- combine or pull out
+        # TODO: put these in globals
+        if path_sep is None:
+            path_sep = os.path.sep
+
+        if inplace_sep is None:
+            inplace_sep = ","
+
+        # Build a regex that matches only the field being processed
+        # todo: pull out regexes into globals?
+        re_str = (
+            r"(?<!\{)\{"  # match { but not {{
+            + r"([^}]*\+)?"  # group 1: optional DELIM+
+            + r"(exiftool:[^\\,}+\?]+)"  # group 3 field name
+            + r"(\([^{}\)]*\))?"  # group 3: optional (PATH_SEP)
+            + r"(\?[^\\,}]*)?"  # group 4: optional ?TRUE_VALUE for boolean fields
+            + r"(,[\w\=\;\-\%. ]*)?"  # group 5: optional ,DEFAULT
+            + r"(?=\}(?!\}))\}"  # match } but not }}
+        )
+        regex_multi = re.compile(re_str)
+
+        # holds each of the new rendered_strings, dict to avoid repeats (dict.keys())
+        new_rendered_strings = []
+        while new_rendered_strings != rendered_strings:
+            new_rendered_strings = rendered_strings
+            new_strings = {}
+            for str_template in rendered_strings:
+                matches = regex_multi.search(str_template)
+                if matches:
+                    # allmatches = regex_multi.finditer(str_template)
+                    # for matches in allmatches:
+                    path_sep = (
+                        matches.group(3).strip("()")
+                        if matches.group(3) is not None
+                        else path_sep
+                    )
+                    field = matches.group(2)
+                    subfield = field[9:]
+
+                    if not self.photo.path:
+                        values = []
+                    else:
+                        exif = ExifTool(self.photo.path)
+                        exifdict = exif.asdict()
+                        exifdict = {k.lower(): v for (k, v) in exifdict.items()}
+                        subfield = subfield.lower()
+                        if subfield in exifdict:
+                            values = exifdict[subfield]
+                            values = (
+                                [values] if not isinstance(values, list) else values
+                            )
+                        else:
+                            values = [None]
+                    if expand_inplace or matches.group(1) is not None:
+                        delim = (
+                            matches.group(1)[:-1]
+                            if matches.group(1) is not None
+                            else inplace_sep
+                        )
+                        # instead of returning multiple strings, join values into a single string
+                        val = (
+                            delim.join(sorted(values)) if values and values[0] else None
+                        )
+
+                        def lookup_template_value_exif(lookup_value, *_):
+                            """ Closure passed to make_subst_function get_func 
+                                    Capture val and field in the closure 
+                                    Allows make_subst_function to be re-used w/o modification
+                                    _ is not used but required so signature matches get_template_value """
+                            if lookup_value == field:
+                                return val
+                            else:
+                                raise ValueError(f"Unexpected value: {lookup_value}")
+
+                        subst = self.make_subst_function(
+                            none_str,
+                            filename,
+                            dirname,
+                            replacement,
+                            get_func=lookup_template_value_exif,
+                        )
+                        new_string = regex_multi.sub(subst, str_template)
+                        # update rendered_strings for the next field to process
+                        rendered_strings = list({new_string})
+                    else:
+                        # create a new template string for each value
+                        for val in values:
+
+                            def lookup_template_value_exif(lookup_value, *_):
+                                """ Closure passed to make_subst_function get_func 
+                                    Capture val and field in the closure 
+                                    Allows make_subst_function to be re-used w/o modification
+                                    _ is not used but required so signature matches get_template_value """
+                                if lookup_value == field:
+                                    return val
+                                else:
+                                    raise ValueError(
+                                        f"Unexpected value: {lookup_value}"
+                                    )
+
+                            subst = self.make_subst_function(
+                                none_str,
+                                filename,
+                                dirname,
+                                replacement,
+                                get_func=lookup_template_value_exif,
+                            )
+                            new_string = regex_multi.sub(subst, str_template)
+                            new_strings[new_string] = 1
+                        # update rendered_strings for the next field to process
+                        rendered_strings = sorted(list(new_strings.keys()))
+        return rendered_strings
+
     def get_template_value(
         self,
         field,
@@ -681,6 +867,7 @@ class PhotoTemplate:
         """
 
         """ return list of values for a multi-valued template field """
+        values = [] 
         if field == "album":
             values = self.photo.albums
         elif field == "keyword":
@@ -724,7 +911,7 @@ class PhotoTemplate:
             values = [
                 f"{comment.user}: {comment.text}" for comment in self.photo.comments
             ]
-        else:
+        elif not field.startswith("exiftool:"):
             raise ValueError(f"Unhandled template value: {field}")
 
         # sanitize directory names if needed, folder_album handled differently above

tests/Test-10.15.7.photoslibrary/database/Photos.sqlite.lock

@@ -7,7 +7,7 @@
 	<key>hostuuid</key>
 	<string>9575E48B-8D5F-5654-ABAC-4431B1167324</string>
 	<key>pid</key>
-	<integer>1797</integer>
+	<integer>464</integer>
 	<key>processname</key>
 	<string>photolibraryd</string>
 	<key>uid</key>

tests/test_cli.py

@@ -17,6 +17,7 @@ PLACES_PHOTOS_DB_13 = "tests/Test-Places-High-Sierra-10.13.6.photoslibrary"
 PHOTOS_DB_15_4 = "tests/Test-10.15.4.photoslibrary"
 PHOTOS_DB_15_5 = "tests/Test-10.15.5.photoslibrary"
 PHOTOS_DB_15_6 = "tests/Test-10.15.6.photoslibrary"
+PHOTOS_DB_15_7 = "tests/Test-10.15.7.photoslibrary"
 PHOTOS_DB_TOUCH = PHOTOS_DB_15_6
 PHOTOS_DB_14_6 = "tests/Test-10.14.6.photoslibrary"
 
@@ -65,6 +66,7 @@ CLI_EXPORT_FILENAMES_ALBUM_UNICODE = ["IMG_4547.jpg"]
 CLI_EXPORT_FILENAMES_DELETED_TWIN = ["wedding.jpg", "wedding_edited.jpeg"]
 
 CLI_EXPORT_EDITED_SUFFIX = "_bearbeiten"
+CLI_EXPORT_ORIGINAL_SUFFIX = "_original"
 
 CLI_EXPORT_FILENAMES_EDITED_SUFFIX = [
     "Pumkins1.jpg",
@@ -77,6 +79,16 @@ CLI_EXPORT_FILENAMES_EDITED_SUFFIX = [
     "wedding_bearbeiten.jpeg",
 ]
 
+CLI_EXPORT_FILENAMES_ORIGINAL_SUFFIX = [
+    "Pumkins1_original.jpg",
+    "Pumkins2_original.jpg",
+    "Pumpkins3_original.jpg",
+    "St James Park_original.jpg",
+    "St James Park_edited.jpeg",
+    "Tulips_original.jpg",
+    "wedding_original.jpg",
+    "wedding_edited.jpeg",
+]
 
 CLI_EXPORT_FILENAMES_CURRENT = [
     "1EB2B765-0765-43BA-A90C-0D0580E6172C.jpeg",
@@ -324,6 +336,31 @@ CLI_EXIFTOOL = {
     }
 }
 
+CLI_EXIFTOOL_QUICKTIME = {
+    "35329C57-B963-48D6-BB75-6AFF9370CBBC": {
+        "File:FileName": "Jellyfish.MOV",
+        "XMP:Description": "Jellyfish Video",
+        "XMP:Title": "Jellyfish",
+        "XMP:TagsList": "Travel",
+        "XMP:Subject": "Travel",
+        "QuickTime:GPSCoordinates": "34.053345 -118.242349",
+        "QuickTime:CreationDate": "2020:01:05 22:13:13",
+        "QuickTime:CreateDate": "2020:01:05 22:13:13",
+        "QuickTime:ModifyDate": "2020:01:05 22:13:13",
+    },
+    "2CE332F2-D578-4769-AEFA-7631BB77AA41": {
+        "File:FileName": "Jellyfish.mp4",
+        "XMP:Description": "Jellyfish Video",
+        "XMP:Title": "Jellyfish",
+        "XMP:TagsList": "Travel",
+        "XMP:Subject": "Travel",
+        "QuickTime:GPSCoordinates": "34.053345 -118.242349",
+        "QuickTime:CreationDate": "2020:12:05 05:21:52",
+        "QuickTime:CreateDate": "2020:12:05 05:21:52",
+        "QuickTime:ModifyDate": "2020:12:05 05:21:52",
+    },
+}
+
 CLI_EXIFTOOL_IGNORE_DATE_MODIFIED = {
     "E9BC5C36-7CD1-40A1-A72B-8B8FAC227D51": {
         "File:FileName": "wedding.jpg",
@@ -976,11 +1013,53 @@ def test_export_exiftool_ignore_date_modified():
             )
             assert result.exit_code == 0
 
-            exif = ExifTool(CLI_EXIFTOOL_IGNORE_DATE_MODIFIED[uuid]["File:FileName"]).asdict()
+            exif = ExifTool(
+                CLI_EXIFTOOL_IGNORE_DATE_MODIFIED[uuid]["File:FileName"]
+            ).asdict()
             for key in CLI_EXIFTOOL_IGNORE_DATE_MODIFIED[uuid]:
                 assert exif[key] == CLI_EXIFTOOL_IGNORE_DATE_MODIFIED[uuid][key]
 
 
+@pytest.mark.skipif(exiftool is None, reason="exiftool not installed")
+def test_export_exiftool_quicktime():
+    """ test --exiftol correctly writes QuickTime tags """
+    import glob
+    import os
+    import os.path
+    from osxphotos.__main__ import export
+    from osxphotos.exiftool import ExifTool
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with runner.isolated_filesystem():
+        for uuid in CLI_EXIFTOOL_QUICKTIME:
+            result = runner.invoke(
+                export,
+                [
+                    os.path.join(cwd, PHOTOS_DB_15_7),
+                    ".",
+                    "-V",
+                    "--exiftool",
+                    "--uuid",
+                    f"{uuid}",
+                ],
+            )
+            assert result.exit_code == 0
+            files = glob.glob("*")
+            assert sorted(files) == sorted(
+                [CLI_EXIFTOOL_QUICKTIME[uuid]["File:FileName"]]
+            )
+
+            exif = ExifTool(CLI_EXIFTOOL_QUICKTIME[uuid]["File:FileName"]).asdict()
+            for key in CLI_EXIFTOOL_QUICKTIME[uuid]:
+                assert exif[key] == CLI_EXIFTOOL_QUICKTIME[uuid][key]
+
+            # clean up exported files to avoid name conflicts
+            for filename in files:
+                os.unlink(filename)
+
+
 def test_export_edited_suffix():
     """ test export with --edited-suffix """
     import glob
@@ -1008,6 +1087,33 @@ def test_export_edited_suffix():
         assert sorted(files) == sorted(CLI_EXPORT_FILENAMES_EDITED_SUFFIX)
 
 
+def test_export_original_suffix():
+    """ test export with --original-suffix """
+    import glob
+    import os
+    import os.path
+    import osxphotos
+    from osxphotos.__main__ import export
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with runner.isolated_filesystem():
+        result = runner.invoke(
+            export,
+            [
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "--original-suffix",
+                CLI_EXPORT_ORIGINAL_SUFFIX,
+                "-V",
+            ],
+        )
+        assert result.exit_code == 0
+        files = glob.glob("*")
+        assert sorted(files) == sorted(CLI_EXPORT_FILENAMES_ORIGINAL_SUFFIX)
+
+
 @pytest.mark.skipif(
     "OSXPHOTOS_TEST_CONVERT" not in os.environ,
     reason="Skip if running in Github actions, no GPU.",
@@ -1734,6 +1840,142 @@ def test_export_sidecar_templates():
         )
 
 
+def test_export_sidecar_update():
+    """ test sidecar don't update if not changed and do update if changed """
+    import datetime
+    import glob
+    import os
+    import os.path
+    import osxphotos
+    from osxphotos.fileutil import FileUtil
+
+    from osxphotos.__main__ import cli
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with runner.isolated_filesystem():
+        result = runner.invoke(
+            cli,
+            [
+                "export",
+                "--db",
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "--sidecar=json",
+                "--sidecar=xmp",
+                f"--uuid={CLI_EXPORT_UUID}",
+                "-V",
+            ],
+        )
+        assert result.exit_code == 0
+        assert "Writing XMP sidecar" in result.output
+        assert "Writing exiftool JSON sidecar" in result.output
+
+        # delete a sidecar file and run update
+        fileutil = FileUtil()
+        fileutil.unlink(CLI_EXPORT_SIDECAR_FILENAMES[1])
+
+        result = runner.invoke(
+            cli,
+            [
+                "export",
+                "--db",
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "--sidecar=json",
+                "--sidecar=xmp",
+                f"--uuid={CLI_EXPORT_UUID}",
+                "-V",
+                "--update",
+            ],
+        )
+        assert result.exit_code == 0
+        assert "Skipped up to date XMP sidecar" in result.output
+        assert "Writing exiftool JSON sidecar" in result.output
+
+        # run update again, no sidecar files should update
+        result = runner.invoke(
+            cli,
+            [
+                "export",
+                "--db",
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "--sidecar=json",
+                "--sidecar=xmp",
+                f"--uuid={CLI_EXPORT_UUID}",
+                "-V",
+                "--update",
+            ],
+        )
+        assert result.exit_code == 0
+        assert "Skipped up to date XMP sidecar" in result.output
+        assert "Skipped up to date exiftool JSON sidecar" in result.output
+
+        # touch a file and export again
+        ts = datetime.datetime.now().timestamp() + 1000
+        fileutil.utime(CLI_EXPORT_SIDECAR_FILENAMES[2], (ts, ts))
+
+        result = runner.invoke(
+            cli,
+            [
+                "export",
+                "--db",
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "--sidecar=json",
+                "--sidecar=xmp",
+                f"--uuid={CLI_EXPORT_UUID}",
+                "-V",
+                "--update",
+            ],
+        )
+        assert result.exit_code == 0
+        assert "Writing XMP sidecar" in result.output
+        assert "Skipped up to date exiftool JSON sidecar" in result.output
+
+        # run update again, no sidecar files should update
+        result = runner.invoke(
+            cli,
+            [
+                "export",
+                "--db",
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "--sidecar=json",
+                "--sidecar=xmp",
+                f"--uuid={CLI_EXPORT_UUID}",
+                "-V",
+                "--update",
+            ],
+        )
+        assert result.exit_code == 0
+        assert "Skipped up to date XMP sidecar" in result.output
+        assert "Skipped up to date exiftool JSON sidecar" in result.output
+
+        # run update again with updated metadata, forcing update
+        result = runner.invoke(
+            cli,
+            [
+                "export",
+                "--db",
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "--sidecar=json",
+                "--sidecar=xmp",
+                f"--uuid={CLI_EXPORT_UUID}",
+                "-V",
+                "--update",
+                "--keyword-template",
+                "foo",
+            ],
+        )
+        assert result.exit_code == 0
+        assert "Writing XMP sidecar" in result.output
+        assert "Writing exiftool JSON sidecar" in result.output
+
+
 def test_export_live():
     import glob
     import os
@@ -2682,8 +2924,7 @@ def test_export_sidecar_keyword_template():
 
         json_expected = json.loads(
             """
-            [{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", 
-            "EXIF:ImageDescription": "Girl holding pumpkin", 
+            [{"EXIF:ImageDescription": "Girl holding pumpkin", 
             "XMP:Description": "Girl holding pumpkin", 
             "XMP:Title": "I found one!", 
             "XMP:TagsList": ["Kids", "Multi Keyword", "Pumpkin Farm", "Test Album"], 
@@ -2742,7 +2983,7 @@ def test_export_update_basic():
         )
         assert result.exit_code == 0
         assert (
-            "Exported: 0 photos, updated: 0 photos, skipped: 8 photos, updated EXIF data: 0 photos"
+            "Processed: 7 photos, exported: 0, updated: 0, skipped: 8, updated EXIF data: 0, missing: 1, error: 0"
             in result.output
         )
 
@@ -2826,7 +3067,7 @@ def test_export_update_exiftool():
         )
         assert result.exit_code == 0
         assert (
-            "Exported: 0 photos, updated: 8 photos, skipped: 0 photos, updated EXIF data: 8 photos"
+            "Processed: 7 photos, exported: 0, updated: 8, skipped: 0, updated EXIF data: 8, missing: 1, error: 0"
             in result.output
         )
 
@@ -2836,7 +3077,7 @@ def test_export_update_exiftool():
         )
         assert result.exit_code == 0
         assert (
-            "Exported: 0 photos, updated: 0 photos, skipped: 8 photos, updated EXIF data: 0 photos"
+            "Processed: 7 photos, exported: 0, updated: 0, skipped: 8, updated EXIF data: 0, missing: 1, error: 0"
             in result.output
         )
 
@@ -2873,7 +3114,7 @@ def test_export_update_hardlink():
         )
         assert result.exit_code == 0
         assert (
-            "Exported: 0 photos, updated: 8 photos, skipped: 0 photos, updated EXIF data: 0 photos"
+            "Processed: 7 photos, exported: 0, updated: 8, skipped: 0, updated EXIF data: 0, missing: 1, error: 0"
             in result.output
         )
         assert not os.path.samefile(CLI_EXPORT_UUID_FILENAME, photo.path)
@@ -2912,7 +3153,7 @@ def test_export_update_hardlink_exiftool():
         )
         assert result.exit_code == 0
         assert (
-            "Exported: 0 photos, updated: 8 photos, skipped: 0 photos, updated EXIF data: 8 photos"
+            "Processed: 7 photos, exported: 0, updated: 8, skipped: 0, updated EXIF data: 8, missing: 1, error: 0"
             in result.output
         )
         assert not os.path.samefile(CLI_EXPORT_UUID_FILENAME, photo.path)
@@ -2950,7 +3191,7 @@ def test_export_update_edits():
         )
         assert result.exit_code == 0
         assert (
-            "Exported: 1 photo, updated: 1 photo, skipped: 6 photos, updated EXIF data: 0 photos"
+            "Processed: 7 photos, exported: 1, updated: 1, skipped: 6, updated EXIF data: 0, missing: 1, error: 0"
             in result.output
         )
 
@@ -2986,7 +3227,7 @@ def test_export_update_no_db():
         # edited files will be re-exported because there won't be an edited signature
         # in the database
         assert (
-            "Exported: 0 photos, updated: 2 photos, skipped: 6 photos, updated EXIF data: 0 photos"
+            "Processed: 7 photos, exported: 0, updated: 2, skipped: 6, updated EXIF data: 0, missing: 1, error: 0"
             in result.output
         )
         assert os.path.isfile(OSXPHOTOS_EXPORT_DB)
@@ -3025,7 +3266,7 @@ def test_export_then_hardlink():
             ],
         )
         assert result.exit_code == 0
-        assert "Exported: 8 photos" in result.output
+        assert "Processed: 7 photos, exported: 8, missing: 1, error: 0" in result.output
         assert os.path.samefile(CLI_EXPORT_UUID_FILENAME, photo.path)
 
 
@@ -3045,7 +3286,7 @@ def test_export_dry_run():
             export, [os.path.join(cwd, CLI_PHOTOS_DB), ".", "-V", "--dry-run"]
         )
         assert result.exit_code == 0
-        assert "Exported: 8 photos" in result.output
+        assert "Processed: 7 photos, exported: 8, missing: 1, error: 0" in result.output
         for filepath in CLI_EXPORT_FILENAMES:
             assert f"Exported {filepath}" in result.output
             assert not os.path.isfile(filepath)
@@ -3089,7 +3330,7 @@ def test_export_update_edits_dry_run():
         )
         assert result.exit_code == 0
         assert (
-            "Exported: 1 photo, updated: 1 photo, skipped: 6 photos, updated EXIF data: 0 photos"
+            "Processed: 7 photos, exported: 1, updated: 1, skipped: 6, updated EXIF data: 0, missing: 1, error: 0"
             in result.output
         )
 
@@ -3124,7 +3365,7 @@ def test_export_directory_template_1_dry_run():
             ],
         )
         assert result.exit_code == 0
-        assert "Exported: 8 photos" in result.output
+        assert "exported: 8" in result.output
         workdir = os.getcwd()
         for filepath in CLI_EXPORTED_DIRECTORY_TEMPLATE_FILENAMES1:
             assert f"Exported {filepath}" in result.output
@@ -3160,7 +3401,8 @@ def test_export_touch_files():
         )
         assert result.exit_code == 0
 
-        assert "Exported: 18 photos, touched date: 16 photos" in result.output
+        assert "exported: 18" in result.output
+        assert "touched date: 16" in result.output
 
         for fname, mtime in zip(CLI_EXPORT_BY_DATE, CLI_EXPORT_BY_DATE_TOUCH_TIMES):
             st = os.stat(fname)
@@ -3192,7 +3434,7 @@ def test_export_touch_files_update():
         )
         assert result.exit_code == 0
 
-        assert "Exported: 18 photos" in result.output
+        assert "exported: 18" in result.output
 
         assert not pathlib.Path(CLI_EXPORT_BY_DATE[0]).is_file()
 
@@ -3202,7 +3444,7 @@ def test_export_touch_files_update():
         )
         assert result.exit_code == 0
 
-        assert "Exported: 18 photos" in result.output
+        assert "exported: 18" in result.output
 
         assert pathlib.Path(CLI_EXPORT_BY_DATE[0]).is_file()
 
@@ -3213,10 +3455,7 @@ def test_export_touch_files_update():
         )
         assert result.exit_code == 0
 
-        assert (
-            "Exported: 0 photos, updated: 0 photos, skipped: 18 photos, updated EXIF data: 0 photos"
-            in result.output
-        )
+        assert "skipped: 18" in result.output
 
         # --update --touch-file --dry-run
         result = runner.invoke(
@@ -3231,10 +3470,8 @@ def test_export_touch_files_update():
             ],
         )
         assert result.exit_code == 0
-        assert (
-            "Exported: 0 photos, updated: 0 photos, skipped: 18 photos, updated EXIF data: 0 photos, touched date: 16 photos"
-            in result.output
-        )
+        assert "skipped: 18" in result.output
+        assert "touched date: 16" in result.output
 
         for fname, mtime in zip(
             CLI_EXPORT_BY_DATE_NEED_TOUCH, CLI_EXPORT_BY_DATE_NEED_TOUCH_TIMES
@@ -3254,10 +3491,8 @@ def test_export_touch_files_update():
             ],
         )
         assert result.exit_code == 0
-        assert (
-            "Exported: 0 photos, updated: 0 photos, skipped: 18 photos, updated EXIF data: 0 photos, touched date: 16 photos"
-            in result.output
-        )
+        assert "skipped: 18" in result.output
+        assert "touched date: 16" in result.output
 
         for fname, mtime in zip(
             CLI_EXPORT_BY_DATE_NEED_TOUCH, CLI_EXPORT_BY_DATE_NEED_TOUCH_TIMES
@@ -3280,10 +3515,8 @@ def test_export_touch_files_update():
             ],
         )
         assert result.exit_code == 0
-        assert (
-            "Exported: 0 photos, updated: 1 photo, skipped: 17 photos, updated EXIF data: 0 photos, touched date: 1 photo"
-            in result.output
-        )
+        assert "updated: 1, skipped: 17" in result.output
+        assert "touched date: 1" in result.output
 
         for fname, mtime in zip(CLI_EXPORT_BY_DATE, CLI_EXPORT_BY_DATE_TOUCH_TIMES):
             st = os.stat(fname)
@@ -3296,10 +3529,7 @@ def test_export_touch_files_update():
         )
         assert result.exit_code == 0
 
-        assert (
-            "Exported: 0 photos, updated: 0 photos, skipped: 18 photos, updated EXIF data: 0 photos"
-            in result.output
-        )
+        assert "skipped: 18" in result.output
 
 
 @pytest.mark.skip("TODO: This fails on some machines but not all")
@@ -3329,7 +3559,7 @@ def test_export_touch_files_exiftool_update():
         )
         assert result.exit_code == 0
 
-        assert "Exported: 18 photos" in result.output
+        assert "exported: 18" in result.output
 
         assert not pathlib.Path(CLI_EXPORT_BY_DATE[0]).is_file()
 
@@ -3339,7 +3569,7 @@ def test_export_touch_files_exiftool_update():
         )
         assert result.exit_code == 0
 
-        assert "Exported: 18 photos" in result.output
+        assert "exported: 18" in result.output
 
         assert pathlib.Path(CLI_EXPORT_BY_DATE[0]).is_file()
 
@@ -3350,10 +3580,7 @@ def test_export_touch_files_exiftool_update():
         )
         assert result.exit_code == 0
 
-        assert (
-            "Exported: 0 photos, updated: 0 photos, skipped: 18 photos, updated EXIF data: 0 photos"
-            in result.output
-        )
+        assert "skipped: 18" in result.output
 
         # --update --exiftool --dry-run
         result = runner.invoke(
@@ -3369,10 +3596,8 @@ def test_export_touch_files_exiftool_update():
         )
         assert result.exit_code == 0
 
-        assert (
-            "Exported: 0 photos, updated: 18 photos, skipped: 0 photos, updated EXIF data: 18 photos"
-            in result.output
-        )
+        assert "updated: 18" in result.output
+        assert "updated EXIF data: 18" in result.output
 
         # --update --exiftool
         result = runner.invoke(
@@ -3386,11 +3611,8 @@ def test_export_touch_files_exiftool_update():
             ],
         )
         assert result.exit_code == 0
-
-        assert (
-            "Exported: 0 photos, updated: 18 photos, skipped: 0 photos, updated EXIF data: 18 photos"
-            in result.output
-        )
+        assert "updated: 18" in result.output
+        assert "updated EXIF data: 18" in result.output
 
         # --update --touch-file --exiftool --dry-run
         result = runner.invoke(
@@ -3406,10 +3628,8 @@ def test_export_touch_files_exiftool_update():
             ],
         )
         assert result.exit_code == 0
-        assert (
-            "Exported: 0 photos, updated: 0 photos, skipped: 18 photos, updated EXIF data: 0 photos, touched date: 18 photos"
-            in result.output
-        )
+        assert "skipped: 18" in result.output
+        assert "touched date: 18" in result.output
 
         # --update --touch-file --exiftool
         result = runner.invoke(
@@ -3424,10 +3644,8 @@ def test_export_touch_files_exiftool_update():
             ],
         )
         assert result.exit_code == 0
-        assert (
-            "Exported: 0 photos, updated: 0 photos, skipped: 18 photos, updated EXIF data: 0 photos, touched date: 18 photos"
-            in result.output
-        )
+        assert "skipped: 18" in result.output
+        assert "touched date: 18" in result.output
 
         for fname, mtime in zip(CLI_EXPORT_BY_DATE, CLI_EXPORT_BY_DATE_TOUCH_TIMES):
             st = os.stat(fname)
@@ -3449,10 +3667,10 @@ def test_export_touch_files_exiftool_update():
             ],
         )
         assert result.exit_code == 0
-        assert (
-            "Exported: 0 photos, updated: 1 photo, skipped: 17 photos, updated EXIF data: 1 photo, touched date: 1 photo"
-            in result.output
-        )
+        assert "updated: 1" in result.output
+        assert "skipped: 17" in result.output
+        assert "updated EXIF data: 1" in result.output
+        assert "touched date: 1" in result.output
 
         for fname, mtime in zip(CLI_EXPORT_BY_DATE, CLI_EXPORT_BY_DATE_TOUCH_TIMES):
             st = os.stat(fname)
@@ -3471,10 +3689,8 @@ def test_export_touch_files_exiftool_update():
             ],
         )
         assert result.exit_code == 0
-        assert (
-            "Exported: 0 photos, updated: 0 photos, skipped: 18 photos, updated EXIF data: 0 photos, touched date: 0 photos"
-            in result.output
-        )
+        assert "exported: 0" in result.output
+        assert "skipped: 18" in result.output
 
         # run update without --touch-file
         result = runner.invoke(
@@ -3489,10 +3705,8 @@ def test_export_touch_files_exiftool_update():
         )
         assert result.exit_code == 0
 
-        assert (
-            "Exported: 0 photos, updated: 0 photos, skipped: 18 photos, updated EXIF data: 0 photos"
-            in result.output
-        )
+        assert "exported: 0" in result.output
+        assert "skipped: 18" in result.output
 
 
 def test_labels():
@@ -3587,3 +3801,116 @@ def test_persons():
 
     json_got = json.loads(result.output)
     assert json_got == PERSONS_JSON
+
+
+def test_export_report():
+    """ test export with --report option """
+    import glob
+    import os
+    import os.path
+    import osxphotos
+    from osxphotos.__main__ import export
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with runner.isolated_filesystem():
+        result = runner.invoke(
+            export,
+            [os.path.join(cwd, CLI_PHOTOS_DB), ".", "-V", "--report", "report.csv"],
+        )
+        assert result.exit_code == 0
+        assert "Writing export report" in result.output
+        assert os.path.exists("report.csv")
+
+
+def test_export_report_not_a_file():
+    """ test export with --report option and bad report value """
+    import glob
+    import os
+    import os.path
+    import osxphotos
+    from osxphotos.__main__ import export
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with runner.isolated_filesystem():
+        result = runner.invoke(
+            export, [os.path.join(cwd, CLI_PHOTOS_DB), ".", "-V", "--report", "."]
+        )
+        assert result.exit_code != 0
+        assert "Aborted!" in result.output
+
+
+def test_export_as_hardlink_download_missing():
+    """ test export with incompatible export options """
+    import glob
+    import os
+    import os.path
+    import osxphotos
+    from osxphotos.__main__ import export
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with runner.isolated_filesystem():
+        result = runner.invoke(
+            export,
+            [
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "-V",
+                "--download-missing",
+                "--export-as-hardlink",
+                ".",
+            ],
+        )
+        assert result.exit_code != 0
+        assert "Aborted!" in result.output
+
+
+def test_export_missing():
+    """ test export with --missing """
+    import glob
+    import os
+    import os.path
+    import osxphotos
+    from osxphotos.__main__ import export
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with runner.isolated_filesystem():
+        result = runner.invoke(
+            export,
+            [
+                os.path.join(cwd, PHOTOS_DB_15_7),
+                ".",
+                "-V",
+                "--missing",
+                "--download-missing",
+                ".",
+            ],
+        )
+        assert result.exit_code == 0
+        assert "Exporting 2 photos" in result.output
+
+
+def test_export_missing_not_download_missing():
+    """ test export with incompatible export options """
+    import glob
+    import os
+    import os.path
+    import osxphotos
+    from osxphotos.__main__ import export
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with runner.isolated_filesystem():
+        result = runner.invoke(
+            export, [os.path.join(cwd, CLI_PHOTOS_DB), ".", "-V", "--missing", "."]
+        )
+        assert result.exit_code != 0
+        assert "Aborted!" in result.output

tests/test_datetime_utils.py

@@ -1,90 +1,96 @@
-""" test datetime_utils """
+from datetime import date, timezone
 import pytest
 
+from osxphotos.datetime_utils import *
+
 
 def test_get_local_tz():
-    """ test get_local_tz during time with no DST """
     import datetime
     import os
-    import time
-
-    from osxphotos.datetime_utils import get_local_tz
 
     os.environ["TZ"] = "US/Pacific"
-    time.tzset()
 
-    dt = datetime.datetime(2018, 12, 31, 0, 0, 0)
-    local_tz = get_local_tz(dt)
-    assert local_tz == datetime.timezone(
-        datetime.timedelta(days=-1, seconds=57600), "PST"
-    )
+    dt = datetime.datetime(2020, 9, 1, 21, 10, 00)
+    tz = get_local_tz(dt)
+    assert tz == datetime.timezone(offset=datetime.timedelta(seconds=-25200))
 
-
-def test_get_local_tz_dst():
-    """ test get_local_tz during time with DST """
-    import datetime
-    import os
-    import time
-
-    from osxphotos.datetime_utils import get_local_tz
-
-    os.environ["TZ"] = "US/Pacific"
-    time.tzset()
-
-    dt = datetime.datetime(2018, 6, 30, 0, 0, 0)
-    local_tz = get_local_tz(dt)
-    assert local_tz == datetime.timezone(
-        datetime.timedelta(days=-1, seconds=61200), "PDT"
-    )
-
-
-def test_datetime_remove_tz():
-    """ test datetime_remove_tz """
-    import datetime
-
-    from osxphotos.datetime_utils import datetime_remove_tz
-
-    dt = datetime.datetime(
-        2018,
-        12,
-        31,
-        tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=57600), "PST"),
-    )
-    dt_no_tz = datetime_remove_tz(dt)
-    assert dt_no_tz.tzinfo is None
+    dt = datetime.datetime(2020, 12, 1, 21, 10, 00)
+    tz = get_local_tz(dt)
+    assert tz == datetime.timezone(offset=datetime.timedelta(seconds=-28800))
 
 
 def test_datetime_has_tz():
-    """ test datetime_has_tz """
     import datetime
 
-    from osxphotos.datetime_utils import datetime_has_tz
-
-    dt = datetime.datetime(
-        2018,
-        12,
-        31,
-        tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=57600), "PST"),
-    )
+    tz = datetime.timezone(offset=datetime.timedelta(seconds=-28800))
+    dt = datetime.datetime(2020, 9, 1, 21, 10, 00, tzinfo=tz)
     assert datetime_has_tz(dt)
 
-    dt_notz = datetime.datetime(2018, 12, 31)
-    assert not datetime_has_tz(dt_notz)
+    dt = datetime.datetime(2020, 9, 1, 21, 10, 00)
+    assert not datetime_has_tz(dt)
+
+
+def test_datetime_tz_to_utc():
+    import datetime
+
+    tz = datetime.timezone(offset=datetime.timedelta(seconds=-25200))
+    dt = datetime.datetime(2020, 9, 1, 22, 6, 0, tzinfo=tz)
+    utc = datetime_tz_to_utc(dt)
+    assert utc == datetime.datetime(2020, 9, 2, 5, 6, 0, tzinfo=datetime.timezone.utc)
+
+
+def test_datetime_remove_tz():
+    import datetime
+    import os
+
+    os.environ["TZ"] = "US/Pacific"
+
+    tz = datetime.timezone(offset=datetime.timedelta(seconds=-25200))
+    dt = datetime.datetime(2020, 9, 1, 22, 6, 0, tzinfo=tz)
+    dt = datetime_remove_tz(dt)
+    assert dt == datetime.datetime(2020, 9, 1, 22, 6, 0)
+    assert not datetime_has_tz(dt)
+
+
+def test_datetime_naive_to_utc():
+    import datetime
+
+    dt = datetime.datetime(2020, 9, 1, 12, 0, 0)
+    utc = datetime_naive_to_utc(dt)
+    assert utc == datetime.datetime(2020, 9, 1, 12, 0, 0, tzinfo=datetime.timezone.utc)
 
 
 def test_datetime_naive_to_local():
-    """ test datetime_naive_to_local """
     import datetime
     import os
-    import time
-
-    from osxphotos.datetime_utils import datetime_naive_to_local
 
     os.environ["TZ"] = "US/Pacific"
-    time.tzset()
 
-    dt = datetime.datetime(2018, 6, 30, 0, 0, 0)
-    dt_local = datetime_naive_to_local(dt)
-    assert dt_local.tzinfo == datetime.timezone(
-        datetime.timedelta(days=-1, seconds=61200), "PDT"
-    )
+    tz = datetime.timezone(offset=datetime.timedelta(seconds=-25200))
+    dt = datetime.datetime(2020, 9, 1, 12, 0, 0)
+    utc = datetime_naive_to_local(dt)
+    assert utc == datetime.datetime(2020, 9, 1, 12, 0, 0, tzinfo=tz)
+
+
+def test_datetime_utc_to_local():
+    import datetime
+    import os
+
+    os.environ["TZ"] = "US/Pacific"
+
+    tz = datetime.timezone(offset=datetime.timedelta(seconds=-25200))
+    utc = datetime.datetime(2020, 9, 1, 19, 0, 0, tzinfo=datetime.timezone.utc)
+    dt = datetime_utc_to_local(utc)
+    assert dt == datetime.datetime(2020, 9, 1, 12, 0, 0, tzinfo=tz)
+
+
+def test_datetime_utc_to_local_2():
+    import datetime
+    import os
+
+    os.environ["TZ"] = "CEST"
+
+    tz = datetime.timezone(offset=datetime.timedelta(seconds=7200))
+    utc = datetime.datetime(2020, 9, 1, 19, 0, 0, tzinfo=datetime.timezone.utc)
+    dt = datetime_utc_to_local(utc)
+    assert dt == datetime.datetime(2020, 9, 1, 21, 0, 0, tzinfo=tz)

tests/test_export_catalina_10_15_7.py

@@ -68,8 +68,7 @@ XMP_JPG_FILENAME = "Pumkins1.jpg"
 
 EXIF_JSON_UUID = UUID_DICT["has_adjustments"]
 EXIF_JSON_EXPECTED = """ 
-    [{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", 
-    "EXIF:ImageDescription": "Bride Wedding day", 
+    [{"EXIF:ImageDescription": "Bride Wedding day", 
     "XMP:Description": "Bride Wedding day", 
     "XMP:TagsList": ["wedding"], 
     "IPTC:Keywords": ["wedding"], 
@@ -84,8 +83,7 @@ EXIF_JSON_EXPECTED = """
     """
 
 EXIF_JSON_EXPECTED_IGNORE_DATE_MODIFIED = """ 
-    [{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", 
-    "EXIF:ImageDescription": "Bride Wedding day", 
+    [{"EXIF:ImageDescription": "Bride Wedding day", 
     "XMP:Description": "Bride Wedding day", 
     "XMP:TagsList": ["wedding"], 
     "IPTC:Keywords": ["wedding"], 
@@ -544,8 +542,7 @@ def test_exiftool_json_sidecar_keyword_template_long(caplog):
 
     json_expected = json.loads(
         """
-        [{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", 
-        "EXIF:ImageDescription": "Bride Wedding day", 
+        [{"EXIF:ImageDescription": "Bride Wedding day", 
         "XMP:Description": "Bride Wedding day", 
         "XMP:TagsList": ["wedding", "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"], 
         "IPTC:Keywords": ["wedding", "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"], 
@@ -594,8 +591,7 @@ def test_exiftool_json_sidecar_keyword_template():
 
     json_expected = json.loads(
         """
-        [{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", 
-        "EXIF:ImageDescription": "Bride Wedding day", 
+        [{"EXIF:ImageDescription": "Bride Wedding day", 
         "XMP:Description": "Bride Wedding day", 
         "XMP:TagsList": ["wedding", "Folder1/SubFolder2/AlbumInFolder", "I have a deleted twin"], 
         "IPTC:Keywords": ["wedding", "Folder1/SubFolder2/AlbumInFolder", "I have a deleted twin"], 
@@ -655,8 +651,7 @@ def test_exiftool_json_sidecar_use_persons_keyword():
 
     json_expected = json.loads(
         """
-        [{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", 
-        "EXIF:ImageDescription": "Girls with pumpkins", 
+        [{"EXIF:ImageDescription": "Girls with pumpkins", 
         "XMP:Description": "Girls with pumpkins", 
         "XMP:Title": "Can we carry this?", 
         "XMP:TagsList": ["Kids", "Suzy", "Katie"], 
@@ -698,8 +693,7 @@ def test_exiftool_json_sidecar_use_albums_keyword():
 
     json_expected = json.loads(
         """
-        [{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", 
-        "EXIF:ImageDescription": "Girls with pumpkins", 
+        [{"EXIF:ImageDescription": "Girls with pumpkins", 
         "XMP:Description": "Girls with pumpkins", 
         "XMP:Title": "Can we carry this?", 
         "XMP:TagsList": ["Kids", "Pumpkin Farm", "Test Album"], 

tests/test_export_db.py

@@ -4,6 +4,8 @@ import pytest
 
 EXIF_DATA = """[{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", "EXIF:ImageDescription": "\u2068Elder Park\u2069, \u2068Adelaide\u2069, \u2068Australia\u2069", "XMP:Description": "\u2068Elder Park\u2069, \u2068Adelaide\u2069, \u2068Australia\u2069", "XMP:Title": "Elder Park", "EXIF:GPSLatitude": "34 deg 55' 8.01\" S", "EXIF:GPSLongitude": "138 deg 35' 48.70\" E", "Composite:GPSPosition": "34 deg 55' 8.01\" S, 138 deg 35' 48.70\" E", "EXIF:GPSLatitudeRef": "South", "EXIF:GPSLongitudeRef": "East", "EXIF:DateTimeOriginal": "2017:06:20 17:18:56", "EXIF:OffsetTimeOriginal": "+09:30", "EXIF:ModifyDate": "2020:05:18 14:42:04"}]"""
 INFO_DATA = """{"uuid": "3DD2C897-F19E-4CA6-8C22-B027D5A71907", "filename": "3DD2C897-F19E-4CA6-8C22-B027D5A71907.jpeg", "original_filename": "IMG_4547.jpg", "date": "2017-06-20T17:18:56.518000+09:30", "description": "\u2068Elder Park\u2069, \u2068Adelaide\u2069, \u2068Australia\u2069", "title": "Elder Park", "keywords": [], "labels": ["Statue", "Art"], "albums": ["AlbumInFolder"], "folders": {"AlbumInFolder": ["Folder1", "SubFolder2"]}, "persons": [], "path": "/Users/rhet/Pictures/Test-10.15.4.photoslibrary/originals/3/3DD2C897-F19E-4CA6-8C22-B027D5A71907.jpeg", "ismissing": false, "hasadjustments": true, "external_edit": false, "favorite": false, "hidden": false, "latitude": -34.91889167000001, "longitude": 138.59686167, "path_edited": "/Users/rhet/Pictures/Test-10.15.4.photoslibrary/resources/renders/3/3DD2C897-F19E-4CA6-8C22-B027D5A71907_1_201_a.jpeg", "shared": false, "isphoto": true, "ismovie": false, "uti": "public.jpeg", "burst": false, "live_photo": false, "path_live_photo": null, "iscloudasset": false, "incloud": null, "date_modified": "2020-05-18T14:42:04.608664+09:30", "portrait": false, "screenshot": false, "slow_mo": false, "time_lapse": false, "hdr": false, "selfie": false, "panorama": false, "has_raw": false, "uti_raw": null, "path_raw": null, "place": {"name": "Elder Park, Adelaide, South Australia, Australia, River Torrens", "names": {"field0": [], "country": ["Australia"], "state_province": ["South Australia"], "sub_administrative_area": ["Adelaide"], "city": ["Adelaide", "Adelaide"], "field5": [], "additional_city_info": ["Adelaide CBD", "Tarndanya"], "ocean": [], "area_of_interest": ["Elder Park", ""], "inland_water": ["River Torrens", "River Torrens"], "field10": [], "region": [], "sub_throughfare": [], "field13": [], "postal_code": [], "field15": [], "field16": [], "street_address": [], "body_of_water": ["River Torrens", "River Torrens"]}, "country_code": "AU", "ishome": false, "address_str": "River Torrens, Adelaide SA, Australia", "address": {"street": null, "sub_locality": "Tarndanya", "city": "Adelaide", "sub_administrative_area": "Adelaide", "state_province": "SA", "postal_code": null, "country": "Australia", "iso_country_code": "AU"}}, "exif": {"flash_fired": false, "iso": 320, "metering_mode": 3, "sample_rate": null, "track_format": null, "white_balance": 0, "aperture": 2.2, "bit_rate": null, "duration": null, "exposure_bias": 0.0, "focal_length": 4.15, "fps": null, "latitude": null, "longitude": null, "shutter_speed": 0.058823529411764705, "camera_make": "Apple", "camera_model": "iPhone 6s", "codec": null, "lens_model": "iPhone 6s back camera 4.15mm f/2.2"}}"""
+SIDECAR_DATA = """FOO_BAR"""
+
 EXIF_DATA2 = """[{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", "XMP:Title": "St. James's Park", "XMP:TagsList": ["London 2018", "St. James's Park", "England", "United Kingdom", "UK", "London"], "IPTC:Keywords": ["London 2018", "St. James's Park", "England", "United Kingdom", "UK", "London"], "XMP:Subject": ["London 2018", "St. James's Park", "England", "United Kingdom", "UK", "London"], "EXIF:GPSLatitude": "51 deg 30' 12.86\" N", "EXIF:GPSLongitude": "0 deg 7' 54.50\" W", "Composite:GPSPosition": "51 deg 30' 12.86\" N, 0 deg 7' 54.50\" W", "EXIF:GPSLatitudeRef": "North", "EXIF:GPSLongitudeRef": "West", "EXIF:DateTimeOriginal": "2018:10:13 09:18:12", "EXIF:OffsetTimeOriginal": "-04:00", "EXIF:ModifyDate": "2019:12:08 14:06:44"}]"""
 INFO_DATA2 = """{"uuid": "F2BB3F98-90F0-4E4C-A09B-25C6822A4529", "filename": "F2BB3F98-90F0-4E4C-A09B-25C6822A4529.jpeg", "original_filename": "IMG_8440.JPG", "date": "2019-06-11T11:42:06.711805-07:00", "description": null, "title": null, "keywords": [], "labels": ["Sky", "Cloudy", "Fence", "Land", "Outdoor", "Park", "Amusement Park", "Roller Coaster"], "albums": [], "folders": {}, "persons": [], "path": "/Volumes/MacBook Catalina - Data/Users/rhet/Pictures/Photos Library.photoslibrary/originals/F/F2BB3F98-90F0-4E4C-A09B-25C6822A4529.jpeg", "ismissing": false, "hasadjustments": false, "external_edit": false, "favorite": false, "hidden": false, "latitude": 33.81558666666667, "longitude": -117.99298, "path_edited": null, "shared": false, "isphoto": true, "ismovie": false, "uti": "public.jpeg", "burst": false, "live_photo": false, "path_live_photo": null, "iscloudasset": true, "incloud": true, "date_modified": "2019-10-14T00:51:47.141950-07:00", "portrait": false, "screenshot": false, "slow_mo": false, "time_lapse": false, "hdr": false, "selfie": false, "panorama": false, "has_raw": false, "uti_raw": null, "path_raw": null, "place": {"name": "Adventure City, Stanton, California, United States", "names": {"field0": [], "country": ["United States"], "state_province": ["California"], "sub_administrative_area": ["Orange"], "city": ["Stanton", "Anaheim", "Anaheim"], "field5": [], "additional_city_info": ["West Anaheim"], "ocean": [], "area_of_interest": ["Adventure City", "Adventure City"], "inland_water": [], "field10": [], "region": [], "sub_throughfare": [], "field13": [], "postal_code": [], "field15": [], "field16": [], "street_address": [], "body_of_water": []}, "country_code": "US", "ishome": false, "address_str": "Adventure City, 1240 S Beach Blvd, Anaheim, CA  92804, United States", "address": {"street": "1240 S Beach Blvd", "sub_locality": "West Anaheim", "city": "Stanton", "sub_administrative_area": "Orange", "state_province": "CA", "postal_code": "92804", "country": "United States", "iso_country_code": "US"}}, "exif": {"flash_fired": false, "iso": 25, "metering_mode": 5, "sample_rate": null, "track_format": null, "white_balance": 0, "aperture": 2.2, "bit_rate": null, "duration": null, "exposure_bias": 0.0, "focal_length": 4.15, "fps": null, "latitude": null, "longitude": null, "shutter_speed": 0.0004940711462450593, "camera_make": "Apple", "camera_model": "iPhone 6s", "codec": null, "lens_model": "iPhone 6s back camera 4.15mm f/2.2"}}"""
 DATABASE_VERSION1 = "tests/export_db_version1.db"
@@ -41,6 +43,8 @@ def test_export_db():
     assert db.get_stat_edited_for_file(filepath) == (10, 11, 12)
     db.set_stat_converted_for_file(filepath, (7, 8, 9))
     assert db.get_stat_converted_for_file(filepath) == (7, 8, 9)
+    db.set_sidecar_for_file(filepath, SIDECAR_DATA, (13, 14, 15))
+    assert db.get_sidecar_for_file(filepath) == (SIDECAR_DATA, (13, 14, 15))
 
     # test set_data which sets all at the same time
     filepath2 = os.path.join(tempdir.name, "test2.jpg")
@@ -109,6 +113,8 @@ def test_export_db_no_op():
     assert db.get_stat_converted_for_file(filepath) is None
     db.set_stat_edited_for_file(filepath, (10, 11, 12))
     assert db.get_stat_edited_for_file(filepath) is None
+    db.set_sidecar_for_file(filepath, SIDECAR_DATA, (13, 14, 15))
+    assert db.get_sidecar_for_file(filepath) == (None, (None, None, None))
 
     # test set_data which sets all at the same time
     filepath2 = os.path.join(tempdir.name, "test2.jpg")
@@ -160,6 +166,7 @@ def test_export_db_in_memory():
     db.set_stat_exif_for_file(filepath, (4, 5, 6))
     db.set_stat_converted_for_file(filepath, (7, 8, 9))
     db.set_stat_edited_for_file(filepath, (10, 11, 12))
+    db.set_sidecar_for_file(filepath, SIDECAR_DATA, (13, 14, 15))
 
     db.close()
 
@@ -176,6 +183,7 @@ def test_export_db_in_memory():
     assert dbram.get_stat_exif_for_file(filepath) == (4, 5, 6)
     assert dbram.get_stat_converted_for_file(filepath) == (7, 8, 9)
     assert dbram.get_stat_edited_for_file(filepath) == (10, 11, 12)
+    assert dbram.get_sidecar_for_file(filepath) == (SIDECAR_DATA, (13, 14, 15))
 
     # change a value
     dbram.set_uuid_for_file(filepath, "FUBAR")
@@ -185,6 +193,7 @@ def test_export_db_in_memory():
     dbram.set_stat_exif_for_file(filepath, (10, 11, 12))
     dbram.set_stat_converted_for_file(filepath, (1, 2, 3))
     dbram.set_stat_edited_for_file(filepath, (4, 5, 6))
+    dbram.set_sidecar_for_file(filepath, "FUBAR", (20, 21, 22))
 
     assert dbram.get_uuid_for_file(filepath_lower) == "FUBAR"
     assert dbram.get_info_for_uuid("FUBAR") == INFO_DATA2
@@ -193,6 +202,7 @@ def test_export_db_in_memory():
     assert dbram.get_stat_exif_for_file(filepath) == (10, 11, 12)
     assert dbram.get_stat_converted_for_file(filepath) == (1, 2, 3)
     assert dbram.get_stat_edited_for_file(filepath) == (4, 5, 6)
+    assert dbram.get_sidecar_for_file(filepath) == ("FUBAR", (20, 21, 22))
 
     dbram.close()
 
@@ -205,6 +215,7 @@ def test_export_db_in_memory():
     assert db.get_stat_exif_for_file(filepath) == (4, 5, 6)
     assert db.get_stat_converted_for_file(filepath) == (7, 8, 9)
     assert db.get_stat_edited_for_file(filepath) == (10, 11, 12)
+    assert db.get_sidecar_for_file(filepath) == (SIDECAR_DATA, (13, 14, 15))
 
     assert db.get_info_for_uuid("FUBAR") is None
 
@@ -232,6 +243,7 @@ def test_export_db_in_memory_nofile():
     dbram.set_stat_exif_for_file(filepath, (10, 11, 12))
     dbram.set_stat_converted_for_file(filepath, (1, 2, 3))
     dbram.set_stat_edited_for_file(filepath, (4, 5, 6))
+    dbram.set_sidecar_for_file(filepath, "FUBAR", (20, 21, 22))
 
     assert dbram.get_uuid_for_file(filepath_lower) == "FUBAR"
     assert dbram.get_info_for_uuid("FUBAR") == INFO_DATA2
@@ -240,5 +252,6 @@ def test_export_db_in_memory_nofile():
     assert dbram.get_stat_exif_for_file(filepath) == (10, 11, 12)
     assert dbram.get_stat_converted_for_file(filepath) == (1, 2, 3)
     assert dbram.get_stat_edited_for_file(filepath) == (4, 5, 6)
+    assert dbram.get_sidecar_for_file(filepath) == ("FUBAR", (20, 21, 22))
 
     dbram.close()

tests/test_export_mojave_10_14_6.py

@@ -46,8 +46,7 @@ UUID_DICT = {
 }
 
 EXIF_JSON_EXPECTED = """
-    [{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", 
-    "XMP:Title": "St. James\'s Park", 
+    [{"XMP:Title": "St. James\'s Park", 
     "XMP:TagsList": ["UK", "England", "London", "United Kingdom", "London 2018", "St. James\'s Park"], 
     "IPTC:Keywords": ["UK", "England", "London", "United Kingdom", "London 2018", "St. James\'s Park"], 
     "XMP:Subject": ["UK", "England", "London", "United Kingdom", "London 2018", "St. James\'s Park"], 

tests/test_template.py

@@ -1,6 +1,13 @@
 """ Test template.py """
 import pytest
 
+from osxphotos.exiftool import get_exiftool_path
+
+try:
+    exiftool = get_exiftool_path()
+except:
+    exiftool = None
+
 PHOTOS_DB_PLACES = (
     "./tests/Test-Places-Catalina-10_15_7.photoslibrary/database/photos.db"
 )
@@ -57,6 +64,29 @@ UUID_BOOL_VALUES = {"hdr": "D11D25FF-5F31-47D2-ABA9-58418878DC15"}
 # Boolean type values that render to False
 UUID_BOOL_VALUES_NOT = {"hdr": "51F2BEF7-431A-4D31-8AC1-3284A57826AE"}
 
+# for exiftool template
+UUID_EXIFTOOL = {
+    "A92D9C26-3A50-4197-9388-CB5F7DB9FA91": {
+        "{exiftool:EXIF:Make}": ["Canon"],
+        "{exiftool:EXIF:Model}": ["Canon PowerShot G10"],
+        "{exiftool:EXIF:Make}/{exiftool:EXIF:Model}": ["Canon/Canon PowerShot G10"],
+        "{exiftool:IPTC:Keywords,foo}": ["foo"],
+    },
+    "DC99FBDD-7A52-4100-A5BB-344131646C30": {
+        "{exiftool:IPTC:Keywords}": [
+            "England",
+            "London",
+            "London 2018",
+            "St. James's Park",
+            "UK",
+            "United Kingdom",
+        ],
+        "{,+exiftool:IPTC:Keywords}": [
+            "England,London,London 2018,St. James's Park,UK,United Kingdom"
+        ],
+    },
+}
+
 TEMPLATE_VALUES = {
     "{name}": "128FB4C6-0B16-4E7D-9108-FB2E90DA1546",
     "{original_name}": "IMG_1064",
@@ -737,3 +767,15 @@ def test_expand_in_place_with_delim_single_value():
     for template in TEMPLATE_VALUES_TITLE:
         rendered, _ = photo.render_template(template)
         assert sorted(rendered) == sorted(TEMPLATE_VALUES_TITLE[template])
+
+
+@pytest.mark.skipif(exiftool is None, reason="exiftool not installed")
+def test_exiftool_template():
+    import osxphotos
+
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
+    for uuid in UUID_EXIFTOOL:
+        photo = photosdb.get_photo(uuid)
+        for template in UUID_EXIFTOOL[uuid]:
+            rendered, _ = photo.render_template(template)
+            assert sorted(rendered) == sorted(UUID_EXIFTOOL[uuid][template])