Fix for issue #257, #275

Add XML escaping to XMP sidecar export, thanks to @jstrine for the fix!

.all-contributorsrc

@@ -100,6 +100,15 @@
       "contributions": [
         "code"
       ]
+    },
+    {
+      "login": "jstrine",
+      "name": "Jonathan Strine",
+      "avatar_url": "https://avatars1.githubusercontent.com/u/33943447?v=4",
+      "profile": "https://github.com/jstrine",
+      "contributions": [
+        "code"
+      ]
     }
   ],
   "contributorsPerLine": 7

CHANGELOG.md

@@ -4,6 +4,101 @@ All notable changes to this project will be documented in this file. Dates are d
 
 Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 
+#### [v0.37.4](https://github.com/RhetTbull/osxphotos/compare/v0.37.3...v0.37.4)
+
+> 5 December 2020
+
+- Merge branch 'master' of github.com:RhetTbull/osxphotos [`69cd236`](https://github.com/RhetTbull/osxphotos/commit/69cd2367122a3a86044df2845e706d3510bdf2c1)
+- Implement fix for issue #282, QuickTime metadata [`4cce9d4`](https://github.com/RhetTbull/osxphotos/commit/4cce9d4939a00ad2d265a510a2c6f0c8e6a8c655)
+- Implement fix for issue #282, QuickTime metadata [`cfb07cb`](https://github.com/RhetTbull/osxphotos/commit/cfb07cbfafaac493f6221be482c432812534ddfa)
+
+#### [v0.37.3](https://github.com/RhetTbull/osxphotos/compare/v0.37.2...v0.37.3)
+
+> 30 November 2020
+
+- Removed --use-photokit authorization check, issue 278 [`ed3a971`](https://github.com/RhetTbull/osxphotos/commit/ed3a9711dc0805aed1aacc30e01eeb9c1077d9e1)
+
+#### [v0.37.2](https://github.com/RhetTbull/osxphotos/compare/v0.37.1...v0.37.2)
+
+> 29 November 2020
+
+- Catch errors in export_photo [`d9dcf09`](https://github.com/RhetTbull/osxphotos/commit/d9dcf0917a541725d1e472e7f918733e4e2613d0)
+- Added --missing to export, see issue #277 [`25eacc7`](https://github.com/RhetTbull/osxphotos/commit/25eacc7caddd6721232b3f77a02532fcd35f7836)
+
+#### [v0.37.1](https://github.com/RhetTbull/osxphotos/compare/v0.37.0...v0.37.1)
+
+> 28 November 2020
+
+- Added --report option to CLI, implements #253 [`d22eaf3`](https://github.com/RhetTbull/osxphotos/commit/d22eaf39edc8b0b489b011d6d21345dcedcc8dff)
+- Updated template values [`af827d7`](https://github.com/RhetTbull/osxphotos/commit/af827d7a5769f41579d300a7cc511251d86b7eed)
+
+#### [v0.37.0](https://github.com/RhetTbull/osxphotos/compare/v0.36.25...v0.37.0)
+
+> 28 November 2020
+
+- Added {exiftool} template, implements issue #259 [`48acb42`](https://github.com/RhetTbull/osxphotos/commit/48acb42631226a71bfc636eea2d3151f1b7165f4)
+
+#### [v0.36.25](https://github.com/RhetTbull/osxphotos/compare/v0.36.24...v0.36.25)
+
+> 27 November 2020
+
+- Added --original-suffix for issue #263 [`399d432`](https://github.com/RhetTbull/osxphotos/commit/399d432a66354b9c235f30d10c6985fbde1b7e4f)
+
+#### [v0.36.24](https://github.com/RhetTbull/osxphotos/compare/v0.36.23...v0.36.24)
+
+> 26 November 2020
+
+- Initial implementation for issue #265 [`382fca3`](https://github.com/RhetTbull/osxphotos/commit/382fca3f92a3c251c12426dd0dc6d7dc21b691cf)
+- More work on issue #265 [`d5a9f76`](https://github.com/RhetTbull/osxphotos/commit/d5a9f767199d25ebd9d5925d05ee39ea7e51ac26)
+- Simplified sidecar table in export_db [`0632a97`](https://github.com/RhetTbull/osxphotos/commit/0632a97f55af67c7e5265b0d3283155c7c087e89)
+
+#### [v0.36.23](https://github.com/RhetTbull/osxphotos/compare/v0.36.22...v0.36.23)
+
+> 26 November 2020
+
+- Fix for missing original_filename, issue #267 [`fa33218`](https://github.com/RhetTbull/osxphotos/commit/fa332186ab3cdbe1bfd6496ff29b652ef984a5f8)
+- version bump [`b5195f9`](https://github.com/RhetTbull/osxphotos/commit/b5195f9d2b81cf6737b65e3cd3793ea9b0da13eb)
+- Updated test [`aa2ebf5`](https://github.com/RhetTbull/osxphotos/commit/aa2ebf55bb50eec14f86a532334b376e407f4bbc)
+
+#### [v0.36.22](https://github.com/RhetTbull/osxphotos/compare/v0.36.21...v0.36.22)
+
+> 26 November 2020
+
+- Add XML escaping to XMP sidecar export, thanks to @jstrine for the fix! [`#272`](https://github.com/RhetTbull/osxphotos/pull/272)
+- Fix EXIF GPS format for XMP sidecar, thanks to @jstrine for the fix! [`#270`](https://github.com/RhetTbull/osxphotos/pull/270)
+- Continue even if the original filename is None, thanks to @jstrine for the fix! [`#268`](https://github.com/RhetTbull/osxphotos/pull/268)
+- Added test for missing original_filename [`116cb66`](https://github.com/RhetTbull/osxphotos/commit/116cb662fbddf9153f6858c6ea97dc7f65c77705)
+-  Add @jstrine as a contributor [`7460bc8`](https://github.com/RhetTbull/osxphotos/commit/7460bc88fcc5e1e7435c9b9bcdf7ec9c7c5e39ea)
+- Escape characters which cause XML parsing issues [`c42050a`](https://github.com/RhetTbull/osxphotos/commit/c42050a10cac40b0b5ac70c587e07f257a9b50dd)
+- Fix tests for apostrophe [`d0d2e80`](https://github.com/RhetTbull/osxphotos/commit/d0d2e8080096bf66f93a830386800ce713680c51)
+- Fix test for XMP sidecar with GPS info [`c27cfb1`](https://github.com/RhetTbull/osxphotos/commit/c27cfb1223fa82b9e5549b93c283e9444693270a)
+
+#### [v0.36.21](https://github.com/RhetTbull/osxphotos/compare/v0.36.20...v0.36.21)
+
+> 25 November 2020
+
+- Exposed --use-photos-export and --use-photokit [`e951e53`](https://github.com/RhetTbull/osxphotos/commit/e951e5361e59060229787bb1ea3fc4e088ffff99)
+
+#### [v0.36.20](https://github.com/RhetTbull/osxphotos/compare/v0.36.19...v0.36.20)
+
+> 23 November 2020
+
+- Added photokit export as hidden --use-photokit option [`26f96d5`](https://github.com/RhetTbull/osxphotos/commit/26f96d582c01ce9816b1f54f0e74c8570f133f7c)
+
+#### [v0.36.19](https://github.com/RhetTbull/osxphotos/compare/v0.36.18...v0.36.19)
+
+> 19 November 2020
+
+- Removed debug statement in _photoinfo_export [`8cb15d1`](https://github.com/RhetTbull/osxphotos/commit/8cb15d15551094dcaf1b0ef32d6ac0273be7fd37)
+
+#### [v0.36.18](https://github.com/RhetTbull/osxphotos/compare/v0.36.17...v0.36.18)
+
+> 14 November 2020
+
+- Moved AppleScript to photoscript [`3c85f26`](https://github.com/RhetTbull/osxphotos/commit/3c85f26f901645ce297685ccd639792757fbc995)
+- Fixed missing data file for photoscript [`2d9429c`](https://github.com/RhetTbull/osxphotos/commit/2d9429c8eefabe6233fc580f65511c48ee6c01e5)
+- Version bump, updated requirements [`3b6dd08`](https://github.com/RhetTbull/osxphotos/commit/3b6dd08d2bb2b20a55064bf24fe7ce788e7268ef)
+
 #### [v0.36.17](https://github.com/RhetTbull/osxphotos/compare/v0.36.15...v0.36.17)
 
 > 12 November 2020

README.md

@@ -3,7 +3,7 @@
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![Python package](https://github.com/RhetTbull/osxphotos/workflows/Python%20package/badge.svg)](https://github.com/RhetTbull/osxphotos/workflows/Python%20package/badge.svg)
 <!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->
-[![All Contributors](https://img.shields.io/badge/all_contributors-10-orange.svg?style=flat-square)](#contributors-)
+[![All Contributors](https://img.shields.io/badge/all_contributors-11-orange.svg?style=flat-square)](#contributors-)
 <!-- ALL-CONTRIBUTORS-BADGE:END -->
 
 - [OSXPhotos](#osxphotos)
@@ -227,6 +227,9 @@ Options:
   --no-comment                    Search for photos with no comments.
   --has-likes                     Search for photos that have likes.
   --no-likes                      Search for photos with no likes.
+  --missing                       Export only photos missing from the Photos
+                                  library; must be used with --download-
+                                  missing.
   --deleted                       Include photos from the 'Recently Deleted'
                                   folder.
   --deleted-only                  Include only photos from the 'Recently
@@ -234,7 +237,8 @@ Options:
   --update                        Only export new or updated files. See notes
                                   below on export and --update.
   --dry-run                       Dry run (test) the export but don't actually
-                                  export any files; most useful with --verbose
+                                  export any files; most useful with
+                                  --verbose.
   --export-as-hardlink            Hardlink files instead of copying them.
                                   Cannot be used with --exiftool which creates
                                   copies of the files with embedded EXIF data.
@@ -351,11 +355,30 @@ Options:
                                   photo would be named
                                   'photoname_bearbeiten.ext'.  The default
                                   suffix is '_edited'.
+  --original-suffix SUFFIX        Optional suffix for naming original photos.
+                                  Default name for original photos is in form
+                                  'filename.ext'. For example, with '--
+                                  original-suffix _original', the original
+                                  photo would be named
+                                  'filename_original.ext'.  The default suffix
+                                  is '' (no suffix).
   --no-extended-attributes        Don't copy extended attributes when
                                   exporting.  You only need this if exporting
                                   to a filesystem that doesn't support Mac OS
                                   extended attributes.  Only use this if you
                                   get an error while exporting.
+  --use-photos-export             Force the use of AppleScript or PhotoKit to
+                                  export even if not missing (see also '--
+                                  download-missing' and '--use-photokit').
+  --use-photokit                  Use with '--download-missing' or '--use-
+                                  photos-export' to use direct Photos
+                                  interface instead of AppleScript to export.
+                                  Highly experimental alpha feature; does not
+                                  work with iTerm2 (use with Terminal.app).
+                                  This is faster and more reliable than the
+                                  default AppleScript interface.
+  --report REPORTNAME.CSV         Write a CSV formatted report of all files
+                                  that were exported.
   -h, --help                      Show this message and exit.
 
 ** Export **
@@ -619,18 +642,26 @@ exported, one to each directory.  For example: --directory
 of the following directories if the photos were created in 2019 and were in
 albums 'Vacation' and 'Family': 2019/Vacation, 2019/Family
 
-Substitution        Description
+Substitution              Description
-{album}             Album(s) photo is contained in
+{album}                   Album(s) photo is contained in
-{folder_album}      Folder path + album photo is contained in. e.g.
+{folder_album}            Folder path + album photo is contained in. e.g.
-                    'Folder/Subfolder/Album' or just 'Album' if no enclosing
+                          'Folder/Subfolder/Album' or just 'Album' if no
-                    folder
+                          enclosing folder
-{keyword}           Keyword(s) assigned to photo
+{keyword}                 Keyword(s) assigned to photo
-{person}            Person(s) / face(s) in a photo
+{person}                  Person(s) / face(s) in a photo
-{label}             Image categorization label associated with a photo
+{label}                   Image categorization label associated with a photo
-                    (Photos 5 only)
+                          (Photos 5 only)
-{label_normalized}  All lower case version of 'label' (Photos 5 only)
+{label_normalized}        All lower case version of 'label' (Photos 5 only)
-{comment}           Comment(s) on shared Photos; format is 'Person name:
+{comment}                 Comment(s) on shared Photos; format is 'Person
-                    comment text' (Photos 5 only)
+                          name: comment text' (Photos 5 only)
+{exiftool:GROUP:TAGNAME}  Use exiftool (https://exiftool.org) to extract
+                          metadata, in form GROUP:TAGNAME, from image.  E.g.
+                          '{exiftool:EXIF:Make}' to get camera make, or
+                          {exiftool:IPTC:Keywords} to extract keywords. See
+                          https://exiftool.org/TagNames/ for list of valid
+                          tag names.  You must specify group (e.g. EXIF,
+                          IPTC, etc) as used in `exiftool -G`. exiftool must
+                          be installed in the path to use this template.
 ```
 
 Example: export all photos to ~/Desktop/export group in folders by date created
@@ -2028,6 +2059,7 @@ The following template field substitutions are availabe for use with `PhotoInfo.
 |{label}|Image categorization label associated with a photo (Photos 5 only)|
 |{label_normalized}|All lower case version of 'label' (Photos 5 only)|
 |{comment}|Comment(s) on shared Photos; format is 'Person name: comment text' (Photos 5 only)|
+|{exiftool:GROUP:TAGNAME}|Use exiftool (https://exiftool.org) to extract metadata, in form GROUP:TAGNAME, from image.  E.g. '{exiftool:EXIF:Make}' to get camera make, or {exiftool:IPTC:Keywords} to extract keywords. See https://exiftool.org/TagNames/ for list of valid tag names.  You must specify group (e.g. EXIF, IPTC, etc) as used in `exiftool -G`. exiftool must be installed in the path to use this template.|
 
 ### Utility Functions
 
@@ -2149,6 +2181,7 @@ Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/d
     <td align="center"><a href="https://github.com/grundsch"><img src="https://avatars0.githubusercontent.com/u/3874928?v=4?s=100" width="100px;" alt=""/><br /><sub><b>grundsch</b></sub></a><br /><a href="https://github.com/RhetTbull/osxphotos/commits?author=grundsch" title="Code">💻</a></td>
     <td align="center"><a href="https://github.com/agprimatic"><img src="https://avatars1.githubusercontent.com/u/4685054?v=4?s=100" width="100px;" alt=""/><br /><sub><b>Ag Primatic</b></sub></a><br /><a href="https://github.com/RhetTbull/osxphotos/commits?author=agprimatic" title="Code">💻</a></td>
     <td align="center"><a href="https://github.com/hhoeck"><img src="https://avatars1.githubusercontent.com/u/6313998?v=4?s=100" width="100px;" alt=""/><br /><sub><b>Horst Höck</b></sub></a><br /><a href="https://github.com/RhetTbull/osxphotos/commits?author=hhoeck" title="Code">💻</a></td>
+    <td align="center"><a href="https://github.com/jstrine"><img src="https://avatars1.githubusercontent.com/u/33943447?v=4?s=100" width="100px;" alt=""/><br /><sub><b>Jonathan Strine</b></sub></a><br /><a href="https://github.com/RhetTbull/osxphotos/commits?author=jstrine" title="Code">💻</a></td>
   </tr>
 </table>
 
@@ -2173,8 +2206,6 @@ This package works by creating a copy of the sqlite3 database that photos uses t
 
 If apple changes the database format this will likely break.
 
-Apple does provide a framework ([PhotoKit](https://developer.apple.com/documentation/photokit?language=objc)) for querying the user's Photos library and I attempted to create the functionality in this package using this framework but unfortunately PhotoKit does not provide access to much of the needed metadata (such as Faces/Persons) and Apple's System Integrity Protection (SIP) made the interface unreliable.  If you'd like to experiment with the PhotoKit interface, here's some sample [code](https://gist.github.com/RhetTbull/41cc85e5bdeb30f761147ce32fba5c94).  While copying the sqlite file is a bit kludgy, it allows osxphotos to provide access to all available metadata.
-
 For additional details about how osxphotos is implemented or if you would like to extend the code, see the [wiki](https://github.com/RhetTbull/osxphotos/wiki).
 
 ## Dependencies

osxphotos/_version.py

@@ -1,4 +1,4 @@
 """ version info """
 
-__version__ = "0.36.19"
+__version__ = "0.37.5"
 

osxphotos/datetime_utils.py

@@ -1,10 +1,10 @@
-""" datetime utilities """
+""" datetime.datetime helper functions for converting to/from UTC """
 
 import datetime
 
 
 def get_local_tz(dt):
-    """ return local timezone as datetime.timezone tzinfo for dt
+    """ Return local timezone as datetime.timezone tzinfo for dt
     
     Args:
         dt: datetime.datetime
@@ -21,21 +21,18 @@ def get_local_tz(dt):
         raise ValueError("dt must be naive datetime.datetime object")
 
 
-def datetime_remove_tz(dt):
-    """ remove timezone from a datetime.datetime object
-        dt: datetime.datetime object with tzinfo
-        returns: dt without any timezone info (naive datetime object) """
-
-    if type(dt) != datetime.datetime:
-        raise TypeError(f"dt must be type datetime.datetime, not {type(dt)}")
-
-    return dt.replace(tzinfo=None)
-
-
 def datetime_has_tz(dt):
-    """ return True if datetime dt has tzinfo else False
+    """ Return True if datetime dt has tzinfo else False
+
+    Args:
         dt: datetime.datetime
-        returns True if dt is timezone aware, else False """
+    
+    Returns:
+        True if dt is timezone aware, else False
+        
+    Raises:
+        TypeError if dt is not a datetime.datetime object
+    """
 
     if type(dt) != datetime.datetime:
         raise TypeError(f"dt must be type datetime.datetime, not {type(dt)}")
@@ -43,11 +40,90 @@ def datetime_has_tz(dt):
     return dt.tzinfo is not None and dt.tzinfo.utcoffset(dt) is not None
 
 
-def datetime_naive_to_local(dt):
+def datetime_tz_to_utc(dt):
-    """ convert naive (timezone unaware) datetime.datetime
+    """ Convert datetime.datetime object with timezone to UTC timezone 
-        to aware timezone in local timezone
+
+    Args:
+        dt: datetime.datetime object
+
+    Returns:
+        datetime.datetime in UTC timezone
+        
+    Raises: 
+        TypeError if dt is not datetime.datetime object
+        ValueError if dt does not have timeone information
+    """
+
+    if type(dt) != datetime.datetime:
+        raise TypeError(f"dt must be type datetime.datetime, not {type(dt)}")
+
+    if dt.tzinfo is not None and dt.tzinfo.utcoffset(dt) is not None:
+        return dt.replace(tzinfo=dt.tzinfo).astimezone(tz=datetime.timezone.utc)
+    else:
+        raise ValueError(f"dt does not have timezone info")
+
+
+def datetime_remove_tz(dt):
+    """ Remove timezone from a datetime.datetime object
+
+    Args:
+        dt: datetime.datetime object with tzinfo
+    
+    Returns:
+        dt without any timezone info (naive datetime object) 
+        
+    Raises:
+        TypeError if dt is not a datetime.datetime object
+    """
+
+    if type(dt) != datetime.datetime:
+        raise TypeError(f"dt must be type datetime.datetime, not {type(dt)}")
+
+    return dt.replace(tzinfo=None)
+
+
+def datetime_naive_to_utc(dt):
+    """ Convert naive (timezone unaware) datetime.datetime
+        to aware timezone in UTC timezone
+
+    Args:
         dt: datetime.datetime without timezone
-        returns: datetime.datetime with local timezone """
+    
+    Returns:
+        datetime.datetime with UTC timezone
+        
+    Raises:
+        TypeError if dt is not a datetime.datetime object
+        ValueError if dt is not a naive/timezone unaware object
+    """
+
+    if type(dt) != datetime.datetime:
+        raise TypeError(f"dt must be type datetime.datetime, not {type(dt)}")
+
+    if dt.tzinfo is not None and dt.tzinfo.utcoffset(dt) is not None:
+        # has timezone info
+        raise ValueError(
+            "dt must be naive/timezone unaware: "
+            f"{dt} has tzinfo {dt.tzinfo} and offset {dt.tzinfo.utcoffset(dt)}"
+        )
+
+    return dt.replace(tzinfo=datetime.timezone.utc)
+
+
+def datetime_naive_to_local(dt):
+    """ Convert naive (timezone unaware) datetime.datetime
+        to aware timezone in local timezone
+
+    Args:
+        dt: datetime.datetime without timezone
+    
+    Returns:
+        datetime.datetime with local timezone
+        
+    Raises:
+        TypeError if dt is not a datetime.datetime object
+        ValueError if dt is not a naive/timezone unaware object
+    """
 
     if type(dt) != datetime.datetime:
         raise TypeError(f"dt must be type datetime.datetime, not {type(dt)}")
@@ -60,3 +136,26 @@ def datetime_naive_to_local(dt):
         )
 
     return dt.replace(tzinfo=get_local_tz(dt))
+
+
+def datetime_utc_to_local(dt):
+    """ Convert datetime.datetime object in UTC timezone to local timezone 
+
+    Args:
+        dt: datetime.datetime object
+
+    Returns:
+        datetime.datetime in local timezone
+
+    Raises:
+        TypeError if dt is not a datetime.datetime object
+        ValueError if dt is not in UTC timezone
+    """
+
+    if type(dt) != datetime.datetime:
+        raise TypeError(f"dt must be type datetime.datetime, not {type(dt)}")
+
+    if dt.tzinfo is not datetime.timezone.utc:
+        raise ValueError(f"{dt} must be in UTC timezone: timezone = {dt.tzinfo}")
+
+    return dt.replace(tzinfo=datetime.timezone.utc).astimezone(tz=None)

osxphotos/export_db.py

@@ -14,7 +14,7 @@ from sqlite3 import Error
 
 from ._version import __version__
 
-OSXPHOTOS_EXPORTDB_VERSION = "2.0"
+OSXPHOTOS_EXPORTDB_VERSION = "3.2"
 
 
 class ExportDB_ABC(ABC):
@@ -76,6 +76,14 @@ class ExportDB_ABC(ABC):
     def set_exifdata_for_file(self, uuid, exifdata):
         pass
 
+    @abstractmethod
+    def set_sidecar_for_file(self, filename, sidecar_data, sidecar_sig):
+        pass
+
+    @abstractmethod
+    def get_sidecar_for_file(self, filename):
+        pass
+
     @abstractmethod
     def set_data(
         self,
@@ -141,6 +149,12 @@ class ExportDBNoOp(ExportDB_ABC):
     def set_exifdata_for_file(self, uuid, exifdata):
         pass
 
+    def set_sidecar_for_file(self, filename, sidecar_data, sidecar_sig):
+        pass
+
+    def get_sidecar_for_file(self, filename):
+        return None, (None, None, None)
+
     def set_data(
         self,
         filename,
@@ -379,6 +393,48 @@ class ExportDB(ExportDB_ABC):
         except Error as e:
             logging.warning(e)
 
+    def get_sidecar_for_file(self, filename):
+        """ returns the sidecar data and signature for a file """
+        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        conn = self._conn
+        try:
+            c = conn.cursor()
+            c.execute(
+                "SELECT sidecar_data, mode, size, mtime FROM sidecar WHERE filepath_normalized = ?",
+                (filename,),
+            )
+            results = c.fetchone()
+            if results:
+                sidecar_data = results[0]
+                sidecar_sig = (
+                    results[1],
+                    results[2],
+                    int(results[3]) if results[3] is not None else None,
+                )
+            else:
+                sidecar_data = None
+                sidecar_sig = (None, None, None)
+        except Error as e:
+            logging.warning(e)
+            sidecar_data = None
+            sidecar_sig = (None, None, None)
+
+        return sidecar_data, sidecar_sig
+
+    def set_sidecar_for_file(self, filename, sidecar_data, sidecar_sig):
+        """ sets the sidecar data and signature for a file """
+        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        conn = self._conn
+        try:
+            c = conn.cursor()
+            c.execute(
+                "INSERT OR REPLACE INTO sidecar(filepath_normalized, sidecar_data, mode, size, mtime) VALUES (?, ?, ?, ?, ?);",
+                (filename, sidecar_data, *sidecar_sig),
+            )
+            conn.commit()
+        except Error as e:
+            logging.warning(e)
+
     def set_data(
         self,
         filename,
@@ -479,13 +535,11 @@ class ExportDB(ExportDB_ABC):
 
         if not os.path.isfile(dbfile):
             conn = self._get_db_connection(dbfile)
-            if conn:
+            if not conn:
-                self._create_db_tables(conn)
-                self.was_created = True
-                self.was_upgraded = ()
-                self.version = OSXPHOTOS_EXPORTDB_VERSION
-            else:
                 raise Exception("Error getting connection to database {dbfile}")
+            self._create_db_tables(conn)
+            self.was_created = True
+            self.was_upgraded = ()
         else:
             conn = self._get_db_connection(dbfile)
             self.was_created = False
@@ -495,8 +549,7 @@ class ExportDB(ExportDB_ABC):
                 self.was_upgraded = (version_info[1], OSXPHOTOS_EXPORTDB_VERSION)
             else:
                 self.was_upgraded = ()
-            self.version = OSXPHOTOS_EXPORTDB_VERSION
+        self.version = OSXPHOTOS_EXPORTDB_VERSION
-
         return conn
 
     def _get_db_connection(self, dbfile):
@@ -570,11 +623,20 @@ class ExportDB(ExportDB_ABC):
                               size INTEGER,
                               mtime REAL
                               ); """,
+            "sql_sidecar_table": """ CREATE TABLE IF NOT EXISTS sidecar (
+                              id INTEGER PRIMARY KEY,
+                              filepath_normalized TEXT NOT NULL,
+                              sidecar_data TEXT,
+                              mode INTEGER,
+                              size INTEGER,
+                              mtime REAL
+                              ); """,
             "sql_files_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_files_filepath_normalized on files (filepath_normalized); """,
             "sql_info_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_info_uuid on info (uuid); """,
             "sql_exifdata_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_exifdata_filename on exifdata (filepath_normalized); """,
             "sql_edited_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_edited_filename on edited (filepath_normalized);""",
             "sql_converted_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_converted_filename on converted (filepath_normalized);""",
+            "sql_sidecar_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_sidecar_filename on sidecar (filepath_normalized);""",
         }
         try:
             c = conn.cursor()

osxphotos/photoinfo/photoinfo.py

@@ -91,9 +91,10 @@ class PhotoInfo:
             and self.raw_original
         ):
             # return the JPEG version as that's what Photos 5+ does
-            return self._info["raw_pair_info"]["originalFilename"]
+            original_name = self._info["raw_pair_info"]["originalFilename"]
         else:
-            return self._info["originalFilename"]
+            original_name = self._info["originalFilename"]
+        return original_name or self.filename
 
     @property
     def date(self):

osxphotos/phototemplate.py

@@ -18,6 +18,7 @@ from functools import partial
 
 from ._constants import _UNKNOWN_PERSON
 from .datetime_formatter import DateTimeFormatter
+from .exiftool import ExifTool
 from .path_utils import sanitize_dirname, sanitize_filename, sanitize_pathpart
 
 # ensure locale set to user's locale
@@ -126,6 +127,10 @@ TEMPLATE_SUBSTITUTIONS_MULTI_VALUED = {
     "{label}": "Image categorization label associated with a photo (Photos 5 only)",
     "{label_normalized}": "All lower case version of 'label' (Photos 5 only)",
     "{comment}": "Comment(s) on shared Photos; format is 'Person name: comment text' (Photos 5 only)",
+    "{exiftool:GROUP:TAGNAME}": "Use exiftool (https://exiftool.org) to extract metadata, in form GROUP:TAGNAME, from image.  "
+    "E.g. '{exiftool:EXIF:Make}' to get camera make, or {exiftool:IPTC:Keywords} to extract keywords. "
+    "See https://exiftool.org/TagNames/ for list of valid tag names.  You must specify group (e.g. EXIF, IPTC, etc) "
+    "as used in `exiftool -G`. exiftool must be installed in the path to use this template.",
 }
 
 # Just the multi-valued substitution names without the braces
@@ -150,6 +155,62 @@ class PhotoTemplate:
         # gets initialized in get_template_value
         self.today = None
 
+    def make_subst_function(
+        self, none_str, filename, dirname, replacement, get_func=None
+    ):
+        """ returns: substitution function for use in re.sub 
+            none_str: value to use if substitution lookup is None and no default provided
+            get_func: function that gets the substitution value for a given template field
+                    default is get_template_value which handles the single-value fields """
+
+        if get_func is None:
+            # used by make_subst_function to get the value for a template substitution
+            get_func = partial(
+                self.get_template_value,
+                filename=filename,
+                dirname=dirname,
+                replacement=replacement,
+            )
+
+        # closure to capture photo, none_str, filename, dirname in subst
+        def subst(matchobj):
+            groups = len(matchobj.groups())
+            if groups != 5:
+                raise ValueError(
+                    f"Unexpected number of groups: expected 4, got {groups}"
+                )
+
+            delim = matchobj.group(1)
+            field = matchobj.group(2)
+            path_sep = matchobj.group(3)
+            bool_val = matchobj.group(4)
+            default = matchobj.group(5)
+
+            # drop the '+' on delim
+            delim = delim[:-1] if delim is not None else None
+            # drop () from path_sep
+            path_sep = path_sep.strip("()") if path_sep is not None else None
+            # drop the ? on bool_val
+            bool_val = bool_val[1:] if bool_val is not None else None
+            # drop the comma on default
+            default_val = default[1:] if default is not None else None
+
+            try:
+                val = get_func(field, default_val, bool_val, delim, path_sep)
+            except ValueError:
+                return matchobj.group(0)
+
+            if val is None:
+                # field valid but didn't match a value
+                if default == ",":
+                    val = ""
+                else:
+                    val = default_val if default_val is not None else none_str
+
+            return val
+
+        return subst
+
     def render(
         self,
         template,
@@ -208,60 +269,7 @@ class PhotoTemplate:
         if type(template) is not str:
             raise TypeError(f"template must be type str, not {type(template)}")
 
-        # used by make_subst_function to get the value for a template substitution
+        subst_func = self.make_subst_function(none_str, filename, dirname, replacement)
-        get_func = partial(
-            self.get_template_value,
-            filename=filename,
-            dirname=dirname,
-            replacement=replacement,
-        )
-
-        def make_subst_function(self, none_str, get_func=get_func):
-            """ returns: substitution function for use in re.sub 
-                none_str: value to use if substitution lookup is None and no default provided
-                get_func: function that gets the substitution value for a given template field
-                        default is get_template_value which handles the single-value fields """
-
-            # closure to capture photo, none_str, filename, dirname in subst
-            def subst(matchobj):
-                groups = len(matchobj.groups())
-                if groups == 5:
-                    delim = matchobj.group(1)
-                    field = matchobj.group(2)
-                    path_sep = matchobj.group(3)
-                    bool_val = matchobj.group(4)
-                    default = matchobj.group(5)
-
-                    # drop the '+' on delim
-                    delim = delim[:-1] if delim is not None else None
-                    # drop () from path_sep
-                    path_sep = path_sep.strip("()") if path_sep is not None else None
-                    # drop the ? on bool_val
-                    bool_val = bool_val[1:] if bool_val is not None else None
-                    # drop the comma on default
-                    default_val = default[1:] if default is not None else None
-
-                    try:
-                        val = get_func(field, default_val, bool_val, delim, path_sep)
-                    except ValueError:
-                        return matchobj.group(0)
-
-                    if val is None:
-                        # field valid but didn't match a value
-                        if default == ",":
-                            val = ""
-                        else:
-                            val = default_val if default_val is not None else none_str
-
-                    return val
-                else:
-                    raise ValueError(
-                        f"Unexpected number of groups: expected 4, got {groups}"
-                    )
-
-            return subst
-
-        subst_func = make_subst_function(self, none_str)
 
         # do the replacements
         rendered = re.sub(regex, subst_func, template)
@@ -289,88 +297,28 @@ class PhotoTemplate:
         #    '2011/Album2/keyword1/person1',
         #    '2011/Album2/keyword2/person1',]
 
-        rendered_strings = [rendered]
+        rendered_strings = self._render_multi_valued_templates(
-        for field in MULTI_VALUE_SUBSTITUTIONS:
+            rendered,
-            # Build a regex that matches only the field being processed
+            none_str,
-            re_str = (
+            path_sep,
-                r"(?<!\{)\{"  # match { but not {{
+            expand_inplace,
-                + r"([^}]*\+)?"  # group 1: optional DELIM+
+            inplace_sep,
-                + r"("
+            filename,
-                + field  # group 2: field name
+            dirname,
-                + r")"
+            replacement,
-                + r"(\([^{}\)]*\))?"  # group 3: optional (PATH_SEP)
+        )
-                + r"(\?[^\\,}]*)?"  # group 4: optional ?TRUE_VALUE for boolean fields
-                + r"(,[\w\=\;\-\%. ]*)?"  # group 5: optional ,DEFAULT
-                + r"(?=\}(?!\}))\}"  # match } but not }}
-            )
-            regex_multi = re.compile(re_str)
 
-            # holds each of the new rendered_strings, dict to avoid repeats (dict.keys())
+        # process exiftool: templates
-            new_strings = {}
+        rendered_strings = self._render_exiftool_template(
-
+            rendered_strings,
-            for str_template in rendered_strings:
+            none_str,
-                matches = regex_multi.search(str_template)
+            path_sep,
-                if matches:
+            expand_inplace,
-                    path_sep = (
+            inplace_sep,
-                        matches.group(3).strip("()")
+            filename,
-                        if matches.group(3) is not None
+            dirname,
-                        else path_sep
+            replacement,
-                    )
+        )
-                    values = self.get_template_value_multi(
-                        field,
-                        path_sep,
-                        filename=filename,
-                        dirname=dirname,
-                        replacement=replacement,
-                    )
-                    if expand_inplace or matches.group(1) is not None:
-                        delim = (
-                            matches.group(1)[:-1] if matches.group(1) is not None else inplace_sep 
-                        )
-                        # instead of returning multiple strings, join values into a single string
-                        val = delim.join(sorted(values)) if values and values[0] else None
-
-                        def lookup_template_value_multi(lookup_value, *_):
-                            """ Closure passed to make_subst_function get_func 
-                                    Capture val and field in the closure 
-                                    Allows make_subst_function to be re-used w/o modification
-                                    _ is not used but required so signature matches get_template_value """
-                            if lookup_value == field:
-                                return val
-                            else:
-                                raise ValueError(f"Unexpected value: {lookup_value}")
-
-                        subst = make_subst_function(
-                            self, none_str, get_func=lookup_template_value_multi
-                        )
-                        new_string = regex_multi.sub(subst, str_template)
-
-                        # update rendered_strings for the next field to process
-                        rendered_strings = {new_string}
-                    else:
-                        # create a new template string for each value
-                        for val in values:
-
-                            def lookup_template_value_multi(lookup_value, *_):
-                                """ Closure passed to make_subst_function get_func 
-                                    Capture val and field in the closure 
-                                    Allows make_subst_function to be re-used w/o modification
-                                    _ is not used but required so signature matches get_template_value """
-                                if lookup_value == field:
-                                    return val
-                                else:
-                                    raise ValueError(
-                                        f"Unexpected value: {lookup_value}"
-                                    )
-
-                            subst = make_subst_function(
-                                self, none_str, get_func=lookup_template_value_multi
-                            )
-                            new_string = regex_multi.sub(subst, str_template)
-                            new_strings[new_string] = 1
-
-                        # update rendered_strings for the next field to process
-                        rendered_strings = list(new_strings.keys())
 
         # find any {fields} that weren't replaced
         unmatched = []
@@ -396,6 +344,244 @@ class PhotoTemplate:
 
         return rendered_strings, unmatched
 
+    def _render_multi_valued_templates(
+        self,
+        rendered,
+        none_str,
+        path_sep,
+        expand_inplace,
+        inplace_sep,
+        filename,
+        dirname,
+        replacement,
+    ):
+        rendered_strings = [rendered]
+        new_rendered_strings = []
+        while new_rendered_strings != rendered_strings:
+            new_rendered_strings = rendered_strings
+            for field in MULTI_VALUE_SUBSTITUTIONS:
+                # Build a regex that matches only the field being processed
+                re_str = (
+                    r"(?<!\{)\{"  # match { but not {{
+                    + r"([^}]*\+)?"  # group 1: optional DELIM+
+                    + r"("
+                    + field  # group 2: field name
+                    + r")"
+                    + r"(\([^{}\)]*\))?"  # group 3: optional (PATH_SEP)
+                    + r"(\?[^\\,}]*)?"  # group 4: optional ?TRUE_VALUE for boolean fields
+                    + r"(,[\w\=\;\-\%. ]*)?"  # group 5: optional ,DEFAULT
+                    + r"(?=\}(?!\}))\}"  # match } but not }}
+                )
+                regex_multi = re.compile(re_str)
+
+                # holds each of the new rendered_strings, dict to avoid repeats (dict.keys())
+                new_strings = {}
+
+                for str_template in rendered_strings:
+                    matches = regex_multi.search(str_template)
+                    if matches:
+                        path_sep = (
+                            matches.group(3).strip("()")
+                            if matches.group(3) is not None
+                            else path_sep
+                        )
+                        values = self.get_template_value_multi(
+                            field,
+                            path_sep,
+                            filename=filename,
+                            dirname=dirname,
+                            replacement=replacement,
+                        )
+                        if expand_inplace or matches.group(1) is not None:
+                            delim = (
+                                matches.group(1)[:-1]
+                                if matches.group(1) is not None
+                                else inplace_sep
+                            )
+                            # instead of returning multiple strings, join values into a single string
+                            val = (
+                                delim.join(sorted(values))
+                                if values and values[0]
+                                else None
+                            )
+
+                            def lookup_template_value_multi(lookup_value, *_):
+                                """ Closure passed to make_subst_function get_func 
+                                        Capture val and field in the closure 
+                                        Allows make_subst_function to be re-used w/o modification
+                                        _ is not used but required so signature matches get_template_value """
+                                if lookup_value == field:
+                                    return val
+                                else:
+                                    raise ValueError(
+                                        f"Unexpected value: {lookup_value}"
+                                    )
+
+                            subst = self.make_subst_function(
+                                none_str,
+                                filename,
+                                dirname,
+                                replacement,
+                                get_func=lookup_template_value_multi,
+                            )
+                            new_string = regex_multi.sub(subst, str_template)
+
+                            # update rendered_strings for the next field to process
+                            rendered_strings = list({new_string})
+                        else:
+                            # create a new template string for each value
+                            for val in values:
+
+                                def lookup_template_value_multi(lookup_value, *_):
+                                    """ Closure passed to make_subst_function get_func 
+                                        Capture val and field in the closure 
+                                        Allows make_subst_function to be re-used w/o modification
+                                        _ is not used but required so signature matches get_template_value """
+                                    if lookup_value == field:
+                                        return val
+                                    else:
+                                        raise ValueError(
+                                            f"Unexpected value: {lookup_value}"
+                                        )
+
+                                subst = self.make_subst_function(
+                                    none_str,
+                                    filename,
+                                    dirname,
+                                    replacement,
+                                    get_func=lookup_template_value_multi,
+                                )
+                                new_string = regex_multi.sub(subst, str_template)
+                                new_strings[new_string] = 1
+
+                            # update rendered_strings for the next field to process
+                            rendered_strings = sorted(list(new_strings.keys()))
+        return rendered_strings
+
+    def _render_exiftool_template(
+        self,
+        rendered_strings,
+        none_str,
+        path_sep,
+        expand_inplace,
+        inplace_sep,
+        filename,
+        dirname,
+        replacement,
+    ):
+        # TODO: lots of code commonality with render_multi_valued_templates -- combine or pull out
+        # TODO: put these in globals
+        if path_sep is None:
+            path_sep = os.path.sep
+
+        if inplace_sep is None:
+            inplace_sep = ","
+
+        # Build a regex that matches only the field being processed
+        # todo: pull out regexes into globals?
+        re_str = (
+            r"(?<!\{)\{"  # match { but not {{
+            + r"([^}]*\+)?"  # group 1: optional DELIM+
+            + r"(exiftool:[^\\,}+\?]+)"  # group 3 field name
+            + r"(\([^{}\)]*\))?"  # group 3: optional (PATH_SEP)
+            + r"(\?[^\\,}]*)?"  # group 4: optional ?TRUE_VALUE for boolean fields
+            + r"(,[\w\=\;\-\%. ]*)?"  # group 5: optional ,DEFAULT
+            + r"(?=\}(?!\}))\}"  # match } but not }}
+        )
+        regex_multi = re.compile(re_str)
+
+        # holds each of the new rendered_strings, dict to avoid repeats (dict.keys())
+        new_rendered_strings = []
+        while new_rendered_strings != rendered_strings:
+            new_rendered_strings = rendered_strings
+            new_strings = {}
+            for str_template in rendered_strings:
+                matches = regex_multi.search(str_template)
+                if matches:
+                    # allmatches = regex_multi.finditer(str_template)
+                    # for matches in allmatches:
+                    path_sep = (
+                        matches.group(3).strip("()")
+                        if matches.group(3) is not None
+                        else path_sep
+                    )
+                    field = matches.group(2)
+                    subfield = field[9:]
+
+                    if not self.photo.path:
+                        values = []
+                    else:
+                        exif = ExifTool(self.photo.path)
+                        exifdict = exif.asdict()
+                        exifdict = {k.lower(): v for (k, v) in exifdict.items()}
+                        subfield = subfield.lower()
+                        if subfield in exifdict:
+                            values = exifdict[subfield]
+                            values = (
+                                [values] if not isinstance(values, list) else values
+                            )
+                        else:
+                            values = [None]
+                    if expand_inplace or matches.group(1) is not None:
+                        delim = (
+                            matches.group(1)[:-1]
+                            if matches.group(1) is not None
+                            else inplace_sep
+                        )
+                        # instead of returning multiple strings, join values into a single string
+                        val = (
+                            delim.join(sorted(values)) if values and values[0] else None
+                        )
+
+                        def lookup_template_value_exif(lookup_value, *_):
+                            """ Closure passed to make_subst_function get_func 
+                                    Capture val and field in the closure 
+                                    Allows make_subst_function to be re-used w/o modification
+                                    _ is not used but required so signature matches get_template_value """
+                            if lookup_value == field:
+                                return val
+                            else:
+                                raise ValueError(f"Unexpected value: {lookup_value}")
+
+                        subst = self.make_subst_function(
+                            none_str,
+                            filename,
+                            dirname,
+                            replacement,
+                            get_func=lookup_template_value_exif,
+                        )
+                        new_string = regex_multi.sub(subst, str_template)
+                        # update rendered_strings for the next field to process
+                        rendered_strings = list({new_string})
+                    else:
+                        # create a new template string for each value
+                        for val in values:
+
+                            def lookup_template_value_exif(lookup_value, *_):
+                                """ Closure passed to make_subst_function get_func 
+                                    Capture val and field in the closure 
+                                    Allows make_subst_function to be re-used w/o modification
+                                    _ is not used but required so signature matches get_template_value """
+                                if lookup_value == field:
+                                    return val
+                                else:
+                                    raise ValueError(
+                                        f"Unexpected value: {lookup_value}"
+                                    )
+
+                            subst = self.make_subst_function(
+                                none_str,
+                                filename,
+                                dirname,
+                                replacement,
+                                get_func=lookup_template_value_exif,
+                            )
+                            new_string = regex_multi.sub(subst, str_template)
+                            new_strings[new_string] = 1
+                        # update rendered_strings for the next field to process
+                        rendered_strings = sorted(list(new_strings.keys()))
+        return rendered_strings
+
     def get_template_value(
         self,
         field,
@@ -681,6 +867,7 @@ class PhotoTemplate:
         """
 
         """ return list of values for a multi-valued template field """
+        values = [] 
         if field == "album":
             values = self.photo.albums
         elif field == "keyword":
@@ -724,7 +911,7 @@ class PhotoTemplate:
             values = [
                 f"{comment.user}: {comment.text}" for comment in self.photo.comments
             ]
-        else:
+        elif not field.startswith("exiftool:"):
             raise ValueError(f"Unhandled template value: {field}")
 
         # sanitize directory names if needed, folder_album handled differently above

osxphotos/templates/xmp_sidecar.mako

@@ -12,7 +12,7 @@
     % if desc is None:
         <dc:description></dc:description>
     % else:
-        <dc:description>${desc}</dc:description>
+        <dc:description>${desc | x}</dc:description>
     % endif
 </%def>
 
@@ -20,7 +20,7 @@
     % if title is None:
         <dc:title></dc:title>
     % else:
-        <dc:title>${title}</dc:title>
+        <dc:title>${title | x}</dc:title>
     % endif
 </%def>
 
@@ -30,7 +30,7 @@
         <dc:subject>
             <rdf:Seq>
             % for subj in subject:
-                <rdf:li>${subj}</rdf:li>
+                <rdf:li>${subj | x}</rdf:li>
             % endfor
             </rdf:Seq>
         </dc:subject>
@@ -48,7 +48,7 @@
         <Iptc4xmpExt:PersonInImage>
             <rdf:Bag>
                 % for person in persons:
-                    <rdf:li>${person}</rdf:li>
+                    <rdf:li>${person | x}</rdf:li>
                 % endfor
             </rdf:Bag>
         </Iptc4xmpExt:PersonInImage>
@@ -60,7 +60,7 @@
         <digiKam:TagsList>
             <rdf:Seq>
             % for keyword in keywords:
-                <rdf:li>${keyword}</rdf:li>
+                <rdf:li>${keyword | x}</rdf:li>
             % endfor
             </rdf:Seq>
         </digiKam:TagsList>
@@ -81,10 +81,8 @@
 
 <%def name="gps_info(latitude, longitude)">
     % if latitude is not None and longitude is not None:
-        <exif:GPSLongitudeRef>${"E" if longitude >= 0 else "W"}</exif:GPSLongitudeRef>
+        <exif:GPSLongitude>${int(abs(longitude))},${(abs(longitude) % 1) * 60}${"E" if longitude >= 0 else "W"}</exif:GPSLongitude>
-        <exif:GPSLongitude>${abs(longitude)}</exif:GPSLongitude>
+        <exif:GPSLatitude>${int(abs(latitude))},${(abs(latitude) % 1) * 60}${"N" if latitude >= 0 else "S"}</exif:GPSLatitude>
-        <exif:GPSLatitude>${abs(latitude)}</exif:GPSLatitude>
-        <exif:GPSLatitudeRef>${"N" if latitude >= 0 else "S"}</exif:GPSLatitudeRef>
     % endif
 </%def>
 

osxphotos/utils.py

@@ -17,7 +17,6 @@ from plistlib import load as plistload
 import CoreFoundation
 import CoreServices
 import objc
-from Foundation import *
 
 from ._constants import UNICODE_FORMAT
 from .fileutil import FileUtil
@@ -202,7 +201,7 @@ def get_last_library_path():
         # pylint: disable=no-member
         # pylint: disable=undefined-variable
         photosurl = CoreFoundation.CFURLCreateByResolvingBookmarkData(
-            kCFAllocatorDefault, photosurlref, 0, None, None, None, None
+            CoreFoundation.kCFAllocatorDefault, photosurlref, 0, None, None, None, None
         )
 
         # the CFURLRef we got is a sruct that python treats as an array

tests/Test-10.15.7.photoslibrary/database/Photos.sqlite.lock

@@ -7,7 +7,7 @@
 	<key>hostuuid</key>
 	<string>9575E48B-8D5F-5654-ABAC-4431B1167324</string>
 	<key>pid</key>
-	<integer>1797</integer>
+	<integer>464</integer>
 	<key>processname</key>
 	<string>photolibraryd</string>
 	<key>uid</key>

tests/test_catalina_10_15_6.py

@@ -177,6 +177,12 @@ RAW_DICT = {
     ),
 }
 
+ORIGINAL_FILENAME_DICT = {
+    "uuid": "D79B8D77-BFFC-460B-9312-034F2877D35B",
+    "filename": "D79B8D77-BFFC-460B-9312-034F2877D35B.jpeg",
+    "original_filename": "Pumkins2.jpg",
+}
+
 
 @pytest.fixture(scope="module")
 def photosdb():
@@ -864,6 +870,27 @@ def test_export_14(photosdb, caplog):
     assert "Invalid destination suffix" not in caplog.text
 
 
+def test_export_no_original_filename(photosdb):
+    # test export OK if original filename is null
+    # issue #267
+
+    tempdir = tempfile.TemporaryDirectory(prefix="osxphotos_")
+    dest = tempdir.name
+    photos = photosdb.photos(uuid=[UUID_DICT["export"]])
+
+    # monkey patch original_filename for testing
+    original_filename = photos[0]._info["originalFilename"]
+    photos[0]._info["originalFilename"] = None
+    filename = f"{photos[0].uuid}.jpeg"
+    expected_dest = os.path.join(dest, filename)
+    got_dest = photos[0].export(dest)[0]
+
+    assert got_dest == expected_dest
+    assert os.path.isfile(got_dest)
+
+    photos[0]._info["originalFilename"] = original_filename
+
+
 def test_eq():
     """ Test equality of two PhotoInfo objects """
 
@@ -1070,3 +1097,18 @@ def test_verbose(capsys):
     photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB, verbose=print)
     captured = capsys.readouterr()
     assert "Processing database" in captured.out
+
+
+def test_original_filename(photosdb):
+    """ test original filename """
+    uuid = ORIGINAL_FILENAME_DICT["uuid"]
+    photo = photosdb.get_photo(uuid)
+    assert photo.original_filename == ORIGINAL_FILENAME_DICT["original_filename"]
+    assert photo.filename == ORIGINAL_FILENAME_DICT["filename"]
+
+    # monkey patch
+    original_filename = photo._info["originalFilename"]
+    photo._info["originalFilename"] = None
+    assert photo.original_filename == ORIGINAL_FILENAME_DICT["filename"]
+    photo._info["originalFilename"] = original_filename
+

tests/test_cli.py

@@ -17,6 +17,7 @@ PLACES_PHOTOS_DB_13 = "tests/Test-Places-High-Sierra-10.13.6.photoslibrary"
 PHOTOS_DB_15_4 = "tests/Test-10.15.4.photoslibrary"
 PHOTOS_DB_15_5 = "tests/Test-10.15.5.photoslibrary"
 PHOTOS_DB_15_6 = "tests/Test-10.15.6.photoslibrary"
+PHOTOS_DB_15_7 = "tests/Test-10.15.7.photoslibrary"
 PHOTOS_DB_TOUCH = PHOTOS_DB_15_6
 PHOTOS_DB_14_6 = "tests/Test-10.14.6.photoslibrary"
 
@@ -65,6 +66,7 @@ CLI_EXPORT_FILENAMES_ALBUM_UNICODE = ["IMG_4547.jpg"]
 CLI_EXPORT_FILENAMES_DELETED_TWIN = ["wedding.jpg", "wedding_edited.jpeg"]
 
 CLI_EXPORT_EDITED_SUFFIX = "_bearbeiten"
+CLI_EXPORT_ORIGINAL_SUFFIX = "_original"
 
 CLI_EXPORT_FILENAMES_EDITED_SUFFIX = [
     "Pumkins1.jpg",
@@ -77,6 +79,16 @@ CLI_EXPORT_FILENAMES_EDITED_SUFFIX = [
     "wedding_bearbeiten.jpeg",
 ]
 
+CLI_EXPORT_FILENAMES_ORIGINAL_SUFFIX = [
+    "Pumkins1_original.jpg",
+    "Pumkins2_original.jpg",
+    "Pumpkins3_original.jpg",
+    "St James Park_original.jpg",
+    "St James Park_edited.jpeg",
+    "Tulips_original.jpg",
+    "wedding_original.jpg",
+    "wedding_edited.jpeg",
+]
 
 CLI_EXPORT_FILENAMES_CURRENT = [
     "1EB2B765-0765-43BA-A90C-0D0580E6172C.jpeg",
@@ -324,6 +336,31 @@ CLI_EXIFTOOL = {
     }
 }
 
+CLI_EXIFTOOL_QUICKTIME = {
+    "35329C57-B963-48D6-BB75-6AFF9370CBBC": {
+        "File:FileName": "Jellyfish.MOV",
+        "XMP:Description": "Jellyfish Video",
+        "XMP:Title": "Jellyfish",
+        "XMP:TagsList": "Travel",
+        "XMP:Subject": "Travel",
+        "QuickTime:GPSCoordinates": "34.053345 -118.242349",
+        "QuickTime:CreationDate": "2020:01:05 22:13:13",
+        "QuickTime:CreateDate": "2020:01:05 22:13:13",
+        "QuickTime:ModifyDate": "2020:01:05 22:13:13",
+    },
+    "2CE332F2-D578-4769-AEFA-7631BB77AA41": {
+        "File:FileName": "Jellyfish.mp4",
+        "XMP:Description": "Jellyfish Video",
+        "XMP:Title": "Jellyfish",
+        "XMP:TagsList": "Travel",
+        "XMP:Subject": "Travel",
+        "QuickTime:GPSCoordinates": "34.053345 -118.242349",
+        "QuickTime:CreationDate": "2020:12:05 05:21:52",
+        "QuickTime:CreateDate": "2020:12:05 05:21:52",
+        "QuickTime:ModifyDate": "2020:12:05 05:21:52",
+    },
+}
+
 CLI_EXIFTOOL_IGNORE_DATE_MODIFIED = {
     "E9BC5C36-7CD1-40A1-A72B-8B8FAC227D51": {
         "File:FileName": "wedding.jpg",
@@ -976,11 +1013,53 @@ def test_export_exiftool_ignore_date_modified():
             )
             assert result.exit_code == 0
 
-            exif = ExifTool(CLI_EXIFTOOL_IGNORE_DATE_MODIFIED[uuid]["File:FileName"]).asdict()
+            exif = ExifTool(
+                CLI_EXIFTOOL_IGNORE_DATE_MODIFIED[uuid]["File:FileName"]
+            ).asdict()
             for key in CLI_EXIFTOOL_IGNORE_DATE_MODIFIED[uuid]:
                 assert exif[key] == CLI_EXIFTOOL_IGNORE_DATE_MODIFIED[uuid][key]
 
 
+@pytest.mark.skipif(exiftool is None, reason="exiftool not installed")
+def test_export_exiftool_quicktime():
+    """ test --exiftol correctly writes QuickTime tags """
+    import glob
+    import os
+    import os.path
+    from osxphotos.__main__ import export
+    from osxphotos.exiftool import ExifTool
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with runner.isolated_filesystem():
+        for uuid in CLI_EXIFTOOL_QUICKTIME:
+            result = runner.invoke(
+                export,
+                [
+                    os.path.join(cwd, PHOTOS_DB_15_7),
+                    ".",
+                    "-V",
+                    "--exiftool",
+                    "--uuid",
+                    f"{uuid}",
+                ],
+            )
+            assert result.exit_code == 0
+            files = glob.glob("*")
+            assert sorted(files) == sorted(
+                [CLI_EXIFTOOL_QUICKTIME[uuid]["File:FileName"]]
+            )
+
+            exif = ExifTool(CLI_EXIFTOOL_QUICKTIME[uuid]["File:FileName"]).asdict()
+            for key in CLI_EXIFTOOL_QUICKTIME[uuid]:
+                assert exif[key] == CLI_EXIFTOOL_QUICKTIME[uuid][key]
+
+            # clean up exported files to avoid name conflicts
+            for filename in files:
+                os.unlink(filename)
+
+
 def test_export_edited_suffix():
     """ test export with --edited-suffix """
     import glob
@@ -1008,6 +1087,33 @@ def test_export_edited_suffix():
         assert sorted(files) == sorted(CLI_EXPORT_FILENAMES_EDITED_SUFFIX)
 
 
+def test_export_original_suffix():
+    """ test export with --original-suffix """
+    import glob
+    import os
+    import os.path
+    import osxphotos
+    from osxphotos.__main__ import export
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with runner.isolated_filesystem():
+        result = runner.invoke(
+            export,
+            [
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "--original-suffix",
+                CLI_EXPORT_ORIGINAL_SUFFIX,
+                "-V",
+            ],
+        )
+        assert result.exit_code == 0
+        files = glob.glob("*")
+        assert sorted(files) == sorted(CLI_EXPORT_FILENAMES_ORIGINAL_SUFFIX)
+
+
 @pytest.mark.skipif(
     "OSXPHOTOS_TEST_CONVERT" not in os.environ,
     reason="Skip if running in Github actions, no GPU.",
@@ -1734,6 +1840,142 @@ def test_export_sidecar_templates():
         )
 
 
+def test_export_sidecar_update():
+    """ test sidecar don't update if not changed and do update if changed """
+    import datetime
+    import glob
+    import os
+    import os.path
+    import osxphotos
+    from osxphotos.fileutil import FileUtil
+
+    from osxphotos.__main__ import cli
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with runner.isolated_filesystem():
+        result = runner.invoke(
+            cli,
+            [
+                "export",
+                "--db",
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "--sidecar=json",
+                "--sidecar=xmp",
+                f"--uuid={CLI_EXPORT_UUID}",
+                "-V",
+            ],
+        )
+        assert result.exit_code == 0
+        assert "Writing XMP sidecar" in result.output
+        assert "Writing exiftool JSON sidecar" in result.output
+
+        # delete a sidecar file and run update
+        fileutil = FileUtil()
+        fileutil.unlink(CLI_EXPORT_SIDECAR_FILENAMES[1])
+
+        result = runner.invoke(
+            cli,
+            [
+                "export",
+                "--db",
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "--sidecar=json",
+                "--sidecar=xmp",
+                f"--uuid={CLI_EXPORT_UUID}",
+                "-V",
+                "--update",
+            ],
+        )
+        assert result.exit_code == 0
+        assert "Skipped up to date XMP sidecar" in result.output
+        assert "Writing exiftool JSON sidecar" in result.output
+
+        # run update again, no sidecar files should update
+        result = runner.invoke(
+            cli,
+            [
+                "export",
+                "--db",
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "--sidecar=json",
+                "--sidecar=xmp",
+                f"--uuid={CLI_EXPORT_UUID}",
+                "-V",
+                "--update",
+            ],
+        )
+        assert result.exit_code == 0
+        assert "Skipped up to date XMP sidecar" in result.output
+        assert "Skipped up to date exiftool JSON sidecar" in result.output
+
+        # touch a file and export again
+        ts = datetime.datetime.now().timestamp() + 1000
+        fileutil.utime(CLI_EXPORT_SIDECAR_FILENAMES[2], (ts, ts))
+
+        result = runner.invoke(
+            cli,
+            [
+                "export",
+                "--db",
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "--sidecar=json",
+                "--sidecar=xmp",
+                f"--uuid={CLI_EXPORT_UUID}",
+                "-V",
+                "--update",
+            ],
+        )
+        assert result.exit_code == 0
+        assert "Writing XMP sidecar" in result.output
+        assert "Skipped up to date exiftool JSON sidecar" in result.output
+
+        # run update again, no sidecar files should update
+        result = runner.invoke(
+            cli,
+            [
+                "export",
+                "--db",
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "--sidecar=json",
+                "--sidecar=xmp",
+                f"--uuid={CLI_EXPORT_UUID}",
+                "-V",
+                "--update",
+            ],
+        )
+        assert result.exit_code == 0
+        assert "Skipped up to date XMP sidecar" in result.output
+        assert "Skipped up to date exiftool JSON sidecar" in result.output
+
+        # run update again with updated metadata, forcing update
+        result = runner.invoke(
+            cli,
+            [
+                "export",
+                "--db",
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "--sidecar=json",
+                "--sidecar=xmp",
+                f"--uuid={CLI_EXPORT_UUID}",
+                "-V",
+                "--update",
+                "--keyword-template",
+                "foo",
+            ],
+        )
+        assert result.exit_code == 0
+        assert "Writing XMP sidecar" in result.output
+        assert "Writing exiftool JSON sidecar" in result.output
+
+
 def test_export_live():
     import glob
     import os
@@ -2682,8 +2924,7 @@ def test_export_sidecar_keyword_template():
 
         json_expected = json.loads(
             """
-            [{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", 
+            [{"EXIF:ImageDescription": "Girl holding pumpkin", 
-            "EXIF:ImageDescription": "Girl holding pumpkin", 
             "XMP:Description": "Girl holding pumpkin", 
             "XMP:Title": "I found one!", 
             "XMP:TagsList": ["Kids", "Multi Keyword", "Pumpkin Farm", "Test Album"], 
@@ -2742,7 +2983,7 @@ def test_export_update_basic():
         )
         assert result.exit_code == 0
         assert (
-            "Exported: 0 photos, updated: 0 photos, skipped: 8 photos, updated EXIF data: 0 photos"
+            "Processed: 7 photos, exported: 0, updated: 0, skipped: 8, updated EXIF data: 0, missing: 1, error: 0"
             in result.output
         )
 
@@ -2826,7 +3067,7 @@ def test_export_update_exiftool():
         )
         assert result.exit_code == 0
         assert (
-            "Exported: 0 photos, updated: 8 photos, skipped: 0 photos, updated EXIF data: 8 photos"
+            "Processed: 7 photos, exported: 0, updated: 8, skipped: 0, updated EXIF data: 8, missing: 1, error: 0"
             in result.output
         )
 
@@ -2836,7 +3077,7 @@ def test_export_update_exiftool():
         )
         assert result.exit_code == 0
         assert (
-            "Exported: 0 photos, updated: 0 photos, skipped: 8 photos, updated EXIF data: 0 photos"
+            "Processed: 7 photos, exported: 0, updated: 0, skipped: 8, updated EXIF data: 0, missing: 1, error: 0"
             in result.output
         )
 
@@ -2873,7 +3114,7 @@ def test_export_update_hardlink():
         )
         assert result.exit_code == 0
         assert (
-            "Exported: 0 photos, updated: 8 photos, skipped: 0 photos, updated EXIF data: 0 photos"
+            "Processed: 7 photos, exported: 0, updated: 8, skipped: 0, updated EXIF data: 0, missing: 1, error: 0"
             in result.output
         )
         assert not os.path.samefile(CLI_EXPORT_UUID_FILENAME, photo.path)
@@ -2912,7 +3153,7 @@ def test_export_update_hardlink_exiftool():
         )
         assert result.exit_code == 0
         assert (
-            "Exported: 0 photos, updated: 8 photos, skipped: 0 photos, updated EXIF data: 8 photos"
+            "Processed: 7 photos, exported: 0, updated: 8, skipped: 0, updated EXIF data: 8, missing: 1, error: 0"
             in result.output
         )
         assert not os.path.samefile(CLI_EXPORT_UUID_FILENAME, photo.path)
@@ -2950,7 +3191,7 @@ def test_export_update_edits():
         )
         assert result.exit_code == 0
         assert (
-            "Exported: 1 photo, updated: 1 photo, skipped: 6 photos, updated EXIF data: 0 photos"
+            "Processed: 7 photos, exported: 1, updated: 1, skipped: 6, updated EXIF data: 0, missing: 1, error: 0"
             in result.output
         )
 
@@ -2986,7 +3227,7 @@ def test_export_update_no_db():
         # edited files will be re-exported because there won't be an edited signature
         # in the database
         assert (
-            "Exported: 0 photos, updated: 2 photos, skipped: 6 photos, updated EXIF data: 0 photos"
+            "Processed: 7 photos, exported: 0, updated: 2, skipped: 6, updated EXIF data: 0, missing: 1, error: 0"
             in result.output
         )
         assert os.path.isfile(OSXPHOTOS_EXPORT_DB)
@@ -3025,7 +3266,7 @@ def test_export_then_hardlink():
             ],
         )
         assert result.exit_code == 0
-        assert "Exported: 8 photos" in result.output
+        assert "Processed: 7 photos, exported: 8, missing: 1, error: 0" in result.output
         assert os.path.samefile(CLI_EXPORT_UUID_FILENAME, photo.path)
 
 
@@ -3045,7 +3286,7 @@ def test_export_dry_run():
             export, [os.path.join(cwd, CLI_PHOTOS_DB), ".", "-V", "--dry-run"]
         )
         assert result.exit_code == 0
-        assert "Exported: 8 photos" in result.output
+        assert "Processed: 7 photos, exported: 8, missing: 1, error: 0" in result.output
         for filepath in CLI_EXPORT_FILENAMES:
             assert f"Exported {filepath}" in result.output
             assert not os.path.isfile(filepath)
@@ -3089,7 +3330,7 @@ def test_export_update_edits_dry_run():
         )
         assert result.exit_code == 0
         assert (
-            "Exported: 1 photo, updated: 1 photo, skipped: 6 photos, updated EXIF data: 0 photos"
+            "Processed: 7 photos, exported: 1, updated: 1, skipped: 6, updated EXIF data: 0, missing: 1, error: 0"
             in result.output
         )
 
@@ -3124,7 +3365,7 @@ def test_export_directory_template_1_dry_run():
             ],
         )
         assert result.exit_code == 0
-        assert "Exported: 8 photos" in result.output
+        assert "exported: 8" in result.output
         workdir = os.getcwd()
         for filepath in CLI_EXPORTED_DIRECTORY_TEMPLATE_FILENAMES1:
             assert f"Exported {filepath}" in result.output
@@ -3160,7 +3401,8 @@ def test_export_touch_files():
         )
         assert result.exit_code == 0
 
-        assert "Exported: 18 photos, touched date: 16 photos" in result.output
+        assert "exported: 18" in result.output
+        assert "touched date: 16" in result.output
 
         for fname, mtime in zip(CLI_EXPORT_BY_DATE, CLI_EXPORT_BY_DATE_TOUCH_TIMES):
             st = os.stat(fname)
@@ -3192,7 +3434,7 @@ def test_export_touch_files_update():
         )
         assert result.exit_code == 0
 
-        assert "Exported: 18 photos" in result.output
+        assert "exported: 18" in result.output
 
         assert not pathlib.Path(CLI_EXPORT_BY_DATE[0]).is_file()
 
@@ -3202,7 +3444,7 @@ def test_export_touch_files_update():
         )
         assert result.exit_code == 0
 
-        assert "Exported: 18 photos" in result.output
+        assert "exported: 18" in result.output
 
         assert pathlib.Path(CLI_EXPORT_BY_DATE[0]).is_file()
 
@@ -3213,10 +3455,7 @@ def test_export_touch_files_update():
         )
         assert result.exit_code == 0
 
-        assert (
+        assert "skipped: 18" in result.output
-            "Exported: 0 photos, updated: 0 photos, skipped: 18 photos, updated EXIF data: 0 photos"
-            in result.output
-        )
 
         # --update --touch-file --dry-run
         result = runner.invoke(
@@ -3231,10 +3470,8 @@ def test_export_touch_files_update():
             ],
         )
         assert result.exit_code == 0
-        assert (
+        assert "skipped: 18" in result.output
-            "Exported: 0 photos, updated: 0 photos, skipped: 18 photos, updated EXIF data: 0 photos, touched date: 16 photos"
+        assert "touched date: 16" in result.output
-            in result.output
-        )
 
         for fname, mtime in zip(
             CLI_EXPORT_BY_DATE_NEED_TOUCH, CLI_EXPORT_BY_DATE_NEED_TOUCH_TIMES
@@ -3254,10 +3491,8 @@ def test_export_touch_files_update():
             ],
         )
         assert result.exit_code == 0
-        assert (
+        assert "skipped: 18" in result.output
-            "Exported: 0 photos, updated: 0 photos, skipped: 18 photos, updated EXIF data: 0 photos, touched date: 16 photos"
+        assert "touched date: 16" in result.output
-            in result.output
-        )
 
         for fname, mtime in zip(
             CLI_EXPORT_BY_DATE_NEED_TOUCH, CLI_EXPORT_BY_DATE_NEED_TOUCH_TIMES
@@ -3280,10 +3515,8 @@ def test_export_touch_files_update():
             ],
         )
         assert result.exit_code == 0
-        assert (
+        assert "updated: 1, skipped: 17" in result.output
-            "Exported: 0 photos, updated: 1 photo, skipped: 17 photos, updated EXIF data: 0 photos, touched date: 1 photo"
+        assert "touched date: 1" in result.output
-            in result.output
-        )
 
         for fname, mtime in zip(CLI_EXPORT_BY_DATE, CLI_EXPORT_BY_DATE_TOUCH_TIMES):
             st = os.stat(fname)
@@ -3296,10 +3529,7 @@ def test_export_touch_files_update():
         )
         assert result.exit_code == 0
 
-        assert (
+        assert "skipped: 18" in result.output
-            "Exported: 0 photos, updated: 0 photos, skipped: 18 photos, updated EXIF data: 0 photos"
-            in result.output
-        )
 
 
 @pytest.mark.skip("TODO: This fails on some machines but not all")
@@ -3329,7 +3559,7 @@ def test_export_touch_files_exiftool_update():
         )
         assert result.exit_code == 0
 
-        assert "Exported: 18 photos" in result.output
+        assert "exported: 18" in result.output
 
         assert not pathlib.Path(CLI_EXPORT_BY_DATE[0]).is_file()
 
@@ -3339,7 +3569,7 @@ def test_export_touch_files_exiftool_update():
         )
         assert result.exit_code == 0
 
-        assert "Exported: 18 photos" in result.output
+        assert "exported: 18" in result.output
 
         assert pathlib.Path(CLI_EXPORT_BY_DATE[0]).is_file()
 
@@ -3350,10 +3580,7 @@ def test_export_touch_files_exiftool_update():
         )
         assert result.exit_code == 0
 
-        assert (
+        assert "skipped: 18" in result.output
-            "Exported: 0 photos, updated: 0 photos, skipped: 18 photos, updated EXIF data: 0 photos"
-            in result.output
-        )
 
         # --update --exiftool --dry-run
         result = runner.invoke(
@@ -3369,10 +3596,8 @@ def test_export_touch_files_exiftool_update():
         )
         assert result.exit_code == 0
 
-        assert (
+        assert "updated: 18" in result.output
-            "Exported: 0 photos, updated: 18 photos, skipped: 0 photos, updated EXIF data: 18 photos"
+        assert "updated EXIF data: 18" in result.output
-            in result.output
-        )
 
         # --update --exiftool
         result = runner.invoke(
@@ -3386,11 +3611,8 @@ def test_export_touch_files_exiftool_update():
             ],
         )
         assert result.exit_code == 0
-
+        assert "updated: 18" in result.output
-        assert (
+        assert "updated EXIF data: 18" in result.output
-            "Exported: 0 photos, updated: 18 photos, skipped: 0 photos, updated EXIF data: 18 photos"
-            in result.output
-        )
 
         # --update --touch-file --exiftool --dry-run
         result = runner.invoke(
@@ -3406,10 +3628,8 @@ def test_export_touch_files_exiftool_update():
             ],
         )
         assert result.exit_code == 0
-        assert (
+        assert "skipped: 18" in result.output
-            "Exported: 0 photos, updated: 0 photos, skipped: 18 photos, updated EXIF data: 0 photos, touched date: 18 photos"
+        assert "touched date: 18" in result.output
-            in result.output
-        )
 
         # --update --touch-file --exiftool
         result = runner.invoke(
@@ -3424,10 +3644,8 @@ def test_export_touch_files_exiftool_update():
             ],
         )
         assert result.exit_code == 0
-        assert (
+        assert "skipped: 18" in result.output
-            "Exported: 0 photos, updated: 0 photos, skipped: 18 photos, updated EXIF data: 0 photos, touched date: 18 photos"
+        assert "touched date: 18" in result.output
-            in result.output
-        )
 
         for fname, mtime in zip(CLI_EXPORT_BY_DATE, CLI_EXPORT_BY_DATE_TOUCH_TIMES):
             st = os.stat(fname)
@@ -3449,10 +3667,10 @@ def test_export_touch_files_exiftool_update():
             ],
         )
         assert result.exit_code == 0
-        assert (
+        assert "updated: 1" in result.output
-            "Exported: 0 photos, updated: 1 photo, skipped: 17 photos, updated EXIF data: 1 photo, touched date: 1 photo"
+        assert "skipped: 17" in result.output
-            in result.output
+        assert "updated EXIF data: 1" in result.output
-        )
+        assert "touched date: 1" in result.output
 
         for fname, mtime in zip(CLI_EXPORT_BY_DATE, CLI_EXPORT_BY_DATE_TOUCH_TIMES):
             st = os.stat(fname)
@@ -3471,10 +3689,8 @@ def test_export_touch_files_exiftool_update():
             ],
         )
         assert result.exit_code == 0
-        assert (
+        assert "exported: 0" in result.output
-            "Exported: 0 photos, updated: 0 photos, skipped: 18 photos, updated EXIF data: 0 photos, touched date: 0 photos"
+        assert "skipped: 18" in result.output
-            in result.output
-        )
 
         # run update without --touch-file
         result = runner.invoke(
@@ -3489,10 +3705,8 @@ def test_export_touch_files_exiftool_update():
         )
         assert result.exit_code == 0
 
-        assert (
+        assert "exported: 0" in result.output
-            "Exported: 0 photos, updated: 0 photos, skipped: 18 photos, updated EXIF data: 0 photos"
+        assert "skipped: 18" in result.output
-            in result.output
-        )
 
 
 def test_labels():
@@ -3587,3 +3801,116 @@ def test_persons():
 
     json_got = json.loads(result.output)
     assert json_got == PERSONS_JSON
+
+
+def test_export_report():
+    """ test export with --report option """
+    import glob
+    import os
+    import os.path
+    import osxphotos
+    from osxphotos.__main__ import export
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with runner.isolated_filesystem():
+        result = runner.invoke(
+            export,
+            [os.path.join(cwd, CLI_PHOTOS_DB), ".", "-V", "--report", "report.csv"],
+        )
+        assert result.exit_code == 0
+        assert "Writing export report" in result.output
+        assert os.path.exists("report.csv")
+
+
+def test_export_report_not_a_file():
+    """ test export with --report option and bad report value """
+    import glob
+    import os
+    import os.path
+    import osxphotos
+    from osxphotos.__main__ import export
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with runner.isolated_filesystem():
+        result = runner.invoke(
+            export, [os.path.join(cwd, CLI_PHOTOS_DB), ".", "-V", "--report", "."]
+        )
+        assert result.exit_code != 0
+        assert "Aborted!" in result.output
+
+
+def test_export_as_hardlink_download_missing():
+    """ test export with incompatible export options """
+    import glob
+    import os
+    import os.path
+    import osxphotos
+    from osxphotos.__main__ import export
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with runner.isolated_filesystem():
+        result = runner.invoke(
+            export,
+            [
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                ".",
+                "-V",
+                "--download-missing",
+                "--export-as-hardlink",
+                ".",
+            ],
+        )
+        assert result.exit_code != 0
+        assert "Aborted!" in result.output
+
+
+def test_export_missing():
+    """ test export with --missing """
+    import glob
+    import os
+    import os.path
+    import osxphotos
+    from osxphotos.__main__ import export
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with runner.isolated_filesystem():
+        result = runner.invoke(
+            export,
+            [
+                os.path.join(cwd, PHOTOS_DB_15_7),
+                ".",
+                "-V",
+                "--missing",
+                "--download-missing",
+                ".",
+            ],
+        )
+        assert result.exit_code == 0
+        assert "Exporting 2 photos" in result.output
+
+
+def test_export_missing_not_download_missing():
+    """ test export with incompatible export options """
+    import glob
+    import os
+    import os.path
+    import osxphotos
+    from osxphotos.__main__ import export
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with runner.isolated_filesystem():
+        result = runner.invoke(
+            export, [os.path.join(cwd, CLI_PHOTOS_DB), ".", "-V", "--missing", "."]
+        )
+        assert result.exit_code != 0
+        assert "Aborted!" in result.output

tests/test_datetime_utils.py

@@ -1,90 +1,96 @@
-""" test datetime_utils """
+from datetime import date, timezone
 import pytest
 
+from osxphotos.datetime_utils import *
+
 
 def test_get_local_tz():
-    """ test get_local_tz during time with no DST """
     import datetime
     import os
-    import time
-
-    from osxphotos.datetime_utils import get_local_tz
 
     os.environ["TZ"] = "US/Pacific"
-    time.tzset()
 
-    dt = datetime.datetime(2018, 12, 31, 0, 0, 0)
+    dt = datetime.datetime(2020, 9, 1, 21, 10, 00)
-    local_tz = get_local_tz(dt)
+    tz = get_local_tz(dt)
-    assert local_tz == datetime.timezone(
+    assert tz == datetime.timezone(offset=datetime.timedelta(seconds=-25200))
-        datetime.timedelta(days=-1, seconds=57600), "PST"
-    )
 
-
+    dt = datetime.datetime(2020, 12, 1, 21, 10, 00)
-def test_get_local_tz_dst():
+    tz = get_local_tz(dt)
-    """ test get_local_tz during time with DST """
+    assert tz == datetime.timezone(offset=datetime.timedelta(seconds=-28800))
-    import datetime
-    import os
-    import time
-
-    from osxphotos.datetime_utils import get_local_tz
-
-    os.environ["TZ"] = "US/Pacific"
-    time.tzset()
-
-    dt = datetime.datetime(2018, 6, 30, 0, 0, 0)
-    local_tz = get_local_tz(dt)
-    assert local_tz == datetime.timezone(
-        datetime.timedelta(days=-1, seconds=61200), "PDT"
-    )
-
-
-def test_datetime_remove_tz():
-    """ test datetime_remove_tz """
-    import datetime
-
-    from osxphotos.datetime_utils import datetime_remove_tz
-
-    dt = datetime.datetime(
-        2018,
-        12,
-        31,
-        tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=57600), "PST"),
-    )
-    dt_no_tz = datetime_remove_tz(dt)
-    assert dt_no_tz.tzinfo is None
 
 
 def test_datetime_has_tz():
-    """ test datetime_has_tz """
     import datetime
 
-    from osxphotos.datetime_utils import datetime_has_tz
+    tz = datetime.timezone(offset=datetime.timedelta(seconds=-28800))
-
+    dt = datetime.datetime(2020, 9, 1, 21, 10, 00, tzinfo=tz)
-    dt = datetime.datetime(
-        2018,
-        12,
-        31,
-        tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=57600), "PST"),
-    )
     assert datetime_has_tz(dt)
 
-    dt_notz = datetime.datetime(2018, 12, 31)
+    dt = datetime.datetime(2020, 9, 1, 21, 10, 00)
-    assert not datetime_has_tz(dt_notz)
+    assert not datetime_has_tz(dt)
+
+
+def test_datetime_tz_to_utc():
+    import datetime
+
+    tz = datetime.timezone(offset=datetime.timedelta(seconds=-25200))
+    dt = datetime.datetime(2020, 9, 1, 22, 6, 0, tzinfo=tz)
+    utc = datetime_tz_to_utc(dt)
+    assert utc == datetime.datetime(2020, 9, 2, 5, 6, 0, tzinfo=datetime.timezone.utc)
+
+
+def test_datetime_remove_tz():
+    import datetime
+    import os
+
+    os.environ["TZ"] = "US/Pacific"
+
+    tz = datetime.timezone(offset=datetime.timedelta(seconds=-25200))
+    dt = datetime.datetime(2020, 9, 1, 22, 6, 0, tzinfo=tz)
+    dt = datetime_remove_tz(dt)
+    assert dt == datetime.datetime(2020, 9, 1, 22, 6, 0)
+    assert not datetime_has_tz(dt)
+
+
+def test_datetime_naive_to_utc():
+    import datetime
+
+    dt = datetime.datetime(2020, 9, 1, 12, 0, 0)
+    utc = datetime_naive_to_utc(dt)
+    assert utc == datetime.datetime(2020, 9, 1, 12, 0, 0, tzinfo=datetime.timezone.utc)
 
 
 def test_datetime_naive_to_local():
-    """ test datetime_naive_to_local """
     import datetime
     import os
-    import time
-
-    from osxphotos.datetime_utils import datetime_naive_to_local
 
     os.environ["TZ"] = "US/Pacific"
-    time.tzset()
 
-    dt = datetime.datetime(2018, 6, 30, 0, 0, 0)
+    tz = datetime.timezone(offset=datetime.timedelta(seconds=-25200))
-    dt_local = datetime_naive_to_local(dt)
+    dt = datetime.datetime(2020, 9, 1, 12, 0, 0)
-    assert dt_local.tzinfo == datetime.timezone(
+    utc = datetime_naive_to_local(dt)
-        datetime.timedelta(days=-1, seconds=61200), "PDT"
+    assert utc == datetime.datetime(2020, 9, 1, 12, 0, 0, tzinfo=tz)
-    )
+
+
+def test_datetime_utc_to_local():
+    import datetime
+    import os
+
+    os.environ["TZ"] = "US/Pacific"
+
+    tz = datetime.timezone(offset=datetime.timedelta(seconds=-25200))
+    utc = datetime.datetime(2020, 9, 1, 19, 0, 0, tzinfo=datetime.timezone.utc)
+    dt = datetime_utc_to_local(utc)
+    assert dt == datetime.datetime(2020, 9, 1, 12, 0, 0, tzinfo=tz)
+
+
+def test_datetime_utc_to_local_2():
+    import datetime
+    import os
+
+    os.environ["TZ"] = "CEST"
+
+    tz = datetime.timezone(offset=datetime.timedelta(seconds=7200))
+    utc = datetime.datetime(2020, 9, 1, 19, 0, 0, tzinfo=datetime.timezone.utc)
+    dt = datetime_utc_to_local(utc)
+    assert dt == datetime.datetime(2020, 9, 1, 21, 0, 0, tzinfo=tz)

tests/test_export_catalina_10_15_7.py

@@ -68,8 +68,7 @@ XMP_JPG_FILENAME = "Pumkins1.jpg"
 
 EXIF_JSON_UUID = UUID_DICT["has_adjustments"]
 EXIF_JSON_EXPECTED = """ 
-    [{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", 
+    [{"EXIF:ImageDescription": "Bride Wedding day", 
-    "EXIF:ImageDescription": "Bride Wedding day", 
     "XMP:Description": "Bride Wedding day", 
     "XMP:TagsList": ["wedding"], 
     "IPTC:Keywords": ["wedding"], 
@@ -84,8 +83,7 @@ EXIF_JSON_EXPECTED = """
     """
 
 EXIF_JSON_EXPECTED_IGNORE_DATE_MODIFIED = """ 
-    [{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", 
+    [{"EXIF:ImageDescription": "Bride Wedding day", 
-    "EXIF:ImageDescription": "Bride Wedding day", 
     "XMP:Description": "Bride Wedding day", 
     "XMP:TagsList": ["wedding"], 
     "IPTC:Keywords": ["wedding"], 
@@ -544,8 +542,7 @@ def test_exiftool_json_sidecar_keyword_template_long(caplog):
 
     json_expected = json.loads(
         """
-        [{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", 
+        [{"EXIF:ImageDescription": "Bride Wedding day", 
-        "EXIF:ImageDescription": "Bride Wedding day", 
         "XMP:Description": "Bride Wedding day", 
         "XMP:TagsList": ["wedding", "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"], 
         "IPTC:Keywords": ["wedding", "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"], 
@@ -594,8 +591,7 @@ def test_exiftool_json_sidecar_keyword_template():
 
     json_expected = json.loads(
         """
-        [{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", 
+        [{"EXIF:ImageDescription": "Bride Wedding day", 
-        "EXIF:ImageDescription": "Bride Wedding day", 
         "XMP:Description": "Bride Wedding day", 
         "XMP:TagsList": ["wedding", "Folder1/SubFolder2/AlbumInFolder", "I have a deleted twin"], 
         "IPTC:Keywords": ["wedding", "Folder1/SubFolder2/AlbumInFolder", "I have a deleted twin"], 
@@ -655,8 +651,7 @@ def test_exiftool_json_sidecar_use_persons_keyword():
 
     json_expected = json.loads(
         """
-        [{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", 
+        [{"EXIF:ImageDescription": "Girls with pumpkins", 
-        "EXIF:ImageDescription": "Girls with pumpkins", 
         "XMP:Description": "Girls with pumpkins", 
         "XMP:Title": "Can we carry this?", 
         "XMP:TagsList": ["Kids", "Suzy", "Katie"], 
@@ -698,8 +693,7 @@ def test_exiftool_json_sidecar_use_albums_keyword():
 
     json_expected = json.loads(
         """
-        [{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", 
+        [{"EXIF:ImageDescription": "Girls with pumpkins", 
-        "EXIF:ImageDescription": "Girls with pumpkins", 
         "XMP:Description": "Girls with pumpkins", 
         "XMP:Title": "Can we carry this?", 
         "XMP:TagsList": ["Kids", "Pumpkin Farm", "Test Album"], 
@@ -1029,7 +1023,7 @@ def test_xmp_sidecar_gps():
             xmlns:photoshop="http://ns.adobe.com/photoshop/1.0/">
         <photoshop:SidecarForExtension>jpg</photoshop:SidecarForExtension>
         <dc:description></dc:description>
-        <dc:title>St. James's Park</dc:title>
+        <dc:title>St. James&#39;s Park</dc:title>
         <!-- keywords and persons listed in <dc:subject> as Photos does -->
         <dc:subject>
             <rdf:Seq>
@@ -1038,7 +1032,7 @@ def test_xmp_sidecar_gps():
                 <rdf:li>London</rdf:li>
                 <rdf:li>United Kingdom</rdf:li>
                 <rdf:li>London 2018</rdf:li>
-                <rdf:li>St. James's Park</rdf:li>
+                <rdf:li>St. James&#39;s Park</rdf:li>
             </rdf:Seq>
         </dc:subject>
         <photoshop:DateCreated>2018-10-13T09:18:12.501000-04:00</photoshop:DateCreated>
@@ -1055,7 +1049,7 @@ def test_xmp_sidecar_gps():
                 <rdf:li>London</rdf:li>
                 <rdf:li>United Kingdom</rdf:li>
                 <rdf:li>London 2018</rdf:li>
-                <rdf:li>St. James's Park</rdf:li>
+                <rdf:li>St. James&#39;s Park</rdf:li>
             </rdf:Seq>
         </digiKam:TagsList>
         </rdf:Description>
@@ -1066,10 +1060,8 @@ def test_xmp_sidecar_gps():
         </rdf:Description>
         <rdf:Description rdf:about=""
             xmlns:exif='http://ns.adobe.com/exif/1.0/'>
-        <exif:GPSLongitudeRef>W</exif:GPSLongitudeRef>
+        <exif:GPSLongitude>0,7.908329999999999W</exif:GPSLongitude>
-        <exif:GPSLongitude>0.1318055</exif:GPSLongitude>
+        <exif:GPSLatitude>51,30.21430019999997N</exif:GPSLatitude>
-        <exif:GPSLatitude>51.50357167</exif:GPSLatitude>
-        <exif:GPSLatitudeRef>N</exif:GPSLatitudeRef>
         </rdf:Description>
    </rdf:RDF>
 </x:xmpmeta>"""

tests/test_export_db.py

@@ -4,6 +4,8 @@ import pytest
 
 EXIF_DATA = """[{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", "EXIF:ImageDescription": "\u2068Elder Park\u2069, \u2068Adelaide\u2069, \u2068Australia\u2069", "XMP:Description": "\u2068Elder Park\u2069, \u2068Adelaide\u2069, \u2068Australia\u2069", "XMP:Title": "Elder Park", "EXIF:GPSLatitude": "34 deg 55' 8.01\" S", "EXIF:GPSLongitude": "138 deg 35' 48.70\" E", "Composite:GPSPosition": "34 deg 55' 8.01\" S, 138 deg 35' 48.70\" E", "EXIF:GPSLatitudeRef": "South", "EXIF:GPSLongitudeRef": "East", "EXIF:DateTimeOriginal": "2017:06:20 17:18:56", "EXIF:OffsetTimeOriginal": "+09:30", "EXIF:ModifyDate": "2020:05:18 14:42:04"}]"""
 INFO_DATA = """{"uuid": "3DD2C897-F19E-4CA6-8C22-B027D5A71907", "filename": "3DD2C897-F19E-4CA6-8C22-B027D5A71907.jpeg", "original_filename": "IMG_4547.jpg", "date": "2017-06-20T17:18:56.518000+09:30", "description": "\u2068Elder Park\u2069, \u2068Adelaide\u2069, \u2068Australia\u2069", "title": "Elder Park", "keywords": [], "labels": ["Statue", "Art"], "albums": ["AlbumInFolder"], "folders": {"AlbumInFolder": ["Folder1", "SubFolder2"]}, "persons": [], "path": "/Users/rhet/Pictures/Test-10.15.4.photoslibrary/originals/3/3DD2C897-F19E-4CA6-8C22-B027D5A71907.jpeg", "ismissing": false, "hasadjustments": true, "external_edit": false, "favorite": false, "hidden": false, "latitude": -34.91889167000001, "longitude": 138.59686167, "path_edited": "/Users/rhet/Pictures/Test-10.15.4.photoslibrary/resources/renders/3/3DD2C897-F19E-4CA6-8C22-B027D5A71907_1_201_a.jpeg", "shared": false, "isphoto": true, "ismovie": false, "uti": "public.jpeg", "burst": false, "live_photo": false, "path_live_photo": null, "iscloudasset": false, "incloud": null, "date_modified": "2020-05-18T14:42:04.608664+09:30", "portrait": false, "screenshot": false, "slow_mo": false, "time_lapse": false, "hdr": false, "selfie": false, "panorama": false, "has_raw": false, "uti_raw": null, "path_raw": null, "place": {"name": "Elder Park, Adelaide, South Australia, Australia, River Torrens", "names": {"field0": [], "country": ["Australia"], "state_province": ["South Australia"], "sub_administrative_area": ["Adelaide"], "city": ["Adelaide", "Adelaide"], "field5": [], "additional_city_info": ["Adelaide CBD", "Tarndanya"], "ocean": [], "area_of_interest": ["Elder Park", ""], "inland_water": ["River Torrens", "River Torrens"], "field10": [], "region": [], "sub_throughfare": [], "field13": [], "postal_code": [], "field15": [], "field16": [], "street_address": [], "body_of_water": ["River Torrens", "River Torrens"]}, "country_code": "AU", "ishome": false, "address_str": "River Torrens, Adelaide SA, Australia", "address": {"street": null, "sub_locality": "Tarndanya", "city": "Adelaide", "sub_administrative_area": "Adelaide", "state_province": "SA", "postal_code": null, "country": "Australia", "iso_country_code": "AU"}}, "exif": {"flash_fired": false, "iso": 320, "metering_mode": 3, "sample_rate": null, "track_format": null, "white_balance": 0, "aperture": 2.2, "bit_rate": null, "duration": null, "exposure_bias": 0.0, "focal_length": 4.15, "fps": null, "latitude": null, "longitude": null, "shutter_speed": 0.058823529411764705, "camera_make": "Apple", "camera_model": "iPhone 6s", "codec": null, "lens_model": "iPhone 6s back camera 4.15mm f/2.2"}}"""
+SIDECAR_DATA = """FOO_BAR"""
+
 EXIF_DATA2 = """[{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", "XMP:Title": "St. James's Park", "XMP:TagsList": ["London 2018", "St. James's Park", "England", "United Kingdom", "UK", "London"], "IPTC:Keywords": ["London 2018", "St. James's Park", "England", "United Kingdom", "UK", "London"], "XMP:Subject": ["London 2018", "St. James's Park", "England", "United Kingdom", "UK", "London"], "EXIF:GPSLatitude": "51 deg 30' 12.86\" N", "EXIF:GPSLongitude": "0 deg 7' 54.50\" W", "Composite:GPSPosition": "51 deg 30' 12.86\" N, 0 deg 7' 54.50\" W", "EXIF:GPSLatitudeRef": "North", "EXIF:GPSLongitudeRef": "West", "EXIF:DateTimeOriginal": "2018:10:13 09:18:12", "EXIF:OffsetTimeOriginal": "-04:00", "EXIF:ModifyDate": "2019:12:08 14:06:44"}]"""
 INFO_DATA2 = """{"uuid": "F2BB3F98-90F0-4E4C-A09B-25C6822A4529", "filename": "F2BB3F98-90F0-4E4C-A09B-25C6822A4529.jpeg", "original_filename": "IMG_8440.JPG", "date": "2019-06-11T11:42:06.711805-07:00", "description": null, "title": null, "keywords": [], "labels": ["Sky", "Cloudy", "Fence", "Land", "Outdoor", "Park", "Amusement Park", "Roller Coaster"], "albums": [], "folders": {}, "persons": [], "path": "/Volumes/MacBook Catalina - Data/Users/rhet/Pictures/Photos Library.photoslibrary/originals/F/F2BB3F98-90F0-4E4C-A09B-25C6822A4529.jpeg", "ismissing": false, "hasadjustments": false, "external_edit": false, "favorite": false, "hidden": false, "latitude": 33.81558666666667, "longitude": -117.99298, "path_edited": null, "shared": false, "isphoto": true, "ismovie": false, "uti": "public.jpeg", "burst": false, "live_photo": false, "path_live_photo": null, "iscloudasset": true, "incloud": true, "date_modified": "2019-10-14T00:51:47.141950-07:00", "portrait": false, "screenshot": false, "slow_mo": false, "time_lapse": false, "hdr": false, "selfie": false, "panorama": false, "has_raw": false, "uti_raw": null, "path_raw": null, "place": {"name": "Adventure City, Stanton, California, United States", "names": {"field0": [], "country": ["United States"], "state_province": ["California"], "sub_administrative_area": ["Orange"], "city": ["Stanton", "Anaheim", "Anaheim"], "field5": [], "additional_city_info": ["West Anaheim"], "ocean": [], "area_of_interest": ["Adventure City", "Adventure City"], "inland_water": [], "field10": [], "region": [], "sub_throughfare": [], "field13": [], "postal_code": [], "field15": [], "field16": [], "street_address": [], "body_of_water": []}, "country_code": "US", "ishome": false, "address_str": "Adventure City, 1240 S Beach Blvd, Anaheim, CA  92804, United States", "address": {"street": "1240 S Beach Blvd", "sub_locality": "West Anaheim", "city": "Stanton", "sub_administrative_area": "Orange", "state_province": "CA", "postal_code": "92804", "country": "United States", "iso_country_code": "US"}}, "exif": {"flash_fired": false, "iso": 25, "metering_mode": 5, "sample_rate": null, "track_format": null, "white_balance": 0, "aperture": 2.2, "bit_rate": null, "duration": null, "exposure_bias": 0.0, "focal_length": 4.15, "fps": null, "latitude": null, "longitude": null, "shutter_speed": 0.0004940711462450593, "camera_make": "Apple", "camera_model": "iPhone 6s", "codec": null, "lens_model": "iPhone 6s back camera 4.15mm f/2.2"}}"""
 DATABASE_VERSION1 = "tests/export_db_version1.db"
@@ -41,6 +43,8 @@ def test_export_db():
     assert db.get_stat_edited_for_file(filepath) == (10, 11, 12)
     db.set_stat_converted_for_file(filepath, (7, 8, 9))
     assert db.get_stat_converted_for_file(filepath) == (7, 8, 9)
+    db.set_sidecar_for_file(filepath, SIDECAR_DATA, (13, 14, 15))
+    assert db.get_sidecar_for_file(filepath) == (SIDECAR_DATA, (13, 14, 15))
 
     # test set_data which sets all at the same time
     filepath2 = os.path.join(tempdir.name, "test2.jpg")
@@ -109,6 +113,8 @@ def test_export_db_no_op():
     assert db.get_stat_converted_for_file(filepath) is None
     db.set_stat_edited_for_file(filepath, (10, 11, 12))
     assert db.get_stat_edited_for_file(filepath) is None
+    db.set_sidecar_for_file(filepath, SIDECAR_DATA, (13, 14, 15))
+    assert db.get_sidecar_for_file(filepath) == (None, (None, None, None))
 
     # test set_data which sets all at the same time
     filepath2 = os.path.join(tempdir.name, "test2.jpg")
@@ -160,6 +166,7 @@ def test_export_db_in_memory():
     db.set_stat_exif_for_file(filepath, (4, 5, 6))
     db.set_stat_converted_for_file(filepath, (7, 8, 9))
     db.set_stat_edited_for_file(filepath, (10, 11, 12))
+    db.set_sidecar_for_file(filepath, SIDECAR_DATA, (13, 14, 15))
 
     db.close()
 
@@ -176,6 +183,7 @@ def test_export_db_in_memory():
     assert dbram.get_stat_exif_for_file(filepath) == (4, 5, 6)
     assert dbram.get_stat_converted_for_file(filepath) == (7, 8, 9)
     assert dbram.get_stat_edited_for_file(filepath) == (10, 11, 12)
+    assert dbram.get_sidecar_for_file(filepath) == (SIDECAR_DATA, (13, 14, 15))
 
     # change a value
     dbram.set_uuid_for_file(filepath, "FUBAR")
@@ -185,6 +193,7 @@ def test_export_db_in_memory():
     dbram.set_stat_exif_for_file(filepath, (10, 11, 12))
     dbram.set_stat_converted_for_file(filepath, (1, 2, 3))
     dbram.set_stat_edited_for_file(filepath, (4, 5, 6))
+    dbram.set_sidecar_for_file(filepath, "FUBAR", (20, 21, 22))
 
     assert dbram.get_uuid_for_file(filepath_lower) == "FUBAR"
     assert dbram.get_info_for_uuid("FUBAR") == INFO_DATA2
@@ -193,6 +202,7 @@ def test_export_db_in_memory():
     assert dbram.get_stat_exif_for_file(filepath) == (10, 11, 12)
     assert dbram.get_stat_converted_for_file(filepath) == (1, 2, 3)
     assert dbram.get_stat_edited_for_file(filepath) == (4, 5, 6)
+    assert dbram.get_sidecar_for_file(filepath) == ("FUBAR", (20, 21, 22))
 
     dbram.close()
 
@@ -205,6 +215,7 @@ def test_export_db_in_memory():
     assert db.get_stat_exif_for_file(filepath) == (4, 5, 6)
     assert db.get_stat_converted_for_file(filepath) == (7, 8, 9)
     assert db.get_stat_edited_for_file(filepath) == (10, 11, 12)
+    assert db.get_sidecar_for_file(filepath) == (SIDECAR_DATA, (13, 14, 15))
 
     assert db.get_info_for_uuid("FUBAR") is None
 
@@ -232,6 +243,7 @@ def test_export_db_in_memory_nofile():
     dbram.set_stat_exif_for_file(filepath, (10, 11, 12))
     dbram.set_stat_converted_for_file(filepath, (1, 2, 3))
     dbram.set_stat_edited_for_file(filepath, (4, 5, 6))
+    dbram.set_sidecar_for_file(filepath, "FUBAR", (20, 21, 22))
 
     assert dbram.get_uuid_for_file(filepath_lower) == "FUBAR"
     assert dbram.get_info_for_uuid("FUBAR") == INFO_DATA2
@@ -240,5 +252,6 @@ def test_export_db_in_memory_nofile():
     assert dbram.get_stat_exif_for_file(filepath) == (10, 11, 12)
     assert dbram.get_stat_converted_for_file(filepath) == (1, 2, 3)
     assert dbram.get_stat_edited_for_file(filepath) == (4, 5, 6)
+    assert dbram.get_sidecar_for_file(filepath) == ("FUBAR", (20, 21, 22))
 
     dbram.close()

tests/test_export_mojave_10_14_6.py

@@ -46,8 +46,7 @@ UUID_DICT = {
 }
 
 EXIF_JSON_EXPECTED = """
-    [{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", 
+    [{"XMP:Title": "St. James\'s Park", 
-    "XMP:Title": "St. James\'s Park", 
     "XMP:TagsList": ["UK", "England", "London", "United Kingdom", "London 2018", "St. James\'s Park"], 
     "IPTC:Keywords": ["UK", "England", "London", "United Kingdom", "London 2018", "St. James\'s Park"], 
     "XMP:Subject": ["UK", "England", "London", "United Kingdom", "London 2018", "St. James\'s Park"], 

tests/test_photokit.py

@@ -0,0 +1,387 @@
+""" test photokit.py methods """
+
+import os
+import pathlib
+import tempfile
+
+import pytest
+
+from osxphotos.photokit import (
+    LivePhotoAsset,
+    PhotoAsset,
+    PhotoLibrary,
+    VideoAsset,
+    PHOTOS_VERSION_CURRENT,
+    PHOTOS_VERSION_ORIGINAL,
+    PHOTOS_VERSION_UNADJUSTED,
+)
+
+skip_test = "OSXPHOTOS_TEST_EXPORT" not in os.environ
+pytestmark = pytest.mark.skipif(
+    skip_test, reason="Skip if not running with author's personal library."
+)
+
+
+UUID_DICT = {
+    "plain_photo": {
+        "uuid": "A8D646C3-89A9-4D74-8001-4EB46BA55B94",
+        "filename": "IMG_8844.JPG",
+    },
+    "hdr": {"uuid": "DA87C6FF-60E8-4DCB-A21D-9C57595667F1", "filename": "IMG_6162.JPG"},
+    "selfie": {
+        "uuid": "316AEBE0-971D-4A33-833C-6BDBFF83469B",
+        "filename": "IMG_1929.JPG",
+    },
+    "video": {
+        "uuid": "5814D9DE-FAB6-473A-9C9A-5A73C6DD1AF5",
+        "filename": "IMG_9411.TRIM.MOV",
+    },
+    "hasadjustments": {
+        "uuid": "2B2D5434-6D31-49E2-BF47-B973D34A317B",
+        "filename": "IMG_2860.JPG",
+        "adjusted_size": 3012634,
+        "unadjusted_size": 2580058,
+    },
+    "slow_mo": {
+        "uuid": "DAABC6D9-1FBA-4485-AA39-0A2B100300B1",
+        "filename": "IMG_4055.MOV",
+    },
+    "live_photo": {
+        "uuid": "612CE30B-3D8F-417A-9B14-EC42CBA10ACC",
+        "filename": "IMG_3259.HEIC",
+        "filename_video": "IMG_3259.mov",
+    },
+    "burst": {
+        "uuid": "CD97EC84-71F0-40C6-BAC1-2BABEE305CAC",
+        "filename": "IMG_8196.JPG",
+        "burst_selected": 3,
+        "burst_all": 5,
+    },
+}
+
+
+def test_fetch_uuid():
+    """ test fetch_uuid """
+    uuid = UUID_DICT["plain_photo"]["uuid"]
+    filename = UUID_DICT["plain_photo"]["filename"]
+
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+    assert isinstance(photo, PhotoAsset)
+
+
+def test_plain_photo():
+    """ test plain_photo """
+    uuid = UUID_DICT["plain_photo"]["uuid"]
+    filename = UUID_DICT["plain_photo"]["filename"]
+
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+    assert photo.original_filename == filename
+    assert photo.isphoto
+    assert not photo.ismovie
+
+
+def test_hdr():
+    """ test hdr """
+    uuid = UUID_DICT["hdr"]["uuid"]
+    filename = UUID_DICT["hdr"]["filename"]
+
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+    assert photo.original_filename == filename
+    assert photo.hdr
+
+
+def test_burst():
+    """ test burst and burstid """
+    test_dict = UUID_DICT["burst"]
+    uuid = test_dict["uuid"]
+    filename = test_dict["filename"]
+
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+    assert photo.original_filename == filename
+    assert photo.burst
+    assert photo.burstid
+
+
+
+# def test_selfie():
+#     """ test selfie """
+#     uuid = UUID_DICT["selfie"]["uuid"]
+#     filename = UUID_DICT["selfie"]["filename"]
+
+#     lib = PhotoLibrary()
+#     photo = lib.fetch_uuid(uuid)
+#     assert photo.original_filename == filename
+#     assert photo.selfie
+
+
+def test_video():
+    """ test ismovie """
+    uuid = UUID_DICT["video"]["uuid"]
+    filename = UUID_DICT["video"]["filename"]
+
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+    assert isinstance(photo, VideoAsset)
+    assert photo.original_filename == filename
+    assert photo.ismovie
+    assert not photo.isphoto
+
+
+def test_slow_mo():
+    """ test slow_mo """
+    test_dict = UUID_DICT["slow_mo"]
+    uuid = test_dict["uuid"]
+    filename = test_dict["filename"]
+
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+    assert isinstance(photo, VideoAsset)
+    assert photo.original_filename == filename
+    assert photo.ismovie
+    assert photo.slow_mo
+    assert not photo.isphoto
+
+
+### PhotoAsset
+
+
+def test_export_photo_original():
+    """ test PhotoAsset.export """
+    test_dict = UUID_DICT["hasadjustments"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, version=PHOTOS_VERSION_ORIGINAL)
+        export_path = pathlib.Path(export_path[0])
+        assert export_path.is_file()
+        filename = test_dict["filename"]
+        assert export_path.stem == pathlib.Path(filename).stem
+        assert export_path.stat().st_size == test_dict["unadjusted_size"]
+
+
+def test_export_photo_unadjusted():
+    """ test PhotoAsset.export """
+    test_dict = UUID_DICT["hasadjustments"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, version=PHOTOS_VERSION_UNADJUSTED)
+        export_path = pathlib.Path(export_path[0])
+        assert export_path.is_file()
+        filename = test_dict["filename"]
+        assert export_path.stem == pathlib.Path(filename).stem
+        assert export_path.stat().st_size == test_dict["unadjusted_size"]
+
+
+def test_export_photo_current():
+    """ test PhotoAsset.export """
+    test_dict = UUID_DICT["hasadjustments"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir)
+        export_path = pathlib.Path(export_path[0])
+        assert export_path.is_file()
+        filename = test_dict["filename"]
+        assert export_path.stem == pathlib.Path(filename).stem
+        assert export_path.stat().st_size == test_dict["adjusted_size"]
+
+
+### VideoAsset
+
+
+def test_export_video_original():
+    """ test VideoAsset.export """
+    test_dict = UUID_DICT["video"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, version=PHOTOS_VERSION_ORIGINAL)
+        export_path = pathlib.Path(export_path[0])
+        assert export_path.is_file()
+        filename = test_dict["filename"]
+        assert export_path.stem == pathlib.Path(filename).stem
+
+
+def test_export_video_unadjusted():
+    """ test VideoAsset.export """
+    test_dict = UUID_DICT["video"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, version=PHOTOS_VERSION_UNADJUSTED)
+        export_path = pathlib.Path(export_path[0])
+        assert export_path.is_file()
+        filename = test_dict["filename"]
+        assert export_path.stem == pathlib.Path(filename).stem
+
+
+def test_export_video_current():
+    """ test VideoAsset.export """
+    test_dict = UUID_DICT["video"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, version=PHOTOS_VERSION_CURRENT)
+        export_path = pathlib.Path(export_path[0])
+        assert export_path.is_file()
+        filename = test_dict["filename"]
+        assert export_path.stem == pathlib.Path(filename).stem
+
+
+### Slow-Mo VideoAsset
+
+
+def test_export_slow_mo_original():
+    """ test VideoAsset.export for slow mo video"""
+    test_dict = UUID_DICT["slow_mo"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, version=PHOTOS_VERSION_ORIGINAL)
+        export_path = pathlib.Path(export_path[0])
+        assert export_path.is_file()
+        filename = test_dict["filename"]
+        assert export_path.stem == pathlib.Path(filename).stem
+
+
+def test_export_slow_mo_unadjusted():
+    """ test VideoAsset.export for slow mo video"""
+    test_dict = UUID_DICT["slow_mo"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, version=PHOTOS_VERSION_UNADJUSTED)
+        export_path = pathlib.Path(export_path[0])
+        assert export_path.is_file()
+        filename = test_dict["filename"]
+        assert export_path.stem == pathlib.Path(filename).stem
+
+
+def test_export_slow_mo_current():
+    """ test VideoAsset.export for slow mo video"""
+    test_dict = UUID_DICT["slow_mo"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, version=PHOTOS_VERSION_CURRENT)
+        export_path = pathlib.Path(export_path[0])
+        assert export_path.is_file()
+        filename = test_dict["filename"]
+        assert export_path.stem == pathlib.Path(filename).stem
+
+
+### LivePhotoAsset
+
+
+def test_export_live_original():
+    """ test LivePhotoAsset.export """
+    test_dict = UUID_DICT["live_photo"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, version=PHOTOS_VERSION_ORIGINAL)
+        for f in export_path:
+            filepath = pathlib.Path(f)
+            assert filepath.is_file()
+            filename = test_dict["filename"]
+            assert filepath.stem == pathlib.Path(filename).stem
+
+
+def test_export_live_unadjusted():
+    """ test LivePhotoAsset.export """
+    test_dict = UUID_DICT["live_photo"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, version=PHOTOS_VERSION_UNADJUSTED)
+        for file in export_path:
+            filepath = pathlib.Path(file)
+            assert filepath.is_file()
+            filename = test_dict["filename"]
+            assert filepath.stem == pathlib.Path(filename).stem
+
+
+def test_export_live_current():
+    """ test LivePhotAsset.export """
+    test_dict = UUID_DICT["live_photo"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, version=PHOTOS_VERSION_CURRENT)
+        for file in export_path:
+            filepath = pathlib.Path(file)
+            assert filepath.is_file()
+            filename = test_dict["filename"]
+            assert filepath.stem == pathlib.Path(filename).stem
+
+
+def test_export_live_current_just_photo():
+    """ test LivePhotAsset.export """
+    test_dict = UUID_DICT["live_photo"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, photo=True, video=False)
+        assert len(export_path) == 1
+        assert export_path[0].lower().endswith(".heic")
+
+
+def test_export_live_current_just_video():
+    """ test LivePhotAsset.export """
+    test_dict = UUID_DICT["live_photo"]
+    uuid = test_dict["uuid"]
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+
+    with tempfile.TemporaryDirectory(prefix="photokit_test") as tempdir:
+        export_path = photo.export(tempdir, photo=False, video=True)
+        assert len(export_path) == 1
+        assert export_path[0].lower().endswith(".mov")
+
+
+def test_fetch_burst_uuid():
+    """ test fetch_burst_uuid """
+    test_dict = UUID_DICT["burst"]
+    uuid = test_dict["uuid"]
+    filename = test_dict["filename"]
+
+    lib = PhotoLibrary()
+    photo = lib.fetch_uuid(uuid)
+    bursts_selected = lib.fetch_burst_uuid(photo.burstid)
+    assert len(bursts_selected) == test_dict["burst_selected"]
+    assert isinstance(bursts_selected[0], PhotoAsset)
+
+    bursts_all = lib.fetch_burst_uuid(photo.burstid, all=True)
+    assert len(bursts_all) == test_dict["burst_all"]
+    assert isinstance(bursts_all[0], PhotoAsset)

tests/test_template.py

@@ -1,6 +1,13 @@
 """ Test template.py """
 import pytest
 
+from osxphotos.exiftool import get_exiftool_path
+
+try:
+    exiftool = get_exiftool_path()
+except:
+    exiftool = None
+
 PHOTOS_DB_PLACES = (
     "./tests/Test-Places-Catalina-10_15_7.photoslibrary/database/photos.db"
 )
@@ -57,6 +64,29 @@ UUID_BOOL_VALUES = {"hdr": "D11D25FF-5F31-47D2-ABA9-58418878DC15"}
 # Boolean type values that render to False
 UUID_BOOL_VALUES_NOT = {"hdr": "51F2BEF7-431A-4D31-8AC1-3284A57826AE"}
 
+# for exiftool template
+UUID_EXIFTOOL = {
+    "A92D9C26-3A50-4197-9388-CB5F7DB9FA91": {
+        "{exiftool:EXIF:Make}": ["Canon"],
+        "{exiftool:EXIF:Model}": ["Canon PowerShot G10"],
+        "{exiftool:EXIF:Make}/{exiftool:EXIF:Model}": ["Canon/Canon PowerShot G10"],
+        "{exiftool:IPTC:Keywords,foo}": ["foo"],
+    },
+    "DC99FBDD-7A52-4100-A5BB-344131646C30": {
+        "{exiftool:IPTC:Keywords}": [
+            "England",
+            "London",
+            "London 2018",
+            "St. James's Park",
+            "UK",
+            "United Kingdom",
+        ],
+        "{,+exiftool:IPTC:Keywords}": [
+            "England,London,London 2018,St. James's Park,UK,United Kingdom"
+        ],
+    },
+}
+
 TEMPLATE_VALUES = {
     "{name}": "128FB4C6-0B16-4E7D-9108-FB2E90DA1546",
     "{original_name}": "IMG_1064",
@@ -737,3 +767,15 @@ def test_expand_in_place_with_delim_single_value():
     for template in TEMPLATE_VALUES_TITLE:
         rendered, _ = photo.render_template(template)
         assert sorted(rendered) == sorted(TEMPLATE_VALUES_TITLE[template])
+
+
+@pytest.mark.skipif(exiftool is None, reason="exiftool not installed")
+def test_exiftool_template():
+    import osxphotos
+
+    photosdb = osxphotos.PhotosDB(dbfile=PHOTOS_DB_15_7)
+    for uuid in UUID_EXIFTOOL:
+        photo = photosdb.get_photo(uuid)
+        for template in UUID_EXIFTOOL[uuid]:
+            rendered, _ = photo.render_template(template)
+            assert sorted(rendered) == sorted(UUID_EXIFTOOL[uuid][template])