Support Open Graph description (#602)

* Support pytest for running tests

* Support extracting description from meta og:description property

* Revert changes to TOC

* Add test

---------

Co-authored-by: Sascha Ißbrücker <sascha.issbruecker@gmail.com>

README.md

@@ -9,11 +9,11 @@
 ## Overview
 - [Introduction](#introduction)
 - [Installation](#installation)
-    - [Using Docker](#using-docker)
+  - [Using Docker](#using-docker)
-    - [Using Docker Compose](#using-docker-compose)
+  - [Using Docker Compose](#using-docker-compose)
-    - [User Setup](#user-setup)
+  - [User Setup](#user-setup)
-    - [Reverse Proxy Setup](#reverse-proxy-setup)
+  - [Reverse Proxy Setup](#reverse-proxy-setup)
-    - [Managed Hosting Options](#managed-hosting-options)
+  - [Managed Hosting Options](#managed-hosting-options)
 - [Documentation](#documentation)
 - [Browser Extension](#browser-extension)
 - [Community](#community)
@@ -103,7 +103,7 @@ docker-compose up -d
 
 To complete the setup, you still have to [create an initial user](#user-setup), so that you can access your installation.
 
-### User setup
+### User Setup
 
 For security reasons, the linkding Docker image does not provide an initial user, so you have to create one after setting up an installation. To do so, replace the credentials in the following command and run it:
 
@@ -119,7 +119,7 @@ docker-compose exec linkding python manage.py createsuperuser --username=joe --e
 
 The command will prompt you for a secure password. After the command has completed you can start using the application by logging into the UI with your credentials.
 
-Alternatively you can automatically create an initial superuser on startup using the [`LD_SUPERUSER_NAME` option](docs/Options.md#LD_SUPERUSER_NAME). 
+Alternatively you can automatically create an initial superuser on startup using the [`LD_SUPERUSER_NAME` option](docs/Options.md#LD_SUPERUSER_NAME).
 
 ### Reverse Proxy Setup
 
@@ -211,7 +211,7 @@ This section lists community projects around using linkding, in alphabetical ord
 - [aiolinkding](https://github.com/bachya/aiolinkding) A Python3, async library to interact with the linkding REST API. By [bachya](https://github.com/bachya)
 - [feed2linkding](https://codeberg.org/strubbl/feed2linkding) A commandline utility to add all web feed item links to linkding via API call. By [Strubbl](https://github.com/Strubbl)
 - [Helm Chart](https://charts.pascaliske.dev/charts/linkding/) Helm Chart for deploying linkding inside a Kubernetes cluster. By [pascaliske](https://github.com/pascaliske)
-- [iOS Shortcut using API and Tagging](https://gist.github.com/andrewdolphin/a7dff49505e588d940bec55132fab8ad) An iOS shortcut using the Linkding API (no extra logins required) that pulls previously used tags and allows tagging at the time of link creation. 
+- [iOS Shortcut using API and Tagging](https://gist.github.com/andrewdolphin/a7dff49505e588d940bec55132fab8ad) An iOS shortcut using the Linkding API (no extra logins required) that pulls previously used tags and allows tagging at the time of link creation.
 - [Linka!](https://github.com/cmsax/linka) Web app (also a PWA) for quickly searching & opening bookmarks in linkding, support multi keywords, exclude mode and other advance options. By [cmsax](https://github.com/cmsax)
 - [linkding-cli](https://github.com/bachya/linkding-cli) A command-line interface (CLI) to interact with the linkding REST API. Powered by [aiolinkding](https://github.com/bachya/aiolinkding). By [bachya](https://github.com/bachya)
 - [linkding-extension](https://github.com/jeroenpardon/linkding-extension) Chromium compatible extension that wraps the linkding bookmarklet. Tested with Chrome, Edge, Brave. By [jeroenpardon](https://github.com/jeroenpardon)
@@ -224,7 +224,7 @@ This section lists community projects around using linkding, in alphabetical ord
 
 ### PikaPods
 
-[PikaPods](https://www.pikapods.com/) has a revenue sharing agreement with this project, sharing some of their revenue from hosting linkding instances. I do not intend to profit from this project financially, so I am in turn donating that revenue. Big thanks to PikaPods for making this possible. 
+[PikaPods](https://www.pikapods.com/) has a revenue sharing agreement with this project, sharing some of their revenue from hosting linkding instances. I do not intend to profit from this project financially, so I am in turn donating that revenue. Big thanks to PikaPods for making this possible.
 
 See the table below for a list of donations.
 
@@ -300,3 +300,8 @@ Start the Django development server with:
 python3 manage.py runserver
 ```
 The frontend is now available under http://localhost:8000
+
+Run all tests with pytest
+```
+pytest
+```

bookmarks/services/website_loader.py

@@ -41,8 +41,13 @@ def load_website_metadata(url: str):
 
         title = soup.title.string.strip() if soup.title is not None else None
         description_tag = soup.find('meta', attrs={'name': 'description'})
-        description = description = description_tag['content'].strip() if description_tag and description_tag[
+        description = description_tag['content'].strip() if description_tag and description_tag[
             'content'] else None
+
+        if not description:
+            description_tag = soup.find('meta', attrs={'property': 'og:description'})
+            description = description_tag['content'].strip() if description_tag and description_tag['content'] else None
+
         end = timezone.now()
         logger.debug(f'Parsing duration: {end - start}')
     finally:

bookmarks/tests/test_bookmark_new_view.py

@@ -183,9 +183,8 @@ class BookmarkNewViewTestCase(TestCase, BookmarkFactoryMixin):
           </div>
         ''', html)
 
+    def test_should_hide_notes_if_there_are_no_notes(self):
+        bookmark = self.setup_bookmark()
+        response = self.client.get(reverse('bookmarks:edit', args=[bookmark.id]))
 
-def test_should_hide_notes_if_there_are_no_notes(self):
+        self.assertContains(response, '<details class="notes">', count=1)
-    bookmark = self.setup_bookmark()
-    response = self.client.get(reverse('bookmarks:edit', args=[bookmark.id]))
-
-    self.assertContains(response, '<details class="notes">', count=1)

bookmarks/tests/test_website_loader.py

@@ -29,14 +29,17 @@ class WebsiteLoaderTestCase(TestCase):
         # clear cached metadata before test run
         website_loader.load_website_metadata.cache_clear()
 
-    def render_html_document(self, title, description):
+    def render_html_document(self, title, description='', og_description=''):
+        meta_description = f'<meta name="description" content="{description}">' if description else ''
+        meta_og_description = f'<meta property="og:description" content="{og_description}">' if og_description else ''
         return f'''
         <!DOCTYPE html>
         <html lang="en">
         <head>
             <meta charset="UTF-8">
             <title>{title}</title>
-            <meta name="description" content="{description}">
+            {meta_description}
+            {meta_og_description}
         </head>
         <body></body>
         </html>
@@ -94,3 +97,19 @@ class WebsiteLoaderTestCase(TestCase):
             metadata = website_loader.load_website_metadata('https://example.com')
             self.assertEqual('test title', metadata.title)
             self.assertEqual('test description', metadata.description)
+
+    def test_load_website_metadata_using_og_description(self):
+        with mock.patch('bookmarks.services.website_loader.load_page') as mock_load_page:
+            mock_load_page.return_value = self.render_html_document('test title', '',
+                                                                    og_description='test og description')
+            metadata = website_loader.load_website_metadata('https://example.com')
+            self.assertEqual('test title', metadata.title)
+            self.assertEqual('test og description', metadata.description)
+
+    def test_load_website_metadata_prefers_description_over_og_description(self):
+        with mock.patch('bookmarks.services.website_loader.load_page') as mock_load_page:
+            mock_load_page.return_value = self.render_html_document('test title', 'test description',
+                                                                    og_description='test og description')
+            metadata = website_loader.load_website_metadata('https://example.com')
+            self.assertEqual('test title', metadata.title)
+            self.assertEqual('test description', metadata.description)

pytest.ini

@@ -0,0 +1,4 @@
+[pytest]
+DJANGO_SETTINGS_MODULE = siteroot.settings.dev
+# -- recommended but optional:
+python_files = tests.py test_*.py *_tests.py

requirements.txt

@@ -1,13 +1,12 @@
 asgiref==3.5.2
 beautifulsoup4==4.11.1
-bleach==6.0.0
 bleach-allowlist==1.0.3
+bleach==6.0.0
 certifi==2023.7.22
 charset-normalizer==2.1.1
 click==8.1.3
 confusable-homoglyphs==3.2.0
 coverage==5.5
-Django==4.1.13
 django-appconf==1.0.5
 django-compressor==4.1
 django-debug-toolbar==3.6.0
@@ -15,6 +14,7 @@ django-generate-secret-key==1.0.2
 django-registration==3.3
 django-sass-processor==1.2.1
 django-widget-tweaks==1.4.12
+Django==4.1.13
 django4-background-tasks==1.2.7
 djangorestframework==3.13.1
 greenlet==3.0.1
@@ -24,6 +24,8 @@ Markdown==3.4.3
 playwright==1.40.0
 psycopg2-binary==2.9.5
 pyee==11.0.1
+pytest-django==4.7.0
+pytest==7.4.4
 python-dateutil==2.8.2
 pytz==2022.2.1
 rcssmin==1.1.0