feat: add a per-recipient Format option to convert HTML bodies (FMG-9) #13
12 changed files with 1773 additions and 77 deletions
75
README.md
75
README.md
|
|
@ -29,6 +29,11 @@ Targets = [
|
||||||
"discord://token@id",
|
"discord://token@id",
|
||||||
"slack://token@channel"
|
"slack://token@channel"
|
||||||
]
|
]
|
||||||
|
# Optional: how an HTML message body should reach these targets.
|
||||||
|
# "raw" forward the message unchanged (default)
|
||||||
|
# "markdown" render an HTML body as Markdown
|
||||||
|
# See "Converting HTML messages" below.
|
||||||
|
# Format = "markdown"
|
||||||
|
|
||||||
# Single Target (Deprecated)
|
# Single Target (Deprecated)
|
||||||
# The Target field is still supported for backward compatibility
|
# The Target field is still supported for backward compatibility
|
||||||
|
|
@ -42,14 +47,17 @@ Target = "ntfy://ntfy.sh/legacy-topic?tags=thing" # Will show deprecation warni
|
||||||
[CatchAll]
|
[CatchAll]
|
||||||
# Shoutrrr services to forward unmatched emails to
|
# Shoutrrr services to forward unmatched emails to
|
||||||
Targets = ["ntfy://ntfy.sh/catch-all-topic?tags=unmatched"]
|
Targets = ["ntfy://ntfy.sh/catch-all-topic?tags=unmatched"]
|
||||||
|
# Format applies to the catch-all as well
|
||||||
|
# Format = "markdown"
|
||||||
```
|
```
|
||||||
|
|
||||||
The server refuses to start if the configuration cannot forward anything — no
|
The server refuses to start if the configuration cannot forward anything — no
|
||||||
recipients and no catch-all, a recipient without addresses or usable targets, or
|
recipients and no catch-all, a recipient without addresses or usable targets, or
|
||||||
a mistyped table name such as `[[Recipient]]`, which TOML would otherwise accept
|
a mistyped table name such as `[[Recipient]]`, which TOML would otherwise accept
|
||||||
silently. It also refuses to start without a `Username` and a `Password`: these
|
silently, or a `Format` that names none of the supported conversions. It also
|
||||||
used to fall back to `username`/`password`, which left the authentication gate
|
refuses to start without a `Username` and a `Password`: these used to fall back
|
||||||
open to the most obvious guess there is.
|
to `username`/`password`, which left the authentication gate open to the most
|
||||||
|
obvious guess there is.
|
||||||
|
|
||||||
Clients must authenticate with the configured `Username` and `Password` before
|
Clients must authenticate with the configured `Username` and `Password` before
|
||||||
starting a mail transaction; an unauthenticated client is refused rather than
|
starting a mail transaction; an unauthenticated client is refused rather than
|
||||||
|
|
@ -86,6 +94,67 @@ docker run -v /path/to/config.toml:/config.toml \
|
||||||
git.nakama.town/fmartingr/smtp2shoutrrr:latest
|
git.nakama.town/fmartingr/smtp2shoutrrr:latest
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Converting HTML messages
|
||||||
|
|
||||||
|
Plenty of mail carries nothing but HTML, and plenty of notification services
|
||||||
|
render none of it — a Mattermost direct message shows the markup verbatim.
|
||||||
|
Setting `Format` on a recipient (or on `[CatchAll]`) rewrites an HTML body
|
||||||
|
before it is forwarded:
|
||||||
|
|
||||||
|
| `Format` | Effect |
|
||||||
|
| ---------- | --------------------------------------------------- |
|
||||||
|
| `raw` | Forward the body as the message wrote it. (default) |
|
||||||
|
| `markdown` | Render an HTML body as Markdown. |
|
||||||
|
|
||||||
|
There is deliberately no plain-text format beside it. Markdown reads as plain
|
||||||
|
text wherever nothing renders it — that is rather the point of Markdown — so a
|
||||||
|
second conversion would only be a worse copy of this one.
|
||||||
|
|
||||||
|
The value is not case sensitive, and an unrecognised one is refused when the
|
||||||
|
server starts rather than silently forwarding raw HTML.
|
||||||
|
|
||||||
|
Only an HTML body is ever rewritten. A message that arrives as plain text is
|
||||||
|
what the sender chose to write and is forwarded untouched whatever `Format`
|
||||||
|
says. Which part of a multipart message that is follows RFC 2046: the parts of
|
||||||
|
a `multipart/alternative` are the same content in several forms, so the
|
||||||
|
`text/plain` one is preferred where it is not empty; the parts of any other
|
||||||
|
multipart are cumulative, so the first one carrying a body is the message and
|
||||||
|
the footers, signatures and attachments after it are not.
|
||||||
|
|
||||||
|
The Markdown itself is produced by
|
||||||
|
[html-to-markdown](https://github.com/JohannesKaufmann/html-to-markdown), which
|
||||||
|
knows CommonMark — escaping, delimiter runs, fencing a code block past the
|
||||||
|
backticks inside it. What it does not know is mail, because it is written for
|
||||||
|
documents. Before the conversion runs, the parsed message is stripped down to
|
||||||
|
what a notification should carry:
|
||||||
|
|
||||||
|
- Hidden elements go. A template opens with a preheader written for the inbox
|
||||||
|
list, which reads as noise anywhere else.
|
||||||
|
- Images without alt text go, which removes tracking pixels, spacers and
|
||||||
|
sliced-up banners. An inline `cid:` attachment leaves its alt text behind as
|
||||||
|
ordinary words.
|
||||||
|
- Destinations a reader cannot open — `#anchors`, `javascript:`, `cid:` — are
|
||||||
|
dropped and the text of the link is kept. Tabs and line breaks are removed
|
||||||
|
from the rest, since a line break inside an `href` is invisible in the
|
||||||
|
document and a fabricated line in the message.
|
||||||
|
- Table rows become lines and their cells stay apart, since mail lays itself
|
||||||
|
out in tables far more often than it tabulates anything.
|
||||||
|
- Quoting and list nesting is flattened past six levels. A line prefix is
|
||||||
|
re-emitted on every line of every level it nests, so depth multiplies against
|
||||||
|
line count: a message at the 1 MB limit nested 250 quotes deep otherwise
|
||||||
|
renders to 131 MB and takes minutes of CPU.
|
||||||
|
|
||||||
|
If a body is too deeply nested for the HTML parser, or renders to nothing at
|
||||||
|
all because it was images and tracking pixels, it is forwarded unchanged and a
|
||||||
|
warning is logged: reformatting is a courtesy to the target, not a condition of
|
||||||
|
delivery. Output past 64 KiB is cut short with a `…`, which no chat target
|
||||||
|
would have displayed anyway.
|
||||||
|
|
||||||
|
Bodies are decoded before they are rendered, so `quoted-printable` and `base64`
|
||||||
|
transfer encodings and non-UTF-8 character sets (`iso-8859-1`, `windows-1252`,
|
||||||
|
…) reach the target as readable text in both formats, `raw` included — which is
|
||||||
|
the one way `raw` is not quite the bytes that arrived.
|
||||||
|
|
||||||
## Development
|
## Development
|
||||||
|
|
||||||
Run the server with:
|
Run the server with:
|
||||||
|
|
|
||||||
14
backend.go
14
backend.go
|
|
@ -56,7 +56,7 @@ type Backend struct {
|
||||||
config *Config
|
config *Config
|
||||||
}
|
}
|
||||||
|
|
||||||
func (bkd *Backend) sendNotifications(recipient ConfigRecipient, email ReceivedEmail) error {
|
func (bkd *Backend) sendNotifications(recipient ConfigRecipient, email *ReceivedEmail) error {
|
||||||
// Get all target URLs (handles merging and caching)
|
// Get all target URLs (handles merging and caching)
|
||||||
targetURLs := recipient.GetTargetURLs()
|
targetURLs := recipient.GetTargetURLs()
|
||||||
|
|
||||||
|
|
@ -66,7 +66,7 @@ func (bkd *Backend) sendNotifications(recipient ConfigRecipient, email ReceivedE
|
||||||
}
|
}
|
||||||
|
|
||||||
// Prepare email body once (reuse for all targets)
|
// Prepare email body once (reuse for all targets)
|
||||||
body, err := email.Body()
|
body, err := email.FormattedBody(recipient.Format)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
slog.Error("Error getting email body", slog.String("err", err.Error()))
|
slog.Error("Error getting email body", slog.String("err", err.Error()))
|
||||||
return fmt.Errorf("failed to get email body: %w", err)
|
return fmt.Errorf("failed to get email body: %w", err)
|
||||||
|
|
@ -74,6 +74,7 @@ func (bkd *Backend) sendNotifications(recipient ConfigRecipient, email ReceivedE
|
||||||
|
|
||||||
slog.Debug("Prepared email body",
|
slog.Debug("Prepared email body",
|
||||||
slog.Int("body_length", len(body)),
|
slog.Int("body_length", len(body)),
|
||||||
|
slog.String("format", string(recipient.Format)),
|
||||||
slog.String("subject", email.Msg.Header.Get("Subject")),
|
slog.String("subject", email.Msg.Header.Get("Subject")),
|
||||||
slog.String("content_type", email.Msg.Header.Get("Content-Type")))
|
slog.String("content_type", email.Msg.Header.Get("Content-Type")))
|
||||||
|
|
||||||
|
|
@ -134,7 +135,7 @@ func (bkd *Backend) NewSession(c *smtp.Conn) (smtp.Session, error) {
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (bkd *Backend) forwardEmail(email ReceivedEmail) error {
|
func (bkd *Backend) forwardEmail(email *ReceivedEmail) error {
|
||||||
slog.Info("forwading message", slog.String("to", strings.Join(email.Recipients, ",")))
|
slog.Info("forwading message", slog.String("to", strings.Join(email.Recipients, ",")))
|
||||||
|
|
||||||
// Try to match configured recipients first
|
// Try to match configured recipients first
|
||||||
|
|
@ -181,7 +182,10 @@ type Session struct {
|
||||||
|
|
||||||
config *Config
|
config *Config
|
||||||
|
|
||||||
forwarderFunc func(ReceivedEmail) error
|
// forwarderFunc takes the message by pointer: the body is read from a
|
||||||
|
// single-use stream and cached on it, so a copy would forward an empty
|
||||||
|
// notification to every recipient after the first.
|
||||||
|
forwarderFunc func(*ReceivedEmail) error
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *Session) isAuthenticated() bool {
|
func (s *Session) isAuthenticated() bool {
|
||||||
|
|
@ -292,7 +296,7 @@ func (s *Session) Data(r io.Reader) error {
|
||||||
|
|
||||||
slog.Info("Received email", slog.String("destination", strings.Join(recipients, ",")))
|
slog.Info("Received email", slog.String("destination", strings.Join(recipients, ",")))
|
||||||
|
|
||||||
if err := s.forwarderFunc(ReceivedEmail{
|
if err := s.forwarderFunc(&ReceivedEmail{
|
||||||
Recipients: recipients,
|
Recipients: recipients,
|
||||||
Msg: msg,
|
Msg: msg,
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
|
|
|
||||||
102
backend_test.go
102
backend_test.go
|
|
@ -699,3 +699,105 @@ func TestRecipientWithoutTargetsIsRejectedPermanently(t *testing.T) {
|
||||||
require.Error(t, err)
|
require.Error(t, err)
|
||||||
require.Contains(t, err.Error(), "550", "a retry cannot add targets to a configuration")
|
require.Contains(t, err.Error(), "550", "a retry cannot add targets to a configuration")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The case the option exists for: a target that renders no HTML, receiving
|
||||||
|
// mail that carries nothing else.
|
||||||
|
func TestHTMLBodyIsConvertedForTheRecipientsFormat(t *testing.T) {
|
||||||
|
notifications := make(chan string, 4)
|
||||||
|
mockServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
body, err := io.ReadAll(r.Body)
|
||||||
|
if err != nil {
|
||||||
|
w.WriteHeader(http.StatusInternalServerError)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
notifications <- string(body)
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
defer mockServer.Close()
|
||||||
|
|
||||||
|
config := &Config{
|
||||||
|
Port: 0,
|
||||||
|
Username: "testuser",
|
||||||
|
Password: "testpass",
|
||||||
|
Recipients: []ConfigRecipient{
|
||||||
|
{
|
||||||
|
Addresses: []string{"markdown@example.com"},
|
||||||
|
Targets: []string{"generic+" + mockServer.URL + "/?template=json"},
|
||||||
|
Format: FormatMarkdown,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Addresses: []string{"raw@example.com"},
|
||||||
|
Targets: []string{"generic+" + mockServer.URL + "/?template=json"},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
addr, _, _ := startServer(t, NewSMTPServer(config))
|
||||||
|
auth := smtp.PlainAuth("", config.Username, config.Password, "localhost")
|
||||||
|
|
||||||
|
message := []byte(strings.Join([]string{
|
||||||
|
"Subject: Deploy failed",
|
||||||
|
`Content-Type: multipart/mixed; boundary="b"`,
|
||||||
|
"",
|
||||||
|
"--b",
|
||||||
|
"Content-Type: text/html; charset=utf-8",
|
||||||
|
"Content-Transfer-Encoding: quoted-printable",
|
||||||
|
"",
|
||||||
|
"<p>The <b>staging</b> deploy =E2=80=94 <a href=3D\"https://ci.example.com/42\">run 4=",
|
||||||
|
"2</a> =E2=80=94 failed.</p>",
|
||||||
|
"--b--",
|
||||||
|
"",
|
||||||
|
}, "\r\n"))
|
||||||
|
|
||||||
|
for _, recipient := range []string{"markdown@example.com", "raw@example.com"} {
|
||||||
|
require.NoError(t, smtp.SendMail(addr, auth, "sender@example.com", []string{recipient}, message))
|
||||||
|
}
|
||||||
|
|
||||||
|
converted := <-notifications
|
||||||
|
require.Contains(t, converted,
|
||||||
|
`The **staging** deploy — [run 42](https://ci.example.com/42) — failed.`)
|
||||||
|
require.NotContains(t, converted, "<b>")
|
||||||
|
require.NotContains(t, converted, "=E2=80=94", "the transfer encoding is undone before the markup is")
|
||||||
|
|
||||||
|
unconverted := <-notifications
|
||||||
|
require.Contains(t, unconverted, `<b>staging</b>`,
|
||||||
|
"a recipient that asked for nothing still receives the message as it arrived")
|
||||||
|
}
|
||||||
|
|
||||||
|
// The message body is read from a single-use stream and cached on the value
|
||||||
|
// that holds it, so passing it by value hands every recipient after the first
|
||||||
|
// an empty notification with a 250.
|
||||||
|
func TestMessageBodySurvivesASecondRecipient(t *testing.T) {
|
||||||
|
notifications := make(chan string, 4)
|
||||||
|
mockServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
body, err := io.ReadAll(r.Body)
|
||||||
|
if err != nil {
|
||||||
|
w.WriteHeader(http.StatusInternalServerError)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
notifications <- string(body)
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
defer mockServer.Close()
|
||||||
|
|
||||||
|
recipient := ConfigRecipient{
|
||||||
|
Addresses: []string{"test@example.com"},
|
||||||
|
Targets: []string{"generic+" + mockServer.URL + "/?template=json"},
|
||||||
|
}
|
||||||
|
|
||||||
|
backend := &Backend{config: &Config{Username: "u", Password: "p"}}
|
||||||
|
email := &ReceivedEmail{
|
||||||
|
Recipients: []string{"test@example.com"},
|
||||||
|
Msg: readMessage(t, "Subject: Test\r\n\r\nthe body"),
|
||||||
|
}
|
||||||
|
|
||||||
|
require.NoError(t, backend.sendNotifications(recipient, email))
|
||||||
|
require.NoError(t, backend.sendNotifications(recipient, email))
|
||||||
|
|
||||||
|
require.Contains(t, <-notifications, "the body")
|
||||||
|
require.Contains(t, <-notifications, "the body")
|
||||||
|
}
|
||||||
|
|
|
||||||
35
config.go
35
config.go
|
|
@ -67,6 +67,14 @@ func (c *Config) SetDefaults() {
|
||||||
if c.Port == 0 {
|
if c.Port == 0 {
|
||||||
c.Port = 11125
|
c.Port = 11125
|
||||||
}
|
}
|
||||||
|
|
||||||
|
for i := range c.Recipients {
|
||||||
|
c.Recipients[i].Format = c.Recipients[i].Format.normalize()
|
||||||
|
}
|
||||||
|
|
||||||
|
if c.CatchAll != nil {
|
||||||
|
c.CatchAll.Format = c.CatchAll.Format.normalize()
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Validate rejects configuration that would leave the server relaying for
|
// Validate rejects configuration that would leave the server relaying for
|
||||||
|
|
@ -95,10 +103,20 @@ func (c *Config) Validate() error {
|
||||||
return fmt.Errorf("recipient %d (%s) has no usable Targets",
|
return fmt.Errorf("recipient %d (%s) has no usable Targets",
|
||||||
i, strings.Join(r.Addresses, ","))
|
i, strings.Join(r.Addresses, ","))
|
||||||
}
|
}
|
||||||
|
if !r.Format.valid() {
|
||||||
|
return fmt.Errorf("recipient %d (%s) has an unknown Format %q, expected one of: %s",
|
||||||
|
i, strings.Join(r.Addresses, ","), r.Format, formatNames())
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if c.CatchAll != nil && len(c.CatchAll.GetTargetURLs()) == 0 {
|
if c.CatchAll != nil {
|
||||||
return errors.New("CatchAll has no usable Targets")
|
if len(c.CatchAll.GetTargetURLs()) == 0 {
|
||||||
|
return errors.New("CatchAll has no usable Targets")
|
||||||
|
}
|
||||||
|
if !c.CatchAll.Format.valid() {
|
||||||
|
return fmt.Errorf("CatchAll has an unknown Format %q, expected one of: %s",
|
||||||
|
c.CatchAll.Format, formatNames())
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if len(c.Recipients) == 0 && c.CatchAll == nil {
|
if len(c.Recipients) == 0 && c.CatchAll == nil {
|
||||||
|
|
@ -109,9 +127,16 @@ func (c *Config) Validate() error {
|
||||||
}
|
}
|
||||||
|
|
||||||
type ConfigRecipient struct {
|
type ConfigRecipient struct {
|
||||||
Addresses []string // email addresses
|
Addresses []string // email addresses
|
||||||
Target string // deprecated: use Targets instead
|
Target string // deprecated: use Targets instead
|
||||||
Targets []string // shoutrrr addresses (supports multiple)
|
Targets []string // shoutrrr addresses (supports multiple)
|
||||||
|
|
||||||
|
// Format is the representation these targets want an HTML message body
|
||||||
|
// in: raw (the default, forwarding the message unchanged), text or
|
||||||
|
// markdown. Targets that render neither HTML nor Markdown — a Mattermost
|
||||||
|
// direct message, for one — otherwise receive the markup verbatim.
|
||||||
|
Format BodyFormat
|
||||||
|
|
||||||
targetURLs []*url.URL // cached parsed URLs
|
targetURLs []*url.URL // cached parsed URLs
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -191,3 +191,61 @@ Targets = ["ntfy://ntfy.sh/catch-all"]
|
||||||
require.Contains(t, logged.String(), "ignoring unknown key in config file")
|
require.Contains(t, logged.String(), "ignoring unknown key in config file")
|
||||||
require.Contains(t, logged.String(), "Recipient")
|
require.Contains(t, logged.String(), "Recipient")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestLoadConfigReadsFormat(t *testing.T) {
|
||||||
|
config, err := LoadConfig(writeConfig(t, credentials+`
|
||||||
|
[[Recipients]]
|
||||||
|
Addresses = ["user@example.com"]
|
||||||
|
Targets = ["ntfy://ntfy.sh/topic"]
|
||||||
|
Format = "Markdown"
|
||||||
|
|
||||||
|
[[Recipients]]
|
||||||
|
Addresses = ["asis@example.com"]
|
||||||
|
Targets = ["ntfy://ntfy.sh/asis"]
|
||||||
|
|
||||||
|
[CatchAll]
|
||||||
|
Targets = ["ntfy://ntfy.sh/catch-all"]
|
||||||
|
Format = "markdown"
|
||||||
|
`))
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
require.Equal(t, FormatMarkdown, config.Recipients[0].Format, "the option is not case sensitive")
|
||||||
|
require.Equal(t, FormatRaw, config.Recipients[1].Format, "an unset Format forwards the message unchanged")
|
||||||
|
require.Equal(t, FormatMarkdown, config.CatchAll.Format)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A typo here is silent otherwise: the server starts and forwards raw HTML to
|
||||||
|
// a target that cannot render it.
|
||||||
|
func TestLoadConfigRejectsUnknownFormat(t *testing.T) {
|
||||||
|
for name, contents := range map[string]string{
|
||||||
|
"on a recipient": `
|
||||||
|
[[Recipients]]
|
||||||
|
Addresses = ["user@example.com"]
|
||||||
|
Targets = ["ntfy://ntfy.sh/topic"]
|
||||||
|
Format = "text"
|
||||||
|
`,
|
||||||
|
"on the catch-all": `
|
||||||
|
[CatchAll]
|
||||||
|
Targets = ["ntfy://ntfy.sh/catch-all"]
|
||||||
|
Format = "md"
|
||||||
|
`,
|
||||||
|
} {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
_, err := LoadConfig(writeConfig(t, credentials+contents))
|
||||||
|
require.Error(t, err)
|
||||||
|
require.Contains(t, err.Error(), "raw, markdown")
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate used to depend on SetDefaults having run, so a Config assembled in
|
||||||
|
// Go failed on a Format nobody had set.
|
||||||
|
func TestValidateAcceptsAConfigBuiltWithoutDefaults(t *testing.T) {
|
||||||
|
config := &Config{
|
||||||
|
Username: "user",
|
||||||
|
Password: "secret",
|
||||||
|
Recipients: []ConfigRecipient{{Addresses: []string{"user@example.com"}, Targets: []string{"ntfy://ntfy.sh/topic"}}},
|
||||||
|
}
|
||||||
|
|
||||||
|
require.NoError(t, config.Validate())
|
||||||
|
}
|
||||||
|
|
|
||||||
442
email.go
442
email.go
|
|
@ -2,14 +2,20 @@ package smtp2shoutrrr
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
|
"encoding/base64"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"mime"
|
"mime"
|
||||||
"mime/multipart"
|
"mime/multipart"
|
||||||
|
"mime/quotedprintable"
|
||||||
"net/mail"
|
"net/mail"
|
||||||
|
"net/textproto"
|
||||||
"strings"
|
"strings"
|
||||||
|
"unicode"
|
||||||
|
|
||||||
|
"golang.org/x/net/html/charset"
|
||||||
)
|
)
|
||||||
|
|
||||||
// errMalformedMessage marks a message this server can never turn into a
|
// errMalformedMessage marks a message this server can never turn into a
|
||||||
|
|
@ -17,81 +23,383 @@ import (
|
||||||
// sender is told to give up rather than retry forever.
|
// sender is told to give up rather than retry forever.
|
||||||
var errMalformedMessage = errors.New("malformed message")
|
var errMalformedMessage = errors.New("malformed message")
|
||||||
|
|
||||||
|
// maxMultipartDepth bounds how deep a message can make this server walk. Real
|
||||||
|
// mail nests three levels at most (mixed → related → alternative); anything
|
||||||
|
// past that is a generated or hostile message rather than one with a body
|
||||||
|
// worth finding.
|
||||||
|
const maxMultipartDepth = 8
|
||||||
|
|
||||||
type ReceivedEmail struct {
|
type ReceivedEmail struct {
|
||||||
Recipients []string
|
Recipients []string
|
||||||
Msg *mail.Message
|
Msg *mail.Message
|
||||||
|
|
||||||
|
bodyRead bool
|
||||||
body string
|
body string
|
||||||
|
bodyIsHTML bool
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Body returns the message body as the message carried it: its text/plain part
|
||||||
|
// when there is one, its text/html part otherwise.
|
||||||
func (re *ReceivedEmail) Body() (string, error) {
|
func (re *ReceivedEmail) Body() (string, error) {
|
||||||
if re.body == "" {
|
if err := re.readBody(); err != nil {
|
||||||
contentType := re.Msg.Header.Get("Content-Type")
|
return "", err
|
||||||
|
|
||||||
if contentType == "" {
|
|
||||||
body, err := io.ReadAll(re.Msg.Body)
|
|
||||||
if err != nil {
|
|
||||||
return "", fmt.Errorf("failed to read email body: %w", err)
|
|
||||||
}
|
|
||||||
re.body = string(body)
|
|
||||||
} else {
|
|
||||||
mediaType, params, err := mime.ParseMediaType(contentType)
|
|
||||||
if err != nil {
|
|
||||||
return "", fmt.Errorf("%w: parsing Content-Type %q: %w", errMalformedMessage, contentType, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if strings.HasPrefix(mediaType, "multipart/") {
|
|
||||||
// Handle any multipart type (alternative, mixed, related, etc.)
|
|
||||||
mr := multipart.NewReader(re.Msg.Body, params["boundary"])
|
|
||||||
var htmlBody string
|
|
||||||
|
|
||||||
for {
|
|
||||||
part, err := mr.NextPart()
|
|
||||||
if err != nil {
|
|
||||||
break
|
|
||||||
}
|
|
||||||
|
|
||||||
partContentType := part.Header.Get("Content-Type")
|
|
||||||
slog.Debug("Processing email part", slog.String("content_type", partContentType))
|
|
||||||
|
|
||||||
// Prefer text/plain, but keep HTML as fallback
|
|
||||||
if strings.HasPrefix(partContentType, "text/plain") {
|
|
||||||
body := new(bytes.Buffer)
|
|
||||||
_, err = body.ReadFrom(part)
|
|
||||||
_ = part.Close()
|
|
||||||
if err != nil {
|
|
||||||
slog.Error("Failed to read part body", slog.String("err", err.Error()))
|
|
||||||
return "", fmt.Errorf("failed to read part body: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
re.body = body.String()
|
|
||||||
break // Found text/plain, use it
|
|
||||||
} else if strings.HasPrefix(partContentType, "text/html") && htmlBody == "" {
|
|
||||||
// Store HTML as fallback if no text/plain found
|
|
||||||
body := new(bytes.Buffer)
|
|
||||||
_, err = body.ReadFrom(part)
|
|
||||||
_ = part.Close()
|
|
||||||
if err == nil {
|
|
||||||
htmlBody = body.String()
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
_ = part.Close()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// If no text/plain was found, use HTML
|
|
||||||
if re.body == "" && htmlBody != "" {
|
|
||||||
re.body = htmlBody
|
|
||||||
}
|
|
||||||
} else if strings.HasPrefix(mediaType, "text/") {
|
|
||||||
// Handle non-multipart text content (text/plain, text/html, etc.)
|
|
||||||
body, err := io.ReadAll(re.Msg.Body)
|
|
||||||
if err != nil {
|
|
||||||
return "", fmt.Errorf("failed to read email body: %w", err)
|
|
||||||
}
|
|
||||||
re.body = string(body)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
return re.body, nil
|
return re.body, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// FormattedBody returns the body in the requested format. Only an HTML body is
|
||||||
|
// ever rewritten: a message that already arrived as plain text is what the
|
||||||
|
// sender chose to write, and no target is better served by a round trip
|
||||||
|
// through a renderer.
|
||||||
|
func (re *ReceivedEmail) FormattedBody(format BodyFormat) (string, error) {
|
||||||
|
if err := re.readBody(); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
|
||||||
|
if format.normalize() == FormatRaw || !re.bodyIsHTML {
|
||||||
|
return re.body, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Reformatting is a courtesy to the target, not a condition of delivery.
|
||||||
|
// A body the renderer cannot take apart, or one that renders to nothing
|
||||||
|
// because it was all images and tracking pixels, is forwarded as it
|
||||||
|
// arrived: an empty notification is refused by most targets, which puts
|
||||||
|
// the message into a retry loop it can never leave.
|
||||||
|
rendered, err := renderMarkdown(re.body)
|
||||||
|
if err != nil {
|
||||||
|
slog.Warn("failed to render HTML body, forwarding it unchanged",
|
||||||
|
slog.String("format", string(format)),
|
||||||
|
slog.String("err", err.Error()))
|
||||||
|
|
||||||
|
return re.body, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
if rendered == "" && strings.TrimSpace(re.body) != "" {
|
||||||
|
slog.Warn("HTML body rendered to nothing, forwarding it unchanged",
|
||||||
|
slog.String("format", string(format)),
|
||||||
|
slog.Int("html_length", len(re.body)))
|
||||||
|
|
||||||
|
return re.body, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return rendered, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (re *ReceivedEmail) readBody() error {
|
||||||
|
if re.bodyRead {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
body, isHTML, err := readEntity(textproto.MIMEHeader(re.Msg.Header), re.Msg.Body, 0)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
re.body, re.bodyIsHTML, re.bodyRead = body, isHTML, true
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// readEntity reads one MIME entity — the message itself or a part of it — and
|
||||||
|
// reports the body it contributes and whether that body is HTML.
|
||||||
|
func readEntity(header textproto.MIMEHeader, body io.Reader, depth int) (string, bool, error) {
|
||||||
|
contentType := header.Get("Content-Type")
|
||||||
|
if contentType == "" {
|
||||||
|
content, err := decodeContent(header, body, "")
|
||||||
|
|
||||||
|
return content, false, err
|
||||||
|
}
|
||||||
|
|
||||||
|
mediaType, params, err := mime.ParseMediaType(contentType)
|
||||||
|
if err != nil {
|
||||||
|
return "", false, fmt.Errorf("%w: parsing Content-Type %q: %w", errMalformedMessage, contentType, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return readMediaType(header, body, mediaType, params, depth)
|
||||||
|
}
|
||||||
|
|
||||||
|
func readMediaType(
|
||||||
|
header textproto.MIMEHeader,
|
||||||
|
body io.Reader,
|
||||||
|
mediaType string,
|
||||||
|
params map[string]string,
|
||||||
|
depth int,
|
||||||
|
) (string, bool, error) {
|
||||||
|
switch {
|
||||||
|
case strings.HasPrefix(mediaType, "multipart/"):
|
||||||
|
return readMultipart(body, mediaType, params["boundary"], depth)
|
||||||
|
|
||||||
|
case strings.HasPrefix(mediaType, "text/"):
|
||||||
|
content, err := decodeContent(header, body, params["charset"])
|
||||||
|
|
||||||
|
return content, mediaType == "text/html", err
|
||||||
|
|
||||||
|
default:
|
||||||
|
// An image or an application/* payload on its own carries no text to
|
||||||
|
// notify with; the subject still reaches the target as the title.
|
||||||
|
return "", false, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func readMultipart(body io.Reader, mediaType, boundary string, depth int) (string, bool, error) {
|
||||||
|
if boundary == "" {
|
||||||
|
return "", false, fmt.Errorf("%w: multipart message without a boundary", errMalformedMessage)
|
||||||
|
}
|
||||||
|
|
||||||
|
if depth >= maxMultipartDepth {
|
||||||
|
slog.Warn("stopped descending into a deeply nested message", slog.Int("depth", depth))
|
||||||
|
|
||||||
|
return "", false, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// RFC 2046 §5.1.4: the parts of a multipart/alternative are one content in
|
||||||
|
// several forms, so exactly one of them is chosen. The parts of any other
|
||||||
|
// multipart are cumulative (§5.1.3): the first one carrying a body is the
|
||||||
|
// message, and what follows it is footers, signatures and enclosures.
|
||||||
|
alternatives := mediaType == "multipart/alternative"
|
||||||
|
|
||||||
|
var chosen bodySelector
|
||||||
|
|
||||||
|
reader := multipart.NewReader(body, boundary)
|
||||||
|
|
||||||
|
for {
|
||||||
|
part, err := reader.NextPart()
|
||||||
|
if errors.Is(err, io.EOF) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
// A truncated container still hands over the parts that were
|
||||||
|
// readable. Only a message that yielded nothing at all is worth
|
||||||
|
// refusing, since there is nothing left to notify with.
|
||||||
|
slog.Warn("stopped reading a multipart message early", slog.String("err", err.Error()))
|
||||||
|
|
||||||
|
if chosen.empty() {
|
||||||
|
return "", false, fmt.Errorf("%w: reading multipart body: %w", errMalformedMessage, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
content, isHTML, err := readPart(part, depth)
|
||||||
|
_ = part.Close()
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
// One unreadable part is not the whole message; the rest of the
|
||||||
|
// tree may still hold a body.
|
||||||
|
slog.Warn("ignoring an unreadable email part", slog.String("err", err.Error()))
|
||||||
|
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
if !alternatives {
|
||||||
|
if strings.TrimSpace(content) != "" {
|
||||||
|
return content, isHTML, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
chosen.offer(content, isHTML)
|
||||||
|
|
||||||
|
if chosen.done() {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
content, isHTML := chosen.result()
|
||||||
|
|
||||||
|
return content, isHTML, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func readPart(part *multipart.Part, depth int) (string, bool, error) {
|
||||||
|
contentType := part.Header.Get("Content-Type")
|
||||||
|
slog.Debug("Processing email part", slog.String("content_type", contentType))
|
||||||
|
|
||||||
|
// mime.ParseMediaType hands back a usable media type alongside the error
|
||||||
|
// it reports for a malformed parameter — an empty charset, an unquoted
|
||||||
|
// file name — which older mailers emit often enough that dropping those
|
||||||
|
// parts would lose the message body itself.
|
||||||
|
if disposition, _, err := mime.ParseMediaType(part.Header.Get("Content-Disposition")); err == nil ||
|
||||||
|
errors.Is(err, mime.ErrInvalidMediaParameter) {
|
||||||
|
if disposition == "attachment" {
|
||||||
|
// An attached .txt or .html file is something the sender enclosed,
|
||||||
|
// not what they wrote.
|
||||||
|
return "", false, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// RFC 2045 §5.2 makes a part without a Content-Type plain US-ASCII text.
|
||||||
|
mediaType, params := "text/plain", map[string]string{}
|
||||||
|
|
||||||
|
if contentType != "" {
|
||||||
|
var err error
|
||||||
|
|
||||||
|
mediaType, params, err = mime.ParseMediaType(contentType)
|
||||||
|
if err != nil && !errors.Is(err, mime.ErrInvalidMediaParameter) {
|
||||||
|
slog.Warn("ignoring email part with an unparseable Content-Type",
|
||||||
|
slog.String("content_type", contentType),
|
||||||
|
slog.String("err", err.Error()))
|
||||||
|
|
||||||
|
return "", false, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
if params == nil {
|
||||||
|
params = map[string]string{}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if !strings.HasPrefix(mediaType, "multipart/") &&
|
||||||
|
mediaType != "text/plain" && mediaType != "text/html" {
|
||||||
|
// text/calendar, an inline image, a signature: nothing to notify with.
|
||||||
|
return "", false, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return readMediaType(part.Header, part, mediaType, params, depth+1)
|
||||||
|
}
|
||||||
|
|
||||||
|
// decodeContent turns one entity's bytes into a UTF-8 string, undoing the
|
||||||
|
// transfer encoding and the character set the message declared. Without this a
|
||||||
|
// quoted-printable body reaches the target as "=E2=80=99" and a Latin-1 one as
|
||||||
|
// mojibake — and neither survives a trip through an HTML renderer.
|
||||||
|
func decodeContent(header textproto.MIMEHeader, body io.Reader, charsetLabel string) (string, error) {
|
||||||
|
raw, err := io.ReadAll(body)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("reading message body: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return decodeCharset(decodeTransferEncoding(raw, header.Get("Content-Transfer-Encoding")), charsetLabel), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// decodeTransferEncoding is deliberately forgiving. A body that will not decode
|
||||||
|
// is still a body: forwarding the bytes as they arrived shows the reader
|
||||||
|
// something, where refusing the message tells the sender to stop retrying and
|
||||||
|
// loses the notification for good.
|
||||||
|
func decodeTransferEncoding(raw []byte, encoding string) []byte {
|
||||||
|
switch strings.ToLower(strings.TrimSpace(encoding)) {
|
||||||
|
case "base64":
|
||||||
|
return decodeBase64(raw)
|
||||||
|
|
||||||
|
case "quoted-printable":
|
||||||
|
// mime/multipart already unwraps this for the parts it hands out and
|
||||||
|
// drops the header with it, so only a single-part message arrives
|
||||||
|
// here still encoded.
|
||||||
|
decoded, err := io.ReadAll(quotedprintable.NewReader(bytes.NewReader(raw)))
|
||||||
|
if err != nil {
|
||||||
|
slog.Warn("failed to decode a quoted-printable body, forwarding what could be decoded",
|
||||||
|
slog.String("err", err.Error()))
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(decoded) == 0 {
|
||||||
|
return raw
|
||||||
|
}
|
||||||
|
|
||||||
|
return decoded
|
||||||
|
|
||||||
|
default:
|
||||||
|
// 7bit, 8bit and binary are the identity encoding; anything else is
|
||||||
|
// something this server has no way to undo.
|
||||||
|
return raw
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// decodeBase64 tolerates the two liberties real mailers take with a base64
|
||||||
|
// body: they wrap it across lines, and they leave the padding off the last
|
||||||
|
// group.
|
||||||
|
func decodeBase64(raw []byte) []byte {
|
||||||
|
compact := bytes.Map(func(r rune) rune {
|
||||||
|
if unicode.IsSpace(r) {
|
||||||
|
return -1
|
||||||
|
}
|
||||||
|
|
||||||
|
return r
|
||||||
|
}, raw)
|
||||||
|
|
||||||
|
encoding := base64.StdEncoding
|
||||||
|
if len(compact)%4 != 0 {
|
||||||
|
encoding = base64.RawStdEncoding
|
||||||
|
}
|
||||||
|
|
||||||
|
decoded, err := encoding.DecodeString(string(compact))
|
||||||
|
if err != nil {
|
||||||
|
slog.Warn("failed to decode a base64 body, forwarding what could be decoded",
|
||||||
|
slog.String("err", err.Error()))
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(decoded) == 0 {
|
||||||
|
return raw
|
||||||
|
}
|
||||||
|
|
||||||
|
return decoded
|
||||||
|
}
|
||||||
|
|
||||||
|
func decodeCharset(raw []byte, label string) string {
|
||||||
|
switch strings.ToLower(strings.TrimSpace(label)) {
|
||||||
|
case "", "utf-8", "utf8", "us-ascii", "ascii":
|
||||||
|
return string(raw)
|
||||||
|
}
|
||||||
|
|
||||||
|
encoding, _ := charset.Lookup(label)
|
||||||
|
if encoding == nil {
|
||||||
|
slog.Warn("unknown message charset, forwarding the body undecoded",
|
||||||
|
slog.String("charset", label))
|
||||||
|
|
||||||
|
return string(raw)
|
||||||
|
}
|
||||||
|
|
||||||
|
decoded, err := encoding.NewDecoder().Bytes(raw)
|
||||||
|
if err != nil {
|
||||||
|
slog.Warn("failed to decode message charset, forwarding the body undecoded",
|
||||||
|
slog.String("charset", label),
|
||||||
|
slog.String("err", err.Error()))
|
||||||
|
|
||||||
|
return string(raw)
|
||||||
|
}
|
||||||
|
|
||||||
|
return string(decoded)
|
||||||
|
}
|
||||||
|
|
||||||
|
// bodySelector picks between the forms of a multipart/alternative. Plain text
|
||||||
|
// beats HTML, and the first of each kind beats the rest, which is the order
|
||||||
|
// the message puts its own preference in. A blank part is held rather than
|
||||||
|
// chosen: senders that build the HTML from a template routinely emit an empty
|
||||||
|
// alternative beside it.
|
||||||
|
type bodySelector struct {
|
||||||
|
plain string
|
||||||
|
html string
|
||||||
|
hasPlain bool
|
||||||
|
hasHTML bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *bodySelector) offer(content string, isHTML bool) {
|
||||||
|
if isHTML {
|
||||||
|
if !s.hasHTML || strings.TrimSpace(s.html) == "" {
|
||||||
|
s.html, s.hasHTML = content, true
|
||||||
|
}
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if !s.hasPlain || strings.TrimSpace(s.plain) == "" {
|
||||||
|
s.plain, s.hasPlain = content, true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *bodySelector) empty() bool {
|
||||||
|
return !s.hasPlain && !s.hasHTML
|
||||||
|
}
|
||||||
|
|
||||||
|
// done reports that nothing later in the container can improve on what has
|
||||||
|
// been found.
|
||||||
|
func (s *bodySelector) done() bool {
|
||||||
|
return s.hasPlain && strings.TrimSpace(s.plain) != ""
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *bodySelector) result() (string, bool) {
|
||||||
|
if s.done() || !s.hasHTML {
|
||||||
|
return s.plain, false
|
||||||
|
}
|
||||||
|
|
||||||
|
return s.html, true
|
||||||
|
}
|
||||||
|
|
|
||||||
418
email_test.go
418
email_test.go
|
|
@ -1,9 +1,11 @@
|
||||||
package smtp2shoutrrr
|
package smtp2shoutrrr
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"encoding/base64"
|
||||||
"net/mail"
|
"net/mail"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
)
|
)
|
||||||
|
|
@ -88,3 +90,419 @@ func TestBodyFallsBackToHTMLPart(t *testing.T) {
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
require.Equal(t, "<p>html body</p>", body)
|
require.Equal(t, "<p>html body</p>", body)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestBodyDecodesQuotedPrintable(t *testing.T) {
|
||||||
|
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||||
|
"Subject: Test",
|
||||||
|
"Content-Type: text/plain; charset=utf-8",
|
||||||
|
"Content-Transfer-Encoding: quoted-printable",
|
||||||
|
"",
|
||||||
|
"caf=C3=A9 =E2=80=94 a very long line that the sender wrapped =",
|
||||||
|
"here",
|
||||||
|
"",
|
||||||
|
}, "\r\n"))}
|
||||||
|
|
||||||
|
body, err := email.Body()
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, "café — a very long line that the sender wrapped here\r\n", body)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBodyDecodesBase64(t *testing.T) {
|
||||||
|
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||||
|
"Subject: Test",
|
||||||
|
"Content-Type: text/plain; charset=utf-8",
|
||||||
|
"Content-Transfer-Encoding: base64",
|
||||||
|
"",
|
||||||
|
base64.StdEncoding.EncodeToString([]byte("café — encoded")),
|
||||||
|
"",
|
||||||
|
}, "\r\n"))}
|
||||||
|
|
||||||
|
body, err := email.Body()
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, "café — encoded", body)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Windows-1252 and Latin-1 are still what a good deal of automated mail is
|
||||||
|
// written in, and their bytes are not valid UTF-8.
|
||||||
|
func TestBodyDecodesNonUTF8Charset(t *testing.T) {
|
||||||
|
email := ReceivedEmail{Msg: readMessage(t,
|
||||||
|
"Subject: Test\r\nContent-Type: text/plain; charset=iso-8859-1\r\n\r\ncaf\xe9\r\n")}
|
||||||
|
|
||||||
|
body, err := email.Body()
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, "café\r\n", body)
|
||||||
|
require.True(t, utf8.ValidString(body))
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBodyUnknownCharsetIsForwardedUndecoded(t *testing.T) {
|
||||||
|
email := ReceivedEmail{Msg: readMessage(t,
|
||||||
|
"Subject: Test\r\nContent-Type: text/plain; charset=not-a-charset\r\n\r\nbody\r\n")}
|
||||||
|
|
||||||
|
body, err := email.Body()
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, "body\r\n", body)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A forwarded message is a multipart/mixed wrapping the multipart/alternative
|
||||||
|
// that holds what the sender actually wrote. Stopping at the outer container
|
||||||
|
// left the notification empty.
|
||||||
|
func TestBodyDescendsIntoNestedMultipart(t *testing.T) {
|
||||||
|
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||||
|
"Subject: Test",
|
||||||
|
`Content-Type: multipart/mixed; boundary="outer"`,
|
||||||
|
"",
|
||||||
|
"--outer",
|
||||||
|
`Content-Type: multipart/alternative; boundary="inner"`,
|
||||||
|
"",
|
||||||
|
"--inner",
|
||||||
|
"Content-Type: text/html; charset=utf-8",
|
||||||
|
"",
|
||||||
|
"<p>html body</p>",
|
||||||
|
"--inner",
|
||||||
|
"Content-Type: text/plain; charset=utf-8",
|
||||||
|
"",
|
||||||
|
"plain body",
|
||||||
|
"--inner--",
|
||||||
|
"--outer--",
|
||||||
|
"",
|
||||||
|
}, "\r\n"))}
|
||||||
|
|
||||||
|
body, err := email.Body()
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, "plain body", body)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBodySkipsAttachments(t *testing.T) {
|
||||||
|
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||||
|
"Subject: Test",
|
||||||
|
`Content-Type: multipart/mixed; boundary="b"`,
|
||||||
|
"",
|
||||||
|
"--b",
|
||||||
|
"Content-Type: text/plain; charset=utf-8",
|
||||||
|
`Content-Disposition: attachment; filename="notes.txt"`,
|
||||||
|
"",
|
||||||
|
"an enclosed file, not the message",
|
||||||
|
"--b",
|
||||||
|
"Content-Type: text/plain; charset=utf-8",
|
||||||
|
"",
|
||||||
|
"the message itself",
|
||||||
|
"--b--",
|
||||||
|
"",
|
||||||
|
}, "\r\n"))}
|
||||||
|
|
||||||
|
body, err := email.Body()
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, "the message itself", body)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Senders that build the HTML alternative from a template routinely emit an
|
||||||
|
// empty text/plain beside it.
|
||||||
|
func TestBodyPrefersHTMLOverAnEmptyPlainTextPart(t *testing.T) {
|
||||||
|
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||||
|
"Subject: Test",
|
||||||
|
`Content-Type: multipart/alternative; boundary="b"`,
|
||||||
|
"",
|
||||||
|
"--b",
|
||||||
|
"Content-Type: text/plain; charset=utf-8",
|
||||||
|
"",
|
||||||
|
" ",
|
||||||
|
"--b",
|
||||||
|
"Content-Type: text/html; charset=utf-8",
|
||||||
|
"",
|
||||||
|
"<p>html body</p>",
|
||||||
|
"--b--",
|
||||||
|
"",
|
||||||
|
}, "\r\n"))}
|
||||||
|
|
||||||
|
body, err := email.Body()
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, "<p>html body</p>", body)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBodyRejectsMultipartWithoutBoundary(t *testing.T) {
|
||||||
|
email := ReceivedEmail{Msg: readMessage(t,
|
||||||
|
"Subject: Test\r\nContent-Type: multipart/alternative\r\n\r\nbody\r\n")}
|
||||||
|
|
||||||
|
_, err := email.Body()
|
||||||
|
require.ErrorIs(t, err, errMalformedMessage)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFormattedBodyConvertsHTML(t *testing.T) {
|
||||||
|
raw := strings.Join([]string{
|
||||||
|
"Subject: Test",
|
||||||
|
"Content-Type: text/html; charset=utf-8",
|
||||||
|
"Content-Transfer-Encoding: quoted-printable",
|
||||||
|
"",
|
||||||
|
"<p>Build <b>failed</b>: <a href=3D\"https://ci.example.com/42\">run 42</a></p>",
|
||||||
|
"",
|
||||||
|
}, "\r\n")
|
||||||
|
|
||||||
|
for format, want := range map[BodyFormat]string{
|
||||||
|
FormatMarkdown: "Build **failed**: [run 42](https://ci.example.com/42)",
|
||||||
|
FormatRaw: "<p>Build <b>failed</b>: <a href=\"https://ci.example.com/42\">run 42</a></p>\r\n",
|
||||||
|
} {
|
||||||
|
t.Run(string(format), func(t *testing.T) {
|
||||||
|
email := ReceivedEmail{Msg: readMessage(t, raw)}
|
||||||
|
|
||||||
|
body, err := email.FormattedBody(format)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, want, body)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A recipient asking for Markdown is asking for HTML to stop reaching it, not
|
||||||
|
// for the plain text a sender wrote to be run through a renderer.
|
||||||
|
func TestFormattedBodyLeavesPlainTextAlone(t *testing.T) {
|
||||||
|
raw := "Subject: Test\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n2 * 3 = 6 <see notes>\r\n"
|
||||||
|
|
||||||
|
for _, format := range []BodyFormat{FormatRaw, FormatMarkdown} {
|
||||||
|
t.Run(string(format), func(t *testing.T) {
|
||||||
|
email := ReceivedEmail{Msg: readMessage(t, raw)}
|
||||||
|
|
||||||
|
body, err := email.FormattedBody(format)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, "2 * 3 = 6 <see notes>\r\n", body)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The zero value of the option means the same as raw, so a recipient built in
|
||||||
|
// code rather than loaded from a file still forwards what it received.
|
||||||
|
func TestFormattedBodyTreatsUnsetFormatAsRaw(t *testing.T) {
|
||||||
|
email := ReceivedEmail{Msg: readMessage(t,
|
||||||
|
"Subject: Test\r\nContent-Type: text/html\r\n\r\n<p>html body</p>\r\n")}
|
||||||
|
|
||||||
|
body, err := email.FormattedBody("")
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, "<p>html body</p>\r\n", body)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Reformatting is a courtesy, so a body the renderer cannot use is still
|
||||||
|
// forwarded rather than costing the reader the notification.
|
||||||
|
func TestFormattedBodyFallsBackToTheRawHTML(t *testing.T) {
|
||||||
|
// The parser refuses a document with more than 512 elements open at once.
|
||||||
|
tooDeep := strings.Repeat("<div>", 600) + "deep" + strings.Repeat("</div>", 600)
|
||||||
|
|
||||||
|
for name, source := range map[string]string{
|
||||||
|
"a body the parser refuses": tooDeep,
|
||||||
|
// Otherwise shoutrrr is handed "", which Mattermost and Discord
|
||||||
|
// reject, so every target fails and the sender retries the same
|
||||||
|
// message forever.
|
||||||
|
"a body that renders to nothing": `<img src="https://x.example/pixel.gif">`,
|
||||||
|
} {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
email := ReceivedEmail{Msg: readMessage(t,
|
||||||
|
"Subject: Test\r\nContent-Type: text/html\r\n\r\n"+source)}
|
||||||
|
|
||||||
|
body, err := email.FormattedBody(FormatMarkdown)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, source, body)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBodyIsReadOnlyOnce(t *testing.T) {
|
||||||
|
email := ReceivedEmail{Msg: readMessage(t,
|
||||||
|
"Subject: Test\r\nContent-Type: text/html\r\n\r\n<p>body</p>\r\n")}
|
||||||
|
|
||||||
|
first, err := email.FormattedBody(FormatMarkdown)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
second, err := email.FormattedBody(FormatMarkdown)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
require.Equal(t, "body", first)
|
||||||
|
require.Equal(t, first, second, "the message body is consumed as it is read")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Regression: an unreadable transfer encoding used to be reported as a
|
||||||
|
// permanent 550, which tells the sender to stop retrying and loses a
|
||||||
|
// notification that main delivered. Real mailers emit unpadded base64.
|
||||||
|
func TestBodyToleratesAnUndecodableTransferEncoding(t *testing.T) {
|
||||||
|
for name, tc := range map[string]struct{ raw, want string }{
|
||||||
|
"unpadded base64": {
|
||||||
|
raw: base64.RawStdEncoding.EncodeToString([]byte("hello world")),
|
||||||
|
want: "hello world",
|
||||||
|
},
|
||||||
|
"base64 that is not base64 at all": {
|
||||||
|
raw: "!!!! not base64 !!!!",
|
||||||
|
want: "!!!! not base64 !!!!",
|
||||||
|
},
|
||||||
|
} {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
email := ReceivedEmail{Msg: readMessage(t,
|
||||||
|
"Subject: Test\r\nContent-Type: text/plain\r\n"+
|
||||||
|
"Content-Transfer-Encoding: base64\r\n\r\n"+tc.raw+"\r\n")}
|
||||||
|
|
||||||
|
body, err := email.Body()
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, tc.want, strings.TrimSpace(body))
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Regression: one part that would not decode used to fail the whole message,
|
||||||
|
// even with a perfectly good alternative already in hand.
|
||||||
|
func TestBodyKeepsAGoodPartBesideAnUnreadableOne(t *testing.T) {
|
||||||
|
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||||
|
"Subject: Test",
|
||||||
|
`Content-Type: multipart/alternative; boundary="b"`,
|
||||||
|
"",
|
||||||
|
"--b",
|
||||||
|
"Content-Type: text/html",
|
||||||
|
"Content-Transfer-Encoding: base64",
|
||||||
|
"",
|
||||||
|
"!!!!not base64!!!!",
|
||||||
|
"--b",
|
||||||
|
"Content-Type: text/plain",
|
||||||
|
"",
|
||||||
|
"the real body",
|
||||||
|
"--b--",
|
||||||
|
"",
|
||||||
|
}, "\r\n"))}
|
||||||
|
|
||||||
|
body, err := email.Body()
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, "the real body", body)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Regression: mime.ParseMediaType returns a usable media type alongside the
|
||||||
|
// error it reports for a malformed parameter. Discarding both dropped the part
|
||||||
|
// that held the message and delivered an empty notification with a 250.
|
||||||
|
func TestBodyReadsPartsWithAMalformedContentType(t *testing.T) {
|
||||||
|
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||||
|
"Subject: Test",
|
||||||
|
`Content-Type: multipart/alternative; boundary="b"`,
|
||||||
|
"",
|
||||||
|
"--b",
|
||||||
|
"Content-Type: text/plain; charset=",
|
||||||
|
"",
|
||||||
|
"still text",
|
||||||
|
"--b--",
|
||||||
|
"",
|
||||||
|
}, "\r\n"))}
|
||||||
|
|
||||||
|
body, err := email.Body()
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, "still text", body)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The same fault the other way round: the attachment guard used to fail open
|
||||||
|
// on the unquoted file names older mailers emit, so the enclosure won.
|
||||||
|
func TestBodySkipsAttachmentsWithAMalformedDisposition(t *testing.T) {
|
||||||
|
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||||
|
"Subject: Test",
|
||||||
|
`Content-Type: multipart/mixed; boundary="b"`,
|
||||||
|
"",
|
||||||
|
"--b",
|
||||||
|
"Content-Type: text/plain",
|
||||||
|
"Content-Disposition: attachment; filename=log file.txt",
|
||||||
|
"",
|
||||||
|
"an enclosed file, not the message",
|
||||||
|
"--b",
|
||||||
|
"Content-Type: text/plain",
|
||||||
|
"",
|
||||||
|
"the message itself",
|
||||||
|
"--b--",
|
||||||
|
"",
|
||||||
|
}, "\r\n"))}
|
||||||
|
|
||||||
|
body, err := email.Body()
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, "the message itself", body)
|
||||||
|
}
|
||||||
|
|
||||||
|
// RFC 2046 §5.1.3: the parts of a multipart/mixed are cumulative, so the first
|
||||||
|
// one carrying a body is the message and the rest are footers and enclosures.
|
||||||
|
// Applying "plain beats HTML" across them handed list mail its unsubscribe
|
||||||
|
// footer as the notification.
|
||||||
|
func TestBodyPrefersTheFirstPartOfACumulativeMultipart(t *testing.T) {
|
||||||
|
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||||
|
"Subject: Test",
|
||||||
|
`Content-Type: multipart/mixed; boundary="b"`,
|
||||||
|
"",
|
||||||
|
"--b",
|
||||||
|
"Content-Type: text/html",
|
||||||
|
"",
|
||||||
|
"<p>the newsletter</p>",
|
||||||
|
"--b",
|
||||||
|
"Content-Type: text/plain",
|
||||||
|
"",
|
||||||
|
"You are receiving this because you subscribed.",
|
||||||
|
"--b--",
|
||||||
|
"",
|
||||||
|
}, "\r\n"))}
|
||||||
|
|
||||||
|
body, err := email.FormattedBody(FormatMarkdown)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, "the newsletter", body,
|
||||||
|
"the HTML body is the message, and asking for Markdown must reach it")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBodyPassesOverAnEmptyHTMLAlternative(t *testing.T) {
|
||||||
|
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||||
|
"Subject: Test",
|
||||||
|
`Content-Type: multipart/alternative; boundary="b"`,
|
||||||
|
"",
|
||||||
|
"--b",
|
||||||
|
"Content-Type: text/html",
|
||||||
|
"",
|
||||||
|
" ",
|
||||||
|
"--b",
|
||||||
|
"Content-Type: text/html",
|
||||||
|
"",
|
||||||
|
"<p>the real one</p>",
|
||||||
|
"--b--",
|
||||||
|
"",
|
||||||
|
}, "\r\n"))}
|
||||||
|
|
||||||
|
body, err := email.Body()
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, "<p>the real one</p>", body)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A container this server cannot finish reading still hands over the parts it
|
||||||
|
// read. Only a message that yielded nothing at all is refused, since there is
|
||||||
|
// then nothing left to notify with and the same bytes fail the same way on
|
||||||
|
// every redelivery.
|
||||||
|
func TestBodyKeepsWhatItReadFromABrokenContainer(t *testing.T) {
|
||||||
|
t.Run("a good part before the break is delivered", func(t *testing.T) {
|
||||||
|
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||||
|
"Subject: Test",
|
||||||
|
`Content-Type: multipart/mixed; boundary="b"`,
|
||||||
|
"",
|
||||||
|
"--b",
|
||||||
|
"Content-Type: text/plain",
|
||||||
|
"",
|
||||||
|
"good body",
|
||||||
|
"--b",
|
||||||
|
"this is not a header",
|
||||||
|
"",
|
||||||
|
"x",
|
||||||
|
"--b--",
|
||||||
|
"",
|
||||||
|
}, "\r\n"))}
|
||||||
|
|
||||||
|
body, err := email.Body()
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, "good body", body)
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("a break before anything readable is permanent", func(t *testing.T) {
|
||||||
|
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||||
|
"Subject: Test",
|
||||||
|
`Content-Type: multipart/alternative; boundary="b"`,
|
||||||
|
"",
|
||||||
|
"--b",
|
||||||
|
"this is not a header",
|
||||||
|
"",
|
||||||
|
"body",
|
||||||
|
"--b--",
|
||||||
|
"",
|
||||||
|
}, "\r\n"))}
|
||||||
|
|
||||||
|
_, err := email.Body()
|
||||||
|
require.ErrorIs(t, err, errMalformedMessage)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
|
||||||
49
format.go
Normal file
49
format.go
Normal file
|
|
@ -0,0 +1,49 @@
|
||||||
|
package smtp2shoutrrr
|
||||||
|
|
||||||
|
import (
|
||||||
|
"slices"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// BodyFormat names the representation a recipient's targets want the message
|
||||||
|
// body in. Only an HTML body is ever rewritten: a message that already arrived
|
||||||
|
// as plain text is forwarded untouched whatever the recipient asked for.
|
||||||
|
type BodyFormat string
|
||||||
|
|
||||||
|
const (
|
||||||
|
// FormatRaw forwards the body exactly as the message carried it.
|
||||||
|
FormatRaw BodyFormat = "raw"
|
||||||
|
// FormatMarkdown renders an HTML body as Markdown. There is deliberately
|
||||||
|
// no plain-text format beside it: Markdown reads as plain text wherever
|
||||||
|
// nothing renders it, so a second conversion would only be a worse copy
|
||||||
|
// of this one.
|
||||||
|
FormatMarkdown BodyFormat = "markdown"
|
||||||
|
)
|
||||||
|
|
||||||
|
var bodyFormats = []BodyFormat{FormatRaw, FormatMarkdown}
|
||||||
|
|
||||||
|
// normalize maps an unset Format to the one that changes nothing, and accepts
|
||||||
|
// the casing a hand-written configuration file is likely to use.
|
||||||
|
func (f BodyFormat) normalize() BodyFormat {
|
||||||
|
normalized := BodyFormat(strings.ToLower(strings.TrimSpace(string(f))))
|
||||||
|
if normalized == "" {
|
||||||
|
return FormatRaw
|
||||||
|
}
|
||||||
|
|
||||||
|
return normalized
|
||||||
|
}
|
||||||
|
|
||||||
|
// valid normalizes first, so a Config assembled in Go rather than loaded from
|
||||||
|
// a file does not fail validation on a Format nobody set.
|
||||||
|
func (f BodyFormat) valid() bool {
|
||||||
|
return slices.Contains(bodyFormats, f.normalize())
|
||||||
|
}
|
||||||
|
|
||||||
|
func formatNames() string {
|
||||||
|
names := make([]string, 0, len(bodyFormats))
|
||||||
|
for _, format := range bodyFormats {
|
||||||
|
names = append(names, string(format))
|
||||||
|
}
|
||||||
|
|
||||||
|
return strings.Join(names, ", ")
|
||||||
|
}
|
||||||
5
go.mod
5
go.mod
|
|
@ -5,15 +5,18 @@ go 1.27
|
||||||
toolchain go1.27.1
|
toolchain go1.27.1
|
||||||
|
|
||||||
require (
|
require (
|
||||||
|
github.com/JohannesKaufmann/html-to-markdown/v2 v2.5.2
|
||||||
github.com/containrrr/shoutrrr v0.8.0
|
github.com/containrrr/shoutrrr v0.8.0
|
||||||
github.com/emersion/go-sasl v0.0.0-20241020182733-b788ff22d5a6
|
github.com/emersion/go-sasl v0.0.0-20241020182733-b788ff22d5a6
|
||||||
github.com/emersion/go-smtp v0.25.0
|
github.com/emersion/go-smtp v0.25.0
|
||||||
github.com/pelletier/go-toml/v2 v2.4.3
|
github.com/pelletier/go-toml/v2 v2.4.3
|
||||||
github.com/stretchr/testify v1.12.1
|
github.com/stretchr/testify v1.12.1
|
||||||
golang.org/x/crypto/x509roots/fallback v0.0.0-20260902180247-86efde54dc70
|
golang.org/x/crypto/x509roots/fallback v0.0.0-20260902180247-86efde54dc70
|
||||||
|
golang.org/x/net v0.58.0
|
||||||
)
|
)
|
||||||
|
|
||||||
require (
|
require (
|
||||||
|
github.com/JohannesKaufmann/dom v0.3.1 // indirect
|
||||||
github.com/fatih/color v1.19.0 // indirect
|
github.com/fatih/color v1.19.0 // indirect
|
||||||
github.com/go-logr/logr v1.4.4 // indirect
|
github.com/go-logr/logr v1.4.4 // indirect
|
||||||
github.com/golang/protobuf v1.5.4 // indirect
|
github.com/golang/protobuf v1.5.4 // indirect
|
||||||
|
|
@ -21,7 +24,7 @@ require (
|
||||||
github.com/mattn/go-colorable v0.1.15 // indirect
|
github.com/mattn/go-colorable v0.1.15 // indirect
|
||||||
github.com/mattn/go-isatty v0.0.24 // indirect
|
github.com/mattn/go-isatty v0.0.24 // indirect
|
||||||
go.yaml.in/yaml/v3 v3.0.5 // indirect
|
go.yaml.in/yaml/v3 v3.0.5 // indirect
|
||||||
golang.org/x/net v0.58.0 // indirect
|
|
||||||
golang.org/x/sys v0.47.0 // indirect
|
golang.org/x/sys v0.47.0 // indirect
|
||||||
|
golang.org/x/text v0.41.0 // indirect
|
||||||
golang.org/x/tools v0.49.0 // indirect
|
golang.org/x/tools v0.49.0 // indirect
|
||||||
)
|
)
|
||||||
|
|
|
||||||
12
go.sum
12
go.sum
|
|
@ -1,3 +1,7 @@
|
||||||
|
github.com/JohannesKaufmann/dom v0.3.1 h1:J16l9JAHWgkFPR3VIPbQ1gvS0cWab6laK1q7PFL3qh0=
|
||||||
|
github.com/JohannesKaufmann/dom v0.3.1/go.mod h1:BZPkf8ZeYrBgABjwJn9iiKt8aiCtkxpHkevms+Yp2DE=
|
||||||
|
github.com/JohannesKaufmann/html-to-markdown/v2 v2.5.2 h1:XFJZFWESIWlUEHHjzBuv8RvrtCWnSGlimEX17ysSDb8=
|
||||||
|
github.com/JohannesKaufmann/html-to-markdown/v2 v2.5.2/go.mod h1:BHWO8lJzttJLqwuV8Rb1B3OG2OSzLbssZDI1FRg2eAA=
|
||||||
github.com/containrrr/shoutrrr v0.8.0 h1:mfG2ATzIS7NR2Ec6XL+xyoHzN97H8WPjir8aYzJUSec=
|
github.com/containrrr/shoutrrr v0.8.0 h1:mfG2ATzIS7NR2Ec6XL+xyoHzN97H8WPjir8aYzJUSec=
|
||||||
github.com/containrrr/shoutrrr v0.8.0/go.mod h1:ioyQAyu1LJY6sILuNyKaQaw+9Ttik5QePU8atnAdO2o=
|
github.com/containrrr/shoutrrr v0.8.0/go.mod h1:ioyQAyu1LJY6sILuNyKaQaw+9Ttik5QePU8atnAdO2o=
|
||||||
github.com/emersion/go-sasl v0.0.0-20241020182733-b788ff22d5a6 h1:oP4q0fw+fOSWn3DfFi4EXdT+B+gTtzx8GC9xsc26Znk=
|
github.com/emersion/go-sasl v0.0.0-20241020182733-b788ff22d5a6 h1:oP4q0fw+fOSWn3DfFi4EXdT+B+gTtzx8GC9xsc26Znk=
|
||||||
|
|
@ -28,8 +32,16 @@ github.com/onsi/gomega v1.27.6 h1:ENqfyGeS5AX/rlXDd/ETokDz93u0YufY1Pgxuy/PvWE=
|
||||||
github.com/onsi/gomega v1.27.6/go.mod h1:PIQNjfQwkP3aQAH7lf7j87O/5FiNr+ZR8+ipb+qQlhg=
|
github.com/onsi/gomega v1.27.6/go.mod h1:PIQNjfQwkP3aQAH7lf7j87O/5FiNr+ZR8+ipb+qQlhg=
|
||||||
github.com/pelletier/go-toml/v2 v2.4.3 h1:GTRvJQutkOSftxIFD5xw9aepkYNuPWmVJpffdDPYVpY=
|
github.com/pelletier/go-toml/v2 v2.4.3 h1:GTRvJQutkOSftxIFD5xw9aepkYNuPWmVJpffdDPYVpY=
|
||||||
github.com/pelletier/go-toml/v2 v2.4.3/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
|
github.com/pelletier/go-toml/v2 v2.4.3/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
|
||||||
|
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||||
|
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||||
|
github.com/sebdah/goldie/v2 v2.8.0 h1:dZb9wR8q5++oplmEiJT+U/5KyotVD+HNGCAc5gNr8rc=
|
||||||
|
github.com/sebdah/goldie/v2 v2.8.0/go.mod h1:oZ9fp0+se1eapSRjfYbsV/0Hqhbuu3bJVvKI/NNtssI=
|
||||||
|
github.com/sergi/go-diff v1.4.0 h1:n/SP9D5ad1fORl+llWyN+D6qoUETXNZARKjyY2/KVCw=
|
||||||
|
github.com/sergi/go-diff v1.4.0/go.mod h1:A0bzQcvG0E7Rwjx0REVgAGH58e96+X0MeOfepqsbeW4=
|
||||||
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
|
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
|
||||||
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
|
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
|
||||||
|
github.com/yuin/goldmark v1.8.2 h1:kEGpgqJXdgbkhcOgBxkC0X0PmoPG1ZyoZ117rDVp4zE=
|
||||||
|
github.com/yuin/goldmark v1.8.2/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg=
|
||||||
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
|
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
|
||||||
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
|
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
|
||||||
golang.org/x/crypto/x509roots/fallback v0.0.0-20260902180247-86efde54dc70 h1:VwViOGcd7C8/Gs18efVlMxvUCS0un6KRKBPFEBvpUXg=
|
golang.org/x/crypto/x509roots/fallback v0.0.0-20260902180247-86efde54dc70 h1:VwViOGcd7C8/Gs18efVlMxvUCS0un6KRKBPFEBvpUXg=
|
||||||
|
|
|
||||||
400
html.go
Normal file
400
html.go
Normal file
|
|
@ -0,0 +1,400 @@
|
||||||
|
package smtp2shoutrrr
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
"unicode"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
|
"github.com/JohannesKaufmann/html-to-markdown/v2/converter"
|
||||||
|
"github.com/JohannesKaufmann/html-to-markdown/v2/plugin/base"
|
||||||
|
"github.com/JohannesKaufmann/html-to-markdown/v2/plugin/commonmark"
|
||||||
|
"github.com/JohannesKaufmann/html-to-markdown/v2/plugin/strikethrough"
|
||||||
|
"golang.org/x/net/html"
|
||||||
|
"golang.org/x/net/html/atom"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
// maxNestingDepth bounds the elements that give every line they contain a
|
||||||
|
// prefix. The converter re-emits that prefix per line and per level, so
|
||||||
|
// depth multiplies against line count: a message at the server's own 1 MB
|
||||||
|
// limit, nested 250 quotes deep, renders to 131 MB and takes minutes.
|
||||||
|
// Real mail quotes a handful of levels and indents fewer.
|
||||||
|
maxNestingDepth = 6
|
||||||
|
|
||||||
|
// maxRenderedBytes caps what is forwarded. No chat target displays this
|
||||||
|
// much, and a notification is not the place to find out.
|
||||||
|
maxRenderedBytes = 64 << 10
|
||||||
|
|
||||||
|
// truncationMarker tells the reader the message goes on, rather than
|
||||||
|
// letting the cap end it mid-sentence and look like the whole of it.
|
||||||
|
truncationMarker = "…"
|
||||||
|
)
|
||||||
|
|
||||||
|
// renderMarkdown rewrites an HTML body as Markdown.
|
||||||
|
//
|
||||||
|
// The conversion itself belongs to html-to-markdown, which knows CommonMark —
|
||||||
|
// escaping, delimiter runs, fencing a code block past the backticks inside it.
|
||||||
|
// What it has no opinion about is mail, since it is written for documents: a
|
||||||
|
// document has no preheader written for an inbox list, no tracking pixel, no
|
||||||
|
// cid: attachment, and lays nothing out in tables. That is what prepare does
|
||||||
|
// to the tree first, so the converter only ever sees what a notification
|
||||||
|
// should carry.
|
||||||
|
func renderMarkdown(source string) (string, error) {
|
||||||
|
document, err := html.Parse(strings.NewReader(source))
|
||||||
|
if err != nil {
|
||||||
|
// The parser recovers from any markup it can hold, so this is a
|
||||||
|
// document too deeply nested for it rather than an invalid one.
|
||||||
|
return "", fmt.Errorf("parsing HTML body: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
prepare(document, 0)
|
||||||
|
|
||||||
|
rendered, err := newConverter().ConvertNode(document)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("converting HTML body: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return truncate(strings.TrimSpace(string(rendered))), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// newConverter builds a converter per message, which is what the library's own
|
||||||
|
// entry points do: a Converter carries the error of the conversion it is
|
||||||
|
// running, and go-smtp serves every connection on its own goroutine.
|
||||||
|
func newConverter() *converter.Converter {
|
||||||
|
return converter.NewConverter(
|
||||||
|
converter.WithPlugins(
|
||||||
|
base.NewBasePlugin(),
|
||||||
|
commonmark.NewCommonmarkPlugin(
|
||||||
|
// The default is "* * *", which reads as a stray line of
|
||||||
|
// asterisks anywhere the Markdown is not rendered.
|
||||||
|
commonmark.WithHorizontalRule("---"),
|
||||||
|
),
|
||||||
|
strikethrough.NewStrikethroughPlugin(),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
func prepare(node *html.Node, depth int) {
|
||||||
|
child := node.FirstChild
|
||||||
|
|
||||||
|
for child != nil {
|
||||||
|
next := child.NextSibling
|
||||||
|
|
||||||
|
switch child.Type {
|
||||||
|
case html.CommentNode:
|
||||||
|
node.RemoveChild(child)
|
||||||
|
|
||||||
|
case html.TextNode:
|
||||||
|
child.Data = sanitizeText(child.Data)
|
||||||
|
|
||||||
|
case html.ElementNode:
|
||||||
|
if unwrapped := prepareElement(node, child, depth); unwrapped != nil {
|
||||||
|
// The element was replaced by its own children, which have
|
||||||
|
// not been looked at yet.
|
||||||
|
next = unwrapped
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
child = next
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func prepareElement(parent, node *html.Node, depth int) *html.Node {
|
||||||
|
if isHidden(node) {
|
||||||
|
// A template opens with a hidden block holding the preview line the
|
||||||
|
// inbox shows, which is written for the list view and reads as noise
|
||||||
|
// anywhere else.
|
||||||
|
parent.RemoveChild(node)
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
switch node.DataAtom {
|
||||||
|
case atom.Img:
|
||||||
|
return prepareImage(parent, node)
|
||||||
|
|
||||||
|
case atom.A:
|
||||||
|
href := sanitizeURL(attrValue(node, "href"))
|
||||||
|
if !isUsableURL(href) {
|
||||||
|
// In-page anchors, script handlers and the message's own inline
|
||||||
|
// attachments give a notification's reader nothing to open, but
|
||||||
|
// the text of the link is still part of the message.
|
||||||
|
return unwrap(parent, node)
|
||||||
|
}
|
||||||
|
|
||||||
|
setAttr(node, "href", href)
|
||||||
|
prepare(node, depth)
|
||||||
|
|
||||||
|
if !hasContent(node) {
|
||||||
|
// A link wrapped around a tracking pixel, or around the spacer
|
||||||
|
// that mail templates use for one, would render as "[](url)" —
|
||||||
|
// invisible to the reader. Its destination is the only thing left
|
||||||
|
// worth forwarding.
|
||||||
|
node.AppendChild(&html.Node{Type: html.TextNode, Data: href})
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
|
||||||
|
case atom.Td, atom.Th:
|
||||||
|
// Mail is laid out in tables far more often than it tabulates
|
||||||
|
// anything, and a converter with no table rules runs the cells of a
|
||||||
|
// row together: "Total4Failed0".
|
||||||
|
parent.InsertBefore(&html.Node{Type: html.TextNode, Data: " "}, node)
|
||||||
|
|
||||||
|
case atom.Tr:
|
||||||
|
if hasElementSibling(node) {
|
||||||
|
parent.InsertBefore(&html.Node{
|
||||||
|
Type: html.ElementNode, DataAtom: atom.Br, Data: "br",
|
||||||
|
}, node)
|
||||||
|
}
|
||||||
|
|
||||||
|
case atom.Blockquote, atom.Ul, atom.Ol:
|
||||||
|
depth++
|
||||||
|
if depth > maxNestingDepth {
|
||||||
|
return unwrap(parent, node)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
prepare(node, depth)
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// prepareImage keeps of an image only what a notification can use. Tracking
|
||||||
|
// pixels, spacers and sliced-up banners carry no alt text and go entirely; an
|
||||||
|
// inline attachment has no address the reader can fetch, so its alt text
|
||||||
|
// stays behind as ordinary words.
|
||||||
|
func prepareImage(parent, node *html.Node) *html.Node {
|
||||||
|
alt := collapseSpace(sanitizeText(attrValue(node, "alt")))
|
||||||
|
source := sanitizeURL(attrValue(node, "src"))
|
||||||
|
|
||||||
|
switch {
|
||||||
|
case alt == "":
|
||||||
|
parent.RemoveChild(node)
|
||||||
|
case !isUsableURL(source):
|
||||||
|
parent.InsertBefore(&html.Node{Type: html.TextNode, Data: alt}, node)
|
||||||
|
parent.RemoveChild(node)
|
||||||
|
default:
|
||||||
|
setAttr(node, "alt", alt)
|
||||||
|
setAttr(node, "src", source)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// unwrap replaces a node with its own children and returns the first of them.
|
||||||
|
func unwrap(parent, node *html.Node) *html.Node {
|
||||||
|
first := node.FirstChild
|
||||||
|
|
||||||
|
for child := node.FirstChild; child != nil; child = node.FirstChild {
|
||||||
|
node.RemoveChild(child)
|
||||||
|
parent.InsertBefore(child, node)
|
||||||
|
}
|
||||||
|
|
||||||
|
parent.RemoveChild(node)
|
||||||
|
|
||||||
|
return first
|
||||||
|
}
|
||||||
|
|
||||||
|
// hasContent reports whether anything left under a node would show. It runs
|
||||||
|
// after the subtree has been prepared, so an element that survived that is one
|
||||||
|
// the reader will see.
|
||||||
|
func hasContent(node *html.Node) bool {
|
||||||
|
for child := node.FirstChild; child != nil; child = child.NextSibling {
|
||||||
|
switch child.Type {
|
||||||
|
case html.ElementNode:
|
||||||
|
if child.DataAtom == atom.Img || hasContent(child) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
case html.TextNode:
|
||||||
|
if strings.TrimSpace(child.Data) != "" {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func hasElementSibling(node *html.Node) bool {
|
||||||
|
for sibling := node.PrevSibling; sibling != nil; sibling = sibling.PrevSibling {
|
||||||
|
if sibling.Type == html.ElementNode {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func truncate(rendered string) string {
|
||||||
|
if len(rendered) <= maxRenderedBytes {
|
||||||
|
return rendered
|
||||||
|
}
|
||||||
|
|
||||||
|
cut := maxRenderedBytes
|
||||||
|
for cut > 0 && !utf8.RuneStart(rendered[cut]) {
|
||||||
|
cut--
|
||||||
|
}
|
||||||
|
|
||||||
|
return strings.TrimRight(rendered[:cut], " \t\n") + "\n" + truncationMarker
|
||||||
|
}
|
||||||
|
|
||||||
|
// isUsableURL rejects the destinations a notification's reader cannot act on:
|
||||||
|
// in-page anchors, inline data, script handlers and the message's own inline
|
||||||
|
// attachments.
|
||||||
|
func isUsableURL(raw string) bool {
|
||||||
|
if raw == "" || strings.HasPrefix(raw, "#") {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
scheme, _, found := strings.Cut(raw, ":")
|
||||||
|
if !found || !isScheme(scheme) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
switch strings.ToLower(scheme) {
|
||||||
|
case "javascript", "data", "cid", "about", "blob":
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
func isScheme(s string) bool {
|
||||||
|
if s == "" || !unicode.IsLetter(rune(s[0])) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
return strings.IndexFunc(s, func(r rune) bool {
|
||||||
|
return !unicode.IsLetter(r) && !unicode.IsDigit(r) &&
|
||||||
|
r != '+' && r != '-' && r != '.'
|
||||||
|
}) < 0
|
||||||
|
}
|
||||||
|
|
||||||
|
// sanitizeURL applies to a URL attribute what a browser applies before it
|
||||||
|
// fetches one: the tabs and line breaks that may sit inside the value are
|
||||||
|
// removed. Left in, they end the line the notification is on, which is a place
|
||||||
|
// to write a sentence the reader will take for the sender's.
|
||||||
|
func sanitizeURL(raw string) string {
|
||||||
|
return strings.Map(func(r rune) rune {
|
||||||
|
if r == ' ' {
|
||||||
|
return ' '
|
||||||
|
}
|
||||||
|
|
||||||
|
if unicode.IsSpace(r) || unicode.IsControl(r) {
|
||||||
|
return -1
|
||||||
|
}
|
||||||
|
|
||||||
|
return r
|
||||||
|
}, strings.TrimSpace(raw))
|
||||||
|
}
|
||||||
|
|
||||||
|
// collapseSpace squeezes the runs of whitespace an attribute value may hold
|
||||||
|
// into single spaces, for the same reason: alt text spans lines in the source
|
||||||
|
// and must not span them in the message.
|
||||||
|
func collapseSpace(s string) string {
|
||||||
|
return strings.Join(strings.Fields(s), " ")
|
||||||
|
}
|
||||||
|
|
||||||
|
// sanitizeText turns the characters mail uses for layout into ones a
|
||||||
|
// notification can show: the non-breaking spaces that hold table cells apart
|
||||||
|
// become ordinary spaces, and the zero-width padding that hides a preheader
|
||||||
|
// from the inbox preview is dropped rather than forwarded invisibly.
|
||||||
|
func sanitizeText(s string) string {
|
||||||
|
return strings.Map(func(r rune) rune {
|
||||||
|
switch r {
|
||||||
|
case zeroWidthSpace, zeroWidthNonJoiner, zeroWidthJoiner,
|
||||||
|
wordJoiner, byteOrderMark, softHyphen:
|
||||||
|
return -1
|
||||||
|
case noBreakSpace, narrowNoBreakSpace, figureSpace:
|
||||||
|
return ' '
|
||||||
|
}
|
||||||
|
|
||||||
|
if unicode.IsSpace(r) {
|
||||||
|
return r
|
||||||
|
}
|
||||||
|
|
||||||
|
// Control characters have no rendering of their own and only muddle
|
||||||
|
// the payloads the targets are sent as.
|
||||||
|
if unicode.IsControl(r) {
|
||||||
|
return -1
|
||||||
|
}
|
||||||
|
|
||||||
|
return r
|
||||||
|
}, s)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The characters mail uses for layout rather than for words: padding that
|
||||||
|
// hides a preheader from the inbox preview, and the spaces that hold table
|
||||||
|
// cells apart without letting them wrap.
|
||||||
|
const (
|
||||||
|
softHyphen = '\u00ad'
|
||||||
|
noBreakSpace = '\u00a0'
|
||||||
|
zeroWidthSpace = '\u200b'
|
||||||
|
zeroWidthNonJoiner = '\u200c'
|
||||||
|
zeroWidthJoiner = '\u200d'
|
||||||
|
narrowNoBreakSpace = '\u202f'
|
||||||
|
wordJoiner = '\u2060'
|
||||||
|
figureSpace = '\u2007'
|
||||||
|
byteOrderMark = '\ufeff'
|
||||||
|
)
|
||||||
|
|
||||||
|
// isHidden reports whether a browser would leave the element out of the page.
|
||||||
|
func isHidden(node *html.Node) bool {
|
||||||
|
for _, attr := range node.Attr {
|
||||||
|
switch attr.Key {
|
||||||
|
case "hidden":
|
||||||
|
return true
|
||||||
|
case "style":
|
||||||
|
if isHidingStyle(spaceStripper.Replace(strings.ToLower(attr.Val))) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func isHidingStyle(style string) bool {
|
||||||
|
for _, declaration := range []string{"display:none", "visibility:hidden", "mso-hide:all"} {
|
||||||
|
if strings.Contains(style, declaration) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// font-size:0 hides a preheader too, but only when the zero is the whole
|
||||||
|
// value: font-size:0.9em is an ordinary line of text.
|
||||||
|
if index := strings.Index(style, "font-size:0"); index >= 0 {
|
||||||
|
rest := strings.TrimPrefix(style[index+len("font-size:0"):], "px")
|
||||||
|
rest = strings.TrimPrefix(rest, "pt")
|
||||||
|
|
||||||
|
return rest == "" || strings.HasPrefix(rest, ";")
|
||||||
|
}
|
||||||
|
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
var spaceStripper = strings.NewReplacer(" ", "", "\t", "", "\n", "", "\r", "")
|
||||||
|
|
||||||
|
func attrValue(node *html.Node, key string) string {
|
||||||
|
for _, attr := range node.Attr {
|
||||||
|
if attr.Key == key {
|
||||||
|
return attr.Val
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
func setAttr(node *html.Node, key, value string) {
|
||||||
|
for i := range node.Attr {
|
||||||
|
if node.Attr[i].Key == key {
|
||||||
|
node.Attr[i].Val = value
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
node.Attr = append(node.Attr, html.Attribute{Key: key, Val: value})
|
||||||
|
}
|
||||||
248
html_test.go
Normal file
248
html_test.go
Normal file
|
|
@ -0,0 +1,248 @@
|
||||||
|
package smtp2shoutrrr
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
func render(t *testing.T, source string) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
out, err := renderMarkdown(source)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// The conversion itself belongs to html-to-markdown; what these pin is the
|
||||||
|
// contract this package keeps on top of it, which is what a notification
|
||||||
|
// should carry rather than what the document said.
|
||||||
|
func TestRenderMarkdown(t *testing.T) {
|
||||||
|
for name, tc := range map[string]struct{ source, want string }{
|
||||||
|
"paragraphs are separated by a blank line": {
|
||||||
|
source: "<p>first</p><p>second</p>",
|
||||||
|
want: "first\n\nsecond",
|
||||||
|
},
|
||||||
|
"headings keep their level": {
|
||||||
|
source: "<h1>one</h1><h3>three</h3>",
|
||||||
|
want: "# one\n\n### three",
|
||||||
|
},
|
||||||
|
"emphasis": {
|
||||||
|
source: "<p><b>bold</b> <i>italic</i> <del>gone</del> <code>x=1</code></p>",
|
||||||
|
want: "**bold** *italic* ~~gone~~ `x=1`",
|
||||||
|
},
|
||||||
|
"two spans of the same emphasis meeting head on become one": {
|
||||||
|
// What every Word and Outlook export does to a bolded label.
|
||||||
|
source: "<p><b><span>Total</span></b><b><span>: 49.99</span></b></p>",
|
||||||
|
want: "**Total: 49.99**",
|
||||||
|
},
|
||||||
|
"emphasis nested inside itself does not double its markers": {
|
||||||
|
source: "<p><b>a<strong>b</strong></b></p>",
|
||||||
|
want: "**ab**",
|
||||||
|
},
|
||||||
|
"links keep their destination": {
|
||||||
|
source: `<p>see <a href="https://example.com/x">the docs</a></p>`,
|
||||||
|
want: "see [the docs](https://example.com/x)",
|
||||||
|
},
|
||||||
|
"a link with nothing to click on falls back to its destination": {
|
||||||
|
// Otherwise a link wrapped around a tracking pixel renders as
|
||||||
|
// "[](url)", which the reader cannot see at all.
|
||||||
|
source: `<p><a href="https://example.com/x"><img src="https://x.example/p.gif"></a></p>`,
|
||||||
|
want: "[https://example.com/x](https://example.com/x)",
|
||||||
|
},
|
||||||
|
"a link around nothing but a spacer falls back the same way": {
|
||||||
|
source: `<p><a href="https://example.com/x"><span> </span></a></p>`,
|
||||||
|
want: "[https://example.com/x](https://example.com/x)",
|
||||||
|
},
|
||||||
|
"unusable destinations are dropped but their text is kept": {
|
||||||
|
source: `<a href="#anchor">up</a> <a href="javascript:evil()">click</a> <a href="cid:part1">img</a>`,
|
||||||
|
want: "up click img",
|
||||||
|
},
|
||||||
|
"line breaks inside a URL are removed rather than forwarded": {
|
||||||
|
// A newline in an href is invisible in the document and a
|
||||||
|
// fabricated line in the notification.
|
||||||
|
source: "<a href=\"https://ok.example/r URGENT: wire funds\">open report</a>",
|
||||||
|
want: "[open report](https://ok.example/rURGENT:%20wire%20funds)",
|
||||||
|
},
|
||||||
|
"unordered lists": {
|
||||||
|
source: "<ul><li>one</li><li>two</li></ul>",
|
||||||
|
want: "- one\n- two",
|
||||||
|
},
|
||||||
|
"ordered lists count from their start attribute": {
|
||||||
|
source: `<ol start="3"><li>three</li><li>four</li></ol>`,
|
||||||
|
want: "3. three\n4. four",
|
||||||
|
},
|
||||||
|
"blockquotes mark every line": {
|
||||||
|
source: "<p>before</p><blockquote><p>quoted</p></blockquote><p>after</p>",
|
||||||
|
want: "before\n\n> quoted\n\nafter",
|
||||||
|
},
|
||||||
|
"a code block is fenced past the backticks it holds": {
|
||||||
|
// Otherwise the sender closes the block and everything after it —
|
||||||
|
// a heading, a link — renders as live Markdown in a message the
|
||||||
|
// reader takes for a forwarded notification.
|
||||||
|
source: "<pre>ok\n```\n## Your account is locked\n</pre>",
|
||||||
|
want: "````\nok\n```\n## Your account is locked\n````",
|
||||||
|
},
|
||||||
|
"table rows become lines and cells stay apart": {
|
||||||
|
// Mail lays itself out in tables far more often than it tabulates
|
||||||
|
// anything, and a converter with no table rules would otherwise
|
||||||
|
// run a row together as "Total4Failed0".
|
||||||
|
source: "<table><tr><td>Total</td><td>4</td></tr><tr><td>Failed</td><td>0</td></tr></table>",
|
||||||
|
want: "Total 4 \nFailed 0",
|
||||||
|
},
|
||||||
|
"images without alt text are dropped": {
|
||||||
|
source: `<p>a<img src="https://x.example/pixel.gif">b</p>`,
|
||||||
|
want: "ab",
|
||||||
|
},
|
||||||
|
"images with alt text are kept": {
|
||||||
|
source: `<img src="https://x.example/logo.png" alt="Logo">`,
|
||||||
|
want: "",
|
||||||
|
},
|
||||||
|
"an inline attachment keeps its alt text without a destination": {
|
||||||
|
source: `<img src="cid:part1.abc" alt="Chart">`,
|
||||||
|
want: "Chart",
|
||||||
|
},
|
||||||
|
"alt text is collapsed onto one line": {
|
||||||
|
source: "<img src=\"https://x.example/i.png\" alt=\"a\n long alt\">",
|
||||||
|
want: "",
|
||||||
|
},
|
||||||
|
"script and style content never reaches the reader": {
|
||||||
|
source: "<html><head><style>p{color:red}</style></head><body><script>x()</script><p>body</p></body></html>",
|
||||||
|
want: "body",
|
||||||
|
},
|
||||||
|
"hidden preheaders are left out": {
|
||||||
|
source: `<div style="display: none">inbox preview</div><p>real body</p>`,
|
||||||
|
want: "real body",
|
||||||
|
},
|
||||||
|
"the other ways a preheader hides are recognised too": {
|
||||||
|
source: `<div style="visibility:hidden">a</div><div style="mso-hide:all">b</div>` +
|
||||||
|
`<div style="font-size:0px">c</div><div style="font-size:0.9em">d</div><p>body</p>`,
|
||||||
|
want: "d\n\nbody",
|
||||||
|
},
|
||||||
|
"entities and non-breaking spaces become ordinary text": {
|
||||||
|
source: "<p>4m 12s & counting — done</p>",
|
||||||
|
want: "4m 12s & counting — done",
|
||||||
|
},
|
||||||
|
"zero width padding is dropped": {
|
||||||
|
source: "<p>​a​b​</p>",
|
||||||
|
want: "ab",
|
||||||
|
},
|
||||||
|
"text that looks like markup is escaped": {
|
||||||
|
source: "<p>2 * 3, a [b] c, `tick`</p>",
|
||||||
|
want: "2 * 3, a \\[b] c, \\`tick\\`",
|
||||||
|
},
|
||||||
|
"markers are only defused where they would take effect": {
|
||||||
|
source: "<p>- not a bullet</p><p># not a heading</p><p>1. not a list</p><p>a - b</p>",
|
||||||
|
want: "\\- not a bullet\n\n\\# not a heading\n\n1\\. not a list\n\na - b",
|
||||||
|
},
|
||||||
|
"a document with nothing to say renders to nothing": {
|
||||||
|
source: `<html><body><img src="https://x.example/pixel.gif"></body></html>`,
|
||||||
|
want: "",
|
||||||
|
},
|
||||||
|
} {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
require.Equal(t, tc.want, render(t, tc.source))
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The parser recovers from anything, so the conversion has to as well: a
|
||||||
|
// notification is worth more than a report that the markup was invalid.
|
||||||
|
func TestRenderMarkdownSurvivesBrokenMarkup(t *testing.T) {
|
||||||
|
for name, tc := range map[string]struct{ source, want string }{
|
||||||
|
"unclosed tags": {"<p><b>bold<i>both<p>next", "**bold*both***\n\n***next***"},
|
||||||
|
"stray closing tags": {"</div></p>text</b>", "text"},
|
||||||
|
"no markup at all": {"just some words", "just some words"},
|
||||||
|
"an empty document": {"", ""},
|
||||||
|
"an unterminated tag": {"<p>text<", "text<"},
|
||||||
|
} {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
require.Equal(t, tc.want, render(t, tc.source))
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Regression: a line prefix is re-emitted on every line of every level it
|
||||||
|
// nests, so depth multiplies against line count. One message at the server's
|
||||||
|
// own 1 MB limit rendered to 131 MB over two and a quarter minutes — the same
|
||||||
|
// class of fault as the remote DoS closed in 7565618. Neither the converter
|
||||||
|
// nor any of the alternatives bounds this on its own.
|
||||||
|
func TestRenderMarkdownBoundsPathologicalInput(t *testing.T) {
|
||||||
|
// Sized to overrun the cap several times over once the nesting is
|
||||||
|
// flattened, rather than to reproduce the original 1 MB message: the
|
||||||
|
// assertion is the same and the suite stays quick.
|
||||||
|
lines := strings.Repeat("<p>x</p>", maxRenderedBytes/maxNestingDepth)
|
||||||
|
|
||||||
|
for name, source := range map[string]string{
|
||||||
|
"quotes nested far past anything real": strings.Repeat("<blockquote>", 250) + lines,
|
||||||
|
"lists nested far past anything real": strings.Repeat("<ul><li>", 250) + lines,
|
||||||
|
} {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
out, err := renderMarkdown(source)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.LessOrEqual(t, len(out), maxRenderedBytes+len(truncationMarker)+1)
|
||||||
|
require.True(t, strings.HasSuffix(out, truncationMarker),
|
||||||
|
"a capped message says so rather than ending mid-sentence")
|
||||||
|
require.True(t, utf8.ValidString(out), "the cap must not fall inside a rune")
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Nesting under the limit is left alone, so an ordinary quoted reply still
|
||||||
|
// reads as one.
|
||||||
|
func TestRenderMarkdownKeepsNestingUnderTheLimit(t *testing.T) {
|
||||||
|
source := strings.Repeat("<blockquote>", maxNestingDepth) + "<p>deep</p>" +
|
||||||
|
strings.Repeat("</blockquote>", maxNestingDepth)
|
||||||
|
|
||||||
|
require.Equal(t, strings.Repeat("> ", maxNestingDepth)+"deep", render(t, source))
|
||||||
|
}
|
||||||
|
|
||||||
|
// A realistic transactional message, pinned end to end. The faults this
|
||||||
|
// feature has had lived in combinations of features rather than in one of
|
||||||
|
// them, which is what a fixture catches and a unit test does not.
|
||||||
|
func TestRenderMarkdownOnARealisticMessage(t *testing.T) {
|
||||||
|
require.Equal(t, strings.Join([]string{
|
||||||
|
"[](https://billing.example.com)",
|
||||||
|
"",
|
||||||
|
"## Invoice #2026-0231",
|
||||||
|
"",
|
||||||
|
"Hi Ferran,",
|
||||||
|
"",
|
||||||
|
"Your invoice for **February 2026 (30 days)** is ready. The total is " +
|
||||||
|
"**€42.00**, charged to the card ending 4242 on *1 March*.",
|
||||||
|
"",
|
||||||
|
"Description Qty Amount ",
|
||||||
|
"Hosting — small 1 €30.00 ",
|
||||||
|
"Backups 2 €12.00",
|
||||||
|
"",
|
||||||
|
"Reference: `inv_2026*0231` ",
|
||||||
|
"[View invoice](https://billing.example.com/invoices/2026-0231?utm_source=email&utm_medium=cta)",
|
||||||
|
"",
|
||||||
|
"You are receiving this because you have an account. " +
|
||||||
|
"[Unsubscribe](https://billing.example.com/unsubscribe?t=abc) · Back to top",
|
||||||
|
}, "\n"), render(t, marketingMessage))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Shaped like the transactional mail this feature exists for: a hidden
|
||||||
|
// preheader, a layout table, a logo wrapped in a link, an Outlook-style split
|
||||||
|
// bold run, a tracking pixel and an in-page anchor.
|
||||||
|
const marketingMessage = `<!DOCTYPE html><html><head><meta charset="utf-8"><title>Invoice</title>
|
||||||
|
<style>.btn{background:#000}</style></head><body style="margin:0">
|
||||||
|
<span style="display:none;font-size:1px">Your February invoice is ready​​</span>
|
||||||
|
<table width="100%"><tbody><tr><td align="center"><table width="600"><tbody>
|
||||||
|
<tr><td><a href="https://billing.example.com"><img src="https://cdn.example.com/logo.png" alt="Example Billing"></a></td></tr>
|
||||||
|
<tr><td><h2>Invoice #2026-0231</h2></td></tr>
|
||||||
|
<tr><td>Hi Ferran,<br><br>Your invoice for <b><span>February 2026</span></b><b><span> (30 days)</span></b> is ready.
|
||||||
|
The total is <strong>€42.00</strong>, charged to the card ending 4242 on <em>1 March</em>.</td></tr>
|
||||||
|
<tr><td><table><tbody><tr><th>Description</th><th>Qty</th><th>Amount</th></tr>
|
||||||
|
<tr><td>Hosting — small</td><td>1</td><td>€30.00</td></tr>
|
||||||
|
<tr><td>Backups</td><td>2</td><td>€12.00</td></tr></tbody></table></td></tr>
|
||||||
|
<tr><td>Reference: <code>inv_2026*0231</code></td></tr>
|
||||||
|
<tr><td align="center"><a class="btn" href="https://billing.example.com/invoices/2026-0231?utm_source=email&utm_medium=cta">View invoice</a></td></tr>
|
||||||
|
<tr><td><p style="font-size:11px">You are receiving this because you have an account.
|
||||||
|
<a href="https://billing.example.com/unsubscribe?t=abc">Unsubscribe</a> · <a href="#top">Back to top</a></p>
|
||||||
|
<img src="https://track.example.com/o.gif?id=abc" width="1" height="1"></td></tr>
|
||||||
|
</tbody></table></td></tr></tbody></table></body></html>`
|
||||||
Loading…
Reference in a new issue