feat: add a per-recipient Format option to convert HTML bodies (FMG-9) #13
12 changed files with 1773 additions and 77 deletions
75
README.md
75
README.md
|
|
@ -29,6 +29,11 @@ Targets = [
|
|||
"discord://token@id",
|
||||
"slack://token@channel"
|
||||
]
|
||||
# Optional: how an HTML message body should reach these targets.
|
||||
# "raw" forward the message unchanged (default)
|
||||
# "markdown" render an HTML body as Markdown
|
||||
# See "Converting HTML messages" below.
|
||||
# Format = "markdown"
|
||||
|
||||
# Single Target (Deprecated)
|
||||
# The Target field is still supported for backward compatibility
|
||||
|
|
@ -42,14 +47,17 @@ Target = "ntfy://ntfy.sh/legacy-topic?tags=thing" # Will show deprecation warni
|
|||
[CatchAll]
|
||||
# Shoutrrr services to forward unmatched emails to
|
||||
Targets = ["ntfy://ntfy.sh/catch-all-topic?tags=unmatched"]
|
||||
# Format applies to the catch-all as well
|
||||
# Format = "markdown"
|
||||
```
|
||||
|
||||
The server refuses to start if the configuration cannot forward anything — no
|
||||
recipients and no catch-all, a recipient without addresses or usable targets, or
|
||||
a mistyped table name such as `[[Recipient]]`, which TOML would otherwise accept
|
||||
silently. It also refuses to start without a `Username` and a `Password`: these
|
||||
used to fall back to `username`/`password`, which left the authentication gate
|
||||
open to the most obvious guess there is.
|
||||
silently, or a `Format` that names none of the supported conversions. It also
|
||||
refuses to start without a `Username` and a `Password`: these used to fall back
|
||||
to `username`/`password`, which left the authentication gate open to the most
|
||||
obvious guess there is.
|
||||
|
||||
Clients must authenticate with the configured `Username` and `Password` before
|
||||
starting a mail transaction; an unauthenticated client is refused rather than
|
||||
|
|
@ -86,6 +94,67 @@ docker run -v /path/to/config.toml:/config.toml \
|
|||
git.nakama.town/fmartingr/smtp2shoutrrr:latest
|
||||
```
|
||||
|
||||
## Converting HTML messages
|
||||
|
||||
Plenty of mail carries nothing but HTML, and plenty of notification services
|
||||
render none of it — a Mattermost direct message shows the markup verbatim.
|
||||
Setting `Format` on a recipient (or on `[CatchAll]`) rewrites an HTML body
|
||||
before it is forwarded:
|
||||
|
||||
| `Format` | Effect |
|
||||
| ---------- | --------------------------------------------------- |
|
||||
| `raw` | Forward the body as the message wrote it. (default) |
|
||||
| `markdown` | Render an HTML body as Markdown. |
|
||||
|
||||
There is deliberately no plain-text format beside it. Markdown reads as plain
|
||||
text wherever nothing renders it — that is rather the point of Markdown — so a
|
||||
second conversion would only be a worse copy of this one.
|
||||
|
||||
The value is not case sensitive, and an unrecognised one is refused when the
|
||||
server starts rather than silently forwarding raw HTML.
|
||||
|
||||
Only an HTML body is ever rewritten. A message that arrives as plain text is
|
||||
what the sender chose to write and is forwarded untouched whatever `Format`
|
||||
says. Which part of a multipart message that is follows RFC 2046: the parts of
|
||||
a `multipart/alternative` are the same content in several forms, so the
|
||||
`text/plain` one is preferred where it is not empty; the parts of any other
|
||||
multipart are cumulative, so the first one carrying a body is the message and
|
||||
the footers, signatures and attachments after it are not.
|
||||
|
||||
The Markdown itself is produced by
|
||||
[html-to-markdown](https://github.com/JohannesKaufmann/html-to-markdown), which
|
||||
knows CommonMark — escaping, delimiter runs, fencing a code block past the
|
||||
backticks inside it. What it does not know is mail, because it is written for
|
||||
documents. Before the conversion runs, the parsed message is stripped down to
|
||||
what a notification should carry:
|
||||
|
||||
- Hidden elements go. A template opens with a preheader written for the inbox
|
||||
list, which reads as noise anywhere else.
|
||||
- Images without alt text go, which removes tracking pixels, spacers and
|
||||
sliced-up banners. An inline `cid:` attachment leaves its alt text behind as
|
||||
ordinary words.
|
||||
- Destinations a reader cannot open — `#anchors`, `javascript:`, `cid:` — are
|
||||
dropped and the text of the link is kept. Tabs and line breaks are removed
|
||||
from the rest, since a line break inside an `href` is invisible in the
|
||||
document and a fabricated line in the message.
|
||||
- Table rows become lines and their cells stay apart, since mail lays itself
|
||||
out in tables far more often than it tabulates anything.
|
||||
- Quoting and list nesting is flattened past six levels. A line prefix is
|
||||
re-emitted on every line of every level it nests, so depth multiplies against
|
||||
line count: a message at the 1 MB limit nested 250 quotes deep otherwise
|
||||
renders to 131 MB and takes minutes of CPU.
|
||||
|
||||
If a body is too deeply nested for the HTML parser, or renders to nothing at
|
||||
all because it was images and tracking pixels, it is forwarded unchanged and a
|
||||
warning is logged: reformatting is a courtesy to the target, not a condition of
|
||||
delivery. Output past 64 KiB is cut short with a `…`, which no chat target
|
||||
would have displayed anyway.
|
||||
|
||||
Bodies are decoded before they are rendered, so `quoted-printable` and `base64`
|
||||
transfer encodings and non-UTF-8 character sets (`iso-8859-1`, `windows-1252`,
|
||||
…) reach the target as readable text in both formats, `raw` included — which is
|
||||
the one way `raw` is not quite the bytes that arrived.
|
||||
|
||||
## Development
|
||||
|
||||
Run the server with:
|
||||
|
|
|
|||
14
backend.go
14
backend.go
|
|
@ -56,7 +56,7 @@ type Backend struct {
|
|||
config *Config
|
||||
}
|
||||
|
||||
func (bkd *Backend) sendNotifications(recipient ConfigRecipient, email ReceivedEmail) error {
|
||||
func (bkd *Backend) sendNotifications(recipient ConfigRecipient, email *ReceivedEmail) error {
|
||||
// Get all target URLs (handles merging and caching)
|
||||
targetURLs := recipient.GetTargetURLs()
|
||||
|
||||
|
|
@ -66,7 +66,7 @@ func (bkd *Backend) sendNotifications(recipient ConfigRecipient, email ReceivedE
|
|||
}
|
||||
|
||||
// Prepare email body once (reuse for all targets)
|
||||
body, err := email.Body()
|
||||
body, err := email.FormattedBody(recipient.Format)
|
||||
if err != nil {
|
||||
slog.Error("Error getting email body", slog.String("err", err.Error()))
|
||||
return fmt.Errorf("failed to get email body: %w", err)
|
||||
|
|
@ -74,6 +74,7 @@ func (bkd *Backend) sendNotifications(recipient ConfigRecipient, email ReceivedE
|
|||
|
||||
slog.Debug("Prepared email body",
|
||||
slog.Int("body_length", len(body)),
|
||||
slog.String("format", string(recipient.Format)),
|
||||
slog.String("subject", email.Msg.Header.Get("Subject")),
|
||||
slog.String("content_type", email.Msg.Header.Get("Content-Type")))
|
||||
|
||||
|
|
@ -134,7 +135,7 @@ func (bkd *Backend) NewSession(c *smtp.Conn) (smtp.Session, error) {
|
|||
}, nil
|
||||
}
|
||||
|
||||
func (bkd *Backend) forwardEmail(email ReceivedEmail) error {
|
||||
func (bkd *Backend) forwardEmail(email *ReceivedEmail) error {
|
||||
slog.Info("forwading message", slog.String("to", strings.Join(email.Recipients, ",")))
|
||||
|
||||
// Try to match configured recipients first
|
||||
|
|
@ -181,7 +182,10 @@ type Session struct {
|
|||
|
||||
config *Config
|
||||
|
||||
forwarderFunc func(ReceivedEmail) error
|
||||
// forwarderFunc takes the message by pointer: the body is read from a
|
||||
// single-use stream and cached on it, so a copy would forward an empty
|
||||
// notification to every recipient after the first.
|
||||
forwarderFunc func(*ReceivedEmail) error
|
||||
}
|
||||
|
||||
func (s *Session) isAuthenticated() bool {
|
||||
|
|
@ -292,7 +296,7 @@ func (s *Session) Data(r io.Reader) error {
|
|||
|
||||
slog.Info("Received email", slog.String("destination", strings.Join(recipients, ",")))
|
||||
|
||||
if err := s.forwarderFunc(ReceivedEmail{
|
||||
if err := s.forwarderFunc(&ReceivedEmail{
|
||||
Recipients: recipients,
|
||||
Msg: msg,
|
||||
}); err != nil {
|
||||
|
|
|
|||
102
backend_test.go
102
backend_test.go
|
|
@ -699,3 +699,105 @@ func TestRecipientWithoutTargetsIsRejectedPermanently(t *testing.T) {
|
|||
require.Error(t, err)
|
||||
require.Contains(t, err.Error(), "550", "a retry cannot add targets to a configuration")
|
||||
}
|
||||
|
||||
// The case the option exists for: a target that renders no HTML, receiving
|
||||
// mail that carries nothing else.
|
||||
func TestHTMLBodyIsConvertedForTheRecipientsFormat(t *testing.T) {
|
||||
notifications := make(chan string, 4)
|
||||
mockServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
body, err := io.ReadAll(r.Body)
|
||||
if err != nil {
|
||||
w.WriteHeader(http.StatusInternalServerError)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
notifications <- string(body)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}))
|
||||
defer mockServer.Close()
|
||||
|
||||
config := &Config{
|
||||
Port: 0,
|
||||
Username: "testuser",
|
||||
Password: "testpass",
|
||||
Recipients: []ConfigRecipient{
|
||||
{
|
||||
Addresses: []string{"markdown@example.com"},
|
||||
Targets: []string{"generic+" + mockServer.URL + "/?template=json"},
|
||||
Format: FormatMarkdown,
|
||||
},
|
||||
{
|
||||
Addresses: []string{"raw@example.com"},
|
||||
Targets: []string{"generic+" + mockServer.URL + "/?template=json"},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
addr, _, _ := startServer(t, NewSMTPServer(config))
|
||||
auth := smtp.PlainAuth("", config.Username, config.Password, "localhost")
|
||||
|
||||
message := []byte(strings.Join([]string{
|
||||
"Subject: Deploy failed",
|
||||
`Content-Type: multipart/mixed; boundary="b"`,
|
||||
"",
|
||||
"--b",
|
||||
"Content-Type: text/html; charset=utf-8",
|
||||
"Content-Transfer-Encoding: quoted-printable",
|
||||
"",
|
||||
"<p>The <b>staging</b> deploy =E2=80=94 <a href=3D\"https://ci.example.com/42\">run 4=",
|
||||
"2</a> =E2=80=94 failed.</p>",
|
||||
"--b--",
|
||||
"",
|
||||
}, "\r\n"))
|
||||
|
||||
for _, recipient := range []string{"markdown@example.com", "raw@example.com"} {
|
||||
require.NoError(t, smtp.SendMail(addr, auth, "sender@example.com", []string{recipient}, message))
|
||||
}
|
||||
|
||||
converted := <-notifications
|
||||
require.Contains(t, converted,
|
||||
`The **staging** deploy — [run 42](https://ci.example.com/42) — failed.`)
|
||||
require.NotContains(t, converted, "<b>")
|
||||
require.NotContains(t, converted, "=E2=80=94", "the transfer encoding is undone before the markup is")
|
||||
|
||||
unconverted := <-notifications
|
||||
require.Contains(t, unconverted, `<b>staging</b>`,
|
||||
"a recipient that asked for nothing still receives the message as it arrived")
|
||||
}
|
||||
|
||||
// The message body is read from a single-use stream and cached on the value
|
||||
// that holds it, so passing it by value hands every recipient after the first
|
||||
// an empty notification with a 250.
|
||||
func TestMessageBodySurvivesASecondRecipient(t *testing.T) {
|
||||
notifications := make(chan string, 4)
|
||||
mockServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
body, err := io.ReadAll(r.Body)
|
||||
if err != nil {
|
||||
w.WriteHeader(http.StatusInternalServerError)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
notifications <- string(body)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}))
|
||||
defer mockServer.Close()
|
||||
|
||||
recipient := ConfigRecipient{
|
||||
Addresses: []string{"test@example.com"},
|
||||
Targets: []string{"generic+" + mockServer.URL + "/?template=json"},
|
||||
}
|
||||
|
||||
backend := &Backend{config: &Config{Username: "u", Password: "p"}}
|
||||
email := &ReceivedEmail{
|
||||
Recipients: []string{"test@example.com"},
|
||||
Msg: readMessage(t, "Subject: Test\r\n\r\nthe body"),
|
||||
}
|
||||
|
||||
require.NoError(t, backend.sendNotifications(recipient, email))
|
||||
require.NoError(t, backend.sendNotifications(recipient, email))
|
||||
|
||||
require.Contains(t, <-notifications, "the body")
|
||||
require.Contains(t, <-notifications, "the body")
|
||||
}
|
||||
|
|
|
|||
35
config.go
35
config.go
|
|
@ -67,6 +67,14 @@ func (c *Config) SetDefaults() {
|
|||
if c.Port == 0 {
|
||||
c.Port = 11125
|
||||
}
|
||||
|
||||
for i := range c.Recipients {
|
||||
c.Recipients[i].Format = c.Recipients[i].Format.normalize()
|
||||
}
|
||||
|
||||
if c.CatchAll != nil {
|
||||
c.CatchAll.Format = c.CatchAll.Format.normalize()
|
||||
}
|
||||
}
|
||||
|
||||
// Validate rejects configuration that would leave the server relaying for
|
||||
|
|
@ -95,10 +103,20 @@ func (c *Config) Validate() error {
|
|||
return fmt.Errorf("recipient %d (%s) has no usable Targets",
|
||||
i, strings.Join(r.Addresses, ","))
|
||||
}
|
||||
if !r.Format.valid() {
|
||||
return fmt.Errorf("recipient %d (%s) has an unknown Format %q, expected one of: %s",
|
||||
i, strings.Join(r.Addresses, ","), r.Format, formatNames())
|
||||
}
|
||||
}
|
||||
|
||||
if c.CatchAll != nil && len(c.CatchAll.GetTargetURLs()) == 0 {
|
||||
return errors.New("CatchAll has no usable Targets")
|
||||
if c.CatchAll != nil {
|
||||
if len(c.CatchAll.GetTargetURLs()) == 0 {
|
||||
return errors.New("CatchAll has no usable Targets")
|
||||
}
|
||||
if !c.CatchAll.Format.valid() {
|
||||
return fmt.Errorf("CatchAll has an unknown Format %q, expected one of: %s",
|
||||
c.CatchAll.Format, formatNames())
|
||||
}
|
||||
}
|
||||
|
||||
if len(c.Recipients) == 0 && c.CatchAll == nil {
|
||||
|
|
@ -109,9 +127,16 @@ func (c *Config) Validate() error {
|
|||
}
|
||||
|
||||
type ConfigRecipient struct {
|
||||
Addresses []string // email addresses
|
||||
Target string // deprecated: use Targets instead
|
||||
Targets []string // shoutrrr addresses (supports multiple)
|
||||
Addresses []string // email addresses
|
||||
Target string // deprecated: use Targets instead
|
||||
Targets []string // shoutrrr addresses (supports multiple)
|
||||
|
||||
// Format is the representation these targets want an HTML message body
|
||||
// in: raw (the default, forwarding the message unchanged), text or
|
||||
// markdown. Targets that render neither HTML nor Markdown — a Mattermost
|
||||
// direct message, for one — otherwise receive the markup verbatim.
|
||||
Format BodyFormat
|
||||
|
||||
targetURLs []*url.URL // cached parsed URLs
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -191,3 +191,61 @@ Targets = ["ntfy://ntfy.sh/catch-all"]
|
|||
require.Contains(t, logged.String(), "ignoring unknown key in config file")
|
||||
require.Contains(t, logged.String(), "Recipient")
|
||||
}
|
||||
|
||||
func TestLoadConfigReadsFormat(t *testing.T) {
|
||||
config, err := LoadConfig(writeConfig(t, credentials+`
|
||||
[[Recipients]]
|
||||
Addresses = ["user@example.com"]
|
||||
Targets = ["ntfy://ntfy.sh/topic"]
|
||||
Format = "Markdown"
|
||||
|
||||
[[Recipients]]
|
||||
Addresses = ["asis@example.com"]
|
||||
Targets = ["ntfy://ntfy.sh/asis"]
|
||||
|
||||
[CatchAll]
|
||||
Targets = ["ntfy://ntfy.sh/catch-all"]
|
||||
Format = "markdown"
|
||||
`))
|
||||
require.NoError(t, err)
|
||||
|
||||
require.Equal(t, FormatMarkdown, config.Recipients[0].Format, "the option is not case sensitive")
|
||||
require.Equal(t, FormatRaw, config.Recipients[1].Format, "an unset Format forwards the message unchanged")
|
||||
require.Equal(t, FormatMarkdown, config.CatchAll.Format)
|
||||
}
|
||||
|
||||
// A typo here is silent otherwise: the server starts and forwards raw HTML to
|
||||
// a target that cannot render it.
|
||||
func TestLoadConfigRejectsUnknownFormat(t *testing.T) {
|
||||
for name, contents := range map[string]string{
|
||||
"on a recipient": `
|
||||
[[Recipients]]
|
||||
Addresses = ["user@example.com"]
|
||||
Targets = ["ntfy://ntfy.sh/topic"]
|
||||
Format = "text"
|
||||
`,
|
||||
"on the catch-all": `
|
||||
[CatchAll]
|
||||
Targets = ["ntfy://ntfy.sh/catch-all"]
|
||||
Format = "md"
|
||||
`,
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
_, err := LoadConfig(writeConfig(t, credentials+contents))
|
||||
require.Error(t, err)
|
||||
require.Contains(t, err.Error(), "raw, markdown")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Validate used to depend on SetDefaults having run, so a Config assembled in
|
||||
// Go failed on a Format nobody had set.
|
||||
func TestValidateAcceptsAConfigBuiltWithoutDefaults(t *testing.T) {
|
||||
config := &Config{
|
||||
Username: "user",
|
||||
Password: "secret",
|
||||
Recipients: []ConfigRecipient{{Addresses: []string{"user@example.com"}, Targets: []string{"ntfy://ntfy.sh/topic"}}},
|
||||
}
|
||||
|
||||
require.NoError(t, config.Validate())
|
||||
}
|
||||
|
|
|
|||
442
email.go
442
email.go
|
|
@ -2,14 +2,20 @@ package smtp2shoutrrr
|
|||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/base64"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"mime"
|
||||
"mime/multipart"
|
||||
"mime/quotedprintable"
|
||||
"net/mail"
|
||||
"net/textproto"
|
||||
"strings"
|
||||
"unicode"
|
||||
|
||||
"golang.org/x/net/html/charset"
|
||||
)
|
||||
|
||||
// errMalformedMessage marks a message this server can never turn into a
|
||||
|
|
@ -17,81 +23,383 @@ import (
|
|||
// sender is told to give up rather than retry forever.
|
||||
var errMalformedMessage = errors.New("malformed message")
|
||||
|
||||
// maxMultipartDepth bounds how deep a message can make this server walk. Real
|
||||
// mail nests three levels at most (mixed → related → alternative); anything
|
||||
// past that is a generated or hostile message rather than one with a body
|
||||
// worth finding.
|
||||
const maxMultipartDepth = 8
|
||||
|
||||
type ReceivedEmail struct {
|
||||
Recipients []string
|
||||
Msg *mail.Message
|
||||
|
||||
bodyRead bool
|
||||
body string
|
||||
bodyIsHTML bool
|
||||
}
|
||||
|
||||
// Body returns the message body as the message carried it: its text/plain part
|
||||
// when there is one, its text/html part otherwise.
|
||||
func (re *ReceivedEmail) Body() (string, error) {
|
||||
if re.body == "" {
|
||||
contentType := re.Msg.Header.Get("Content-Type")
|
||||
|
||||
if contentType == "" {
|
||||
body, err := io.ReadAll(re.Msg.Body)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to read email body: %w", err)
|
||||
}
|
||||
re.body = string(body)
|
||||
} else {
|
||||
mediaType, params, err := mime.ParseMediaType(contentType)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("%w: parsing Content-Type %q: %w", errMalformedMessage, contentType, err)
|
||||
}
|
||||
|
||||
if strings.HasPrefix(mediaType, "multipart/") {
|
||||
// Handle any multipart type (alternative, mixed, related, etc.)
|
||||
mr := multipart.NewReader(re.Msg.Body, params["boundary"])
|
||||
var htmlBody string
|
||||
|
||||
for {
|
||||
part, err := mr.NextPart()
|
||||
if err != nil {
|
||||
break
|
||||
}
|
||||
|
||||
partContentType := part.Header.Get("Content-Type")
|
||||
slog.Debug("Processing email part", slog.String("content_type", partContentType))
|
||||
|
||||
// Prefer text/plain, but keep HTML as fallback
|
||||
if strings.HasPrefix(partContentType, "text/plain") {
|
||||
body := new(bytes.Buffer)
|
||||
_, err = body.ReadFrom(part)
|
||||
_ = part.Close()
|
||||
if err != nil {
|
||||
slog.Error("Failed to read part body", slog.String("err", err.Error()))
|
||||
return "", fmt.Errorf("failed to read part body: %w", err)
|
||||
}
|
||||
|
||||
re.body = body.String()
|
||||
break // Found text/plain, use it
|
||||
} else if strings.HasPrefix(partContentType, "text/html") && htmlBody == "" {
|
||||
// Store HTML as fallback if no text/plain found
|
||||
body := new(bytes.Buffer)
|
||||
_, err = body.ReadFrom(part)
|
||||
_ = part.Close()
|
||||
if err == nil {
|
||||
htmlBody = body.String()
|
||||
}
|
||||
} else {
|
||||
_ = part.Close()
|
||||
}
|
||||
}
|
||||
|
||||
// If no text/plain was found, use HTML
|
||||
if re.body == "" && htmlBody != "" {
|
||||
re.body = htmlBody
|
||||
}
|
||||
} else if strings.HasPrefix(mediaType, "text/") {
|
||||
// Handle non-multipart text content (text/plain, text/html, etc.)
|
||||
body, err := io.ReadAll(re.Msg.Body)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to read email body: %w", err)
|
||||
}
|
||||
re.body = string(body)
|
||||
}
|
||||
}
|
||||
if err := re.readBody(); err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
return re.body, nil
|
||||
}
|
||||
|
||||
// FormattedBody returns the body in the requested format. Only an HTML body is
|
||||
// ever rewritten: a message that already arrived as plain text is what the
|
||||
// sender chose to write, and no target is better served by a round trip
|
||||
// through a renderer.
|
||||
func (re *ReceivedEmail) FormattedBody(format BodyFormat) (string, error) {
|
||||
if err := re.readBody(); err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
if format.normalize() == FormatRaw || !re.bodyIsHTML {
|
||||
return re.body, nil
|
||||
}
|
||||
|
||||
// Reformatting is a courtesy to the target, not a condition of delivery.
|
||||
// A body the renderer cannot take apart, or one that renders to nothing
|
||||
// because it was all images and tracking pixels, is forwarded as it
|
||||
// arrived: an empty notification is refused by most targets, which puts
|
||||
// the message into a retry loop it can never leave.
|
||||
rendered, err := renderMarkdown(re.body)
|
||||
if err != nil {
|
||||
slog.Warn("failed to render HTML body, forwarding it unchanged",
|
||||
slog.String("format", string(format)),
|
||||
slog.String("err", err.Error()))
|
||||
|
||||
return re.body, nil
|
||||
}
|
||||
|
||||
if rendered == "" && strings.TrimSpace(re.body) != "" {
|
||||
slog.Warn("HTML body rendered to nothing, forwarding it unchanged",
|
||||
slog.String("format", string(format)),
|
||||
slog.Int("html_length", len(re.body)))
|
||||
|
||||
return re.body, nil
|
||||
}
|
||||
|
||||
return rendered, nil
|
||||
}
|
||||
|
||||
func (re *ReceivedEmail) readBody() error {
|
||||
if re.bodyRead {
|
||||
return nil
|
||||
}
|
||||
|
||||
body, isHTML, err := readEntity(textproto.MIMEHeader(re.Msg.Header), re.Msg.Body, 0)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
re.body, re.bodyIsHTML, re.bodyRead = body, isHTML, true
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// readEntity reads one MIME entity — the message itself or a part of it — and
|
||||
// reports the body it contributes and whether that body is HTML.
|
||||
func readEntity(header textproto.MIMEHeader, body io.Reader, depth int) (string, bool, error) {
|
||||
contentType := header.Get("Content-Type")
|
||||
if contentType == "" {
|
||||
content, err := decodeContent(header, body, "")
|
||||
|
||||
return content, false, err
|
||||
}
|
||||
|
||||
mediaType, params, err := mime.ParseMediaType(contentType)
|
||||
if err != nil {
|
||||
return "", false, fmt.Errorf("%w: parsing Content-Type %q: %w", errMalformedMessage, contentType, err)
|
||||
}
|
||||
|
||||
return readMediaType(header, body, mediaType, params, depth)
|
||||
}
|
||||
|
||||
func readMediaType(
|
||||
header textproto.MIMEHeader,
|
||||
body io.Reader,
|
||||
mediaType string,
|
||||
params map[string]string,
|
||||
depth int,
|
||||
) (string, bool, error) {
|
||||
switch {
|
||||
case strings.HasPrefix(mediaType, "multipart/"):
|
||||
return readMultipart(body, mediaType, params["boundary"], depth)
|
||||
|
||||
case strings.HasPrefix(mediaType, "text/"):
|
||||
content, err := decodeContent(header, body, params["charset"])
|
||||
|
||||
return content, mediaType == "text/html", err
|
||||
|
||||
default:
|
||||
// An image or an application/* payload on its own carries no text to
|
||||
// notify with; the subject still reaches the target as the title.
|
||||
return "", false, nil
|
||||
}
|
||||
}
|
||||
|
||||
func readMultipart(body io.Reader, mediaType, boundary string, depth int) (string, bool, error) {
|
||||
if boundary == "" {
|
||||
return "", false, fmt.Errorf("%w: multipart message without a boundary", errMalformedMessage)
|
||||
}
|
||||
|
||||
if depth >= maxMultipartDepth {
|
||||
slog.Warn("stopped descending into a deeply nested message", slog.Int("depth", depth))
|
||||
|
||||
return "", false, nil
|
||||
}
|
||||
|
||||
// RFC 2046 §5.1.4: the parts of a multipart/alternative are one content in
|
||||
// several forms, so exactly one of them is chosen. The parts of any other
|
||||
// multipart are cumulative (§5.1.3): the first one carrying a body is the
|
||||
// message, and what follows it is footers, signatures and enclosures.
|
||||
alternatives := mediaType == "multipart/alternative"
|
||||
|
||||
var chosen bodySelector
|
||||
|
||||
reader := multipart.NewReader(body, boundary)
|
||||
|
||||
for {
|
||||
part, err := reader.NextPart()
|
||||
if errors.Is(err, io.EOF) {
|
||||
break
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
// A truncated container still hands over the parts that were
|
||||
// readable. Only a message that yielded nothing at all is worth
|
||||
// refusing, since there is nothing left to notify with.
|
||||
slog.Warn("stopped reading a multipart message early", slog.String("err", err.Error()))
|
||||
|
||||
if chosen.empty() {
|
||||
return "", false, fmt.Errorf("%w: reading multipart body: %w", errMalformedMessage, err)
|
||||
}
|
||||
|
||||
break
|
||||
}
|
||||
|
||||
content, isHTML, err := readPart(part, depth)
|
||||
_ = part.Close()
|
||||
|
||||
if err != nil {
|
||||
// One unreadable part is not the whole message; the rest of the
|
||||
// tree may still hold a body.
|
||||
slog.Warn("ignoring an unreadable email part", slog.String("err", err.Error()))
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
if !alternatives {
|
||||
if strings.TrimSpace(content) != "" {
|
||||
return content, isHTML, nil
|
||||
}
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
chosen.offer(content, isHTML)
|
||||
|
||||
if chosen.done() {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
content, isHTML := chosen.result()
|
||||
|
||||
return content, isHTML, nil
|
||||
}
|
||||
|
||||
func readPart(part *multipart.Part, depth int) (string, bool, error) {
|
||||
contentType := part.Header.Get("Content-Type")
|
||||
slog.Debug("Processing email part", slog.String("content_type", contentType))
|
||||
|
||||
// mime.ParseMediaType hands back a usable media type alongside the error
|
||||
// it reports for a malformed parameter — an empty charset, an unquoted
|
||||
// file name — which older mailers emit often enough that dropping those
|
||||
// parts would lose the message body itself.
|
||||
if disposition, _, err := mime.ParseMediaType(part.Header.Get("Content-Disposition")); err == nil ||
|
||||
errors.Is(err, mime.ErrInvalidMediaParameter) {
|
||||
if disposition == "attachment" {
|
||||
// An attached .txt or .html file is something the sender enclosed,
|
||||
// not what they wrote.
|
||||
return "", false, nil
|
||||
}
|
||||
}
|
||||
|
||||
// RFC 2045 §5.2 makes a part without a Content-Type plain US-ASCII text.
|
||||
mediaType, params := "text/plain", map[string]string{}
|
||||
|
||||
if contentType != "" {
|
||||
var err error
|
||||
|
||||
mediaType, params, err = mime.ParseMediaType(contentType)
|
||||
if err != nil && !errors.Is(err, mime.ErrInvalidMediaParameter) {
|
||||
slog.Warn("ignoring email part with an unparseable Content-Type",
|
||||
slog.String("content_type", contentType),
|
||||
slog.String("err", err.Error()))
|
||||
|
||||
return "", false, nil
|
||||
}
|
||||
|
||||
if params == nil {
|
||||
params = map[string]string{}
|
||||
}
|
||||
}
|
||||
|
||||
if !strings.HasPrefix(mediaType, "multipart/") &&
|
||||
mediaType != "text/plain" && mediaType != "text/html" {
|
||||
// text/calendar, an inline image, a signature: nothing to notify with.
|
||||
return "", false, nil
|
||||
}
|
||||
|
||||
return readMediaType(part.Header, part, mediaType, params, depth+1)
|
||||
}
|
||||
|
||||
// decodeContent turns one entity's bytes into a UTF-8 string, undoing the
|
||||
// transfer encoding and the character set the message declared. Without this a
|
||||
// quoted-printable body reaches the target as "=E2=80=99" and a Latin-1 one as
|
||||
// mojibake — and neither survives a trip through an HTML renderer.
|
||||
func decodeContent(header textproto.MIMEHeader, body io.Reader, charsetLabel string) (string, error) {
|
||||
raw, err := io.ReadAll(body)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("reading message body: %w", err)
|
||||
}
|
||||
|
||||
return decodeCharset(decodeTransferEncoding(raw, header.Get("Content-Transfer-Encoding")), charsetLabel), nil
|
||||
}
|
||||
|
||||
// decodeTransferEncoding is deliberately forgiving. A body that will not decode
|
||||
// is still a body: forwarding the bytes as they arrived shows the reader
|
||||
// something, where refusing the message tells the sender to stop retrying and
|
||||
// loses the notification for good.
|
||||
func decodeTransferEncoding(raw []byte, encoding string) []byte {
|
||||
switch strings.ToLower(strings.TrimSpace(encoding)) {
|
||||
case "base64":
|
||||
return decodeBase64(raw)
|
||||
|
||||
case "quoted-printable":
|
||||
// mime/multipart already unwraps this for the parts it hands out and
|
||||
// drops the header with it, so only a single-part message arrives
|
||||
// here still encoded.
|
||||
decoded, err := io.ReadAll(quotedprintable.NewReader(bytes.NewReader(raw)))
|
||||
if err != nil {
|
||||
slog.Warn("failed to decode a quoted-printable body, forwarding what could be decoded",
|
||||
slog.String("err", err.Error()))
|
||||
}
|
||||
|
||||
if len(decoded) == 0 {
|
||||
return raw
|
||||
}
|
||||
|
||||
return decoded
|
||||
|
||||
default:
|
||||
// 7bit, 8bit and binary are the identity encoding; anything else is
|
||||
// something this server has no way to undo.
|
||||
return raw
|
||||
}
|
||||
}
|
||||
|
||||
// decodeBase64 tolerates the two liberties real mailers take with a base64
|
||||
// body: they wrap it across lines, and they leave the padding off the last
|
||||
// group.
|
||||
func decodeBase64(raw []byte) []byte {
|
||||
compact := bytes.Map(func(r rune) rune {
|
||||
if unicode.IsSpace(r) {
|
||||
return -1
|
||||
}
|
||||
|
||||
return r
|
||||
}, raw)
|
||||
|
||||
encoding := base64.StdEncoding
|
||||
if len(compact)%4 != 0 {
|
||||
encoding = base64.RawStdEncoding
|
||||
}
|
||||
|
||||
decoded, err := encoding.DecodeString(string(compact))
|
||||
if err != nil {
|
||||
slog.Warn("failed to decode a base64 body, forwarding what could be decoded",
|
||||
slog.String("err", err.Error()))
|
||||
}
|
||||
|
||||
if len(decoded) == 0 {
|
||||
return raw
|
||||
}
|
||||
|
||||
return decoded
|
||||
}
|
||||
|
||||
func decodeCharset(raw []byte, label string) string {
|
||||
switch strings.ToLower(strings.TrimSpace(label)) {
|
||||
case "", "utf-8", "utf8", "us-ascii", "ascii":
|
||||
return string(raw)
|
||||
}
|
||||
|
||||
encoding, _ := charset.Lookup(label)
|
||||
if encoding == nil {
|
||||
slog.Warn("unknown message charset, forwarding the body undecoded",
|
||||
slog.String("charset", label))
|
||||
|
||||
return string(raw)
|
||||
}
|
||||
|
||||
decoded, err := encoding.NewDecoder().Bytes(raw)
|
||||
if err != nil {
|
||||
slog.Warn("failed to decode message charset, forwarding the body undecoded",
|
||||
slog.String("charset", label),
|
||||
slog.String("err", err.Error()))
|
||||
|
||||
return string(raw)
|
||||
}
|
||||
|
||||
return string(decoded)
|
||||
}
|
||||
|
||||
// bodySelector picks between the forms of a multipart/alternative. Plain text
|
||||
// beats HTML, and the first of each kind beats the rest, which is the order
|
||||
// the message puts its own preference in. A blank part is held rather than
|
||||
// chosen: senders that build the HTML from a template routinely emit an empty
|
||||
// alternative beside it.
|
||||
type bodySelector struct {
|
||||
plain string
|
||||
html string
|
||||
hasPlain bool
|
||||
hasHTML bool
|
||||
}
|
||||
|
||||
func (s *bodySelector) offer(content string, isHTML bool) {
|
||||
if isHTML {
|
||||
if !s.hasHTML || strings.TrimSpace(s.html) == "" {
|
||||
s.html, s.hasHTML = content, true
|
||||
}
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
if !s.hasPlain || strings.TrimSpace(s.plain) == "" {
|
||||
s.plain, s.hasPlain = content, true
|
||||
}
|
||||
}
|
||||
|
||||
func (s *bodySelector) empty() bool {
|
||||
return !s.hasPlain && !s.hasHTML
|
||||
}
|
||||
|
||||
// done reports that nothing later in the container can improve on what has
|
||||
// been found.
|
||||
func (s *bodySelector) done() bool {
|
||||
return s.hasPlain && strings.TrimSpace(s.plain) != ""
|
||||
}
|
||||
|
||||
func (s *bodySelector) result() (string, bool) {
|
||||
if s.done() || !s.hasHTML {
|
||||
return s.plain, false
|
||||
}
|
||||
|
||||
return s.html, true
|
||||
}
|
||||
|
|
|
|||
418
email_test.go
418
email_test.go
|
|
@ -1,9 +1,11 @@
|
|||
package smtp2shoutrrr
|
||||
|
||||
import (
|
||||
"encoding/base64"
|
||||
"net/mail"
|
||||
"strings"
|
||||
"testing"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
|
@ -88,3 +90,419 @@ func TestBodyFallsBackToHTMLPart(t *testing.T) {
|
|||
require.NoError(t, err)
|
||||
require.Equal(t, "<p>html body</p>", body)
|
||||
}
|
||||
|
||||
func TestBodyDecodesQuotedPrintable(t *testing.T) {
|
||||
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||
"Subject: Test",
|
||||
"Content-Type: text/plain; charset=utf-8",
|
||||
"Content-Transfer-Encoding: quoted-printable",
|
||||
"",
|
||||
"caf=C3=A9 =E2=80=94 a very long line that the sender wrapped =",
|
||||
"here",
|
||||
"",
|
||||
}, "\r\n"))}
|
||||
|
||||
body, err := email.Body()
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "café — a very long line that the sender wrapped here\r\n", body)
|
||||
}
|
||||
|
||||
func TestBodyDecodesBase64(t *testing.T) {
|
||||
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||
"Subject: Test",
|
||||
"Content-Type: text/plain; charset=utf-8",
|
||||
"Content-Transfer-Encoding: base64",
|
||||
"",
|
||||
base64.StdEncoding.EncodeToString([]byte("café — encoded")),
|
||||
"",
|
||||
}, "\r\n"))}
|
||||
|
||||
body, err := email.Body()
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "café — encoded", body)
|
||||
}
|
||||
|
||||
// Windows-1252 and Latin-1 are still what a good deal of automated mail is
|
||||
// written in, and their bytes are not valid UTF-8.
|
||||
func TestBodyDecodesNonUTF8Charset(t *testing.T) {
|
||||
email := ReceivedEmail{Msg: readMessage(t,
|
||||
"Subject: Test\r\nContent-Type: text/plain; charset=iso-8859-1\r\n\r\ncaf\xe9\r\n")}
|
||||
|
||||
body, err := email.Body()
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "café\r\n", body)
|
||||
require.True(t, utf8.ValidString(body))
|
||||
}
|
||||
|
||||
func TestBodyUnknownCharsetIsForwardedUndecoded(t *testing.T) {
|
||||
email := ReceivedEmail{Msg: readMessage(t,
|
||||
"Subject: Test\r\nContent-Type: text/plain; charset=not-a-charset\r\n\r\nbody\r\n")}
|
||||
|
||||
body, err := email.Body()
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "body\r\n", body)
|
||||
}
|
||||
|
||||
// A forwarded message is a multipart/mixed wrapping the multipart/alternative
|
||||
// that holds what the sender actually wrote. Stopping at the outer container
|
||||
// left the notification empty.
|
||||
func TestBodyDescendsIntoNestedMultipart(t *testing.T) {
|
||||
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||
"Subject: Test",
|
||||
`Content-Type: multipart/mixed; boundary="outer"`,
|
||||
"",
|
||||
"--outer",
|
||||
`Content-Type: multipart/alternative; boundary="inner"`,
|
||||
"",
|
||||
"--inner",
|
||||
"Content-Type: text/html; charset=utf-8",
|
||||
"",
|
||||
"<p>html body</p>",
|
||||
"--inner",
|
||||
"Content-Type: text/plain; charset=utf-8",
|
||||
"",
|
||||
"plain body",
|
||||
"--inner--",
|
||||
"--outer--",
|
||||
"",
|
||||
}, "\r\n"))}
|
||||
|
||||
body, err := email.Body()
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "plain body", body)
|
||||
}
|
||||
|
||||
func TestBodySkipsAttachments(t *testing.T) {
|
||||
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||
"Subject: Test",
|
||||
`Content-Type: multipart/mixed; boundary="b"`,
|
||||
"",
|
||||
"--b",
|
||||
"Content-Type: text/plain; charset=utf-8",
|
||||
`Content-Disposition: attachment; filename="notes.txt"`,
|
||||
"",
|
||||
"an enclosed file, not the message",
|
||||
"--b",
|
||||
"Content-Type: text/plain; charset=utf-8",
|
||||
"",
|
||||
"the message itself",
|
||||
"--b--",
|
||||
"",
|
||||
}, "\r\n"))}
|
||||
|
||||
body, err := email.Body()
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "the message itself", body)
|
||||
}
|
||||
|
||||
// Senders that build the HTML alternative from a template routinely emit an
|
||||
// empty text/plain beside it.
|
||||
func TestBodyPrefersHTMLOverAnEmptyPlainTextPart(t *testing.T) {
|
||||
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||
"Subject: Test",
|
||||
`Content-Type: multipart/alternative; boundary="b"`,
|
||||
"",
|
||||
"--b",
|
||||
"Content-Type: text/plain; charset=utf-8",
|
||||
"",
|
||||
" ",
|
||||
"--b",
|
||||
"Content-Type: text/html; charset=utf-8",
|
||||
"",
|
||||
"<p>html body</p>",
|
||||
"--b--",
|
||||
"",
|
||||
}, "\r\n"))}
|
||||
|
||||
body, err := email.Body()
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "<p>html body</p>", body)
|
||||
}
|
||||
|
||||
func TestBodyRejectsMultipartWithoutBoundary(t *testing.T) {
|
||||
email := ReceivedEmail{Msg: readMessage(t,
|
||||
"Subject: Test\r\nContent-Type: multipart/alternative\r\n\r\nbody\r\n")}
|
||||
|
||||
_, err := email.Body()
|
||||
require.ErrorIs(t, err, errMalformedMessage)
|
||||
}
|
||||
|
||||
func TestFormattedBodyConvertsHTML(t *testing.T) {
|
||||
raw := strings.Join([]string{
|
||||
"Subject: Test",
|
||||
"Content-Type: text/html; charset=utf-8",
|
||||
"Content-Transfer-Encoding: quoted-printable",
|
||||
"",
|
||||
"<p>Build <b>failed</b>: <a href=3D\"https://ci.example.com/42\">run 42</a></p>",
|
||||
"",
|
||||
}, "\r\n")
|
||||
|
||||
for format, want := range map[BodyFormat]string{
|
||||
FormatMarkdown: "Build **failed**: [run 42](https://ci.example.com/42)",
|
||||
FormatRaw: "<p>Build <b>failed</b>: <a href=\"https://ci.example.com/42\">run 42</a></p>\r\n",
|
||||
} {
|
||||
t.Run(string(format), func(t *testing.T) {
|
||||
email := ReceivedEmail{Msg: readMessage(t, raw)}
|
||||
|
||||
body, err := email.FormattedBody(format)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, want, body)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A recipient asking for Markdown is asking for HTML to stop reaching it, not
|
||||
// for the plain text a sender wrote to be run through a renderer.
|
||||
func TestFormattedBodyLeavesPlainTextAlone(t *testing.T) {
|
||||
raw := "Subject: Test\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n2 * 3 = 6 <see notes>\r\n"
|
||||
|
||||
for _, format := range []BodyFormat{FormatRaw, FormatMarkdown} {
|
||||
t.Run(string(format), func(t *testing.T) {
|
||||
email := ReceivedEmail{Msg: readMessage(t, raw)}
|
||||
|
||||
body, err := email.FormattedBody(format)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "2 * 3 = 6 <see notes>\r\n", body)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The zero value of the option means the same as raw, so a recipient built in
|
||||
// code rather than loaded from a file still forwards what it received.
|
||||
func TestFormattedBodyTreatsUnsetFormatAsRaw(t *testing.T) {
|
||||
email := ReceivedEmail{Msg: readMessage(t,
|
||||
"Subject: Test\r\nContent-Type: text/html\r\n\r\n<p>html body</p>\r\n")}
|
||||
|
||||
body, err := email.FormattedBody("")
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "<p>html body</p>\r\n", body)
|
||||
}
|
||||
|
||||
// Reformatting is a courtesy, so a body the renderer cannot use is still
|
||||
// forwarded rather than costing the reader the notification.
|
||||
func TestFormattedBodyFallsBackToTheRawHTML(t *testing.T) {
|
||||
// The parser refuses a document with more than 512 elements open at once.
|
||||
tooDeep := strings.Repeat("<div>", 600) + "deep" + strings.Repeat("</div>", 600)
|
||||
|
||||
for name, source := range map[string]string{
|
||||
"a body the parser refuses": tooDeep,
|
||||
// Otherwise shoutrrr is handed "", which Mattermost and Discord
|
||||
// reject, so every target fails and the sender retries the same
|
||||
// message forever.
|
||||
"a body that renders to nothing": `<img src="https://x.example/pixel.gif">`,
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
email := ReceivedEmail{Msg: readMessage(t,
|
||||
"Subject: Test\r\nContent-Type: text/html\r\n\r\n"+source)}
|
||||
|
||||
body, err := email.FormattedBody(FormatMarkdown)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, source, body)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestBodyIsReadOnlyOnce(t *testing.T) {
|
||||
email := ReceivedEmail{Msg: readMessage(t,
|
||||
"Subject: Test\r\nContent-Type: text/html\r\n\r\n<p>body</p>\r\n")}
|
||||
|
||||
first, err := email.FormattedBody(FormatMarkdown)
|
||||
require.NoError(t, err)
|
||||
|
||||
second, err := email.FormattedBody(FormatMarkdown)
|
||||
require.NoError(t, err)
|
||||
|
||||
require.Equal(t, "body", first)
|
||||
require.Equal(t, first, second, "the message body is consumed as it is read")
|
||||
}
|
||||
|
||||
// Regression: an unreadable transfer encoding used to be reported as a
|
||||
// permanent 550, which tells the sender to stop retrying and loses a
|
||||
// notification that main delivered. Real mailers emit unpadded base64.
|
||||
func TestBodyToleratesAnUndecodableTransferEncoding(t *testing.T) {
|
||||
for name, tc := range map[string]struct{ raw, want string }{
|
||||
"unpadded base64": {
|
||||
raw: base64.RawStdEncoding.EncodeToString([]byte("hello world")),
|
||||
want: "hello world",
|
||||
},
|
||||
"base64 that is not base64 at all": {
|
||||
raw: "!!!! not base64 !!!!",
|
||||
want: "!!!! not base64 !!!!",
|
||||
},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
email := ReceivedEmail{Msg: readMessage(t,
|
||||
"Subject: Test\r\nContent-Type: text/plain\r\n"+
|
||||
"Content-Transfer-Encoding: base64\r\n\r\n"+tc.raw+"\r\n")}
|
||||
|
||||
body, err := email.Body()
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, tc.want, strings.TrimSpace(body))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Regression: one part that would not decode used to fail the whole message,
|
||||
// even with a perfectly good alternative already in hand.
|
||||
func TestBodyKeepsAGoodPartBesideAnUnreadableOne(t *testing.T) {
|
||||
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||
"Subject: Test",
|
||||
`Content-Type: multipart/alternative; boundary="b"`,
|
||||
"",
|
||||
"--b",
|
||||
"Content-Type: text/html",
|
||||
"Content-Transfer-Encoding: base64",
|
||||
"",
|
||||
"!!!!not base64!!!!",
|
||||
"--b",
|
||||
"Content-Type: text/plain",
|
||||
"",
|
||||
"the real body",
|
||||
"--b--",
|
||||
"",
|
||||
}, "\r\n"))}
|
||||
|
||||
body, err := email.Body()
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "the real body", body)
|
||||
}
|
||||
|
||||
// Regression: mime.ParseMediaType returns a usable media type alongside the
|
||||
// error it reports for a malformed parameter. Discarding both dropped the part
|
||||
// that held the message and delivered an empty notification with a 250.
|
||||
func TestBodyReadsPartsWithAMalformedContentType(t *testing.T) {
|
||||
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||
"Subject: Test",
|
||||
`Content-Type: multipart/alternative; boundary="b"`,
|
||||
"",
|
||||
"--b",
|
||||
"Content-Type: text/plain; charset=",
|
||||
"",
|
||||
"still text",
|
||||
"--b--",
|
||||
"",
|
||||
}, "\r\n"))}
|
||||
|
||||
body, err := email.Body()
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "still text", body)
|
||||
}
|
||||
|
||||
// The same fault the other way round: the attachment guard used to fail open
|
||||
// on the unquoted file names older mailers emit, so the enclosure won.
|
||||
func TestBodySkipsAttachmentsWithAMalformedDisposition(t *testing.T) {
|
||||
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||
"Subject: Test",
|
||||
`Content-Type: multipart/mixed; boundary="b"`,
|
||||
"",
|
||||
"--b",
|
||||
"Content-Type: text/plain",
|
||||
"Content-Disposition: attachment; filename=log file.txt",
|
||||
"",
|
||||
"an enclosed file, not the message",
|
||||
"--b",
|
||||
"Content-Type: text/plain",
|
||||
"",
|
||||
"the message itself",
|
||||
"--b--",
|
||||
"",
|
||||
}, "\r\n"))}
|
||||
|
||||
body, err := email.Body()
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "the message itself", body)
|
||||
}
|
||||
|
||||
// RFC 2046 §5.1.3: the parts of a multipart/mixed are cumulative, so the first
|
||||
// one carrying a body is the message and the rest are footers and enclosures.
|
||||
// Applying "plain beats HTML" across them handed list mail its unsubscribe
|
||||
// footer as the notification.
|
||||
func TestBodyPrefersTheFirstPartOfACumulativeMultipart(t *testing.T) {
|
||||
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||
"Subject: Test",
|
||||
`Content-Type: multipart/mixed; boundary="b"`,
|
||||
"",
|
||||
"--b",
|
||||
"Content-Type: text/html",
|
||||
"",
|
||||
"<p>the newsletter</p>",
|
||||
"--b",
|
||||
"Content-Type: text/plain",
|
||||
"",
|
||||
"You are receiving this because you subscribed.",
|
||||
"--b--",
|
||||
"",
|
||||
}, "\r\n"))}
|
||||
|
||||
body, err := email.FormattedBody(FormatMarkdown)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "the newsletter", body,
|
||||
"the HTML body is the message, and asking for Markdown must reach it")
|
||||
}
|
||||
|
||||
func TestBodyPassesOverAnEmptyHTMLAlternative(t *testing.T) {
|
||||
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||
"Subject: Test",
|
||||
`Content-Type: multipart/alternative; boundary="b"`,
|
||||
"",
|
||||
"--b",
|
||||
"Content-Type: text/html",
|
||||
"",
|
||||
" ",
|
||||
"--b",
|
||||
"Content-Type: text/html",
|
||||
"",
|
||||
"<p>the real one</p>",
|
||||
"--b--",
|
||||
"",
|
||||
}, "\r\n"))}
|
||||
|
||||
body, err := email.Body()
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "<p>the real one</p>", body)
|
||||
}
|
||||
|
||||
// A container this server cannot finish reading still hands over the parts it
|
||||
// read. Only a message that yielded nothing at all is refused, since there is
|
||||
// then nothing left to notify with and the same bytes fail the same way on
|
||||
// every redelivery.
|
||||
func TestBodyKeepsWhatItReadFromABrokenContainer(t *testing.T) {
|
||||
t.Run("a good part before the break is delivered", func(t *testing.T) {
|
||||
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||
"Subject: Test",
|
||||
`Content-Type: multipart/mixed; boundary="b"`,
|
||||
"",
|
||||
"--b",
|
||||
"Content-Type: text/plain",
|
||||
"",
|
||||
"good body",
|
||||
"--b",
|
||||
"this is not a header",
|
||||
"",
|
||||
"x",
|
||||
"--b--",
|
||||
"",
|
||||
}, "\r\n"))}
|
||||
|
||||
body, err := email.Body()
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "good body", body)
|
||||
})
|
||||
|
||||
t.Run("a break before anything readable is permanent", func(t *testing.T) {
|
||||
email := ReceivedEmail{Msg: readMessage(t, strings.Join([]string{
|
||||
"Subject: Test",
|
||||
`Content-Type: multipart/alternative; boundary="b"`,
|
||||
"",
|
||||
"--b",
|
||||
"this is not a header",
|
||||
"",
|
||||
"body",
|
||||
"--b--",
|
||||
"",
|
||||
}, "\r\n"))}
|
||||
|
||||
_, err := email.Body()
|
||||
require.ErrorIs(t, err, errMalformedMessage)
|
||||
})
|
||||
}
|
||||
|
|
|
|||
49
format.go
Normal file
49
format.go
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
package smtp2shoutrrr
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// BodyFormat names the representation a recipient's targets want the message
|
||||
// body in. Only an HTML body is ever rewritten: a message that already arrived
|
||||
// as plain text is forwarded untouched whatever the recipient asked for.
|
||||
type BodyFormat string
|
||||
|
||||
const (
|
||||
// FormatRaw forwards the body exactly as the message carried it.
|
||||
FormatRaw BodyFormat = "raw"
|
||||
// FormatMarkdown renders an HTML body as Markdown. There is deliberately
|
||||
// no plain-text format beside it: Markdown reads as plain text wherever
|
||||
// nothing renders it, so a second conversion would only be a worse copy
|
||||
// of this one.
|
||||
FormatMarkdown BodyFormat = "markdown"
|
||||
)
|
||||
|
||||
var bodyFormats = []BodyFormat{FormatRaw, FormatMarkdown}
|
||||
|
||||
// normalize maps an unset Format to the one that changes nothing, and accepts
|
||||
// the casing a hand-written configuration file is likely to use.
|
||||
func (f BodyFormat) normalize() BodyFormat {
|
||||
normalized := BodyFormat(strings.ToLower(strings.TrimSpace(string(f))))
|
||||
if normalized == "" {
|
||||
return FormatRaw
|
||||
}
|
||||
|
||||
return normalized
|
||||
}
|
||||
|
||||
// valid normalizes first, so a Config assembled in Go rather than loaded from
|
||||
// a file does not fail validation on a Format nobody set.
|
||||
func (f BodyFormat) valid() bool {
|
||||
return slices.Contains(bodyFormats, f.normalize())
|
||||
}
|
||||
|
||||
func formatNames() string {
|
||||
names := make([]string, 0, len(bodyFormats))
|
||||
for _, format := range bodyFormats {
|
||||
names = append(names, string(format))
|
||||
}
|
||||
|
||||
return strings.Join(names, ", ")
|
||||
}
|
||||
5
go.mod
5
go.mod
|
|
@ -5,15 +5,18 @@ go 1.27
|
|||
toolchain go1.27.1
|
||||
|
||||
require (
|
||||
github.com/JohannesKaufmann/html-to-markdown/v2 v2.5.2
|
||||
github.com/containrrr/shoutrrr v0.8.0
|
||||
github.com/emersion/go-sasl v0.0.0-20241020182733-b788ff22d5a6
|
||||
github.com/emersion/go-smtp v0.25.0
|
||||
github.com/pelletier/go-toml/v2 v2.4.3
|
||||
github.com/stretchr/testify v1.12.1
|
||||
golang.org/x/crypto/x509roots/fallback v0.0.0-20260902180247-86efde54dc70
|
||||
golang.org/x/net v0.58.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/JohannesKaufmann/dom v0.3.1 // indirect
|
||||
github.com/fatih/color v1.19.0 // indirect
|
||||
github.com/go-logr/logr v1.4.4 // indirect
|
||||
github.com/golang/protobuf v1.5.4 // indirect
|
||||
|
|
@ -21,7 +24,7 @@ require (
|
|||
github.com/mattn/go-colorable v0.1.15 // indirect
|
||||
github.com/mattn/go-isatty v0.0.24 // indirect
|
||||
go.yaml.in/yaml/v3 v3.0.5 // indirect
|
||||
golang.org/x/net v0.58.0 // indirect
|
||||
golang.org/x/sys v0.47.0 // indirect
|
||||
golang.org/x/text v0.41.0 // indirect
|
||||
golang.org/x/tools v0.49.0 // indirect
|
||||
)
|
||||
|
|
|
|||
12
go.sum
12
go.sum
|
|
@ -1,3 +1,7 @@
|
|||
github.com/JohannesKaufmann/dom v0.3.1 h1:J16l9JAHWgkFPR3VIPbQ1gvS0cWab6laK1q7PFL3qh0=
|
||||
github.com/JohannesKaufmann/dom v0.3.1/go.mod h1:BZPkf8ZeYrBgABjwJn9iiKt8aiCtkxpHkevms+Yp2DE=
|
||||
github.com/JohannesKaufmann/html-to-markdown/v2 v2.5.2 h1:XFJZFWESIWlUEHHjzBuv8RvrtCWnSGlimEX17ysSDb8=
|
||||
github.com/JohannesKaufmann/html-to-markdown/v2 v2.5.2/go.mod h1:BHWO8lJzttJLqwuV8Rb1B3OG2OSzLbssZDI1FRg2eAA=
|
||||
github.com/containrrr/shoutrrr v0.8.0 h1:mfG2ATzIS7NR2Ec6XL+xyoHzN97H8WPjir8aYzJUSec=
|
||||
github.com/containrrr/shoutrrr v0.8.0/go.mod h1:ioyQAyu1LJY6sILuNyKaQaw+9Ttik5QePU8atnAdO2o=
|
||||
github.com/emersion/go-sasl v0.0.0-20241020182733-b788ff22d5a6 h1:oP4q0fw+fOSWn3DfFi4EXdT+B+gTtzx8GC9xsc26Znk=
|
||||
|
|
@ -28,8 +32,16 @@ github.com/onsi/gomega v1.27.6 h1:ENqfyGeS5AX/rlXDd/ETokDz93u0YufY1Pgxuy/PvWE=
|
|||
github.com/onsi/gomega v1.27.6/go.mod h1:PIQNjfQwkP3aQAH7lf7j87O/5FiNr+ZR8+ipb+qQlhg=
|
||||
github.com/pelletier/go-toml/v2 v2.4.3 h1:GTRvJQutkOSftxIFD5xw9aepkYNuPWmVJpffdDPYVpY=
|
||||
github.com/pelletier/go-toml/v2 v2.4.3/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/sebdah/goldie/v2 v2.8.0 h1:dZb9wR8q5++oplmEiJT+U/5KyotVD+HNGCAc5gNr8rc=
|
||||
github.com/sebdah/goldie/v2 v2.8.0/go.mod h1:oZ9fp0+se1eapSRjfYbsV/0Hqhbuu3bJVvKI/NNtssI=
|
||||
github.com/sergi/go-diff v1.4.0 h1:n/SP9D5ad1fORl+llWyN+D6qoUETXNZARKjyY2/KVCw=
|
||||
github.com/sergi/go-diff v1.4.0/go.mod h1:A0bzQcvG0E7Rwjx0REVgAGH58e96+X0MeOfepqsbeW4=
|
||||
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
|
||||
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
|
||||
github.com/yuin/goldmark v1.8.2 h1:kEGpgqJXdgbkhcOgBxkC0X0PmoPG1ZyoZ117rDVp4zE=
|
||||
github.com/yuin/goldmark v1.8.2/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg=
|
||||
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
|
||||
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
|
||||
golang.org/x/crypto/x509roots/fallback v0.0.0-20260902180247-86efde54dc70 h1:VwViOGcd7C8/Gs18efVlMxvUCS0un6KRKBPFEBvpUXg=
|
||||
|
|
|
|||
400
html.go
Normal file
400
html.go
Normal file
|
|
@ -0,0 +1,400 @@
|
|||
package smtp2shoutrrr
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/JohannesKaufmann/html-to-markdown/v2/converter"
|
||||
"github.com/JohannesKaufmann/html-to-markdown/v2/plugin/base"
|
||||
"github.com/JohannesKaufmann/html-to-markdown/v2/plugin/commonmark"
|
||||
"github.com/JohannesKaufmann/html-to-markdown/v2/plugin/strikethrough"
|
||||
"golang.org/x/net/html"
|
||||
"golang.org/x/net/html/atom"
|
||||
)
|
||||
|
||||
const (
|
||||
// maxNestingDepth bounds the elements that give every line they contain a
|
||||
// prefix. The converter re-emits that prefix per line and per level, so
|
||||
// depth multiplies against line count: a message at the server's own 1 MB
|
||||
// limit, nested 250 quotes deep, renders to 131 MB and takes minutes.
|
||||
// Real mail quotes a handful of levels and indents fewer.
|
||||
maxNestingDepth = 6
|
||||
|
||||
// maxRenderedBytes caps what is forwarded. No chat target displays this
|
||||
// much, and a notification is not the place to find out.
|
||||
maxRenderedBytes = 64 << 10
|
||||
|
||||
// truncationMarker tells the reader the message goes on, rather than
|
||||
// letting the cap end it mid-sentence and look like the whole of it.
|
||||
truncationMarker = "…"
|
||||
)
|
||||
|
||||
// renderMarkdown rewrites an HTML body as Markdown.
|
||||
//
|
||||
// The conversion itself belongs to html-to-markdown, which knows CommonMark —
|
||||
// escaping, delimiter runs, fencing a code block past the backticks inside it.
|
||||
// What it has no opinion about is mail, since it is written for documents: a
|
||||
// document has no preheader written for an inbox list, no tracking pixel, no
|
||||
// cid: attachment, and lays nothing out in tables. That is what prepare does
|
||||
// to the tree first, so the converter only ever sees what a notification
|
||||
// should carry.
|
||||
func renderMarkdown(source string) (string, error) {
|
||||
document, err := html.Parse(strings.NewReader(source))
|
||||
if err != nil {
|
||||
// The parser recovers from any markup it can hold, so this is a
|
||||
// document too deeply nested for it rather than an invalid one.
|
||||
return "", fmt.Errorf("parsing HTML body: %w", err)
|
||||
}
|
||||
|
||||
prepare(document, 0)
|
||||
|
||||
rendered, err := newConverter().ConvertNode(document)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("converting HTML body: %w", err)
|
||||
}
|
||||
|
||||
return truncate(strings.TrimSpace(string(rendered))), nil
|
||||
}
|
||||
|
||||
// newConverter builds a converter per message, which is what the library's own
|
||||
// entry points do: a Converter carries the error of the conversion it is
|
||||
// running, and go-smtp serves every connection on its own goroutine.
|
||||
func newConverter() *converter.Converter {
|
||||
return converter.NewConverter(
|
||||
converter.WithPlugins(
|
||||
base.NewBasePlugin(),
|
||||
commonmark.NewCommonmarkPlugin(
|
||||
// The default is "* * *", which reads as a stray line of
|
||||
// asterisks anywhere the Markdown is not rendered.
|
||||
commonmark.WithHorizontalRule("---"),
|
||||
),
|
||||
strikethrough.NewStrikethroughPlugin(),
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
func prepare(node *html.Node, depth int) {
|
||||
child := node.FirstChild
|
||||
|
||||
for child != nil {
|
||||
next := child.NextSibling
|
||||
|
||||
switch child.Type {
|
||||
case html.CommentNode:
|
||||
node.RemoveChild(child)
|
||||
|
||||
case html.TextNode:
|
||||
child.Data = sanitizeText(child.Data)
|
||||
|
||||
case html.ElementNode:
|
||||
if unwrapped := prepareElement(node, child, depth); unwrapped != nil {
|
||||
// The element was replaced by its own children, which have
|
||||
// not been looked at yet.
|
||||
next = unwrapped
|
||||
}
|
||||
}
|
||||
|
||||
child = next
|
||||
}
|
||||
}
|
||||
|
||||
func prepareElement(parent, node *html.Node, depth int) *html.Node {
|
||||
if isHidden(node) {
|
||||
// A template opens with a hidden block holding the preview line the
|
||||
// inbox shows, which is written for the list view and reads as noise
|
||||
// anywhere else.
|
||||
parent.RemoveChild(node)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
switch node.DataAtom {
|
||||
case atom.Img:
|
||||
return prepareImage(parent, node)
|
||||
|
||||
case atom.A:
|
||||
href := sanitizeURL(attrValue(node, "href"))
|
||||
if !isUsableURL(href) {
|
||||
// In-page anchors, script handlers and the message's own inline
|
||||
// attachments give a notification's reader nothing to open, but
|
||||
// the text of the link is still part of the message.
|
||||
return unwrap(parent, node)
|
||||
}
|
||||
|
||||
setAttr(node, "href", href)
|
||||
prepare(node, depth)
|
||||
|
||||
if !hasContent(node) {
|
||||
// A link wrapped around a tracking pixel, or around the spacer
|
||||
// that mail templates use for one, would render as "[](url)" —
|
||||
// invisible to the reader. Its destination is the only thing left
|
||||
// worth forwarding.
|
||||
node.AppendChild(&html.Node{Type: html.TextNode, Data: href})
|
||||
}
|
||||
|
||||
return nil
|
||||
|
||||
case atom.Td, atom.Th:
|
||||
// Mail is laid out in tables far more often than it tabulates
|
||||
// anything, and a converter with no table rules runs the cells of a
|
||||
// row together: "Total4Failed0".
|
||||
parent.InsertBefore(&html.Node{Type: html.TextNode, Data: " "}, node)
|
||||
|
||||
case atom.Tr:
|
||||
if hasElementSibling(node) {
|
||||
parent.InsertBefore(&html.Node{
|
||||
Type: html.ElementNode, DataAtom: atom.Br, Data: "br",
|
||||
}, node)
|
||||
}
|
||||
|
||||
case atom.Blockquote, atom.Ul, atom.Ol:
|
||||
depth++
|
||||
if depth > maxNestingDepth {
|
||||
return unwrap(parent, node)
|
||||
}
|
||||
}
|
||||
|
||||
prepare(node, depth)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// prepareImage keeps of an image only what a notification can use. Tracking
|
||||
// pixels, spacers and sliced-up banners carry no alt text and go entirely; an
|
||||
// inline attachment has no address the reader can fetch, so its alt text
|
||||
// stays behind as ordinary words.
|
||||
func prepareImage(parent, node *html.Node) *html.Node {
|
||||
alt := collapseSpace(sanitizeText(attrValue(node, "alt")))
|
||||
source := sanitizeURL(attrValue(node, "src"))
|
||||
|
||||
switch {
|
||||
case alt == "":
|
||||
parent.RemoveChild(node)
|
||||
case !isUsableURL(source):
|
||||
parent.InsertBefore(&html.Node{Type: html.TextNode, Data: alt}, node)
|
||||
parent.RemoveChild(node)
|
||||
default:
|
||||
setAttr(node, "alt", alt)
|
||||
setAttr(node, "src", source)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// unwrap replaces a node with its own children and returns the first of them.
|
||||
func unwrap(parent, node *html.Node) *html.Node {
|
||||
first := node.FirstChild
|
||||
|
||||
for child := node.FirstChild; child != nil; child = node.FirstChild {
|
||||
node.RemoveChild(child)
|
||||
parent.InsertBefore(child, node)
|
||||
}
|
||||
|
||||
parent.RemoveChild(node)
|
||||
|
||||
return first
|
||||
}
|
||||
|
||||
// hasContent reports whether anything left under a node would show. It runs
|
||||
// after the subtree has been prepared, so an element that survived that is one
|
||||
// the reader will see.
|
||||
func hasContent(node *html.Node) bool {
|
||||
for child := node.FirstChild; child != nil; child = child.NextSibling {
|
||||
switch child.Type {
|
||||
case html.ElementNode:
|
||||
if child.DataAtom == atom.Img || hasContent(child) {
|
||||
return true
|
||||
}
|
||||
case html.TextNode:
|
||||
if strings.TrimSpace(child.Data) != "" {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
func hasElementSibling(node *html.Node) bool {
|
||||
for sibling := node.PrevSibling; sibling != nil; sibling = sibling.PrevSibling {
|
||||
if sibling.Type == html.ElementNode {
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
func truncate(rendered string) string {
|
||||
if len(rendered) <= maxRenderedBytes {
|
||||
return rendered
|
||||
}
|
||||
|
||||
cut := maxRenderedBytes
|
||||
for cut > 0 && !utf8.RuneStart(rendered[cut]) {
|
||||
cut--
|
||||
}
|
||||
|
||||
return strings.TrimRight(rendered[:cut], " \t\n") + "\n" + truncationMarker
|
||||
}
|
||||
|
||||
// isUsableURL rejects the destinations a notification's reader cannot act on:
|
||||
// in-page anchors, inline data, script handlers and the message's own inline
|
||||
// attachments.
|
||||
func isUsableURL(raw string) bool {
|
||||
if raw == "" || strings.HasPrefix(raw, "#") {
|
||||
return false
|
||||
}
|
||||
|
||||
scheme, _, found := strings.Cut(raw, ":")
|
||||
if !found || !isScheme(scheme) {
|
||||
return true
|
||||
}
|
||||
|
||||
switch strings.ToLower(scheme) {
|
||||
case "javascript", "data", "cid", "about", "blob":
|
||||
return false
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
func isScheme(s string) bool {
|
||||
if s == "" || !unicode.IsLetter(rune(s[0])) {
|
||||
return false
|
||||
}
|
||||
|
||||
return strings.IndexFunc(s, func(r rune) bool {
|
||||
return !unicode.IsLetter(r) && !unicode.IsDigit(r) &&
|
||||
r != '+' && r != '-' && r != '.'
|
||||
}) < 0
|
||||
}
|
||||
|
||||
// sanitizeURL applies to a URL attribute what a browser applies before it
|
||||
// fetches one: the tabs and line breaks that may sit inside the value are
|
||||
// removed. Left in, they end the line the notification is on, which is a place
|
||||
// to write a sentence the reader will take for the sender's.
|
||||
func sanitizeURL(raw string) string {
|
||||
return strings.Map(func(r rune) rune {
|
||||
if r == ' ' {
|
||||
return ' '
|
||||
}
|
||||
|
||||
if unicode.IsSpace(r) || unicode.IsControl(r) {
|
||||
return -1
|
||||
}
|
||||
|
||||
return r
|
||||
}, strings.TrimSpace(raw))
|
||||
}
|
||||
|
||||
// collapseSpace squeezes the runs of whitespace an attribute value may hold
|
||||
// into single spaces, for the same reason: alt text spans lines in the source
|
||||
// and must not span them in the message.
|
||||
func collapseSpace(s string) string {
|
||||
return strings.Join(strings.Fields(s), " ")
|
||||
}
|
||||
|
||||
// sanitizeText turns the characters mail uses for layout into ones a
|
||||
// notification can show: the non-breaking spaces that hold table cells apart
|
||||
// become ordinary spaces, and the zero-width padding that hides a preheader
|
||||
// from the inbox preview is dropped rather than forwarded invisibly.
|
||||
func sanitizeText(s string) string {
|
||||
return strings.Map(func(r rune) rune {
|
||||
switch r {
|
||||
case zeroWidthSpace, zeroWidthNonJoiner, zeroWidthJoiner,
|
||||
wordJoiner, byteOrderMark, softHyphen:
|
||||
return -1
|
||||
case noBreakSpace, narrowNoBreakSpace, figureSpace:
|
||||
return ' '
|
||||
}
|
||||
|
||||
if unicode.IsSpace(r) {
|
||||
return r
|
||||
}
|
||||
|
||||
// Control characters have no rendering of their own and only muddle
|
||||
// the payloads the targets are sent as.
|
||||
if unicode.IsControl(r) {
|
||||
return -1
|
||||
}
|
||||
|
||||
return r
|
||||
}, s)
|
||||
}
|
||||
|
||||
// The characters mail uses for layout rather than for words: padding that
|
||||
// hides a preheader from the inbox preview, and the spaces that hold table
|
||||
// cells apart without letting them wrap.
|
||||
const (
|
||||
softHyphen = '\u00ad'
|
||||
noBreakSpace = '\u00a0'
|
||||
zeroWidthSpace = '\u200b'
|
||||
zeroWidthNonJoiner = '\u200c'
|
||||
zeroWidthJoiner = '\u200d'
|
||||
narrowNoBreakSpace = '\u202f'
|
||||
wordJoiner = '\u2060'
|
||||
figureSpace = '\u2007'
|
||||
byteOrderMark = '\ufeff'
|
||||
)
|
||||
|
||||
// isHidden reports whether a browser would leave the element out of the page.
|
||||
func isHidden(node *html.Node) bool {
|
||||
for _, attr := range node.Attr {
|
||||
switch attr.Key {
|
||||
case "hidden":
|
||||
return true
|
||||
case "style":
|
||||
if isHidingStyle(spaceStripper.Replace(strings.ToLower(attr.Val))) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
func isHidingStyle(style string) bool {
|
||||
for _, declaration := range []string{"display:none", "visibility:hidden", "mso-hide:all"} {
|
||||
if strings.Contains(style, declaration) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
// font-size:0 hides a preheader too, but only when the zero is the whole
|
||||
// value: font-size:0.9em is an ordinary line of text.
|
||||
if index := strings.Index(style, "font-size:0"); index >= 0 {
|
||||
rest := strings.TrimPrefix(style[index+len("font-size:0"):], "px")
|
||||
rest = strings.TrimPrefix(rest, "pt")
|
||||
|
||||
return rest == "" || strings.HasPrefix(rest, ";")
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
var spaceStripper = strings.NewReplacer(" ", "", "\t", "", "\n", "", "\r", "")
|
||||
|
||||
func attrValue(node *html.Node, key string) string {
|
||||
for _, attr := range node.Attr {
|
||||
if attr.Key == key {
|
||||
return attr.Val
|
||||
}
|
||||
}
|
||||
|
||||
return ""
|
||||
}
|
||||
|
||||
func setAttr(node *html.Node, key, value string) {
|
||||
for i := range node.Attr {
|
||||
if node.Attr[i].Key == key {
|
||||
node.Attr[i].Val = value
|
||||
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
node.Attr = append(node.Attr, html.Attribute{Key: key, Val: value})
|
||||
}
|
||||
248
html_test.go
Normal file
248
html_test.go
Normal file
|
|
@ -0,0 +1,248 @@
|
|||
package smtp2shoutrrr
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func render(t *testing.T, source string) string {
|
||||
t.Helper()
|
||||
|
||||
out, err := renderMarkdown(source)
|
||||
require.NoError(t, err)
|
||||
|
||||
return out
|
||||
}
|
||||
|
||||
// The conversion itself belongs to html-to-markdown; what these pin is the
|
||||
// contract this package keeps on top of it, which is what a notification
|
||||
// should carry rather than what the document said.
|
||||
func TestRenderMarkdown(t *testing.T) {
|
||||
for name, tc := range map[string]struct{ source, want string }{
|
||||
"paragraphs are separated by a blank line": {
|
||||
source: "<p>first</p><p>second</p>",
|
||||
want: "first\n\nsecond",
|
||||
},
|
||||
"headings keep their level": {
|
||||
source: "<h1>one</h1><h3>three</h3>",
|
||||
want: "# one\n\n### three",
|
||||
},
|
||||
"emphasis": {
|
||||
source: "<p><b>bold</b> <i>italic</i> <del>gone</del> <code>x=1</code></p>",
|
||||
want: "**bold** *italic* ~~gone~~ `x=1`",
|
||||
},
|
||||
"two spans of the same emphasis meeting head on become one": {
|
||||
// What every Word and Outlook export does to a bolded label.
|
||||
source: "<p><b><span>Total</span></b><b><span>: 49.99</span></b></p>",
|
||||
want: "**Total: 49.99**",
|
||||
},
|
||||
"emphasis nested inside itself does not double its markers": {
|
||||
source: "<p><b>a<strong>b</strong></b></p>",
|
||||
want: "**ab**",
|
||||
},
|
||||
"links keep their destination": {
|
||||
source: `<p>see <a href="https://example.com/x">the docs</a></p>`,
|
||||
want: "see [the docs](https://example.com/x)",
|
||||
},
|
||||
"a link with nothing to click on falls back to its destination": {
|
||||
// Otherwise a link wrapped around a tracking pixel renders as
|
||||
// "[](url)", which the reader cannot see at all.
|
||||
source: `<p><a href="https://example.com/x"><img src="https://x.example/p.gif"></a></p>`,
|
||||
want: "[https://example.com/x](https://example.com/x)",
|
||||
},
|
||||
"a link around nothing but a spacer falls back the same way": {
|
||||
source: `<p><a href="https://example.com/x"><span> </span></a></p>`,
|
||||
want: "[https://example.com/x](https://example.com/x)",
|
||||
},
|
||||
"unusable destinations are dropped but their text is kept": {
|
||||
source: `<a href="#anchor">up</a> <a href="javascript:evil()">click</a> <a href="cid:part1">img</a>`,
|
||||
want: "up click img",
|
||||
},
|
||||
"line breaks inside a URL are removed rather than forwarded": {
|
||||
// A newline in an href is invisible in the document and a
|
||||
// fabricated line in the notification.
|
||||
source: "<a href=\"https://ok.example/r URGENT: wire funds\">open report</a>",
|
||||
want: "[open report](https://ok.example/rURGENT:%20wire%20funds)",
|
||||
},
|
||||
"unordered lists": {
|
||||
source: "<ul><li>one</li><li>two</li></ul>",
|
||||
want: "- one\n- two",
|
||||
},
|
||||
"ordered lists count from their start attribute": {
|
||||
source: `<ol start="3"><li>three</li><li>four</li></ol>`,
|
||||
want: "3. three\n4. four",
|
||||
},
|
||||
"blockquotes mark every line": {
|
||||
source: "<p>before</p><blockquote><p>quoted</p></blockquote><p>after</p>",
|
||||
want: "before\n\n> quoted\n\nafter",
|
||||
},
|
||||
"a code block is fenced past the backticks it holds": {
|
||||
// Otherwise the sender closes the block and everything after it —
|
||||
// a heading, a link — renders as live Markdown in a message the
|
||||
// reader takes for a forwarded notification.
|
||||
source: "<pre>ok\n```\n## Your account is locked\n</pre>",
|
||||
want: "````\nok\n```\n## Your account is locked\n````",
|
||||
},
|
||||
"table rows become lines and cells stay apart": {
|
||||
// Mail lays itself out in tables far more often than it tabulates
|
||||
// anything, and a converter with no table rules would otherwise
|
||||
// run a row together as "Total4Failed0".
|
||||
source: "<table><tr><td>Total</td><td>4</td></tr><tr><td>Failed</td><td>0</td></tr></table>",
|
||||
want: "Total 4 \nFailed 0",
|
||||
},
|
||||
"images without alt text are dropped": {
|
||||
source: `<p>a<img src="https://x.example/pixel.gif">b</p>`,
|
||||
want: "ab",
|
||||
},
|
||||
"images with alt text are kept": {
|
||||
source: `<img src="https://x.example/logo.png" alt="Logo">`,
|
||||
want: "",
|
||||
},
|
||||
"an inline attachment keeps its alt text without a destination": {
|
||||
source: `<img src="cid:part1.abc" alt="Chart">`,
|
||||
want: "Chart",
|
||||
},
|
||||
"alt text is collapsed onto one line": {
|
||||
source: "<img src=\"https://x.example/i.png\" alt=\"a\n long alt\">",
|
||||
want: "",
|
||||
},
|
||||
"script and style content never reaches the reader": {
|
||||
source: "<html><head><style>p{color:red}</style></head><body><script>x()</script><p>body</p></body></html>",
|
||||
want: "body",
|
||||
},
|
||||
"hidden preheaders are left out": {
|
||||
source: `<div style="display: none">inbox preview</div><p>real body</p>`,
|
||||
want: "real body",
|
||||
},
|
||||
"the other ways a preheader hides are recognised too": {
|
||||
source: `<div style="visibility:hidden">a</div><div style="mso-hide:all">b</div>` +
|
||||
`<div style="font-size:0px">c</div><div style="font-size:0.9em">d</div><p>body</p>`,
|
||||
want: "d\n\nbody",
|
||||
},
|
||||
"entities and non-breaking spaces become ordinary text": {
|
||||
source: "<p>4m 12s & counting — done</p>",
|
||||
want: "4m 12s & counting — done",
|
||||
},
|
||||
"zero width padding is dropped": {
|
||||
source: "<p>​a​b​</p>",
|
||||
want: "ab",
|
||||
},
|
||||
"text that looks like markup is escaped": {
|
||||
source: "<p>2 * 3, a [b] c, `tick`</p>",
|
||||
want: "2 * 3, a \\[b] c, \\`tick\\`",
|
||||
},
|
||||
"markers are only defused where they would take effect": {
|
||||
source: "<p>- not a bullet</p><p># not a heading</p><p>1. not a list</p><p>a - b</p>",
|
||||
want: "\\- not a bullet\n\n\\# not a heading\n\n1\\. not a list\n\na - b",
|
||||
},
|
||||
"a document with nothing to say renders to nothing": {
|
||||
source: `<html><body><img src="https://x.example/pixel.gif"></body></html>`,
|
||||
want: "",
|
||||
},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
require.Equal(t, tc.want, render(t, tc.source))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The parser recovers from anything, so the conversion has to as well: a
|
||||
// notification is worth more than a report that the markup was invalid.
|
||||
func TestRenderMarkdownSurvivesBrokenMarkup(t *testing.T) {
|
||||
for name, tc := range map[string]struct{ source, want string }{
|
||||
"unclosed tags": {"<p><b>bold<i>both<p>next", "**bold*both***\n\n***next***"},
|
||||
"stray closing tags": {"</div></p>text</b>", "text"},
|
||||
"no markup at all": {"just some words", "just some words"},
|
||||
"an empty document": {"", ""},
|
||||
"an unterminated tag": {"<p>text<", "text<"},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
require.Equal(t, tc.want, render(t, tc.source))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Regression: a line prefix is re-emitted on every line of every level it
|
||||
// nests, so depth multiplies against line count. One message at the server's
|
||||
// own 1 MB limit rendered to 131 MB over two and a quarter minutes — the same
|
||||
// class of fault as the remote DoS closed in 7565618. Neither the converter
|
||||
// nor any of the alternatives bounds this on its own.
|
||||
func TestRenderMarkdownBoundsPathologicalInput(t *testing.T) {
|
||||
// Sized to overrun the cap several times over once the nesting is
|
||||
// flattened, rather than to reproduce the original 1 MB message: the
|
||||
// assertion is the same and the suite stays quick.
|
||||
lines := strings.Repeat("<p>x</p>", maxRenderedBytes/maxNestingDepth)
|
||||
|
||||
for name, source := range map[string]string{
|
||||
"quotes nested far past anything real": strings.Repeat("<blockquote>", 250) + lines,
|
||||
"lists nested far past anything real": strings.Repeat("<ul><li>", 250) + lines,
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
out, err := renderMarkdown(source)
|
||||
require.NoError(t, err)
|
||||
require.LessOrEqual(t, len(out), maxRenderedBytes+len(truncationMarker)+1)
|
||||
require.True(t, strings.HasSuffix(out, truncationMarker),
|
||||
"a capped message says so rather than ending mid-sentence")
|
||||
require.True(t, utf8.ValidString(out), "the cap must not fall inside a rune")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Nesting under the limit is left alone, so an ordinary quoted reply still
|
||||
// reads as one.
|
||||
func TestRenderMarkdownKeepsNestingUnderTheLimit(t *testing.T) {
|
||||
source := strings.Repeat("<blockquote>", maxNestingDepth) + "<p>deep</p>" +
|
||||
strings.Repeat("</blockquote>", maxNestingDepth)
|
||||
|
||||
require.Equal(t, strings.Repeat("> ", maxNestingDepth)+"deep", render(t, source))
|
||||
}
|
||||
|
||||
// A realistic transactional message, pinned end to end. The faults this
|
||||
// feature has had lived in combinations of features rather than in one of
|
||||
// them, which is what a fixture catches and a unit test does not.
|
||||
func TestRenderMarkdownOnARealisticMessage(t *testing.T) {
|
||||
require.Equal(t, strings.Join([]string{
|
||||
"[](https://billing.example.com)",
|
||||
"",
|
||||
"## Invoice #2026-0231",
|
||||
"",
|
||||
"Hi Ferran,",
|
||||
"",
|
||||
"Your invoice for **February 2026 (30 days)** is ready. The total is " +
|
||||
"**€42.00**, charged to the card ending 4242 on *1 March*.",
|
||||
"",
|
||||
"Description Qty Amount ",
|
||||
"Hosting — small 1 €30.00 ",
|
||||
"Backups 2 €12.00",
|
||||
"",
|
||||
"Reference: `inv_2026*0231` ",
|
||||
"[View invoice](https://billing.example.com/invoices/2026-0231?utm_source=email&utm_medium=cta)",
|
||||
"",
|
||||
"You are receiving this because you have an account. " +
|
||||
"[Unsubscribe](https://billing.example.com/unsubscribe?t=abc) · Back to top",
|
||||
}, "\n"), render(t, marketingMessage))
|
||||
}
|
||||
|
||||
// Shaped like the transactional mail this feature exists for: a hidden
|
||||
// preheader, a layout table, a logo wrapped in a link, an Outlook-style split
|
||||
// bold run, a tracking pixel and an in-page anchor.
|
||||
const marketingMessage = `<!DOCTYPE html><html><head><meta charset="utf-8"><title>Invoice</title>
|
||||
<style>.btn{background:#000}</style></head><body style="margin:0">
|
||||
<span style="display:none;font-size:1px">Your February invoice is ready​​</span>
|
||||
<table width="100%"><tbody><tr><td align="center"><table width="600"><tbody>
|
||||
<tr><td><a href="https://billing.example.com"><img src="https://cdn.example.com/logo.png" alt="Example Billing"></a></td></tr>
|
||||
<tr><td><h2>Invoice #2026-0231</h2></td></tr>
|
||||
<tr><td>Hi Ferran,<br><br>Your invoice for <b><span>February 2026</span></b><b><span> (30 days)</span></b> is ready.
|
||||
The total is <strong>€42.00</strong>, charged to the card ending 4242 on <em>1 March</em>.</td></tr>
|
||||
<tr><td><table><tbody><tr><th>Description</th><th>Qty</th><th>Amount</th></tr>
|
||||
<tr><td>Hosting — small</td><td>1</td><td>€30.00</td></tr>
|
||||
<tr><td>Backups</td><td>2</td><td>€12.00</td></tr></tbody></table></td></tr>
|
||||
<tr><td>Reference: <code>inv_2026*0231</code></td></tr>
|
||||
<tr><td align="center"><a class="btn" href="https://billing.example.com/invoices/2026-0231?utm_source=email&utm_medium=cta">View invoice</a></td></tr>
|
||||
<tr><td><p style="font-size:11px">You are receiving this because you have an account.
|
||||
<a href="https://billing.example.com/unsubscribe?t=abc">Unsubscribe</a> · <a href="#top">Back to top</a></p>
|
||||
<img src="https://track.example.com/o.gif?id=abc" width="1" height="1"></td></tr>
|
||||
</tbody></table></td></tr></tbody></table></body></html>`
|
||||
Loading…
Reference in a new issue