butterrobot
06de333118
The hand-written renderer is replaced by github.com/JohannesKaufmann/html-to-markdown/v2 plus the mail-specific policy it has no opinion about. 986 lines of html.go become 372; the conversion itself — CommonMark escaping, delimiter runs, fencing a code block past the backticks inside it — is now a maintained library's problem rather than ours. The original justification for writing it by hand was that the library would drag in goquery and its dependencies. That was true of v1 and wrong for v2, which dropped it: the measured cost is two modules, html-to-markdown/v2 and JohannesKaufmann/dom, on top of the golang.org/x/net this already used. What the library does not know is mail, because it is written for documents. The parsed message is prepared before conversion: - Hidden preheaders, written for the inbox list, are removed. - Images without alt text go, which takes the tracking pixels, spacers and sliced-up banners with them. An inline cid: attachment leaves its alt text behind as ordinary words. - Destinations a reader cannot open are dropped and the link text kept; tabs and line breaks are stripped from the rest, since a line break inside an href is invisible in the document and a fabricated line in the message. - Table rows become lines and their cells stay apart, which the converter has no rule for: "Total4Failed0" otherwise. - A link left holding nothing but a pixel falls back to its own destination rather than rendering as an invisible "[](url)". - Quote and list nesting is flattened past six levels, and the output is capped at 64 KiB with a marker. That last one is not something any of the candidates solved. html-to-markdown amplifies exactly as the hand-written renderer did before it was capped, from the same cause — a line prefix re-emitted per line and per level. Measured on one message at the server's own 1 MB limit, nested 250 deep: 131 MB of output over 2m16s, against 64 KiB in 1.5s and 85 MiB of peak heap with the flattening in place. Format = "text" is dropped, leaving raw and markdown. Markdown reads as plain text wherever nothing renders it, so a second conversion would only have been a worse copy of this one, and the plain-text libraries surveyed were the weak half of the field. Nothing has shipped with "text", so no released configuration names it; an unknown Format is still refused at startup. The test suite carries over almost unchanged, because it asserts output rather than internals — which is what made the swap safe to judge. Every mail-policy and injection case still holds, and the pathological-input test is sized from the constants now so the suite stays quick. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
248 lines
11 KiB
Go
248 lines
11 KiB
Go
package smtp2shoutrrr
|
|
|
|
import (
|
|
"strings"
|
|
"testing"
|
|
"unicode/utf8"
|
|
|
|
"github.com/stretchr/testify/require"
|
|
)
|
|
|
|
func render(t *testing.T, source string) string {
|
|
t.Helper()
|
|
|
|
out, err := renderMarkdown(source)
|
|
require.NoError(t, err)
|
|
|
|
return out
|
|
}
|
|
|
|
// The conversion itself belongs to html-to-markdown; what these pin is the
|
|
// contract this package keeps on top of it, which is what a notification
|
|
// should carry rather than what the document said.
|
|
func TestRenderMarkdown(t *testing.T) {
|
|
for name, tc := range map[string]struct{ source, want string }{
|
|
"paragraphs are separated by a blank line": {
|
|
source: "<p>first</p><p>second</p>",
|
|
want: "first\n\nsecond",
|
|
},
|
|
"headings keep their level": {
|
|
source: "<h1>one</h1><h3>three</h3>",
|
|
want: "# one\n\n### three",
|
|
},
|
|
"emphasis": {
|
|
source: "<p><b>bold</b> <i>italic</i> <del>gone</del> <code>x=1</code></p>",
|
|
want: "**bold** *italic* ~~gone~~ `x=1`",
|
|
},
|
|
"two spans of the same emphasis meeting head on become one": {
|
|
// What every Word and Outlook export does to a bolded label.
|
|
source: "<p><b><span>Total</span></b><b><span>: 49.99</span></b></p>",
|
|
want: "**Total: 49.99**",
|
|
},
|
|
"emphasis nested inside itself does not double its markers": {
|
|
source: "<p><b>a<strong>b</strong></b></p>",
|
|
want: "**ab**",
|
|
},
|
|
"links keep their destination": {
|
|
source: `<p>see <a href="https://example.com/x">the docs</a></p>`,
|
|
want: "see [the docs](https://example.com/x)",
|
|
},
|
|
"a link with nothing to click on falls back to its destination": {
|
|
// Otherwise a link wrapped around a tracking pixel renders as
|
|
// "[](url)", which the reader cannot see at all.
|
|
source: `<p><a href="https://example.com/x"><img src="https://x.example/p.gif"></a></p>`,
|
|
want: "[https://example.com/x](https://example.com/x)",
|
|
},
|
|
"a link around nothing but a spacer falls back the same way": {
|
|
source: `<p><a href="https://example.com/x"><span> </span></a></p>`,
|
|
want: "[https://example.com/x](https://example.com/x)",
|
|
},
|
|
"unusable destinations are dropped but their text is kept": {
|
|
source: `<a href="#anchor">up</a> <a href="javascript:evil()">click</a> <a href="cid:part1">img</a>`,
|
|
want: "up click img",
|
|
},
|
|
"line breaks inside a URL are removed rather than forwarded": {
|
|
// A newline in an href is invisible in the document and a
|
|
// fabricated line in the notification.
|
|
source: "<a href=\"https://ok.example/r URGENT: wire funds\">open report</a>",
|
|
want: "[open report](https://ok.example/rURGENT:%20wire%20funds)",
|
|
},
|
|
"unordered lists": {
|
|
source: "<ul><li>one</li><li>two</li></ul>",
|
|
want: "- one\n- two",
|
|
},
|
|
"ordered lists count from their start attribute": {
|
|
source: `<ol start="3"><li>three</li><li>four</li></ol>`,
|
|
want: "3. three\n4. four",
|
|
},
|
|
"blockquotes mark every line": {
|
|
source: "<p>before</p><blockquote><p>quoted</p></blockquote><p>after</p>",
|
|
want: "before\n\n> quoted\n\nafter",
|
|
},
|
|
"a code block is fenced past the backticks it holds": {
|
|
// Otherwise the sender closes the block and everything after it —
|
|
// a heading, a link — renders as live Markdown in a message the
|
|
// reader takes for a forwarded notification.
|
|
source: "<pre>ok\n```\n## Your account is locked\n</pre>",
|
|
want: "````\nok\n```\n## Your account is locked\n````",
|
|
},
|
|
"table rows become lines and cells stay apart": {
|
|
// Mail lays itself out in tables far more often than it tabulates
|
|
// anything, and a converter with no table rules would otherwise
|
|
// run a row together as "Total4Failed0".
|
|
source: "<table><tr><td>Total</td><td>4</td></tr><tr><td>Failed</td><td>0</td></tr></table>",
|
|
want: "Total 4 \nFailed 0",
|
|
},
|
|
"images without alt text are dropped": {
|
|
source: `<p>a<img src="https://x.example/pixel.gif">b</p>`,
|
|
want: "ab",
|
|
},
|
|
"images with alt text are kept": {
|
|
source: `<img src="https://x.example/logo.png" alt="Logo">`,
|
|
want: "",
|
|
},
|
|
"an inline attachment keeps its alt text without a destination": {
|
|
source: `<img src="cid:part1.abc" alt="Chart">`,
|
|
want: "Chart",
|
|
},
|
|
"alt text is collapsed onto one line": {
|
|
source: "<img src=\"https://x.example/i.png\" alt=\"a\n long alt\">",
|
|
want: "",
|
|
},
|
|
"script and style content never reaches the reader": {
|
|
source: "<html><head><style>p{color:red}</style></head><body><script>x()</script><p>body</p></body></html>",
|
|
want: "body",
|
|
},
|
|
"hidden preheaders are left out": {
|
|
source: `<div style="display: none">inbox preview</div><p>real body</p>`,
|
|
want: "real body",
|
|
},
|
|
"the other ways a preheader hides are recognised too": {
|
|
source: `<div style="visibility:hidden">a</div><div style="mso-hide:all">b</div>` +
|
|
`<div style="font-size:0px">c</div><div style="font-size:0.9em">d</div><p>body</p>`,
|
|
want: "d\n\nbody",
|
|
},
|
|
"entities and non-breaking spaces become ordinary text": {
|
|
source: "<p>4m 12s & counting — done</p>",
|
|
want: "4m 12s & counting — done",
|
|
},
|
|
"zero width padding is dropped": {
|
|
source: "<p>​a​b​</p>",
|
|
want: "ab",
|
|
},
|
|
"text that looks like markup is escaped": {
|
|
source: "<p>2 * 3, a [b] c, `tick`</p>",
|
|
want: "2 * 3, a \\[b] c, \\`tick\\`",
|
|
},
|
|
"markers are only defused where they would take effect": {
|
|
source: "<p>- not a bullet</p><p># not a heading</p><p>1. not a list</p><p>a - b</p>",
|
|
want: "\\- not a bullet\n\n\\# not a heading\n\n1\\. not a list\n\na - b",
|
|
},
|
|
"a document with nothing to say renders to nothing": {
|
|
source: `<html><body><img src="https://x.example/pixel.gif"></body></html>`,
|
|
want: "",
|
|
},
|
|
} {
|
|
t.Run(name, func(t *testing.T) {
|
|
require.Equal(t, tc.want, render(t, tc.source))
|
|
})
|
|
}
|
|
}
|
|
|
|
// The parser recovers from anything, so the conversion has to as well: a
|
|
// notification is worth more than a report that the markup was invalid.
|
|
func TestRenderMarkdownSurvivesBrokenMarkup(t *testing.T) {
|
|
for name, tc := range map[string]struct{ source, want string }{
|
|
"unclosed tags": {"<p><b>bold<i>both<p>next", "**bold*both***\n\n***next***"},
|
|
"stray closing tags": {"</div></p>text</b>", "text"},
|
|
"no markup at all": {"just some words", "just some words"},
|
|
"an empty document": {"", ""},
|
|
"an unterminated tag": {"<p>text<", "text<"},
|
|
} {
|
|
t.Run(name, func(t *testing.T) {
|
|
require.Equal(t, tc.want, render(t, tc.source))
|
|
})
|
|
}
|
|
}
|
|
|
|
// Regression: a line prefix is re-emitted on every line of every level it
|
|
// nests, so depth multiplies against line count. One message at the server's
|
|
// own 1 MB limit rendered to 131 MB over two and a quarter minutes — the same
|
|
// class of fault as the remote DoS closed in 7565618. Neither the converter
|
|
// nor any of the alternatives bounds this on its own.
|
|
func TestRenderMarkdownBoundsPathologicalInput(t *testing.T) {
|
|
// Sized to overrun the cap several times over once the nesting is
|
|
// flattened, rather than to reproduce the original 1 MB message: the
|
|
// assertion is the same and the suite stays quick.
|
|
lines := strings.Repeat("<p>x</p>", maxRenderedBytes/maxNestingDepth)
|
|
|
|
for name, source := range map[string]string{
|
|
"quotes nested far past anything real": strings.Repeat("<blockquote>", 250) + lines,
|
|
"lists nested far past anything real": strings.Repeat("<ul><li>", 250) + lines,
|
|
} {
|
|
t.Run(name, func(t *testing.T) {
|
|
out, err := renderMarkdown(source)
|
|
require.NoError(t, err)
|
|
require.LessOrEqual(t, len(out), maxRenderedBytes+len(truncationMarker)+1)
|
|
require.True(t, strings.HasSuffix(out, truncationMarker),
|
|
"a capped message says so rather than ending mid-sentence")
|
|
require.True(t, utf8.ValidString(out), "the cap must not fall inside a rune")
|
|
})
|
|
}
|
|
}
|
|
|
|
// Nesting under the limit is left alone, so an ordinary quoted reply still
|
|
// reads as one.
|
|
func TestRenderMarkdownKeepsNestingUnderTheLimit(t *testing.T) {
|
|
source := strings.Repeat("<blockquote>", maxNestingDepth) + "<p>deep</p>" +
|
|
strings.Repeat("</blockquote>", maxNestingDepth)
|
|
|
|
require.Equal(t, strings.Repeat("> ", maxNestingDepth)+"deep", render(t, source))
|
|
}
|
|
|
|
// A realistic transactional message, pinned end to end. The faults this
|
|
// feature has had lived in combinations of features rather than in one of
|
|
// them, which is what a fixture catches and a unit test does not.
|
|
func TestRenderMarkdownOnARealisticMessage(t *testing.T) {
|
|
require.Equal(t, strings.Join([]string{
|
|
"[](https://billing.example.com)",
|
|
"",
|
|
"## Invoice #2026-0231",
|
|
"",
|
|
"Hi Ferran,",
|
|
"",
|
|
"Your invoice for **February 2026 (30 days)** is ready. The total is " +
|
|
"**€42.00**, charged to the card ending 4242 on *1 March*.",
|
|
"",
|
|
"Description Qty Amount ",
|
|
"Hosting — small 1 €30.00 ",
|
|
"Backups 2 €12.00",
|
|
"",
|
|
"Reference: `inv_2026*0231` ",
|
|
"[View invoice](https://billing.example.com/invoices/2026-0231?utm_source=email&utm_medium=cta)",
|
|
"",
|
|
"You are receiving this because you have an account. " +
|
|
"[Unsubscribe](https://billing.example.com/unsubscribe?t=abc) · Back to top",
|
|
}, "\n"), render(t, marketingMessage))
|
|
}
|
|
|
|
// Shaped like the transactional mail this feature exists for: a hidden
|
|
// preheader, a layout table, a logo wrapped in a link, an Outlook-style split
|
|
// bold run, a tracking pixel and an in-page anchor.
|
|
const marketingMessage = `<!DOCTYPE html><html><head><meta charset="utf-8"><title>Invoice</title>
|
|
<style>.btn{background:#000}</style></head><body style="margin:0">
|
|
<span style="display:none;font-size:1px">Your February invoice is ready​​</span>
|
|
<table width="100%"><tbody><tr><td align="center"><table width="600"><tbody>
|
|
<tr><td><a href="https://billing.example.com"><img src="https://cdn.example.com/logo.png" alt="Example Billing"></a></td></tr>
|
|
<tr><td><h2>Invoice #2026-0231</h2></td></tr>
|
|
<tr><td>Hi Ferran,<br><br>Your invoice for <b><span>February 2026</span></b><b><span> (30 days)</span></b> is ready.
|
|
The total is <strong>€42.00</strong>, charged to the card ending 4242 on <em>1 March</em>.</td></tr>
|
|
<tr><td><table><tbody><tr><th>Description</th><th>Qty</th><th>Amount</th></tr>
|
|
<tr><td>Hosting — small</td><td>1</td><td>€30.00</td></tr>
|
|
<tr><td>Backups</td><td>2</td><td>€12.00</td></tr></tbody></table></td></tr>
|
|
<tr><td>Reference: <code>inv_2026*0231</code></td></tr>
|
|
<tr><td align="center"><a class="btn" href="https://billing.example.com/invoices/2026-0231?utm_source=email&utm_medium=cta">View invoice</a></td></tr>
|
|
<tr><td><p style="font-size:11px">You are receiving this because you have an account.
|
|
<a href="https://billing.example.com/unsubscribe?t=abc">Unsubscribe</a> · <a href="#top">Back to top</a></p>
|
|
<img src="https://track.example.com/o.gif?id=abc" width="1" height="1"></td></tr>
|
|
</tbody></table></td></tr></tbody></table></body></html>`
|