Converting png to jpg in ffmpeg with scale2ref

Posted on Dec 20, 2023

source: a human posted something like “why does this happen?” on instagram. picked it up from there.

If I attempt converting a png to jpg using ffmpeg -i input.png output.jpg, it usually works out but with this png file (patterns) in particular, we’re getting this output jpg file (a solid color from the image). Which is weird. Let’s try finding whatever we can.
First I confirmed that this does infact get converted correctly using some online conversion tool. It’s getting converted, the background in jpeg is set to white, otherwise everything seems alright with that.
Did not want to trace the codepath yet so tried playing with the cli options for a while.
Now, looking at the file with ffprobe -v error -hide_banner -print_format json -show_streams <filename>.<ext>
- I initially thought this might be happening because of pixel format(Try ffmpeg -h encoder=mjpeg -hide_banner|less),
- Then I could create some variations in results with playing around with in_range and out_range for scale filter
- Also see A journey through color space with FFmpeg - Canva Engineering Blog
It occurred to me that I could follow the steps imagemagick convert uses
When I tried converting input.png to output.jpg in imagemagick convert (with convert input.png output.jpg), surprisingly it also gave the same “visual” output. This ensured the issue was in the process for the particular image and not necessarily an issue in ffmpeg.
What fixed it for imagemagick was using -flatten : convert input.png -flatten output.jpg

We then try to emulate the same in ffmpeg and probably even follow the codepath if that works out.

  ffmpeg -i input.png -vf "split[a][b];[a]format=rgba,colorchannelmixer=aa=1[aint];[b]geq=r='r(X,Y)':a='0*alpha(X,Y)'[bint];[aint][bint]blend=all_mode='addition'" output.jpg

Improving on that, we finally got it working with scale2ref

  ffmpeg -i input.png -vf "color=white[bg];[bg][0]scale2ref[bgs][0s];[bgs][0s]overlay=shortest=1" output.jpg

So basically, the background for the converted image was not getting correctly set, even though we arrived at the correct outcome, we could not figure out what exactly caused the initial issue.

Doubts

I am still not sure what makes imagemagick and ffmpeg pick the particular color that it picks for background.

$ magick input.png -colors 20 -unique-colors txt:-
# ImageMagick pixel enumeration: 5,1,0,255,srgba
0,0: (198,96,83,40)  #C6605328  srgba(198,96,83,0.156863)
1,0: (198,96,83,120)  #C6605378  srgba(77.6471%,37.6471%,32.549%,0.470588)
2,0: (197,98,83,141)  #C562538D  srgba(197,98,83,0.552941)
3,0: (198,98,84,183)  #C66254B7  srgba(198,98,84,0.717647)
4,0: (197,97,83,184)  #C56153B8  srgba(197,97,83,0.721569)

$ magick incorrect_output.jpg -colors 20 -unique-colors txt:-
# ImageMagick pixel enumeration: 3,1,0,255,srgb
0,0: (199,95,82)  #C75F52  srgb(199,95,82)
1,0: (198,96,82)  #C66052  srgb(198,96,82)
2,0: (199,99,84)  #C76354  srgb(199,99,84)

$ magick correct_output.jpg -colors 20 -unique-colors txt:-
# ImageMagick pixel enumeration: 14,1,0,255,srgb
0,0: (217,138,133)  #D98A85  srgb(85.0478%,54.1277%,52.3379%)
1,0: (224,141,137)  #E08D89  srgb(224,141,137)
2,0: (218,170,156)  #DAAA9C  srgb(85.6361%,66.6211%,61.3497%)
3,0: (226,168,158)  #E2A89E  srgb(88.7802%,65.7781%,62.0547%)
4,0: (223,168,161)  #DFA8A1  srgb(87.3387%,65.9835%,63.2184%)
5,0: (227,170,166)  #E3AAA6  srgb(88.8466%,66.659%,65.2358%)
6,0: (247,218,214)  #F7DAD6  srgb(247,218,214)
7,0: (245,230,223)  #F5E6DF  srgb(96.1077%,90.1333%,87.3522%)
8,0: (246,231,227)  #F6E7E3  srgb(96.4678%,90.5964%,89.2112%)
9,0: (246,231,227)  #F6E7E3  srgb(96.4918%,90.567%,89.1715%)
10,0: (245,233,227)  #F5E9E3  srgb(95.8824%,91.1765%,89.0196%)
11,0: (243,232,228)  #F3E8E4  srgb(243,232,228)
12,0: (246,232,231)  #F6E8E7  srgb(246,232,231)
13,0: (250,232,230)  #FAE8E6  srgb(250,232,230)