Discussion:
[Arm-netbook] [EOMA68] A20 G2D / VAAPI / VDPAU video acceleration in web browsers
Luke Kenneth Casson Leighton
2016-08-09 21:36:05 UTC
Permalink
.... ain't gonna work with the current code-base of either xulrunner
(firefox and ice* types) or webkit (chrome, midori, qupzilla). darn.

the reason for webkit is: although gstreamer supports vdpau and vaapi,
and even can be activated to do hardware-accelerated decode, webkit
(at least as best i can find) the gtk+ variant then targets *opengl*
as the output rendering engine, rather than asks VAAPI / VDPAU to
handle it directly on the webkit engine's behalf.

what that means is: with software-rendering OpenGL of the decoded
pixels it's simply too much for the A20 to handle.

amazingly i have actually seen icecat do about 2 or 3 fps video
decoding in software, i really don't know how it managed, and i
haven't really been able to replicate it.

anyway if people are expecting videos to be played in web browsers,
it's gonna take some work. for now it's possible to download them
(youtube-dl or other methods) and then use a video player.

l.

---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68

_______________________________________________
arm-netbook mailing list arm-***@lists.phcomp.co.uk
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
Send lar
Russell Hyer
2016-08-09 21:40:47 UTC
Permalink
Hi Luke,

So, if I've understood this post correctly, video playback on vlc
would perhaps be within the bounds of what the A20 hardware / software
could do?

Russell
Just a Fourier transform away from creating a computer game...
Post by Luke Kenneth Casson Leighton
.... ain't gonna work with the current code-base of either xulrunner
(firefox and ice* types) or webkit (chrome, midori, qupzilla). darn.
the reason for webkit is: although gstreamer supports vdpau and vaapi,
and even can be activated to do hardware-accelerated decode, webkit
(at least as best i can find) the gtk+ variant then targets *opengl*
as the output rendering engine, rather than asks VAAPI / VDPAU to
handle it directly on the webkit engine's behalf.
what that means is: with software-rendering OpenGL of the decoded
pixels it's simply too much for the A20 to handle.
amazingly i have actually seen icecat do about 2 or 3 fps video
decoding in software, i really don't know how it managed, and i
haven't really been able to replicate it.
anyway if people are expecting videos to be played in web browsers,
it's gonna take some work. for now it's possible to download them
(youtube-dl or other methods) and then use a video player.
l.
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
_______________________________________________
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
_______________________________________________
arm-netbook mailing list arm-***@lists.phcomp.co.uk
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
Send large at
Luke Kenneth Casson Leighton
2016-08-09 22:16:23 UTC
Permalink
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
Post by Russell Hyer
Hi Luke,
So, if I've understood this post correctly, video playback on vlc
would perhaps be within the bounds of what the A20 hardware / software
could do?
yes. ffplay, vlc, mpv, mplayer - all work fine, that's what this
update was about:
https://www.crowdsupply.com/eoma68/micro-desktop/updates/cedrus

basically by using the full capabilities of the A20's accelerated
hardware (both CEDAR and G2D), the CPU usage is somewhere around the
10% mark. peanuts in other words.

the reason why those work is, through vdpau plugins there's two
parts: "decode" part and "actual on-screen display" part. all those
video players have an option to do *both* things, and the vdpau-sunxi
plugin that the Cedrus team wrote goes "okay, we're configured to use
G2D, let's just go for it".

that's why you end up with the overlay issues, btw, because the G2D
hardware is told to write direct to an area on-screen, bypassing the
X11 protocol.

anyway, it's *ASSUMED* in the webkit and xulrunner engines that the
"on-screen display" bit can be handled by OpenGL or...
something-else-don't-know-don't-care-but-basically-it's-not.

and that means a bit of rewriting of both webkit and xulrunner so
that they hand over the display bit to G2D. that's going to require
some specialist work.

l.

_______________________________________________
arm-netbook mailing list arm-***@lists.phcomp.co.uk
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
Send large attachments to arm-***@file
Russell Hyer
2016-08-10 05:50:28 UTC
Permalink
thanks for the detailed answer.

russ
sent from a non libre device
Post by Luke Kenneth Casson Leighton
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
Post by Russell Hyer
Hi Luke,
So, if I've understood this post correctly, video playback on vlc
would perhaps be within the bounds of what the A20 hardware / software
could do?
yes. ffplay, vlc, mpv, mplayer - all work fine, that's what this
https://www.crowdsupply.com/eoma68/micro-desktop/updates/cedrus
basically by using the full capabilities of the A20's accelerated
hardware (both CEDAR and G2D), the CPU usage is somewhere around the
10% mark. peanuts in other words.
the reason why those work is, through vdpau plugins there's two
parts: "decode" part and "actual on-screen display" part. all those
video players have an option to do *both* things, and the vdpau-sunxi
plugin that the Cedrus team wrote goes "okay, we're configured to use
G2D, let's just go for it".
that's why you end up with the overlay issues, btw, because the G2D
hardware is told to write direct to an area on-screen, bypassing the
X11 protocol.
anyway, it's *ASSUMED* in the webkit and xulrunner engines that the
"on-screen display" bit can be handled by OpenGL or...
something-else-don't-know-don't-care-but-basically-it's-not.
and that means a bit of rewriting of both webkit and xulrunner so
that they hand over the display bit to G2D. that's going to require
some specialist work.
l.
_______________________________________________
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
Andreas Baierl
2016-08-10 07:23:25 UTC
Permalink
Hi Luke,
Post by Luke Kenneth Casson Leighton
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
Post by Russell Hyer
Hi Luke,
So, if I've understood this post correctly, video playback on vlc
would perhaps be within the bounds of what the A20 hardware / software
could do?
yes. ffplay, vlc, mpv, mplayer - all work fine, that's what this
https://www.crowdsupply.com/eoma68/micro-desktop/updates/cedrus
basically by using the full capabilities of the A20's accelerated
hardware (both CEDAR and G2D), the CPU usage is somewhere around the
10% mark. peanuts in other words.
the reason why those work is, through vdpau plugins there's two
parts: "decode" part and "actual on-screen display" part. all those
video players have an option to do *both* things, and the vdpau-sunxi
plugin that the Cedrus team wrote goes "okay, we're configured to use
G2D, let's just go for it".
that's why you end up with the overlay issues, btw, because the G2D
hardware is told to write direct to an area on-screen, bypassing the
X11 protocol.
Just a minor correction. The G2D (mixer processor) hardware (only on
A10/A20) is only used within libvdpau-sunxi to blit one rgba surface
into another one. It has nothing to do with bringing something *really*
to screen, because it only handles the blit or fill actions on rgba
surfaces. The only use of rgba surfaces that i know of, is to realize an
OSD, subtitles etc... as an overlay over the video.

NVidia and the others most likely do *real* blits of that rgba into the
the yuv video. Maybe with the use of OpenGL.
Our libvdpau-sunxi does separate these surfaces and hold them both on
separater layers. The display enginge takes care of these two layers and
pushes the video into layer 1 and then the OSD (if existent) into
another layer 2 which lays over the video layer. So data keeps separated.
This is the problem why some things with libvdpau-sunxi do not work
exactly like someone expects. Especially in windowed mode. We use a
colorkey to mask the video area and don't render with X11's mechanism,
but bypass the whole X11 system and write directly to framebuffer as you
said.
So to sum up, overlay issues are caused by the display engine, or the
way libvdpau-sunxi build his surfaces, but not by G2D.
As we don't have G2D available on future Allwinner SoCs, the blits there
are done in software using pixman.
Post by Luke Kenneth Casson Leighton
anyway, it's *ASSUMED* in the webkit and xulrunner engines that the
"on-screen display" bit can be handled by OpenGL or...
something-else-don't-know-don't-care-but-basically-it's-not.
Oh. Maybe i completely misunderstood and my previous comment was
b*****it, but is there really an o-s-d over the video within the browser
window? If so, they most likely use something with OpenGL, yes.
Post by Luke Kenneth Casson Leighton
and that means a bit of rewriting of both webkit and xulrunner so
that they hand over the display bit to G2D. that's going to require
some specialist work.
It would theoretically be possible to let libvdpau-sunxi use OpenGL/ES
under the hood and blit everything into one surface/texture. But this
maybe would drop the performance advantages compared to how we do know.
And you depend on Mali....
There are some efforts to integrate/simulate the nv_gl_interop extension
with libvdpau-sunxi and OpenGL/ES, which lets OpenGL use vdpau surface
and vice versa. So it would be possible to let vdpau do the rendering
and OpenGL/ES do the presentation. Problem is, that it's not finished
and it's not intended by the spec anyway to either use VDPAU on ARM nor
combine it with OpenGL/ES, as the nv_gl_interop only talks about i386
and OpenGL. <- This is the way kodi works with vdpau for example.

And the whole problem of this all is, that i currently cannot used with
mainline kernel :) So i'd look further in having an open source video
decoding driver and the needed display driver bits getting mainline ;)

Andreas
Post by Luke Kenneth Casson Leighton
l.
_______________________________________________
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
_______________________________________________
arm-netbook mailing list arm-***@lists.phcomp.co.uk
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
Send large attachments
Luke Kenneth Casson Leighton
2016-08-10 14:30:55 UTC
Permalink
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
Post by Russell Hyer
Hi Luke,
Post by Luke Kenneth Casson Leighton
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
Post by Russell Hyer
Hi Luke,
So, if I've understood this post correctly, video playback on vlc
would perhaps be within the bounds of what the A20 hardware / software
could do?
yes. ffplay, vlc, mpv, mplayer - all work fine, that's what this
https://www.crowdsupply.com/eoma68/micro-desktop/updates/cedrus
basically by using the full capabilities of the A20's accelerated
hardware (both CEDAR and G2D), the CPU usage is somewhere around the
10% mark. peanuts in other words.
the reason why those work is, through vdpau plugins there's two
parts: "decode" part and "actual on-screen display" part. all those
video players have an option to do *both* things, and the vdpau-sunxi
plugin that the Cedrus team wrote goes "okay, we're configured to use
G2D, let's just go for it".
that's why you end up with the overlay issues, btw, because the G2D
hardware is told to write direct to an area on-screen, bypassing the
X11 protocol.
Just a minor correction. The G2D (mixer processor) hardware (only on
A10/A20) is only used within libvdpau-sunxi to blit one rgba surface into
another one. It has nothing to do with bringing something *really* to
screen, because it only handles the blit or fill actions on rgba surfaces.
The only use of rgba surfaces that i know of, is to realize an OSD,
subtitles etc... as an overlay over the video.
interesting.... because if i don't enable OSD i get absolutely
nothing (black screen) on mpv and other players.
Post by Russell Hyer
NVidia and the others most likely do *real* blits of that rgba into the the
yuv video. Maybe with the use of OpenGL.
Our libvdpau-sunxi does separate these surfaces and hold them both on
separater layers. The display enginge takes care of these two layers and
pushes the video into layer 1 and then the OSD (if existent) into another
layer 2 which lays over the video layer. So data keeps separated.
This is the problem why some things with libvdpau-sunxi do not work exactly
like someone expects. Especially in windowed mode. We use a colorkey to mask
the video area and don't render with X11's mechanism, but bypass the whole
X11 system and write directly to framebuffer as you said.
.... i knew it was _something_ like that... :)
Post by Russell Hyer
So to sum up, overlay issues are caused by the display engine, or the way
libvdpau-sunxi build his surfaces, but not by G2D.
As we don't have G2D available on future Allwinner SoCs, the blits there are
done in software using pixman.
let's hope there's enough horsepower in those!
Post by Russell Hyer
Post by Luke Kenneth Casson Leighton
anyway, it's *ASSUMED* in the webkit and xulrunner engines that the
"on-screen display" bit can be handled by OpenGL or...
something-else-don't-know-don't-care-but-basically-it's-not.
Oh. Maybe i completely misunderstood and my previous comment was b*****it,
but is there really an o-s-d over the video within the browser window? If
so, they most likely use something with OpenGL, yes.
google "webkit gtk vaapi gstreamer" there's a blog by the (pretty
much world's only) person dealing with it.
Post by Russell Hyer
Post by Luke Kenneth Casson Leighton
and that means a bit of rewriting of both webkit and xulrunner so
that they hand over the display bit to G2D. that's going to require
some specialist work.
It would theoretically be possible to let libvdpau-sunxi use OpenGL/ES under
the hood and blit everything into one surface/texture. But this maybe would
drop the performance advantages compared to how we do know. And you depend
on Mali....
... which would be bad. every person who's running with libre
software at the moment would never be able to view videos.
Post by Russell Hyer
There are some efforts to integrate/simulate the nv_gl_interop extension
with libvdpau-sunxi and OpenGL/ES, which lets OpenGL use vdpau surface and
vice versa. So it would be possible to let vdpau do the rendering and
OpenGL/ES do the presentation. Problem is, that it's not finished and it's
not intended by the spec anyway to either use VDPAU on ARM nor combine it
with OpenGL/ES, as the nv_gl_interop only talks about i386 and OpenGL. <-
This is the way kodi works with vdpau for example.
And the whole problem of this all is, that i currently cannot used with
mainline kernel :)
i knoooow....

l.

_______________________________________________
arm-netbook mailing list arm-***@lists.phcomp.co.uk
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
Send large attachments to arm-***@files.phcomp.co.
Andreas Baierl
2016-08-10 14:37:17 UTC
Permalink
Post by Luke Kenneth Casson Leighton
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
Post by Russell Hyer
Hi Luke,
Post by Luke Kenneth Casson Leighton
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
Post by Russell Hyer
Hi Luke,
So, if I've understood this post correctly, video playback on vlc
would perhaps be within the bounds of what the A20 hardware / software
could do?
yes. ffplay, vlc, mpv, mplayer - all work fine, that's what this
https://www.crowdsupply.com/eoma68/micro-desktop/updates/cedrus
basically by using the full capabilities of the A20's accelerated
hardware (both CEDAR and G2D), the CPU usage is somewhere around the
10% mark. peanuts in other words.
the reason why those work is, through vdpau plugins there's two
parts: "decode" part and "actual on-screen display" part. all those
video players have an option to do *both* things, and the vdpau-sunxi
plugin that the Cedrus team wrote goes "okay, we're configured to use
G2D, let's just go for it".
that's why you end up with the overlay issues, btw, because the G2D
hardware is told to write direct to an area on-screen, bypassing the
X11 protocol.
Just a minor correction. The G2D (mixer processor) hardware (only on
A10/A20) is only used within libvdpau-sunxi to blit one rgba surface into
another one. It has nothing to do with bringing something *really* to
screen, because it only handles the blit or fill actions on rgba surfaces.
The only use of rgba surfaces that i know of, is to realize an OSD,
subtitles etc... as an overlay over the video.
interesting.... because if i don't enable OSD i get absolutely
nothing (black screen) on mpv and other players.
Arghh... You are the second one, that reports that issue :(
Now i really have to investigate into this :p
With enable you mean setting VDPAU_OSD=1 ? And I suppose you do this all
in windows within X11?

But getting offtopic now. Maybe we should continue on irc...

rellla
Post by Luke Kenneth Casson Leighton
i knoooow....
l.
_______________________________________________
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
_______________________________________________
arm-netbook mailing list arm-***@lists.phcomp.co.uk
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
Send large attachments to arm-***@files.phcomp.co
Luke Kenneth Casson Leighton
2016-08-10 14:48:33 UTC
Permalink
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
Post by Andreas Baierl
Arghh... You are the second one, that reports that issue :(
Now i really have to investigate into this :p
:) using 16-bit 1280x720. i think.
Post by Andreas Baierl
With enable you mean setting VDPAU_OSD=1 ?
yes.
Post by Andreas Baierl
And I suppose you do this all in
windows within X11?
yes.
Post by Andreas Baierl
But getting offtopic now. Maybe we should continue on irc...
sure.

_______________________________________________
arm-netbook mailing list arm-***@lists.phcomp.co.uk
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
Send large attachments to arm-n
Xavi Drudis Ferran
2016-08-10 15:02:29 UTC
Permalink
Post by Andreas Baierl
But getting offtopic now. Maybe we should continue on irc...
sure.
Continue wherever you want. But I don't think it is off-topic here.

Not sure about the list orginal intent, but these are issues that come
up when you take certain arm SOCs and use them for a netbook, and look
for free software support for the hardware so I'd say it is on
topic. I assume most people here are interested in how to solve these
issues, or what the status is while they are solved, or what kind of
issues to expect in future projects...

At least for me it's been a very instructive discussion to read so
far. IRC might be faster or more confortable for you, so fell free to
use it but my vote is for welcoming you to talk about this here and
thank you for teaching us. You don't get this kind of knowledge when
you buy a wintel laptop.


_______________________________________________
arm-netbook mailing list arm-***@lists.phcomp.co.uk
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
Send lar
Luke Kenneth Casson Leighton
2016-08-10 15:36:19 UTC
Permalink
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
Post by Xavi Drudis Ferran
Post by Andreas Baierl
But getting offtopic now. Maybe we should continue on irc...
sure.
Continue wherever you want. But I don't think it is off-topic here.
Not sure about the list orginal intent, but these are issues that come
up when you take certain arm SOCs and use them for a netbook, and look
for free software support for the hardware so I'd say it is on
topic. I assume most people here are interested in how to solve these
issues, or what the status is while they are solved, or what kind of
issues to expect in future projects...
At least for me it's been a very instructive discussion to read so
far. IRC might be faster or more confortable for you, so fell free to
use it but my vote is for welcoming you to talk about this here and
thank you for teaching us. You don't get this kind of knowledge when
you buy a wintel laptop.
good point. subject line even seems relevant which is amazing.

_______________________________________________
arm-netbook mailing list arm-***@lists.phcomp.co.uk
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
Send large attachments to arm-***@f

Luke Kenneth Casson Leighton
2016-08-10 01:39:49 UTC
Permalink
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68


On Wed, Aug 10, 2016 at 2:35 AM, Alexander .S.T. Ross
for firefox there is the addon Video Assistant that will auto play video
from embedded & video sites in your chosen system video player.
ah ha! ok. i'll give it a shot.

_______________________________________________
arm-netbook mailing list arm-***@lists.phcomp.co.uk
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
Send
Lauri Kasanen
2016-08-10 08:09:23 UTC
Permalink
On Tue, 9 Aug 2016 22:36:05 +0100
Post by Luke Kenneth Casson Leighton
.... ain't gonna work with the current code-base of either xulrunner
(firefox and ice* types) or webkit (chrome, midori, qupzilla). darn.
the reason for webkit is: although gstreamer supports vdpau and vaapi,
and even can be activated to do hardware-accelerated decode, webkit
(at least as best i can find) the gtk+ variant then targets *opengl*
as the output rendering engine, rather than asks VAAPI / VDPAU to
handle it directly on the webkit engine's behalf.
Background: the website may need to do transformations on the data,
read the data, or overlay several dozen layers of ads on top of the
video, each with different clicky reactions.

Thus, a web browser outputting to a hw video layer would be
non-compliant, as it would in most cases prevent those things (and
especially for the ads, we can't have that, can we?).

So, you can't expect it to be mainlined in any popular browser, even if
you develop such. It would break some sites, and block ads on others.

(plug: Fifth, my browser, is happily non-compliant, as it replaces all
HTML5 video blocks with "download" and "stream" buttons, the stream
button calling whatever video player you want. Similar to the FF
extension mentioned ITT)

- Lauri

_______________________________________________
arm-netbook mailing list arm-***@lists.phcomp.co.uk
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
Send larg
Xavi Drudis Ferran
2016-08-10 08:14:32 UTC
Permalink
for firefox there is the addon Video Assistant that will auto play video
from embedded & video sites in your chosen system video player. I find
mpv works well for this because it has a very quick load time and a
basic, minimal window that i then put on-top and have the video playing
in the corner of my screen while i do other things.
And maybe go to about:config in firefox and disable mediasource with

media.mediasource.enabled=false

mediasource (MSE) is a modern WebAPI used by vimeo, youtube and some
popular video websites to throtle streams (or download stream chunks
at different moments and resolutions/quality depending on instant
network bandwith available) from javascript in the client. Besides
cross-origin problems (I guess it's the implementation fault) and
possible LibreJS issues, Video Asssitant can't get an URL to give the
video player if you have mediasource enabled and go to websites that
use mediasource. At least for now those sites degrade to http(s) urls
if mediasource is unavailable in the browser.


Well, I just tried some examples, maybe it's not completely true
generally, but it seems to work like this in my short experience.

_______________________________________________
arm-netbook mailing list arm-***@lists.phcomp.co.uk
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
Send large attac
Tzafrir Cohen
2016-08-10 09:07:53 UTC
Permalink
Post by Xavi Drudis Ferran
And maybe go to about:config in firefox and disable mediasource with
media.mediasource.enabled=false
If that turns out to be required: it is rather simple to create a
package (at least on Debian) of a xul extension that merely sets
several config directives. They will be applied the next time iceweasel
/ firefox is started.

I'm not sure how this works with the need for signing addons in recent
versions of firefox, though.
--
Tzafrir Cohen | ***@jabber.org | VIM is
http://tzafrir.org.il | | a Mutt's
***@cohens.org.il | | best
***@debian.org | | friend

_______________________________________________
arm-netbook mailing list arm-***@lists.phcomp.co.uk
http://lists.phcomp.co.uk/mailman/listinfo/arm-netbook
Send large attachments to arm-net
Loading...