1150923 - [Crash] [@ mozilla::MediaStreamGraphImpl::AddBlockingRelatedStreamsToSet ]

Status: RESOLVED WORKSFORME

(Firefox OS Graveyard :: Stability, defect)

Description

[Nicholas Troast [:ntroast]]

We observed the following crash signature during testing.

[@ mozilla::MediaStreamGraphImpl::AddBlockingRelatedStreamsToSet | mozilla::MediaStreamGraphImpl::AddBlockingRelatedStreamsToSet | mozilla::MediaStreamGraphImpl::AddBlockingRelatedStreamsToSet | mozilla::MediaStreamGraphImpl::RecomputeBlocking ]

Cafbot will upload the decoded minidump and extra file.

Comment 1

[cafbot (PoC: ggrisco)]

Attachment: EXTRA file attachment -

Comment 2

[cafbot (PoC: ggrisco)]

Attachment: decoded minidump -

Comment 3

[Mike Lee [:mlee]]

Johnny & Andrew,

Can you have someone(s) on the DOM team familiar with the media codebase look into this critical fxOS 2.2 crash asap. It's currently blocking CAF and is directly contributing to their inability to reach our MTBF goal.

Thanks,
Mike

Comment 4

[cafbot (PoC: ggrisco)]

Observed on: 

Device: msm8909
Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.119
Moz BuildID: 20150330002503
Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.119.xml?h=release
Gecko Version: 37.0
Gaia:  http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=473cd63f53c855299b719285d9b95e3f2910782f
Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=e44e28d2a37258fe0c908c59f2e91e5287777b40
Patches: bug 1133398, bug 1143694, bug 1146987, bug 1145724, bug 1147646, bug 1133147, bug 1150271, bug 1142770

Comment 5

[cafbot (PoC: ggrisco)]

Attachment: EXTRA file attachment - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.119

Comment 6

[cafbot (PoC: ggrisco)]

Attachment: decoded minidump - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.119

Comment 7

[StevenLee[:slee]]

Hi roc, 

Could you please check this problem? I checked out b2g-2.2. It crashes at the first line of MediaStreamGraphImpl::AddBlockingRelatedStreamsToSet, "if(aStream->mInBlockingSet)". It looks like aStream has beeb deleted.

Comment 8

[StevenLee[:slee]]

Bug 1113927 crashes at the same place.

Comment 9

[StevenLee[:slee]]

ni padenot, too.

Comment 10

[Paul Adenot (:padenot)]

This also looks like an UAF (like 1150918). Having steps to reproduce would make this actionable.

From the stacks, we can see that this is a video-only application (there is no audio flowing in the graph), but it's not clear to me what it is?

Comment 11

[Inder]

This crash was produced during stability tests which involves monkey testing for several hours and there is no clear STR for this. If we are not able to identify the issue using provided logs then please feel free to provide us a debug patch with additional logging to identify the issue.

Comment 12

[Robert O'Callahan (:roc) (email my personal email if necessary)]

(In reply to StevenLee[:slee] from comment #8)
> Bug 1113927 crashes at the same place.

Unfortunately we only have two crashes reported there, both in December last year.

I've tried running dom/media/tests many times locally on desktop Linux, with chaos mode and --shuffle, with no success.

(In reply to StevenLee[:slee] from comment #7)
> Could you please check this problem? I checked out b2g-2.2. It crashes at
> the first line of MediaStreamGraphImpl::AddBlockingRelatedStreamsToSet,
> "if(aStream->mInBlockingSet)". It looks like aStream has beeb deleted.

Can you explain the reasoning that lead to that conclusion? Can you tell from the disassembly and the crash dump that aStream is a non-null pointer to unmapped memory?

Is this crash occurring frequently enough that we can add some logging to get some additional information?

Comment 13

[Robert O'Callahan (:roc) (email my personal email if necessary)]

(In reply to Inder from comment #11)
> This crash was produced during stability tests which involves monkey testing
> for several hours and there is no clear STR for this.

You mean tests that drive the interface of a phone? So Gaia application code involved in triggering this?

Comment 14

[StevenLee[:slee]]

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #12)
> (In reply to StevenLee[:slee] from comment #7)
> > Could you please check this problem? I checked out b2g-2.2. It crashes at
> > the first line of MediaStreamGraphImpl::AddBlockingRelatedStreamsToSet,
> > "if(aStream->mInBlockingSet)". It looks like aStream has beeb deleted.
> Can you explain the reasoning that lead to that conclusion? Can you tell
> from the disassembly and the crash dump that aStream is a non-null pointer
> to unmapped memory?

I was guessing. From the crash dump below, it crashed at [1]. I assumed r1 and r2 are the arguments, nsTArray<MediaStream*>* aStreams and MediaStream* aStream. The offset from r2 to crash address, 0x2d6f665a, is 0xF6. I checked the class definition if MediaStream and it looks like the address should be located in the object. So that I think the memory address pointed by aStream should be deleted in other places. 

Crash address: 0x2d6f665a
 0  libxul.so!mozilla::MediaStreamGraphImpl::AddBlockingRelatedStreamsToSet [MediaStreamGraph.cpp : 793 + 0x0]
     r0 = 0xb1f01600    r1 = 0xaef8fb60    r2 = 0x2d6f6564    r3 = 0xb1f5b920
 1  libxul.so!mozilla::MediaStreamGraphImpl::AddBlockingRelatedStreamsToSet [MediaStreamGraph.cpp : 800 + 0x9]
     r0 = 0xb1f01600    r1 = 0xaef8fb60    r2 = 0x2d6f6564    r4 = 0x0000008e

[1] http://git.mozilla.org/?p=integration/gecko-dev.git;a=blob;f=dom/media/MediaStreamGraph.cpp;h=4a50ea3185be2f8f01ea44d2e52444e119c2e11a;hb=643f37b68305d90ecfa148f81db63891c30e4620#l793

> 
> Is this crash occurring frequently enough that we can add some logging to
> get some additional information?
Hi Inder,
Can you tell us the frequency of the problem happening?

Comment 15

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Attachment: log MediaStream destruction

This patch will log the addresses of destroyed MediaStreams. It should help us tell if a MediaStream is being referenced after it has been destroyed.

Comment 16

[Kai-Chih Hu [:khu]]

Steven, can you help to find someone to do the testing based on the patch provided by :roc and provide the logs again? Thanks.

Comment 17

[Kai-Chih Hu [:khu]]

Nicholas, since you're the reporter, can you also test it with the patch from :roc? 
Thanks.

Comment 18

[StevenLee[:slee]]

Hi Kevin,

Since we have no any information about how this bug happens, we may need Nicholas to test this.

Comment 19

[Nicholas Troast [:ntroast]]

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #13)
> (In reply to Inder from comment #11)
> > This crash was produced during stability tests which involves monkey testing
> > for several hours and there is no clear STR for this.
> 
> You mean tests that drive the interface of a phone? So Gaia application code
> involved in triggering this?

Testing is done for several hours with monkey testing so no assumptions can be made about how this crash was triggered. For all we know the crash could happen if we just leave the phone on for several hours with no UI interaction. All we know is that there was a crash, and we have collected as much as we could at the time of the crash.

I have added the logging patch. It will be included in either the very next AU or the one after. Thanks!

Comment 20

[Nicholas Troast [:ntroast]]

(In reply to StevenLee[:slee] from comment #14)
> (In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #12)
> > (In reply to StevenLee[:slee] from comment #7)
> > > Could you please check this problem? I checked out b2g-2.2. It crashes at
> > > the first line of MediaStreamGraphImpl::AddBlockingRelatedStreamsToSet,
> > > "if(aStream->mInBlockingSet)". It looks like aStream has beeb deleted.
> > Can you explain the reasoning that lead to that conclusion? Can you tell
> > from the disassembly and the crash dump that aStream is a non-null pointer
> > to unmapped memory?
> 
> I was guessing. From the crash dump below, it crashed at [1]. I assumed r1
> and r2 are the arguments, nsTArray<MediaStream*>* aStreams and MediaStream*
> aStream. The offset from r2 to crash address, 0x2d6f665a, is 0xF6. I checked
> the class definition if MediaStream and it looks like the address should be
> located in the object. So that I think the memory address pointed by aStream
> should be deleted in other places. 
> 
> Crash address: 0x2d6f665a
>  0  libxul.so!mozilla::MediaStreamGraphImpl::AddBlockingRelatedStreamsToSet
> [MediaStreamGraph.cpp : 793 + 0x0]
>      r0 = 0xb1f01600    r1 = 0xaef8fb60    r2 = 0x2d6f6564    r3 = 0xb1f5b920
>  1  libxul.so!mozilla::MediaStreamGraphImpl::AddBlockingRelatedStreamsToSet
> [MediaStreamGraph.cpp : 800 + 0x9]
>      r0 = 0xb1f01600    r1 = 0xaef8fb60    r2 = 0x2d6f6564    r4 = 0x0000008e
> 
> [1]
> http://git.mozilla.org/?p=integration/gecko-dev.git;a=blob;f=dom/media/
> MediaStreamGraph.cpp;h=4a50ea3185be2f8f01ea44d2e52444e119c2e11a;
> hb=643f37b68305d90ecfa148f81db63891c30e4620#l793
> 
> > 
> > Is this crash occurring frequently enough that we can add some logging to
> > get some additional information?
> Hi Inder,
> Can you tell us the frequency of the problem happening?

We saw this crash twice in a single run on AU 119

Comment 21

[Robert O'Callahan (:roc) (email my personal email if necessary)]

(In reply to Nicholas Troast [:ntroast] from comment #19)
> Testing is done for several hours with monkey testing so no assumptions can
> be made about how this crash was triggered. For all we know the crash could
> happen if we just leave the phone on for several hours with no UI
> interaction. All we know is that there was a crash, and we have collected as
> much as we could at the time of the crash.

Right, but just to be clear: this testing doesn't run specific Mozilla test code, e.g. our mochitests?

Comment 22

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Also, do these builds have #ifdef DEBUG enabled (e.g. NS_ASSERTIONs)?

Comment 23

[Nicholas Troast [:ntroast]]

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #21)
> (In reply to Nicholas Troast [:ntroast] from comment #19)
> > Testing is done for several hours with monkey testing so no assumptions can
> > be made about how this crash was triggered. For all we know the crash could
> > happen if we just leave the phone on for several hours with no UI
> > interaction. All we know is that there was a crash, and we have collected as
> > much as we could at the time of the crash.
> 
> Right, but just to be clear: this testing doesn't run specific Mozilla test
> code, e.g. our mochitests?

Right, this testing does not run specific Mozilla test code

Comment 24

[Nicholas Troast [:ntroast]]

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #22)
> Also, do these builds have #ifdef DEBUG enabled (e.g. NS_ASSERTIONs)?

These builds do not have DEBUG enabled

Comment 25

[Nicholas Troast [:ntroast]]

I removed attachment 8589477 [details] [diff] [review] from our build due to a conflict with the logging patch from bug 1152439. I believe the patch from bug 1152439 supersedes this one.

Comment 26

[cafbot (PoC: ggrisco)]

"Closing issue which has not been seen since 04/01/15 18:39"

Comment 27

[Pi Wei Cheng [:piwei] (inactive)]

See comment 26.