Shot detection issues

[cinematicity]

I had good results using the shot detect feature on one film, but now I'm trying a second film and having no luck.

I've adjusted the sensitivity value from 45 to 85 but the results are always the same: shotdetect finds shot boundaries where there are none, while it misses other, obvious ones. For example, it misses a scene transition from a nearly white background to a dark interior.

It's not even getting a single shot boundary correct in the entire clip, and so I'm beginning to suspect that it's having a problem parsing the images or there is some codec problem. The first film that I tried was in avi format, which worked fine. This one is in m4v format (I used Handbrake to pull it from a DVD).

Are there any known limitations with shotdetect? Should I try a different video file format? Any other suggestions/workarounds?

TIA,

M

Addendum: I tried converting the file to both AVI and MOV formats -- same problems.

Addendum on the addendum: to try and see if shotdetect is able to parse my video file, I decided to run it standalone. There's some problem doing this under OS X (as mentioned in my previous thread), but it works okay on Windows. After it finished, I looked in the /thumbs directory and there I found proper thumbnails for my clip, and most all of the shot boundaries seemed to have been detected as I would have expected. However, when I try the same thing in Advene on MS-Windows, once again the shot boundaries are all off.

So, my hypothesis now is that perhaps there is a problem when Advene tries to process the results from shotdetect. If it would help, I can provide my video clip (20 MB) and the result.xml file from shotdetect.

[oaubert]

> Are there any known limitations with shotdetect? Should I try a different video file format? Any other suggestions/workarounds?
The MacOSX version of shotdetect is older than the one on Windows, so the (statically linked) ffmpeg libs that provide access to the video may have trouble with some codecs.

> Addendum on the addendum: to try and see if shotdetect is able to parse my video file, I decided to run it standalone. There's some
> problem doing this under OS X (as mentioned in my previous thread), but it works okay on Windows. After it finished, I looked in the
>/thumbs directory and there I found proper thumbnails for my clip, and most all of the shot boundaries seemed to have been detected as I
> would have expected. However, when I try the same thing in Advene on MS-Windows, once again the shot boundaries are all off.
How did you validate the shot boundaries coming directly from shotdetect ? It cannot be done on the basis of the screenshots alone.
There may be conversion issues: shotdetect processes the video frame-by-frame and outputs timestamps in ms, using fps information to convert frame number to ms. Advene gets the ms information from shotdetect, and uses gstreamer to playback the movie at the specified time. This can be one of the conversion issues.

> So, my hypothesis now is that perhaps there is a problem when Advene tries to process the results from shotdetect.
There is basically no interpretation: Advene just gets the (integer) ms values from shotdetect.

> If it would help, I can provide my video clip (20 MB) and the result.xml file from shotdetect.
Sure, I can have a look. Use a downloader service like http://dl.free.fr/ and send the reference to olivier.aubert@liris.cnrs.fr

[cinematicity]

> Addendum on the addendum: to try and see if shotdetect is able to parse my video file, I decided to run it standalone. There's some
> problem doing this under OS X (as mentioned in my previous thread), but it works okay on Windows. After it finished, I looked in the
>/thumbs directory and there I found proper thumbnails for my clip, and most all of the shot boundaries seemed to have been detected as I
> would have expected. However, when I try the same thing in Advene on MS-Windows, once again the shot boundaries are all off.

How did you validate the shot boundaries coming directly from shotdetect ? It cannot be done on the basis of the screenshots alone.
There may be conversion issues: shotdetect processes the video frame-by-frame and outputs timestamps in ms, using fps information to convert frame number to ms. Advene gets the ms information from shotdetect, and uses gstreamer to playback the movie at the specified time. This can be one of the conversion issues.

> So, my hypothesis now is that perhaps there is a problem when Advene tries to process the results from shotdetect.

There is basically no interpretation: Advene just gets the (integer) ms values from shotdetect.

Well, right, I didn't validate the shot boundaries in the XML file. I was assuming they would be correct if the thumbnails looked right, but of course there could still be a problem inside shotdetect with that. Or maybe there's something funny with the frame rate of the video file(?).

I will upload the files and mail you a link.

Entretemps, if I can help with figuring out the problem with running shotdetect on OS X, just let me know what I need to do.

Thanks,

M

[cinematicity]

Since I made an overly hasty hypothesis before, I decided to look at the problem more closely.

I found two things that might be suspicious.

First, when I run the video clip through shotdetect, I get the following result for the first five shots:

Code: Select all

<shot fbegin="0" fduration="297" id="0" msbegin="0" msduration="4954">
<img height="150" size="thumb" src="/thumbs/0_in.jpg" type="in" width="355"></img>
</shot>
<shot fbegin="297" fduration="288" id="1" msbegin="4954" msduration="4804">
<img height="150" size="thumb" src="/thumbs/1_in.jpg" type="in" width="355"></img>
</shot>
<shot fbegin="585" fduration="507" id="2" msbegin="9759" msduration="8458">
<img height="150" size="thumb" src="/thumbs/2_in.jpg" type="in" width="355"></img>
</shot>
<shot fbegin="1092" fduration="964" id="3" msbegin="18218" msduration="16082">
<img height="150" size="thumb" src="/thumbs/3_in.jpg" type="in" width="355"></img>
</shot>
<shot fbegin="2056" fduration="48" id="4" msbegin="34300" msduration="800">
<img height="150" size="thumb" src="/thumbs/4_in.jpg" type="in" width="355"></img>
</shot>
...


I am assuming that the "fbegin" and "fduration" attributes measure frames, while the "ms*" attributes measure milliseconds.

In the <media> element, I find this:

Code: Select all

<fps>59.9401</fps>

Is this normal? According to the properties display for the video file, it's 23 fps.

Next, when I load the same clip into Advene, run "Detect shots" and then inspect the annotations, I get this:

Code: Select all

num=0   00:00:00.000   00:00:04.954
num=1   00:00:04.954   00:00:09.758
num=2   00:00:09.758   00:00:18.217
num=3   00:00:18.217   00:00:34.300
...

So, these numbers pretty much line up, through I notice that the start and end times are always the same. Shouldn't the start time of the annotation match the "msbegin" attribute for the shot? (This might(?) explain the off-by-one issue that I saw with a different film.)

Next, when I look at the thumbnails generated by shotdetect, they don't seem to match the msbegin values. That is, the thumbnail image "1_in.jpg" doesn't correspond to what I find in the film clip at 4.954s.

So, could there be a problem in shotdetect? Or, could it be getting confused by an incorrect frame rate?

M

[oaubert]

From the example you have sent to me, the source video file appears to use variable frame rate. And shotdetect frame number to timestamp conversion assumes a constant frame-rate. This is a shotdetect shortcoming, not easily addressable ATM.

Your best bet would be to convert the movie again using a constant frame rate.

[cinematicity]

More information:

I looked at the source code for shotdetect. In "film.cpp", it calculates the msbegin value like this:

Code: Select all

s.msbegin = int ((frame_number * 1000) / fps);


The value of fps is set in update_metadata ():

Code: Select all

this->fps = av_q2d (pFormatCtx->streams[videoStream]->[b]r_frame_rate[/b]);


Next, I checked the ffmpeg faq, and found this:

4.15 AVStream.r_frame_rate is wrong, it is much larger than the frame rate.

r_frame_rate is NOT the average frame rate, it is the smallest frame rate that can accurately represent all timestamps. So no, it is not wrong if it is larger than the average! For example, if you have mixed 25 and 30 fps content, then r_frame_rate will be 150.

I find other threads relating to ffmpeg which indicate concern about r_frame_rate, and one says "should be fixed in ffmpeg at videolan".

If you google "r_frame_rate", you'll find some suggested patch code, too:

http://libav-users.943685.n4.nabble.com/Retrieving-Frames-Per-Second-FPS-td946533.html

I am not knowledgable enough to suggest a real solution, but these leads might save some time.

HTH,

M

[cinematicity]

Your best bet would be to convert the movie again using a constant frame rate.

OK, I just tried that and now "Detect shots" produces much better results. Thanks a lot for this suggestion!

Instead of converting the DVD to an mpeg file, it would be simpler if I could just work directly from the DVD itself. But, "Associate a DVD" seems to be broken. Is that a known problem?

Thanks,

M