Navigation

    VEYE IMAGING Forum

    • Register
    • Login
    • Search
    • Categories
    • Tags
    • Recent
    • Popular
    • Users
    • WIKI
    • veye.cc

    SOLVED Get high framerate V4L2 in python/opencv?

    CS MIPI camera
    2
    9
    2097
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • T
      tj last edited by

      I'm trying to get 720p@50fps into python with a IMX307 with the V4L2 driver installed. Somehow, I can't get above about 43fps, with the following code:

      from time import time
      import cv2
      
      cap = cv2.VideoCapture(0, cv2.CAP_V4L2)
      
      t0 = time()
      
      for i in range(256):
          _, frame = cap.read()
      
      t1 = time()
      
      print("FPS:", 256/(t1-t0))
      print("Frame shape:", frame.shape)
      

      This outputs:

      FPS: 44.038153429151514
      Frame shape: (720, 1280, 3)
      

      (To get the right width/height/fps in opencv, I hardcoded them into OpenCV as suggested here.)

      I'm running on a Raspberry Pi 4 with 4GB RAM. Now my question: how can I get more than 44 fps into python? Should I throw opencv out and look at pure python implementations (1, 2)? Why is this so hard?

      T 1 Reply Last reply Reply Quote 0
      • T
        tj @tj last edited by

        Expanding a bit, I tried the gstreamer-command, and this may not be a python problem:

        gst-launch-1.0 v4l2src device=/dev/video0 ! "video/x-raw,format=(string)UYVY, width=(int)1280, height=(int)720,framerate=(fraction)50/1" ! videoconvert ! fpsdisplaysink video-sink=fakesink -v
        /GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 495, dropped: 0, current: 41.52, average: 48.30
        /GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 520, dropped: 0, current: 49.69, average: 48.37
        /GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 520, dropped: 0, current: 49.69, average: 48.37
        /GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 546, dropped: 0, current: 49.99, average: 48.44
        /GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 546, dropped: 0, current: 49.99, average: 48.44
        /GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 572, dropped: 0, current: 50.33, average: 48.53
        /GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 572, dropped: 0, current: 50.33, average: 48.53
        /GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 597, dropped: 0, current: 49.92, average: 48.58
        /GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 597, dropped: 0, current: 49.92, average: 48.58
        /GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 618, dropped: 0, current: 41.27, average: 48.29
        /GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 618, dropped: 0, current: 41.27, average: 48.29
        /GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 643, dropped: 0, current: 49.78, average: 48.35
        /GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 643, dropped: 0, current: 49.78, average: 48.35
        /GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 669, dropped: 0, current: 50.22, average: 48.42
        

        Is this a hardware problem? Or a driver problem?

        T 1 Reply Last reply Reply Quote 0
        • T
          tj @tj last edited by

          I'm diving into this a bit more. Thanks to Yevhenii I'm now using the gstreamer backend to opencv.

          I've tested the time it takes to capture individual frames, with Now at 60fps, so a single frame should take 16.7ms. This works out for most frames, but there's a strange pattern of frames that take a lot longer:
          fps.png
          The number of frames between the frame that stalls is 153, 146, 153, 145, 153, 146, 153, 146, ... (this goes on for at least 100 times, so it's a real pattern).

          The opencv v4l2 backend gives a similar result:
          fps_v4l2.png

          I have not tried to make this plot with the D_mipicamera python library, but I suspect a very similar behavior.

          This is probably also the reason the average goes below 50 in the above questions. Does this ring a bell?

          veye_xumm 1 Reply Last reply Reply Quote 0
          • veye_xumm
            veye_xumm @tj last edited by

            @tj
            You have done very nice work.

            1. The current driver only supports 720p@60fps, but it doesn't support 720p@50ps yet. But we have plans to support flexible frame rate configuration.
            2. This is indeed a bug, that seems to lose a few frames in a while. I'll arrange to fix it.
            3. There is now a temporary solution.Configure the camera as the master of sync mode.
            ./cs_mipi_i2c.sh -w -f streammode -p1 1 -p2 0
            
            veye_xumm T 2 Replies Last reply Reply Quote 0
            • veye_xumm
              veye_xumm @veye_xumm last edited by

              @tj

              We have fixed this bug. Please upgrade the camera firmware and test.
              http://wiki.veye.cc/index.php/CS-MIPI-IMX307_version_log

              Regards!

              1 Reply Last reply Reply Quote 0
              • T
                tj @veye_xumm last edited by

                @veye_xumm Thanks for the quick reply and the workaround!

                I tried setting the camera to master and redid the experiments. It seems that the cv2.CAP_V4L2 gives a slightly more constant framerate, so I'll skip gstreamer.

                fps_master.png
                fps_master_v4l2.png

                veye_xumm 1 Reply Last reply Reply Quote 0
                • veye_xumm
                  veye_xumm @tj last edited by

                  @tj

                  We have fixed this bug. Please upgrade the camera firmware and test.
                  http://wiki.veye.cc/index.php/CS-MIPI-IMX307_version_log

                  Regards!

                  T 1 Reply Last reply Reply Quote 0
                  • T
                    tj @veye_xumm last edited by

                    @veye_xumm Thanks for the quick fix! I've installed 2.39 and tested; the regular pattern of missed frames is now gone.
                    fps_master_v4l2_239.png

                    I've also tried with 2.39 in master mode, it gives a similar pattern. The regular pattern sometimes a somewhat longer read time (this pattern appears about every 48 frames) is very acceptable. Also, missing one frame in 2000 frames isn't a big problem.

                    Thanks again! Next problem in a next topic 😉

                    veye_xumm 1 Reply Last reply Reply Quote 0
                    • veye_xumm
                      veye_xumm @tj last edited by

                      @tj You are welcome.

                      1 Reply Last reply Reply Quote 0
                      • First post
                        Last post