I believe this is the side-effect of shortening the AESpeed setting for day mode. Just be mindful of ALL settings when switching between modes. The purple hue can be misinterpreted as a faulty cut filter -- which was not the case here.
BTW, the IMX327 seems to be performing quite well in an extreme environment. We've already lost a light meter sensor, but the camera module seems to be operating just fine. 
Okay, maybe this was a false alarm. Seems it corrected itself after a capture session and I2C calls were made to set exposure. Very strange tho. This can be closed -- yet still curious about the hue. Will continue to monitor.
Now that I think about it, the camera may be in color mode and the IR cut filter is stuck in open. Seems our IR cut filter has died?
Our remote camera trap system (IMX327) started returning purple hued images. This is when the camera is in day mode with shots taken during the daytime (no IR lighting turned on). This just started happening. It was working for several weeks before (system uptime 43 days).
Thoughts? Image attached.
For our system, we utilize 940nm IR lighting at night that is triggered by motion. I noticed that there is a 1-3 second period for the camera to adjust each time if we start streaming when the lighting is activated. The first few images are washed out until AE kicks in. What's the recommended approach to eliminate this latency and the initial washout? Should we explore manual exposure and gain? Thx!
@newleaf BTW, the I2C features are nice and work quite well. Was able to port the code over to Go.
@veye_xumm We haven’t had time to try JP 5. It is definitely related to exactly 16348 camera invocations. That is, starting and stoping streaming. We actually took a different approach for the 1-2 second rapid timelapse images and utilized the dropped frames and capture modes via I2C and Go. It effectively delays the issue a bit as we only need to invoke streaming when a motion event is detected via a radar sensor. Regardless, it’s still a mystery and reproducible on JP 4.6. I was going to try another camera, like a Pi HQ one to see if it’s firmware or OS related. Any suggestions or thoughts?
@newleaf Adding an update here. Upon repeated testing, it was discovered that the issue described above happens after about 16348 still image captures (individual camera capture invocations). After that time, the camera becomes unresponsive until after a reboot.
Being that this number is close to 2^14 (16384), is there a register or counter that is overflowing/wrapping? Perhaps a bug in the driver or camera firmware?
After speaking with out of our engineers, he noted that that the NULL pointer dereference at virtual address 00000000 could simply be the side effect of an internal issue.
We will be exploring an upgrade to JetPack 5 and continue testing.
Thank you!
@newleaf After some high-level research, this may require an update to JetPack 5. Seeing some issues reported about vi-output misbehaving.
We've been doing some stress testing with timelapse captures and inference. So far the Xavier+327S has been a great combo (just ordered 2 more 327S cams too).
Running into an issue, which could be resource related, but just seeing if anyone else has run into it.
We ran a stress test capturing still images in a loop with a 250ms delay. Each image is fed to an inference pipeline (TensorRT). Everything works great and our services appear free of memory leaks. After about 3 hours (approximately 21K captures), the video device becomes unusable and our service freezes up. It requires a reboot for the device to work again.
dmesg reports:
[Fri Apr 5 01:23:41 2024] tegra194-vi5 15c10000.vi: dma_alloc_coherent failed
[Fri Apr 5 01:23:41 2024] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[Fri Apr 5 01:23:41 2024] Mem abort info:
[Fri Apr 5 01:23:41 2024] ESR = 0x96000046
[Fri Apr 5 01:23:41 2024] Exception class = DABT (current EL), IL = 32 bits
[Fri Apr 5 01:23:41 2024] SET = 0, FnV = 0
[Fri Apr 5 01:23:41 2024] EA = 0, S1PTW = 0
[Fri Apr 5 01:23:41 2024] Data abort info:
[Fri Apr 5 01:23:41 2024] ISV = 0, ISS = 0x00000046
[Fri Apr 5 01:23:41 2024] CM = 0, WnR = 1
[Fri Apr 5 01:23:41 2024] user pgtable: 4k pages, 39-bit VAs, pgd = 00000000ca31e3b8
[Fri Apr 5 01:23:41 2024] [0000000000000000] *pgd=000000025b1d1003, *pud=000000025b1d1003, *pmd=0000000000000000
[Fri Apr 5 01:23:41 2024] Internal error: Oops: 96000046 [#1] PREEMPT SMP
[Fri Apr 5 01:23:41 2024] Modules linked in: fuse xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter zram cdc_acm overlay userspace_alert binfmt_misc veyecam nvgpu ip_tables x_tables
[Fri Apr 5 01:23:41 2024] CPU: 3 PID: 8996 Comm: vi-output, veye Not tainted 4.9.337-tegra #1
[Fri Apr 5 01:23:41 2024] Hardware name: NVIDIA Jetson Xavier NX Developer Kit (DT)
[Fri Apr 5 01:23:41 2024] task: 000000006596177f task.stack: 000000004c535bec
[Fri Apr 5 01:23:41 2024] PC is at __memcpy+0x110/0x180
[Fri Apr 5 01:23:41 2024] LR is at tegra_channel_kthread_capture_enqueue+0x14c/0x498
[Fri Apr 5 01:23:41 2024] pc : [<ffffff800845fed0>] lr : [<ffffff8008b3c6a4>] pstate: 20c00045
[Fri Apr 5 01:23:41 2024] sp : ffffffc0bb0abd40
[Fri Apr 5 01:23:41 2024] x29: ffffffc0bb0abd40 x28: ffffffc1f4db6018
[Fri Apr 5 01:23:41 2024] x27: 0000000000000000 x26: 0000000000000f00
[Fri Apr 5 01:23:41 2024] x25: ffffff80301f0000 x24: ffffffc0bb0abdf8
[Fri Apr 5 01:23:41 2024] x23: ffffffc1f4db6934 x22: ffffffc1b5af6000
[Fri Apr 5 01:23:41 2024] x21: 0000000000000001 x20: ffffffc1f4db69f8
[Fri Apr 5 01:23:41 2024] x19: 0000000000000000 x18: 0000000000000400
[Fri Apr 5 01:23:41 2024] x17: 0000007f9382d4b0 x16: 0000000000000001
[Fri Apr 5 01:23:41 2024] x15: 0000000000000219 x14: 0000000000000000
[Fri Apr 5 01:23:41 2024] x13: 0000000000000000 x12: 0000000000000000
[Fri Apr 5 01:23:41 2024] x11: 0000000000000000 x10: 0000000000000000
[Fri Apr 5 01:23:41 2024] x9 : 0000000000000000 x8 : 0000000000000000
[Fri Apr 5 01:23:41 2024] x7 : 0000000300000000 x6 : 0000000000000000
[Fri Apr 5 01:23:41 2024] x5 : 00000004bfc00000 x4 : 0000000000000000
[Fri Apr 5 01:23:41 2024] x3 : 0000000000000000 x2 : 0000000000000240
[Fri Apr 5 01:23:41 2024] x1 : ffffff8009092980 x0 : 0000000000000000
[Fri Apr 5 01:23:41 2024] Process vi-output, veye (pid: 8996, stack limit = 0x000000004c535bec)
[Fri Apr 5 01:23:41 2024] Call trace:
[Fri Apr 5 01:23:41 2024] [<000000004dcbfb33>] __memcpy+0x110/0x180
[Fri Apr 5 01:23:41 2024] [<00000000cc923fd9>] kthread+0xec/0xf0
[Fri Apr 5 01:23:41 2024] [<000000002de8d5bf>] ret_from_fork+0x10/0x30
[Fri Apr 5 01:23:41 2024] ---[ end trace 6389bf17dc0e47e6 ]---
Could this be an issue with the VEYE driver?
Thank you!
Just wanted to provide a followup on this. Reached out to Seeed and they said they would order a camera and investigate further.
I also had some success on the Nano by reducing the cable length between the Nano and the camera (using a CSI-HDMI adapter from Arducam). Worked sometimes on first boot. Then stopped.
Regardless, we are nearing a deadline and decided to switch to the Xavier for a prototype. The Nano has been challenging due to reduced support from NVIDIA.
I believe you are right about device tree. Something different from first boot compared to reboot. Even with the longer cable, it was 100% consistent with this strange behavior.
I have successfully built and flashed a custom kernel image for the Nano to use the 327S.
I used the instructions at https://wiki.veye.cc/index.php/VEYE_CS_Camera_source_for_Jetson and the VEYE-MIPI-CAM2M source.
Everything works. I can stream video, capture images (event from Go code), and control the cut filter using Go's periph library.
Observation and question:
With the Nano, I am experiencing an odd behavior where the video0 device is not available after a cold boot (where the device is off for some period of time and powered up). From dmesg, I see
[ 3.469294] veyecam 7-003b: probing v4l2 sensor
[ 3.469545] veyecam 7-003b: tegracam sensor driver:veyecam_v2.0.6
[ 3.480088] tegra-vii2c 546c0000.i2c: no acknowledge from address 0x3b
[ 3.488825] veyecam 7-003b: probe failed
[ 3.494165] veyecam 7-003b: board setup failed
[ 3.502582] veyecam 8-003b: probing v4l2 sensor
[ 3.502824] veyecam 8-003b: tegracam sensor driver:veyecam_v2.0.6
[ 3.513797] Mass Storage Function, version: 2009/09/11
[ 3.513803] LUN: removable file: (no medium)
[ 3.528349] using random self ethernet address
[ 3.535722] tegra-vii2c 546c0000.i2c: no acknowledge from address 0x3b
[ 3.536048] veyecam 8-003b: probe failed
[ 3.536051] veyecam 8-003b: board setup failed
If I reboot the device, then everything works. This is consistent. I must do a reboot after a cold boot in order for the camera to be seen.
Also, I am unable to access an I2C device that is plugged into the board. The I2C cut works fine with bus 8. But trying a device on another bus fails. I did successfully apply the USB bus firmware patch from Seeed if that has any impact.
i2cdetect -l
i2c-3 i2c 7000c700.i2c I2C adapter
i2c-1 i2c 7000c400.i2c I2C adapter
i2c-101 i2c tegradc.1 I2C adapter
i2c-8 i2c i2c-6-mux (chan_id 1) I2C adapter
i2c-6 i2c Tegra I2C adapter I2C adapter
i2c-4 i2c 7000d000.i2c I2C adapter
i2c-2 i2c 7000c500.i2c I2C adapter
i2c-0 i2c 7000c000.i2c I2C adapter
i2c-7 i2c i2c-6-mux (chan_id 0) I2C adapter
i2c-5 i2c 7000d100.i2c I2C adapter
Any thoughts on why a reboot is needed? As far as the not seeing other bus devices, it may be unrelated.
Thx!
The main VEYE website from the US is not loading. Some subpages seem to work fine.
Getting ERR_TOO_MANY_REDIRECTS on Firefox, Chrome, Brave and Safari.
@veye_xumm Thank you. And agreed, it's heat transfer. Everything is stable and works great. Good to know it's normal.
@newleaf Actually, this could be residual heat from when the sensor was streaming some video. I had video off for 30 minutes and its cooled down a bit. I was a little surprise how much the IR-cut filter heated up however. I will keep testing.
Currently testing the IMX327S with a reComputer Jetson Xavier. Everything works perfectly, including the I2C tool to control the IR-cut filter. However, I am noticing that that the IR-cut filter device seems to generate a good amount of heat. To the point you can't touch it for too long. I am using the recommended wiring as documented on the Wiki, where 5V is also provided to the module in addition to the CSI connection. Is this normal? Thx!
Sorry for the late reply.
Applied the USB bus firmware update from Seeed and the camera WORKS!!!
Thank you for the help. The 327S module is very nice BTW. Also got it to work on the Pi.
@veye_xumm For anyone else. This works perfectly. Note that when recompiling on 64-bit Pi, the following warning is produced. It still builds. Just copy the outputs to bin and the scripts work fine.
./make.sh
i2c_read.c: In function ‘main’:
i2c_read.c:112:35: warning: passing argument 4 of ‘i2c_rd’ from incompatible pointer type [-Wincompatible-pointer-types]
112 | i2c_rd(fd, device_addr,reg_addr, &value, num);
| ^~~~~~
| |
| unsigned char (*)[512]
i2c_read.c:20:68: note: expected ‘unsigned char *’ but argument is of type ‘unsigned char (*)[512]’
20 | static int i2c_rd(int fd, uint8_t i2c_addr, uint16_t reg, uint8_t *values, uint32_t n)
@veye_xumm This seems to align with what I saw on one of their forums. Thx for the firmware update filename, I can reference it in an email to them.
@veye_xumm Below are the results with the camera connected as shown.
ls /proc/device-tree/cam_i2cmux/i2c@*
/proc/device-tree/cam_i2cmux/i2c@0:
'#address-cells' linux,phandle name phandle rbpcv2_veyecam_a@3b reg '#size-cells'
/proc/device-tree/cam_i2cmux/i2c@1:
'#address-cells' linux,phandle name phandle rbpcv2_veyecam_c@3b reg '#size-cells'
ls /sys/bus/i2c/drivers/
bq27xxx-battery dummy ina230x lp855x nct1008_nct72 pca954x rt5640 rtc-rx8025 stepper_pca tegra_edid tps65132 usb3503
cs53l30 ina219x ina3221x max77620 pca953x 'PEX9749 thermal sensor' rt5659 sgtl5000 tas2552 tmpm32xi2c ucsi_ccg veyecam
dmesg | grep veye
[ 6.214128] veyecam 9-003b: probing v4l2 sensor
[ 6.214385] veyecam 9-003b: tegracam sensor driver:veyecam_v2.0.6
[ 16.243950] veyecam 9-003b: probe failed
[ 16.244034] veyecam 9-003b: board setup failed
[ 16.244245] veyecam 10-003b: probing v4l2 sensor
[ 16.244499] veyecam 10-003b: tegracam sensor driver:veyecam_v2.0.6
[ 26.481745] veyecam 10-003b: probe failed
[ 26.481863] veyecam 10-003b: board setup failed
Thank you!